Cloud Watching
A few years ago we moved away from hosting our website infrastructure from its dusty basement to the Cloud. This brought a certain peace of mind in knowing that even if the museum building’s internet connection or electricity was interrupted, the site would still stay up. As it turns out, the Cloud is also dependent on electricity and network connectivity, so while a storm in Brooklyn would leave our digital infrastructure unscathed, one in Virginia might make a dent. Since that fateful summer we’ve progressed in fine-tuning our virtual servers, databases, and content storage and distribution. Without going so far as to build Google/Facebook/Netflix-scale high-availability infrastructure and the 24/7 DevOps team that goes with it, we’ve gotten pretty far in making sure our website stays online.
As with building any infrastructure, a disaster plan should also be in place to make sure people know what’s happening when something goes wrong. Part of the alphabet soup of Amazon Web Services, Route53, is configured with the ability to automatically route web visitors away from a server having—or about to have—issues to a static placeholder page hosted in an S3 bucket based in Oregon, independent of website assets or server-side code. This is called a DNS Failover. The switch is triggered by an AWS health check which we’ve set up on our production server to check for whether a web or database server is unavailable. If that’s the case, the health check, a simple PHP page that only returns an HTTP header response, returns an HTTP 503 error, otherwise it returns an HTTP 200 OK response. The end result is a “fail whale” page that shows up when the site is going down or already there.
Aside from letting site visitors know when things are amiss, the same AWS health check triggers an email notification to our developer team, which is then picked up by their smartphones (or, in my case, a Nokia 515 which happens to have Exchange support). At the office, we’ve created a glowing 3D printed status indicator based on the 3D scan of Emma-o, King and Judge of Hell aka Yamma aka 閻魔 who we scanned for a 3D printed chess project some time ago.
Emma-Ohnoes, King and Judge of Cloud Computing uses an Arduino Yún and Temboo to connect to the same health check page that Route 53 uses. Like the DNS failover setup, it connects to the health check page every minute, however, if a 200 OK is detected, it glows blue, otherwise it pulses red using one of the Arduino’s analog inputs with pulse width modulation (PWM).
Our health check page is pretty specifically catered to just our systems, but Amazon has put together a neat guide on how to create one for your own architecture. The Arduino sketch and schematic and 3D files for Emma Ohnoes, however, can easily be adapted to any website by changing the targetUrl to either your own health check page or the website URL directly to see if it’s up or not.
Download Emma-Ohnoes’s Arduino sketch and schematics (MIT license) on Github
Start the conversation