Splitting out Web App Health Check URLs for Basic and Database Checks
More specifically a bare bones health check and another separate check that connects to databases and other services you might have.
Quick Jump: Demo Video
A couple of years ago I made a video about setting up health checks. Since then I still think they’re very much worth using but I’ve changed a few things around.
Back then I was a fan of having a single health check endpoint that would connect to your database, Redis and any other services you might have that’s necessary to make your app work. That wouldn’t include external services like Stripe’s API, but perhaps a micro-service you’re calling from within your app.
With Flask, that could be a matter of setting up a view like this:
@app.get("/up")
def up():
redis.ping()
db.engine.execute("SELECT 1")
return ""
This works but it has downsides in the sense that if you’re deploying to multiple servers such as a Kubernetes cluster and you have various Kubernetes health check probes hitting this endpoint it could trigger a rolling restart of containers if you’re unable to connect to your database for a few seconds.
There’s other things that could cause issues too like maybe one of your dependent services goes down for 5 seconds or you change around your networking set up which kills the connection between 2 internal services.
In any case, a temporary disconnect between 2 services should not trigger all app containers to restart. It could lead to cascading restarts of all of your containers during network blips when technically each app may be able to function in isolation.
All of the above could likely be fixed without having to push a code change out or restart your app’s containers.
During a state where certain external services can’t be connected to you should have the flexibility to either let your apps run as they are or perhaps pull them out from your load balancer. In both cases you allow the apps themselves to continue to run since technically they’re independently healthy.
I recently refactored a bunch of example Docker web apps to now provide 2 different health check URLs. There’s examples for Flask, Rails, Django, Node and Phoenix. Now you can pick which health check to use based on the type of health check you want to perform.
The video below goes into more detail:
Demo Video
Timestamps
- 0:28 – How I had things set up a few years ago
- 1:13 – This could lead to getting stuck in an app restart loop
- 2:29 – Supplying 2 different health checks depending on what’s monitoring it
- 3:32 – Examples of how checking your DB in all types of health checks might be bad
- 5:11 – Using the different health checks for different probes
- 6:47 – All of the example Docker apps I created have been changed for this
Reference Links
How do you do health checks in your web apps? Let me know below.