Elixir, K8s and Graceful Shutdown

I am now working on a large application that use k8s (for better or worse).

We have an interesting issue in that during deployments there is a time window in which some of our services are called during the time that they are being shut down. This is a consequence of the lag between the liveness probe (or one of the probes) being called periodically. If the router does not stop sending requests to the server (we are using https|) the only way to ensure that these are neatly handled is to have a retry policy for http requests.

It does not matter how well you wire up notifications there is still the possibility of a failed call.