Avoiding dropped connections in nginx containers with “STOPSIGNAL SIGQUIT”

Robin Winslow

on November 19, 2019

nginx is a very popular web server.

It may have just become the most popular web server – Netcraft’s October survey found nginx had 33% market share. This may be thanks to growth in Kubernetes adoption (most Kubernetes installations use nginx as the default ingress controller).

Unsurprisingly, its use within Kubernetes configurations is just as popular. 33 thousand public projects on GitHub use the nginx image in configs, and at least as many run it in their own Docker images.

We run nginx in the images for our 15 static websites running on our team’s Charmed Kubernetes cluster.

Closing connections in usn.ubuntu.com

About 2 years ago we noticed a problem in downloads of our USN database: Many clients were their connections reset during the download.

This may have had a few causes, but one likely culprit is Kubernetes restarting or replacing containers. This would happen when Kubernetes reschedules pods to respond to load. It will also happen every time we release a new version of the site.

The trouble with SIGTERM and nginx

When we release a new version of a site to Kubernetes, we first build and push a new image to the registry, then we ask Kubernetes to gradually roll out new containers based off the new image.

Kubernetes will gradually switch over to the new containers, removing the old ones as it goes. It does this using the same mechanism as docker stop: It will send a SIGTERM signal to the container, allow 30 seconds for it to stop gracefully, then send SIGKILL. It expects SIGTERM to result in a graceful shutdown:

It’s important that your application handle termination gracefully so that there is minimal impact on the end user and the time-to-recovery is as fast as possible!

In practice, this means your application needs to handle the SIGTERM message and begin shutting down when it receives it. This means saving all data that needs to be saved, closing down network connections, finishing any work that is left, and other similar tasks.


Unfortunately, this isn’t what happens when nginx receives a TERM signal:

The master process supports the following signals:

TERM, INT    fast shutdown
QUIT        graceful shutdown


What that “fast shutdown” means is that any open connections will be immediately closed. So if Kubernetes sends SIGTERM to my running containers, any open connection to those containers will break.

We can illustrate this using a Dockerfile which simply uses nginx to proxy to httpbin.org/delay/10. When we stop the container, we can see the container exit within less than a second, and the curl exit with “Empty reply from server”:

$ docker build -t delay-image .
Successfully tagged delay-image:latest

$ docker run --rm --name delay-ctnr --detach --publish 80:80 delay-image

$ curl -I localhost 2>&1 | egrep 'curl:|HTTP' &
[1] 14531

$ /usr/bin/time --format '%E' docker stop delay-ctnr
curl: (52) Empty reply from server

SIGQUIT to the rescue

What we need instead is to send a SIGQUIT signal to ask nginx to perform a graceful shutdown, where it will wait for open connections to close before quitting.

If we  add STOPSIGNAL SIGQUIT to the Dockerfile then we can instead see curl exit gracefully with a successful 200 response, and docker stop wait (in this case, for 8.51 seconds) until it’s done so:

$ docker build -t delay-image .
Successfully tagged delay-image:latest

$ docker run --rm --name delay-ctnr --detach --publish 80:80 delay-image

$ curl -I localhost 2>&1 | egrep 'curl:|HTTP' &
[1] 14533

$ /usr/bin/time --format '%E' docker stop delay-ctnr
HTTP/1.1 200 OK

Bear in mind that `docker kill` will only wait 10 seconds, and Kubernetes will only wait 30, before killing the container with `SIGKILL`. So if you might need longer than that to close connections then you may need to increase the grace period.

An exception to this rule is if you are relying on unix sockets in your nginx config. In this case, SIGQUIT will fail to close the sockets properly, resulting in containers potentially not restarting correctly. So if you’re using sockets, be careful with SIGQUIT.

Why isn’t this default?

I can’t find any reference to why nginx made the decision not to treat SIGTERM more gracefully, as a graceful termination seems to be the norm with SIGTERM:

The SIGTERM signal is sent to a process to request its termination. […] This allows the process to perform nice termination releasing resources and saving state if appropriate.


What is a shame is that the Dockerfile for the default nginx Docker image explicitly uses STOPSIGNAL SIGTERM, meaning that anyone using the default image (and anyone copying it) will get this connection closing issue.

They have made the decision to use SIGTERM rather than SIGQUIT because of the issue with sockets. But if you’re not using sockets, you should definitely use SIGQUIT instead.

Newsletter signup

Select topics you’re
interested in

In submitting this form, I confirm that I have read and agree to Canonical’s Privacy Notice and Privacy Policy.

Related posts

We are changing the way you build snaps from GitHub repos

On the 11th March 2020 we introduced a new process for building a snap using GitHub repos to snapcraft.io. Here is all you need to know about this update....

GNOME 3.34 snapcraft extension

We constantly strive to empower developers. Part of that aim extends to making development easier, for example improving build tools and documentation. As an...

An adventure through the Snap Store

An application store with a large number of entries is a double-edged sword. It’s often a good sign of a vibrant, thriving community of software creators,...