Uvicorn Graceful Shutdown within kubernetes ecosystem

[nayanmanojgupta]

Context

Hello, our team is currently using Uvicorn and we are experiencing issues with graceful shutdown. During a graceful shutdown, Uvicorn stops accepting new requests while finishing up any in-progress requests. However, we are dealing with a situation where our Application Load Balancer (ALB) is still sending traffic to the server for a short period of time after receiving the SIGTERM signal. Due to which, service will encounter connection refused error. In order to handle this, we need to keep the port open for a few extra seconds in order to receive these remaining requests for certain period of time.

One possible way to handle this could be to put sleep in shutdown function of Server Class.

async def shutdown(self, sockets: list[socket.socket] | None = None) -> None:
        logger.info("Shutting down")
   
        await asyncio.sleep(self.wait_time_before_accepting_new_connections)

        # Stop accepting new connections.
        for server in self.servers:
            server.close()
        for sock in sockets or []:
            sock.close()
        ...........
        .....
        ..

However, this is currently not possible to be configured.

We would appreciate insights from the community on how to handle this issue.

We are using

uvicorn = {version = ">=0.23.2", extras = ["standard"]}
fastapi = ">=0.98.0"
python = "^3.11"

[Kludex]

Is there a way to create a MRE?

[nayanmanojgupta]

We can use this code as MRE:

from fastapi import FastAPI
from contextlib import asynccontextmanager
import asyncio

@asynccontextmanager
async def lifespan(app):
    print("Before app starts taking request")
    yield
    print("After app finished taking new request")

app = FastAPI(lifespan = lifespan)

@app.get("/")
async def root():
    for i in range(1,11):
        print(i)
        await asyncio.sleep(1)
    return {"Hello": "World"}

@app.get("/health")
async def health():
    return {"status": "ok"}

if __name__ == "__main__":
    import uvicorn
    uvicorn.run(app, host="0.0.0.0", port=8000)

Let's say there is a server located in a container within a Kubernetes POD. When this POD is scheduled for deletion, a SIGTERM signal is sent to the server and uvicorn after receiving SIGTERM signal stops taking new requests. However, removing the server endpoint from all Kubernetes components is not an immediate process. It may take some time for the interaction between Kubernetes API Server, kube-proxy, and kubelet to update the endpoints.

To reproduce the issue, follow these steps:

• Install required packages.
• Start the uvicorn server.
• Send a request to the "http://localhost:8000/" route.
• The route is designed to take 10 seconds to respond. While waiting for the response, send a SIGTERM signal to the server.
• If another request is sent to the server after SIGTERM is received, uvicorn will not accept it as it has stopped taking new requests.

So if we have more than one pod for a service and we already block incoming requests in the terminating pod, we will encounter a connection refused error for this service although there is a healthy pod to handle the request, it will create bad user experience.

To handle this, we should keep processing incoming traffic for some time despite the SIGTERM signal and use solution suggested in question tab.

To get more clarity on this, you can through section "Graceful Shutdown" in this link:https://learnk8s.io/graceful-shutdown

[nayanmanojgupta]

#2242 (comment)

@Kludex, please see this comment, this will help us to solve above issue.

[nayanmanojgupta]

Any update on this @Kludex ?

[johnkoehn]

RIP - this is so annoying that uvicorn just straight up doesn't support this

[danieldevere2]

Couldn't you run the uvicorn server so that it will wait for a few seconds before acting on the shutdown command? Something like trapping the signal and then waiting before propagating it.

[nayanmanojgupta]

Handling async code is the issue here for me.

[raf-moreau]

Hi. I'm also using uvicorn in a K8S context.
What i'm doing is having 2 separates uvicorn servers in 2 different ports. First one is for regular queries. Second one is for K8S probes and Prometheus metrics.
I'm handling signals on my side (which is why i'm stuck before 0.30 for now) and when receiving a SIGTERM, i first say to probe server to fail the readyness probe, wait a few seconds, and then do a graceful shutdown of both servers.
That way, kube detect the readyness failure, kick the pod out of the service before the server shuts down ... and i'm done.

I could have done that with a single server, of course, but i didn't want to expose my metrics and probes to everyone.

Just my 2 cents

[nayanmanojgupta]

Handling SIGTERM signal by our side is great but how are you handling async code?

[aiguofer]

You can use a preStop hook to add a sleep before SIGTERM is sent to the process. This will give some time for k8s to stop sending requests to the pod before your process stops handling new requests: https://freecontent.manning.com/handling-client-requests-properly-with-kubernetes/

[nayanmanojgupta]

Yes
But here in this scenario, it is ALB which will keep sending request until readiness probe fails for consecutive 3 times.

[HenningDinero]

I'm struggling with the same issue and have forked and implemented the sleep-timer before the shutdown https://github.com/Jakobhenningjensen/uvicorn
Note, it also sets an attribute uvicorn_shutdown_triggered = True in your app i.e if you have a FastAPI app it would have uvicorn_shutdown_triggered = True when the shutdown is triggered, which you can use as a check for your ready response to the load-balancer (i.e make it return not 200 if uvicorn_shutdown_triggered = True to make the load-balancer ditch the pod while it finishes it's current requests).

I'm trying to make a PR but I can't really make a unittest for it to work. I can see that we are not the only one who are having that issue i.e if we somehow can make the PR and upvote it enough, we could maybe get it merged?

[nayanmanojgupta]

Check this #2242 (comment)
If this is taken into consideration, we are good.