Implementing graceful shutdown in go
The purpose of a graceful shutdown is to make a go application stop receiving new requests while completing the ongoing request before finally going to shutdown. That usually happens when rolling update pods. The old pods will terminate after new pods are ready.
Usually, the pod will terminate after receiving the OS interrupt signal. To achieve graceful shutdown, we can create a context with a cancel callback and a built-in shutdown method provided by http.Server interface. We can use the created context to keep listening for OS interrupts via a channel and call the cancel function right after receiving the signal.
Here is an example of a server that implements the graceful shutdown. We will breakdown the code later.
To see how this graceful shutdown works, we can modify the handler to simulate the processing time when there is a request.
func handler() http.Handler {
mux := http.NewServeMux()
mux.Handle("/healthz", http.HandlerFunc(
func(w http.ResponseWriter, r *http.Request) {
fmt.Fprintf(w, "ok")
for i := 1; i <= 7; i++ {
log.Println(i)
time.Sleep(time.Second)
}
},
))
return mux
}Then send a request to the endpoint. While the program process the request, press Ctrl+C to send an interrupt signal. The server will stop, but it still processes the ongoing requests. On the other hand, if there is another request after the server stop, it will be refused.
Let’s breakdown the code. First, in the main function, we create a channel that will listen to os.Interrupt signal.
c := make(chan os.Signal, 1)
signal.Notify(c, os.Interrupt)Then we create a context with a cancel function. The context will block a process (in this case, preventing the server from shutdown) until we call the cancel function.
ctx, cancel := context.WithCancel(context.Background())After that, we create an anonymous function that runs in a separated goroutine, which waits for an interrupt signal then calls the cancel function.
go func() {
osCall := <-c
log.Printf("system call: %+v", osCall)
cancel()
}()We also create a &http.Server and logger instance for later use.
server := &http.Server{
Addr: ":8080",
Handler: handler(),
}
stdLog := log.New(os.Stderr, "", log.LstdFlags)Finally, pass the context, server, and logger instance to the serve function.
if err := serve(ctx, server, stdLog); err != nil {
log.Printf("serve(ctx, server, stdLog)=%+s", err)
}In the serve function, we will run the server in a separate goroutine. Then, block the process with <-ctx.Done() so the program will wait for the call of cancel function (called in the main function after receiving the interrupt signal).
go func() {
if err = server.ListenAndServe(); err != nil && err != http.ErrServerClosed {
logger.Fatalf("ListenAndServe()=%+s", err)
}
}()<-ctx.Done()
We will gracefully shutdown the server by calling the Shutdown function from the server instance after calling the cancel function. Because the function calls require a context, we also create a context with a timeout to give a time window for the server to shutdown.
ctxShutDown, cancel := context.WithTimeout(context.Background(), 30*time.Second)
defer func() {
cancel()
}()if err = server.Shutdown(ctxShutDown); err != nil {
logger.Fatalf("server.Shutdown(ctxShutdown)=%+s", err)
}
You might notice that I use Server and Logger interface. The purpose of that is to make the serve function easier to test. I will update the story soon regarding this.
That’s it! The application should finish the ongoing process and reject the upcoming request.
