Graceful Shutdown | The Garnet Wiki

Graceful shutdown is the process of stopping a long-running server or service in a way that minimises data loss, client disruption, and resource contention. It matters when a service must handle ongoing requests, maintain state across restarts, and coordinate with other services during deployment, scaling, or infrastructure maintenance—especially in systems where downtime is unacceptable, such as payment processors, real-time messaging platforms, or distributed databases.

Preparing for Shutdown

To achieve graceful shutdown, the service must first signal that it is entering a shutdown phase. This involves two key actions: ceasing to accept new work and completing all in-flight operations.

The service should begin rejecting new incoming requests by updating its health status, typically via an HTTP endpoint that returns 200 OK and a payload indicating readiness. A common pattern is to expose a /health endpoint that reports status: "ready" during normal operation and status: "shutting-down" during shutdown.

The server must also register a signal handler for SIGTERM, the standard signal used to initiate shutdown in Unix environments. When a process receives SIGTERM, it begins the shutdown sequence. The signal is sent by the process supervisor (e.g., systemd, supervisord, or a container orchestration system like Kubernetes) during a rolling update, instance termination, or manual restart.


[Service]
ExecStart=/usr/bin/myapp --config /etc/myapp/config.yaml
ExecStop=/usr/bin/myapp --shutdown
Restart=on-failure
TimeoutStopSec=30

The TimeoutStopSec=30 directive ensures that the system waits up to 30 seconds for the process to exit after sending SIGTERM. If the process does not terminate within this window, SIGKILL is sent to force termination.

Coordinating In-Flight Work

While the service is shutting down, it must complete all active requests before exiting. This includes processing requests already in the request queue, handling background jobs, and flushing buffers.

The server should maintain a list of active request contexts—such as HTTP request objects, database transactions, or message consumers. Each of these should be tracked and waited upon during shutdown.

For example, in a Node.js application using Express, the server can register a SIGTERM handler that triggers a shutdown sequence:

const server = app.listen(3000);

process.on('SIGTERM', async () => {
  console.log('SIGTERM received: shutting down gracefully');

  // Stop accepting new requests
  server.close(async (err) => {
    if (err) {
      console.error('Error during server shutdown:', err);
      process.exit(1);
    }

    console.log('Server closed');

    // Wait for all database connections to flush
    await db.close();

    // Wait for all background jobs to finish
    await jobQueue.drain();

    // Close all open file handles, caches, and streams
    await cache.flush();

    console.log('Graceful shutdown complete');
    process.exit(0);
  });
});

In this example, server.close() is an asynchronous method that waits for all active HTTP connections to finish. It returns a promise that resolves when all requests have been processed and the connection pool is drained.

The key insight is that server.close() must be called *before* the process exits. If the process exits immediately after sending SIGTERM, new connections may be accepted but never processed, leading to request loss.

Handling Long-Running Tasks

Some services perform long-running operations that should not be interrupted during shutdown. These include background jobs, file processing, and periodic data syncs.

These tasks must be designed to be interruptible. The service should periodically check a global "shutdown" flag or use a cancellation token. When the shutdown is initiated, the flag is set, and all long-running tasks must respond by finishing their current work and exiting.

For instance, in a Go service using context.Context, a background worker might look like:

func startWorker(ctx context.Context) {
  for {
    select {
    case <-ctx.Done():
      // Shutdown signal received
      log.Println("Worker shutting down")
      return
    default:
      // Perform work
      task, err := fetchNextTask()
      if err != nil {
        log.Printf("Error fetching task: %v", err)
        continue
      }

      // Process task
      err = processTask(task)
      if err != nil {
        log.Printf("Error processing task: %v", err)
      }

      // Mark task as complete
      updateTaskStatus(task.ID, "completed")
    }
  }
}

The ctx.Done() channel is used to signal that the context has been cancelled—typically via context.WithCancel or context.WithTimeout. During graceful shutdown, the main process calls cancel() on the context, which propagates the shutdown signal to all workers.

If the worker is processing a task that takes 15 seconds, but the shutdown timeout is only 10 seconds, the worker will receive the signal during task processing and finish the current task, but may not start a new one. The service must account for such timing mismatches.

Managing External Dependencies

A service rarely operates in isolation. It depends on databases, message queues, caches, and external APIs. During graceful shutdown, these dependencies must be given time to acknowledge the service’s departure.

The service should close database connections in a coordinated way. For example, in a PostgreSQL application using libpq, the following sequence ensures data consistency:

Begin a transaction to update a service_status table.
Set the service’s status to shutting-down.
Flush all pending writes to disk.
Wait for all database connections to be released.
Finalize the transaction.

BEGIN;
UPDATE services SET status = 'shutting-down', last_updated = NOW()
WHERE name = 'order-processing';
COMMIT;

If the service is also a consumer of a Kafka stream, it should commit offsets before shutting down. The consumer must call commitSync() to ensure that all consumed messages are acknowledged.

# In a Kafka consumer script
kafka-console-consumer.sh --bootstrap-server kafka.example.com:9092 \
  --topic orders \
  --partition 0 \
  --offsets \
  --group my-group

The consumer should poll messages in a loop and commit offsets every N messages or every T seconds. During shutdown, it calls consumer.commitSync() and waits for the commit to complete.

If the service is using a message broker like RabbitMQ, it should declare a shutdown exchange and publish a shutdown event before closing.

{
  "event": "service-shutdown",
  "service": "payment-gateway",
  "timestamp": "2024-03-10T12:00:00Z",
  "reason": "rolling-update"
}

This event can be consumed by other services to trigger cascading shutdowns or health checks.

Detecting and Responding to Failure

Even with a well-designed shutdown, failures can occur. The most common silent failure is a service that exits too early—before all in-flight requests are processed.

This happens when the server calls server.close() but does not await the returned promise. In Node.js, calling server.close() without await results in the process exiting immediately, even though the server is still accepting requests.

Another failure mode is the difference between SIGTERM and SIGINT. While SIGTERM is the standard graceful shutdown signal, SIGINT (Ctrl+C) is often used for interactive shutdowns. A service that only listens for SIGTERM may not respond to SIGINT, leading to inconsistent behavior across environments.

A third issue arises when multiple services are shutting down simultaneously. If a service depends on another that is also shutting down, it must handle timeouts and partial availability.

For example, if a service A depends on service B, and both are shutting down, service B may stop accepting requests before service A finishes processing its final batch. Service A must be resilient to B’s unavailability during this window.

A common pattern is to use a retry loop with exponential backoff for external calls during shutdown.

import time
import requests

def call_service_b_with_retry():
    max_retries = 5
    base_delay = 1
    for attempt in range(max_retries):
        try:
            response = requests.get("http://service-b:8080/api/notify", timeout=3)
            response.raise_for_status()
            return response.json()
        except requests.exceptions.RequestException as e:
            if attempt == max_retries - 1:
                raise
            delay = base_delay * (2 ** attempt)
            time.sleep(delay)
            print(f"Attempt {attempt + 1} failed: {e}. Retrying in {delay}s...")

This ensures that the service continues to make progress even if the dependency is slow to respond.

The Complete Shutdown Sequence

The full shutdown sequence, from signal to process exit, follows a predictable path:

SIGTERM is sent to the process.
Process registers handler and begins shutdown.
Service updates health status to shutting-down.
Process calls server.close() and awaits completion.
All active requests are processed.
Background workers receive cancellation signals and finish.
External dependencies are flushed and committed.
Final data is written to disk.
Process exits with process.exit(0).

If the process exits with 1, the system assumes a failure. The most common cause is a timeout during shutdown.

# In a systemd service, a shutdown that takes 35 seconds
# but TimeoutStopSec=30
# results in SIGKILL being sent after 30 seconds
# causing the service to exit prematurely

To detect such issues, logs should include timestamps for each stage of shutdown:

2024-03-10T12:00:00Z [INFO]  SIGTERM received: shutting down gracefully
2024-03-10T12:00:01Z [INFO]  Server closing
2024-03-10T12:00:05Z [INFO]  Database connections flushed
2024-03-10T12:00:10Z [INFO]  Background jobs drained
2024-03-10T12:00:15Z [INFO]  Cache flushed
2024-03-10T12:00:16Z [INFO]  Graceful shutdown complete

A missing 16Z timestamp—indicating that the process exited immediately after Graceful shutdown complete—points to a missing await on server.close().

Summary of Key Practices

Use SIGTERM as the primary shutdown signal.
Always await server.close() in the shutdown handler.
Maintain a list of active contexts (requests, jobs, transactions).
Allow external dependencies to acknowledge the service’s departure.
Use cancellation tokens or context propagation for long-running tasks.
Commit data and update health status during shutdown.
Handle both SIGTERM and SIGINT for consistency across environments.
Log each stage of shutdown to detect timing issues.
Use exponential backoff for external calls during shutdown.

Graceful shutdown is not a feature—it is a discipline. A service that shuts down gracefully is one that respects its clients, its data, and its environment.

This page was rewritten on 18 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.