Kubernetes Probes

Applications deployed on Kubernetes can provide information about their internal state with Container Probes. Depending on your Kubernetes configuration, the kubelet calls those probes and reacts to the result.

By default, Spring Boot manages your Application Availability State. If deployed in a Kubernetes environment, actuator gathers the “Liveness” and “Readiness” information from the ApplicationAvailability interface and uses that information in dedicated health indicators: LivenessStateHealthIndicator and ReadinessStateHealthIndicator. These indicators are shown on the global health endpoint ("/actuator/health"). They are also exposed as separate HTTP Probes by using health groups: "/actuator/health/liveness" and "/actuator/health/readiness".

You can then configure your Kubernetes infrastructure with the following endpoint information:

livenessProbe:
  httpGet:
    path: "/actuator/health/liveness"
    port: <actuator-port>
  failureThreshold: ...
  periodSeconds: ...

readinessProbe:
  httpGet:
    path: "/actuator/health/readiness"
    port: <actuator-port>
  failureThreshold: ...
  periodSeconds: ...
<actuator-port> should be set to the port that the actuator endpoints are available on. It could be the main web server port or a separate management port if the "management.server.port" property has been set.

These health groups are automatically enabled only if the application runs in a Kubernetes environment. You can enable them in any environment by using the management.endpoint.health.probes.enabled configuration property.

If an application takes longer to start than the configured liveness period, Kubernetes mentions the "startupProbe" as a possible solution. Generally speaking, the "startupProbe" is not necessarily needed here, as the "readinessProbe" fails until all startup tasks are done. This means your application will not receive traffic until it is ready. However, if your application takes a long time to start, consider using a "startupProbe" to make sure that Kubernetes won’t kill your application while it is in the process of starting. See the section that describes how probes behave during the application lifecycle.

If your Actuator endpoints are deployed on a separate management context, the endpoints do not use the same web infrastructure (port, connection pools, framework components) as the main application. In this case, a probe check could be successful even if the main application does not work properly (for example, it cannot accept new connections). For this reason, is it a good idea to make the liveness and readiness health groups available on the main server port. This can be done by setting the following property:

management.endpoint.health.probes.add-additional-paths=true

This would make liveness available at /livez and readiness at readyz on the main server port.

Checking External State With Kubernetes Probes

Actuator configures the “liveness” and “readiness” probes as Health Groups. This means that all the health groups features are available for them. You can, for example, configure additional Health Indicators:

  • Properties

  • YAML

management.endpoint.health.group.readiness.include=readinessState,customCheck
management:
  endpoint:
    health:
      group:
        readiness:
          include: "readinessState,customCheck"

By default, Spring Boot does not add other health indicators to these groups.

The “liveness” probe should not depend on health checks for external systems. If the liveness state of an application is broken, Kubernetes tries to solve that problem by restarting the application instance. This means that if an external system (such as a database, a Web API, or an external cache) fails, Kubernetes might restart all application instances and create cascading failures.

As for the “readiness” probe, the choice of checking external systems must be made carefully by the application developers. For this reason, Spring Boot does not include any additional health checks in the readiness probe. If the readiness state of an application instance is unready, Kubernetes does not route traffic to that instance. Some external systems might not be shared by application instances, in which case they could be included in a readiness probe. Other external systems might not be essential to the application (the application could have circuit breakers and fallbacks), in which case they definitely should not be included. Unfortunately, an external system that is shared by all application instances is common, and you have to make a judgement call: Include it in the readiness probe and expect that the application is taken out of service when the external service is down or leave it out and deal with failures higher up the stack, perhaps by using a circuit breaker in the caller.

If all instances of an application are unready, a Kubernetes Service with type=ClusterIP or NodePort does not accept any incoming connections. There is no HTTP error response (503 and so on), since there is no connection. A service with type=LoadBalancer might or might not accept connections, depending on the provider. A service that has an explicit ingress also responds in a way that depends on the implementation — the ingress service itself has to decide how to handle the “connection refused” from downstream. HTTP 503 is quite likely in the case of both load balancer and ingress.

Also, if an application uses Kubernetes autoscaling, it may react differently to applications being taken out of the load-balancer, depending on its autoscaler configuration.

Application Lifecycle and Probe States

An important aspect of the Kubernetes Probes support is its consistency with the application lifecycle. There is a significant difference between the AvailabilityState (which is the in-memory, internal state of the application) and the actual probe (which exposes that state). Depending on the phase of application lifecycle, the probe might not be available.

Spring Boot publishes application events during startup and shutdown, and probes can listen to such events and expose the AvailabilityState information.

The following tables show the AvailabilityState and the state of HTTP connectors at different stages.

When a Spring Boot application starts:

Startup phase LivenessState ReadinessState HTTP server Notes

Starting

BROKEN

REFUSING_TRAFFIC

Not started

Kubernetes checks the "liveness" Probe and restarts the application if it takes too long.

Started

CORRECT

REFUSING_TRAFFIC

Refuses requests

The application context is refreshed. The application performs startup tasks and does not receive traffic yet.

Ready

CORRECT

ACCEPTING_TRAFFIC

Accepts requests

Startup tasks are finished. The application is receiving traffic.

When a Spring Boot application shuts down:

Shutdown phase Liveness State Readiness State HTTP server Notes

Running

CORRECT

ACCEPTING_TRAFFIC

Accepts requests

Shutdown has been requested.

Graceful shutdown

CORRECT

REFUSING_TRAFFIC

New requests are rejected

If enabled, graceful shutdown processes in-flight requests.

Shutdown complete

N/A

N/A

Server is shut down

The application context is closed and the application is shut down.
See Kubernetes container lifecycle section for more information about Kubernetes deployment.