Health & Readiness¶

On this page you will learn the difference between the health check and the readiness probe of a Workload, and how to set them up in the app.yaml.

Available Guides¶

• Declare the Probes in app.yaml
• Choose the Check Method
• Tune the Timing
• Example: A Workload With a Slow Startup
• Make Your Application Answer the Probes

Overview¶

Two Probes, Two Questions¶

A running Workload can be asked two different questions. Each question has its own probe.

Both probes are optional, and you can use one, both or neither.

Why You Need Readiness¶

With only a liveness probe, you have to make a bad choice.

• If the probe is lenient enough to survive a slow startup, it also hides real problems, because a frozen Workload is found much later.
• If the probe is strict enough to find a frozen Workload quickly, it also restarts a healthy Workload that is only warming up.

A readiness probe removes the choice. Liveness answers "restart me", and readiness answers "do not rely on me yet".

Good to Know¶

• Readiness is optional. If you do not declare it, your Workload behaves exactly as before.
• It works wherever Workloads run with a liveness probe today, in the cloud and on the edge.
• The probes use the same mechanism as the Kubernetes liveness and readiness probes. You can read more on the Kubernetes Website.
• Startup probes are not supported. For a very slow startup, use an Initial Delay on the readiness probe.

Guides¶

Declare the Probes in app.yaml¶

Add the probes to the health_check setting in the system section of the app.yaml. This sets the default for every deployment of the Application.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

spec_version: 5.0.0

  ...

defaults:
  system:
    health_check:
      # Restart the Workload if it stops responding
      liveness_probe:
        type: http_get
        http_get:
          path: /health/live
          port: 8080
        period_seconds: 10
        failure_threshold: 3

      # Do not rely on the Workload until it reports it is ready
      readiness_probe:
        type: http_get
        http_get:
          path: /health/ready
          port: 8080
        initial_delay_seconds: 5
        period_seconds: 10
        failure_threshold: 3
        success_threshold: 1

You can change these defaults when you deploy a Workload, in the Advanced section.

Choose the Check Method¶

Each probe has a type that chooses the check method. The settings of the method are in the block with the same name as the type.

On a readiness probe, type is required and the block that matches the type must be present.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

# HTTP GET
readiness_probe:
  type: http_get
  http_get:
    path: /health/ready
    port: 8080

# TCP Socket
readiness_probe:
  type: tcp_socket
  tcp_socket:
    port: 5432

# Exec
readiness_probe:
  type: exec
  exec:
    command: ["cat", "/tmp/ready"]

Tune the Timing¶

The timing settings are on the probe itself, next to type.

The values period_seconds, timeout_seconds, failure_threshold and success_threshold must be 1 or more. initial_delay_seconds can be 0 or more.

success_threshold is only for readiness. A liveness probe is happy with one success.

Example: A Workload With a Slow Startup¶

An Application loads a machine learning model when it starts. This takes about 60 seconds. Afterwards it answers requests quickly.

With only a liveness probe

1
2
3
4
5
6
7
8

health_check:
  liveness_probe:
    type: http_get
    http_get:
      path: /health
      port: 8080
    period_seconds: 10
    failure_threshold: 3

The probe checks every 10 seconds and fails after 3 failures, so the Workload is restarted after about 30 seconds. The model needs 60 seconds, so the Workload is restarted again and again and never finishes starting.

To fix it you must make the probe lenient, for example failure_threshold: 10. Now the Workload starts, but a frozen Workload is also only found after 100 seconds.

With a liveness and a readiness probe

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

health_check:
  liveness_probe:
    type: http_get
    http_get:
      path: /health/live
      port: 8080
    period_seconds: 10
    failure_threshold: 3

  readiness_probe:
    type: http_get
    http_get:
      path: /health/ready
      port: 8080
    period_seconds: 10
    failure_threshold: 3

• /health/live answers as soon as the process is running, so the liveness probe stays strict and still finds a frozen Workload in about 30 seconds.
• /health/ready only answers after the model is loaded. Until then the Workload is marked as not ready, and it is not restarted.

Make Your Application Answer the Probes¶

The probes only work if your Application answers them.

• For http_get, the Application must serve the path on the port and answer with a successful response when the check passes.
• For tcp_socket, the Application must listen on the port.
• For exec, the command must exit with 0 when the check passes.