Skip to navigation

Health checks

A health check is a small test the load balancer sends to each backend on a schedule, for example a request for /health. A backend that fails several checks in a row is taken out of rotation, so users don’t reach a broken server. When it passes again, it is put back automatically. You don’t have to remove or re-add anything yourself.

When to use each type of check

CheckWhat passesUse it whenSet it in
HTTP (default)The backend answers a request for the Path with HTTP status 200Your app serves HTTP. This is the best choice: it shows the app can answer, not just that the port is open.Portal and API
HTTPSThe same, over TLSThe backend serves its health endpoint over HTTPSPortal and API
TCPA TCP connection to the backend port opensThe backend doesn’t speak HTTP: a database, a queue, a game serverPortal and API
PassiveThe backend keeps answering real traffic without 5xx errorsYou want a backend removed quickly when it starts returning errors, on top of an active checkAPI only

The first three are active checks: the load balancer sends its own test traffic. A passive check only watches real requests.

How it works

  1. Every Interval (default 10 seconds), the load balancer sends one check to each backend and waits up to the Timeout (default 2 seconds) for an answer.
  2. After Unhealthy threshold failures in a row (default 3), the backend is taken out of rotation. A failure is a wrong status code, a closed connection or no answer within the timeout.
  3. Checks keep running against the removed backend. After Healthy threshold passes in a row (default 2), it is put back.
  4. If no backend is healthy, requests fail until at least one backend passes its checks again.

How long it takes, with the defaults

EventTime
A backend stops answering → it is taken outAbout 30 seconds (3 failed checks, 10 seconds apart), plus up to 2 seconds for the last timeout
A backend recovers → it gets traffic againAbout 20 seconds (2 passed checks, 10 seconds apart)

To react faster, lower the interval: an Interval of 5,000 ms with an Unhealthy threshold of 2 takes a backend out after about 10 seconds. Shorter intervals send more check requests to your backends.

Before you begin

  • A health endpoint on every backend, such as /health, that returns HTTP 200 without needing a login. Return 200 only when the app can really serve requests (for example when it can reach its database).
  • The backend port open to the load balancer in each VM’s firewall group. Health checks use the same port as traffic. See Create firewall rules.
  • For a backend served over TLS: the load balancer connects to backends without encryption (the backend TLS option isn’t applied yet). Use a TCP check, or serve the health endpoint over plain HTTP on the backend port.

Test the endpoint from your computer before you create the load balancer:

curl -i http://203.0.113.10:8080/health

The first line of the answer should be HTTP/1.1 200 OK. Any other status, including a redirect such as 301, can count as a failure.

Set a health check in the portal

The portal sets the health check only when you create the load balancer.

1

Open the create form

Under Network & Security, click Load Balancer, then Create Load Balancer.

2

Find Active health check

Under Routing and health, find Active health check: “Unhealthy backends are removed from traffic rotation.”

3

Fill in the check

FieldDefaultAllowed
TypeHTTPHTTP, HTTPS, TCP
Path/healthAny path. Shown for HTTP and HTTPS only.
Interval (ms)10000100–120,000 ms
Timeout (ms)2000100–120,000 ms
Healthy threshold21–20
Unhealthy threshold31–20

Keep the Timeout shorter than the Interval.

4

Create

Finish the rest of the form and click Create Load Balancer. The Configuration summary shows your check under Health check.

Set or change a health check with the API

Send health_check when you create the load balancer, or change it later with PATCH. This request sets an HTTP check every 5 seconds and a passive check that takes a backend out after 5 errors in a row:

curl -X PATCH "https://api.ibee.ai/v1/networking/load-balancers/l7/3f9c2b7e-5d1a-4c8e-9b6f-0a2d4e6f8a1c?workspace_id=710995" \
-H "Authorization: Bearer $IBEE_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"health_check": {
"active": {
"type": "http",
"path": "/health",
"interval_ms": 5000,
"timeout_ms": 2000,
"healthy_threshold": 2,
"unhealthy_threshold": 2
},
"passive": {
"enabled": true,
"consecutive_5xx": 5,
"interval_ms": 10000,
"base_ejection_time_ms": 30000
}
}
}'

For a TCP check, use "type": "tcp" and leave out path; a TCP check with a path is refused. For an L4 load balancer, send the request to /networking/load-balancers/l4/LB-ID. If you create a load balancer through the API without health_check, it has no active check.

Passive check fields (API only):

FieldMeaningAllowedDefault
consecutive_5xx5xx answers in a row that take a backend out1–1005
interval_msHow often the error count is evaluated100–120,000 ms10,000 ms
base_ejection_time_msHow long the backend stays out the first time1,000–600,000 ms30,000 ms (30 seconds)

Verify

  1. Stop the app on one backend (or make its /health return 503).
  2. Wait for the removal time from the table above: about 30 seconds with the defaults.
  3. Send requests to the load balancer. They should all reach the other backends.
  4. Start the app again. After about 20 seconds, that backend gets traffic again.

Limits

SettingAllowedDefault
Interval100–120,000 ms10,000 ms (10 seconds)
Timeout100–120,000 ms2,000 ms (2 seconds)
Healthy threshold1–202
Unhealthy threshold1–203
PathHTTP and HTTPS checks only/health
Active checks per load balancer1, applied to every backend—

Troubleshooting

You seeWhat it meansWhat to do
Requests fail right after creating the load balancer (often 503 on L7)No backend passes the check yetWait 20 seconds. If it continues, test curl -i http://BACKEND-IP:PORT/health and check the steps below.
curl to the backend times outThe VM’s firewall group doesn’t accept the backend portAdd an accept rule for the port. See Create firewall rules. Check the OS firewall (ufw, firewalld) too.
curl returns 301, 302, 401 or 404The path redirects, needs a login, or doesn’t existUse a path that returns 200 directly, without a login.
The backend only accepts HTTPS on that portThe load balancer connects without encryptionUse a TCP check, or serve the health endpoint over HTTP.
A backend flaps in and out of rotationThe app is slow to answer, or the timeout is too shortRaise Timeout (ms) (keep it below the interval), or make the endpoint lighter.
”Health check interval must be between 100 and 120000 ms”The value is out of rangeEnter a value from 100 to 120000.
”path is not supported for tcp active health checks”An API request sent path with "type": "tcp"Remove path.