Skip to navigation

Rate limits

A rate limit is a cap on how much work a client may ask for in a period of time. When a limit is reached, the API answers 429 Too Many Requests instead of doing the work, and you try again later. This page says which limits the IBEE API applies, and how to retry without doing a write twice.

Which limits apply

LimitValue todayWhat you get when you pass it
Requests per second or minute, per token or workspaceNot published. IBEE has not announced a request-rate quota for the public API.—
Work in progress inside a product (concurrency)Set by each product service429
Body size of a request that creates something billable65,536 bytes (64 KiB)413 request_body_too_large
Records per page of a listDiffers per list, for example 10 by default and at most 100 for cloud VMsEach list’s default and maximum are in Lists and paging

Because no request rate is published, don’t build on an assumed number. Write your client so that it slows down when it sees 429, and contact support before you run large batch jobs in production.

When to use a retry

A retry helps only when the problem is temporary. For everything else, the same request gets the same answer.

You getRetry?Why
429Yes, after a waitA limit was reached; it clears with time.
503Yes, after a waitA service didn’t answer. With an error code, the gateway refused it and nothing was done; with detail, read the resource before you repeat a write.
502, 504 on a readYesThe read didn’t complete; reading again is harmless.
402 billing_deniedNoAdd credit or raise the limit first.
409 idempotency_in_progressYes, after Retry-After, with the same key and bodyThe outcome of a networking create isn’t known yet. See Idempotency.
400, 401, 403, 404, any other 409, 413, 422NoFix the request, the token or the resource state first.
423 organization_suspendedNoWrites stay refused until the suspension is lifted.
No answer at all (timeout, dropped connection)Reads: yes. Writes: only with the same idempotency key, and never a VM createThe write may have happened.

How it works

  1. A request reaches a limit, or a service it depends on is briefly unavailable.
  2. The API answers 429 (limit) or 503 (dependency unavailable) and does nothing else. A 503 from the gateway’s check carries an error code such as workspace_verification_unavailable or billing_admission_unavailable; the request never reached the product.
  3. If the response has a Retry-After header, it says how many seconds to wait.
  4. Your client waits, then sends the same request again, if it is safe to repeat.

Before you begin

  • Know which of your requests are safe to repeat: reads (GET) always are. A write is safe to repeat only when it carries the same idempotency key as the first attempt: a unique string, sent in the X-Idempotency-Key header, that lets the API recognise a repeated request. The VM writes that take one are listed in Idempotency.
  • Use a client that can read the response status and headers. curl -D saves headers to a file; the SDKs expose them on every error.
  • If you use an SDK, the CLI or Terraform, retries are already built in. See Retry settings in the IBEE tools.

Handle a 429 or 503

1

Check the status and the body

Retry only 429, 503 and 409 idempotency_in_progress. Every other error needs a change to the request, the token or your account first; retrying the same request gets the same answer.

2

Don’t retry a billing answer

If the body has a billing_reason, an admission_context_id, or an error code containing billing, admission, catalog or quote, Billing is involved. A 503 such as billing_admission_unavailable can be retried later; a 402 billing_denied cannot until you fix the cause.

3

Wait

Wait the number of seconds in Retry-After if it is there. Otherwise wait 1 second, then 2, then 4: double the wait each time, and add a little randomness (jitter) so many clients don’t retry at the same moment.

4

Retry only what is safe

Resend reads freely. Resend a write only with the same X-Idempotency-Key as before. Never resend a VM create that got no answer at all: list your VMs or read the operation first, so you don’t create a second billable VM.

5

Stop after a few attempts

After 3 or 4 attempts, stop and report the error with its x-request-id header, if the response has one.

This script reads the list of compute sites and retries 429 and 503 up to 3 times, honouring Retry-After when it is a number of seconds:

#!/usr/bin/env bash
# Needs IBEE_TOKEN and IBEE_WORKSPACE_ID in the environment.
set -u
url="https://api.ibee.ai/v1/compute/sites?workspace_id=${IBEE_WORKSPACE_ID}"
for attempt in 0 1 2 3; do
status=$(curl -sS -o response.json -D headers.txt -w '%{http_code}' \
-H "Authorization: Bearer ${IBEE_TOKEN}" "$url")
case "$status" in
429|503) ;; # worth retrying
*) break ;; # success, or an error that a retry won't fix
esac
[ "$attempt" -eq 3 ] && break
delay=$(awk 'tolower($1) == "retry-after:" && $2 ~ /^[0-9]+/ { print $2 + 0 }' headers.txt)
sleep "${delay:-$((1 << attempt))}" # Retry-After, else 1, 2, 4 seconds
done
echo "HTTP $status"
cat response.json

Retry settings in the IBEE tools

The SDKs, the CLI and the Terraform provider retry for you. Their defaults:

ToolRetriesRetried answersWait between triesLongest waitTimeout per request
Python SDK 0.7.02Reads: 429, 502, 503, 504. Writes with an idempotency key: 429, 503Retry-After, or 1 second doubling, ±10%60 seconds60 seconds
CLI 0.7.0 (uses the Python SDK)2Same as the Python SDKSame as the Python SDK60 seconds30 seconds
TypeScript SDK 0.7.02Reads and writes with an idempotency key: 429, 502, 503, 504Retry-After, or 1 second doubling, ±10%30 seconds60 seconds
Terraform provider 0.8.03Reads: 429, 502, 503, 504. Writes are never retriedRetry-After, or 250 ms doubling30 seconds90 seconds (request_timeout)

The Python SDK, the CLI and the Terraform provider never retry an answer that carries a billing or admission code. The TypeScript SDK doesn’t check for those codes, so it can retry a 503 such as billing_admission_unavailable on a write with an idempotency key. No tool retries a VM create, or a read of a secret identity’s login details, because each read issues a new secret ID. Set max_retries (Python) or maxRetries (TypeScript) to change the number of retries; 0 turns retries off.

Keep your request count down

  • Read sites, plans and images once per run and reuse the IDs, instead of before every create.
  • Ask for bigger pages (the list’s maximum limit) rather than many small ones.
  • Poll asynchronous work at a steady pace, every 2 to 5 seconds. Terraform checks an operation every 2 seconds; the SDKs and the CLI leave polling to you. See Asynchronous operations.
  • Don’t run the same create from several processes; give each action one owner and one idempotency key.

Troubleshooting

You seeWhat it meansWhat to do
429A product’s concurrency limit was reachedWait (Retry-After, or 1, 2, 4 seconds) and retry. If it keeps happening, slow down or run fewer jobs in parallel.
503 with workspace_verification_unavailable, principal_verification_unavailable, principal_authorization_unavailable or product_check_unavailableThe gateway couldn’t finish its checks; nothing was doneRetry after a few seconds.
503 with billing_admission_unavailable, billing_admission_timeout or a …catalog… codeBilling or the price catalog didn’t answer; nothing was createdRetry later with the same idempotency key.
503 with detailThe product service is unavailableRetry reads. For writes, read the resource first to see whether the change happened.
413 request_body_too_largeThe body of a billable create is over 64 KiBSend a smaller body.
The same 429 after every retryYou are sending faster than the limit allowsLower your request rate and contact support with your expected volume.