Service mesh simulator: retries and outlier detection

The client calls service A, which calls B, and so on down the chain; every call passes through the caller’s and the callee’s sidecar proxies. Some instances of the last service fail, or the whole service is down. Choose where retries are configured, how many tries each call gets, and whether proxies eject failing instances. Twenty requests a second run for a minute, and the table shows the same failure under five setups.

In a microservice system, every service needs the same plumbing for calling others: load balancing, retries, timeouts, encryption, metrics. A service mesh moves that plumbing out of the services into proxies. Each instance gets a sidecar proxy that all its traffic passes through, and a control plane pushes routing and policy to every proxy, so behaviour can change without redeploying code. Istio, for example, builds on the Envoy proxy.

Because the mesh makes these features easy to switch on everywhere, it also makes it easy to switch them on in combinations that interact badly, which is what the simulator explores.

Istio retries a failed HTTP request twice by default, three tries in all. Against an instance that fails now and then, that is a good trade: in a four-service chain where one of five instances of the last service always fails, success went from 80% without retries to 100%, at the cost of a longer tail (55 ms at the 99th percentile instead of 24 ms).

The danger is a dependency that is really down. Each hop that retries multiplies the calls below it, so with three tries at each of four hops a single user request becomes 3⁴ = 81 calls to the dead service, and the request takes 2.7 seconds to fail. With retries only at the edge, it is 3 calls. Google’s SRE book warns about exactly this combinatorial growth when several layers retry; the usual advice is to retry in one place and give each request a deadline.

Outlier detection is passive health checking: the proxy watches real responses and stops sending to an instance that keeps failing. Envoy’s defaults eject a host after 5 consecutive 5xx responses, for 30 seconds multiplied by the number of times it has been ejected, and never eject more than 10% of a cluster at once. With outlier detection and no retries, success rose to 99.2%.

But errors propagate: a healthy instance of service C that calls the bad instance of D returns an error to B, and to B’s proxy it looks just as faulty. In that run 8 instances were ejected and 6 of them were healthy, pushed out because of a fault below them. Retrying at each hop first, so the bad instance’s errors are absorbed where they happen, kept every healthy instance in service: 2 ejections, 0 of them healthy, and 100% success.

Every hop passes through two proxies, the caller’s and the callee’s, so a chain of four services adds eight proxy traversals to each request. With 0.5 ms per proxy and 5 ms of work per service, the proxies are 17% of the request’s time; the per-proxy cost is a setting in the simulator because it depends on the proxy, its configuration and the hardware. Every sidecar also uses CPU and memory of its own.

A mesh does more than retry. Timeouts cap how long a call may take; Istio’s documentation notes that the Envoy timeout for HTTP requests is disabled by default, so a hung dependency can hold a request indefinitely until one is set. Mutual TLS encrypts and authenticates traffic between proxies with certificates the mesh issues and rotates. Traffic splitting sends a percentage of requests to a new version, the basis of canary releases.