Service discovery, finding live instances.

Service discovery is how services find each other in a dynamic environment where instances come and go. The two patterns: client-side discovery (client queries a registry then load-balances) and server-side discovery (a load balancer fronts the service, hiding instance changes). DNS is the oldest registry; Consul, etcd, ZooKeeper, Eureka, Kubernetes Services are the modern ones.

In a static world, you write api.example.com in your config and call it forever. In a dynamic world (autoscaling groups, container orchestrators, blue-green deploys, spot fleets, rolling restarts) instances appear and disappear constantly. Every restart can mean a new IP, so a hardcoded list goes stale within minutes.

Service discovery fills that gap with a registry of which instances are currently up, and a way to query it. The design questions are who maintains the registry, who reads from it, and how stale the answer is allowed to be.

The pattern you choose affects deployment complexity, language portability, and how much logic lives in the client rather than the platform load balancer.

There are two approaches. With self-registration the instance announces itself on startup ("I am here, IP 10.0.0.42, port 8080") and re-announces (heartbeats) periodically. Eureka, Consul agents and mDNS work this way. It is simple, but the workload has to know about the registry.

With third-party registration a separate process watches infrastructure events and registers instances on the workload's behalf. Kubernetes does this: the kubelet reports pod status to the API server; the endpoints controller writes them into the Service's endpoint set. The workload doesn't need to know about discovery.

Liveness, readiness and startup probes answer different questions. Kubernetes and most modern systems keep them separate, while older systems combine them and can mistake an overloaded instance for a crashed one.

The simplest discovery mechanism is DNS. Configure A records for each instance; clients resolve a name and get a list. It is cheap, universal and needs no new infrastructure. The downside is TTL: clients cache records for as long as the TTL says, even after the instance is gone.

SRV records add port/weight/priority, but most systems do not use them. DNS-SD (RFC 6763) standardises service lookup over DNS, and Kubernetes headless services take a similar approach, with very low TTLs and DNS servers tied to the orchestrator. DNS-based discovery still runs into caching problems, since the JVM has its own DNS cache, glibc can use nscd and browsers keep their own.

Consul (HashiCorp): DNS interface plus an HTTP API; gossip-based agent network; multi-datacenter primitives. Works well in mixed fleets of VMs and containers.

etcd (CoreOS, now CNCF): the registry behind Kubernetes Services. Watch-based; you subscribe to a key prefix and get a stream of changes. It is strongly consistent and Raft-replicated.

Zookeeper (Apache): older, the original "coordination service". It offers low-level primitives (sequential nodes, ephemeral nodes) but is heavier to operate. Kafka used it for years; some newer systems have moved off it (Kafka's KRaft).

Eureka (Netflix OSS): designed for AP, happy to return slightly stale data during partitions if it keeps clients running. Pairs with Ribbon on the client.

A registry is a distributed system; CAP applies. Strong-consistency registries (etcd, Zookeeper) refuse to return stale data during partitions, so clients fail closed. AP registries (Eureka) keep returning the last known list, so clients fail open and might call dead instances.

The right pick depends on your topology. For the Kubernetes etcd store, an AP design could corrupt the control plane, so CP is required. For user-facing service discovery, "slightly stale" is sometimes better than "no answer", which is Eureka's design choice.

This is a composite failure based on several real postmortems. An orders service runs six instances behind nginx. The upstream block is rendered by consul-template: when the Consul catalog changes, the template re-renders and nginx reloads. A routine afternoon deploy rotates the autoscaling group: six new instances up, six old ones down.

Four of the old instances shut down cleanly: the SIGTERM handler deregisters from Consul before exit, and they leave the catalog immediately. Two do not: the orchestrator gives them ten seconds and then kills them hard, before the deregistration call goes out. The registry still lists them as passing. The health check runs every ten seconds and needs three consecutive failures before marking an instance critical, so for roughly thirty seconds the catalog is wrong: two of the "healthy" IPs answer nothing.

Now the consumers. consul-template notices the change, waits out its quiescence window (a few seconds, to batch flapping), renders the new upstream block, and reloads nginx: five to fifteen seconds from change to traffic, for the entries the registry knows about. The two dead-but-passing instances stay in the rendered config until the health check catches up, so nginx keeps sending a quarter of requests to IPs that refuse connections. With proxy_next_upstream configured, those requests retry another backend and users see a latency blip. Without it, they see 502s.

Clients that resolve the registry's DNS directly do worse. The records carry a low TTL, but the JVM's resolver caches positive lookups for thirty seconds by default (indefinitely on some older defaults), and there may be an nscd or node-local DNS cache in between. Their staleness window is the registry's window plus every cache on the path.

An Envoy-style consumer behaves differently. Endpoints arrive by xDS push, so propagation is sub-second instead of render-and-reload, and outlier detection ejects the dead endpoints after a few consecutive failures without waiting for the registry at all. One caveat is the panic threshold. If too large a fraction of a cluster looks unhealthy at once, as it can during a fast rolling deploy, Envoy assumes the health data is wrong and routes to all endpoints, dead ones included.

Staleness adds up from several terms: health-check interval times failure threshold, plus propagation to each consumer, plus reload time, plus client DNS caches. Summing them gives the worst case. Then shrink the largest term. Deregistering on SIGTERM helps more than faster health checks, and retrying on connect failure helps more than either.

Consul · HashiCorp 2014

Built for service discovery. CP via Raft. Built-in health checks, multi-datacenter replication, an ACL system and a DNS interface. It is the most full-featured of the dedicated registries, and HashiCorp's commercial tier (Consul Enterprise) adds federation, namespace isolation and network segmentation.

etcd · CoreOS 2013, CNCF 2018

The control plane datastore behind Kubernetes. CP via Raft. Lower-level than Consul: no built-in health checks, no DNS interface; you build service discovery on top. Tuned for the watch protocol. A cluster is a single Raft group, and 3 or 5 members is the usual size.

Apache ZooKeeper · 2008

The oldest of these registries. CP via ZAB (a Paxos-like protocol). Used by Kafka, HBase, HDFS, Solr, Druid for cluster coordination rather than direct service discovery. Its API is harder to use correctly than Consul or etcd; its clients are lower-level.

Netflix Eureka · 2012

The best-known AP registry. It returns the last-known list during partitions; clients fail open. It has seen little active development since Netflix moved to an Envoy-based service mesh and still ships with Spring Cloud. It is the standard example of AP being the right choice for some user-facing service discovery.

Kubernetes Services + CoreDNS

If you're already on Kubernetes, you already have service discovery. The Service object is the registry; etcd is the durable store; CoreDNS resolves names; kube-proxy programs the kernel for routing. No additional registry to operate.

Choosing by ecosystem. Already on Kubernetes? Use Services. For cross-platform discovery (VMs, containers and bare metal), use Consul. If you run Kafka you already have ZooKeeper or KRaft. A new JVM service on Spring can still use Eureka. For low-level coordination primitives in a custom system, use etcd directly.

Netflix: Eureka, then service mesh. Eureka was Netflix's service discovery for over a decade. Netflix chose AP because, during a region failure, returning a stale instance is better than returning nothing. Around 2018-2020 Netflix migrated most internal traffic to a service-mesh model (custom Envoy data plane), where service discovery happens via xDS push from a central control plane rather than a per-instance lookup.

Lyft: Envoy + xDS. Lyft, where Envoy originated, runs it as a per-host proxy with xDS-based service discovery. The control plane holds the registry and pushes endpoint updates to every Envoy as the cluster changes. The data plane already sits in the request path, so making it the discovery client adds little.

Airbnb: SmartStack, then service mesh. Airbnb's home-grown SmartStack was an early (~2013) implementation of client-side discovery via local HAProxy and Synapse. They migrated to Istio + Envoy in the late 2010s. The lesson Airbnb drew was that client-side discovery libraries tend to reinvent service-mesh features (retries, circuit breakers, observability), and once you have built those you have a service mesh of your own.