KMS & Secrets.

1 · What KMS actually is

The mental model that survives every conversation: KMS is a fleet of FIPS 140-2 Level 3 HSMs that AWS operates on your behalf. The customer master key material never leaves those HSMs in plaintext. You can't export it; AWS engineers can't see it; there is no GetPrivateKey API. Every cryptographic operation (encrypt, decrypt, sign, generate data key) is an RPC into the HSM quorum that performs the operation inside the secure boundary and returns the result. The whole security story rests on "the keys are unexportable, full stop."

What KMS isn't: a database for secrets. The maximum direct-encrypt payload is 4 KB. KMS doesn't store ciphertext for you; you store the ciphertext, KMS holds only the keys. KMS isn't a per-region cluster you provision — it's a regional shared service with published TPS quotas (5,500-30,000 rps per region depending on operation, per account, with on-request limit increases). And KMS is not free for customer-managed keys: $1/key/month plus $0.03/10k API calls. A high-traffic application that calls Decrypt on every request is calling KMS, and KMS is one of the more expensive AWS services per-API-call.

2 · How KMS is built — the public sketch

AWS published a cryptographic-details whitepaper with enough of the request path to draw the shape. The key fact: every KMS request fans out to a quorum of HSMs, and only a quorum response is acked.

Two implications. The HSMs are the security boundary — the front-end is just authentication, policy evaluation, and routing. Compromise the front-end and you can deny service; you cannot exfiltrate keys. And every KMS API call is logged to CloudTrail with the principal, the key ARN, the encryption context, and the request ID. If something decrypts your data, you have an audit record of it.

3 · Envelope encryption, explained

Naive approach: send every byte of plaintext to KMS, get ciphertext back. Breaks immediately — KMS has a 4 KB ciphertext limit per direct call and you can't move 100 MB of data through it. Envelope encryption is the workaround.

1. Call GenerateDataKey with a Customer Master Key (the CMK / KEK — key-encryption-key). KMS returns two blobs: the data key in plaintext (256 bits of randomness) and the data key encrypted under the CMK.
2. Use the plaintext data key locally to AES-GCM-encrypt your data. Then zero the plaintext key from memory.
3. Store the ciphertext alongside the encrypted data key. The CMK never left KMS.
4. To decrypt: send the encrypted data key to KMS's Decrypt, get the plaintext key back, use it locally, zero it.

Every "encrypted at rest" S3 object, EBS volume, RDS row, and DynamoDB item works this way. AWS-managed CMKs handle it transparently; with a customer-managed CMK, you control the key policy and rotation.

The EncryptionContext is the underrated power here. It's a free-form key-value pair that becomes additional authenticated data — encrypted-key handles are useless without the same context on Decrypt. S3 uses it (passes the bucket and key name as context), preventing an attacker from moving a ciphertext blob to a different object slot. RDS, EBS, and DynamoDB do the same. If you write your own envelope-encryption code, always pass a context that identifies what is being encrypted.

4 · S3 Bucket Keys — how the optimisation works

Without S3 Bucket Keys, every SSE-KMS object PUT/GET triggers a KMS API call. A bucket doing 50,000 GETs/sec on KMS-encrypted objects burns 50,000 KMS Decrypts/sec — well over the per-region quota and a meaningful chunk of the KMS bill.

S3 Bucket Keys (2020) is the fix. With Bucket Keys enabled, S3 asks KMS for a time-limited per-bucket data key once and derives per-object data keys from it locally (via HKDF). Across the lifetime of the bucket key (about a week), S3 can encrypt millions of objects without further KMS calls. AWS quotes a ~99% reduction in KMS calls; for high-traffic buckets, this drops the KMS bill from "noticeable" to "rounding error."

5 · Key types

Within customer-managed, you also choose symmetric vs asymmetric. Symmetric (AES-256) for envelope encryption — the common case. Asymmetric (RSA, ECC) when you need Sign/Verify or to share a public key with someone outside AWS who needs to encrypt for you. Symmetric is cheaper and faster.

6 · Key policies, grants, and IAM

A KMS key has three layers of access control. Key policy: a JSON document on the key itself; the root of trust. If the key policy doesn't grant access, no IAM policy can override that. IAM policies: on principals; effective only if the key policy also allows access (the key policy must delegate to IAM with the standard "Principal": {"AWS": "arn:aws:iam::123:root"} + IAM-allow pattern). Grants: programmatic, revocable, scoped delegations — typically used by AWS services on your behalf (e.g., RDS creates a grant to encrypt/decrypt for the duration of the volume's life).

7 · Rotation — what it really does

"Automatic key rotation" on a CMK rotates the backing key material every year (or every 90 days if you opt in to the faster cadence). The KMS key ID and ARN don't change. All prior backing keys are retained — KMS automatically uses the right one to decrypt anything previously encrypted. From the caller's perspective, nothing changes; from a compliance perspective, the keys-in-use rotate.

What this does not do: re-encrypt your existing data with the new key. That's a separate job — call ReEncrypt on each ciphertext if you actually need the data re-wrapped under the new material. For most use cases (audit windows, compliance attestation), you don't.

8 · Secrets Manager vs SSM Parameter Store

Practical rule: anything that can be auto-rotated belongs in Secrets Manager (the $0.40/month buys real value); everything else — flags, endpoints, non-secret config — belongs in Parameter Store. Mixing the two intentionally is fine.

9 · Build it yourself — envelope encryption end-to-end

1. Create a customer-managed key.
   KEY_ID=$(aws kms create-key --description "lab envelope key" \ --query KeyMetadata.KeyId --output text) aws kms create-alias --alias-name alias/lab-envelope --target-key-id $KEY_ID echo $KEY_ID
2. Generate a data key and encrypt some data locally.
   # Get a data key — KMS returns plaintext + encrypted versions. aws kms generate-data-key --key-id alias/lab-envelope --key-spec AES_256 \ --query '{p:Plaintext,c:CiphertextBlob}' --output json > /tmp/dk.json PT_KEY=$(jq -r .p /tmp/dk.json) # base64 plaintext key ENC_KEY=$(jq -r .c /tmp/dk.json) # base64 encrypted key (store this) # Encrypt a payload with openssl using the plaintext key. echo $PT_KEY | base64 -d > /tmp/dk.bin echo "ssn: 123-45-6789" | openssl enc -aes-256-cbc -salt -pbkdf2 \ -pass file:/tmp/dk.bin -out /tmp/payload.enc # Zero the plaintext key from disk. shred -u /tmp/dk.bin 2>/dev/null || rm -f /tmp/dk.bin echo "Stored: /tmp/payload.enc + encrypted data key in /tmp/dk.json"
3. Decrypt — only the encrypted key plus the CMK can recover it.
   # Ask KMS to decrypt the data key. echo $ENC_KEY | base64 -d > /tmp/enc.bin aws kms decrypt --ciphertext-blob fileb:///tmp/enc.bin \ --query Plaintext --output text | base64 -d > /tmp/dk.bin # Decrypt the payload locally with the recovered key. openssl enc -d -aes-256-cbc -pbkdf2 -pass file:/tmp/dk.bin -in /tmp/payload.enc shred -u /tmp/dk.bin 2>/dev/null || rm -f /tmp/dk.bin
4. Stash a real secret in Secrets Manager and read it back.
   aws secretsmanager create-secret --name lab/dbcreds \ --kms-key-id alias/lab-envelope \ --secret-string '{"user":"app","pass":"hunter2"}' aws secretsmanager get-secret-value --secret-id lab/dbcreds \ --query SecretString --output text # Returns the JSON. KMS Decrypt happened transparently.
5. Tear down. Customer-managed keys must be scheduled for deletion — minimum 7 days.
   aws secretsmanager delete-secret --secret-id lab/dbcreds --force-delete-without-recovery aws kms delete-alias --alias-name alias/lab-envelope aws kms schedule-key-deletion --key-id $KEY_ID --pending-window-in-days 7

You've now done by hand what every AWS service does behind the scenes for "encrypted at rest." The CMK never left KMS; only encrypted data keys traveled over the wire.

10 · What breaks

• "Access denied" on KMS when calling S3 / RDS / Secrets Manager. The issuing service tries to use a CMK on your behalf via a grant; if the principal lacks kms:Decrypt on that key, you get a generic AWS-service error that gives no hint that KMS is the cause. Always check CloudTrail for KMS AccessDenied events when debugging encryption issues.
• Cross-account: both sides need permission. Account A's key policy must allow Account B's principals AND Account B's IAM must allow the action. Most teams remember one half and miss the other.
• KMS request throttling. Quotas are per-region per-account (5.5k–30k rps depending on operation). High-throughput services that call Decrypt on every request — like a Lambda fetching a secret on every cold start — hit the wall. Cache decrypted values.
• Forgotten Secrets Manager rotation Lambda. The rotation Lambda runs in your account; if its execution role lacks kms:Decrypt on your CMK, rotation fails silently — you get an alarm in CloudWatch only if you set one up.
• Imported key material expires. Bring-your-own-key keys can be imported with an expiration; data encrypted under them becomes permanently unreadable if you don't re-import in time. There is no recovery.

11 · Real-world case studies

Three public stories show how KMS shapes — and sometimes fails to save — production security architectures.

Capital One — the 2019 breach (what KMS does and doesn't protect against). A misconfigured WAF on an EC2 instance allowed an attacker to perform a Server-Side Request Forgery against the EC2 metadata service, retrieve the instance's IAM role credentials, and use them to read S3 buckets containing roughly 100 million records. KMS was working correctly the entire time — every object was encrypted at rest, every Decrypt call was authorised by valid IAM. The lesson isn't that encryption failed; it's that encryption only protects against threats that don't have a legitimate principal's credentials. Defence in depth — IMDSv2 (which mitigates the original SSRF), least-privilege IAM, encryption context conditions on KMS key policies, and S3 Block Public Access — is what fills the gap.

Financial sector — KMS for PCI-DSS and FedRAMP compliance. AWS publishes a Cryptographic Details whitepaper describing the KMS HSM fleet (FIPS 140-2 Level 3 validated; quorum-based key operations; documented secure boot and operator separation). For banks, broker-dealers, and federal contractors, this whitepaper is what makes KMS acceptable as the system of record for cryptographic key material — auditors can map the KMS architecture onto NIST SP 800-57 key-management requirements without a custom HSM deployment. The same compliance story is why AWS GovCloud's KMS is FedRAMP High-authorised and why most regulated workloads on AWS use KMS rather than CloudHSM (which gives you more control but you operate the cluster, including failover and durability).

Bucket Keys at scale — the KMS bill that disappeared. Before S3 Bucket Keys (launched 2020), every object PUT/GET on an SSE-KMS-encrypted bucket cost one KMS API call. A high-traffic data lake doing a billion operations a month was looking at $3,000+ in KMS API charges alone. S3 Bucket Keys let S3 derive a short-lived bucket-level data key from your CMK once, then use it locally for many object operations — cutting KMS calls (and KMS cost) by ~99% for typical workloads. The mechanism is interesting because it shows the envelope-encryption pattern recursing: KMS protects the bucket key, the bucket key protects the data keys, the data keys protect the objects. Enable it on every SSE-KMS bucket; there is no downside.

12 · Further reading

• KMS developer guide.
• KMS Cryptographic Details whitepaper. The HSM fleet, quorum protocol, and key lifecycle in full.
• Secrets Manager user guide.
• Envelope encryption — the canonical explanation.
• S3 Bucket Keys — the cost optimisation.
• Capital One 2019 breach analysis. The IAM/SSRF / KMS interaction story.
• Cloud identity. The IAM model KMS key policies extend.