Serializing a hash chain on Aurora DSQL: the ceiling, measured

An audit log has one requirement that most databases never promise: a strict total order. Record N embeds the hash of record N−1. Two writers that both believe they are writing seq 6 will fork the chain, and a forked audit log is worse than no audit log, because it looks authoritative.

AWS used to sell a database that solved this — QLDB — and retired it on July 31, 2025, pointing customers at Aurora PostgreSQL with an explicit note that the migration loses cryptographic verifiability. I rebuilt that layer on Aurora DSQL, AWS's newest database. DSQL is serverless, multi-region, and active-active, and it hands you exactly one concurrency primitive to build an ordering on: optimistic concurrency control. No advisory locks, no blocking SELECT ... FOR UPDATE, no sequences, no triggers. Transactions run against a snapshot; when two of them touch the same rows, the loser aborts at COMMIT with SQLSTATE 40001 (DSQL marker OC000).

I wrote earlier about how to build a correct hash chain on that primitive. This post is about what it costs. I benchmarked the append path under contention on a real multi-region pair (us-east-1 + us-east-2, witness us-west-2), found the ceiling, and then moved it with batching. Everything here ties to a number in the repo.

The design being measured

Each stream keeps a head row, (head_seq, head_hash), and an append is one transaction:

BEGIN;
SELECT head_seq, head_hash FROM streams WHERE id = $1;
-- app-side: build record seq = head_seq + 1, prev_hash = head_hash
INSERT INTO records (...);
UPDATE streams SET head_seq = $2, head_hash = $3
  WHERE id = $1 AND head_seq = $4;   -- guarded CAS
COMMIT;

Every append updates the same head row, so two concurrent appends to one stream cannot both commit. That is the point: the head row turns "please serialize this" into something the database enforces at commit time, across regions, with no locks held.

The retry is where correctness lives. A lost race means the head moved, which means the failed attempt's prev_hash is stale. Replaying the same statements would either fail forever on the guard or link a record to the wrong predecessor. The correct loop re-reads the head and rebuilds the record — seq, prev_hash, hash — on every attempt, with bounded exponential backoff and full jitter (lib/db/append.ts).

So the design is honest about its bottleneck: one CAS on one row per record. The question is what that costs under real contention.

Method, briefly

The harness is scripts/bench.ts (npm run bench). Each config runs N concurrent writers against one or more streams; by default writers alternate between the two regional endpoints of the pair, so half the load enters through us-east-1 and half through us-east-2, both writing the same chains — the pair is active-active. Three reps per config. Appends get up to 25 OCC attempts (backoff 25 ms doubling to a 1 s cap, full jitter); an append that exhausts them counts as starved, and a rep aborts after three starvations rather than hammering the cluster. After every rep the harness re-reads each chain through the second region's endpoint and re-verifies every hash; the streams are then deleted. Raw per-append JSONL and summary JSONs for every config are committed in bench/results/.

Two caveats to keep in mind. The client was a single Node process on my workstation over the public internet, not an EC2 box in-region, so absolute latencies include my round trip to AWS; the comparisons between configs are what matter. And payloads were small (about 100 bytes of JSON) — this measures the ordering machinery, not bulk ingest.

Sweep A: the ceiling

One stream, 128 records per rep, writers from 1 to 64.

A single writer commits about 5.5 appends per second — a 181 ms median round trip per append, one CAS at a time. Sixty-four concurrent writers commit 6.8 per second. That is the entire headline: on one stream, concurrency buys almost nothing. The head row admits roughly one commit per round trip, and everything else queues in retry loops.

What grows instead is the price per append:

At 4 writers, 91.9% of appends still commit on the first try. At 16 writers that is 65.1%. At 64 writers it is 17.5%, and a committed append needed 10.8 attempts on average:

Past 8 writers a new failure mode appears: starvation. Some appends burn all 25 attempts — 14 to 16 seconds of retrying — and give up. In the 64-writer config, 18 of 359 appends starved. The harness records every one (they are the "ok":false rows in the JSONL, and the reps carry an aborted flag when the failure cap stopped them early).

The integrity result is the part I care about most: across every rep of every config in this post — including the aborted ones — the chain re-verified gap-free from the other region. Contention on this design costs latency and, at the extreme, starvation. It never cost a fork, a gap, or a wrong link. That is the OCC contract doing exactly what it promises: the database refuses to let a stale writer commit, and the punishment for load lands on availability, not correctness.

Sweep B: the cross-region tax that wasn't

Same matrix, but with every writer connecting to the us-east-1 endpoint instead of splitting across both. I expected the split-writer runs to pay something for cross-region contention. They did not:

The two curves are the same line with noise. The reason, I think, is that the commit path does not care where the SQL entered: this is one peered cluster, and a durable commit coordinates across regions either way. To be precise about what this does and does not show: "single-region" here means all writers used one regional endpoint of the same multi-region pair. It is not a comparison against a true single-region cluster, which would be a different (and cheaper) commit protocol.

The asymmetry that did show up surprised me: the us-east-2 endpoint is faster. Across the low-contention sweep A runs, a first-try append through us-east-1 took a median 181 ms; through us-east-2, 126 ms (n = 900 and 509). Same chain, same cluster, 55 ms apart by entry point — presumably the placement of the transaction coordination relative to my client. If you run active-active and care about write latency, measure per endpoint; the label on the region is not the number.

Sweep D: batching moves the ceiling

The ceiling exists because each record costs one CAS round trip. The fix is to stop paying it per record. appendMany (lib/db/append.ts) appends K records in one transaction: it reads the head once, chains K records in memory (record i's prev_hashis record i−1's hash; record 1 chains to the stored head), inserts them in one multi-row INSERT, and advances the head once, still guarded by AND head_seq = $expected. On a lost race the entire batch is rebuilt from the fresh head — the same re-chain rule as a single append, applied K records at a time. DSQL caps a transaction at 3,000 modified rows and 10 MiB; I cap K at 100, far under both.

Eight writers, one stream, 400 records per rep:

K=1 is sweep A's 8-writer config again, and again some appends starved. At K=10 the same offered load commits six times faster with zero failures. At K=50, the stream that could not exceed 7 appends per second does 182, because 8 writers × 8 transactions replace 400 CAS rounds with 24.

Batching is not free, and the table says where the bill goes. A batch commit holds more work behind one round trip, so op latency rises — 882 ms median for a 50-record batch — and a lost race now rebuilds and re-hashes 50 records instead of one (83% of batch ops at K=50 retried at least once). The semantics change too: the batch is atomic, all-or-nothing, and the caller has to buffer records before writing, which trades a little ingest freshness for throughput. For an audit log fed by an application that can buffer even 100 ms of events, that trade is nearly always right.

Sweep C: scale across streams, not within one

The linear chain serializes per stream by design. The other axis is free: streams do not share a head row, so they do not contend.

Eight writers per stream, unbatched: 1 stream does 5.4 appends per second, 2 do 12.4, 4 do 25.4, 8 do 44.4 — against a linear projection of 43.1 from the one-stream baseline. Contention does not leak across streams; within the range I tested, aggregate throughput is streams × the per-stream ceiling. (The 8-stream config ran as three single-rep invocations because the harness hard-caps any one invocation at 2,000 records.)

The practical reading: a stream should map to the natural unit of audit consistency — a tenant, a service, a workflow — and a deployment scales by having many of them, each individually ordered, each individually verifiable. Combined with batching, the measured ranges here span roughly 5.5 appends/s (one stream, unbatched, worst case) to well over a thousand (many streams × K=50), on a database with no locks and no sequences.

Honest limits

• One day, one cluster pair, one client. These are 2026-07-19 numbers from a single Aurora DSQL pair, driven from one machine over the public internet. The shapes (flat ceiling, near-linear stream scaling, batch amortization) should transfer; the absolute numbers will move with your client placement and payload size.
• Starvation is real at high contention. Past 8 unbatched writers on one stream, tail appends can exhaust a generous retry budget. If your workload looks like that, the answer in order: batch, then shard streams, then queue writers upstream. More retries is the wrong lever — the ceiling does not move.
• The ledger is tamper-evident, not tamper-proof. Verification detects rewrites; it does not prevent them, and anchored verification is only as fresh as the last signed checkpoint. That part of the system is measured by a different kind of test — the live demo lets you run the attack yourself.
• Serial per stream is the design, not a bug. If you need total order across everything, you get one stream and its ceiling. Most audit domains do not need that.

Run it yourself

The harness, the raw JSONL of every append in this post, and the summary JSONs are committed. The whole matrix — about 14,700 records — cost cents in DPUs.

git clone https://github.com/V-3604/indelible && cd indelible && npm install
# .env.local with your DSQL endpoints — see docs/DB_ACCESS.md

npm run bench -- --writers 8 --appends-per-writer 16 --dry-run   # prints the plan
npm run bench -- --writers 8 --appends-per-writer 16             # sweep A point
npm run bench -- --writers 8 --appends-per-writer 50 --batch 50  # sweep D point
npm run bench -- --writers 8 --appends-per-writer 16 --streams 4 # sweep C point

Every run ends by re-verifying every chain it touched from the second region and fails loudly if a single hash is wrong. That check never fired.

The ledger itself is live at indelible-eta.vercel.app— the console has a fenced Danger Zone that runs a malicious DBA's raw SQL for you, and an offline verifier (npm run verify:offline) that checks the chain and its signed anchor without trusting me or my database.