Repository navigation
Description
Summary
When N readers open one new link at once, the server starts N scrapes of the same page. The laptop bench saw each request reach the origin, up to the burst size. A layer-2 sketch that joins them cut origin hits to 1 per burst. It also halved p99 at a burst of 500.
Measured
See Benchmark. Coalescing is measured on layer 2 only, with k = 1, and it is a cross-group reading against layer 1. Herd did not run on Linux.
Cause
runPipeline(modules/link-metadata/domain/pipeline.ts) has no in-flight join. Each miss runs its own stages until the first write lands.- The image route calls
runPipelinetoo (modules/link-metadata/http/imageController.ts). - The social-card module already has
createSingleFlight(modules/social-card/infra/renderGuards.ts:54).
What to change
- Wrap the stage run in
runPipelinein a singleflight keyed by the cache key: URL, language and pad generation. - Delete the entry when it settles. The image route shares the same entry.
Decisions
- Per process or across replicas? Recommended: per process. At 2 replicas, the worst case is 2 scrapes per link. Other: a Redis
SET NXlock, which adds round trips and a stale-lock case. - Reuse
createSingleFlightfrom social-card? Recommended: yes, if its contract fits. Other: a 15-line map in the module.
Benchmark
Laptop before, measured 2026-11-14
Condition: the laptop condition in Bench conditions below. Scenario herd: 5 bursts per size on 5 distinct URLs, 300 ms origin delay.
| Burst | Layer 1 (20261114T020638, 20261114T033808) origin hits per URL (maxPerUrl) |
Layer 1 p99 ms | Layer 2 module p99 ms | Layer 2 coalesce (k = 1): origin hits / p99 ms |
|---|---|---|---|---|
| 200 | 200 (resets 7.2 % in both) | 355.44 / 364.16 | 364.96-438.78 | 1 / 321.82 |
| 500 | 500 (origin active max 256) | 687.2 / 770.44 | 686.35-691.75 | 1 / 343.37 |
- Layer 1 sent no coalescing: each request that got through reached the origin. At burst 500 in
20261114T033808, the origin total was 2469 against 2448 answered. - At burst 500 the uncoalesced burst runs in two waves behind the 256-slot pool.
- The layer-1 resets at burst 200 did not occur on layer 2. Their cause is not proven, and coalescing may not fix them.
- Coalescing lowered REST CPU by 14 to 16 points at 200 and 9 to 14 points at 500.
After
Use the after method in Bench conditions below, on layer 1, at bursts 10, 50, 200 and 500. Report resets apart from the origin-hit count.
Condition
The laptop condition in Bench conditions below. A Linux run of herd is owed (#793).
Content-safety checks
- The key holds URL, language and pad generation. A joined caller never gets another language or an older pad card.
- The join shares exactly what the Redis cache already shares across readers. It adds no new exposure.
UX checks
- Joined callers wait for one scrape, so the card arrives at about the same time for all of them.
- One slow scrape delays every joined caller. Link metadata: one deadline per request #790's deadline bounds that.
Rollback
One setting turns the join off. Otherwise, revert the commit.
Acceptance criteria
- Origin hits per burst are 1 at every burst size, on one replica.
- Every answer is a full card, with 0 degraded.
- At burst 500, p99 sits below 2 x the origin delay plus 100 ms.
- A
Measuredcomment posts the after beside the before, with the condition.
Related
- REPORT.md §2 (herd), §4 rank 6 and §6.7.
- Link metadata: time-bound Redis calls on the request path #789: a fire-and-forget write needs this change.
Bench conditions
The bench is apps/hocuspocus.server/scripts/bench-link-tolerance/ (commit 1a0d21b7f). The headline numbers are in apps/hocuspocus.server/src/modules/link-metadata/BENCHMARK.md, section Load tolerance. REPORT.md, LINUX-RESULTS.md, research-*.md and raw/ are local bench notes (Notes/local-docs/2026-11-13-link-scraper-tolerance/), not in the repo. The Linux runner is local too.
Linux condition, used by every "Linux before": one REST container at --cpus=1 --memory=512m --memory-swap=512m, in the OrbStack VM on one laptop. Kernel 7.2.12-orbstack-00394-gbecdd6130068, booted with cgroup.memory=nokmem,nosocket, so socket and slab memory are not charged. --no-healthcheck. Bun 1.4.4+38afee6c8. Redis 8.8.0 (redis:alpine) in REST's network namespace. --network none. Pooled loopback to one local origin, no DNS, no TLS. benchTree 8774a6275a05005f, srcTree a9b6ca73ca68c2ec, runnerTree 516099a8c9839eac. Measured 2026-11-14. The core speed ratio to a production vCPU is not measured.
Laptop condition, used by every "Laptop before": local, one laptop (Apple M5 Max, 18 cores, 48 GB, macOS), Bun 1.4.4+38afee6c8, native Redis 8.10.3. REST runs under the sandbox-exec fence. Traffic is plain HTTP to one local origin, pooled loopback, no DNS, no TLS. No CPU quota and no cgroup. benchTree 8774a6275a05005f, srcTree a9b6ca73ca68c2ec. Measured 2026-11-14. Never compare it with a Linux number as one series.
"(void session)" marks a value from a valid step inside a void session. On Linux the OOM kill voided those sessions. The kill is the measured break.
After method, used by every "After":
- Base = the commit before the change. Head = the change.
- Run ABAB, at least 3 sessions per arm, each with its control step, at n ≥ 2,000 per step.
- The change moves
srcTree, so the before must run again. Do not reuse the 2026-11-14 numbers as the before. - The runner takes
src/from the working tree. So it first needs a source-folder option for base (REPORT.md §5.7 item 5). - On Linux, start each session in a fresh REST container, and keep the run agent alive until the footer is written.
- Quote a change only when every head session sits outside the base range. Give σ and the MDE beside it.
Paths are under apps/hocuspocus.server/src/ unless they say otherwise.
Activity
perf(link-metadata): fetch a URL once per burst
HMarzban commentedon Nov 15, 2026
Measured: before and after (slot 3, 2026-11-15)
Setup. One REST container at
--cpus=1 --memory=512m(OrbStack, Linux cgroup v2), on one laptop. Both arms were built fromgit archive: base061fee04d, headf62a5e383, with the same bench. Base and head ran in turns (ABAB, BABA, ABAB). 3 valid sessions per arm. A step is void above 250 % other-process CPU (maintainer ruling, 2026-11-14).Bursts of requests for one link that is not in the cache:
Status. Shipped in 2.3.6 (
f62a5e383). Canonical table: moduleBENCHMARK.md, "After #787 to #792". Raw data:Notes/local-docs/2026-11-13-link-scraper-tolerance/raw/linux/s3/(local).HMarzban commentedon Nov 15, 2026
Shipped in 2.3.6 (
f62a5e383). The before and after is in the Measured comment: #791 (comment)Closing as done. The issue stays in milestone v2.3.8 as the record of the load-tolerance work.