Repository navigation
Description
Summary
REST has no limit on link-metadata work in flight. When arrivals outrun one CPU, the queue grows until the kernel kills REST. On one Linux replica at production limits, this happened at 6400 fast misses/s and at 96 hostile pages/s. When the driver bounded the work in flight, memory stayed flat at the same CPU ceiling.
Measured
See Benchmark. In short:
miss-fast@6400(void session): CPU throttled in 90 of 140 periods.inUseMean743.92,outboundInUseMax1570, anon 400.3 MiB. The OOM kill came during the drain.hostile@96(void session): CPU hit the quota in the warm-up. Memory reached 512 MiB at t 1.7 s, and the kill came at about t 2.9 s.- At
hostile@48, only 5 fetches were in flight (outboundInflightMax5). So the hostile queue is CPU work, and the outbound cap in Link metadata: cap outbound fetches so fast lanes stay alive when origins hang #788 does not bound it. - Closed loop, valid session: the driver held 16 to 64 requests in flight. Throughput reached 5665.53/s with 0 errors, and memory.current stayed at 265.3 MiB or less. The driver set this bound, not REST.
Cause
- No route has a concurrency limit, a singleflight or an in-flight dedupe (
research-code-limits.md). - The limiter caps volume per IP per 15 min. With
execEvenly: false, one IP may spend its whole bucket at once. It does not cap concurrency (research-code-limits.md). - Traefik's
rate-limit-api(average: 800,burst: 900) is likely per source IP (unverified). It sits above the hostile break rate (96/s). For one source IP, it sits below the 6400/s miss kill, but it does not bound a flood from many addresses.
What to change
- Count cold link-metadata requests in flight per process, in the module shared with Link metadata: stream image relays or cap image bytes in flight #786 and Link metadata: cap outbound fetches so fast lanes stay alive when origins hang #788.
- Over the cap, skip the stages and answer at once. Cache hits are always served. Health routes are never shed.
- Count shed answers (Link metadata: metrics for in-flight work, the outbound pool, cache, stages and parse time #792).
Decisions
- What cap? Recommended: 64 cold requests in flight per replica. In the closed loop, 64 in flight reached the CPU plateau (5665.53/s) with flat memory. The cost: real-delay misses cap near 64 / 0.36 s, about 178/s per replica (a projection). Production serves about 0.023 link requests per second. Other: 128.
- What does a shed request get? Recommended: the fallback card with
miss: 'unreachable'andCache-Control: no-store, with no Redis write.research-scaling-options.mdR3 says the webapp needs no change for it. Other: 503 withRetry-After, which needs a client check. - Shed on event-loop lag too? Recommended: not now. Record lag beside the shed count first.
- Does the image route count? Recommended: yes, against the same cap, plus its own relay limit from Link metadata: stream image relays or cap image bytes in flight #786.
Benchmark
Linux before, measured 2026-11-14
Condition: the Linux condition in Bench conditions below. Open loop: void session 20261114T040347. Closed loop: valid session 20261114T050553. Hostile: void session 20261114T042447.
| Step | Answered | p99 ms | CPU % | In flight | mem.current max MiB | Result |
|---|---|---|---|---|---|---|
| miss-fast@1600 (open loop) | 24000 of 24000 | 3.02 | 44.2 | inUseMean 0.14 |
258.1 | held (void session) |
| miss-fast@3200 (open loop) | 48000 of 48000 | 8.13 | 77.3 | inUseMean 0.21 |
286.5 | void step (Spotlight) |
| miss-fast@6400 (open loop) | 43775 of 96000 | 14809.41 | 99.5 | inUseMean 743.92 |
478.9 | OOM kill (void session) |
| miss-fast@c16 (closed loop) | 5186.73/s | 5.92 | 91.6 | 16 | 260.5 | held |
| miss-fast@c64 (closed loop) | 5665.53/s | 24.84 | 96.5 | 64 | 265.3 | held; plateau from c32 |
| hostile@48 | 1440 of 1440 | per lane only | 60.9 | outboundInflightMax 5 |
390.4 | held (void session) |
| hostile@96 | 110 of 2016 | — | — | — | 493.4 | OOM kill (void session) |
- Closed-loop CPU % is the cgroup reading.
- Hostile reports per lane only (knee rule 7). At
@48, lane p50 was 1.5 to 38.8 ms (void session).
Laptop before, measured 2026-11-14 (context)
Condition: the laptop condition in Bench conditions below. At 6400/s REST used 115.1-115.7 % CPU, with p99 10.63-36.37 ms and 0 errors. There was no quota, so no queue grew to a kill.
After
Use the after method in Bench conditions below, on Linux: miss-fast open loop up to 6400/s and hostile up to 96/s. Expected, not measured: no OOM kill; shed answers instead; accepted answers near the closed-loop plateau.
Condition
The after runs on the Linux condition in Bench conditions below, with the same ramps.
Content-safety checks
- A shed answer is never written to the Redis link cache. A good link must not keep a fallback card for 10 min.
- A shed answer names only the requested URL. The bench
cardUrlLeakscount stays 0. - Cache hits and pad links (own cards) are served as today.
UX checks
- Under the cap, the hover card shows the full card as today.
- A shed request shows the fallback card. The next hover asks again, because nothing was cached.
- The webapp does not retry a shed answer in a tight loop.
Rollback
One setting sets the cap to 0, which turns shedding off. Otherwise, revert the commit.
Acceptance criteria
- On Linux at
miss-fast@6400andhostile@96, REST is not OOM-killed. - memory.current stays under 409.6 MiB in both steps.
- Shed answers write 0 Redis keys.
- Accepted answers at
miss-fast@6400keep p99 within 2x of the 1600/s step. - A
Measuredcomment posts the after beside the before, with the condition.
Related
- REPORT.md §1 and §4 rank 2.
research-scaling-options.mdR3. - Link metadata: cap outbound fetches so fast lanes stay alive when origins hang #788 shares the counter. Link metadata: stream image relays or cap image bytes in flight #786 bounds the image route's memory.
Bench conditions
The bench is apps/hocuspocus.server/scripts/bench-link-tolerance/ (commit 1a0d21b7f). The headline numbers are in apps/hocuspocus.server/src/modules/link-metadata/BENCHMARK.md, section Load tolerance. REPORT.md, LINUX-RESULTS.md, research-*.md and raw/ are local bench notes (Notes/local-docs/2026-11-13-link-scraper-tolerance/), not in the repo. The Linux runner is local too.
Linux condition, used by every "Linux before": one REST container at --cpus=1 --memory=512m --memory-swap=512m, in the OrbStack VM on one laptop. Kernel 7.2.12-orbstack-00394-gbecdd6130068, booted with cgroup.memory=nokmem,nosocket, so socket and slab memory are not charged. --no-healthcheck. Bun 1.4.4+38afee6c8. Redis 8.8.0 (redis:alpine) in REST's network namespace. --network none. Pooled loopback to one local origin, no DNS, no TLS. benchTree 8774a6275a05005f, srcTree a9b6ca73ca68c2ec, runnerTree 516099a8c9839eac. Measured 2026-11-14. The core speed ratio to a production vCPU is not measured.
Laptop condition, used by every "Laptop before": local, one laptop (Apple M5 Max, 18 cores, 48 GB, macOS), Bun 1.4.4+38afee6c8, native Redis 8.10.3. REST runs under the sandbox-exec fence. Traffic is plain HTTP to one local origin, pooled loopback, no DNS, no TLS. No CPU quota and no cgroup. benchTree 8774a6275a05005f, srcTree a9b6ca73ca68c2ec. Measured 2026-11-14. Never compare it with a Linux number as one series.
"(void session)" marks a value from a valid step inside a void session. On Linux the OOM kill voided those sessions. The kill is the measured break.
After method, used by every "After":
- Base = the commit before the change. Head = the change.
- Run ABAB, at least 3 sessions per arm, each with its control step, at n ≥ 2,000 per step.
- The change moves
srcTree, so the before must run again. Do not reuse the 2026-11-14 numbers as the before. - The runner takes
src/from the working tree. So it first needs a source-folder option for base (REPORT.md §5.7 item 5). - On Linux, start each session in a fresh REST container, and keep the run agent alive until the footer is written.
- Quote a change only when every head session sits outside the base range. Give σ and the MDE beside it.
Paths are under apps/hocuspocus.server/src/ unless they say otherwise.
Activity
fix(link-metadata): shed cold work past 64 in flight
docs(link-metadata): record the slot 0 results and shed answers
HMarzban commentedon Nov 15, 2026
Measured: before and after (slot 1, 2026-11-15)
Setup. One REST container at
--cpus=1 --memory=512m(OrbStack, Linux cgroup v2), on one laptop. The client reads at full speed. Both arms were built fromgit archive: basedf910c6b1, heade5d3e92b9(cap 64), with the same bench. Base and head ran in turns (ABAB, BABA, ABAB). Each cell is the range over 3 valid sessions per arm. A step is void above 250 % other-process CPU. An OOM kill counts as a valid memory break (maintainer rulings, 2026-11-14 and 2026-11-15).200a012c1d7fand on38a969bc0, where base survived. It was not committed.Status. Shipped in 2.3.6 (
e5d3e92b9). Canonical table: moduleBENCHMARK.md, "After #787 to #792". Raw data and runner:Notes/local-docs/2026-11-13-link-scraper-tolerance/raw/linux/s1r/,s5/,s9/(local).HMarzban commentedon Nov 15, 2026
Shipped in 2.3.6 (
e5d3e92b9). The before and after is in the Measured comment: #787 (comment)The hang cost stays open: beside hanging hosts, good links get a fallback card (60 % at 20 hanging links/s, 81 % at 40). That work continues in #790 (a smaller waiting bound) and #788 (a per-host cap).
Closing as done. The issue stays in milestone v2.3.8 as the record of the load-tolerance work.