Skip to content

Link metadata: shed cold work before the queue grows to an OOM kill #787

Closed

Description

@HMarzban

Summary

REST has no limit on link-metadata work in flight. When arrivals outrun one CPU, the queue grows until the kernel kills REST. On one Linux replica at production limits, this happened at 6400 fast misses/s and at 96 hostile pages/s. When the driver bounded the work in flight, memory stayed flat at the same CPU ceiling.

Measured

See Benchmark. In short:

  • miss-fast@6400 (void session): CPU throttled in 90 of 140 periods. inUseMean 743.92, outboundInUseMax 1570, anon 400.3 MiB. The OOM kill came during the drain.
  • hostile@96 (void session): CPU hit the quota in the warm-up. Memory reached 512 MiB at t 1.7 s, and the kill came at about t 2.9 s.
  • At hostile@48, only 5 fetches were in flight (outboundInflightMax 5). So the hostile queue is CPU work, and the outbound cap in Link metadata: cap outbound fetches so fast lanes stay alive when origins hang #788 does not bound it.
  • Closed loop, valid session: the driver held 16 to 64 requests in flight. Throughput reached 5665.53/s with 0 errors, and memory.current stayed at 265.3 MiB or less. The driver set this bound, not REST.

Cause

  • No route has a concurrency limit, a singleflight or an in-flight dedupe (research-code-limits.md).
  • The limiter caps volume per IP per 15 min. With execEvenly: false, one IP may spend its whole bucket at once. It does not cap concurrency (research-code-limits.md).
  • Traefik's rate-limit-api (average: 800, burst: 900) is likely per source IP (unverified). It sits above the hostile break rate (96/s). For one source IP, it sits below the 6400/s miss kill, but it does not bound a flood from many addresses.

What to change

  1. Count cold link-metadata requests in flight per process, in the module shared with Link metadata: stream image relays or cap image bytes in flight #786 and Link metadata: cap outbound fetches so fast lanes stay alive when origins hang #788.
  2. Over the cap, skip the stages and answer at once. Cache hits are always served. Health routes are never shed.
  3. Count shed answers (Link metadata: metrics for in-flight work, the outbound pool, cache, stages and parse time #792).

Decisions

  1. What cap? Recommended: 64 cold requests in flight per replica. In the closed loop, 64 in flight reached the CPU plateau (5665.53/s) with flat memory. The cost: real-delay misses cap near 64 / 0.36 s, about 178/s per replica (a projection). Production serves about 0.023 link requests per second. Other: 128.
  2. What does a shed request get? Recommended: the fallback card with miss: 'unreachable' and Cache-Control: no-store, with no Redis write. research-scaling-options.md R3 says the webapp needs no change for it. Other: 503 with Retry-After, which needs a client check.
  3. Shed on event-loop lag too? Recommended: not now. Record lag beside the shed count first.
  4. Does the image route count? Recommended: yes, against the same cap, plus its own relay limit from Link metadata: stream image relays or cap image bytes in flight #786.

Benchmark

Linux before, measured 2026-11-14

Condition: the Linux condition in Bench conditions below. Open loop: void session 20261114T040347. Closed loop: valid session 20261114T050553. Hostile: void session 20261114T042447.

Step Answered p99 ms CPU % In flight mem.current max MiB Result
miss-fast@1600 (open loop) 24000 of 24000 3.02 44.2 inUseMean 0.14 258.1 held (void session)
miss-fast@3200 (open loop) 48000 of 48000 8.13 77.3 inUseMean 0.21 286.5 void step (Spotlight)
miss-fast@6400 (open loop) 43775 of 96000 14809.41 99.5 inUseMean 743.92 478.9 OOM kill (void session)
miss-fast@c16 (closed loop) 5186.73/s 5.92 91.6 16 260.5 held
miss-fast@c64 (closed loop) 5665.53/s 24.84 96.5 64 265.3 held; plateau from c32
hostile@48 1440 of 1440 per lane only 60.9 outboundInflightMax 5 390.4 held (void session)
hostile@96 110 of 2016 — — — 493.4 OOM kill (void session)
  • Closed-loop CPU % is the cgroup reading.
  • Hostile reports per lane only (knee rule 7). At @48, lane p50 was 1.5 to 38.8 ms (void session).

Laptop before, measured 2026-11-14 (context)

Condition: the laptop condition in Bench conditions below. At 6400/s REST used 115.1-115.7 % CPU, with p99 10.63-36.37 ms and 0 errors. There was no quota, so no queue grew to a kill.

After

Use the after method in Bench conditions below, on Linux: miss-fast open loop up to 6400/s and hostile up to 96/s. Expected, not measured: no OOM kill; shed answers instead; accepted answers near the closed-loop plateau.

Condition

The after runs on the Linux condition in Bench conditions below, with the same ramps.

Content-safety checks

  • A shed answer is never written to the Redis link cache. A good link must not keep a fallback card for 10 min.
  • A shed answer names only the requested URL. The bench cardUrlLeaks count stays 0.
  • Cache hits and pad links (own cards) are served as today.

UX checks

  • Under the cap, the hover card shows the full card as today.
  • A shed request shows the fallback card. The next hover asks again, because nothing was cached.
  • The webapp does not retry a shed answer in a tight loop.

Rollback

One setting sets the cap to 0, which turns shedding off. Otherwise, revert the commit.

Acceptance criteria

  • On Linux at miss-fast@6400 and hostile@96, REST is not OOM-killed.
  • memory.current stays under 409.6 MiB in both steps.
  • Shed answers write 0 Redis keys.
  • Accepted answers at miss-fast@6400 keep p99 within 2x of the 1600/s step.
  • A Measured comment posts the after beside the before, with the condition.

Related

Bench conditions

The bench is apps/hocuspocus.server/scripts/bench-link-tolerance/ (commit 1a0d21b7f). The headline numbers are in apps/hocuspocus.server/src/modules/link-metadata/BENCHMARK.md, section Load tolerance. REPORT.md, LINUX-RESULTS.md, research-*.md and raw/ are local bench notes (Notes/local-docs/2026-11-13-link-scraper-tolerance/), not in the repo. The Linux runner is local too.

Linux condition, used by every "Linux before": one REST container at --cpus=1 --memory=512m --memory-swap=512m, in the OrbStack VM on one laptop. Kernel 7.2.12-orbstack-00394-gbecdd6130068, booted with cgroup.memory=nokmem,nosocket, so socket and slab memory are not charged. --no-healthcheck. Bun 1.4.4+38afee6c8. Redis 8.8.0 (redis:alpine) in REST's network namespace. --network none. Pooled loopback to one local origin, no DNS, no TLS. benchTree 8774a6275a05005f, srcTree a9b6ca73ca68c2ec, runnerTree 516099a8c9839eac. Measured 2026-11-14. The core speed ratio to a production vCPU is not measured.

Laptop condition, used by every "Laptop before": local, one laptop (Apple M5 Max, 18 cores, 48 GB, macOS), Bun 1.4.4+38afee6c8, native Redis 8.10.3. REST runs under the sandbox-exec fence. Traffic is plain HTTP to one local origin, pooled loopback, no DNS, no TLS. No CPU quota and no cgroup. benchTree 8774a6275a05005f, srcTree a9b6ca73ca68c2ec. Measured 2026-11-14. Never compare it with a Linux number as one series.

"(void session)" marks a value from a valid step inside a void session. On Linux the OOM kill voided those sessions. The kill is the measured break.

After method, used by every "After":

  • Base = the commit before the change. Head = the change.
  • Run ABAB, at least 3 sessions per arm, each with its control step, at n ≥ 2,000 per step.
  • The change moves srcTree, so the before must run again. Do not reuse the 2026-11-14 numbers as the before.
  • The runner takes src/ from the working tree. So it first needs a source-folder option for base (REPORT.md §5.7 item 5).
  • On Linux, start each session in a fresh REST container, and keep the run agent alive until the footer is written.
  • Quote a change only when every head session sits outside the base range. Give σ and the MDE beside it.

Paths are under apps/hocuspocus.server/src/ unless they say otherwise.

Activity

  1. added this to the v2.3.6 milestone on Nov 14, 2026
  2. added
    bugSomething isn't working
    SecuritySecurity, access control, and data exposure
    PerformanceQuery cost, load, and capacity
    on Nov 14, 2026
  3. modified the milestones: v2.3.6, v2.3.8 on Nov 14, 2026
  4. HMarzban commented on Nov 15, 2026

    @HMarzban
    CollaboratorAuthor

    Measured: before and after (slot 1, 2026-11-15)

    Setup. One REST container at --cpus=1 --memory=512m (OrbStack, Linux cgroup v2), on one laptop. The client reads at full speed. Both arms were built from git archive: base df910c6b1, head e5d3e92b9 (cap 64), with the same bench. Base and head ran in turns (ABAB, BABA, ABAB). Each cell is the range over 3 valid sessions per arm. A step is void above 250 % other-process CPU. An OOM kill counts as a valid memory break (maintainer rulings, 2026-11-14 and 2026-11-15).

    Lane Before After Change
    Fast misses, 6400/s OOM-killed in 3 of 3 Up in 3 of 3; 96000 answered, 30194–45519 fallback cards, 267.6–307.5 MiB no OOM kill
    Images, 200/s: answers 200 1133–1166 1471–1480 26.2–30.6 % more
    Images, 200/s: p95 7573.41–7600.39 ms 4026.97–4048.37 ms 46.5–47.0 % less time
    Images, 200/s: reset streams 808–830 0 100 % fewer
    Good links beside 20 hanging links/s: fallback 0 of 2000 1208–1211 of 2000 cost: 60 %
    Good links beside 40 hanging links/s: fallback 0 of 2000 1616–1617 of 2000 cost: 81 %
    • The cap does what it was built for: a miss flood and an image flood no longer kill REST.
    • The open cost: a hanging host holds one of the 64 slots for seconds. So beside hanging hosts, good links get a fallback card.
    • A fix was tried twice and failed. W2 lets a request that waits for headers hold no working slot (192 waiting plus 64 working, with one deadline per request, Link metadata: one deadline per request #790). It brought the fallback to 0 and 417–418 of 2000. But REST was OOM-killed under hostile pages in 3 of 3 sessions, both on a012c1d7f and on 38a969bc0, where base survived. It was not committed.
    • Options left: a smaller waiting bound (needs a bench), the per-host cap from Link metadata: cap outbound fetches so fast lanes stay alive when origins hang #788, or accept the cost.

    Status. Shipped in 2.3.6 (e5d3e92b9). Canonical table: module BENCHMARK.md, "After #787 to #792". Raw data and runner: Notes/local-docs/2026-11-13-link-scraper-tolerance/raw/linux/s1r/, s5/, s9/ (local).

  5. HMarzban commented on Nov 15, 2026

    @HMarzban
    CollaboratorAuthor

    Shipped in 2.3.6 (e5d3e92b9). The before and after is in the Measured comment: #787 (comment)

    The hang cost stays open: beside hanging hosts, good links get a fallback card (60 % at 20 hanging links/s, 81 % at 40). That work continues in #790 (a smaller waiting bound) and #788 (a per-host cap).

    Closing as done. The issue stays in milestone v2.3.8 as the record of the load-tolerance work.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    PerformanceQuery cost, load, and capacitySecuritySecurity, access control, and data exposurebugSomething isn't working

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions