Repository navigation
Description
Summary
Production has no link-metadata metric. Nobody can see requests in flight, outbound pool use, queue waits, cache hits or parse time. The bench found an OOM kill and a full pool at production limits, and production gets no warning before either one. Two Rust reopen triggers cannot be read today: parse above 10 % of request time at p95, and lag above 50 ms from parsing.
Measured
- Production, 7 days to 2026-11-10 (
research-prod-facts.md):/api/metadatap95 0.92 s and icon p95 0.82 s. The REST p95 SLO is 1 s, and it mixes every route. - The
stageslog field mixes DNS, fetch, read and parse in onedurationMs. - Linux bench: 4 OOM kills, and the pool bound first for hanging hosts (REPORT.md §1). Hostile and image lag passed 50 ms, with no parse attribution (REPORT.md §3).
Cause
- The module emits no Prometheus series. The only trace is the
stageslog field.
What to change
- A gauge of cold link-metadata requests in flight, and a counter of shed answers (with Link metadata: shed cold work before the queue grows to an OOM kill #787).
- A gauge of link fetches in flight and queued (with Link metadata: cap outbound fetches so fast lanes stay alive when origins hang #788).
- A gauge of image bytes in flight, and counters of image relay bytes and 204 answers (with Link metadata: stream image relays or cap image bytes in flight #786).
- A histogram of stage duration, by stage and outcome.
- A histogram of parse time (the
HTMLRewriterpass), and of parse share of request time. - A counter of parses over 50 ms.
- A counter of cache read outcomes: hit, miss and timeout (with Link metadata: time-bound Redis calls on the request path #789).
- A panel in
app-performance.json, and a non-paging rule for pool saturation.
Decisions
- Label by host? Recommended: no. Use stage and outcome only, to keep cardinality low.
- Time every parse, or a sample? Recommended: every parse, if the overhead A/B shows no cost. Other: a 1-in-10 sample.
- Alert on the Rust trigger? Recommended: a non-paging rule on parse share p95 above 10 % for 1 h.
Benchmark
Laptop before, measured 2026-11-14
Condition: the laptop condition in Bench conditions below. Layer-1 miss-fast: 0.000174-0.000189 CPU-s per answer (7 fits). p99 at 3200/s 1.82-2.07 ms (3 sessions).
After
Use the after method in Bench conditions below, on miss-fast at 3200/s. Accept the change when every head session sits inside the base range for CPU per answer and p99.
Condition
The laptop condition in Bench conditions below for the overhead A/B. A Linux A/B is optional.
Content-safety checks
- No metric label holds a URL, a host or a pad id.
UX checks
- No reader-facing change. The overhead A/B must sit inside noise.
Rollback
Revert the commit. The series have no reader-facing effect.
Acceptance criteria
- The series show on the production dashboard.
- The overhead A/B is in a comment and sits inside noise.
- After 7 days, a comment posts the parse share p95 and the count of parses over 50 ms.
Related
- REPORT.md §3 (Rust ruling check) and §4 rank 7.
- Link metadata: stream image relays or cap image bytes in flight #786 to Link metadata: time-bound Redis calls on the request path #789 feed the in-flight and cache series.
Bench conditions
The bench is apps/hocuspocus.server/scripts/bench-link-tolerance/ (commit 1a0d21b7f). The headline numbers are in apps/hocuspocus.server/src/modules/link-metadata/BENCHMARK.md, section Load tolerance. REPORT.md, LINUX-RESULTS.md, research-*.md and raw/ are local bench notes (Notes/local-docs/2026-11-13-link-scraper-tolerance/), not in the repo. The Linux runner is local too.
Linux condition, used by every "Linux before": one REST container at --cpus=1 --memory=512m --memory-swap=512m, in the OrbStack VM on one laptop. Kernel 7.2.12-orbstack-00394-gbecdd6130068, booted with cgroup.memory=nokmem,nosocket, so socket and slab memory are not charged. --no-healthcheck. Bun 1.4.4+38afee6c8. Redis 8.8.0 (redis:alpine) in REST's network namespace. --network none. Pooled loopback to one local origin, no DNS, no TLS. benchTree 8774a6275a05005f, srcTree a9b6ca73ca68c2ec, runnerTree 516099a8c9839eac. Measured 2026-11-14. The core speed ratio to a production vCPU is not measured.
Laptop condition, used by every "Laptop before": local, one laptop (Apple M5 Max, 18 cores, 48 GB, macOS), Bun 1.4.4+38afee6c8, native Redis 8.10.3. REST runs under the sandbox-exec fence. Traffic is plain HTTP to one local origin, pooled loopback, no DNS, no TLS. No CPU quota and no cgroup. benchTree 8774a6275a05005f, srcTree a9b6ca73ca68c2ec. Measured 2026-11-14. Never compare it with a Linux number as one series.
"(void session)" marks a value from a valid step inside a void session. On Linux the OOM kill voided those sessions. The kill is the measured break.
After method, used by every "After":
- Base = the commit before the change. Head = the change.
- Run ABAB, at least 3 sessions per arm, each with its control step, at n ≥ 2,000 per step.
- The change moves
srcTree, so the before must run again. Do not reuse the 2026-11-14 numbers as the before. - The runner takes
src/from the working tree. So it first needs a source-folder option for base (REPORT.md §5.7 item 5). - On Linux, start each session in a fresh REST container, and keep the run agent alive until the footer is written.
- Quote a change only when every head session sits outside the base range. Give σ and the MDE beside it.
Paths are under apps/hocuspocus.server/src/ unless they say otherwise.
Activity
feat(link-metadata): count shed, joined and timed-out work
HMarzban commentedon Nov 15, 2026
Measured: overhead (slot 4, 2026-11-15)
Setup. One REST container at
--cpus=1 --memory=512m(OrbStack, Linux cgroup v2), on one laptop. Both arms were built fromgit archive: basef62a5e383, headd216267d9, with the same bench. Base and head ran in turns (ABAB, BABA, ABAB), plus one head rerun. 3 valid sessions per arm. A step is void above 250 % other-process CPU (maintainer ruling, 2026-11-14).Fast misses at 16 in flight. This lane is CPU-bound (about 92.5 % of one core), so any cost of the metrics would show here first.
chrome-headless-shell(1144.1 % other-process CPU), so it is void. Its rerun is valid and inside the range above.Status. Shipped in 2.3.6 (
d216267d9). Canonical text: moduleBENCHMARK.md, "After #787 to #792". Raw data:Notes/local-docs/2026-11-13-link-scraper-tolerance/raw/linux/s4/(local).HMarzban commentedon Nov 15, 2026
Shipped in 2.3.6 (
d216267d9). The before and after is in the Measured comment: #792 (comment)Closing as done. The issue stays in milestone v2.3.8 as the record of the load-tolerance work.