Skip to content

Link metadata: metrics for in-flight work, the outbound pool, cache, stages and parse time #792

Closed

Description

@HMarzban

Summary

Production has no link-metadata metric. Nobody can see requests in flight, outbound pool use, queue waits, cache hits or parse time. The bench found an OOM kill and a full pool at production limits, and production gets no warning before either one. Two Rust reopen triggers cannot be read today: parse above 10 % of request time at p95, and lag above 50 ms from parsing.

Measured

  • Production, 7 days to 2026-11-10 (research-prod-facts.md): /api/metadata p95 0.92 s and icon p95 0.82 s. The REST p95 SLO is 1 s, and it mixes every route.
  • The stages log field mixes DNS, fetch, read and parse in one durationMs.
  • Linux bench: 4 OOM kills, and the pool bound first for hanging hosts (REPORT.md §1). Hostile and image lag passed 50 ms, with no parse attribution (REPORT.md §3).

Cause

  • The module emits no Prometheus series. The only trace is the stages log field.

What to change

  1. A gauge of cold link-metadata requests in flight, and a counter of shed answers (with Link metadata: shed cold work before the queue grows to an OOM kill #787).
  2. A gauge of link fetches in flight and queued (with Link metadata: cap outbound fetches so fast lanes stay alive when origins hang #788).
  3. A gauge of image bytes in flight, and counters of image relay bytes and 204 answers (with Link metadata: stream image relays or cap image bytes in flight #786).
  4. A histogram of stage duration, by stage and outcome.
  5. A histogram of parse time (the HTMLRewriter pass), and of parse share of request time.
  6. A counter of parses over 50 ms.
  7. A counter of cache read outcomes: hit, miss and timeout (with Link metadata: time-bound Redis calls on the request path #789).
  8. A panel in app-performance.json, and a non-paging rule for pool saturation.

Decisions

  1. Label by host? Recommended: no. Use stage and outcome only, to keep cardinality low.
  2. Time every parse, or a sample? Recommended: every parse, if the overhead A/B shows no cost. Other: a 1-in-10 sample.
  3. Alert on the Rust trigger? Recommended: a non-paging rule on parse share p95 above 10 % for 1 h.

Benchmark

Laptop before, measured 2026-11-14

Condition: the laptop condition in Bench conditions below. Layer-1 miss-fast: 0.000174-0.000189 CPU-s per answer (7 fits). p99 at 3200/s 1.82-2.07 ms (3 sessions).

After

Use the after method in Bench conditions below, on miss-fast at 3200/s. Accept the change when every head session sits inside the base range for CPU per answer and p99.

Condition

The laptop condition in Bench conditions below for the overhead A/B. A Linux A/B is optional.

Content-safety checks

  • No metric label holds a URL, a host or a pad id.

UX checks

  • No reader-facing change. The overhead A/B must sit inside noise.

Rollback

Revert the commit. The series have no reader-facing effect.

Acceptance criteria

  • The series show on the production dashboard.
  • The overhead A/B is in a comment and sits inside noise.
  • After 7 days, a comment posts the parse share p95 and the count of parses over 50 ms.

Related

Bench conditions

The bench is apps/hocuspocus.server/scripts/bench-link-tolerance/ (commit 1a0d21b7f). The headline numbers are in apps/hocuspocus.server/src/modules/link-metadata/BENCHMARK.md, section Load tolerance. REPORT.md, LINUX-RESULTS.md, research-*.md and raw/ are local bench notes (Notes/local-docs/2026-11-13-link-scraper-tolerance/), not in the repo. The Linux runner is local too.

Linux condition, used by every "Linux before": one REST container at --cpus=1 --memory=512m --memory-swap=512m, in the OrbStack VM on one laptop. Kernel 7.2.12-orbstack-00394-gbecdd6130068, booted with cgroup.memory=nokmem,nosocket, so socket and slab memory are not charged. --no-healthcheck. Bun 1.4.4+38afee6c8. Redis 8.8.0 (redis:alpine) in REST's network namespace. --network none. Pooled loopback to one local origin, no DNS, no TLS. benchTree 8774a6275a05005f, srcTree a9b6ca73ca68c2ec, runnerTree 516099a8c9839eac. Measured 2026-11-14. The core speed ratio to a production vCPU is not measured.

Laptop condition, used by every "Laptop before": local, one laptop (Apple M5 Max, 18 cores, 48 GB, macOS), Bun 1.4.4+38afee6c8, native Redis 8.10.3. REST runs under the sandbox-exec fence. Traffic is plain HTTP to one local origin, pooled loopback, no DNS, no TLS. No CPU quota and no cgroup. benchTree 8774a6275a05005f, srcTree a9b6ca73ca68c2ec. Measured 2026-11-14. Never compare it with a Linux number as one series.

"(void session)" marks a value from a valid step inside a void session. On Linux the OOM kill voided those sessions. The kill is the measured break.

After method, used by every "After":

  • Base = the commit before the change. Head = the change.
  • Run ABAB, at least 3 sessions per arm, each with its control step, at n ≥ 2,000 per step.
  • The change moves srcTree, so the before must run again. Do not reuse the 2026-11-14 numbers as the before.
  • The runner takes src/ from the working tree. So it first needs a source-folder option for base (REPORT.md §5.7 item 5).
  • On Linux, start each session in a fresh REST container, and keep the run agent alive until the footer is written.
  • Quote a change only when every head session sits outside the base range. Give σ and the MDE beside it.

Paths are under apps/hocuspocus.server/src/ unless they say otherwise.

Activity

  1. added this to the v2.3.6 milestone on Nov 14, 2026
  2. modified the milestones: v2.3.6, v2.3.8 on Nov 14, 2026
  3. HMarzban commented on Nov 15, 2026

    @HMarzban
    CollaboratorAuthor

    Measured: overhead (slot 4, 2026-11-15)

    Setup. One REST container at --cpus=1 --memory=512m (OrbStack, Linux cgroup v2), on one laptop. Both arms were built from git archive: base f62a5e383, head d216267d9, with the same bench. Base and head ran in turns (ABAB, BABA, ABAB), plus one head rerun. 3 valid sessions per arm. A step is void above 250 % other-process CPU (maintainer ruling, 2026-11-14).

    Fast misses at 16 in flight. This lane is CPU-bound (about 92.5 % of one core), so any cost of the metrics would show here first.

    Before After
    Answers per second 4288.2–4974.07 4282.67–4998.47
    p95 4.7–5.63 ms 4.61–5.67 ms
    p99 7.21–9.92 ms 7.15–8.79 ms
    • The ranges overlap fully. The metrics cost is below what this bench can see.
    • So ruling D5 (time every parse) stands.
    • One head session ran beside another project's chrome-headless-shell (1144.1 % other-process CPU), so it is void. Its rerun is valid and inside the range above.

    Status. Shipped in 2.3.6 (d216267d9). Canonical text: module BENCHMARK.md, "After #787 to #792". Raw data: Notes/local-docs/2026-11-13-link-scraper-tolerance/raw/linux/s4/ (local).

  4. HMarzban commented on Nov 15, 2026

    @HMarzban
    CollaboratorAuthor

    Shipped in 2.3.6 (d216267d9). The before and after is in the Measured comment: #792 (comment)

    Closing as done. The issue stays in milestone v2.3.8 as the record of the load-tolerance work.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions