Skip to content

Link metadata: stream image relays or cap image bytes in flight #786

Closed

Description

@HMarzban

Summary

At production limits, the image relay is the one route whose first break is memory. One Linux replica at --cpus=1 --memory=512m reached memory.max at 100 images/s. REST was OOM-killed at 100/s in one run, and in the 200/s warm-up in the other. The relay holds each image whole in memory, up to 2 MiB, in about three copies. Nothing caps relays in flight.

Measured

See Benchmark. In short:

  • Linux, at 100/s: memory.peak 512 MiB and 2092 memory.max events, with 2000 of 2000 answered and CPU at 42.1 % (void session).
  • The side RSS fit is 1.0868 MiB per image in flight, intercept 227.6 MiB, r² 0.997 (void session).
  • The 80 % line (409.6 MiB) fell at 25/s (run 1) and 50/s (run 2), with page cache counted. anon alone was 379.2 MiB and 400.2 MiB there.
  • research-scaling-options.md R6 said it "Moves up if RSS passes about 400M before 200 concurrent relays". Linux read 417.1 MiB with 131 in flight (void session).
  • The laptop showed side RSS peaks of 263.1-787.5 MiB at 100/s. No laptop RSS is compared with 512 MiB.

Cause

  • readCappedBody keeps a chunk list, then Buffer.concats it (lib/readCappedBody.ts:15-32).
  • fetchImage copies again with new Uint8Array(bytes) (modules/link-metadata/domain/relay.ts:59).
  • The own-card path reads response.arrayBuffer() and copies it into a Uint8Array (modules/link-metadata/http/imageController.ts:30).
  • IMAGE_MAX_BYTES is 2 MiB (modules/link-metadata/http/imageController.ts:11). No cap limits relays in flight, and no server cache keeps image bytes.
  • The oEmbed stage reads response.json() with no byte cap (modules/link-metadata/domain/stages/oembed.ts:100).

What to change

  1. Stream the relay. Read until at least 16 bytes arrive, sniff them with sniffImageType, then pass the rest through a counting stream. Over the cap, abort the upstream read.
  2. If streaming is not chosen, cap bytes in flight instead: a process-wide budget of in-flight image bytes, and a cap on relays in flight. Over budget, answer 204 at once.
  3. Remove the extra copy in relay.ts:59 in either case.
  4. Read oEmbed bodies through readApiJson, with its 64 KiB cap.
  5. Count relays in flight, bytes in flight and 204 answers (Link metadata: metrics for in-flight work, the outbound pool, cache, stages and parse time #792).

Decisions

  1. Stream, or cap bytes? Recommended: stream when the upstream declares a Content-Length of 2 MiB or less. Memory then follows chunks in flight, not whole images. Other: buffer with a 128 MiB byte budget and a cap of 32 relays, which keeps today's clean 204.
  2. What about a body with no Content-Length? Recommended: buffer it under the byte budget from step 2. Then a cut stream happens only when a peer sends more than it declared.
  3. Share the in-flight counter with Link metadata: shed cold work before the queue grows to an OOM kill #787 and Link metadata: cap outbound fetches so fast lanes stay alive when origins hang #788? Recommended: yes, one module for every in-flight count.

Benchmark

Linux before, measured 2026-11-14

Condition: the Linux condition in Bench conditions below. Sessions 20261114T041855 (run 2) and 20261114T041519 (run 1), both void by OOM kill. 30 s steps; @100 20 s. Run 2 is the main read.

Step Throughput/s p95 ms mem.current max / peak MiB memory.max events CPU % inUseMean
@10 9.97 2585.36 332.3 / 376.4 0 14.6 26.41
@25 24.9 2595.85 384.1 / 396.5 0 21.4 65.72
@50 49.9 2570.88 417.1 / 439.1 0 32.4 131
@100 99.5 2600.71 497.6 / 512 2092 42.1 261.47
@200 0 of 3000 answered — OOM kill about 2.3 s into the warm-up — — —
  • Run 1: memory.current 436.6 MiB at @25. @100 was void: 1623 of 2000 answered, 18.85 % errors, 731 memory.max events, OOM kill. A host bfs process ran at 236 to 517 % CPU.
  • The outbound pool also hit its cap at @100 in both runs (originStreamingMax 256).

Laptop before, measured 2026-11-14 (context)

Condition: the laptop condition in Bench conditions below. Sessions 20261114T003057, 20261114T003512, 20261114T003927, 20261114T005905. RSS is side memoryUsage().rss at 1 Hz. It is never compared with 512 MiB.

Offered Degraded % Fetches in use RSS steady / max MiB
10 0 26.2 133.8-138.9 / 141-165.6
50 0 129.6 134.3-139.6 / 195.3-406.1
100 0 258.21-258.47 147.5-197.2 / 263.1-787.5
125 9.95 504.1 137.9 / 198.5
200 61.2-61.3 1124-1132 159.5-218.3 / 493.6-856.7

After

Use the after method in Bench conditions below, on Linux, with the image ramp at 10, 25, 50, 100 and 200/s. Expected, not measured: memory.current stops growing with images in flight. Report each step's memory.current max, memory.max events, oom_kill and 204 count.

Condition

The after runs on the Linux condition in Bench conditions below, with the same image ramp and origin. Name the VM, nokmem,nosocket and the missing healthcheck in the Measured comment.

Content-safety checks

  • The served Content-Type still comes from the sniffed bytes only. SVG never passes.
  • The sniff runs before any header or byte goes to the client.
  • Every hop still goes through safeFetch, and the reader's browser still never contacts the linked site.
  • A cut stream must not stay in a browser cache under max-age=86400. Check this in Chrome and Safari before the change ships.
  • Pad links (own cards) keep their current path.

UX checks

  • At 10 to 100/s, the hover card and the link preview show the image as today.
  • A 204 shows the card without an image, as today.
  • The time to the first image byte does not rise.

Rollback

One setting turns streaming off and restores the buffered path. Otherwise, revert the commit.

Acceptance criteria

  • On Linux at image 200/s, REST is not OOM-killed, and memory.events oom_kill stays 0.
  • At image 100/s, memory.max events are 0, and memory.current stays under 409.6 MiB.
  • SVG and non-image bodies still answer 204.
  • A Measured comment posts the after beside the before, with the condition.

Related

Bench conditions

The bench is apps/hocuspocus.server/scripts/bench-link-tolerance/ (commit 1a0d21b7f). The headline numbers are in apps/hocuspocus.server/src/modules/link-metadata/BENCHMARK.md, section Load tolerance. REPORT.md, LINUX-RESULTS.md, research-*.md and raw/ are local bench notes (Notes/local-docs/2026-11-13-link-scraper-tolerance/), not in the repo. The Linux runner is local too.

Linux condition, used by every "Linux before": one REST container at --cpus=1 --memory=512m --memory-swap=512m, in the OrbStack VM on one laptop. Kernel 7.2.12-orbstack-00394-gbecdd6130068, booted with cgroup.memory=nokmem,nosocket, so socket and slab memory are not charged. --no-healthcheck. Bun 1.4.4+38afee6c8. Redis 8.8.0 (redis:alpine) in REST's network namespace. --network none. Pooled loopback to one local origin, no DNS, no TLS. benchTree 8774a6275a05005f, srcTree a9b6ca73ca68c2ec, runnerTree 516099a8c9839eac. Measured 2026-11-14. The core speed ratio to a production vCPU is not measured.

Laptop condition, used by every "Laptop before": local, one laptop (Apple M5 Max, 18 cores, 48 GB, macOS), Bun 1.4.4+38afee6c8, native Redis 8.10.3. REST runs under the sandbox-exec fence. Traffic is plain HTTP to one local origin, pooled loopback, no DNS, no TLS. No CPU quota and no cgroup. benchTree 8774a6275a05005f, srcTree a9b6ca73ca68c2ec. Measured 2026-11-14. Never compare it with a Linux number as one series.

"(void session)" marks a value from a valid step inside a void session. On Linux the OOM kill voided those sessions. The kill is the measured break.

After method, used by every "After":

  • Base = the commit before the change. Head = the change.
  • Run ABAB, at least 3 sessions per arm, each with its control step, at n ≥ 2,000 per step.
  • The change moves srcTree, so the before must run again. Do not reuse the 2026-11-14 numbers as the before.
  • The runner takes src/ from the working tree. So it first needs a source-folder option for base (REPORT.md §5.7 item 5).
  • On Linux, start each session in a fresh REST container, and keep the run agent alive until the footer is written.
  • Quote a change only when every head session sits outside the base range. Give σ and the MDE beside it.

Paths are under apps/hocuspocus.server/src/ unless they say otherwise.

Activity

  1. added this to the v2.3.6 milestone on Nov 14, 2026
  2. added
    bugSomething isn't working
    SecuritySecurity, access control, and data exposure
    PerformanceQuery cost, load, and capacity
    on Nov 14, 2026
  3. modified the milestones: v2.3.6, v2.3.8 on Nov 14, 2026
  4. HMarzban commented on Nov 14, 2026

    @HMarzban
    CollaboratorAuthor

    Measured: after, 2026-11-14

    Change. relayImage (domain/relay.ts) streams an image that declares its Content-Length (2 MiB at most), after the sniff. An image with no declared length is read whole, at most 32 at once. Past that the route answers 204 with Cache-Control: no-store. oEmbed now reads through readApiJson (64 KiB cap). The copy in relay.ts:59 is gone.

    Commits. 9d446b65d (bench: a declared-length image lane) and daad326fe (the fix), on main. Not deployed yet.

    Condition. The Linux condition in this issue: one REST container at --cpus=1 --memory=512m --memory-swap=512m, OrbStack VM, one Apple M5 Max laptop, Bun 1.4.4+38afee6c8, Redis 8.8.0, pooled loopback, no DNS, no TLS. 2 MiB images from a 2.5 s origin.

    • Base: the code at 9f0e2be02 (srcTreeNoTests 7d1df39a060799fb). Head: the change (srcTreeNoTests e9c71b4893743850). Same bench in both images (benchTree e45612b12a568b64).
    • Base and head ran in turns (ABAB), 3 sessions per arm per lane. Sessions 20261114T075104 to 20261114T083003.
    • Two lanes. "Declared length" is new: the fake origin serves it from a node:http listener, because Bun's server sends every stream chunked. "No declared length" is the lane of the before.

    Every session is void. Another project's Next dev server and Playwright tests ran on the host the whole time (other-process CPU above 150 %). Each base session also ended in an OOM kill, which is the measured break. Base and head took turns, so both arms shared the noise. Values are the range over 3 sessions.

    Lane, rate Base memory.current max Base answers Head memory.current max Head answers
    Declared length, 25/s 329.3-389 MiB 750 × 200 218.8-223.3 MiB 750 × 200
    Declared length, 50/s 414.4-433.8 MiB 1500 × 200 247.1-252 MiB 1500 × 200
    Declared length, 100/s 486.3-496.4 MiB 1568-2000 × 200 332.2-334.9 MiB 2000 × 200
    Declared length, 200/s OOM kill (3 of 3) 0 371.7-395.3 MiB 1163-1172 × 200, 999-1003 × 204, 825-835 reset
    No declared length, 25/s 340.1-360.3 MiB 750 × 200 270.1-280.3 MiB 388-390 × 200, 360-362 × 204
    No declared length, 100/s 496.1-510.8 MiB 2000 × 200 298.9-302.7 MiB 256 × 200, 1744 × 204
    No declared length, 200/s OOM kill (3 of 3) 0 269.2-319.1 MiB 192 × 200, 2808 × 204

    Base at 100/s also had up to 469 memory.max events (no declared length) and 87 (declared length). Head had 0 at every step. One base session was killed at 100/s.

    Read.

    • Declared length: every head session sits outside the base range on memory at 25, 50 and 100/s. At 200/s REST stays up. The outbound pool then fills (p95 7513.58-7562.45 ms, Link metadata: cap outbound fetches so fast lanes stay alive when origins hang #788). The 5 s timer turns fetches still waiting into 204, and cuts 825-835 streams already sending. Before, a late image was a 204 cached for 600 s. Now a cut stream is a failed load, never cached.
    • No declared length: the head serves about 13 images per second (32 reads of 2.5 s) and answers 204 for the rest. Memory stays near 300 MiB.

    Final code. A review after these runs changed three things. A coded body (content-encoding other than identity) now counts as undeclared, because fetch unzips it but keeps the coded length. One AbortSignal.timeout holds the 5 s bound. The stream's pull is shorter. One declared-length session ran on the final code (20261114T084906, void by host CPU). Its answers match: 2000 × 200 at 100/s; at 200/s 1164 × 200, 1007 × 204, 829 reset, no OOM kill. At 100/s memory.current was 369 MiB and anon 313.8 MiB. The head range for anon was 312.2-315.1 MiB, so page cache made the difference. Both stay under 409.6 MiB. The acceptance metric (anon or memory.current) still needs a ruling.

    Content safety. A cut chunked image with max-age=86400 was fetched again on each of 3 loads in Chrome 156, while a whole one with the same header was fetched once. Safari was not checked. SVG and non-image bodies still answer 204 (tests).

    Acceptance criteria.

    • At 200/s REST is not OOM-killed (declared length 3 of 3, no declared length 3 of 3).
    • At 100/s, memory.max events are 0 and memory.current stays under 409.6 MiB (332.2-334.9 and 298.9-302.7 MiB).
    • SVG and non-image bodies still answer 204.
    • This comment.

    Open before close.

    1. The UX check "images show as today at 10 to 100/s" holds only for images that declare their length. Without a declared length, images show to about 13/s, then 204. Accept the split, or raise the cap from 32 to 64? Base data suggests 64 stays under 409.6 MiB (base held 340.1-360.3 MiB at 25/s), but 64 is not measured.
    2. Safari is not checked.
    3. The issue's rollback "one setting" was not built. Rollback is a revert.
  5. HMarzban commented on Nov 14, 2026

    @HMarzban
    CollaboratorAuthor

    Measured again: valid before and after (slot 0, 2026-11-14)

    This replaces the numbers in the earlier Measured comment. Every session in that run was void from host CPU, so those numbers are not quotable. Read this one instead.

    Setup. One REST container at --cpus=1 --memory=512m (OrbStack, Linux cgroup v2), on one laptop. 2 MiB images from a 2.5 s origin. The client reads at full speed. Both arms were built from git archive: base 9f0e2be02, head daad326fe, with the same bench. 3 valid sessions per arm and lane.

    Validity rule (maintainer ruling, 2026-11-14). A step is void when other processes use more than 250 % CPU. An OOM kill counts as a valid memory break.

    Images that declare their length Before: memory.current max After: memory.current max
    50/s 419.6–453.6 MiB 256.5–285.5 MiB
    100/s 503–511.4 MiB 323.2–359.7 MiB
    200/s OOM-killed in 3 of 3 383–442.5 MiB, killed in 0 of 3
    • At 25, 50 and 100/s, every after session used less memory than every before session.
    • At 200/s the outbound pool fills (p95 7511.69–7653.9 ms). The 5 s timer cut 819–832 of 3000 streams, and memory.peak reached 422.7–456.9 MiB.
    • Images with no declared length are served at about 13 per second, and the rest get a 204 with no-store (100/s: 256 × 200, 1744 × 204).
    • Chrome 156 did not cache a cut stream in one check of 3 loads. Safari was not checked.

    Status. Committed locally as daad326fe. Not pushed or deployed yet. Raw data, judge script and runner: Notes/local-docs/2026-11-13-link-scraper-tolerance/raw/linux/s0/ (local). Canonical table: module BENCHMARK.md, section "After #786" (061fee04d).

  6. HMarzban commented on Nov 15, 2026

    @HMarzban
    CollaboratorAuthor

    Shipped in 2.3.6 (daad326fe). The before and after is in the Measured comment: #786 (comment)

    Closing as done. The issue stays in milestone v2.3.8 as the record of the load-tolerance work.

  7. HMarzban commented on Nov 15, 2026

    @HMarzban
    CollaboratorAuthor

    Rulings on the two open questions (2026-11-15)

    1. Images with no declared length: keep 32 reads at once. A relay must hold such an image in memory until it ends, so 64 reads would double that memory. 64 is not measured, and REST memory is already the tightest limit. 32 serves about 13 such images per second per replica. Production sends about 0.023 link requests per second. Revisit only if the Link metadata: metrics for in-flight work, the outbound pool, cache, stages and parse time #792 metrics show 204 answers for this reason in production.
    2. Safari and a cut stream: closed without a Safari run. HTTP caching (RFC 9111 §3.3) does not let a cache reuse an incomplete response as complete. Chrome 156 agreed in one check of 3 loads. If a browser did cache one, the cost is one broken image until its cache entry expires. Recorded as "not checked in Safari".
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    PerformanceQuery cost, load, and capacitySecuritySecurity, access control, and data exposurebugSomething isn't working

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions