Repository navigation
Description
Summary
At production limits, the image relay is the one route whose first break is memory. One Linux replica at --cpus=1 --memory=512m reached memory.max at 100 images/s. REST was OOM-killed at 100/s in one run, and in the 200/s warm-up in the other. The relay holds each image whole in memory, up to 2 MiB, in about three copies. Nothing caps relays in flight.
Measured
See Benchmark. In short:
- Linux, at 100/s:
memory.peak512 MiB and 2092memory.maxevents, with 2000 of 2000 answered and CPU at 42.1 % (void session). - The side RSS fit is 1.0868 MiB per image in flight, intercept 227.6 MiB, r² 0.997 (void session).
- The 80 % line (409.6 MiB) fell at 25/s (run 1) and 50/s (run 2), with page cache counted. anon alone was 379.2 MiB and 400.2 MiB there.
research-scaling-options.mdR6 said it "Moves up if RSS passes about 400M before 200 concurrent relays". Linux read 417.1 MiB with 131 in flight (void session).- The laptop showed side RSS peaks of 263.1-787.5 MiB at 100/s. No laptop RSS is compared with 512 MiB.
Cause
readCappedBodykeeps a chunk list, thenBuffer.concats it (lib/readCappedBody.ts:15-32).fetchImagecopies again withnew Uint8Array(bytes)(modules/link-metadata/domain/relay.ts:59).- The own-card path reads
response.arrayBuffer()and copies it into aUint8Array(modules/link-metadata/http/imageController.ts:30). IMAGE_MAX_BYTESis 2 MiB (modules/link-metadata/http/imageController.ts:11). No cap limits relays in flight, and no server cache keeps image bytes.- The oEmbed stage reads
response.json()with no byte cap (modules/link-metadata/domain/stages/oembed.ts:100).
What to change
- Stream the relay. Read until at least 16 bytes arrive, sniff them with
sniffImageType, then pass the rest through a counting stream. Over the cap, abort the upstream read. - If streaming is not chosen, cap bytes in flight instead: a process-wide budget of in-flight image bytes, and a cap on relays in flight. Over budget, answer 204 at once.
- Remove the extra copy in
relay.ts:59in either case. - Read oEmbed bodies through
readApiJson, with its 64 KiB cap. - Count relays in flight, bytes in flight and 204 answers (Link metadata: metrics for in-flight work, the outbound pool, cache, stages and parse time #792).
Decisions
- Stream, or cap bytes? Recommended: stream when the upstream declares a
Content-Lengthof 2 MiB or less. Memory then follows chunks in flight, not whole images. Other: buffer with a 128 MiB byte budget and a cap of 32 relays, which keeps today's clean 204. - What about a body with no
Content-Length? Recommended: buffer it under the byte budget from step 2. Then a cut stream happens only when a peer sends more than it declared. - Share the in-flight counter with Link metadata: shed cold work before the queue grows to an OOM kill #787 and Link metadata: cap outbound fetches so fast lanes stay alive when origins hang #788? Recommended: yes, one module for every in-flight count.
Benchmark
Linux before, measured 2026-11-14
Condition: the Linux condition in Bench conditions below. Sessions 20261114T041855 (run 2) and 20261114T041519 (run 1), both void by OOM kill. 30 s steps; @100 20 s. Run 2 is the main read.
| Step | Throughput/s | p95 ms | mem.current max / peak MiB | memory.max events |
CPU % | inUseMean |
|---|---|---|---|---|---|---|
| @10 | 9.97 | 2585.36 | 332.3 / 376.4 | 0 | 14.6 | 26.41 |
| @25 | 24.9 | 2595.85 | 384.1 / 396.5 | 0 | 21.4 | 65.72 |
| @50 | 49.9 | 2570.88 | 417.1 / 439.1 | 0 | 32.4 | 131 |
| @100 | 99.5 | 2600.71 | 497.6 / 512 | 2092 | 42.1 | 261.47 |
| @200 | 0 of 3000 answered | — | OOM kill about 2.3 s into the warm-up | — | — | — |
- Run 1: memory.current 436.6 MiB at
@25.@100was void: 1623 of 2000 answered, 18.85 % errors, 731memory.maxevents, OOM kill. A hostbfsprocess ran at 236 to 517 % CPU. - The outbound pool also hit its cap at
@100in both runs (originStreamingMax256).
Laptop before, measured 2026-11-14 (context)
Condition: the laptop condition in Bench conditions below. Sessions 20261114T003057, 20261114T003512, 20261114T003927, 20261114T005905. RSS is side memoryUsage().rss at 1 Hz. It is never compared with 512 MiB.
| Offered | Degraded % | Fetches in use | RSS steady / max MiB |
|---|---|---|---|
| 10 | 0 | 26.2 | 133.8-138.9 / 141-165.6 |
| 50 | 0 | 129.6 | 134.3-139.6 / 195.3-406.1 |
| 100 | 0 | 258.21-258.47 | 147.5-197.2 / 263.1-787.5 |
| 125 | 9.95 | 504.1 | 137.9 / 198.5 |
| 200 | 61.2-61.3 | 1124-1132 | 159.5-218.3 / 493.6-856.7 |
After
Use the after method in Bench conditions below, on Linux, with the image ramp at 10, 25, 50, 100 and 200/s. Expected, not measured: memory.current stops growing with images in flight. Report each step's memory.current max, memory.max events, oom_kill and 204 count.
Condition
The after runs on the Linux condition in Bench conditions below, with the same image ramp and origin. Name the VM, nokmem,nosocket and the missing healthcheck in the Measured comment.
Content-safety checks
- The served
Content-Typestill comes from the sniffed bytes only. SVG never passes. - The sniff runs before any header or byte goes to the client.
- Every hop still goes through
safeFetch, and the reader's browser still never contacts the linked site. - A cut stream must not stay in a browser cache under
max-age=86400. Check this in Chrome and Safari before the change ships. - Pad links (own cards) keep their current path.
UX checks
- At 10 to 100/s, the hover card and the link preview show the image as today.
- A 204 shows the card without an image, as today.
- The time to the first image byte does not rise.
Rollback
One setting turns streaming off and restores the buffered path. Otherwise, revert the commit.
Acceptance criteria
- On Linux at image 200/s, REST is not OOM-killed, and
memory.eventsoom_killstays 0. - At image 100/s,
memory.maxevents are 0, and memory.current stays under 409.6 MiB. - SVG and non-image bodies still answer 204.
- A
Measuredcomment posts the after beside the before, with the condition.
Related
- REPORT.md §1, §3 and §4 rank 1.
- Link metadata: shed cold work before the queue grows to an OOM kill #787 and Link metadata: cap outbound fetches so fast lanes stay alive when origins hang #788 share the in-flight counter. Bench: Linux series follow-up at 1.0 CPU and 512M #793 runs the Linux before and after.
Bench conditions
The bench is apps/hocuspocus.server/scripts/bench-link-tolerance/ (commit 1a0d21b7f). The headline numbers are in apps/hocuspocus.server/src/modules/link-metadata/BENCHMARK.md, section Load tolerance. REPORT.md, LINUX-RESULTS.md, research-*.md and raw/ are local bench notes (Notes/local-docs/2026-11-13-link-scraper-tolerance/), not in the repo. The Linux runner is local too.
Linux condition, used by every "Linux before": one REST container at --cpus=1 --memory=512m --memory-swap=512m, in the OrbStack VM on one laptop. Kernel 7.2.12-orbstack-00394-gbecdd6130068, booted with cgroup.memory=nokmem,nosocket, so socket and slab memory are not charged. --no-healthcheck. Bun 1.4.4+38afee6c8. Redis 8.8.0 (redis:alpine) in REST's network namespace. --network none. Pooled loopback to one local origin, no DNS, no TLS. benchTree 8774a6275a05005f, srcTree a9b6ca73ca68c2ec, runnerTree 516099a8c9839eac. Measured 2026-11-14. The core speed ratio to a production vCPU is not measured.
Laptop condition, used by every "Laptop before": local, one laptop (Apple M5 Max, 18 cores, 48 GB, macOS), Bun 1.4.4+38afee6c8, native Redis 8.10.3. REST runs under the sandbox-exec fence. Traffic is plain HTTP to one local origin, pooled loopback, no DNS, no TLS. No CPU quota and no cgroup. benchTree 8774a6275a05005f, srcTree a9b6ca73ca68c2ec. Measured 2026-11-14. Never compare it with a Linux number as one series.
"(void session)" marks a value from a valid step inside a void session. On Linux the OOM kill voided those sessions. The kill is the measured break.
After method, used by every "After":
- Base = the commit before the change. Head = the change.
- Run ABAB, at least 3 sessions per arm, each with its control step, at n ≥ 2,000 per step.
- The change moves
srcTree, so the before must run again. Do not reuse the 2026-11-14 numbers as the before. - The runner takes
src/from the working tree. So it first needs a source-folder option for base (REPORT.md §5.7 item 5). - On Linux, start each session in a fresh REST container, and keep the run agent alive until the footer is written.
- Quote a change only when every head session sits outside the base range. Give σ and the MDE beside it.
Paths are under apps/hocuspocus.server/src/ unless they say otherwise.
Activity
HMarzban commentedon Nov 14, 2026
Measured: after, 2026-11-14
Change.
relayImage(domain/relay.ts) streams an image that declares itsContent-Length(2 MiB at most), after the sniff. An image with no declared length is read whole, at most 32 at once. Past that the route answers204withCache-Control: no-store. oEmbed now reads throughreadApiJson(64 KiB cap). The copy inrelay.ts:59is gone.Commits.
9d446b65d(bench: a declared-length image lane) anddaad326fe(the fix), onmain. Not deployed yet.Condition. The Linux condition in this issue: one REST container at
--cpus=1 --memory=512m --memory-swap=512m, OrbStack VM, one Apple M5 Max laptop, Bun1.4.4+38afee6c8, Redis 8.8.0, pooled loopback, no DNS, no TLS. 2 MiB images from a 2.5 s origin.9f0e2be02(srcTreeNoTests7d1df39a060799fb). Head: the change (srcTreeNoTestse9c71b4893743850). Same bench in both images (benchTreee45612b12a568b64).20261114T075104to20261114T083003.node:httplistener, because Bun's server sends every stream chunked. "No declared length" is the lane of the before.Every session is void. Another project's Next dev server and Playwright tests ran on the host the whole time (other-process CPU above 150 %). Each base session also ended in an OOM kill, which is the measured break. Base and head took turns, so both arms shared the noise. Values are the range over 3 sessions.
memory.currentmaxmemory.currentmax200200200200200200200, 999-1003 ×204, 825-835 reset200200, 360-362 ×204200200, 1744 ×204200, 2808 ×204Base at 100/s also had up to 469
memory.maxevents (no declared length) and 87 (declared length). Head had 0 at every step. One base session was killed at 100/s.Read.
204, and cuts 825-835 streams already sending. Before, a late image was a204cached for 600 s. Now a cut stream is a failed load, never cached.204for the rest. Memory stays near 300 MiB.Final code. A review after these runs changed three things. A coded body (
content-encodingother thanidentity) now counts as undeclared, because fetch unzips it but keeps the coded length. OneAbortSignal.timeoutholds the 5 s bound. The stream'spullis shorter. One declared-length session ran on the final code (20261114T084906, void by host CPU). Its answers match: 2000 ×200at 100/s; at 200/s 1164 ×200, 1007 ×204, 829 reset, no OOM kill. At 100/smemory.currentwas 369 MiB andanon313.8 MiB. The head range foranonwas 312.2-315.1 MiB, so page cache made the difference. Both stay under 409.6 MiB. The acceptance metric (anonormemory.current) still needs a ruling.Content safety. A cut chunked image with
max-age=86400was fetched again on each of 3 loads in Chrome 156, while a whole one with the same header was fetched once. Safari was not checked. SVG and non-image bodies still answer204(tests).Acceptance criteria.
memory.maxevents are 0 andmemory.currentstays under 409.6 MiB (332.2-334.9 and 298.9-302.7 MiB).204.Open before close.
204. Accept the split, or raise the cap from 32 to 64? Base data suggests 64 stays under 409.6 MiB (base held 340.1-360.3 MiB at 25/s), but 64 is not measured.HMarzban commentedon Nov 14, 2026
Measured again: valid before and after (slot 0, 2026-11-14)
This replaces the numbers in the earlier Measured comment. Every session in that run was void from host CPU, so those numbers are not quotable. Read this one instead.
Setup. One REST container at
--cpus=1 --memory=512m(OrbStack, Linux cgroup v2), on one laptop. 2 MiB images from a 2.5 s origin. The client reads at full speed. Both arms were built fromgit archive: base9f0e2be02, headdaad326fe, with the same bench. 3 valid sessions per arm and lane.Validity rule (maintainer ruling, 2026-11-14). A step is void when other processes use more than 250 % CPU. An OOM kill counts as a valid memory break.
memory.currentmaxmemory.currentmaxmemory.peakreached 422.7–456.9 MiB.204withno-store(100/s: 256 ×200, 1744 ×204).Status. Committed locally as
daad326fe. Not pushed or deployed yet. Raw data, judge script and runner:Notes/local-docs/2026-11-13-link-scraper-tolerance/raw/linux/s0/(local). Canonical table: moduleBENCHMARK.md, section "After #786" (061fee04d).test(link-metadata): serve images with a declared length in the bench
fix(link-metadata): stream page images that declare their length
fix(link-metadata): shed cold work past 64 in flight
docs(link-metadata): record the slot 0 results and shed answers
HMarzban commentedon Nov 15, 2026
Shipped in 2.3.6 (
daad326fe). The before and after is in the Measured comment: #786 (comment)Closing as done. The issue stays in milestone v2.3.8 as the record of the load-tolerance work.
HMarzban commentedon Nov 15, 2026
Rulings on the two open questions (2026-11-15)
204answers for this reason in production.