Repository navigation
Description
Summary
A Redis stall holds every link-metadata request until Redis answers. On the laptop bench, a 10 s pause held each request for the whole pause. The code allows up to 60 s. The limiter already gives up after 500 ms, but the cache calls do not.
Measured
See Benchmark. Laptop only: redis-stall did not run on Linux.
Cause
cache.get(modules/link-metadata/domain/pipeline.ts:231) andcache.set(:272,:292) have no timeout of their own. A miss awaits the write before it answers.- They use the shared ioredis client. Its
commandTimeoutisREDIS_COMMAND_TIMEOUT, default 60000 ms (config/env.schema.ts:88), andenableOfflineQueueis true. jsonStore(modules/link-metadata/infra/redisCache.ts) only catches errors. A stall does not throw, so a stall is not a miss.- The limiter bounds itself with
consumeWithinat 500 ms (lib/consumeWithin.ts). Then it fails open, and the request waits on the cache instead.
What to change
- Race
cache.getand the padgenerationread againstwithinTimeout(lib/withinTimeout.ts). A timed-out read counts as a miss. A timed-out generation read skips the cache, as a failed read does today. - Bound
cache.setthe same way, and keep it awaited. Skip the write when the read timed out, so writes do not pile into the offline queue. - Count each timeout in a metric (Link metadata: metrics for in-flight work, the outbound pool, cache, stages and parse time #792).
Decisions
- What bound for a read? Recommended: 150 ms. It is far above a normal round trip and far below the scrape time. Other: 250 ms.
- Await the write, or fire and forget? Recommended: await it with the same bound. A fire-and-forget write lets the image request arrive before the write lands, and scrape again. Other: fire and forget, only together with Link metadata: join concurrent misses for one URL #791.
- Add a Redis breaker now? Recommended: no. Add it only if the after shows timeouts piling up.
Benchmark
Laptop before, measured 2026-11-14
Condition: the laptop condition in Bench conditions below. Scenario redis-stall, session 20261114T012145 (k = 1). 50/s; Redis paused (SIGSTOP) from 8 s to 18 s. REST used the default REDIS_COMMAND_TIMEOUT of 60000 ms.
| Step | n | p50 ms | p99 ms | Health p99 / max ms | Errors |
|---|---|---|---|---|---|
| ip-none | 1500 | 1.27 | 9719.63 | 1.38 / 32.32 | 0 |
| ip-pool | 1500 | 1.39 | 9715.39 | 1.07 / 10.78 | 0 |
- Per-second p99, ip-none: 10059.6 ms in second 8, then about 1 s less each second, to 1100.8 ms in second 17. The limiter path (ip-pool) has the same shape.
- Every answer was 200. REST logged no Redis error.
- A second valid session, a Docker-backend run and a Linux run are owed (Bench: Linux series follow-up at 1.0 CPU and 512M #793).
After
Use the after method in Bench conditions below, plus one step with a pause of 70 s. Expected, not measured: p99 during a pause near the scrape time plus the bound. Origin hits rise during a stall, because each read becomes a miss. Read them beside #788 and #791.
Condition
The laptop condition in Bench conditions below; the Linux condition once #793 runs redis-stall.
Content-safety checks
- A timed-out generation read never serves a cached pad card. It skips the cache, as a failed read does today.
- A timed-out write leaves no partial key.
UX checks
- During a stall, cards arrive at about the scrape time, not after the stall.
- After the stall, cards come from the cache again.
Rollback
One setting raises the bound to the client's commandTimeout, which restores today's behaviour. Otherwise, revert the commit.
Acceptance criteria
- During a 10 s pause, no request waits longer than its scrape time plus 2 x the bound.
- During a 70 s pause, no request is held for 60 s.
- After the pause, the cache serves hits again, with no restart.
- A
Measuredcomment posts the after beside the before, with the condition.
Related
- REPORT.md §2 (Redis stall) and §4 rank 4.
- Link metadata: join concurrent misses for one URL #791: a fire-and-forget write needs singleflight.
Bench conditions
The bench is apps/hocuspocus.server/scripts/bench-link-tolerance/ (commit 1a0d21b7f). The headline numbers are in apps/hocuspocus.server/src/modules/link-metadata/BENCHMARK.md, section Load tolerance. REPORT.md, LINUX-RESULTS.md, research-*.md and raw/ are local bench notes (Notes/local-docs/2026-11-13-link-scraper-tolerance/), not in the repo. The Linux runner is local too.
Linux condition, used by every "Linux before": one REST container at --cpus=1 --memory=512m --memory-swap=512m, in the OrbStack VM on one laptop. Kernel 7.2.12-orbstack-00394-gbecdd6130068, booted with cgroup.memory=nokmem,nosocket, so socket and slab memory are not charged. --no-healthcheck. Bun 1.4.4+38afee6c8. Redis 8.8.0 (redis:alpine) in REST's network namespace. --network none. Pooled loopback to one local origin, no DNS, no TLS. benchTree 8774a6275a05005f, srcTree a9b6ca73ca68c2ec, runnerTree 516099a8c9839eac. Measured 2026-11-14. The core speed ratio to a production vCPU is not measured.
Laptop condition, used by every "Laptop before": local, one laptop (Apple M5 Max, 18 cores, 48 GB, macOS), Bun 1.4.4+38afee6c8, native Redis 8.10.3. REST runs under the sandbox-exec fence. Traffic is plain HTTP to one local origin, pooled loopback, no DNS, no TLS. No CPU quota and no cgroup. benchTree 8774a6275a05005f, srcTree a9b6ca73ca68c2ec. Measured 2026-11-14. Never compare it with a Linux number as one series.
"(void session)" marks a value from a valid step inside a void session. On Linux the OOM kill voided those sessions. The kill is the measured break.
After method, used by every "After":
- Base = the commit before the change. Head = the change.
- Run ABAB, at least 3 sessions per arm, each with its control step, at n ≥ 2,000 per step.
- The change moves
srcTree, so the before must run again. Do not reuse the 2026-11-14 numbers as the before. - The runner takes
src/from the working tree. So it first needs a source-folder option for base (REPORT.md §5.7 item 5). - On Linux, start each session in a fresh REST container, and keep the run agent alive until the footer is written.
- Quote a change only when every head session sits outside the base range. Give σ and the MDE beside it.
Paths are under apps/hocuspocus.server/src/ unless they say otherwise.
Activity
fix(link-metadata): bound each Redis cache call to 150 ms
docs(link-metadata): record the slot 0 results and shed answers
HMarzban commentedon Nov 15, 2026
Measured: before and after (slot 2, 2026-11-15)
Setup. One REST container at
--cpus=1 --memory=512m(OrbStack, Linux cgroup v2), on one laptop. Both arms were built fromgit archive: baseca9d555ce, headdf910c6b1, with the same bench. Base and head ran in turns (ABAB, BABA, ABAB), plus one head rerun. Base has 3 valid sessions, head 2. A step is void above 250 % other-process CPU (maintainer ruling, 2026-11-14).Redis is frozen for 10 s while 50 requests per second arrive, with no IP rate limit (
redis-stall-ip-none@50):redis-stall-ip-pool@50), head p95 was 806.6–806.66 ms (2 valid sessions) against 8295.96–8319.07 ms. The limiter adds up to 500 ms. This lane is reported, not judged.Status. Shipped in 2.3.6 (
df910c6b1). Canonical table: moduleBENCHMARK.md, "After #787 to #792". Raw data:Notes/local-docs/2026-11-13-link-scraper-tolerance/raw/linux/s2/(local).HMarzban commentedon Nov 15, 2026
Shipped in 2.3.6 (
df910c6b1). The before and after is in the Measured comment: #789 (comment)Closing as done. The issue stays in milestone v2.3.8 as the record of the load-tolerance work.