Skip to content

perf(pool): keep Singleton waiters in a Slab - #353

Merged
seanmonstar merged 6 commits into
hyperium:masterfrom
patrick-immometrica:perf-singleton-waiter-lookup
Nov 18, 2026
Merged

seanmonstar merged 6 commits into
hyperium:masterfrom
patrick-immometrica:perf-singleton-waiter-lookup

Conversation

@patrick-immometrica

Copy link
Copy Markdown
Contributor

Batch::store_waker and Batch::remove looked a waiter up with a linear scan under the lock. When many requests wait on the first connection that's O(n) per poll; polling 30k waiters once took ~150ms.

A map keyed by WaiterId makes both lookups O(log n), ~1ms for the same run. Driver hand-off still promotes the newest waiter, as Vec::pop did.

Batch::store_waker and Batch::remove looked a waiter up with a linear
scan under the lock. When many requests wait on the first connection
that's O(n) per poll; polling 30k waiters once took ~150ms.

A map keyed by WaiterId makes both lookups O(log n), ~1ms for the same
run. Driver hand-off still promotes the newest waiter, as Vec::pop did.
@seanmonstar

Copy link
Copy Markdown
Member

Interesting! I'd be interested in hearing about use cases, was this found by having a case where there were 30k requests queued?

@patrick-immometrica

Copy link
Copy Markdown
Contributor Author

I didn't have a use case. I'm just interested in optimizing things. Anything I can improve here?

@seanmonstar

Copy link
Copy Markdown
Member

Hm, would using a Slab be better? It should be less expensive for smaller batches, but also provide O(1) operations even as a batch grows. (Also, I don't know if we care about exactly which gets promoted to driver.)

@patrick-immometrica

Copy link
Copy Markdown
Contributor Author

Thanks for pointing me to Slab. It performed even better.

Waiters Operation Vec (master) BTreeMap (previous) Slab
30k poll all once 127 ms 1.2 ms 0.32 ms
30k cancel all 65 ms 1.2 ms 0.55 ms
30k register all 0.45 ms 1.5 ms 0.47 ms
1024 poll / cancel all 109 / 105 µs 23 / 38 µs 11 / 19 µs
16 poll / cancel all 0.23 / 0.31 µs 0.25 / 0.58 µs 0.19 / 0.28 µs
1 register ~100 ns ~103 ns ~115 ns

@seanmonstar seanmonstar changed the title perf(pool): keep Singleton waiters in a BTreeMap perf(pool): keep Singleton waiters in a Slab Nov 18, 2026

@seanmonstar seanmonstar left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wonderful!

Comment thread Cargo.toml Outdated
tower-layer = { version = "0.3", optional = true }
tower-service = { version = "0.3", optional = true }
sync_wrapper = { version = "1", optional = true }
slab = { version = "0.4", optional = true }

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we need to set the minimum version to 0.4.4, because of using try_remove.

@seanmonstar
seanmonstar merged commit 8a1e01a into hyperium:master Nov 18, 2026
38 of 39 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants