Repository navigation
perf(mpmc): back bounded queues with a ring buffer - #382
ariesdevil wants to merge 2 commits into
Conversation
ariesdevil
changed the title
94d174b to
4ea81c4
Compare
|
@ariesdevil You may update the branch to resolve the conflict with the main branch, and I will also adopt the new benchmark best practice that you can leverage. Once this pull request is clean, I will do another review. |
Replace the bounded queue's VecDeque behind one mutex with a preallocated ring of stamped slots adapted from crossbeam-channel's array flavor. Sends, receives, and reservations take no lock unless a sender or receiver has to wait, while wait-queue order, exact capacity, permits, cancellation, and disconnection behave as before. Storage for the full capacity is allocated when the queue is created, and capacity is limited to 2,097,151. The capacity that permits and grants hold, and a bit marking queued senders, live in the tail cursor beside the position, so every exchange on it compares the reservation state it was computed from. While senders are queued, a send with a permit grants the capacity that receivers free until it finishes, so a single receiver does not have to wake every queued sender itself. Positions are 64 bits on every target, so a stalled exchange can succeed only after at least 2^40 sends. Targets without 64-bit atomics, such as ARMv5TE and 32-bit PowerPC and MIPS, keep each cursor and stamp as a u64 behind a lock, which is slower but runs the same algorithm. The unbounded queue keeps its mutex and drops the bounded-only code. Copy CachePadded from crossbeam-utils and record the incorporated crossbeam portions in LICENSE, licenserc.toml, and .asf.yaml. The benchmarks gain an AsyncbandMpmcReserve adapter that sends through permits, a crossbeam-channel adapter, fixed-work scenarios for 1P/8C, capacity 1, and a spinning batch, and reservation micro-benchmarks. Co-authored-by: tison <wander4096@gmail.com>
9600dab to
065940a
Compare
|
|
||
| ### Notable changes | ||
|
|
||
| * Bounded MPMC queues now panic when created with a capacity greater than 2,097,151. |
There was a problem hiding this comment.
Why do we now have this limitation?
There was a problem hiding this comment.
The ring's tail packs the slot index, the disconnect mark, the capacity that permits and grants hold, the queued-senders bit, and a lap count into one 64-bit word, so every exchange on it checks all of that state at once. That is what makes a send that stalled before its exchange fail if reservations changed meanwhile; the versioned counter this replaced could be fooled after 2^24 reservations. The index and the reserved count each need about log2(capacity) bits, leaving the rest for the lap count, and a stale exchange can only succeed after the lap count wraps. Capping capacity at 2^21 − 1 keeps at least 20 lap bits, so a wrap takes at least 2^40 sends. At 2^28 it would take about 2^34 sends, which a fast producer reaches within minutes while another thread is descheduled. Larger buffers can use mpmc::unbounded.
Partial related: #298
Summary
Back the bounded MPMC queue (added in #265) with a preallocated ring of stamped slots, adapted from crossbeam-channel's array flavor, instead of a
VecDequebehind one mutex. Sends, receives, and reservations no longer take a lock unless a sender or receiver has to wait. Pendingsendandreserveoperations still receive capacity in wait-queue order, permits still hold capacity without a slot, and cancellation and disconnection behave as before. Storage for the full capacity is now allocated when a queue is created, and capacity is limited to 2,097,151.The unbounded queue keeps its mutex and drops the bounded-only code.
CachePaddedis copied from crossbeam-utils, andLICENSE,licenserc.toml, and.asf.yamlrecord the incorporated crossbeam portions.WakerBatch::wake_allskips its overflow storage when nothing spilled, which makes waking a blocked sender 3% to 10% faster.The benchmarks gain an
AsyncbandMpmcReserveadapter that sends each value throughreserve().awaitand a permit, a crossbeam-channel adapter, fixed-work scenarios for 1P/8C, capacity 1, and a spinning batch that isolates queue contention from wakeups, and reservation micro-benchmarks.Design Notes
tailbetween the slot index and the lap, so every exchange ontailcompares the reservation state it was computed from. The other values a sender reads,headand the slot stamps, only advance, so reading them early can only understate free capacity.powerpc-unknown-linux-gnu.reserveandtry_reservetake no lock while capacity is free and no sender is queued, and neither does dropping an unused permit unless senders are queued.sendjoins the wait queue directly oncetry_sendfails, because another attempt would only add contention. Granting the last queued sender clears the queued bit in the same update. A grant removes the waiter's node and sets a flag in its pinned future, so the woken sender claims its permit without the lock.perfprofiles on x86. Keeping the count in its own word behind a version intailwas incorrect: 2^24 reservations wrap the version, after which a stalled send can claim capacity that a permit holds. A regression test covers that interleaving. Taking the wait-queue lock in everyreservewas dropped as well: removing it made multi-producer reservations on a multi-thread Tokio runtime 29% to 40% faster on Linux, but blocking 1P/1C and 8P/1C reservations 59% and 20% slower, because the lock kept a lone producer from outrunning its consumer and waiting for grants.Benchmarks
Fixed-work comparisons from the harness in #383, with Asyncband at
c37b1a4(main) as the reference library. The new scenarios use round counts that put the reference near 300 ms on macOS, as the existing ones do. Each binary ran--repeat 10in four alternating passes (main, this PR, this PR,main), so every cell pools 20 runs. Cells show the median and the minimum-to-maximum range in milliseconds; negative changes are faster. Theprimitivesrows omit theprimitives::prefix. Built with rustc 1.99.0.macOS, Apple M5 Pro:
mainmsmpmc/bounded/tasks-2p2c-cap64-4wAsyncbandMpmcReservempmc/bounded/threads-2p2c-cap64AsyncbandMpmcReservempmc/bounded/threads-1p8c-cap64mpmc/bounded/threads-2p2c-cap1mpmc/bounded/spin-2p2c-cap64mpmc/unbounded/tasks-2p2c-4wmpmc/unbounded/threads-2p2cmpmc::bounded::try_send_then_try_recvmpmc::bounded::send_then_recvmpmc::bounded::try_reserve_then_sendmpmc::bounded::reserve_then_sendmpmc::bounded::wake_blocked_sendermpmc::unbounded::send_then_try_recvmpmc::unbounded::send_then_recvmpmc::unbounded::wake_pending_receivermpmc::unbounded::repoll_pending_receiverLinux, Intel Xeon 6982P-C, 16 cores, KVM guest, Ubuntu 26.04, kernel 7.0:
mainmsmpmc/bounded/tasks-2p2c-cap64-4wAsyncbandMpmcReservempmc/bounded/threads-2p2c-cap64AsyncbandMpmcReservempmc/bounded/threads-1p8c-cap64mpmc/bounded/threads-2p2c-cap1mpmc/bounded/spin-2p2c-cap64mpmc/unbounded/tasks-2p2c-4wmpmc/unbounded/threads-2p2cmpmc::bounded::try_send_then_try_recvmpmc::bounded::send_then_recvmpmc::bounded::try_reserve_then_sendmpmc::bounded::reserve_then_sendmpmc::bounded::wake_blocked_sendermpmc::unbounded::send_then_try_recvmpmc::unbounded::send_then_recvmpmc::unbounded::wake_pending_receivermpmc::unbounded::repoll_pending_receiverEach Linux
send_then_recvprocess onmainsettles at either 833 or 1022 ms, so both modes are shown. The macOS-onlympmc::unbounded::send_then_try_recvchange is not reproduced on Linux, and the unbounded queue's send and receive paths only lost code.Peers in this PR's binaries, as elapsed time relative to Asyncband's
send(macOS / Linux):AsyncbandMpmcReservempmc/bounded/tasks-2p2c-cap64-4wmpmc/bounded/threads-2p2c-cap64mpmc/bounded/threads-1p8c-cap64mpmc/bounded/threads-2p2c-cap1mpmc/bounded/spin-2p2c-cap64mpmc/unbounded/tasks-2p2c-4wmpmc/unbounded/threads-2p2cThe remaining regressions are on paths where every operation waits: capacity 1 and a blocked sender's handoff. Peers ran the same code in both binaries. Between them, peer medians moved by at most 4.3% on macOS; on Linux they stayed within 4% except crossbeam-channel, which moved 5.5% in
threads-2p2c-cap64and 23% inspin-2p2c-cap64, so treat Linux spinning comparisons as noisy.