-
Zero-RMW Reader Scaling: Eliminates per-object atomic read-modify-write (RMW) instructions for transient readers (
0atomic RMWs for RCU; CPU-sharded acquisition for Rundown), reducing cross-core cache invalidation on multi-core systems. -
NUMA-Aware Retirement Sharding (RCU & Rundown): Routes displaced control blocks into node-local, cache-aligned lock-free
SLIST_HEADERshards to reduce cross-socket interconnect traffic during pointer updates. -
$O(1)$ Batch Synchronization: Replaces per-object OS synchronization waits with driver-wide Epoch-based and Quiescent-state collectors that drain and reclaim retired objects in batches. - Cache-Line Isolation: Single-allocation factories place payload data adjacent to control blocks at cache-aligned offsets, separating read-mostly payload fields from control-block metadata mutations.
- User-Controlled Garbage Collection: Does not spawn internal background threads. Memory reclamation is scheduled and invoked explicitly by the client driver via high-watermark signaling or periodic maintenance routines.
- Cross-Architecture Memory Ordering: Supports x64 Total Store Order (TSO) and weakly-ordered ARM64 pipelines using inner-shareable hardware memory fences and WDK-compliant IRQL transitions.
- Overview
- Architecture & Algorithms
- High-Level Design: Transient Access vs. Persistent Ownership
- User-Controlled Garbage Collection (RCU & Rundown)
- 1. DPC-Based Quiescent State RCU (
KmRcuSharedPtr.h) - 2. Epoch-Based Cache-Aware Rundown (
KmRundownSharedPtr.h) - 3. 128-Bit Split-Count Atomic Shared Pointer (
KmAtomicSharedPtr.h) - 4. Single-Allocation Cache-Isolated Memory Layout
- Quick Start & API Overview
- Benchmarks & Scaling Performance
- Project Layout
- Building Instructions
- Choosing the Right Shared Pointer
- Conclusion
- License
Kernel-mode drivers frequently maintain global or per-device state—such as security policies, routing tables, filter configurations, and connection maps—that is read on every I/O fast path and updated periodically in the background. Protecting these pointers with traditional kernel primitives (EX_PUSH_LOCK wrapped in critical regions, spinlocks, or standard atomic reference counting) introduces scalability bottlenecks on multi-core and multi-socket NUMA systems. Even lock-free reference counting forces every concurrent reader to execute atomic read-modify-write (LOCK XADD / LDADD) instructions against a shared control-block cache line, causing cross-core cache invalidations and tail-latency spikes as core counts increase.
This repository provides three header-only smart pointer implementations with lock-free read and update paths designed for Windows kernel drivers (IRQL <= DISPATCH_LEVEL on non-paged memory). Primary emphasis is placed on two deferred-reclamation architectures—DPC-Based Quiescent State RCU (KmRcuSharedPtr) and Epoch-Based Cache-Aware Rundown (KmRundownSharedPtr)—which decouple transient read access from per-object reference counting and batch memory reclamation in KmAtomicSharedPtr) based on Anthony Williams' algorithm is also provided.
| Primitive & Header | Reader Progress & Mechanism | Reader RMWs | Reclamation Model | Primary Use Case |
|---|---|---|---|---|
CRcuSharedPtr<T>KmRcuSharedPtr.h |
Wait-freeDISPATCH_LEVEL preemption guard (KeRaiseIrqlToDpcLevel) |
0 (Acquire load) |
Blocking GC pass: targeted HighImportance KDPC broadcast + batch free |
Low-latency non-blocking reads at <= DISPATCH_LEVEL |
CRundownSharedPtr<T>KmRundownSharedPtr.h |
Lock-free Per-CPU hazard slot + cache-aware OS rundown epoch shard |
0 per-object (CPU-local shard) |
Blocking GC pass: 64-bit epoch swap + single OS rundown wait per batch | Read-heavy PASSIVE_LEVEL / APC_LEVEL paths without raising IRQL |
CAtomicSharedPtr<T>KmAtomicSharedPtr.h |
Lock-free 128-bit CAS loop + 64-bit atomic fetch-add reconciliation |
2 on load()1 on release |
Immediate free when internal and external counts reach zero | General shared ownership, CAS state machines, zero-GC paths |
- Decoupled Transient & Persistent Access (
KmRcuSharedPtr/KmRundownSharedPtr): Fast-path readers execute inside a scopedRead()lambda without mutating object reference counts. When a caller needs to retain an object across asynchronous I/O completions, work items, or blockingPASSIVE_LEVELcalls,LoadPersistent()promotes the active object to a reference-counted handle (CPersistentRcuPtr<T>orCPersistentRundownPtr<T>) while protected by the transient read section. - Explicit User-Controlled Garbage Collection: Neither deferred-reclamation pointer spawns internal background threads. Clients control when the garbage collection pipeline is scheduled and invoked.
- Compile-Time Anti-Pattern Enforcement:
Read()inspects the callback's return type viastatic_assertand blocks returning raw pointers or C++ references, preventing callers from leaking transient pointers outside the protected scope. - Cache-Line Isolation & Direct Payload Resolution: Single-allocation factories (
MakeRcuShared,MakeRundownShared,MakeAtomicShared) place the control block and payload in adjacent, cache-line-aligned regions of a single non-paged pool block. RCU and Rundown readers resolve the payload address via compile-time constant offset arithmetic (GetPayloadOffset()), whileCLocalSharedPtr<T>caches a direct payload pointer (m_pPtr), avoiding loads from the control block's mutable cache line during access. - NUMA-Affinitized Lock-Free Retirement Queues: Displaced RCU and Rundown blocks are pushed via lock-free
InterlockedPushEntrySListoperations into per-NUMA-node, cache-alignedSLIST_HEADERshards (ExAllocatePool3node-affinitized allocations) with a statically embedded fallbackSLIST_HEADER. - Cross-Architecture x64 & ARM64 Support: Implements explicit memory ordering across both x64 Total Store Order (TSO) and weakly-ordered ARM64 pipelines using
ReadAcquire64(LDAR),WriteRelease64(STLR), inner-shareable hardware data memory barriers (__dmb(_ARM64_BARRIER_ISH)), and 128-bit atomic compare-and-swap intrinsics (_InterlockedCompareExchange128/_acq/_rel). - IRQL & Driver Verifier Compliance: All anchor reads, persistent promotions, and pointer replacements (
Store,load,store,exchange,compare_exchange) are safe atIRQL <= DISPATCH_LEVEL. All IRQL transitions use documented WDK kernel APIs (KeRaiseIrqlToDpcLevel/KeLowerIrql) to maintain compatibility with Driver Verifier IRQL and concurrency checks.
In many kernel workloads, accesses to shared configuration or state structures are brief, read-only inspections: a dispatch routine or filter callback reads a few fields, makes a routing or policy decision, and exits. Standard smart pointers force these short-lived readers to pay the same atomic reference-counting cost as long-lived owners, generating cache invalidation traffic across CPUs.
KmRcuSharedPtr and KmRundownSharedPtr address this by splitting access into two distinct paths:
- Transient Read Path (
Read): Binds read access to a lexical scope via an execute-around lambda (anchor.Read([](const T* pObj) { ... })). Protection is established at the processor or epoch level rather than per object. The reader performs an acquire load of the anchor pointer, computespObjby adding a compile-time constant byte offset, and executes the callback without modifying the object's control block. - Persistent Ownership Path (
LoadPersistent): When an object must outlive the immediate scope (for example, attaching policy state to a pended IRP or reading fields across blockingPASSIVE_LEVELcalls),LoadPersistent()enters the transient read critical section, atomically incrementsm_lPersistentRefCounton the control block, exits the transient critical section, and returns aCPersistentRcuPtr<T>orCPersistentRundownPtr<T>.
While pointer retirement (Store) is lock-free and executes in a bounded number of instructions via InterlockedExchangePointer and InterlockedPushEntrySList, reclaiming retired memory requires waiting for in-flight readers across other processors to complete their read sections. Rather than spawning internal background worker threads that consume CPU cycles or queue DPCs unpredictably, the RCU and Rundown implementations rely on user-controlled garbage collection.
The underlying garbage collection engines (CQuiescentGarbageCollector for RCU, and CRundownPtrGarbageCollector for Rundown) operate as driver-wide global singletons. A single instance of each collector multiplexes and reclaims displaced blocks for all RCU or Rundown anchors instantiated within the driver module, avoiding per-anchor queue fragmentation.
The client driver is responsible for invoking ProcessGarbagePipeline() at PASSIVE_LEVEL through one of two mechanisms:
- Event-Driven Wake Callback: The library provides a
SetWakeCallbacknotification mechanism that triggers when the aggregate queue of retired objects reaches a configurable high-watermark threshold. Note: This callback must not execute the reclamation pass directly. Since the callback may be invoked atDISPATCH_LEVELduring a pointer swap, its role is to signal a synchronization event (KeSetEvent) or queue a system work item (IoQueueWorkItem) that wakes a client-managedPASSIVE_LEVELthread to callProcessGarbagePipeline(). - Periodic Maintenance: Alternatively, drivers can omit the callback and invoke
ProcessGarbagePipeline()periodically from an existing maintenance timer or housekeeping thread.
This separation keeps writer updates non-blocking while allowing driver authors to defer reclamation during latency-sensitive I/O bursts and consolidate cleanup into existing maintenance threads.
KmRcuSharedPtr implements Quiescent State-Based Reclamation (QSBR) by mapping Read-Copy-Update (RCU) principles onto the Windows NT dispatcher's IRQL hierarchy.
In operating systems with built-in RCU primitives, the scheduler tracks read-side critical sections across context switches. Since Windows kernel drivers cannot instrument the scheduler, KmRcuSharedPtr uses documented NT IRQL rules to enforce and observe quiescent states without per-object reference counting.
By binding transient read sections to DISPATCH_LEVEL, the implementation uses the NT DPC dispatcher's preemption masking to define a quiescent state constraint. To establish a system-wide grace period during garbage collection, the collector queues a targeted HighImportance DPC to every online processor and waits on a kernel event. Since a logical processor cannot service a newly queued DPC while an existing DISPATCH_LEVEL routine is executing on that core, waiting for all targeted DPCs to complete ensures that all pre-existing read sections have finished across the system.
- Zero-RMW Reader Scaling: Since grace-period synchronization is deferred to the collector and relies on DPC scheduling invariants, the
Read()path is wait-free and executes zero atomic read-modify-write (RMW) instructions. This avoids the cross-socketLOCKcontention and cache-line invalidation traffic associated with atomic reference counting on multi-core systems. - Degraded-Mode Reclamation Fallbacks: The collector handles edge cases where non-paged pool exhaustion prevents
KDPCarray allocation, or where an anchor is destroyed at an elevated IRQL where synchronous waiting is illegal. In these scenarios, the collector falls back to a sequential processor sweep (synchronous via an embeddedKDPCatPASSIVE_LEVEL, or asynchronous via a state-machineKDPCwhen triggered at elevated IRQL) so reclamation can proceed without violating OS blocking rules.
While RCU avoids shared memory writes on the read path, elevating to DISPATCH_LEVEL is incompatible with read callbacks that must access pageable memory or invoke PASSIVE_LEVEL kernel APIs. KmRundownSharedPtr supports read scaling at IRQL <= DISPATCH_LEVEL without raising IRQL by combining Epoch-Based Reclamation (EBR) with Windows' native EX_RUNDOWN_REF_CACHE_AWARE facility.
Allocating a dedicated EX_RUNDOWN_REF_CACHE_AWARE structure for every shared object scales well for readers but introduces reclamation overhead: it requires per-CPU memory allocations for each object and forces the garbage collector to wait sequentially on ExWaitForRundownProtectionReleaseCacheAware for every displaced pointer, creating Head-of-Line blocking.
KmRundownSharedPtr decouples rundown lifetime from individual objects by maintaining two global cache-aware rundown shards multiplexed across all shared pointer instances in the driver, representing Epoch 0 and Epoch 1:
- Readers sample the global 64-bit monotonic epoch counter and acquire rundown protection on the active epoch's CPU-sharded
EX_RUNDOWN_REF_CACHE_AWAREinstance (entering a critical region viaKeEnterCriticalRegion()when called atPASSIVE_LEVELso normal kernel APCs cannot suspend the thread while holding the epoch shard). - During garbage collection, the collector flushes all retired objects into a pending batch and increments the global epoch, directing new readers to the alternate shard.
- The collector then blocks once on
ExWaitForRundownProtectionReleaseCacheAwarefor the outgoing epoch shard to drain. Once active readers on that shard exit, the collector re-initializes the shard and frees the entire batch of retired objects across all NUMA nodes in a single pass.
Multiplexing epoch rings with OS rundown protection introduces a lock-free linearization hazard: a reader thread at PASSIVE_LEVEL could load the anchor pointer and sample the epoch, experience an OS scheduling delay before calling ExAcquireRundownProtectionCacheAware, and resume after the collector has already advanced the epoch and re-initialized the shard.
To close this window without reader-side locks, the implementation combines a cache-line-isolated hazard slot array with a Dekker-style memory barrier handshake:
- The Reader (Lock-Free Retry Loop): Before acquiring the OS rundown shard, the reader increments a CPU-indexed hazard slot (
InterlockedIncrement), re-verifies that the anchor pointer has not changed, and acquires rundown protection on the sampled epoch shard before decrementing the hazard slot. If the anchor pointer changed or the epoch advanced prior to acquisition, the reader releases its claim and retries. - The Collector (Two-Phase Drain): The collector flushes displaced objects, waits for all hazard slots to reach zero (Barrier 1), increments the 64-bit epoch counter, and waits for all hazard slots to reach zero a second time (Barrier 2) before waiting on the outgoing OS rundown shard.
This handshake ensures that any preempted reader either observes the updated state and retries, or registered in its hazard slot prior to the epoch increment, causing the collector to wait until the reader finishes attaching to the outgoing OS rundown shard.
For subsystems requiring standard value-semantics handles, lock-free atomic state-machine transitions (compare_exchange, exchange), and immediate deterministic destruction without a deferred garbage collection pipeline, KmAtomicSharedPtr.h adapts Anthony Williams' split-reference-counting algorithm for the Windows kernel.
In a lock-free reference-counted pointer, loading a control-block address from a shared anchor and incrementing the reference count inside that control block are separate memory operations; a concurrent writer can displace and free the control block between the load and the increment.
KmAtomicSharedPtr resolves this without hazard pointers or epochs by splitting the reference count into two differential components:
- The External Count (Anchor): The shared anchor is a 128-bit structure packing the control-block pointer alongside a transient "External" reader count and an ABA generation counter. A reader checking out a handle executes a 128-bit Compare-and-Swap (CAS) loop on the anchor to increment this External count, pinning the control block while the checkout is in flight.
- The Internal Count (Control Block): The control block maintains a packed 64-bit split counter containing the "Internal" reference count (tracking active handles and anchor ownership) and a mirrored "External" balance.
- Deferred Reconciliation: Once the 128-bit CAS succeeds on the anchor, the reader reconciles its checkout on the control block by atomically incrementing the Internal count and decrementing the mirrored External count. When an anchor is displaced via
store()orexchange(), the displaced anchor's accumulated External count is added into the control block's External balance. The control block and payload are freed when both the Internal and External balances reach zero.
- Single-Instruction Carry-Compensated Reconciliation: Updating two 32-bit counters inside the control block via a 64-bit CAS loop would introduce retry contention under heavy reader traffic. Instead,
apply_counts_atomic()packs both signed 32-bit deltas into a single bounded 64-bit atomic fetch-and-add (InterlockedExchangeAdd64), using two's-complement carry compensation to account for borrow/carry propagation across bit 32. - Bounded External-Count Flushing: If a thread displacing an anchor is preempted while concurrent readers continue checking out handles, the 32-bit External count on the anchor could approach overflow. To bound accumulation, any reader observing an External count at or above
65,536flushes the accumulated claims into the control block's reconciliation counter via a 128-bit CAS and increments the anchor's generation counter.
All three primitives provide single-allocation factories (MakeRcuShared, MakeRundownShared, and MakeAtomicShared) to avoid secondary pool allocations and separate synchronization metadata from user payload data across cache lines (CACHE_LINE_SIZE: 64 bytes on x64, 128 bytes on ARM64).
Placing read-only payload data on the same CPU cache line as frequently mutated synchronization fields (such as reference counts or intrusive list entries) causes false sharing, invalidating the payload cache line in the L1/L2 caches of reader cores whenever metadata is updated.
The factories allocate a single, cache-aligned non-paged pool block that places the payload behind the control block on a cache-line boundary. Since the deferred garbage collection mechanisms require different tracking metadata than the immediate-reclamation atomic model, the memory layouts are structured into two configurations:
KmRcuSharedPtr and KmRundownSharedPtr embed the garbage collection tracking node (SLIST_ENTRY) within the control block. To avoid extra prefix padding for over-aligned pool allocations, the raw pool base pointer (m_pRawAllocation) is stored at byte offset 32 inside the control block's existing cache-line tail padding.
+-----------------------------------------------------------------------+
| Cache Line 0 (Offset 0..63 x64 / 0..127 ARM64): Control Block |
| - SLIST_ENTRY m_SListEntry (16 bytes, 16-byte aligned) |
| - m_pfnDeleteCallback (8 bytes) |
| - volatile LONG m_lAnchorState (4 bytes) |
| - volatile LONG m_lPersistentRefCount (4 bytes) |
| - void* m_pRawAllocation (8 bytes, packed at offset 32) |
| - [Tail Padding to CACHE_LINE_SIZE Boundary] |
+-----------------------------------------------------------------------+
| Cache Line 1+ (Offset 64+ x64 / 128+ ARM64): Embedded Payload (T) |
| - Constructed in-place via placement new at GetPayloadOffset() |
| - Isolated on separate cache line(s) from control-block metadata |
+-----------------------------------------------------------------------+
KmAtomicSharedPtr does not use deferred garbage collection queues. Its control block contains only the payload pointer, the 64-bit split counter, and an embedded-allocation flag, placing the payload on the next cache line boundary while CLocalSharedPtr caches a direct pointer to the payload.
+-----------------------------------------------------------------------+
| Cache Line 0 (Offset 0..63 x64 / 0..127 ARM64): TAtomicControlBlock |
| - T* m_pPtr (8 bytes) |
| - volatile LONG64 m_llCounts (8 bytes: Internal & External) |
| - bool m_bIsEmbedded (1 byte) |
| - [Tail Padding to CACHE_LINE_SIZE Boundary] |
+-----------------------------------------------------------------------+
| Cache Line 1+ (Offset 64+ x64 / 128+ ARM64): Embedded Payload (T) |
| - Constructed in-place via placement new at cbPayloadOffset |
| - Isolated on separate cache line(s) from control-block metadata |
+-----------------------------------------------------------------------+
CRcuSharedPtr<T> and CRundownSharedPtr<T> expose an identical high-level API surface (Read, LoadPersistent, Store, and singleton garbage-collector management), allowing drivers to switch between DISPATCH_LEVEL RCU protection and PASSIVE_LEVEL-compatible Rundown protection without restructuring call sites. CAtomicSharedPtr<T> follows the standard std::atomic<std::shared_ptr<T>> handle checkout model.
Comprehensive API specifications, edge-case contracts, and full driver integration patterns (including event-driven GC worker threads, IRP queue-node caching, and batched DISPATCH_LEVEL fast paths) are provided in dedicated documents:
| Primitive | Header | Usage Guide & Design Patterns | API Reference |
|---|---|---|---|
| DPC Quiescent RCU | KmRcuSharedPtr.h |
RCU Usage Guide | RCU API Reference |
| Epoch Cache-Aware Rundown | KmRundownSharedPtr.h |
Rundown Usage Guide | Rundown API Reference |
| 128-Bit Split-Count Atomic | KmAtomicSharedPtr.h |
Atomic Usage Guide | Atomic API Reference |
| Component / Operation | RCU (KmRcuSharedPtr.h) |
Rundown (KmRundownSharedPtr.h) |
Max IRQL | Description |
|---|---|---|---|---|
| Shared Anchor | CRcuSharedPtr<T>CCacheAlignedRcuSharedPtr<T> |
CRundownSharedPtr<T>CCacheAlignedRundownSharedPtr<T> |
<= DISPATCH |
Thread-safe publication slot (standard 8-byte and cache-isolated variants). |
| Persistent Handle | CPersistentRcuPtr<T> |
CPersistentRundownPtr<T> |
<= DISPATCH |
Reference-counted handle keeping payload T alive outside Read() scopes. |
| Single-Alloc Factory | MakeRcuShared<T>(...)MakeRcuSharedNuma<T>(...) |
MakeRundownShared<T>(...)MakeRundownSharedNuma<T>(...) |
<= DISPATCH |
Allocates the control block and cache-isolated payload T in one non-paged block. |
| Transient Read | anchor.Read(lambda) |
anchor.Read(lambda) |
<= DISPATCH |
Invokes lambda(T*) under RCU (DISPATCH_LEVEL) or Epoch Rundown with 0 per-object RMWs. |
| Persistent Checkout | anchor.LoadPersistent() |
anchor.LoadPersistent() |
<= DISPATCH |
Promotes the active object to a persistent handle and immediately exits the transient guard. |
| Persistent Access | handle->, *handle,handle.Get() |
handle->, *handle,handle.Get() |
<= DISPATCH |
Accesses payload fields at the caller's unchanged IRQL (supports PASSIVE_LEVEL blocking). |
| Pointer Replacement | anchor.Store(desired) |
anchor.Store(desired) |
<= DISPATCH |
Atomically swaps the active pointer and stages the displaced block into the NUMA GC queue. |
| GC Singleton | CQuiescentGarbageCollector |
CRundownPtrGarbageCollector |
PASSIVE(Sweep/Teardown) |
Driver-wide singleton: GetInstance(), SetWakeCallback(), ProcessGarbagePipeline(), Teardown(). |
| Component / Operation | Primitive / Method | Max IRQL | Description |
|---|---|---|---|
| Shared Anchor | CAtomicSharedPtr<T> |
<= DISPATCH |
Cache-line-isolated anchor (64B on x64, 128B on ARM64) holding a 128-bit counted-pointer state managed via 128-bit CAS. |
| Local Handle | CLocalSharedPtr<T> |
<= DISPATCH |
Thread-local reference-counted handle caching a direct T* payload pointer (->, *, get()). |
| Single-Alloc Factory | MakeAtomicShared<T>(...) |
<= DISPATCH |
Allocates TAtomicControlBlock<T> and cache-aligned payload T in one non-paged block. |
| Checkout / Replace | load() / store() /exchange() |
<= DISPATCH |
Lock-free handle checkout, unconditional replacement, and atomic swap-and-extract. |
| Conditional CAS | compare_exchange_strong()compare_exchange_weak() |
<= DISPATCH |
Lock-free 128-bit CAS supporting lvalue and rvalue desired handles with spurious-failure filtering. |
The example below demonstrates the primary RCU workflow: wait-free transient reads via Read(), persistent checkout and PASSIVE_LEVEL read access via LoadPersistent() (CPersistentRcuPtr<T>), lock-free writer updates via Store(), and explicit garbage collection maintenance. Substituting CRundownSharedPtr, CPersistentRundownPtr, MakeRundownShared, and CRundownPtrGarbageCollector follows the exact same pattern when the Read() callback itself must remain at PASSIVE_LEVEL.
#include <ntddk.h>
#include "KmRcuSharedPtr.h" // Or "KmRundownSharedPtr.h" if Read() accesses pageable memory
struct SFilterPolicy
{
ULONG ulMaxConnections;
ULONG ulTimeoutMs;
BOOLEAN bInspectionEnabled;
SFilterPolicy(ULONG maxConn,
ULONG timeoutMs,
BOOLEAN enableInspection) noexcept : ulMaxConnections(maxConn),
ulTimeoutMs(timeoutMs),
bInspectionEnabled(enableInspection)
{
}
};
// Shared publication slot (can be global or embedded in a device/filter context)
CRcuSharedPtr<SFilterPolicy> g_PolicyAnchor;
// ----------------------------------------------------------------------------
// 1. Fast-Path Transient Reader (IRQL <= DISPATCH_LEVEL, 0 atomic RMWs)
// ----------------------------------------------------------------------------
BOOLEAN ShouldInspectPacket(ULONG ulCurrentConnections) noexcept
{
return g_PolicyAnchor.Read([ulCurrentConnections](const SFilterPolicy* pPolicy) noexcept -> BOOLEAN
{
if (!pPolicy || !pPolicy->bInspectionEnabled)
{
return FALSE;
}
return (ulCurrentConnections < pPolicy->ulMaxConnections) ? TRUE : FALSE;
});
}
// ----------------------------------------------------------------------------
// 2. Persistent Read via CPersistentRcuPtr (Safe for PASSIVE_LEVEL Blocking)
// ----------------------------------------------------------------------------
void ExecutePassiveInspectionWork() noexcept
{
// LoadPersistent() briefly enters the RCU critical section, increments
// m_lPersistentRefCount, and immediately restores the caller's IRQL.
CPersistentRcuPtr<SFilterPolicy> pPolicy = g_PolicyAnchor.LoadPersistent();
if (!pPolicy)
{
return;
}
// Read payload fields directly via pPolicy-> at PASSIVE_LEVEL outside RCU.
// The caller can freely access pageable memory, acquire push locks, or invoke
// blocking kernel APIs while pPolicy keeps the underlying object alive.
if (pPolicy->bInspectionEnabled)
{
ULONG ulTimeout = pPolicy->ulTimeoutMs;
UNREFERENCED_PARAMETER(ulTimeout);
}
}
// ----------------------------------------------------------------------------
// 3. Writer Update (IRQL <= DISPATCH_LEVEL)
// ----------------------------------------------------------------------------
bool UpdateFilterPolicy(ULONG ulNewMaxConn,
ULONG ulNewTimeoutMs,
BOOLEAN bEnableInspection) noexcept
{
CPersistentRcuPtr<SFilterPolicy> pNewPolicy = MakeRcuShared<SFilterPolicy>(ulNewMaxConn,
ulNewTimeoutMs,
bEnableInspection);
if (!pNewPolicy)
{
return false;
}
// Atomically swaps the active pointer and pushes the displaced block to the NUMA SList
return g_PolicyAnchor.Store(Ktl::move(pNewPolicy));
}
// ----------------------------------------------------------------------------
// 4. Garbage Collection & DriverUnload Teardown (PASSIVE_LEVEL)
// ----------------------------------------------------------------------------
void OnMaintenanceWorkerTick() noexcept
{
// Invoked periodically or when woken by SetWakeCallback()
CQuiescentGarbageCollector::GetInstance().ProcessGarbagePipeline(false);
}
void OnDriverUnload() noexcept
{
g_PolicyAnchor.Store(nullptr);
CQuiescentGarbageCollector::GetInstance().Teardown();
}When deferred garbage collection is not desired—or when threads must perform conditional Compare-And-Swap (compare_exchange_strong) state transitions—use CAtomicSharedPtr<T> and CLocalSharedPtr<T>:
#include <ntddk.h>
#include "KmAtomicSharedPtr.h"
// Uses struct SFilterPolicy defined in the example above
CAtomicSharedPtr<SFilterPolicy> g_AtomicPolicy;
// ----------------------------------------------------------------------------
// 1. Reader Checkout (IRQL <= DISPATCH_LEVEL)
// ----------------------------------------------------------------------------
BOOLEAN ShouldInspectPacketAtomic(ULONG ulCurrentConnections) noexcept
{
// Atomically checks out a local reference-counted handle via 128-bit CAS.
// The payload remains valid for the lifetime of pPolicy.
CLocalSharedPtr<SFilterPolicy> pPolicy = g_AtomicPolicy.load();
if (!pPolicy || !pPolicy->bInspectionEnabled)
{
return FALSE;
}
return (ulCurrentConnections < pPolicy->ulMaxConnections) ? TRUE : FALSE;
// pPolicy destructor drops the reference; frees immediately if last owner
}
// ----------------------------------------------------------------------------
// 2. Unconditional Writer Update (IRQL <= DISPATCH_LEVEL)
// ----------------------------------------------------------------------------
bool UpdateFilterPolicyAtomic(ULONG ulNewMaxConn,
ULONG ulNewTimeoutMs,
BOOLEAN bEnableInspection) noexcept
{
CLocalSharedPtr<SFilterPolicy> pNewPolicy = MakeAtomicShared<SFilterPolicy>(ulNewMaxConn,
ulNewTimeoutMs,
bEnableInspection);
if (!pNewPolicy)
{
return false;
}
g_AtomicPolicy.store(Ktl::move(pNewPolicy));
return true;
}
// ----------------------------------------------------------------------------
// 3. Conditional CAS State Transition (IRQL <= DISPATCH_LEVEL)
// ----------------------------------------------------------------------------
bool TryBumpConnectionLimit() noexcept
{
CLocalSharedPtr<SFilterPolicy> pExpected = g_AtomicPolicy.load();
if (!pExpected)
{
return false;
}
CLocalSharedPtr<SFilterPolicy> pDesired = MakeAtomicShared<SFilterPolicy>(pExpected->ulMaxConnections + 64,
pExpected->ulTimeoutMs,
pExpected->bInspectionEnabled);
if (!pDesired)
{
return false;
}
// Atomically commits if g_AtomicPolicy still points to pExpected.
// Displaced objects are freed immediately when the last handle goes out of scope.
return g_AtomicPolicy.compare_exchange_strong(pExpected,
Ktl::move(pDesired));
}Kernel-mode synchronization primitives must be evaluated against hardware cache-coherency traffic, core migration, and scheduler preemption under multi-core contention. The included performance test suite measures memory reclamation correctness, throughput scaling, and read tail latency across bare-metal and virtualized SMP configurations.
All tests were performed on Windows 11, and the complete WDM test driver source code is included in the repository.
The test harness is implemented as a native Windows kernel-mode driver (TestKmSharedPtrDrv.sys) executing across PASSIVE_LEVEL and DISPATCH_LEVEL. The suite evaluates four shared-pointer architectures under identical test conditions:
- Baseline (
CBaselineAtomicSharedPtr): A conventional Windows driver synchronization model protecting pointer swaps with an exclusiveEX_PUSH_LOCKand managing object lifetimes with atomic reference counting. - Atomic (
CAtomicSharedPtr): The 128-bit interlocked split-reference-counting implementation using Anthony Williams' algorithm. - Rundown (
CRundownSharedPtr): The two-phase epoch-based reclamation implementation backed by OS cache-aware rundown protection (PEX_RUNDOWN_REF_CACHE_AWARE) and per-CPU hazard counters. - RCU (
CRcuSharedPtr): The DPC-based quiescent state implementation usingDISPATCH_LEVELpreemption suppression and targetedHighImportanceDPC broadcasts.
The benchmark framework controls for allocator contention and measurement overhead:
- Functional & Stress Validation: Covers handle lifecycle and CAS rollback correctness, sustained multi-producer multi-consumer contention, 6x CPU thread oversubscription, concurrent object replacement under active read dereferencing, 100% atomic exchange contention, voluntary thread yields, hardware-timed tail latency profiling, and multi-core thread scaling sweeps.
-
Kernel Memory Allocator Isolation: During thread scaling sweeps, worker threads operate on pre-allocated payload arrays. This keeps the working set cache-resident and prevents lock contention within the Windows kernel pool allocator (
ExAllocatePool2) from skewing synchronization throughput measurements. -
Amortized Batch Guards & Checkpointing: Readers in the scaling sweep use batch guards (
CRundownBatchGuardandCRcuBatchGuard) that amortize IRQL and rundown setup across read bursts. To prevent targeted quiescent DPCs from being delayed during long read-heavy bursts (e.g., 99/1 and 99.9/0.1),CRcuBatchGuard::Checkpoint()periodically lowers IRQL every 32 reads if a quiescent barrier is active. -
Tail-Latency Capture: Reader threads record individual operation durations into 1,048,576-entry power-of-two ring buffers using hardware timestamp counters (
__rdtscon x64,CNTVCT_EL0on ARM64) while writer threads pace mutations every ~3 microseconds. Samples are sorted in-place using an iterative, non-recursive QuickSort with an explicit stack bounded to$\le 64$ entries. - Integer & Fixed-Point Timekeeping: Timer calibration and cooldown calculations use 64-bit integer cycle arithmetic and Q16 fixed-point math, avoiding kernel floating-point or vector register state saves.
- Reclamation Verification: The harness tracks live allocations globally and asserts that all allocated objects are freed after every test run. Between scaling steps, synchronous garbage collection flushes drain pending queues to prevent cross-step interference.
Performance benchmarks show that lock-based synchronization and per-checkout atomic CAS reference counting hit scalability ceilings as thread counts increase.
Multi-Core Throughput Scaling (80/20, 99/1, and 99.9/0.1 Read/Write Ratios) — Scaling comparison across Baseline (
EX_PUSH_LOCK),KmAtomicSharedPtr(128-bit CAS),KmRundownSharedPtr(Epoch), andKmRcuSharedPtr(DPC Quiescent).
Figure 1: Bare-Metal Client (Intel Core i7-1165G7 @ 2.80GHz — 4 Cores / 8 Threads)
Intel Core i7-1165G7: Across all three workload ratios (80/20, 99/1, and 99.9/0.1), KmRcuSharedPtr and KmRundownSharedPtr track closely and scale from ~7.5–8.1 Mops/s at 1 thread up to 50.2 Mops/s (80/20), 53.8 Mops/s (99/1), and 54.1 Mops/s (99.9/0.1) at 8 threads. In contrast, KmAtomicSharedPtr peaks near 4 threads (~12.4–14.5 Mops/s) and flattens at ~11.6–12.0 Mops/s due to cross-core 128-bit CAS cache-line invalidations, while EX_PUSH_LOCK peaks at 2–3 threads (~9.9–11.6 Mops/s) before regressing to ~8.2–8.5 Mops/s at 8 threads.
Figure 2: Bare-Metal Legacy (Intel Core i7-4800MQ @ 2.70GHz — 4 Cores / 8 Threads)
Intel Core i7-4800MQ: On Haswell silicon, KmRcuSharedPtr and KmRundownSharedPtr scale from ~7.6–8.4 Mops/s at 1 thread up to 63.0 Mops/s and 60.5 Mops/s under 80/20 write load, and reach 68.8 Mops/s (99/1) and 69.3 Mops/s (99.9/0.1) at 8 threads (~8.3x scaling). KmAtomicSharedPtr peaks at 3 threads (~12.8–14.1 Mops/s) and declines to ~10.3–10.9 Mops/s at 8 threads as ring-bus coherency traffic saturates, while EX_PUSH_LOCK peaks at 2 threads (~9.0–10.5 Mops/s) and drops to ~8.1–8.5 Mops/s.
Figure 3: Virtualized Guest (VMware Workstation on Intel Core i7-8086K @ 4.00GHz — 8 vCPUs)
Intel Core i7-8086K, VMware Guest: Under virtualized SMP scheduling, KmRcuSharedPtr and KmRundownSharedPtr scale from ~8.1–8.9 Mops/s on 1 vCPU to 63.0 Mops/s and 58.7 Mops/s at 8 vCPUs under 80/20 write contention, and reach 66.4–67.2 Mops/s across the 99/1 and 99.9/0.1 read-dominant sweeps (~7.5x scaling). KmAtomicSharedPtr plateaus at ~13.7–14.4 Mops/s beyond 3 vCPUs, and EX_PUSH_LOCK peaks at 2 vCPUs (~11.3–11.8 Mops/s) before declining to ~8.9–9.2 Mops/s at 8 vCPUs.
The baseline EX_PUSH_LOCK experiences contention bottlenecks as thread counts increase:
- Lock Convoys & Negative Scaling: Across all tested platforms, PushLock throughput peaks at 2 threads (~10–11 Mops/sec) and regresses as core counts rise, dropping to ~8.2–9.2 Mops/sec at 8 threads. At an 80/20 read/write ratio, contending exclusive acquisitions force reader threads into kernel wait states.
- High Tail Latency Under Contention: Since acquiring an exclusive push lock deschedules contending threads, writer activity creates lock convoys. While median latency (P50) remains between 36 ns and 73 ns, tail latency rises to 39–55 microseconds at P99.9 and 368–520 microseconds at P99.99.
KmAtomicSharedPtr avoids OS thread blocking and lock convoys, keeping bare-metal P99.9 latency under 510 ns and P99.99 latency near 3.2–3.5 µs:
- Cache-Line Bouncing: Every checkout via
load()executes a 128-bit atomic Compare-and-Swap (LOCK CMPXCHG16B/CASP) against the shared anchor to incrementm_lExternalCount. - MESI Interconnect Saturation: The 128-bit atomic CAS transitions the cache line containing the anchor into the Exclusive/Modified state on the executing core, invalidating L1/L2 copies on other cores. Beyond 3 to 4 threads, coherency traffic limits throughput to ~10–14 Mops/sec across all read/write ratios.
KmRundownSharedPtr and KmRcuSharedPtr avoid shared control-block cache-line contention on the read path:
- Zero Per-Object RMWs on Read Paths: Neither primitive modifies the object's control block during
Read(). RCU executes a wait-free read path using local IRQL elevation and an acquire load, while Rundown uses lock-free CPU-local hazard registration and CPU-sharded OS rundown acquisition. - Multi-Core Throughput Scaling: Both primitives scale nearly linearly across the tested core counts. On an 11th Gen Core i7, throughput increases from ~8 Mops/sec at 1 thread to 53.8 Mops/sec at 8 threads (a 6.7x speedup on 8 logical cores). On an i7-4800MQ, throughput increases from ~8.3 Mops/sec to 69.3 Mops/sec (an 8.3x speedup).
- Bounded Read Tail Latency: In tail latency benchmarks,
KmRcuSharedPtrachieves a median read latency of 19–62 ns and a P99.99 latency of 262–3,238 ns.KmRundownSharedPtrachieves a median latency of 35–80 ns and a P99.99 latency of 327–5,273 ns.
Reader latency was captured over 10-to-15-second intervals with 75% readers and 25% writers, where writers mutate the anchor every ~3 microseconds. All metrics are reported in nanoseconds.
4 physical Performance cores / 8 logical threads, 12MB L3 Cache.
| Architecture | P50 (Median) | P90 | P99.9 | P99.99 |
|---|---|---|---|---|
Baseline (EX_PUSH_LOCK) |
65 ns | 98 ns | 46,062 ns | 447,055 ns |
| KmAtomicSharedPtr | 114 ns | 139 ns | 427 ns | 3,483 ns |
| KmRundownSharedPtr | 80 ns | 105 ns | 358 ns | 3,330 ns |
| KmRcuSharedPtr | 62 ns | 77 ns | 277 ns | 3,238 ns |
4 physical cores / 8 logical threads, 6MB L3 Cache.
| Architecture | P50 (Median) | P90 | P99.9 | P99.99 |
|---|---|---|---|---|
Baseline (EX_PUSH_LOCK) |
73 ns | 144 ns | 54,649 ns | 519,943 ns |
| KmAtomicSharedPtr | 162 ns | 258 ns | 506 ns | 3,177 ns |
| KmRundownSharedPtr | 65 ns | 83 ns | 208 ns | 327 ns |
| KmRcuSharedPtr | 40 ns | 42 ns | 136 ns | 283 ns |
Windows 11 VM, 8 assigned virtual CPUs.
| Architecture | P50 (Median) | P90 | P99.9 | P99.99 |
|---|---|---|---|---|
Baseline (EX_PUSH_LOCK) |
36 ns | 94 ns | 39,389 ns | 368,364 ns |
| KmAtomicSharedPtr | 94 ns | 130 ns | 326 ns | 6,195 ns |
| KmRundownSharedPtr | 35 ns | 51 ns | 138 ns | 5,273 ns |
| KmRcuSharedPtr | 19 ns | 26 ns | 113 ns | 262 ns |
- Median & P90 Fast-Path Overhead:
KmRcuSharedPtrrecords the lowest P50 (19–62 ns) and P90 (26–77 ns) latencies across all three environments because its read path performs a local IRQL elevation and a single acquire load with zero atomic RMW instructions.KmRundownSharedPtrfollows closely (35–80 ns P50, 51–105 ns P90), as its CPU-indexed pre-acquire slot and cache-aware rundown shard typically remain resident in the executing core's L1/L2 cache.KmAtomicSharedPtrincurs 2x–4x higher baseline latency (94–162 ns P50, 130–258 ns P90) because every checkout and destruction executes a 128-bit CAS on the shared anchor followed by a 64-bitLOCK XADDreconciliation on the control block. - P99.9 & P99.99 Tail Behavior: Under 3 µs writer mutations,
EX_PUSH_LOCKlatency increases by three orders of magnitude—reaching 39.4–54.6 µs at P99.9 and 368.4–519.9 µs at P99.99—because readers colliding with an exclusive writer are descheduled by the kernel dispatcher. The three lock-free read paths avoid OS thread blocking, keeping P99.9 latency between 113 ns and 506 ns (more than 75x–100x lower thanEX_PUSH_LOCK). At the P99.99 percentile,KmRcuSharedPtrholds to 262–283 ns on the i7-4800MQ and VMware guest because executing atDISPATCH_LEVELprevents thread preemption during the measurement window, whereasKmRundownSharedPtr(327–5,273 ns) andKmAtomicSharedPtr(3,177–6,195 ns) reflect occasional retry backoff iterations or dispatcher interrupts when an anchor swap or epoch flip coincides with a reader probe atPASSIVE_LEVEL.
- MPMC Soak (75% Read / 25% Write at 1x Concurrency): Spawns one system thread per logical processor (6 dedicated readers continuously invoking
Read()and 2 dedicated writers continuously allocating and publishing new objects viaStore()). Unlike the isolated thread-scaling sweep, writers in this workload perform live non-paged pool allocations (ExAllocatePool3/ExAllocatePool2) and trigger the test driver's client-managed garbage-collection pipeline and pool reclamation. - Heavy CPU Oversubscription (6x Concurrency, 48 Threads): Runs 48 concurrent kernel threads on an 8-logical-core system (36 readers and 12 writers, with writers continuously publishing new objects via
Store()for RCU/Rundown and alternatingstore()/exchange()forCAtomicSharedPtr). This induces frequent OS scheduler preemption inside read and write windows, testingKmRundownSharedPtr's pre-acquire hazard slots,KmAtomicSharedPtr's 65,536 external-count flush threshold, andKmRcuSharedPtr'sDISPATCH_LEVELcritical sections. - Concurrent Replacement (1.5x Concurrency, Extended Read Hold): Runs 12 concurrent threads (8 readers and 4 writers) where writers continuously displace the shared anchor while each reader executes a 25-iteration memory accumulation and invariant check on the payload inside
Read(). This widens the active read window to verify grace-period enforcement against premature reclamation while measuring throughput across longer critical sections.
Across 10-to-15-second soak workloads under contention and oversubscription, the deferred-reclamation architectures sustain higher throughput than the lock-based and split-counting baselines (all values below are reported in Mops/s — millions of completed pointer operations per second):
| Workload Scenario | Platform | Baseline (EX_PUSH_LOCK) |
Atomic (128-Bit CAS) |
Rundown (Epoch GC) |
RCU (DPC Quiescent) |
|---|---|---|---|---|---|
| MPMC Soak (75% Read / 25% Write, 1x Cores) |
i7-1165G7 | 13.53 | 17.11 | 66.84 | 56.85 |
| i7-4800MQ | 16.86 | 9.87 | 59.31 | 105.66 | |
| i7-8086K VM | 31.10 | 18.05 | 89.06 | 118.73 | |
| Heavy Oversubscription (6x Cores, 48 Threads) |
i7-1165G7 | 12.31 | 14.84 | 42.20 | 48.55 |
| i7-4800MQ | 15.28 | 9.80 | 45.05 | 73.56 | |
| i7-8086K VM | 29.44 | 18.49 | 64.10 | 80.98 | |
| Concurrent Replacement (1.5x Cores, 25-Load Hold) |
i7-1165G7 | 12.38 | 15.50 | 55.67 | 68.12 |
| i7-4800MQ | 14.52 | 9.88 | 53.41 | 109.52 | |
| i7-8086K VM | 30.38 | 19.84 | 79.44 | 143.41 |
-
Live Allocation & Batch Reclamation Efficiency (MPMC Soak): With 25% of threads continuously allocating from the kernel non-paged pool and retiring displaced objects,
KmRundownSharedPtr(59.31–89.06 Mops/s) andKmRcuSharedPtr(56.85–118.73 Mops/s) achieve 3.3x to 10.7x the throughput ofEX_PUSH_LOCKandKmAtomicSharedPtr. Offloading reclamation into$O(1)$ batched background sweeps reduces writer publication to anInterlockedExchangePointerandInterlockedPushEntrySList, whereasKmAtomicSharedPtrdrops to 9.87–18.05 Mops/s as writers and readers contend on the 128-bit CAS anchor. -
Behavior Under 6x Thread Oversubscription: When 48 threads compete for 8 logical processors,
KmRcuSharedPtrachieves the highest throughput across all tested platforms (48.55–80.98 Mops/s) because elevating toDISPATCH_LEVELprevents the OS dispatcher from preempting a reader thread mid-access.KmRundownSharedPtrsustains 42.20–64.10 Mops/s because its pre-acquire hazard window spans only a few instructions andCScopedRundownGuarddisables normal kernel APCs while holding rundown protection. -
Impact of Critical Section Duration (Concurrent Replacement): Expanding the reader callback to 25 consecutive memory loads widens the gap between deferred reclamation and traditional primitives:
KmRcuSharedPtrreaches 68.12–143.41 Mops/s andKmRundownSharedPtrreaches 53.41–79.44 Mops/s, compared to 12.38–30.38 Mops/s forEX_PUSH_LOCKand 9.88–19.84 Mops/s forKmAtomicSharedPtr. Since RCU and Rundown readers do not hold exclusive locks or dirty the control block's cache line, all 8 reader threads execute the 25-load loop concurrently from local L1 cache while writers continue replacing the anchor without blocking.
The repository separates the header-only kernel smart pointer implementations from the documentation, benchmark telemetry logs, and the WDM verification/benchmark driver.
├── Assets/ # Benchmark throughput and scaling charts (PNG)
├── Docs/ # Detailed usage guides and API reference documents
│ ├── AtomicSharedPtrApi.md # API specification for KmAtomicSharedPtr.h
│ ├── AtomicSharedPtrUsage.md # Usage patterns for CAtomicSharedPtr & CLocalSharedPtr
│ ├── RcuSharedPtrApi.md # API specification for KmRcuSharedPtr.h
│ ├── RcuSharedPtrUsage.md # Usage patterns for CRcuSharedPtr & DPC Quiescent GC
│ ├── RundownSharedPtrApi.md # API specification for KmRundownSharedPtr.h
│ └── RundownSharedPtrUsage.md # Usage patterns for CRundownSharedPtr & Epoch Rundown GC
├── PerfData/ # Raw verification & benchmark telemetry collected by the test driver
│ ├── 1165G7-KmSharedPtrPerf.txt # Bare-metal client telemetry (Intel Core i7-1165G7)
│ ├── 4800MQ-KmSharedPtrPerf.txt # Bare-metal legacy telemetry (Intel Core i7-4800MQ)
│ └── 8086K-VM_Guest_KmSharedPtrPerf.txt # Virtualized guest telemetry (VMware on Intel Core i7-8086K)
├── SharedPtr/
│ ├── KmAtomicSharedPtr.h # 128-bit CAS split-reference-counted shared pointer
│ ├── KmRcuSharedPtr.h # DPC-based quiescent state RCU shared pointer
│ ├── KmRundownSharedPtr.h # Epoch-based cache-aware rundown shared pointer
│ └── KmSharedPtrUtils.h # Shared allocators, memory barriers, NUMA & backoff utilities
└── TestDrv/
├── TestKmSharedPtr.h # Cross-architecture verification, soak, and benchmark suite
└── TestKmSharedPtrDrv.cpp # WDM kernel driver host, PAL, and telemetry logger
- Visual Studio 2026: MSVC C++ compiler toolset supporting x64 (
_M_AMD64) and ARM64 (_M_ARM64/_M_ARM64EC). - C++20 Language Standard: Enable
/std:c++20in the project's C/C++ language flags. - Windows Driver Kit (WDK): Matching Windows 11 WDK installation integrated with Visual Studio.
The smart pointer implementations in SharedPtr/ are header-only and require no static or dynamic library linking:
- Copy the
SharedPtr/directory into your driver project or add it to your project's Additional Include Directories. - Include the desired header (
KmRcuSharedPtr.h,KmRundownSharedPtr.h, orKmAtomicSharedPtr.h) after<ntddk.h>or<wdm.h>. - Compile with
/std:c++20enabled.
Native MSVC 2026 solution and project files are provided in the repository to build the evaluation driver:
TestDrv(TestKmSharedPtrDrv.sys): Compiles the WDM kernel driver hosting the functional verification, stress-soak, tail-latency, and multi-core scaling suite (Release/x64orARM64configurations recommended for benchmarking).
- Asynchronous Execution & Safe Teardown:
DriverEntryspawns the benchmark coordinator and garbage-collection worker threads asynchronously so the Service Control Manager (SCM) start request returnsSTATUS_SUCCESSimmediately. Stopping the service (DriverUnload) signals a global abort flag that slices sleeps into 50 ms ticks, allowing the driver to unwind active worker threads, flush both GC singletons, and unload cleanly at any point during the run. - Configurable Soak Duration (Optional): By default, each soak and latency phase runs for 10 seconds. Before starting the driver service, you can override this duration by creating a
REG_DWORDvalue namedTestPhaseDuration(in seconds) under the driver's parameters key:HKLM\SYSTEM\CurrentControlSet\Services\<ServiceName>\Parameters\TestPhaseDuration - Live Kernel Debugger Tracing: The driver emits real-time progress and throughput measurements via
DbgPrintExunder theDPFLTR_IHVDRIVER_IDcomponent filter atDPFLTR_ERROR_LEVEL, making output visible immediately in WinDbg or Sysinternals DebugView without modifying component filter masks. - Persistent Telemetry File: Upon completing the test suite, the driver flushes its 4 MB in-memory log buffer to
\SystemRoot\Temp\KmSharedPtrPerf.txt(C:\Windows\Temp\KmSharedPtrPerf.txt).
Selecting between CRcuSharedPtr, CRundownSharedPtr, and CAtomicSharedPtr depends on multi-core scaling requirements, tail-latency bounds under contention, read-path IRQL constraints, and whether the subsystem requires immediate reclamation or conditional Compare-And-Swap (CAS) primitives.
| Decision Criterion | CRcuSharedPtr<T> |
CRundownSharedPtr<T> |
CAtomicSharedPtr<T> |
|---|---|---|---|
| Multi-Core Read Scaling | Linear ( 0 atomic RMWs in Read()) |
Linear ( 0 per-object RMWs in Read()) |
Poor (plateaus at 3–4 threads due to 128-bit CAS bouncing) |
| Sustained Contention & Oversubscription | Highest (DISPATCH_LEVEL prevents mid-read preemption) |
High (15–40% below RCU under heavy oversubscription) | Low (128-bit CAS + 64-bit XADD cache-line contention) |
| Read Latency (P50 / P99.99 Tail) | Lowest (19–62 ns P50; tightest P99.99 bounds) | Low (35–80 ns P50; occasional epoch-flip retry spikes) | Highest (2x–4x higher P50; elevated P99.99 under writes) |
IRQL Inside Read() |
DISPATCH_LEVEL(elevated if < DISPATCH_LEVEL) |
Caller's IRQL (<= DISPATCH; normal APCs disabled) |
N/A (handle checkout via load()) |
| Pageable Memory / Blocking APIs | Via LoadPersistent()(restores IRQL immediately) |
Directly inside Read()(at PASSIVE_LEVEL) or LoadPersistent() |
Directly via load()(at PASSIVE_LEVEL) |
| Per-Object Atomic RMWs | 0 in Read()2 via LoadPersistent() (+1 / -1) |
0 in Read()2 via LoadPersistent() (+1 / -1) |
2 per load() checkout1 on handle release |
| Reclamation & Update Model | Deferred batch GC; unconditional Store() |
Deferred batch GC; unconditional Store() |
Immediate free at refcount 0;store(), exchange(), compare_exchange |
- Best overall contention and latency profile: Matches
CRundownSharedPtron linear multi-core scaling while delivering the lowest median and P99.99 tail latencies and the highest throughput under sustained write contention and 6x thread oversubscription (elevating toDISPATCH_LEVELprevents the scheduler from preempting readers mid-access). - Non-blocking hot path with
LoadPersistent()for blocking work: UseRead()for fast non-paged inspections (0atomic RMWs), and callLoadPersistent()whenever aPASSIVE_LEVELcaller needs aCPersistentRcuPtr<T>handle to access pageable memory, acquire push locks, or invoke blocking APIs. - Native
DISPATCH_LEVELexecution: Incurs zero IRQL transition overhead when called from NDIS receive paths, StorPort completion routines, or timer DPCs.
2. Choose CRundownSharedPtr<T> (KmRundownSharedPtr.h) — When Every Hot-Path Read Must Stay at PASSIVE_LEVEL
- Linear scaling without raising IRQL: Provides the same multi-core scaling curve as RCU when the primary fast path itself must touch paged pool/code or invoke synchronous
PASSIVE_LEVELAPIs on every read (where callingCRcuSharedPtr::LoadPersistent()on every access would reintroduce per-object refcount contention). - Trade-off vs. RCU: Because readers execute at
PASSIVE_LEVEL(with normal kernel APCs disabled), they remain preemptible byDISPATCH_LEVEL/DIRQL activity and can experience retry backoff during concurrent epoch flips—resulting in slightly higher P99.99 tail latencies and lower throughput than RCU under heavy CPU oversubscription or rapid pointer replacement. - Extended read hold times: Holding an epoch rundown shard does not mask DPC delivery or thread scheduling on the local processor.
3. Choose CAtomicSharedPtr<T> (KmAtomicSharedPtr.h) — Only for Immediate Reclamation or CAS State Machines
- Significant scaling and latency trade-off: Every
load()and handle destruction executes a 128-bit CAS on the shared anchor plus a 64-bit atomic fetch-add on the control block. Cross-core cache-line invalidation causes throughput to plateau beyond 3–4 threads (~4x–6x slower than RCU/Rundown at 8 threads) and produces 2x–4x higher baseline and tail read latencies. - When to use: Restrict to low-to-moderate concurrency paths that strictly require immediate deterministic destruction upon releasing the final handle (no periodic/event-driven GC sweep) or lock-free
compare_exchange/exchangestate transitions.
This library addresses a common scalability bottleneck in Windows kernel driver design: protecting shared, read-mostly state across multi-core and multi-socket NUMA topologies without incurring the lock convoys of EX_PUSH_LOCK or the cross-core cache-line invalidation traffic of per-access atomic reference counting.
By separating transient read access (Read) from persistent reference-counted ownership (LoadPersistent), mapping Quiescent State-Based Reclamation (QSBR) and Epoch-Based Reclamation (EBR) onto documented Windows NT dispatcher and rundown facilities, and isolating payloads from control-block metadata within single-allocation pool blocks, CRcuSharedPtr and CRundownSharedPtr eliminate per-object atomic RMW operations from the read fast path. Paired with CAtomicSharedPtr for immediate-reclamation and CAS state-machine workloads, the suite provides x64 and ARM64 kernel drivers with Driver Verifier-compliant synchronization primitives that scale with core count while maintaining sub-microsecond read tail latency.
This project is licensed under the Apache License, Version 2.0.
You may not use this file except in compliance with the License. You may obtain a copy of the License at: http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations.