md-kmec - Many-Parity Software RAID for Lustre OSTs - LAD26
In June, Hiroshi Nishida, an engineer at The Lustre Collective, posted an eight-patch series to the Linux kernel's RAID list.
The problem
Every write to a RAID-5 or RAID-6 array passes through md’s stripe cache, a pool of in-memory stripes that the parity calculation works on. The ceiling on its size, the number of hash locks that guard it and the number of stripes handled in one batch are compile-time constants, chosen for much smaller machines. Stripe handling runs on one thread per array unless worker groups are turned on, and they are off by default. With many cores and many drives that means contention on every stripe and on the cache itself, and throughput stops rising long before the drives or the CPUs are busy.
What the patches change
Scalability - More hash locks on the stripe cache, 8 to 32, a far higher ceiling on its size, 32,768 to 262,144 stripes, and larger batches, 8 to 32 stripes, so that the extra worker threads spend less time waiting on each other.
The rebuild path - A window of 32 stripes per resync call instead of one, and a share of the stripe cache held back for user I/O so a rebuild does not crowd it out.
Types - Several size and count fields widened for larger arrays; groundwork rather than behaviour. With a single handling thread the series is neutral. The gains appear as worker threads are added.
With a single handling thread the series is neutral. The gains appear as worker threads are added.
The numbers
Stock md, tuned stock md and md-kmec on an 8+2 NVMe array, from the LAD’26 talk
Hiroshi's cover letter measured a RAID-6 array of six ramdisks on a 22-core host, chosen so that device latency would not mask the CPU-side effects. With four worker threads, 4 KiB random writes improved by 17 per cent and database-style mixed workloads by about 10 per cent. Ramdisks overstate the contention relative to real drives; the cover letter says so.
On real hardware the defaults matter more. A table from Hiroshi's benchmarks, shown in the md-kmec talk at LAD'26, has an 8+2 NVMe array where setting the values md-kmec ships with (worker threads, a larger stripe cache, skipping redundant copies) more than doubled random 4 KiB write throughput over stock md. That is stock md tuned through sysfs, with no patches at all. Since June the work has been about making those settings the default.
Where it stands
Phoronix covered the series, under the headline "Linux MD RAID5 Seeing Scalability Improvements Up To 17%". Yu Kuai, who maintains md, reviewed it, and in July Hiroshi reposted the work as three series: the type widening, the rebuild-path changes, and a new series that sizes worker groups, hash locks and the stripe cache from the machine's CPUs and memory, with worker groups on by default. On a 16-disk NVMe array, turning worker groups on by default measured 2.1 to 3.2 times the throughput of the stock single-threaded default.
One patch from the type series is in Linux 7.3, released in October. The rest are being revised with the maintainers' feedback: per-array controls through sysfs and mdadm rather than module parameters, and testing across architectures before any defaults change.
md-kmec’s declustered layout: narrow groups at permuted positions, so a rebuild is spread across the pool. From the LAD’26 slides.
Why we are doing this
The patches are one strand of md-kmec, Hiroshi’s many-parity RAID for wide Lustre OSTs, presented by Artem Blagodarenko at LAD’26. The aim is to bring md up to what a wide array needs: surviving more than two failures, healing silent corruption and rebuilding many times faster, while keeping md’s direct write path and the tools around it. md-kmec sits on the same write path as RAID-5 and RAID-6, so the scaling limits it ran into on wide arrays are common to all three. Those fixes go upstream first; md-kmec builds on top of them.
