Alpine Linux Persistence and Storage Summit 2022

In October 2022 the 5th Alpine Linux Persistence and Storage Summit (ALPSS) was held. The organizers Christoph Hellwig, Johannes Thumshirn and our own Richard Weinberger were proud to welcome a big and motivated bunch of the Linux Kernel Storage community from various nations around the world. In old tradition the get-together took place at Lizumer Hütte (2019 m) in the Tuxer Alps.

Day 1: It’s all about hiking – and food

The main ingredients for a successful ALPSS: hot topics, Linux-focused brains, hiking and food. Or was it the other way round? However, we started with a pre-conference lunch. Afterwards we felt ready for the hike, starting at Lager Walchen in Wattental. With sweat and some photo-stops on the picturesque Zirbenweg we reached the Lizumer Hütte after some 2 hours. The lodge-keeper gave us a warm welcome and we switched back to where we started the day: food (plus some drinks).

Day 2: Storage mixed with good weather

The first round of presentations started with a talk by Richard Weinberger on his efforts to improve NFS re-export. While most ideas to reduce the overhead of file handle wrapping didn’t find love because they fix problems that exist only on NFSv3, the suggested approach to fix the ongoing cross-mount issue at least didn’t result in rejection. In the next talk Luis Chamberlain gave an overview of kdevops, blk- and fstests where he outlined where to expect determinism and where not. The conses was that testing the kernel is hard and traditional approaches often don’t apply.

After a tea break both Alexei Starovoitov and Christoph Hellwig spoke about eBPF and how to use it outside of the Linux kernel, for example directly on computational storage devices. The general conses was that having small programs on network or storage devices makes sense for some workloads - but instead of having eBPF support in hardware it’d make more sense to have a tiny CPU on these systems which can run a JIT. eBPF is way too Linux specific and restricted.

The final two presentations of the first day were given by Daniel Wagner and Hannes Reinecke. Daniel spoke about his ongoing work on libnvme and nvme-cli. Over the last year the userspace tooling around NVMe saw a major cleanup which will hopefully ease further maintenance. Hannes explored the possibilities on how to have an isolated userspace helper process around which can initiate a kTLS session for the kernel when needed. Especially when the root filesystem needs a multipath failover. One promising approach is using Linux’s usermode driver where the helper program is part of the kernel binary.

Day 3: More Storage

Despite the early hour the very first presentation of the third day resulted quickly in a heated discussion. Pankaj Raghav gave a talk on how to support non power of two (npo2) zoned devices in Linux. While the topic sounds not too scary it caused immediate fear and anger among certain developers. Currently Linux enforces that zones have power of two sizes. While this is considered as a sane technical decision, no specification enforces this constraint. Therefore some storage vendors expect Linux to support also non power of two sized zones. So far no consent was found on whether supporting npo2 devices is worth the technical burden nor whether there are users in the wild. Pankaj’s presentation was followed by Alex Graf. He discussed the possibilities of having thousands of NVMe disks attached to a single virtual machine. Using more disks per NVMe namespace would allow much more disks but due to shared queues this could cause performance problems. The other option is using NVMe over fabrics. While this scheme allows the needed amount of disk without performance penalties the user experience changes. Instead of having the disks directly attached to the VM, the operating system has to initiate the network attached disks. It was suggested to have single discovery server which helps the operating system finding all targets.

After a short break Paul McKenney presented a new way to work with RCU. Instead of having a global grace period were all pending RCU work is waited for, he proposed a way to avoid waiting. Alexei Starovoitov followed and gave a short presentation on how to use C extensions to catch common programming errors - mostly for use in eBPF.

The final two presentations where held by Tudor Ambarus, Pratyush Yadav and Keith Busch. Tudor and Pratyush gave an overview of Linux’s SPI-NOR flash framework and their ongoing effort to work with various flash chips that do not follow the specification. Last but not least, Keith gave insights in various performance improvements for Direct-IO he is currently working on. Especially combined with an IOMMU Linux has plenty of potential for optimizations.

Day 4: Back to reality

Time goes fast when you’re having a good time. In a blink it was time to pack our rucksacks again and leave for the valley. To make it a round trip – and be a bit quicker – we took the road down to Lager Walchen. With everybody safe back in lower altitudes, we said our farewells. Again, ALPSS kept what it promised: an ideal mix of in-depth, to-the-point talks and discussions on the latest in Linux Storage, peppered with the stunning panorama of the Tyrolean Alps. Sure thing, we are already looking forward to the next ALPSS.