Lightbits: Linux & NVMe/TCP

Mainstream Linux Distributions Now Include Support for NVMe/TCP

By Sagi Grimberg

Co-founder and Chief Technology Officer

November 26, 2019

Since its ratification in last November, NVMe/TCP is gaining significant momentum in the storage arena - in large part because of its availability in the mainline Linux kernel and the fact that it is based on an industry standard guaranteeing vendor neutrality and interoperability. With NVMe/TCP transport drivers included in the Linux kernel v5.0 (since March) and with more of the main Linux distributions adding support for this technology, it is becoming increasingly easy for users to trust in the stability and maturity of the different NVMe-oF solutions.

Technology MUST first be consumable in order to be viable, regardless of how innovative it is. Now as with any new technology, propagation from upstream into the mainstream distributions takes time, but it is key to its success. The main Linux distributions are the ones that really power the vast majority of the world's computer systems. One cannot underestimate the power of technology working "out-of-the-box" eliminating the friction of SW compatibility when it is upgrading constantly. Hence, the availability of NVMe/TCP support in the main Linux distributions plays a key role in its adoption in the field, making it ready for "prime time".

Now those of us that are really serious about NVMe-oF, know that the most resilient disaggregated storage systems are highly available, clustered and distributed. Multipathing is an important piece of the puzzle when it comes to fabrics. NVMe offers native multipathing capabilities since kernel v4.15 and added support for the standard "Asymmetric Namespace Access" multipathing scheme in kernel v4.19. NVMe native multipathing is extremely efficient and will guarantee interoperability and correct operation across the vendors in the industry. So the adoption of NVMe-oF products also depends on proper multipathing supported by the OS distribution.

Primary Certified Linux Distributions with NVMe/TCP and NVMe Native Multipathing

Here is the current list of some of the primary certified Linux distributions that now include more recent upstream Linux features such as NVMe/TCP and NVMe native multipathing:

Red Hat - The major distributor of Linux on the planet.
- RHEL 8.1 - Supports all NVMe-oF transports including NVMe/TCP and also supports native NVMe symmetric and asymmetric multipathing. - CentOS 8-Stream - This new distribution from Red Hat is essentially a rolling release that is more advanced than CentOS and RHEL, but that essentially previews technologies that will be introduced in RHEL 8.1, CentOS 8.1 as well as future releases. It is based on Linux kernel 5.3 (currently) and supports NVMe/TCP as well as NVMe native multipathing. Red Hat describes CentOS Stream as a midstream distribution between Fedora and RHEL/CentOS, providing a cleared-path for participation in creating the next versions of Red Hat releases. - Coming soon: Red Hat community edition CentOS 8.1, which will inherit from RHEL 8.1 support for NVMe/TCP and NVMe native multipathing.
SUSE - One of the most popular Linux distributions in the world.
- SUSE Linux Enterprise Server 15 - Supports both NVMe/TCP and NVMe native multipathing in-box. - openSUSE 15.1/15.2 - An open source community project independent of but sponsored by SUSE and promotes the use of Linux everywhere. openSUSE offers similar support as SUSE Linux Enterprise 15.
Ubuntu - Another one of the most popular Linux distributions in the world.
- Ubuntu 19.04 (Disco Dingo) and 19.10 (Eoan Ermine) both support NVMe/TCP and NVMe native multipathing.
Fedora and Debian - Upstream rolling distributions
- Debian kernels buster and sid support NVMe/TCP and NVMe native multipathing. - Fedora core 28 and on support NVMe/TCP and NVMe native multipathing.

For those that have yet to upgrade to one of these new Linux distributions, Lightbits also offers back-ported NVMe/TCP host drivers for several popular OS distributions and can add more based on customer and user demand.

A year since its ratification, most of the major Linux distributions are adopting NVMe/TCP, making it a mainstream technology that is viable and consumable. In the coming months, you can expect more of the same. We anticipate all Linux distributors will soon include support for a modern NVMe stack including NVMe/TCP and native multipathing. Stay tuned!

Learn More

FMS 2020 Keynotes & Sessions

FMS 2020 Sponsors & Exhibitors