Find out ways to future proof your storage

Non Volatile Memory Express (NVMe) and Storage Class Memories (SCMs) are taking the solid-state drive revolution to the next level. This comes as no surprise and here's why: NAND flash based Solid-State Drives (SSDs) had no rotating parts and brought down drive latencies from the order of a few milliseconds offered by rotating media to sub 100 microseconds. Considering that a microsecond is literally 1/1,000 milliseconds, that’s pretty fast. The interconnect to SSDs was predominantly Serial Attached SCSI (SAS), which added about 20us to 40us to the sub 100us media latency offered by NAND flash. This kind of latency offered by the interconnect managed to hide behind the media latency. The emergence of Storage Class Memories has brought down media latencies to below 10us, thus rendering SAS and other existing interconnects such as Fibre Channel(FC) unviable to be used with SCMs. They are not only slow, but they also inhibit the true potential and powers of SCM. This is where Non Volatile Memory Express comes in: it’s the answer to the current storage protocol and interconnect bottlenecks.

It is to be noted that the low media latency offered by any SCM also brings it closer to the main memory, and it can be used as an extension to system main memory as NVDIMMs (Non Volatile Dual Inline Memory Module).

The Rise of a new Class of Storage

NVM is the umbrella term which expands to Non Volatile Memory. As the name suggests, it is persistent, non-rotating, electronic and memory class data storage media. It includes NVRAM which is essentially battery backed RAM and often used by storage arrays to store the operation log of recent changes, SSDs which are based on NAND flash and used either as a cache or a capacity tier, and the newly emerging SCM media such as 3D-Xpoint, ReRAM, Z-NAND, etc.

Non Volatile Memory Express (NVMe) is a storage transport protocol just like SAS and SATA which can be used to access all types of NVM media. But unlike SAS and SATA, it unlocks the real potential of NVM media by offering low interconnect latency and leverages the inherent parallelism of the NVM devices. What makes NVMe so powerful is the access to those NVM devices over PCIe, a newer set of commands, a streamlined stack and most importantly an incredibly massive number of IO queues. A true data superhighway!

NVMe and NVMe-of adoption

NVMe is used for directly attached NVM media i.e. NVMe is used as the backend interface between storage controllers (or servers) and NVM devices . When connected to the back-end of storage controllers NVMe significantly lowers the access latency.

NVMe-of is used to connect servers to external storage arrays over storage fabrics like Ethernet, FC or InfiniBand using low latency RDMA interfaces. When used for connectivity to external storage arrays, the dramatic boost in speeds and protocol efficiencies translates to significant improvements in cost and performance of data centres.

Fig1: Storage controller with NVMe and SATA devices. This figure shows NVMe as the back-end interface to connect NVM devices, and some basic differences between the NVMe and SATA interfaces.

Fig2: Using NVMe-of as the front-end interface to connect servers/hosts to shared PCIe storage controllers.

NVMe and NVMe-of adoption is expected to revolutionize data centres and have a huge positive impact on businesses. Traditional data centre applications such as databases, CRM and analytics will run more quickly and more efficiently - because NVMe and NVMe-of reduces access latencies and storage CPU utilization. What this means is that organizations would be able to get more out of their Oracle, Microsoft, VMWare, etc. license costs, as these applications would be able to run more efficiently by (a) utilizing CPU time earlier used by storage protocol processing and (b) accessing data faster . Organizations will be able to apply machine learning with real-time training and real-time inference to much larger datasets. Organizations will be able to analyse huge amount of information that is being generated today, by handheld devices, IoT sensors, etc. It could go some way in helping to tame the explosion of data.

What NetApp is up to in the NVMe & SCM space

NetApp serves on the Non Volatile Memory Express board where the NVMe standard is being developed. Along with other leading technology companies, NetApp has worked to develop and standardize multi-stream writes to improve the performance and endurance of SSDs. This feature is being incorporated in the NVMe standard too. NetApp already had a Flash Cache product where the CPU could directly talk to the cache module over PCIe. This proprietary technology has been changed to the NVMe standard. As of today, the company has already shipped more than 6PB of NVMe flash. In the near future, NetApp AFF and E-Series will use NVMe-of. NetApp software stack is also undergoing changes to improve the stack latency to be able to fully exploit the advantages of faster interconnects such as NVMe and faster storage media such as SCM. All these in a way that lets customers continue to use the full suite of NetApp data management capabilities.