Digital Storage and MemoryTechnology (Part 1)Tom Coughlin, Roger Hoyt, and Jim HandyNovember 2017This work is licensed under a Creative Commons Attribution-NonCommercial 3.0 United States License.

This report is the first of a two-part series for the IEEE discussing developments indigital storage technology. In this first part, the opening section looks at developmentsin basic digital storage and memory technology at the storage and memory devicelevel, extending to interfaces and protocols for connecting storage systems such ashard disk drives (HDDs) and solid-state drives (SDDs) to other computer hardware tocreate storage systems.The second section discusses developments in storage systems and software tomanage that storage. Developments and expected developments in semiconductormemories are examined as well as the move from volatile to nonvolatile memory,including flash memory and emerging nonvolatile memories such as threedimensional (3-D) XPoint and magnetic random-access memory (MRAM). Currentand expected developments in HDD and magnetic tape technology as well as opticalstorage technology are reviewed. Important factors that prompt storage-systemdevelopers to use one or more of these storage technologies to create workingstorage systems are described.The final section looks at the business of storage and memory, comparing capitalequipment costs, industry consolidation trends, regional developments, andemployment, ending with recommendations for industry professionals to remainrelevant and employed during this time of intense change in the memory and storageindustry.2

Table of ContentsTable of Contents . 3Storage and Memory Device Developments. 5Hard-Disk Drives . 6Magnetic Tape . 8Optical Storage . 9Flash Memory . 10Emerging Nonvolatile Memory Technologies . 12Choosing Storage Technologies. 13Response time . 14Touch rate . 15Touch rate versus response time . 15Technology regions . 16I/O object size curve . 17Digital Storage Interfaces, Protocols, and Form Factors . 20Device Interfaces: SATA, SAS, NVMe, and the Memory Channel . 20SATA . 20SAS. 22NVMe over PCIe. 22Memory Channel . 23Network Interfaces: Ethernet, Infiniband, and NVMe-oF . 25Ethernet . 25InfiniBand . 26NVMe over fabric . 26Business Trends for Storage and Memory Devices. 27Capital Investments, Regional Developments, and Industry Consolidation . 27Recommendations for Industry Professionals. 28How IEEE Can Help You Stay Employable. 29Biographies . 313

Table of FiguresFigure 1. Data created compared to data stored. (Reproduced by Tom Coughlin from theSeagate Technology keynote talk at the 2017 Flash Memory Summit.) . 5Figure 2. The ASTC HDD areal density road map. (Courtesy of ASTC.). 7Figure 3. The LTO magnetic tape road map. (Courtesy of the LTO Consortium.) . 9Figure 4. The Blu-ray optical disc road map. . 10Figure 5. The NAND flash technology road map. (Courtesy of TechInsights.) . 11Figure 6. A smart modular nvNITRO NVMe storage device. (Courtesy of Everspin.) . 12Figure 7. Memory-storage hierarchy versus performance. (Objective Analysis, 2017.). . 14Figure 8 The touch rate versus response time, indicating various types of uses. . 16Figure 9. The storage technology regions overlaid on the touch rate/response time chart inFigure 8. . 17Figure 10. The touch per year and response time for 100% random I/O in a 4-TB capacityHDD. . 18Figure 11. Touch per year and response time for 4-TB capacity HDD, LTO tape, and Blu-raydiscs. . 19Figure 12. History and projections for annual shipped capacity of magnetic tape, SSDs (withNAND flash), and HDDs. (Coughlin Associates, 2017.). 20Figure 13. SATA cable and connector. (Courtesy of SATA-IO.). . 21Figure 14. An M.2 form factor NVMe storage device. (Courtesy of Samsung.). . 23Figure 15. An NVDIMM-N block diagram. (Objective Analysis, 2017.). . 24Figure 16. A Samsung flash factory in Xi’an, China. (Courtesy of Samsung.) . 284

Storage and Memory Device DevelopmentsInternational Data Corporation has estimated that, in 2016, 16 zettabytes (ZB, 10 tothe 21st power bytes) of data were created and that this amount of generated data willincrease to 163 ZB by 2023. At the same time, the shipped storage capacity for HDDs,magnetic tape, and flash memory was about 4.3% of the estimated generated ZB in2016 (0.7 ZB) and is expected to be between 2 and 3% of the estimated total ZBgenerated in 2023. This is because most of the data generated are meant to beprocessed and used, rather than stored as raw data. For instance, sensor data fromthe Internet of Things (IoT) devices or driving assistance in cars are meant to beprocessed to make real-time decisions and require memory and storage to enable thisprocessing. The results of this processing might be stored for a longer time. Figure 1shows a visual representation of estimated data generated and stored out to 2025.Figure 1. Data created compared to data stored. (Reproduced by Tom Coughlinfrom the Seagate Technology keynote talk at the 2017 Flash Memory Summit.)Thus, different types of nonvolatile memory and digital storage are required fordifferent applications. Further, more than one technology might be used together withothers to achieve the optimal tradeoff in cost versus performance. This results in ahierarchy of memory and storage technologies.Digital storage and memory technologies began in the 1950s with the introduction ofdigital magnetic tape, magnetic HDDs, magnetic drums, and various early memorytechnologies. Since their introduction, digital storage and nonvolatile memory havebroadened to encompass a wide range of technologies and applications. Currenttechnologies include solid-state nonvolatile flash memory based on NANDsemiconductor cells, ferroelectric, MRAM, magnetic recording on rigid disks and tape,and a number of different optical storage technologies.5

The list of potential additional emerging digital storage and nonvolatile memorytechnologies that have already or may soon enter the mainstream continues toincrease, including solid-state phase-change memory (PCM) such as Intel’s Optane,hybrid flash/disk drives, various resistive memory storage, spin-torque MRAM, andother magnetic spin-based memories and optical holographic-based storage.Hard-Disk DrivesMagnetic-recording-based HDD technology continues to play a large and importantrole in mass data storage. With about 424 million shipped units in 2016 and expectedvolumes slowing to 317 million units by 2022, HDD technology continues as a costeffective, nonvolatile online storage solution (particularly for cooler long-term datastorage). Price reductions and cost efficiencies led to vertical integration of all but oneof the remaining three HDD manufacturers (Western Digital, Seagate Technology, andToshiba) and the further consolidation of head, disk, electronics, and mechanicalsuppliers.Enablers for continued HDD areal density and performance improvements includecurrent perpendicular-to-plane giant magnetoresistance (CPP GMR) heads, heatassisted magnetic recording, microwave-assisted magnetic recording, shingledmagnetic writing, bit patterned media, advanced signal processing, and improvedsecurity through disk data encryption.HDDs are gradually declining in the personal computer market although, for users whoneed 1-TB or higher capacity, they are still the preferred storage media because oftheir lower cost. These HDDs may be combined with an SSD or act as a hybrid HDDwith embedded flash memory to improve the overall drive performance. The maingrowth in HDDs will be for inexpensive storage for content repositories that need lowerlatencies than magnetic tape and optical library systems (note that HDD costs are nowlower than US .04 per raw gigabyte). While total unit shipments are expected todecline going forward, the consolidation in the industry and the growing importance ofenterprise HDD storage will limit the decline in total HDD revenue (and profit).The Advanced Storage Technology Consortium (ASTC), consisting of representativesfrom HDD companies as well as major HDD component vendors, creates annual roadmaps for HDD technology. In particular, the ASTC charts the areal density of magneticrecording over time. The areal density is the number of bits that can be stored on aunit of disk surface and the product of the average track density on a disk surfacetimes the average linear density around those tracks. Figure 2 shows the 2016 ASTCHDD technology road map.Perpendicular magnetic recording (PMR) was introduced into HDDs in 2004. At thattime, HDD areal density growth was over 100% annually (so you could buy over twicethe storage capacity for the same sized disk drive from one year to the next). Today,nearing the end of the density improvements possible with perpendicular recordingalone, annual areal density growth is between 10 and 15%. HDD areal densities willneed to continue growing to retain their cost advantage over other storagetechnologies.6

Figure 2. The ASTC HDD areal density road map. (Courtesy of ASTC.)To continue areal density growth, new technologies must be used in HDDs. Shingledmagnetic recording writes adjacent tracks overlapping one another (like shingles on aroof) to increase the track density, while two-dimensional magnetic recordingcombines this with multiple read heads on adjacent tracks. These technologies canincrease the areal density, but they are best used for HDDs that write data only oncebecause overwriting previous data involves a multistep process that increases thewrite time of the drive. For applications that need to rewrite often, other approaches toincrease areal density must be used.Up until October 2017, this additional improvement was expected to be what is calledheat-assisted magnetic recording (HAMR), where a laser is used to allow the HDDwrite transducer to write on very-high-coercivity magnetic recording media on a disksurface. The high coercivity allows writing higher linear densities, but conventionalwrite heads can’t produce sufficient field to write this media. The heat of the laserlowers the media coercivity, so the write head can write on the media. HAMR is theapproach that Seagate Technology has announced it will use for the next generationof HDDs, with engineering samples available in 2018 and products in 2019.In October 2017, HGST, a division of Western Digital, announced it would manufactureHDDs that used another technology to help write heads write on high-coercivity media.HGST plans to put a spin-torque oscillator into the gap of the write head that willgenerate microwaves of high enough energy to help the write head write on the media.This approach is called microwave-assisted magnetic recording (MAMR). WesternDigital says that engineering samples of these products will be available in 2018, with7

production in 2019. The company also says that MAMR will allow it to increase arealdensity by 15% annually, making 40-TB HDDs possible by 2025.Whether using HAMR or MAMR, higher-capacity direct-overwrite HDDs will beavailable with higher storage capacities into the next decade. These HDDs will enablestoring the vast bulk of the world’s data with acceptable latency and data rate at a lowprice.Magnetic TapeLinear tape technology leverages as much as possible from the much higher-volumeand more advanced disk drive industry. Tape-drive developers use a number oftechnologies available from disk drives. These include read and write heads, readpreamplifiers, write driver electronics, channel electronics such as partial-responsemaximum-likelihood (PRML) coding and detection, servo control systems, and errorcorrection codes.The challenge for the tape-drive developer is to adapt these technologies tomultichannel devices. With multiple channels recording or reading simultaneously, theerror correction code can be spread across all of the channels and thus provide addedreliability in recovering the data. It is noteworthy that, although tape has dependencieson technology developed earlier for disk drives, tape products currently deliver one totwo orders of magnitude higher bit-error reliability than disk products.Linear tape-open (LTO) tape cartridges (the most popular digital tape format) includea built-in file system capability. This file system, called the linear tape file system(LTFS), was first introduced with the LTO-5 tape format and expands the capabilitiesof tape library systems. The LTFS allows network-attached storage (NAS)-like singletape behavior and, in library systems, also allows faster access time to data. Recently,RESTful application programming interfaces using LTFS tapes have also enabledobject-based storage built around the use of magnetic tapes. The LTO road map wasrecently extended two more generations to LTO-12; see Figure 3.8

Figure 3. The LTO magnetic tape road map. (Courtesy of the LTO Consortium.)LTO tapes are now available with a price below US .01 per gigabyte of storagecapacity. The latest LTO-8 tapes offer 12 TB of native (uncompressed) storagecapacity and over 470 MB/s sustained data rates.LTO-12 will support 192 TB of uncompressed storage capacity. With new LTOgenerations being introduced roughly every three years, LTO-12 should be availableby 2029. Recent laboratory demonstrations by IBM and Sony have shown magnetictape areal densities on sputtered film tape of 201 Gb/i2, which would lead to a 330-TBuncompressed LTO-style cartridge.Optical StorageDigital optical discs have been used to record and store information since the 1980s.Optical discs record information by creating areas of differing contrast on the discsurface that are used for encoding digital bits. While the bulk of optical discs havebeen used for the distribution of music and video, there has long been a market forconsumer and enterprise write-once and write-multiple-time products. While opticaldrives are becoming scarce in computers, they are still being used for write-oncearchiving of content by some organizations. The latest generation of optical mediaemployed for data archiving uses Blu-ray technology. Figure 4 shows a road map forwrite-once Blu-ray optical storage technology for archiving applications released bySony and Panasonic.9

Figure 4. The Blu-ray optical disc road map.Blu-ray write-once optical discs are now available with 300 GB per disc capacity.These discs are combined into cartridges containing 12 discs (resulting in a 3.3-TBtotal cartridge capacity) that are loaded in a library system, like the tape libraries usedby many organizations. By 2018 or 2019, it is expected that 500-GB discs will beavailable, with 1-TB discs expected by 2021–2022 (based upon Blu-ray Archive Discstatements). Like magnetic tape, optical discs are used for high-latency archiveapplications.Flash MemoryAs the fastest-growing market in the history of semiconductors, NAND flash hasexpanded to over US 35 billion in annual revenues, largely by displacing other storagemedia and enabling new applications.This significant growth was fueled by NAND’s cannibalization of the film market forcameras, the videotape market in camcorders, the floppy disk market with universalserial bus (USB) flash drives, and the compact disc (CD) market in MP3 players anddigital maps, as well as the use of CD-R and CD-RW for data sharing. The technologyalso grew by displacing small form factor hard drives in MP3 players and camcordersand some HDDs in laptop computers, and it is gradually replacing high-performanceHDDs used in enterprise and data-center applications.Although NAND flash solid-state drives have replaced HDDs in several markets, theyhave also served to complement HDDs as a nonvolatile cache between HDD storageand dynamic RAM (DRAM) main memory. Spending an equivalent amount of moneyon flash memory capacity generally provides a greater performance improvement thanspending that money on more DRAM. Thus, NAND flash represents a threat to DRAMmemory growth.10

Flash memory-based SSDs are poised to become the primary storage for manyenterprise applications, with HDDs being used for higher-latency, cooler storage.Although, due to greater manufacturing costs but lower storage capacity, flashmemory is more expensive than HDD technology on a raw cost per terabyte basis,there are some applications where flash memory is approaching or has achievedoperating cost parity with HDDs.It is anticipated that flash memory will continue to play a larger role in both low- andhigh-end computing systems as well as consumer products, as prices decline andcapacity and reliability improvements are realized. The price decline may be fasterthan that expected based on Moore’s law, enabled by capacity improvements frommultibit cells and three-dimensional (3-D) multilayer flash memory, although scalinglimits due to small cell size and fundamental signal-to-noise limits may eventuallyinhibit the technology’s advance. Also, the endurance tradeoffs for increasing thenumber of bits per cell may not be acceptable for high write applications. Current flashmemory products are up to 4 b per cell and 64 3-D layer stacks. In 2017, WesternDigital (SanDisk), Toshiba, and Samsung announced that they would start tomanufacture 92- to 96-layer 3-D NAND products.AsFigure 5 shows, products with 128 layers and higher should become available by theend of the decade, likely with up to 4 b per cell. Laboratory demonstrations show thatseveral-hundred-layer 3-D flash memory may be possible.Figure 5. The NAND flash technology road map. (Courtesy of TechInsights.)Flash memory is temporarily in short supply compared to demand (and thus highercost) as the industry transitions to 3-D structures to continue the increase in storagecapacity per chip, because planar flash memory line widths have nearly reached their11

minimum. While prices for flash memory at the end of 2017 were higher than a yearago, these should start to decrease again in 2018 as the 3-D fabricators become fullyonline.Emerging Nonvolatile Memory TechnologiesNew solid-state storage technologies will displace today’s prevailing technologies onlyby offering better value in the markets they serve. All technologies have technicalweaknesses that position them for replacement. For example, HDDs drain battery life;and, although NAND flash memory may offer nonvolatility, it suffers from slow writingspeeds and a limited number of rewrite cycles. However, these emerging memorieshave found niche applications and are likely to find ways to displace other memorytechnologies or enable new applications.Today’s most promising alternative technologies are MRAM, viewed as a potentialsuccessor to DRAM; resistive RAM (ReRAM), which many expect to inherit the marketfor NAND flash; and PCM, which is being newly promoted in the role of an additionalmemory/storage layer to fit between NAND flash and DRAM.MRAM is a nonvolatile memory based on magnetic storage elements integrated withcomplementary–metal-oxide-semiconductor circuitry. Magnetoresistive-based MRAMnonvolatile memory products are established in the marketplace, although at fairly lowcapacity and high prices. The long-term success of this technology critically dependson a successful transition to more areal-efficient and less costly spin-torque or thermalswitched cells, which enable higher capacities and a wider application spectrum. Spintorque switched MRAM products are now available from Everspin and as anembedded memory from its manufacturing partner, GlobalFoundries. An expected2018 delivery of spin-transfer torque MRAM products by Toshiba and Samsung(among others) is indicative of vibrant and growing progress in this technology. Figure6 shows an image of an MRAM nonvolatile memory express (NVMe) Everspin device.Figure 6. A SMART Modular nvNITRO NVMe storage device. (Courtesy ofEverspin.)12

ReRAM is a large umbrella category that covers a number of subcategories, namelyPCM, programmable metallization cell memory (PMCm), and oxygen-depletionmemory. All of these products use changes in resistance to store data.The announcement by Intel and Micron of their 3-D XPoint technology, now availableas Optane SSDs, could lead to a wider use of PCM technology. 3-D XPoint is expectedto play an important intermediate role between DRAM and flash memory in the nearfuture. Intel currently offers Optane as SSDs, but Micron and Intel have said that theywill deliver this technology as dual in-line memory modules (DIMMs) in computermemory channels.Some companies such as Adesto and Crossbar are shipping or have announcedPMCm-type ReRAM products. The benefits of these devices are similar to those ofPCM. But, where PCM is temperature sensitive, requiring special solder flowprocesses to attach a preprogrammed PCM chip to a PC board, PMCm is nottemperature sensitive and can withstand the rigors of a standard soldering process.HP introduced a version of oxygen-depletion ReRAM several years ago with itsmemristor, but this has not resulted in actual products.Choosing Storage TechnologiesHere, we will first consider why people use different storage and memory technologies.Our initial approach is to look at the tradeoffs between bandwidth (data rate) and cost,and the second is a more in-depth look at demands for different applications and howthe accessibility to data and responsiveness to commands make different storagetechnologies attractive for these applications, either singly or in combination. Next, weprovide some projections for shipping capacity of NAND flash, HDDs, and magnetictape out to 2022.In general, the higher-performance memory or storage is also the most expensive ona dollar per terabyte basis, as shown in Figure 7.13

Figure 7. Memory-storage hierarchy versus performance. (Objective Analysis,2017.)1071.E 0761.E 0610L1L251.E 05Bandwidth (MB/s)10L31041.E 04DRAM1031.E 03NAND21.E 02101011.E 0101.E 00-11.E-011010HDDTape01.E 001011.E 011021.E 021031.E 031041.E 041051.E 051061.E 0610Price per GigabyteIn computer systems, the various memory cache technologies (L1, L2, and L3) are thefastest in terms of performance, followed by DRAM. All of these technologies areeffectively volatile memories that lose their data when power is removed. NAND flashis nonvolatile memory and provides performance and cost per terabyte that liebetween DRAM and HDDs. Below HDDs are technologies like tape (or optical discs)that have slower performance (primarily due to mechanical latencies) but also havethe lowest overall storage costs in dollars per terabyte. Note that most of the emergingsolid-state storage technologies are aiming to replace either NAND flash, DRAM, or,possibly, the various processor cache memories.We will now take another look at how to decide which memory or storage technologyis best for which applications using what is called the touch rate and the response timeof the storage technologies for moving stored objects of various sizes.We define a scale-independent method for classifying performance of storagesystems for differing workloads (see Hetzler and Coughlin, 2015). The goal is to finda metric that could be used as a rule of thumb as well as for detailed analysis. Thisanalysis is designed to be easy to understand and useful for comparing differentsystem designs. Furthermore, this metric should reflect the behavior of the system asit is being used, not when it is idle (unless that is how it will be used). As notedpreviously, we call this new metric touch rate.Response TimeBecause we want touch rate to be a measure for a busy system operating at load, weneed first to define what is meant by being “at load.” We define the load as back-toback input/output (I/O) operations, which represent 100% utilization without queuing.We leave queuing aside, as it makes the analysis simpler.The response time is the time to complete an I/O operation, including the transfer ofdata and restoring the system for a subsequent I/O operation. The response time is,14

therefore, a function of the I/O object size as well as the speed of ancillary supportoperations. This is distinct from the access time, which is the time to the first byte ofdata after a request to an idle system. The response time is thus a measure of howrapidly an object can be retrieved under operating conditions.Touch RateTouch rate is defined as the portion of the total system capacity that can be accessedin a given interval of time. Using a shopkeeper’s metric, it can be thought of as thenumber of inventory turns on the data set. This analogy leads us to look at touch rateas a measure of the value that can be extracted from the data set. We need to pick atime interval for measuring the touch rate that is appropriate to an application—forinstance, a year is a suitable period for archival data.Equation (1) gives a definition of the touch rate over a 𝑩) 𝟑𝟏.𝟓𝟑𝟔𝑇𝑜𝑢𝑐ℎ 𝑅𝑎𝑡𝑒 ���) 𝑪𝒂𝒑𝒂𝒄𝒊𝒕𝒚(𝑻𝑩) .(1)There are 31,536,000 s in a year; after the terms are cancelled for the object and totalcapacity size, we are left with the factor of 31.536. Equation (1) assumes that theobject size is in megabytes and the system or unit capacity is in terabytes. Responsetime is the steady-state response time in seconds (as described previously) for thisobject size undergoing back-to-back 100% random I/Os.Note that a touch rate greater than one doesn’t necessarily mean that the same dataare accessed repeatedly, although they may. It can also mean that new data arecoming in, which is also counted in the touch rate. What matters here is the amount ofdata accessed. The touch rate is thus a measure of how much of the data in a systemcan be accessed during a given interval of time.Touch Rate Versus Response TimeWe can learn a great deal about the behavior of a system by plotting the touch rateversus the response time. The response time measures the time to act on a singleobject, while the touch rate relates to the time to act on the data set as a whole. Figure8 shows such a chart that includes some indications of various sorts of applications(performance regions) and general tradeoffs important to the system design indicated.Note that this is a log-log chart with log base-10 scales on both the vertical andhorizontal axes.Figure 8 shows touch rate as log touch per year on the vertical axis and log responsetime on the horizontal axis, with faster response times on the left. A shorter responsetime means data can be accessed more quickly, increasing the data’s value. Highertouch rate means more data can be processed in a given time period, incre

hybrid flash/disk drives, various resistive memory storage, spin-torque MRAM, and other magnetic spin-based memories and optical holographic-based storage. Hard-Disk Drives Magnetic-recording-based HDD technology continues to play a large and important role in mass data storage.