Programming Persistent Memory Steve Scargall download
Programming Persistent Memory Steve Scargall download
download
https://ebookbell.com/product/programming-persistent-memory-
steve-scargall-59042598
https://ebookbell.com/product/programming-persistent-memory-a-
comprehensive-guide-for-developers-1st-edition-steve-scargall-34689828
https://ebookbell.com/product/vertically-integrated-architectures-
versioned-data-models-implicit-services-and-persistenceaware-
programming-1st-edition-jos-jong-7293810
https://ebookbell.com/product/programming-101-learn-to-code-using-the-
processing-programming-language-2nd-edition-2nd-jeanine-meyer-46238180
Programming 101 The How And Why Of Programming Revealed Using The
Processing Programming Language Jeanine Meyer
https://ebookbell.com/product/programming-101-the-how-and-why-of-
programming-revealed-using-the-processing-programming-language-
jeanine-meyer-46318424
https://ebookbell.com/product/programming-and-gui-fundamentals-tcltk-
for-electronic-design-automation-suman-lata-tripathi-46318712
https://ebookbell.com/product/programming-with-openscad-a-beginners-
guide-to-coding-3dprintable-objects-1st-edition-justin-gohde-46410140
https://ebookbell.com/product/programming-in-two-semesters-using-
python-and-java-quentin-charatan-46494972
https://ebookbell.com/product/programming-the-future-politics-
resistance-and-utopia-in-contemporary-speculative-tv-sherryl-vint-
jonathan-alexander-46771488
Programming
Persistent
Memory
A Comprehensive Guide for Developers
—
Steve Scargall
Programming Persistent
Memory
A Comprehensive Guide for
Developers
Steve Scargall
Programming Persistent Memory: A Comprehensive Guide for Developers
Steve Scargall
Santa Clara, CA, USA
iii
Table of Contents
iv
Table of Contents
Persistent Libraries��������������������������������������������������������������������������������������������������������������������� 67
libpmem�������������������������������������������������������������������������������������������������������������������������������� 67
libpmemobj���������������������������������������������������������������������������������������������������������������������������� 68
libpmemobj-cpp�������������������������������������������������������������������������������������������������������������������� 68
libpmemkv����������������������������������������������������������������������������������������������������������������������������� 69
libpmemlog���������������������������������������������������������������������������������������������������������������������������� 69
libpmemblk���������������������������������������������������������������������������������������������������������������������������� 69
Tools and Command Utilities������������������������������������������������������������������������������������������������������� 70
pmempool������������������������������������������������������������������������������������������������������������������������������ 70
pmemcheck��������������������������������������������������������������������������������������������������������������������������� 70
pmreorder������������������������������������������������������������������������������������������������������������������������������ 71
Summary������������������������������������������������������������������������������������������������������������������������������������ 71
v
Table of Contents
vi
Table of Contents
vii
Table of Contents
viii
Table of Contents
ix
Table of Contents
Chapter 16: PMDK Internals: Important Algorithms and Data Structures������������ 313
A Pool of Persistent Memory: High-Level Architecture Overview��������������������������������������������� 313
The Uncertainty of Memory Mapping: Persistent Memory Object Identifier����������������������������� 315
Persistent Thread Local Storage: Using Lanes�������������������������������������������������������������������������� 318
Ensuring Power-Fail Atomicity: Redo and Undo Logging���������������������������������������������������������� 320
Transaction Redo Logging��������������������������������������������������������������������������������������������������� 320
Transaction Undo Logging��������������������������������������������������������������������������������������������������� 321
libpmemobj Unified Logging������������������������������������������������������������������������������������������������ 322
Persistent Allocations: The Interface of a Transactional Persistent Allocator���������������������������� 323
Persistent Memory Heap Management: Allocator Design for Persistent Memory�������������������� 324
ACID Transactions: Efficient Low-Level Persistent Transactions����������������������������������������������� 328
Lazy Reinitialization of Variables: Storing the Volatile State on Persistent Memory����������������� 330
Summary���������������������������������������������������������������������������������������������������������������������������������� 331
x
Table of Contents
xi
Table of Contents
Appendix B: How to Install the Persistent Memory Development Kit (PMDK)������ 395
PMDK Prerequisites������������������������������������������������������������������������������������������������������������������ 395
Installing PMDK Using the Linux Distribution Package Repository������������������������������������������� 395
Package Naming Convention����������������������������������������������������������������������������������������������� 396
Searching for Packages Within a Package Repository�������������������������������������������������������� 396
Installing PMDK Libraries from the Package Repository����������������������������������������������������� 398
Installing PMDK on Microsoft Windows������������������������������������������������������������������������������������ 402
Glossary���������������������������������������������������������������������������������������������������������������� 425
Index��������������������������������������������������������������������������������������������������������������������� 429
xii
About the Author
Steve Scargall is a persistent memory software and cloud architect at Intel
Corporation. As a technology evangelist, he supports the enabling and development
effort to integrate persistent memory technology into software stacks, applications,
and hardware architectures. This includes working with independent software
vendors (ISVs) on both proprietary and open source development, original equipment
manufacturers (OEMs), and cloud service providers (CSPs).
Steve holds a Bachelor of Science in computer science and cybernetics from the
University of Reading, UK, where he studied neural networks, AI, and robotics. He
has over 19 years’ experience providing performance analysis on x86 architecture and
SPARC for Solaris Kernel, ZFS, and UFS file system. He performed DTrace debugging in
enterprise and cloud environments during his tenures at Sun Microsystems and Oracle.
xiii
About the Technical Reviewer
Andy Rudoff is a principal engineer at Intel Corporation, focusing on non-volatile
memory programming. He is a contributor to the SNIA NVM Programming Technical
Work Group. His more than 30 years’ industry experience includes design and
development work in operating systems, file systems, networking, and fault management
at companies large and small, including Sun Microsystems and VMware. Andy has
taught various operating systems classes over the years and is a coauthor of the popular
UNIX Network Programming textbook.
xv
About the Contributors
Piotr Balcer is a software engineer at Intel Corporation with many years’ experience
working on storage-related technologies. He holds a Bachelor of Science in engineering
from the Gdańsk University of Technology, Poland, where he studied system software
engineering. Piotr has been working on the software ecosystem for next-generation
persistent memory since 2014.
Eduardo Berrocal joined Intel Corporation as a cloud software engineer in 2017 after
receiving his PhD in computer science from the Illinois Institute of Technology. His
doctoral research focused on data analytics and fault tolerance for high-performance
computing. Past experience includes working as an intern at Bell Labs (Nokia), a research
aid at Argonne National Laboratory, a scientific programmer and web developer at the
University of Chicago, and an intern in the CESVIMA laboratory in Spain.
xvii
About the Contributors
Chet Douglas is a principal software engineer at Intel Corporation and focuses on cloud
software architecture along with operating system and OEM enabling of non-volatile
memory technologies. He has over 14 years’ experience working on various enterprise
and client programs and 28 years of total storage experience. Chet has worked in all
aspects of storage, including storage controller hardware design, SCSI disk/tape/CD
writer firmware architecture, storage management software architecture, Microsoft
Windows* and Linux kernel-mode drivers, enterprise hardware RAID, and client/
workstation software RAID. He holds seven storage-related hardware and software
patents and has a dual Bachelor of Science in electrical engineering and computer
engineering from Clarkson University, New York.
Ken Gibson is the director of persistent memory software architecture within Intel
Corporation’s Data Center Group. Since 2012, Ken and his team have been working with
Intel’s server and software partners to create the open persistent memory programming
model.
Kishor Kharbas is a software engineer on the Java runtime engineering team at Intel
Corporation. For the past eight years, he has been working to optimize Oracle’s OpenJDK
on Intel platforms. This involves Java garbage collection and compiler back-end
optimization.
xviii
About the Contributors
Nicholas Moulin is a cloud software architect at Intel Corporation. Since joining Intel
in 2012, he has focused on enabling and developing persistent memory software for
operating systems and platform firmware and managing persistent memory hardware.
Nicholas is currently working with industry partners to define and improve RAS features
relevant to the persistent memory programming model.
Vineet Singh is a memory and storage tools software engineer at Intel Corporation.
He develops techniques to help developers adapt to the latest memory technologies.
Vineet holds a PhD in philosophy from the University of California and has a Bachelor
of Technology degree from the Indian Institute of Information Technology, Design, and
Manufacturing in Jabalpur.
xix
About the Contributors
Usha Upadhyayula has been with Intel Corporation for 20 years serving in many
different roles. Usha holds a master’s degree in computer science from the University
of South Carolina, and she spent the first few years at Intel developing user-level
applications in C and C++. She later moved to customer-enabling roles for Intel media
processors and support for Intel RAID software. Usha is currently part of the Data Center
Group where she is focused on enabling cloud service providers to fully utilize and
accelerate the adoption of Intel persistent memory products.
xx
Acknowledgments
First and foremost, I would like to thank Ken Gibson for masterminding this book idea
and for gifting me the pleasure of writing and managing it. Your support, guidance, and
contributions have been instrumental in delivering a high-quality product.
If the Vulcan mind-meld or The Matrix Headjack were possible, I could have cloned
Andy Rudoff’s mind and allowed him to work on his daily activities. Instead, Andy’s
infinite knowledge of persistent memory had to be tapped through good old verbal
communication and e-mail. I sincerely thank you for devoting so much time to me and
this project. The results read for themselves.
Debbie Graham was instrumental in helping me manage this colossal project. Her
dedication and support helped drive the project to an on-time completion.
To my friends and colleagues at Intel who contributed content, supported
discussions, helped with decision-making, and reviewed drafts during the book-writing
process. These are the real heroes. Without your heavily invested time and support, this
book would have taken considerably longer to complete. It is a much better product as a
result of the collaborative effort. A huge thanks to all of you.
I'd like to express my sincerest gratitude and appreciation to the people at Apress,
without whom this book could not have been published. From the initial contact and
outline discussions through the entire publishing process to this final polished product,
the Apress team delivered continuous support and assistance. Many thanks to Susan,
Jessica, and Rita. It was a real pleasure working with you.
xxi
Preface
About This Book
Persistent memory is often referred to as non-volatile memory (NVM) or storage
class memory (SCM). In this book, we purposefully use persistent memory as an all-
encompassing term to represent all the current and future memory technologies that
fall under this umbrella. This book introduces the persistent memory technology and
provides answers to key questions. For software developers, those questions include:
What is persistent memory? How do I use it? What APIs and libraries are available?
What benefits can it provide for my application? What new programming methods do I
need to learn? How do I design applications to use persistent memory? Where can I find
information, documentation, and help?
System and cloud architects will be provided with answers to questions such as:
What is persistent memory? How does it work? How is it different than DRAM or SSD/
NVMe storage devices? What are the hardware and operating system requirements?
What applications need or could benefit from persistent memory? Can my existing
applications use persistent memory without being modified?
Persistent memory is not a plug-and-play technology for software applications.
Although it may look and feel like traditional DRAM memory, applications need to be
modified to fully utilize the persistence feature of persistent memory. That is not to say
that applications cannot run unmodified on systems with persistent memory installed,
they can, but they will not see the full potential of what persistent memory offers without
code modification.
Thankfully, server and operating system vendors collaborated very early in the
design phase and already have products available on the market. Linux and Microsoft
Windows already provide native support for persistent memory technologies. Many
popular virtualization technologies also support persistent memory.
For ISVs and the developer community at large, the journey is just beginning. Some
software has already been modified and is available on the market. However, it will
take time for the enterprise and cloud computing industries to adopt and make the
hardware available to the general marketplace. ISVs and software developers need time
to understand what changes to existing applications are required and implement them.
xxiii
Preface
To make the required development work easier, Intel developed and open sourced
the Persistent Memory Development Kit (PMDK) available from https://pmem.io/
pmdk/. We introduce the PMDK in more detail in Chapter 5 and walk through most of
the available libraries in subsequent chapters. Each chapter provides an in-depth guide
so developers can understand what library or libraries to use. PMDK is a set of open
source libraries and tools based on the Storage Networking Industry Association (SNIA)
NVM programming model designed and implemented by over 50 industry partners. The
latest NVM programming model document can be found at https://www.snia.org/
tech_activities/standards/curr_standards/npm. The model describes how software
can utilize persistent memory features and enables designers to develop APIs that take
advantage of NVM features and performance.
Available for both Linux and Windows, PMDK facilitates persistent memory
programming adoption with higher-level language support. C and C++ support is fully
validated. Support for other languages such as Java and Python is work in progress
at the time this book was written. Other languages are expected to also adopt the
programming model and provide native persistent memory APIs for developers. The
PMDK development team welcomes and encourages new contributions to core code,
new language bindings, or new storage engines for the persistent memory key-value
store called pmemkv.
This book assumes no prior knowledge of persistent memory hardware devices
or software development. The book layout allows you to freely navigate the content in
the order you want. It is not required to read all chapters in order, though we do build
upon concepts and knowledge described in previous chapters. In such cases, we make
backward and forward references to relevant chapters and sections so you can learn or
refresh your memory.
B
ook Structure
This book has 19 chapters, each one focusing on a different topic. The book has three
main sections. Chapters 1-4 provide an introduction to persistent memory architecture,
hardware, and operating system support. Chapters 5-16 allow developers to understand
the PMDK libraries and how to use them in applications. Finally, Chapters 17-19 provide
information on advanced topics such as RAS and replication of data using RDMA.
xxiv
Preface
xxv
Preface
xxvi
Preface
The Appendixes have separate procedures for installing the PMDK and utilities
required for managing persistent memory. We also included an update for Java and the
future of the RDMA protocols. All of this content is considered temporal, so we did not
want to include it in the main body of the book.
Intended Audience
This book has been written for experienced application developers in mind. We
intend the content to be useful to a wider readership such as system administrators
and architects, students, lecturers, and academic research fellows to name but a few.
System designers, kernel developers, and anyone with a vested or passing interest in this
emerging technology will find something useful within this book.
Every reader will learn what persistent memory is, how it works, and how operating
systems and applications can utilize it. Provisioning and managing persistent memory
are vendor specific, so we include some resources in the Appendix sections to avoid
overcomplicating the main chapter content.
Application developers will learn, by example, how to integrate persistent memory
in to existing or new applications. We use examples extensively throughout this book
using a variety of libraries available within the Persistent Memory Development Kit
(PMDK). Example code is provided in a variety of programming languages such as C,
C++, JavaScript, and others. We want developers to feel comfortable using these libraries
in their own projects. The book provides extensive links to resources where you can find
help and information.
System administrators and architects of Cloud, high-performance computing,
and enterprise environments can use most of the content of this book to
understand persistent memory features and benefits to support applications and
developers. Imagine being able to deploy more virtual machines per physical server or
provide applications with this new memory/storage tier such that they can keep more
data closer to the CPU or restart in a fraction of the time they could before while keeping
a warm cache of data.
Students, lecturers, and academic research fellows will also benefit from many
chapters within this book. Computer science classes can learn about the hardware,
operating system features, and programming techniques. Lecturers are free use the
content in student classes or to form the basis of research projects such as new persistent
memory file systems, algorithms, or caching implementations.
xxvii
Preface
We introduce tools that profile the server and applications to better understand CPU,
memory, and disk IO access patterns. Using this knowledge, we show how applications
can be modified to take full advantage of persistence using the Persistent Memory
Development Kit (PMDK).
A Future Reference
The book content has been written to provide value for many years. Industry
specification such as ACPI, UEFI, and the SNIA non-volatile programming model will,
unless otherwise stated by the specification, remain backward compatible as new
versions are released. If new form factors are introduced, the approach to programming
remains the same. We do not limit ourselves to one specific persistent memory vendor
or implementation. In places where it is necessary to describe vendor-specific features
or implementations, we specifically call this out as it may change between vendors or
between product generations. We encourage you to read the vendor documentation for
the persistent memory product to learn more.
Developers using the Persistent Memory Development Kit (PMDK) will retain a stable
API interface. PMDK will deliver new features and performance improvements with each
major release. It will evolve with new persistent memory products, CPU instructions,
platform designs, industry specifications, and operating system feature support.
xxviii
Preface
The code examples provided with this book have been tested and validated using
Intel Optane DC persistent memory. Since the PMDK is vendor neutral, they will also
work on NVDIMM-N devices. PMDK will support any future persistent memory product
that enters the market.
The code examples used throughout this book are current at the time of
publication. All code examples have been validated and tested to ensure they compile
and execute without error. For brevity, some of the examples in this book use assert()
statements to indicate unexpected errors. Any production code would likely replace
these with the appropriate error handling actions which would include friendlier
error messages and appropriate error recovery actions. Additionally, some of the code
examples use different mount points to represent persistent memory aware file systems,
for example “/daxfs”, “/pmemfs”, and “/mnt/pmemfs”. This demonstrates persistent
memory file systems can be mounted and named appropriately for the application, just
like regular block-based file systems. Source code is from the repository that accompanies
this book – https://github.com/Apress/programming-persistent-memory.
Since this is a rapidly evolving technology, the software and APIs references
throughout this book may change over time. While every effort is made to be backward
compatible, sometimes software must evolve and invalidate previous versions. For this
reason, it is therefore expected that some of the code samples may not compile on newer
hardware or operating systems and may need to be changed accordingly.
B
ook Conventions
This book uses several conventions to draw your attention to specific pieces of
information. The convention used depends on the type of information displayed.
C
omputer Commands
Commands, programming library, and API function references may be presented in line
with the paragraph text using a monospaced font. For example:
To illustrate how persistent memory is used, let’s start with a sample program
demonstrating the key-value store provided by a library called libpmemkv.
xxix
Preface
S
ource Code
Source code examples taken from the accompanying GitHub repository are shown with
relevant line numbers in a monospaced font. Below each code listing is a reference to
the line number or line number range with a brief description. Code comments use
language native styling. Most languages use the same syntax. Single line comments
will use // and block/multiline comments should use /*..*/. An example is shown in
Listing 1.
xxx
Preface
• Line 45: Here we define a small helper routine, kvprint(), which prints
a key-value pair when called.
Notes
We use a standard format for notes, cautions, and tips when we want to direct your
attention to an important point, for example.
xxxi
CHAPTER 1
Introduction to Persistent
Memory Programming
This book describes programming techniques for writing applications that use persistent
memory. It is written for experienced software developers, but we assume no previous
experience using persistent memory. We provide many code examples in a variety of
programming languages. Most programmers will understand these examples, even if
they have not previously used the specific language.
1
© The Author(s) 2020
S. Scargall, Programming Persistent Memory, https://doi.org/10.1007/978-1-4842-4932-1_1
Chapter 1 Introduction to Persistent Memory Programming
37 #include <iostream>
38 #include <cassert>
39 #include <libpmemkv.hpp>
40
41 using namespace pmem::kv;
42 using std::cerr;
43 using std::cout;
44 using std::endl;
45 using std::string;
46
47 /*
48 * for this example, create a 1 Gig file
49 * called "/daxfs/kvfile"
50 */
51 auto PATH = "/daxfs/kvfile";
52 const uint64_t SIZE = 1024 * 1024 * 1024;
53
54 /*
55 * kvprint -- print a single key-value pair
56 */
57 int kvprint(string_view k, string_view v) {
58 cout << "key: " << k.data() <<
59 " value: " << v.data() << endl;
60 return 0;
61 }
62
2
Chapter 1 Introduction to Persistent Memory Programming
63 int main() {
64 // start by creating the db object
65 db *kv = new db();
66 assert(kv != nullptr);
67
68 // create the config information for
69 // libpmemkv's open method
70 config cfg;
71
72 if (cfg.put_string("path", PATH) != status::OK) {
73 cerr << pmemkv_errormsg() << endl;
74 exit(1);
75 }
76 if (cfg.put_uint64("force_create", 1) != status::OK) {
77 cerr << pmemkv_errormsg() << endl;
78 exit(1);
79 }
80 if (cfg.put_uint64("size", SIZE) != status::OK) {
81 cerr << pmemkv_errormsg() << endl;
82 exit(1);
83 }
84
85
86 // open the key-value store, using the cmap engine
87 if (kv->open("cmap", std::move(cfg)) != status::OK) {
88 cerr << db::errormsg() << endl;
89 exit(1);
90 }
91
92 // add some keys and values
93 if (kv->put("key1", "value1") != status::OK) {
94 cerr << db::errormsg() << endl;
95 exit(1);
96 }
3
Chapter 1 Introduction to Persistent Memory Programming
W
hat’s Different?
A wide variety of key-value libraries are available in practically every programming
language. The persistent memory example in Listing 1-1 is different because the key-
value store itself resides in persistent memory. For comparison, Figure 1-1 shows how a
key-value store using traditional storage is laid out.
When the application in Figure 1-1 wants to fetch a value from the key-value store,
a buffer must be allocated in memory to hold the result. This is because the values are
kept on block storage, which cannot be addressed directly by the application. The only
way to access a value is to bring it into memory, and the only way to do that is to read
full blocks from the storage device, which can only be accessed via block I/O. Now
consider Figure 1-2, where the key-value store resides in persistent memory like our
sample code.
5
Chapter 1 Introduction to Persistent Memory Programming
With the persistent memory key-value store, values are accessed by the application
directly, without the need to first allocate buffers in memory. The kvprint() routine in
Listing 1-1 will be called with references to the actual keys and values, directly where
they live in persistence – something that is not possible with traditional storage. In
fact, even the data structures used by the key-value store library to organize its data are
accessed directly. When a storage-based key-value store library needs to make a small
update, for example, 64 bytes, it must read the block of storage containing those 64 bytes
into a memory buffer, update the 64 bytes, and then write out the entire block to make it
persistent. That is because storage accesses can only happen using block I/O, typically
4K bytes at a time, so the task to update 64 bytes requires reading 4K and then writing
4K. But with persistent memory, the same example of changing 64 bytes would only
write the 64 bytes directly to persistence.
6
Chapter 1 Introduction to Persistent Memory Programming
Program Complexity
Perhaps the most important point of our example is that the programmer still uses
the familiar get/put interfaces normally associated with key-value stores. The fact that
the data structures are in persistent memory is abstracted away by the high-level API
provided by libpmemkv. This principle of using the highest level of abstraction possible,
as long as it meets the application’s needs, will be a recurring theme throughout this
book. We start by introducing very high-level APIs; later chapters delve into the lower-
level details for programmers who need them. At the lowest level, programming directly
to raw persistent memory requires detailed knowledge of things like hardware atomicity,
cache flushing, and transactions. High-level libraries like libpmemkv abstract away all
that complexity and provide much simpler, less error-prone interfaces.
7
Chapter 1 Introduction to Persistent Memory Programming
Starting from the bottom of Figure 1-4 and working upward are these components:
8
Chapter 1 Introduction to Persistent Memory Programming
• And finally, the application that uses the API provided by libpmemkv.
Although there is quite a stack of components in use here, it does not mean there
is necessarily a large amount of code that runs for each operation. Some components
are only used during the initial setup. For example, the pmem-aware file system is
used to find the persistent memory file and perform permission checks; it is out of the
application’s data path after that. The PMDK libraries are designed to leverage the direct
access allowed by persistent memory as much as possible.
W
hat’s Next?
Chapters 1 through 3 provide the essential background that programmers need to know to
start persistent memory programming. The stage is now set with a simple example; the next
two chapters provide details about persistent memory at the hardware and operating system
levels. The later and more advanced chapters provide much more detail for those interested.
Because the immediate goal is to get you programming quickly, we recommend
reading Chapters 2 and 3 to gain the essential background and then dive into Chapter 4
where we start to show more detailed persistent memory programming examples.
S
ummary
This chapter shows how high-level APIs like libpmemkv can be used for persistent
memory programming, hiding complex details of persistent memory from the
application developer. Using persistent memory can allow finer-grained access and
higher performance than block-based storage. We recommend using the highest-level,
simplest APIs possible and only introducing the complexity of lower-level persistent
memory programming as necessary.
9
Chapter 1 Introduction to Persistent Memory Programming
Open Access This chapter is licensed under the terms of the Creative
Commons Attribution 4.0 International License (http://creativecommons.
org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and
reproduction in any medium or format, as long as you give appropriate credit to the
original author(s) and the source, provide a link to the Creative Commons license and
indicate if changes were made.
The images or other third party material in this chapter are included in the chapter’s
Creative Commons license, unless indicated otherwise in a credit line to the material. If
material is not included in the chapter’s Creative Commons license and your intended
use is not permitted by statutory regulation or exceeds the permitted use, you will need
to obtain permission directly from the copyright holder.
10
CHAPTER 2
Persistent Memory
Architecture
This chapter provides an overview of the persistent memory architecture while focusing
on the hardware to emphasize requirements and decisions that developers need to know.
Applications that are designed to recognize the presence of persistent memory in
a system can run much faster than using other storage devices because data does not
have to transfer back and forth between the CPU and slower storage devices. Because
applications that only use persistent memory may be slower than dynamic random-
access memory (DRAM), they should decide what data resides in DRAM, persistent
memory, and storage.
The capacity of persistent memory is expected to be many times larger than DRAM;
thus, the volume of data that applications can potentially store and process in place is
also much larger. This significantly reduces the number of disk I/Os, which improves
performance and reduces wear on the storage media.
On systems without persistent memory, large datasets that cannot fit into DRAM
must be processed in segments or streamed. This introduces processing delays as the
application stalls waiting for data to be paged from disk or streamed from the network.
If the working dataset size fits within the capacity of persistent memory and DRAM,
applications can perform in-memory processing without needing to checkpoint or page
data to or from storage. This significantly improves performance.
11
© The Author(s) 2020
S. Scargall, Programming Persistent Memory, https://doi.org/10.1007/978-1-4842-4932-1_2
Chapter 2 Persistent Memory Architecture
12
Chapter 2 Persistent Memory Architecture
13
Chapter 2 Persistent Memory Architecture
C
ache Hierarchy
We use load and store operations to read and write to persistent memory rather than
using block-based I/O to read and write to traditional storage. We suggest reading the
CPU architecture documentation for an in-depth description because each successive
CPU generation may introduce new features, methods, and optimizations.
Using the Intel architecture as an example, a CPU cache typically has three
distinct levels: L1, L2, and L3. The hierarchy makes references to the distance
from the CPU core, its speed, and size of the cache. The L1 cache is closest to
the CPU. It is extremely fast but very small. L2 and L3 caches are increasingly
larger in capacity, but they are relatively slower. Figure 2-1 shows a typical CPU
microarchitecture with three levels of CPU cache and a memory controller with
three memory channels. Each memory channel has a single DRAM and persistent
memory attached. On platforms where the CPU caches are not contained within
the power-fail protected domain, any modified data within the CPU caches that has
not been flushed to persistent memory will be lost when the system loses power or
crashes. Platforms that do include CPU caches in the power-fail protected domain
will ensure modified data within the CPU caches are flushed to the persistent
memory should the system crash or loses power. We describe these requirements
and features in the upcoming “Power-Fail Protected Domains” section.
14
Chapter 2 Persistent Memory Architecture
The L1 (Level 1) cache is the fastest memory in a computer system. In terms of access
priority, the L1 cache has the data the CPU is most likely to need while completing a
specific task. The L1 cache is also usually split two ways, into the instruction cache (L1 I)
and the data cache (L1 D). The instruction cache deals with the information about the
operation that the CPU has to perform, while the data cache holds the data on which the
operation is to be performed.
The L2 (Level 2) cache has a larger capacity than the L1 cache, but it is slower. L2
cache holds data that is likely to be accessed by the CPU next. In most modern CPUs,
the L1 and L2 caches are present on the CPU cores themselves, with each core getting
dedicated caches.
The L3 (Level 3) cache is the largest cache memory, but it is also the slowest of the
three. It is also a commonly shared resource among all the cores on the CPU and may be
internally partitioned to allow each core to have dedicated L3 resources.
Data read from DRAM or persistent memory is transferred through the memory
controller into the L3 cache, then propagated into the L2 cache, and finally the L1 cache
where the CPU core consumes it. When the processor is looking for data to carry out an
operation, it first tries to find it into the L1 cache. If the CPU can find it, the condition is
called a cache hit. If the CPU cannot find the data within the L1 cache, it then proceeds to
15
Chapter 2 Persistent Memory Architecture
search for it first within L2, then L3. If it cannot find the data in any of the three, it tries to
access it from memory. Each failure to find data in a cache is called a cache miss. Failure
to locate the data in memory requires the operating system to page the data into memory
from a storage device.
When the CPU writes data, it is initially written to the L1 cache. Due to ongoing
activity within the CPU, at some point in time, the data will be evicted from the L1 cache
into the L2 cache. The data may be further evicted from L2 and placed into L3 and
eventually evicted from L3 into the memory controller’s write buffers where it is then
written to the memory device.
In a system that does not possess persistent memory, software persists data by writing it
to a non-volatile storage device such as an SSD, HDD, SAN, NAS, or a volume in the cloud.
This protects data from application or system crashes. Critical data can be manually flushed
using calls such as msync(), fsync(), or fdatasync(), which flush uncommitted dirty
pages from volatile memory to the non-volatile storage device. File systems provide fdisk
or chkdsk utilities to check and attempt repairs on damaged file systems if required. File
systems do not protect user data from torn blocks. Applications have a responsibility to
detect and recovery from this situation. That’s why databases, for example, use a variety of
techniques such as transactional updates, redo/undo logging, and checksums.
Applications memory map the persistent memory address range directly into its
own memory address space. Therefore, the application must assume responsibility
for checking and guaranteeing data integrity. The rest of this chapter describes
your responsibilities in a persistent memory environment and how to achieve data
consistency and integrity.
16
Chapter 2 Persistent Memory Architecture
from the power-fail protected domain using stored energy guaranteed by the platform for
this purpose. Data that has not yet made it into the protected domain will be lost.
Multiple persistence domains may exist within the same system, for example, on
systems with more than one physical CPU. Systems may also provide a mechanism for
partitioning the platform resources for isolation. This must be done in such a way that
SNIA NVM programming model behavior is assured from each compliant volume or file
system. (Chapter 3 describes the programming model as it applies to operating systems
and file systems. The “Detecting Platform Capabilities” section in that chapter describes
the logic that applications should perform to detect platform capabilities including
power failure protected domains. Later chapters provide in-depth discussions into why,
how, and when applications should flush data, if required, to guarantee the data is safe
within the protected domain and persistent memory.)
Volatile memory loses its contents when the computer system’s power is interrupted.
Just like non-volatile storage devices, persistent memory keeps its contents even in the
absence of system power. Data that has been physically saved to the persistent memory
media is called data at rest. Data in-flight has the following meanings:
• Writes sent to the persistent memory device but have not yet been
physically committed to the media
• Data that has been temporarily buffered or cached in either the CPU
caches or memory controller
When a system is gracefully rebooted or shut down, the system maintains power
and can ensure all contents of the CPU caches and memory controllers are flushed such
that any in-flight or uncommitted data is successfully written to persistent memory
or non-volatile storage. When an unexpected power failure occurs, and assuming no
uninterruptable power supply (UPS) is available, the system must have enough stored
energy within the power supplies and capacitors dotted around it to flush data before the
power is completely exhausted. Any data that is not flushed is lost and not recoverable.
Asynchronous DRAM Refresh (ADR) is a feature supported on Intel products which
flushes the write-protected data buffers and places the DRAM in self-refresh. This
process is critical during a power loss event or system crash to ensure the data is in a safe
and consistent state on persistent memory. By default, ADR does not flush the processor
caches. A platform that supports ADR only includes persistent memory and the memory
controller’s write pending queues within the persistence domain. This is the reason
17
Chapter 2 Persistent Memory Architecture
data in the CPU caches must be flushed by the application using the CLWB, CLFLUSHOPT,
CLFLUSH, non-temporal stores, or WBINVD machine instructions.
Enhanced Asynchronous DRAM Refresh (eADR) requires that a non-maskable
interrupt (NMI) routine be called to flush the CPU caches before the ADR event can begin.
Applications running on an eADR platform do not need to perform flush operations
because the hardware should flush the data automatically, but they are still required
to perform an SFENCE operation to maintain write order correctness. Stores should be
considered persistent only when they are globally visible, which the SFENCE guarantees.
Figure 2-2 shows both the ADR and eADR persistence domains.
18
Chapter 2 Persistent Memory Architecture
which introduces additional maintenance routines that reduce server uptime. There
is also an environmental impact when using batteries as they must be disposed of
or recycled correctly. It is entirely possible for server or appliance OEMs to include a
battery in their product.
Because some appliance and server vendors plan to use batteries, and because
platforms will someday include the CPU caches in the persistence domain, a property is
available within ACPI such that the BIOS can notify software when the CPU flushes can
be skipped. On platforms with eADR, there is no need for manual cache line flushing.
19
Chapter 2 Persistent Memory Architecture
2. Update the node pointer (next pointer) to point to the last node in
the list (Node 2 → Node 1).
3. Update the head pointer to point at the new node (Head → Node 2).
Figure 2-3 (Step 3) shows that the head pointer was updated in the CPU cached version,
but the Node 2 to Node 1 pointer has not yet been updated in persistent memory. This
is because the hardware can choose which cache lines to commit and the order may not
match the source code flow. If the system or application were to crash at this point, the
persistent memory state would be inconsistent, and the data structure would no longer
be usable.
1
NIA NVM programming model spec: https://www.snia.org/tech_activities/standards/
S
curr_standards/npm
20
Chapter 2 Persistent Memory Architecture
Figure 2-3. Adding a new node to an existing linked list without a store barrier
To solve this problem, we introduce a memory store barrier to ensure the order of the
write operations is maintained. Starting from the same initial state, the pseudocode now
looks like this:
2. Update the node pointer (next pointer) to point to the last node in
the list, and perform a store barrier/fence operation.
3. Update the head pointer to point at the new node.
Figure 2-4 shows that the addition of the store barrier allows the code to work as
expected and maintains a consistent data structure in the volatile CPU caches and on
21
Random documents with unrelated
content Scribd suggests to you:
PREMIERE PARTIE.
De la formation des fluides aériformes & de leur
décomposition; de la combustion des corps
simples & de la formation des acides.
CHAPITRE PREMIER.
ebookbell.com