0% found this document useful (0 votes)
75 views

Programming Persistent Memory A Comprehensive Guide For Developers 1st Edition Steve Scargall pdf download

The document is a comprehensive guide on programming persistent memory, authored by Steve Scargall, covering various aspects such as architecture, operating system support, and development kits. It includes detailed chapters on fundamental concepts, libraries, and tools for developers to effectively utilize persistent memory in their applications. The guide is published under a Creative Commons license, allowing for sharing and adaptation with proper credit to the author.

Uploaded by

calajelukova
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
0% found this document useful (0 votes)
75 views

Programming Persistent Memory A Comprehensive Guide For Developers 1st Edition Steve Scargall pdf download

The document is a comprehensive guide on programming persistent memory, authored by Steve Scargall, covering various aspects such as architecture, operating system support, and development kits. It includes detailed chapters on fundamental concepts, libraries, and tools for developers to effectively utilize persistent memory in their applications. The guide is published under a Creative Commons license, allowing for sharing and adaptation with proper credit to the author.

Uploaded by

calajelukova
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 65

Programming Persistent Memory A Comprehensive

Guide For Developers 1st Edition Steve Scargall


download

https://ebookbell.com/product/programming-persistent-memory-a-
comprehensive-guide-for-developers-1st-edition-steve-
scargall-34689828

Explore and download more ebooks at ebookbell.com


Here are some recommended products that we believe you will be
interested in. You can click the link to download.

Programming Persistent Memory Steve Scargall

https://ebookbell.com/product/programming-persistent-memory-steve-
scargall-59042598

Vertically Integrated Architectures Versioned Data Models Implicit


Services And Persistenceaware Programming 1st Edition Jos Jong

https://ebookbell.com/product/vertically-integrated-architectures-
versioned-data-models-implicit-services-and-persistenceaware-
programming-1st-edition-jos-jong-7293810

Programming Languages And Systems 31st European Symposium On


Programming Esop 2022 Held As Part Of The European Joint Conferences
On Theory And Practice Of Software Etaps 2022 Munich Germany April 27
2022 Proceedings Ilya Sergey
https://ebookbell.com/product/programming-languages-and-systems-31st-
european-symposium-on-programming-esop-2022-held-as-part-of-the-
european-joint-conferences-on-theory-and-practice-of-software-
etaps-2022-munich-germany-april-27-2022-proceedings-ilya-
sergey-44887738

Programming 101 Learn To Code Using The Processing Programming


Language 2nd Edition 2nd Jeanine Meyer

https://ebookbell.com/product/programming-101-learn-to-code-using-the-
processing-programming-language-2nd-edition-2nd-jeanine-meyer-46238180
Programming 101 The How And Why Of Programming Revealed Using The
Processing Programming Language Jeanine Meyer

https://ebookbell.com/product/programming-101-the-how-and-why-of-
programming-revealed-using-the-processing-programming-language-
jeanine-meyer-46318424

Programming And Gui Fundamentals Tcltk For Electronic Design


Automation Suman Lata Tripathi

https://ebookbell.com/product/programming-and-gui-fundamentals-tcltk-
for-electronic-design-automation-suman-lata-tripathi-46318712

Programming With Openscad A Beginners Guide To Coding 3dprintable


Objects 1st Edition Justin Gohde

https://ebookbell.com/product/programming-with-openscad-a-beginners-
guide-to-coding-3dprintable-objects-1st-edition-justin-gohde-46410140

Programming In Two Semesters Using Python And Java Quentin Charatan

https://ebookbell.com/product/programming-in-two-semesters-using-
python-and-java-quentin-charatan-46494972

Programming The Future Politics Resistance And Utopia In Contemporary


Speculative Tv Sherryl Vint Jonathan Alexander

https://ebookbell.com/product/programming-the-future-politics-
resistance-and-utopia-in-contemporary-speculative-tv-sherryl-vint-
jonathan-alexander-46771488
Programming
Persistent
Memory
A Comprehensive Guide for Developers

Steve Scargall
Programming Persistent
Memory
A Comprehensive Guide for
Developers

Steve Scargall
Programming Persistent Memory: A Comprehensive Guide for Developers

Steve Scargall
Santa Clara, CA, USA

ISBN-13 (pbk): 978-1-4842-4931-4 ISBN-13 (electronic): 978-1-4842-4932-1


https://doi.org/10.1007/978-1-4842-4932-1
Copyright © 2020 by Intel
This work is subject to copyright. All rights are reserved by the Publisher, whether the whole or part of the material is
concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction
on microfilms or in any other physical way, and transmission or information storage and retrieval, electronic
adaptation, computer software, or by similar or dissimilar methodology now known or hereafter developed.
Open Access This book is licensed under the terms of the Creative Commons Attribution 4.0
International License (http://creativecommons.org/licenses/by/4.0/), which permits use,
sharing, adaptation, distribution and reproduction in any medium or format, as long as you give
appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and
indicate if changes were made.
The images or other third party material in this book are included in the book’s Creative Commons license, unless
indicated otherwise in a credit line to the material. If material is not included in the book’s Creative Commons
license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need
to obtain permission directly from the copyright holder.
Trademarked names, logos, and images may appear in this book. Rather than use a trademark symbol with every
occurrence of a trademarked name, logo, or image we use the names, logos, and images only in an editorial fashion
and to the benefit of the trademark owner, with no intention of infringement of the trademark.
The use in this publication of trade names, trademarks, service marks, and similar terms, even if they are not identified
as such, is not to be taken as an expression of opinion as to whether or not they are subject to proprietary rights.
While the advice and information in this book are believed to be true and accurate at the date of publication, neither
the authors nor the editors nor the publisher can accept any legal responsibility for any errors or omissions that may
be made. The publisher makes no warranty, express or implied, with respect to the material contained herein.
Managing Director, Apress Media LLC: Welmoed Spahr
Acquisitions Editor: Susan McDermott
Development Editor: Laura Berendson
Coordinating Editor: Jessica Vakili
Distributed to the book trade worldwide by Springer Science+Business Media New York, 233 Spring Street,
6th Floor, New York, NY 10013. Phone 1-800-SPRINGER, fax (201) 348-4505, e-mail orders-ny@springer-sbm.com,
or visit www.springeronline.com. Apress Media, LLC is a California LLC and the sole member (owner) is Springer
Science + Business Media Finance Inc (SSBM Finance Inc). SSBM Finance Inc is a Delaware corporation.
For information on translations, please e-mail rights@apress.com, or visit http://www.apress.com/
rights-permissions.
Apress titles may be purchased in bulk for academic, corporate, or promotional use. eBook versions and licenses
are also available for most titles. For more information, reference our Print and eBook Bulk Sales web page at
http://www.apress.com/bulk-sales.
Any source code or other supplementary material referenced by the author in this book is available to readers on
GitHub via the book's product page, located at www.apress.com/978-1-4842-4931-4. For more detailed
information, please visit http://www.apress.com/source-code.
Printed on acid-free paper
Table of Contents
About the Author��������������������������������������������������������������������������������������������������� xiii
About the Technical Reviewer���������������������������������������������������������������������������������xv
About the Contributors������������������������������������������������������������������������������������������xvii
Acknowledgments��������������������������������������������������������������������������������������������������xxi
Preface�����������������������������������������������������������������������������������������������������������������xxiii

Chapter 1: Introduction to Persistent Memory Programming���������������������������������� 1


A High-Level Example Program���������������������������������������������������������������������������������������������������� 2
What’s Different?��������������������������������������������������������������������������������������������������������������������� 5
The Performance Difference��������������������������������������������������������������������������������������������������� 6
Program Complexity���������������������������������������������������������������������������������������������������������������� 7
How Does libpmemkv Work?��������������������������������������������������������������������������������������������������� 8
What’s Next?��������������������������������������������������������������������������������������������������������������������������������� 9
Summary�������������������������������������������������������������������������������������������������������������������������������������� 9

Chapter 2: Persistent Memory Architecture����������������������������������������������������������� 11


Persistent Memory Characteristics��������������������������������������������������������������������������������������������� 12
Platform Support for Persistent Memory������������������������������������������������������������������������������������ 13
Cache Hierarchy�������������������������������������������������������������������������������������������������������������������������� 14
Power-Fail Protected Domains���������������������������������������������������������������������������������������������������� 16
The Need for Flushing, Ordering, and Fencing���������������������������������������������������������������������������� 19
Data Visibility������������������������������������������������������������������������������������������������������������������������������ 23
Intel Machine Instructions for Persistent Memory���������������������������������������������������������������������� 24
Detecting Platform Capabilities�������������������������������������������������������������������������������������������������� 25

iii
Table of Contents

Application Startup and Recovery����������������������������������������������������������������������������������������������� 27


What’s Next?������������������������������������������������������������������������������������������������������������������������������� 29
Summary������������������������������������������������������������������������������������������������������������������������������������ 29

Chapter 3: Operating System Support for Persistent Memory������������������������������� 31


Operating System Support for Memory and Storage������������������������������������������������������������������ 31
Persistent Memory As Block Storage������������������������������������������������������������������������������������������ 33
Persistent Memory-Aware File Systems������������������������������������������������������������������������������������� 34
Memory-Mapped Files���������������������������������������������������������������������������������������������������������������� 35
Persistent Memory Direct Access (DAX)������������������������������������������������������������������������������������� 43
Summary������������������������������������������������������������������������������������������������������������������������������������ 53

Chapter 4: Fundamental Concepts of Persistent Memory Programming��������������� 55


What’s Different?������������������������������������������������������������������������������������������������������������������������ 55
Atomic Updates��������������������������������������������������������������������������������������������������������������������������� 56
Transactions�������������������������������������������������������������������������������������������������������������������������������� 57
Atomicity������������������������������������������������������������������������������������������������������������������������������� 57
Consistency��������������������������������������������������������������������������������������������������������������������������� 58
Isolation��������������������������������������������������������������������������������������������������������������������������������� 58
Durability������������������������������������������������������������������������������������������������������������������������������� 58
Flushing Is Not Transactional������������������������������������������������������������������������������������������������������ 59
Start-Time Responsibilities��������������������������������������������������������������������������������������������������������� 59
Tuning for Hardware Configurations������������������������������������������������������������������������������������������� 60
Summary������������������������������������������������������������������������������������������������������������������������������������ 60

Chapter 5: Introducing the Persistent Memory Development Kit��������������������������� 63


Background��������������������������������������������������������������������������������������������������������������������������������� 63
Choosing the Right Semantics���������������������������������������������������������������������������������������������������� 64
Volatile Libraries������������������������������������������������������������������������������������������������������������������������� 65
libmemkind���������������������������������������������������������������������������������������������������������������������������� 65
libvmemcache����������������������������������������������������������������������������������������������������������������������� 66
libvmem��������������������������������������������������������������������������������������������������������������������������������� 67

iv
Table of Contents

Persistent Libraries��������������������������������������������������������������������������������������������������������������������� 67
libpmem�������������������������������������������������������������������������������������������������������������������������������� 67
libpmemobj���������������������������������������������������������������������������������������������������������������������������� 68
libpmemobj-cpp�������������������������������������������������������������������������������������������������������������������� 68
libpmemkv����������������������������������������������������������������������������������������������������������������������������� 69
libpmemlog���������������������������������������������������������������������������������������������������������������������������� 69
libpmemblk���������������������������������������������������������������������������������������������������������������������������� 69
Tools and Command Utilities������������������������������������������������������������������������������������������������������� 70
pmempool������������������������������������������������������������������������������������������������������������������������������ 70
pmemcheck��������������������������������������������������������������������������������������������������������������������������� 70
pmreorder������������������������������������������������������������������������������������������������������������������������������ 71
Summary������������������������������������������������������������������������������������������������������������������������������������ 71

Chapter 6: libpmem: Low-Level Persistent Memory Support��������������������������������� 73


Using the Library������������������������������������������������������������������������������������������������������������������������� 74
Mapping a File���������������������������������������������������������������������������������������������������������������������������� 75
Copying to Persistent Memory���������������������������������������������������������������������������������������������������� 76
Separating the Flush Steps��������������������������������������������������������������������������������������������������������� 77
Summary������������������������������������������������������������������������������������������������������������������������������������ 79

Chapter 7: libpmemobj: A Native Transactional Object Store��������������������������������� 81


What is libpmemobj?������������������������������������������������������������������������������������������������������������������ 81
Why not malloc( )?����������������������������������������������������������������������������������������������������������������������� 82
Grouping Operations������������������������������������������������������������������������������������������������������������������� 83
Memory Pools����������������������������������������������������������������������������������������������������������������������������� 83
Creating Memory Pools��������������������������������������������������������������������������������������������������������� 83
Pool Object Pointer (POP) and the Root Object���������������������������������������������������������������������� 87
Opening and Reading from Memory Pools���������������������������������������������������������������������������� 88
Memory Poolsets������������������������������������������������������������������������������������������������������������������������ 90
Concatenated Poolsets���������������������������������������������������������������������������������������������������������� 90
Replica Poolsets�������������������������������������������������������������������������������������������������������������������� 91
Managing Memory Pools and Poolsets��������������������������������������������������������������������������������������� 92

v
Table of Contents

Typed Object Identifiers (TOIDs)�������������������������������������������������������������������������������������������������� 92


Allocating Memory���������������������������������������������������������������������������������������������������������������������� 93
Persisting Data���������������������������������������������������������������������������������������������������������������������������� 94
Atomic Operations����������������������������������������������������������������������������������������������������������������� 94
Reserve/Publish API�������������������������������������������������������������������������������������������������������������� 97
Transactional API����������������������������������������������������������������������������������������������������������������� 100
Optional Flags���������������������������������������������������������������������������������������������������������������������� 104
Persisting Data Summary���������������������������������������������������������������������������������������������������� 104
Guarantees of libpmemobj’s APIs��������������������������������������������������������������������������������������������� 105
Managing Library Behavior������������������������������������������������������������������������������������������������������� 106
Debugging and Error Handling�������������������������������������������������������������������������������������������������� 106
Summary���������������������������������������������������������������������������������������������������������������������������������� 108

Chapter 8: libpmemobj-cpp: The Adaptable Language - C++ and


Persistent Memory����������������������������������������������������������������������������������������������� 111
Introduction������������������������������������������������������������������������������������������������������������������������������� 111
Metaprogramming to the Rescue��������������������������������������������������������������������������������������������� 112
Persistent Pointers�������������������������������������������������������������������������������������������������������������� 112
Transactions������������������������������������������������������������������������������������������������������������������������ 113
Snapshotting����������������������������������������������������������������������������������������������������������������������� 115
Allocating����������������������������������������������������������������������������������������������������������������������������� 116
C++ Standard limitations���������������������������������������������������������������������������������������������������������� 118
An Object’s Lifetime������������������������������������������������������������������������������������������������������������ 119
Trivial Types������������������������������������������������������������������������������������������������������������������������� 120
Object Layout����������������������������������������������������������������������������������������������������������������������� 122
Pointers������������������������������������������������������������������������������������������������������������������������������� 123
Limitations Summary���������������������������������������������������������������������������������������������������������� 125
Persistence Simplified�������������������������������������������������������������������������������������������������������������� 126
The Ecosystem�������������������������������������������������������������������������������������������������������������������������� 133
Persistent Containers���������������������������������������������������������������������������������������������������������� 134
Examples of Persistent Containers�������������������������������������������������������������������������������������� 135
Summary���������������������������������������������������������������������������������������������������������������������������������� 138

vi
Table of Contents

Chapter 9: pmemkv: A Persistent In-­Memory Key-Value Store���������������������������� 141


pmemkv Architecture���������������������������������������������������������������������������������������������������������������� 143
A Phonebook Example�������������������������������������������������������������������������������������������������������������� 147
Bringing Persistent Memory Closer to the Cloud���������������������������������������������������������������������� 151
Summary���������������������������������������������������������������������������������������������������������������������������������� 152

Chapter 10: Volatile Use of Persistent Memory���������������������������������������������������� 155


Introduction������������������������������������������������������������������������������������������������������������������������������� 155
Background������������������������������������������������������������������������������������������������������������������������������� 156
Memory Allocation��������������������������������������������������������������������������������������������������������������� 156
How it Works����������������������������������������������������������������������������������������������������������������������� 156
Supported “Kinds” of Memory��������������������������������������������������������������������������������������������� 157
The memkind API���������������������������������������������������������������������������������������������������������������������� 159
Kind Management API��������������������������������������������������������������������������������������������������������� 159
Heap Management API�������������������������������������������������������������������������������������������������������� 164
Kind Configuration Management����������������������������������������������������������������������������������������� 167
Additional memkind Code Examples����������������������������������������������������������������������������������� 168
C++ Allocator for PMEM Kind��������������������������������������������������������������������������������������������������� 168
pmem::allocator methods���������������������������������������������������������������������������������������������������� 169
Nested Containers��������������������������������������������������������������������������������������������������������������� 169
C++ Examples�������������������������������������������������������������������������������������������������������������������������� 170
Using the pmem::allocator�������������������������������������������������������������������������������������������������� 170
Creating a Vector of Strings������������������������������������������������������������������������������������������������� 171
Expanding Volatile Memory Using Persistent Memory�������������������������������������������������������������� 173
libvmemcache: An Efficient Volatile Key-Value Cache for Large-Capacity
Persistent Memory�������������������������������������������������������������������������������������������������������������������� 177
libvmemcache Overview����������������������������������������������������������������������������������������������������� 178
libvmemcache Design��������������������������������������������������������������������������������������������������������� 180
Using libvmemcache����������������������������������������������������������������������������������������������������������� 183
Summary���������������������������������������������������������������������������������������������������������������������������������� 186

vii
Table of Contents

Chapter 11: Designing Data Structures for Persistent Memory��������������������������� 187


Contiguous Data Structures and Fragmentation����������������������������������������������������������������������� 187
Internal and External Fragmentation����������������������������������������������������������������������������������� 188
Atomicity and Consistency�������������������������������������������������������������������������������������������������� 189
Selective Persistence���������������������������������������������������������������������������������������������������������� 193
Example Data Structures����������������������������������������������������������������������������������������������������� 193
Summary���������������������������������������������������������������������������������������������������������������������������������� 206

Chapter 12: Debugging Persistent Memory Applications������������������������������������� 207


pmemcheck for Valgrind����������������������������������������������������������������������������������������������������������� 208
Stack Overflow Example������������������������������������������������������������������������������������������������������ 208
Memory Leak Example�������������������������������������������������������������������������������������������������������� 209
Intel Inspector – Persistence Inspector������������������������������������������������������������������������������������ 210
Stack Overflow Example������������������������������������������������������������������������������������������������������ 211
Memory Leak Example�������������������������������������������������������������������������������������������������������� 212
Common Persistent Memory Programming Problems�������������������������������������������������������������� 214
Nonpersistent Stores����������������������������������������������������������������������������������������������������������� 214
Stores Not Added into a Transaction������������������������������������������������������������������������������������ 228
Memory Added to Two Different Transactions��������������������������������������������������������������������� 233
Memory Overwrites������������������������������������������������������������������������������������������������������������� 240
Unnecessary Flushes���������������������������������������������������������������������������������������������������������� 242
Out-of-Order Writes������������������������������������������������������������������������������������������������������������� 247
Summary���������������������������������������������������������������������������������������������������������������������������������� 259

Chapter 13: Enabling Persistence Using a Real-World Application���������������������� 261


The Database Example������������������������������������������������������������������������������������������������������������� 262
Different Persistent Memory Enablement Approaches������������������������������������������������������������� 262
Developing a Persistent Memory-Aware MariaDB* Storage Engine����������������������������������������� 263
Understanding the Storage Layer���������������������������������������������������������������������������������������� 264
Creating a Storage Engine Class����������������������������������������������������������������������������������������� 265
Summary���������������������������������������������������������������������������������������������������������������������������������� 276

viii
Table of Contents

Chapter 14: Concurrency and Persistent Memory������������������������������������������������ 277


Transactions and Multithreading����������������������������������������������������������������������������������������������� 278
Mutexes on Persistent Memory������������������������������������������������������������������������������������������������ 282
Atomic Operations and Persistent Memory������������������������������������������������������������������������������ 285
Lock-Free Algorithms and Persistent Memory�������������������������������������������������������������������� 285
Concurrent Data Structures for Persistent Memory������������������������������������������������������������������ 286
Concurrent Ordered Map����������������������������������������������������������������������������������������������������� 287
Concurrent Hash Map���������������������������������������������������������������������������������������������������������� 291
Summary���������������������������������������������������������������������������������������������������������������������������������� 293

Chapter 15: Profiling and Performance���������������������������������������������������������������� 295


Introduction������������������������������������������������������������������������������������������������������������������������������� 295
Performance Analysis Concepts����������������������������������������������������������������������������������������������� 295
Compute-Bound vs. Memory-Bound����������������������������������������������������������������������������������� 295
Memory Latency vs. Memory Capacity������������������������������������������������������������������������������� 296
Read vs. Write Performance������������������������������������������������������������������������������������������������ 296
Memory Access Patterns����������������������������������������������������������������������������������������������������� 296
I/O Storage Bound Workloads���������������������������������������������������������������������������������������������� 297
Determining the Suitability of Workloads for Persistent Memory��������������������������������������������� 297
Volatile Use Cases��������������������������������������������������������������������������������������������������������������� 298
Use Cases Requiring Persistence���������������������������������������������������������������������������������������� 301
Performance Analysis of Workloads Using Persistent Memory������������������������������������������������ 302
Characterizing the Workload����������������������������������������������������������������������������������������������� 303
Memory Bandwidth and Latency����������������������������������������������������������������������������������������� 303
Persistent Memory Read-Write Ratio���������������������������������������������������������������������������������� 305
Working Set Size and Memory Footprint����������������������������������������������������������������������������� 305
Non-Uniform Memory Architecture (NUMA) Behavior��������������������������������������������������������� 305
Optimizing the Software for Persistent Memory����������������������������������������������������������������� 307
Summary���������������������������������������������������������������������������������������������������������������������������������� 311

ix
Table of Contents

Chapter 16: PMDK Internals: Important Algorithms and Data Structures������������ 313
A Pool of Persistent Memory: High-Level Architecture Overview��������������������������������������������� 313
The Uncertainty of Memory Mapping: Persistent Memory Object Identifier����������������������������� 315
Persistent Thread Local Storage: Using Lanes�������������������������������������������������������������������������� 318
Ensuring Power-Fail Atomicity: Redo and Undo Logging���������������������������������������������������������� 320
Transaction Redo Logging��������������������������������������������������������������������������������������������������� 320
Transaction Undo Logging��������������������������������������������������������������������������������������������������� 321
libpmemobj Unified Logging������������������������������������������������������������������������������������������������ 322
Persistent Allocations: The Interface of a Transactional Persistent Allocator���������������������������� 323
Persistent Memory Heap Management: Allocator Design for Persistent Memory�������������������� 324
ACID Transactions: Efficient Low-Level Persistent Transactions����������������������������������������������� 328
Lazy Reinitialization of Variables: Storing the Volatile State on Persistent Memory����������������� 330
Summary���������������������������������������������������������������������������������������������������������������������������������� 331

Chapter 17: Reliability, Availability, and Serviceability (RAS)������������������������������ 333


Dealing with Uncorrectable Errors�������������������������������������������������������������������������������������������� 333
Consumed Uncorrectable Error Handling���������������������������������������������������������������������������� 334
Unconsumed Uncorrectable Error Handling������������������������������������������������������������������������ 336
Clearing Uncorrectable Errors��������������������������������������������������������������������������������������������� 339
Device Health���������������������������������������������������������������������������������������������������������������������������� 339
ACPI-Defined Health Functions (_NCH, _NBS)��������������������������������������������������������������������� 342
Vendor-Specific Device Health (_DSMs)������������������������������������������������������������������������������ 342
ACPI NFIT Health Event Notification������������������������������������������������������������������������������������ 343
Unsafe/Dirty Shutdown������������������������������������������������������������������������������������������������������������� 343
Application Utilization of Data Loss Count (DLC)����������������������������������������������������������������� 344
Summary���������������������������������������������������������������������������������������������������������������������������������� 346

Chapter 18: Remote Persistent Memory��������������������������������������������������������������� 347


RDMA Networking Protocols����������������������������������������������������������������������������������������������������� 348
Goals of the Initial Remote Persistent Memory Architecture���������������������������������������������������� 351
Guaranteeing Remote Persistence�������������������������������������������������������������������������������������������� 351
General-Purpose Remote Replication Method��������������������������������������������������������������������� 353

x
Table of Contents

Appliance Remote Replication Method�������������������������������������������������������������������������������� 355


General Software Architecture�������������������������������������������������������������������������������������������������� 357
librpmem Architecture and Its Use in Replication��������������������������������������������������������������������� 358
Configuring Remote Replication Using Poolsets����������������������������������������������������������������� 362
Performance Considerations����������������������������������������������������������������������������������������������� 362
Remote Replication Error Handling������������������������������������������������������������������������������������� 364
Say Hello to the Replicated World��������������������������������������������������������������������������������������� 364
Summary���������������������������������������������������������������������������������������������������������������������������������� 370

Chapter 19: Advanced Topics������������������������������������������������������������������������������� 373


Nonuniform Memory Access (NUMA)���������������������������������������������������������������������������������������� 373
NUMACTL Linux Utility��������������������������������������������������������������������������������������������������������� 374
NDCTL Linux Utility�������������������������������������������������������������������������������������������������������������� 376
Intel Memory Latency Checker Utility���������������������������������������������������������������������������������� 378
NUMASTAT Utility����������������������������������������������������������������������������������������������������������������� 380
Intel VTune Profiler – Platform Profiler�������������������������������������������������������������������������������� 381
IPMCTL Utility���������������������������������������������������������������������������������������������������������������������� 381
BIOS Tuning Options������������������������������������������������������������������������������������������������������������ 382
Automatic NUMA Balancing������������������������������������������������������������������������������������������������� 382
Using Volume Managers with Persistent Memory�������������������������������������������������������������������� 383
The mmap( ) MAP_SYNC Flag��������������������������������������������������������������������������������������������������� 385
Summary���������������������������������������������������������������������������������������������������������������������������������� 386

Appendix A: How to Install NDCTL and DAXCTL on Linux������������������������������������� 389


Prerequisites����������������������������������������������������������������������������������������������������������������������������� 389
Installing NDCTL and DAXCTL Using the Linux Distribution Package Repository���������������������� 390
Searching for Packages Within a Package Repository�������������������������������������������������������� 391
Installing NDCTL and DAXCTL from the Package Repository���������������������������������������������� 392

xi
Table of Contents

Appendix B: How to Install the Persistent Memory Development Kit (PMDK)������ 395
PMDK Prerequisites������������������������������������������������������������������������������������������������������������������ 395
Installing PMDK Using the Linux Distribution Package Repository������������������������������������������� 395
Package Naming Convention����������������������������������������������������������������������������������������������� 396
Searching for Packages Within a Package Repository�������������������������������������������������������� 396
Installing PMDK Libraries from the Package Repository����������������������������������������������������� 398
Installing PMDK on Microsoft Windows������������������������������������������������������������������������������������ 402

Appendix C: How to Install IPMCTL on Linux and Windows���������������������������������� 403


IPMCTL Linux Prerequisites������������������������������������������������������������������������������������������������������ 404
libsafec�������������������������������������������������������������������������������������������������������������������������������� 404
IPMCTL Linux Packages������������������������������������������������������������������������������������������������������������ 404
IPMCTL for Microsoft Windows������������������������������������������������������������������������������������������������� 404
Using ipmctl������������������������������������������������������������������������������������������������������������������������������ 405

Appendix D: Java for Persistent Memory������������������������������������������������������������� 411


Volatile Use of Persistent Memory�������������������������������������������������������������������������������������������� 411
Heap Allocation on Alternative Memory Devices����������������������������������������������������������������� 412
Persistent Collections for Java (PCJ)���������������������������������������������������������������������������������������� 416
Using PCJ in Java Applications������������������������������������������������������������������������������������������������� 417
Low-Level Persistent Library (LLPL)����������������������������������������������������������������������������������������� 418
Using LLPL in Java Applications������������������������������������������������������������������������������������������������ 419
Summary���������������������������������������������������������������������������������������������������������������������������������� 419

Appendix E: The Future of Remote Persistent Memory Replication��������������������� 421

Glossary���������������������������������������������������������������������������������������������������������������� 425

Index��������������������������������������������������������������������������������������������������������������������� 429

xii
About the Author
Steve Scargall is a persistent memory software and cloud architect at Intel
Corporation. As a technology evangelist, he supports the enabling and development
effort to integrate persistent memory technology into software stacks, applications,
and hardware architectures. This includes working with independent software
vendors (ISVs) on both proprietary and open source development, original equipment
manufacturers (OEMs), and cloud service providers (CSPs).

Steve holds a Bachelor of Science in computer science and cybernetics from the
University of Reading, UK, where he studied neural networks, AI, and robotics. He
has over 19 years’ experience providing performance analysis on x86 architecture and
SPARC for Solaris Kernel, ZFS, and UFS file system. He performed DTrace debugging in
enterprise and cloud environments during his tenures at Sun Microsystems and Oracle.

xiii
About the Technical Reviewer
Andy Rudoff is a principal engineer at Intel Corporation, focusing on non-volatile
memory programming. He is a contributor to the SNIA NVM Programming Technical
Work Group. His more than 30 years’ industry experience includes design and
development work in operating systems, file systems, networking, and fault management
at companies large and small, including Sun Microsystems and VMware. Andy has
taught various operating systems classes over the years and is a coauthor of the popular
UNIX Network Programming textbook.

xv
About the Contributors
Piotr Balcer is a software engineer at Intel Corporation with many years’ experience
working on storage-related technologies. He holds a Bachelor of Science in engineering
from the Gdańsk University of Technology, Poland, where he studied system software
engineering. Piotr has been working on the software ecosystem for next-generation
persistent memory since 2014.

Eduardo Berrocal joined Intel Corporation as a cloud software engineer in 2017 after
receiving his PhD in computer science from the Illinois Institute of Technology. His
doctoral research focused on data analytics and fault tolerance for high-performance
computing. Past experience includes working as an intern at Bell Labs (Nokia), a research
aid at Argonne National Laboratory, a scientific programmer and web developer at the
University of Chicago, and an intern in the CESVIMA laboratory in Spain.

Adam Borowski is a software engineer at Intel Corporation, hailing from the


University of Warsaw, Poland. He is a Debian developer and has made many open
source contributions over the past two decades. Adam is currently working on
persistent memory stacks, both on upstream code and integrating it with downstream
distributions.

Igor Chorazewicz is a software engineer at Intel Corporation. His main focus is on


persistent memory data structures and enabling C++ applications for persistent
memory. Igor holds a Bachelor of Science in engineering from the Gdańsk University of
Technology, Poland.

Adam Czapski is a technical writer at Intel Corporation. He writes technical


documentation in the Data Center Group and is currently working in the persistent
memory department. Adam holds a Bachelor of Arts in English philology and a master’s
degree in natural language processing from the Gdańsk University of Technology, Poland.

Steve Dohrmann is a software engineer at Intel Corporation. He has worked on a variety


of projects over the past 20 years, including media frameworks, mobile agent software,
secure collaboration software, and parallel programming language implementation. He
is currently working on enabling the use of persistent memory in Java*.

xvii
About the Contributors

Chet Douglas is a principal software engineer at Intel Corporation and focuses on cloud
software architecture along with operating system and OEM enabling of non-volatile
memory technologies. He has over 14 years’ experience working on various enterprise
and client programs and 28 years of total storage experience. Chet has worked in all
aspects of storage, including storage controller hardware design, SCSI disk/tape/CD
writer firmware architecture, storage management software architecture, Microsoft
Windows* and Linux kernel-mode drivers, enterprise hardware RAID, and client/
workstation software RAID. He holds seven storage-related hardware and software
patents and has a dual Bachelor of Science in electrical engineering and computer
engineering from Clarkson University, New York.

Ken Gibson is the director of persistent memory software architecture within Intel
Corporation’s Data Center Group. Since 2012, Ken and his team have been working with
Intel’s server and software partners to create the open persistent memory programming
model.

Tomasz Gromadzki is a software architect in Intel Corporation’s Non-Volatile Memory


Solutions Group. His focus is on remote persistent memory access, which includes
proper integration of persistent memory with other (networking) technologies as well as
optimal persistent memory replication procedures and algorithms.

Kishor Kharbas is a software engineer on the Java runtime engineering team at Intel
Corporation. For the past eight years, he has been working to optimize Oracle’s OpenJDK
on Intel platforms. This involves Java garbage collection and compiler back-end
optimization.

Jackson Marusarz is a senior technical consulting engineer (TCE) in Intel Corporation's


Compute Performance and Developer Products Division. As the lead TCE for Intel
VTune Profiler, his main focus is on software performance analysis and tuning for both
serial and multithreaded applications. Jackson’s time is split between determining how
to analyze and tune software and creating tools that help others do the same.

Jan Michalski is a software engineer in Intel Corporation’s Non-­Volatile Memory


Solutions Group. His focus is on remote persistent memory access, which includes
proper integration of persistent memory with other technologies, as well as looking for
optimal persistent memory replication procedures and algorithms. He holds a master's
degree in computer engineering from the Gdańsk University of Technology, Poland,
where he studied system software engineering.

xviii
About the Contributors

Nicholas Moulin is a cloud software architect at Intel Corporation. Since joining Intel
in 2012, he has focused on enabling and developing persistent memory software for
operating systems and platform firmware and managing persistent memory hardware.
Nicholas is currently working with industry partners to define and improve RAS features
relevant to the persistent memory programming model.

Szymon Romik is a software engineer at Intel Corporation and is currently focused on


persistent memory programming. He previously worked as a lead software engineer
on 5G technologies at Ericsson. Szymon holds a master’s degree in mathematics from
Jagiellonian University, Poland.

Jakub Schmiegel is a software architect in Intel Corporation’s Non-Volatile Memory


Solutions Group where he has been focused on enabling existing applications to
persistent memory and analyzing their performance for more than four years. Jakub
holds a master’s degree in computer science from the Gdańsk University of Technology,
Poland.

Kevin Shalkowsky is a Telly Award–winning creative director, graphic designer, and


animator with more than a decade of experience. While his contributions are in
technology today, Kevin has spent time in broadcast journalism and selling numerous
products through 30-minute late-night infomercials. He resides in Oregon with his wife
and son. From time to time, you can find Kevin lost in the woods, lost in a parking lot,
or lost in his design process – but this somehow got him to where he is today, and he
wouldn’t have it any other way.

Vineet Singh is a memory and storage tools software engineer at Intel Corporation.
He develops techniques to help developers adapt to the latest memory technologies.
Vineet holds a PhD in philosophy from the University of California and has a Bachelor
of Technology degree from the Indian Institute of Information Technology, Design, and
Manufacturing in Jabalpur.

Pawel Skowron is a software engineering manager at Intel Corporation with 20 years'


experience in the software industry. Pawel has worked in various roles related to the
whole-software development life cycle. His software engineering background lies in the
areas of embedded systems, database systems, and applications. For the past few years,
Pawel has led the development and validation of the Persistent Memory Development
Kit (https://github.com/pmem/pmdk).

xix
About the Contributors

Usha Upadhyayula has been with Intel Corporation for 20 years serving in many
different roles. Usha holds a master’s degree in computer science from the University
of South Carolina, and she spent the first few years at Intel developing user-level
applications in C and C++. She later moved to customer-­enabling roles for Intel media
processors and support for Intel RAID software. Usha is currently part of the Data Center
Group where she is focused on enabling cloud service providers to fully utilize and
accelerate the adoption of Intel persistent memory products.

Sergey Vinogradov is a senior software development engineer at Intel Corporation


where he spent more than seven years working on performance profiling tools and
threading runtime libraries. During the past four years, Sergey has been working on C++
programming models and performance profiling methodologies for persistent memory.

xx
Acknowledgments
First and foremost, I would like to thank Ken Gibson for masterminding this book idea
and for gifting me the pleasure of writing and managing it. Your support, guidance, and
contributions have been instrumental in delivering a high-quality product.
If the Vulcan mind-meld or The Matrix Headjack were possible, I could have cloned
Andy Rudoff’s mind and allowed him to work on his daily activities. Instead, Andy’s
infinite knowledge of persistent memory had to be tapped through good old verbal
communication and e-mail. I sincerely thank you for devoting so much time to me and
this project. The results read for themselves.
Debbie Graham was instrumental in helping me manage this colossal project. Her
dedication and support helped drive the project to an on-time completion.
To my friends and colleagues at Intel who contributed content, supported
discussions, helped with decision-making, and reviewed drafts during the book-writing
process. These are the real heroes. Without your heavily invested time and support, this
book would have taken considerably longer to complete. It is a much better product as a
result of the collaborative effort. A huge thanks to all of you.
I'd like to express my sincerest gratitude and appreciation to the people at Apress,
without whom this book could not have been published. From the initial contact and
outline discussions through the entire publishing process to this final polished product,
the Apress team delivered continuous support and assistance. Many thanks to Susan,
Jessica, and Rita. It was a real pleasure working with you.

xxi
Preface
About This Book
Persistent memory is often referred to as non-volatile memory (NVM) or storage
class memory (SCM). In this book, we purposefully use persistent memory as an all-­
encompassing term to represent all the current and future memory technologies that
fall under this umbrella. This book introduces the persistent memory technology and
provides answers to key questions. For software developers, those questions include:
What is persistent memory? How do I use it? What APIs and libraries are available?
What benefits can it provide for my application? What new programming methods do I
need to learn? How do I design applications to use persistent memory? Where can I find
information, documentation, and help?
System and cloud architects will be provided with answers to questions such as:
What is persistent memory? How does it work? How is it different than DRAM or SSD/
NVMe storage devices? What are the hardware and operating system requirements?
What applications need or could benefit from persistent memory? Can my existing
applications use persistent memory without being modified?
Persistent memory is not a plug-and-play technology for software applications.
Although it may look and feel like traditional DRAM memory, applications need to be
modified to fully utilize the persistence feature of persistent memory. That is not to say
that applications cannot run unmodified on systems with persistent memory installed,
they can, but they will not see the full potential of what persistent memory offers without
code modification.
Thankfully, server and operating system vendors collaborated very early in the
design phase and already have products available on the market. Linux and Microsoft
Windows already provide native support for persistent memory technologies. Many
popular virtualization technologies also support persistent memory.
For ISVs and the developer community at large, the journey is just beginning. Some
software has already been modified and is available on the market. However, it will
take time for the enterprise and cloud computing industries to adopt and make the
hardware available to the general marketplace. ISVs and software developers need time
to understand what changes to existing applications are required and implement them.
xxiii
Preface

To make the required development work easier, Intel developed and open sourced
the Persistent Memory Development Kit (PMDK) available from https://pmem.io/
pmdk/. We introduce the PMDK in more detail in Chapter 5 and walk through most of
the available libraries in subsequent chapters. Each chapter provides an in-depth guide
so developers can understand what library or libraries to use. PMDK is a set of open
source libraries and tools based on the Storage Networking Industry Association (SNIA)
NVM programming model designed and implemented by over 50 industry partners. The
latest NVM programming model document can be found at https://www.snia.org/
tech_activities/standards/curr_standards/npm. The model describes how software
can utilize persistent memory features and enables designers to develop APIs that take
advantage of NVM features and performance.
Available for both Linux and Windows, PMDK facilitates persistent memory
programming adoption with higher-level language support. C and C++ support is fully
validated. Support for other languages such as Java and Python is work in progress
at the time this book was written. Other languages are expected to also adopt the
programming model and provide native persistent memory APIs for developers. The
PMDK development team welcomes and encourages new contributions to core code,
new language bindings, or new storage engines for the persistent memory key-value
store called pmemkv.
This book assumes no prior knowledge of persistent memory hardware devices
or software development. The book layout allows you to freely navigate the content in
the order you want. It is not required to read all chapters in order, though we do build
upon concepts and knowledge described in previous chapters. In such cases, we make
backward and forward references to relevant chapters and sections so you can learn or
refresh your memory.

B
 ook Structure
This book has 19 chapters, each one focusing on a different topic. The book has three
main sections. Chapters 1-4 provide an introduction to persistent memory architecture,
hardware, and operating system support. Chapters 5-16 allow developers to understand
the PMDK libraries and how to use them in applications. Finally, Chapters 17-19 provide
information on advanced topics such as RAS and replication of data using RDMA.

xxiv
Preface

• Chapter 1. Introduction to Persistent Memory – Introduces persistent


memory and dips our toes in the water with a simple persistent key-­
value store example using libpmemkv.

• Chapter 2. Persistent Memory Architecture – Describes the persistent


memory architecture and focuses on the hardware requirements
developers should know.

• Chapter 3. Operating System Support for Persistent Memory –


Provides information relating to operating system changes, new
features, and how persistent memory is seen by the OS.

• Chapter 4. Fundamental Concepts of Persistent Memory


Programming – Builds on the first three chapters and describes the
fundamental concepts of persistent memory programming.

• Chapter 5. Introducing the Persistent Memory Development Kit


(PMDK) – Introduces the Persistent Memory Development Kit
(PMDK), a suite of libraries to assist software developers.

• Chapter 6. libpmem: Low-Level Persistent Memory Support –


Describes and shows how to use libpmem from the PMDK, a low-­level
library providing persistent memory support.

• Chapter 7. libpmemobj: A Native Transactional Object Store –


Provides information and examples using libpmemobj, a C native
object store library from the PMDK.
• Chapter 8. libpmemobj-cpp: The Adaptable Language - C++ and
Persistent Memory – Demonstrates the C++ libpmemobj-cpp object
store from the PMDK, built using C++ headers on top of libpmemobj.

• Chapter 9. pmemkv: A Persistent In-Memory Key-Value Store –


Expands upon the introduction to libpmemkv from Chapter 1 with a
more in-depth discussion using examples.

• Chapter 10. Volatile Use of Persistent Memory – This chapter is


for those who want to take advantage of persistent memory but
do not require data to be stored persistently. libmemkind is a user-
extensible heap manager built on top of jemalloc which enables
control of memory characteristics and a partitioning of the heap

xxv
Preface

between different kinds of memory, including persistent memory.


libvmemcache is an embeddable and lightweight in-memory caching
solution. It is designed to fully take advantage of large-capacity
memory, such as persistent memory with DAX, through memory
mapping in an efficient and scalable way.

• Chapter 11. Designing Data Structures for Persistent Memory –


Provides a wealth of information for designing data structures for
persistent memory.
• Chapter 12. Debugging Persistent Memory Applications – Introduces
tools and walks through several examples for how software developers
can debug persistent memory–enabled applications.

• Chapter 13. Enabling Persistence using a Real-World Application –


Discusses how a real-world application was modified to enable
persistent memory features.

• Chapter 14. Concurrency and Persistent Memory – Describes how


concurrency in applications should be implemented for use with
persistent memory.

• Chapter 15. Profiling and Performance – Teaches performance


concepts and demonstrates how to use the Intel VTune suite of tools
to profile systems and applications before and after code changes are
made.

• Chapter 16. PMDK Internals: Important Algorithms and Data


Structures – Takes us on a deep dive of the PMDK design, architecture,
algorithms, and memory allocator implementation.

• Chapter 17. Reliability, Availability, and Serviceability (RAS) –


Describes the implementation of reliability, availability, and
serviceability (RAS) with the hardware and operating system layers.

• Chapter 18. Remote Persistent Memory – Discusses how applications


can scale out across multiple systems using local and remote persistent
memory.

• Chapter 19. Advanced Topics – Describes things such as NUMA, using


software volume managers, and the mmap() MAP_SYNC flag.

xxvi
Preface

The Appendixes have separate procedures for installing the PMDK and utilities
required for managing persistent memory. We also included an update for Java and the
future of the RDMA protocols. All of this content is considered temporal, so we did not
want to include it in the main body of the book.

Intended Audience
This book has been written for experienced application developers in mind. We
intend the content to be useful to a wider readership such as system administrators
and architects, students, lecturers, and academic research fellows to name but a few.
System designers, kernel developers, and anyone with a vested or passing interest in this
emerging technology will find something useful within this book.
Every reader will learn what persistent memory is, how it works, and how operating
systems and applications can utilize it. Provisioning and managing persistent memory
are vendor specific, so we include some resources in the Appendix sections to avoid
overcomplicating the main chapter content.
Application developers will learn, by example, how to integrate persistent memory
in to existing or new applications. We use examples extensively throughout this book
using a variety of libraries available within the Persistent Memory Development Kit
(PMDK). Example code is provided in a variety of programming languages such as C,
C++, JavaScript, and others. We want developers to feel comfortable using these libraries
in their own projects. The book provides extensive links to resources where you can find
help and information.
System administrators and architects of Cloud, high-performance computing,
and enterprise environments can use most of the content of this book to
understand persistent memory features and benefits to support applications and
developers. Imagine being able to deploy more virtual machines per physical server or
provide applications with this new memory/storage tier such that they can keep more
data closer to the CPU or restart in a fraction of the time they could before while keeping
a warm cache of data.
Students, lecturers, and academic research fellows will also benefit from many
chapters within this book. Computer science classes can learn about the hardware,
operating system features, and programming techniques. Lecturers are free use the
content in student classes or to form the basis of research projects such as new persistent
memory file systems, algorithms, or caching implementations.

xxvii
Preface

We introduce tools that profile the server and applications to better understand CPU,
memory, and disk IO access patterns. Using this knowledge, we show how applications
can be modified to take full advantage of persistence using the Persistent Memory
Development Kit (PMDK).

A Future Reference
The book content has been written to provide value for many years. Industry
specification such as ACPI, UEFI, and the SNIA non-volatile programming model will,
unless otherwise stated by the specification, remain backward compatible as new
versions are released. If new form factors are introduced, the approach to programming
remains the same. We do not limit ourselves to one specific persistent memory vendor
or implementation. In places where it is necessary to describe vendor-specific features
or implementations, we specifically call this out as it may change between vendors or
between product generations. We encourage you to read the vendor documentation for
the persistent memory product to learn more.
Developers using the Persistent Memory Development Kit (PMDK) will retain a stable
API interface. PMDK will deliver new features and performance improvements with each
major release. It will evolve with new persistent memory products, CPU instructions,
platform designs, industry specifications, and operating system feature support.

Source Code Examples


Concepts and source code samples within this book adhere to the vendor neutral
SNIA non-volatile memory programming model. SNIA which is the Storage
Networking Industry Association is a non-profit global organization dedicated to
developing standards and education programs to advance storage and information
technology. The model was designed, developed, and is maintained by the SNIA NVM
Technical Working Group (TWG) which includes many leading operating system,
hardware, and server vendors. You can join this group or find information at https://
www.snia.org/forums/sssi/nvmp.

xxviii
Preface

The code examples provided with this book have been tested and validated using
Intel Optane DC persistent memory. Since the PMDK is vendor neutral, they will also
work on NVDIMM-N devices. PMDK will support any future persistent memory product
that enters the market.
The code examples used throughout this book are current at the time of
publication. All code examples have been validated and tested to ensure they compile
and execute without error. For brevity, some of the examples in this book use assert()
statements to indicate unexpected errors. Any production code would likely replace
these with the appropriate error handling actions which would include friendlier
error messages and appropriate error recovery actions. Additionally, some of the code
examples use different mount points to represent persistent memory aware file systems,
for example “/daxfs”, “/pmemfs”, and “/mnt/pmemfs”. This demonstrates persistent
memory file systems can be mounted and named appropriately for the application, just
like regular block-based file systems. Source code is from the repository that accompanies
this book – https://github.com/Apress/programming-persistent-memory.
Since this is a rapidly evolving technology, the software and APIs references
throughout this book may change over time. While every effort is made to be backward
compatible, sometimes software must evolve and invalidate previous versions. For this
reason, it is therefore expected that some of the code samples may not compile on newer
hardware or operating systems and may need to be changed accordingly.

B
 ook Conventions
This book uses several conventions to draw your attention to specific pieces of
information. The convention used depends on the type of information displayed.

C
 omputer Commands
Commands, programming library, and API function references may be presented in line
with the paragraph text using a monospaced font. For example:
To illustrate how persistent memory is used, let’s start with a sample program
demonstrating the key-value store provided by a library called libpmemkv.

xxix
Preface

Computer Terminal Output


Computer terminal output is usually taken directly from a computer terminal presented
in a monospaced font such as the following example demonstrating cloning the
Persistent Memory Development Kit (PMDK) from the GitHub project:

$ git clone https://github.com/pmem/pmdk


Cloning into 'pmdk'...
remote: Enumerating objects: 12, done.
remote: Counting objects: 100% (12/12), done.
remote: Compressing objects: 100% (10/10), done.
remote: Total 100169 (delta 2), reused 7 (delta 2), pack-reused 100157
Receiving objects: 100% (100169/100169), 34.71 MiB | 4.85 MiB/s, done.
Resolving deltas: 100% (83447/83447), done.

S
 ource Code
Source code examples taken from the accompanying GitHub repository are shown with
relevant line numbers in a monospaced font. Below each code listing is a reference to
the line number or line number range with a brief description. Code comments use
language native styling. Most languages use the same syntax. Single line comments
will use // and block/multiline comments should use /*..*/. An example is shown in
Listing 1.

Listing 1. A sample program using libpmemkv


    37  #include <iostream>
    38  #include "libpmemkv.h"
    39  
    40  using namespace pmemkv;
    41  
    42  /*
    43   * kvprint -- print a single key-value pair
    44   */
    45  void kvprint(const string& k, const string& v) {
    46      std::cout << "key: " << k << ", value: " << v << "\n";
    47  }

xxx
Preface

• Line 45: Here we define a small helper routine, kvprint(), which prints
a key-value pair when called.

Notes
We use a standard format for notes, cautions, and tips when we want to direct your
attention to an important point, for example.

Note Notes are tips, shortcuts, or alternative approaches to the current


discussion topic. Ignoring a note should have no negative consequences, but you
might miss out on a nugget of information that makes your life easier.

xxxi
CHAPTER 1

Introduction to Persistent
Memory Programming
This book describes programming techniques for writing applications that use persistent
memory. It is written for experienced software developers, but we assume no previous
experience using persistent memory. We provide many code examples in a variety of
programming languages. Most programmers will understand these examples, even if
they have not previously used the specific language.

Note All code examples are available on a GitHub repository (https://


github.com/Apress/programming-persistent-memory), along with
instructions for building and running it.

Additional documentation for persistent memory, example programs, tutorials, and


details on the Persistent Memory Development Kit (PMDK), which is used heavily in this
book, can be found on http://pmem.io.
The persistent memory products on the market can be used in various ways, and
many of these ways are transparent to applications. For example, all persistent memory
products we encountered support the storage interfaces and standard file API’s just like
any solid-state disk (SSD). Accessing data on an SSD is simple and well-understood, so
we consider these use cases outside the scope of this book. Instead, we concentrate on
memory-style access, where applications manage byte-addressable data structures that
reside in persistent memory. Some use cases we describe are volatile, using the persistent
memory only for its capacity and ignoring the fact it is persistent. However, most of this
book is dedicated to the persistent use cases, where data structures placed in persistent
memory are expected to survive crashes and power failures, and the techniques
described in this book keep those data structures consistent across those events.

1
© The Author(s) 2020
S. Scargall, Programming Persistent Memory, https://doi.org/10.1007/978-1-4842-4932-1_1
Chapter 1 Introduction to Persistent Memory Programming

A High-Level Example Program


To illustrate how persistent memory is used, we start with a sample program
demonstrating the key-value store provided by a library called libpmemkv. Listing 1-1
shows a full C++ program that stores three key-value pairs in persistent memory and
then iterates through the key-value store, printing all the pairs. This example may seem
trivial, but there are several interesting components at work here. Descriptions below the
listing show what the program does.

Listing 1-1. A sample program using libpmemkv

    37  #include <iostream>
    38  #include <cassert>
    39  #include <libpmemkv.hpp>
    40
    41  using namespace pmem::kv;
    42  using std::cerr;
    43  using std::cout;
    44  using std::endl;
    45  using std::string;
    46
    47  /*
    48   * for this example, create a 1 Gig file
    49   * called "/daxfs/kvfile"
    50   */
    51  auto PATH = "/daxfs/kvfile";
    52  const uint64_t SIZE = 1024 * 1024 * 1024;
    53
    54  /*
    55   * kvprint -- print a single key-value pair
    56   */
    57  int kvprint(string_view k, string_view v) {
    58      cout << "key: "    << k.data() <<
    59          " value: " << v.data() << endl;
    60      return 0;
    61  }
    62
2
Chapter 1 Introduction to Persistent Memory Programming

    63  int main() {
    64      // start by creating the db object
    65      db *kv = new db();
    66      assert(kv != nullptr);
    67
    68      // create the config information for
    69      // libpmemkv's open method
    70      config cfg;
    71
    72      if (cfg.put_string("path", PATH) != status::OK) {
    73          cerr << pmemkv_errormsg() << endl;
    74          exit(1);
    75      }
    76      if (cfg.put_uint64("force_create", 1) != status::OK) {
    77          cerr << pmemkv_errormsg() << endl;
    78          exit(1);
    79      }
    80      if (cfg.put_uint64("size", SIZE) != status::OK) {
    81          cerr << pmemkv_errormsg() << endl;
    82          exit(1);
    83      }
    84
    85
    86      // open the key-value store, using the cmap engine
    87      if (kv->open("cmap", std::move(cfg)) != status::OK) {
    88          cerr << db::errormsg() << endl;
    89          exit(1);
    90      }
    91
    92      // add some keys and values
    93      if (kv->put("key1", "value1") != status::OK) {
    94          cerr << db::errormsg() << endl;
    95          exit(1);
    96      }

3
Chapter 1 Introduction to Persistent Memory Programming

    97      if (kv->put("key2", "value2") != status::OK) {


    98          cerr << db::errormsg() << endl;
    99          exit(1);
   100      }
   101      if (kv->put("key3", "value3") != status::OK) {
   102          cerr << db::errormsg() << endl;
   103          exit(1);
   104      }
   105
   106      // iterate through the key-value store, printing them
   107      kv->get_all(kvprint);
   108
   109      // stop the pmemkv engine
   110      delete kv;
   111
   112      exit(0);
   113  }

• Line 57: We define a small helper routine, kvprint(), which prints a


key-value pair when called.
• Line 63: This is the first line of main() which is where every C++
program begins execution. We start by instantiating a key-value
engine using the engine name "cmap". We discuss other engine types
in Chapter 9.
• Line 70: The cmap engine takes config parameters from a config
structure. The parameter "path" is configured to "/daxfs/kvfile",
which is the path to a persistent memory file on a DAX file system;
the parameter "size" is set to SIZE. Chapter 3 describes how to
create and mount DAX file systems.
• Line 93: We add several key-value pairs to the store. The trademark
of a key-value store is the use of simple operations like put() and
get(); we only show put() in this example.
• Line 107: Using the get_all() method, we iterate through the
entire key-value store, printing each pair when get_all() calls our
kvprint() routine.
4
Chapter 1 Introduction to Persistent Memory Programming

W
 hat’s Different?
A wide variety of key-value libraries are available in practically every programming
language. The persistent memory example in Listing 1-1 is different because the key-­
value store itself resides in persistent memory. For comparison, Figure 1-1 shows how a
key-value store using traditional storage is laid out.

Figure 1-1. A key-value store on traditional storage

When the application in Figure 1-1 wants to fetch a value from the key-value store,
a buffer must be allocated in memory to hold the result. This is because the values are
kept on block storage, which cannot be addressed directly by the application. The only
way to access a value is to bring it into memory, and the only way to do that is to read
full blocks from the storage device, which can only be accessed via block I/O. Now
consider Figure 1-2, where the key-value store resides in persistent memory like our
sample code.

5
Chapter 1 Introduction to Persistent Memory Programming

Figure 1-2. A key-value store in persistent memory

With the persistent memory key-value store, values are accessed by the application
directly, without the need to first allocate buffers in memory. The kvprint() routine in
Listing 1-1 will be called with references to the actual keys and values, directly where
they live in persistence – something that is not possible with traditional storage. In
fact, even the data structures used by the key-value store library to organize its data are
accessed directly. When a storage-based key-value store library needs to make a small
update, for example, 64 bytes, it must read the block of storage containing those 64 bytes
into a memory buffer, update the 64 bytes, and then write out the entire block to make it
persistent. That is because storage accesses can only happen using block I/O, typically
4K bytes at a time, so the task to update 64 bytes requires reading 4K and then writing
4K. But with persistent memory, the same example of changing 64 bytes would only
write the 64 bytes directly to persistence.

The Performance Difference


Moving a data structure from storage to persistent memory does not just mean smaller
I/O sizes are supported; there is a fundamental performance difference. To illustrate this,
Figure 1-3 shows a hierarchy of latency among the different types of media where data
can reside at any given time in a program.

6
Chapter 1 Introduction to Persistent Memory Programming

Figure 1-3. The memory/storage hierarchy pyramid with estimated latencies

As the pyramid shows, persistent memory provides latencies similar to memory,


measured in nanoseconds, while providing persistency. Block storage provides
persistency with latencies starting in the microseconds and increasing from there,
depending on the technology. Persistent memory is unique in its ability to act like both
memory and storage at the same time.

Program Complexity
Perhaps the most important point of our example is that the programmer still uses
the familiar get/put interfaces normally associated with key-value stores. The fact that
the data structures are in persistent memory is abstracted away by the high-level API
provided by libpmemkv. This principle of using the highest level of abstraction possible,
as long as it meets the application’s needs, will be a recurring theme throughout this
book. We start by introducing very high-level APIs; later chapters delve into the lower-­
level details for programmers who need them. At the lowest level, programming directly
to raw persistent memory requires detailed knowledge of things like hardware atomicity,
cache flushing, and transactions. High-level libraries like libpmemkv abstract away all
that complexity and provide much simpler, less error-prone interfaces.

7
Chapter 1 Introduction to Persistent Memory Programming

How Does libpmemkv Work?


All the complexity hidden by high-level libraries like libpmemkv are described more fully
in later chapters, but let’s look at the building blocks used to construct a library like this.
Figure 1-4 shows the full software stack involved when an application uses libpmemkv.

Figure 1-4. The software stack when using libpmemkv

Starting from the bottom of Figure 1-4 and working upward are these components:

• The persistent memory hardware, typically connected to the system


memory bus and accessed using common memory load/store
operations.

• A pmem-aware file system, which is a kernel module that exposes


persistent memory to applications as files. Those files can be memory
mapped to give applications direct access (abbreviated as DAX).
This method of exposing persistent memory was published by SNIA
(Storage Networking Industry Association) and is described in detail
in Chapter 3.

• The libpmem library is part of the PMDK. This library abstracts


away some of the low-level hardware details like cache flushing
instructions.

8
Chapter 1 Introduction to Persistent Memory Programming

• The libpmemobj library is a full-featured transaction and allocation


library for persistent memory. (Chapters 7 and 8 describe libpmemobj
and its C++ cousin in more detail.) If you cannot find data structures
that meet your needs, you will most likely have to implement what
you need using this library, as described in Chapter 11.

• The cmap engine, a concurrent hash map optimized for persistent


memory.
• The libpmemkv library, providing the API demonstrated in Listing 1-1.

• And finally, the application that uses the API provided by libpmemkv.

Although there is quite a stack of components in use here, it does not mean there
is necessarily a large amount of code that runs for each operation. Some components
are only used during the initial setup. For example, the pmem-aware file system is
used to find the persistent memory file and perform permission checks; it is out of the
application’s data path after that. The PMDK libraries are designed to leverage the direct
access allowed by persistent memory as much as possible.

W
 hat’s Next?
Chapters 1 through 3 provide the essential background that programmers need to know to
start persistent memory programming. The stage is now set with a simple example; the next
two chapters provide details about persistent memory at the hardware and operating system
levels. The later and more advanced chapters provide much more detail for those interested.
Because the immediate goal is to get you programming quickly, we recommend
reading Chapters 2 and 3 to gain the essential background and then dive into Chapter 4
where we start to show more detailed persistent memory programming examples.

S
 ummary
This chapter shows how high-level APIs like libpmemkv can be used for persistent
memory programming, hiding complex details of persistent memory from the
application developer. Using persistent memory can allow finer-grained access and
higher performance than block-based storage. We recommend using the highest-level,
simplest APIs possible and only introducing the complexity of lower-level persistent
memory programming as necessary.
9
Chapter 1 Introduction to Persistent Memory Programming

Open Access This chapter is licensed under the terms of the Creative
Commons Attribution 4.0 International License (http://creativecommons.
org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and
reproduction in any medium or format, as long as you give appropriate credit to the
original author(s) and the source, provide a link to the Creative Commons license and
indicate if changes were made.
The images or other third party material in this chapter are included in the chapter’s
Creative Commons license, unless indicated otherwise in a credit line to the material. If
material is not included in the chapter’s Creative Commons license and your intended
use is not permitted by statutory regulation or exceeds the permitted use, you will need
to obtain permission directly from the copyright holder.

10
CHAPTER 2

Persistent Memory
Architecture
This chapter provides an overview of the persistent memory architecture while focusing
on the hardware to emphasize requirements and decisions that developers need to know.
Applications that are designed to recognize the presence of persistent memory in
a system can run much faster than using other storage devices because data does not
have to transfer back and forth between the CPU and slower storage devices. Because
applications that only use persistent memory may be slower than dynamic random-­
access memory (DRAM), they should decide what data resides in DRAM, persistent
memory, and storage.
The capacity of persistent memory is expected to be many times larger than DRAM;
thus, the volume of data that applications can potentially store and process in place is
also much larger. This significantly reduces the number of disk I/Os, which improves
performance and reduces wear on the storage media.
On systems without persistent memory, large datasets that cannot fit into DRAM
must be processed in segments or streamed. This introduces processing delays as the
application stalls waiting for data to be paged from disk or streamed from the network.
If the working dataset size fits within the capacity of persistent memory and DRAM,
applications can perform in-memory processing without needing to checkpoint or page
data to or from storage. This significantly improves performance.

11
© The Author(s) 2020
S. Scargall, Programming Persistent Memory, https://doi.org/10.1007/978-1-4842-4932-1_2
Chapter 2 Persistent Memory Architecture

Persistent Memory Characteristics


As with every new technology, there are always new things to consider. Persistent
memory is no exception. Consider these characteristics when architecting and
developing solutions:

• Performance (throughput, latency, and bandwidth) of persistent


memory is much better than NAND but potentially slower than
DRAM.

• Persistent memory is durable unlike DRAM. Its endurance is usually


orders of magnitude better than NAND and should exceed the
lifetime of the server without wearing out.

• Persistent memory module capacities can be much larger than


DRAM DIMMs and can coexist on the same memory channels.

• Persistent memory-enabled applications can update data in place


without needing to serialize/deserialize the data.

• Persistent memory is byte addressable like memory. Applications


can update only the data needed without any read-modify-write
overhead.

• Data is CPU cache coherent.

• Persistent memory provides direct memory access (DMA) and


remote DMA (RDMA) operations.

• Data written to persistent memory is not lost when power is removed.

• After permission checks are completed, data located on persistent


memory is directly accessible from user space. No kernel code, file
system page caches, or interrupts are in the data path.

12
Chapter 2 Persistent Memory Architecture

• Data on persistent memory is instantly available, that is:

• Data is available as soon as power is applied to the system.

• Applications do not need to spend time warming up caches. They


can access the data immediately upon memory mapping it.

• Data residing on persistent memory has no DRAM footprint


unless the application copies data to DRAM for faster access.

• Data written to persistent memory modules is local to the system.


Applications are responsible for replicating data across systems.

Platform Support for Persistent Memory


Platform vendors such as Intel, AMD, ARM, and others will decide how persistent
memory should be implemented at the lowest hardware levels. We try to provide a
vendor-agnostic perspective and only occasionally call out platform-specific details.
For systems with persistent memory, failure atomicity guarantees that systems can
always recover to a consistent state following a power or system failure. Failure atomicity
for applications can be achieved using logging, flushing, and memory store barriers that
order such operations. Logging, either undo or redo, ensures atomicity when a failure
interrupts the last atomic operation from completion. Cache flushing ensures that
data held within volatile caches reach the persistence domain so it will not be lost if a
sudden failure occurs. Memory store barriers, such as an SFENCE operation on the x86
architecture, help prevent potential reordering in the memory hierarchy, as caches and
memory controllers may reorder memory operations. For example, a barrier ensures
that the undo log copy of the data gets persisted onto the persistent memory before the
actual data is modified in place. This guarantees that the last atomic operation can be
rolled back should a failure occur. However, it is nontrivial to add such failure atomicity
in user applications with low-level operations such as write logging, cache flushing, and
barriers. The Persistent Memory Development Kit (PMDK) was developed to isolate
developers from having to re-implement the hardware intricacies.
Failure atomicity should be a familiar concept, since most file systems implement
and perform journaling and flushing of their metadata to storage devices.

13
Chapter 2 Persistent Memory Architecture

C
 ache Hierarchy
We use load and store operations to read and write to persistent memory rather than
using block-based I/O to read and write to traditional storage. We suggest reading the
CPU architecture documentation for an in-depth description because each successive
CPU generation may introduce new features, methods, and optimizations.
Using the Intel architecture as an example, a CPU cache typically has three
distinct levels: L1, L2, and L3. The hierarchy makes references to the distance
from the CPU core, its speed, and size of the cache. The L1 cache is closest to
the CPU. It is extremely fast but very small. L2 and L3 caches are increasingly
larger in capacity, but they are relatively slower. Figure 2-1 shows a typical CPU
microarchitecture with three levels of CPU cache and a memory controller with
three memory channels. Each memory channel has a single DRAM and persistent
memory attached. On platforms where the CPU caches are not contained within
the power-fail protected domain, any modified data within the CPU caches that has
not been flushed to persistent memory will be lost when the system loses power or
crashes. Platforms that do include CPU caches in the power-fail protected domain
will ensure modified data within the CPU caches are flushed to the persistent
memory should the system crash or loses power. We describe these requirements
and features in the upcoming “Power-Fail Protected Domains” section.

14
Chapter 2 Persistent Memory Architecture

Figure 2-1. CPU cache and memory hierarchy

The L1 (Level 1) cache is the fastest memory in a computer system. In terms of access
priority, the L1 cache has the data the CPU is most likely to need while completing a
specific task. The L1 cache is also usually split two ways, into the instruction cache (L1 I)
and the data cache (L1 D). The instruction cache deals with the information about the
operation that the CPU has to perform, while the data cache holds the data on which the
operation is to be performed.
The L2 (Level 2) cache has a larger capacity than the L1 cache, but it is slower. L2
cache holds data that is likely to be accessed by the CPU next. In most modern CPUs,
the L1 and L2 caches are present on the CPU cores themselves, with each core getting
dedicated caches.
The L3 (Level 3) cache is the largest cache memory, but it is also the slowest of the
three. It is also a commonly shared resource among all the cores on the CPU and may be
internally partitioned to allow each core to have dedicated L3 resources.
Data read from DRAM or persistent memory is transferred through the memory
controller into the L3 cache, then propagated into the L2 cache, and finally the L1 cache
where the CPU core consumes it. When the processor is looking for data to carry out an
operation, it first tries to find it into the L1 cache. If the CPU can find it, the condition is
called a cache hit. If the CPU cannot find the data within the L1 cache, it then proceeds to

15
Chapter 2 Persistent Memory Architecture

search for it first within L2, then L3. If it cannot find the data in any of the three, it tries to
access it from memory. Each failure to find data in a cache is called a cache miss. Failure
to locate the data in memory requires the operating system to page the data into memory
from a storage device.
When the CPU writes data, it is initially written to the L1 cache. Due to ongoing
activity within the CPU, at some point in time, the data will be evicted from the L1 cache
into the L2 cache. The data may be further evicted from L2 and placed into L3 and
eventually evicted from L3 into the memory controller’s write buffers where it is then
written to the memory device.
In a system that does not possess persistent memory, software persists data by writing it
to a non-volatile storage device such as an SSD, HDD, SAN, NAS, or a volume in the cloud.
This protects data from application or system crashes. Critical data can be manually flushed
using calls such as msync(), fsync(), or fdatasync(), which flush uncommitted dirty
pages from volatile memory to the non-volatile storage device. File systems provide fdisk
or chkdsk utilities to check and attempt repairs on damaged file systems if required. File
systems do not protect user data from torn blocks. Applications have a responsibility to
detect and recovery from this situation. That’s why databases, for example, use a variety of
techniques such as transactional updates, redo/undo logging, and checksums.
Applications memory map the persistent memory address range directly into its
own memory address space. Therefore, the application must assume responsibility
for checking and guaranteeing data integrity. The rest of this chapter describes
your responsibilities in a persistent memory environment and how to achieve data
consistency and integrity.

Power-Fail Protected Domains


A computer system may include one or more CPUs, volatile or persistent memory
modules, and non-volatile storage devices such as SSDs or HDDs.
System platform hardware supports the concept of a persistence domain, also called
power-fail protected domains. Depending on the platform, a persistence domain may
include the persistent memory controller and write queues, memory controller write
queues, and CPU caches. Once data has reached the persistence domain, it may be
recoverable during a process that results from a system restart. That is, if data is located
within hardware write queues or buffers protected by power failure, domain applications
should assume it is persistent. For example, if a power failure occurs, the data will be flushed

16
Chapter 2 Persistent Memory Architecture

from the power-fail protected domain using stored energy guaranteed by the platform for
this purpose. Data that has not yet made it into the protected domain will be lost.
Multiple persistence domains may exist within the same system, for example, on
systems with more than one physical CPU. Systems may also provide a mechanism for
partitioning the platform resources for isolation. This must be done in such a way that
SNIA NVM programming model behavior is assured from each compliant volume or file
system. (Chapter 3 describes the programming model as it applies to operating systems
and file systems. The “Detecting Platform Capabilities” section in that chapter describes
the logic that applications should perform to detect platform capabilities including
power failure protected domains. Later chapters provide in-depth discussions into why,
how, and when applications should flush data, if required, to guarantee the data is safe
within the protected domain and persistent memory.)
Volatile memory loses its contents when the computer system’s power is interrupted.
Just like non-volatile storage devices, persistent memory keeps its contents even in the
absence of system power. Data that has been physically saved to the persistent memory
media is called data at rest. Data in-flight has the following meanings:

• Writes sent to the persistent memory device but have not yet been
physically committed to the media

• Any writes that are in progress but not yet complete

• Data that has been temporarily buffered or cached in either the CPU
caches or memory controller

When a system is gracefully rebooted or shut down, the system maintains power
and can ensure all contents of the CPU caches and memory controllers are flushed such
that any in-flight or uncommitted data is successfully written to persistent memory
or non-volatile storage. When an unexpected power failure occurs, and assuming no
uninterruptable power supply (UPS) is available, the system must have enough stored
energy within the power supplies and capacitors dotted around it to flush data before the
power is completely exhausted. Any data that is not flushed is lost and not recoverable.
Asynchronous DRAM Refresh (ADR) is a feature supported on Intel products which
flushes the write-protected data buffers and places the DRAM in self-refresh. This
process is critical during a power loss event or system crash to ensure the data is in a safe
and consistent state on persistent memory. By default, ADR does not flush the processor
caches. A platform that supports ADR only includes persistent memory and the memory
controller’s write pending queues within the persistence domain. This is the reason

17
Chapter 2 Persistent Memory Architecture

data in the CPU caches must be flushed by the application using the CLWB, CLFLUSHOPT,
CLFLUSH, non-temporal stores, or WBINVD machine instructions.
Enhanced Asynchronous DRAM Refresh (eADR) requires that a non-maskable
interrupt (NMI) routine be called to flush the CPU caches before the ADR event can begin.
Applications running on an eADR platform do not need to perform flush operations
because the hardware should flush the data automatically, but they are still required
to perform an SFENCE operation to maintain write order correctness. Stores should be
considered persistent only when they are globally visible, which the SFENCE guarantees.
Figure 2-2 shows both the ADR and eADR persistence domains.

Figure 2-2. ADR and eADR power-fail protection domains

ADR is a mandatory platform requirement for persistent memory. The write


pending queue (WPQ) within the memory controller acknowledges receipt of the data
to the writer once all the data is received. Although the data has not yet made it to the
persistent media, a platform supporting ADR guarantees that it will be successfully
written should a power loss event occur. During a crash or power failure, data that is in-­
flight through the CPU caches can only be guaranteed to be flushed to persistent media
if the platform supports eADR. It will be lost on platforms that only support ADR.
The challenge with extending the persistence domain to include the CPU caches is
that the CPU caches are quite large and it would take considerably more energy than the
capacitors in a typical power supply can practically provide. This means the platform
would have to contain batteries or utilize an external uninterruptable power supply.
Requiring a battery for every server supporting persistent memory is not generally
practical or cost-effective. The lifetime of a battery is typically shorter than the server,

18
Chapter 2 Persistent Memory Architecture

which introduces additional maintenance routines that reduce server uptime. There
is also an environmental impact when using batteries as they must be disposed of
or recycled correctly. It is entirely possible for server or appliance OEMs to include a
battery in their product.
Because some appliance and server vendors plan to use batteries, and because
platforms will someday include the CPU caches in the persistence domain, a property is
available within ACPI such that the BIOS can notify software when the CPU flushes can
be skipped. On platforms with eADR, there is no need for manual cache line flushing.

The Need for Flushing, Ordering, and Fencing


Except for WBINVD, which is a kernel-mode-only operation, the machine instructions
in Table 2-1 (in the “Intel Machine Instructions for Persistent Memory” section)
are supported in user space by Intel and AMD CPUs. Intel adopted the SNIA NVM
programming model for working with persistent memory. This model allows for
direct access (DAX) using byte-addressable operations (i.e., load/store). However, the
persistence of the data in the cache is not guaranteed until it has entered the persistence
domain. The x86 architecture provides a set of instructions for flushing cache lines in
a more optimized way. In addition to existing x86 instructions, such as non-temporal
stores, CLFLUSH, and WBINVD, two new instructions were added: CLFLUSHOPT and
CLWB. Both new instructions must be followed by an SFENCE to ensure all flushes are
completed before continuing. Flushing a cache line using CLWB, CLFLUSHOPT, or CLFLUSH
and using non-temporal stores are all supported from user space. You can find details
for each machine instruction in the software developer manuals for the architecture.
On Intel platforms, for example, this information can be found in the Intel 64 and 32
Architectures Software Developer Manuals (https://software.intel.com/en-us/
articles/intel-sdm).
Non-temporal stores imply that the data being written is not going to be read again
soon, so we bypass the CPU caches. That is, there is no temporal locality, so there is no
benefit to keeping the data in the processor’s cache(s), and there may be a penalty if the
stored data displaces other useful data from the cache(s).
Flushing to persistent memory directly from user space negates calling into the
kernel, which makes it highly efficient. The feature is documented in the SNIA persistent
memory programming model specification as an optimized flush. The specification

19
Chapter 2 Persistent Memory Architecture

document1 describes optimized flush as optionally supported by the platform,


depending on the hardware and operating system support. Despite the CPU support,
it is essential for applications to use only optimized flushes when the operating system
indicates that it is safe to use. The operating system may require the control point
provided by calls like msync() when, for example, there are changes to file system
metadata that need to be written as part of the msync() operation.
To better understand instruction ordering, consider a very simple linked list
example. Our pseudocode described in the following has three simple steps to add a new
node into an existing list that already contains two nodes. These steps are depicted in
Figure 2-3.

1. Create the new node (Node 2).

2. Update the node pointer (next pointer) to point to the last node in
the list (Node 2 → Node 1).

3. Update the head pointer to point at the new node (Head → Node 2).

Figure 2-3 (Step 3) shows that the head pointer was updated in the CPU cached version,
but the Node 2 to Node 1 pointer has not yet been updated in persistent memory. This
is because the hardware can choose which cache lines to commit and the order may not
match the source code flow. If the system or application were to crash at this point, the
persistent memory state would be inconsistent, and the data structure would no longer
be usable.

1
 NIA NVM programming model spec: https://www.snia.org/tech_activities/standards/
S
curr_standards/npm

20
Chapter 2 Persistent Memory Architecture

Figure 2-3. Adding a new node to an existing linked list without a store barrier

To solve this problem, we introduce a memory store barrier to ensure the order of the
write operations is maintained. Starting from the same initial state, the pseudocode now
looks like this:

1. Create the new node.

2. Update the node pointer (next pointer) to point to the last node in
the list, and perform a store barrier/fence operation.
3. Update the head pointer to point at the new node.

Figure 2-4 shows that the addition of the store barrier allows the code to work as
expected and maintains a consistent data structure in the volatile CPU caches and on

21
Discovering Diverse Content Through
Random Scribd Documents
1.C. The Project Gutenberg Literary Archive Foundation (“the
Foundation” or PGLAF), owns a compilation copyright in the
collection of Project Gutenberg™ electronic works. Nearly all the
individual works in the collection are in the public domain in the
United States. If an individual work is unprotected by copyright
law in the United States and you are located in the United
States, we do not claim a right to prevent you from copying,
distributing, performing, displaying or creating derivative works
based on the work as long as all references to Project
Gutenberg are removed. Of course, we hope that you will
support the Project Gutenberg™ mission of promoting free
access to electronic works by freely sharing Project Gutenberg™
works in compliance with the terms of this agreement for
keeping the Project Gutenberg™ name associated with the
work. You can easily comply with the terms of this agreement
by keeping this work in the same format with its attached full
Project Gutenberg™ License when you share it without charge
with others.

1.D. The copyright laws of the place where you are located also
govern what you can do with this work. Copyright laws in most
countries are in a constant state of change. If you are outside
the United States, check the laws of your country in addition to
the terms of this agreement before downloading, copying,
displaying, performing, distributing or creating derivative works
based on this work or any other Project Gutenberg™ work. The
Foundation makes no representations concerning the copyright
status of any work in any country other than the United States.

1.E. Unless you have removed all references to Project


Gutenberg:

1.E.1. The following sentence, with active links to, or other


immediate access to, the full Project Gutenberg™ License must
appear prominently whenever any copy of a Project
Gutenberg™ work (any work on which the phrase “Project
Gutenberg” appears, or with which the phrase “Project
Gutenberg” is associated) is accessed, displayed, performed,
viewed, copied or distributed:

This eBook is for the use of anyone anywhere


in the United States and most other parts of
the world at no cost and with almost no
restrictions whatsoever. You may copy it, give it
away or re-use it under the terms of the
Project Gutenberg License included with this
eBook or online at www.gutenberg.org. If you
are not located in the United States, you will
have to check the laws of the country where
you are located before using this eBook.

1.E.2. If an individual Project Gutenberg™ electronic work is


derived from texts not protected by U.S. copyright law (does not
contain a notice indicating that it is posted with permission of
the copyright holder), the work can be copied and distributed to
anyone in the United States without paying any fees or charges.
If you are redistributing or providing access to a work with the
phrase “Project Gutenberg” associated with or appearing on the
work, you must comply either with the requirements of
paragraphs 1.E.1 through 1.E.7 or obtain permission for the use
of the work and the Project Gutenberg™ trademark as set forth
in paragraphs 1.E.8 or 1.E.9.

1.E.3. If an individual Project Gutenberg™ electronic work is


posted with the permission of the copyright holder, your use and
distribution must comply with both paragraphs 1.E.1 through
1.E.7 and any additional terms imposed by the copyright holder.
Additional terms will be linked to the Project Gutenberg™
License for all works posted with the permission of the copyright
holder found at the beginning of this work.
1.E.4. Do not unlink or detach or remove the full Project
Gutenberg™ License terms from this work, or any files
containing a part of this work or any other work associated with
Project Gutenberg™.

1.E.5. Do not copy, display, perform, distribute or redistribute


this electronic work, or any part of this electronic work, without
prominently displaying the sentence set forth in paragraph 1.E.1
with active links or immediate access to the full terms of the
Project Gutenberg™ License.

1.E.6. You may convert to and distribute this work in any binary,
compressed, marked up, nonproprietary or proprietary form,
including any word processing or hypertext form. However, if
you provide access to or distribute copies of a Project
Gutenberg™ work in a format other than “Plain Vanilla ASCII” or
other format used in the official version posted on the official
Project Gutenberg™ website (www.gutenberg.org), you must,
at no additional cost, fee or expense to the user, provide a copy,
a means of exporting a copy, or a means of obtaining a copy
upon request, of the work in its original “Plain Vanilla ASCII” or
other form. Any alternate format must include the full Project
Gutenberg™ License as specified in paragraph 1.E.1.

1.E.7. Do not charge a fee for access to, viewing, displaying,


performing, copying or distributing any Project Gutenberg™
works unless you comply with paragraph 1.E.8 or 1.E.9.

1.E.8. You may charge a reasonable fee for copies of or


providing access to or distributing Project Gutenberg™
electronic works provided that:

• You pay a royalty fee of 20% of the gross profits you


derive from the use of Project Gutenberg™ works
calculated using the method you already use to
calculate your applicable taxes. The fee is owed to the
owner of the Project Gutenberg™ trademark, but he has
agreed to donate royalties under this paragraph to the
Project Gutenberg Literary Archive Foundation. Royalty
payments must be paid within 60 days following each
date on which you prepare (or are legally required to
prepare) your periodic tax returns. Royalty payments
should be clearly marked as such and sent to the
Project Gutenberg Literary Archive Foundation at the
address specified in Section 4, “Information about
donations to the Project Gutenberg Literary Archive
Foundation.”

• You provide a full refund of any money paid by a user


who notifies you in writing (or by e-mail) within 30 days
of receipt that s/he does not agree to the terms of the
full Project Gutenberg™ License. You must require such
a user to return or destroy all copies of the works
possessed in a physical medium and discontinue all use
of and all access to other copies of Project Gutenberg™
works.

• You provide, in accordance with paragraph 1.F.3, a full


refund of any money paid for a work or a replacement
copy, if a defect in the electronic work is discovered and
reported to you within 90 days of receipt of the work.

• You comply with all other terms of this agreement for


free distribution of Project Gutenberg™ works.

1.E.9. If you wish to charge a fee or distribute a Project


Gutenberg™ electronic work or group of works on different
terms than are set forth in this agreement, you must obtain
permission in writing from the Project Gutenberg Literary
Archive Foundation, the manager of the Project Gutenberg™
trademark. Contact the Foundation as set forth in Section 3
below.
1.F.

1.F.1. Project Gutenberg volunteers and employees expend


considerable effort to identify, do copyright research on,
transcribe and proofread works not protected by U.S. copyright
law in creating the Project Gutenberg™ collection. Despite these
efforts, Project Gutenberg™ electronic works, and the medium
on which they may be stored, may contain “Defects,” such as,
but not limited to, incomplete, inaccurate or corrupt data,
transcription errors, a copyright or other intellectual property
infringement, a defective or damaged disk or other medium, a
computer virus, or computer codes that damage or cannot be
read by your equipment.

1.F.2. LIMITED WARRANTY, DISCLAIMER OF DAMAGES - Except


for the “Right of Replacement or Refund” described in
paragraph 1.F.3, the Project Gutenberg Literary Archive
Foundation, the owner of the Project Gutenberg™ trademark,
and any other party distributing a Project Gutenberg™ electronic
work under this agreement, disclaim all liability to you for
damages, costs and expenses, including legal fees. YOU AGREE
THAT YOU HAVE NO REMEDIES FOR NEGLIGENCE, STRICT
LIABILITY, BREACH OF WARRANTY OR BREACH OF CONTRACT
EXCEPT THOSE PROVIDED IN PARAGRAPH 1.F.3. YOU AGREE
THAT THE FOUNDATION, THE TRADEMARK OWNER, AND ANY
DISTRIBUTOR UNDER THIS AGREEMENT WILL NOT BE LIABLE
TO YOU FOR ACTUAL, DIRECT, INDIRECT, CONSEQUENTIAL,
PUNITIVE OR INCIDENTAL DAMAGES EVEN IF YOU GIVE
NOTICE OF THE POSSIBILITY OF SUCH DAMAGE.

1.F.3. LIMITED RIGHT OF REPLACEMENT OR REFUND - If you


discover a defect in this electronic work within 90 days of
receiving it, you can receive a refund of the money (if any) you
paid for it by sending a written explanation to the person you
received the work from. If you received the work on a physical
medium, you must return the medium with your written
explanation. The person or entity that provided you with the
defective work may elect to provide a replacement copy in lieu
of a refund. If you received the work electronically, the person
or entity providing it to you may choose to give you a second
opportunity to receive the work electronically in lieu of a refund.
If the second copy is also defective, you may demand a refund
in writing without further opportunities to fix the problem.

1.F.4. Except for the limited right of replacement or refund set


forth in paragraph 1.F.3, this work is provided to you ‘AS-IS’,
WITH NO OTHER WARRANTIES OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO WARRANTIES OF
MERCHANTABILITY OR FITNESS FOR ANY PURPOSE.

1.F.5. Some states do not allow disclaimers of certain implied


warranties or the exclusion or limitation of certain types of
damages. If any disclaimer or limitation set forth in this
agreement violates the law of the state applicable to this
agreement, the agreement shall be interpreted to make the
maximum disclaimer or limitation permitted by the applicable
state law. The invalidity or unenforceability of any provision of
this agreement shall not void the remaining provisions.

1.F.6. INDEMNITY - You agree to indemnify and hold the


Foundation, the trademark owner, any agent or employee of the
Foundation, anyone providing copies of Project Gutenberg™
electronic works in accordance with this agreement, and any
volunteers associated with the production, promotion and
distribution of Project Gutenberg™ electronic works, harmless
from all liability, costs and expenses, including legal fees, that
arise directly or indirectly from any of the following which you
do or cause to occur: (a) distribution of this or any Project
Gutenberg™ work, (b) alteration, modification, or additions or
deletions to any Project Gutenberg™ work, and (c) any Defect
you cause.
Section 2. Information about the Mission
of Project Gutenberg™
Project Gutenberg™ is synonymous with the free distribution of
electronic works in formats readable by the widest variety of
computers including obsolete, old, middle-aged and new
computers. It exists because of the efforts of hundreds of
volunteers and donations from people in all walks of life.

Volunteers and financial support to provide volunteers with the


assistance they need are critical to reaching Project
Gutenberg™’s goals and ensuring that the Project Gutenberg™
collection will remain freely available for generations to come. In
2001, the Project Gutenberg Literary Archive Foundation was
created to provide a secure and permanent future for Project
Gutenberg™ and future generations. To learn more about the
Project Gutenberg Literary Archive Foundation and how your
efforts and donations can help, see Sections 3 and 4 and the
Foundation information page at www.gutenberg.org.

Section 3. Information about the Project


Gutenberg Literary Archive Foundation
The Project Gutenberg Literary Archive Foundation is a non-
profit 501(c)(3) educational corporation organized under the
laws of the state of Mississippi and granted tax exempt status
by the Internal Revenue Service. The Foundation’s EIN or
federal tax identification number is 64-6221541. Contributions
to the Project Gutenberg Literary Archive Foundation are tax
deductible to the full extent permitted by U.S. federal laws and
your state’s laws.

The Foundation’s business office is located at 809 North 1500


West, Salt Lake City, UT 84116, (801) 596-1887. Email contact
links and up to date contact information can be found at the
Foundation’s website and official page at
www.gutenberg.org/contact

Section 4. Information about Donations to


the Project Gutenberg Literary Archive
Foundation
Project Gutenberg™ depends upon and cannot survive without
widespread public support and donations to carry out its mission
of increasing the number of public domain and licensed works
that can be freely distributed in machine-readable form
accessible by the widest array of equipment including outdated
equipment. Many small donations ($1 to $5,000) are particularly
important to maintaining tax exempt status with the IRS.

The Foundation is committed to complying with the laws


regulating charities and charitable donations in all 50 states of
the United States. Compliance requirements are not uniform
and it takes a considerable effort, much paperwork and many
fees to meet and keep up with these requirements. We do not
solicit donations in locations where we have not received written
confirmation of compliance. To SEND DONATIONS or determine
the status of compliance for any particular state visit
www.gutenberg.org/donate.

While we cannot and do not solicit contributions from states


where we have not met the solicitation requirements, we know
of no prohibition against accepting unsolicited donations from
donors in such states who approach us with offers to donate.

International donations are gratefully accepted, but we cannot


make any statements concerning tax treatment of donations
received from outside the United States. U.S. laws alone swamp
our small staff.
Please check the Project Gutenberg web pages for current
donation methods and addresses. Donations are accepted in a
number of other ways including checks, online payments and
credit card donations. To donate, please visit:
www.gutenberg.org/donate.

Section 5. General Information About


Project Gutenberg™ electronic works
Professor Michael S. Hart was the originator of the Project
Gutenberg™ concept of a library of electronic works that could
be freely shared with anyone. For forty years, he produced and
distributed Project Gutenberg™ eBooks with only a loose
network of volunteer support.

Project Gutenberg™ eBooks are often created from several


printed editions, all of which are confirmed as not protected by
copyright in the U.S. unless a copyright notice is included. Thus,
we do not necessarily keep eBooks in compliance with any
particular paper edition.

Most people start at our website which has the main PG search
facility: www.gutenberg.org.

This website includes information about Project Gutenberg™,


including how to make donations to the Project Gutenberg
Literary Archive Foundation, how to help produce our new
eBooks, and how to subscribe to our email newsletter to hear
about new eBooks.
Welcome to our website – the perfect destination for book lovers and
knowledge seekers. We believe that every book holds a new world,
offering opportunities for learning, discovery, and personal growth.
That’s why we are dedicated to bringing you a diverse collection of
books, ranging from classic literature and specialized publications to
self-development guides and children's books.

More than just a book-buying platform, we strive to be a bridge


connecting you with timeless cultural and intellectual values. With an
elegant, user-friendly interface and a smart search system, you can
quickly find the books that best suit your interests. Additionally,
our special promotions and home delivery services help you save time
and fully enjoy the joy of reading.

Join us on a journey of knowledge exploration, passion nurturing, and


personal growth every day!

ebookbell.com

You might also like