100% found this document useful (1 vote)
2 views

Programming Persistent Memory: A Comprehensive Guide For Developers 1st Edition Steve Scargall instant download

The document is a comprehensive guide for developers on programming with persistent memory, authored by Steve Scargall. It covers various aspects including persistent memory architecture, operating system support, and fundamental programming concepts, along with practical libraries and tools. The guide aims to equip developers with the knowledge and tools necessary for effective persistent memory programming.

Uploaded by

gallireskars
Copyright
© © All Rights Reserved
Available Formats
Download as PDF, TXT or read online on Scribd
100% found this document useful (1 vote)
2 views

Programming Persistent Memory: A Comprehensive Guide For Developers 1st Edition Steve Scargall instant download

The document is a comprehensive guide for developers on programming with persistent memory, authored by Steve Scargall. It covers various aspects including persistent memory architecture, operating system support, and fundamental programming concepts, along with practical libraries and tools. The guide aims to equip developers with the knowledge and tools necessary for effective persistent memory programming.

Uploaded by

gallireskars
Copyright
© © All Rights Reserved
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 65

Programming Persistent Memory: A Comprehensive

Guide For Developers 1st Edition Steve Scargall


pdf download

https://textbookfull.com/product/programming-persistent-memory-a-
comprehensive-guide-for-developers-1st-edition-steve-scargall/

Download more ebook from https://textbookfull.com


We believe these products will be a great fit for you. Click
the link to download now, or visit textbookfull.com
to discover even more!

Camping Washington A Comprehensive Guide to Public Tent


and RV Campgrounds 3rd Edition Steve Giordano

https://textbookfull.com/product/camping-washington-a-
comprehensive-guide-to-public-tent-and-rv-campgrounds-3rd-
edition-steve-giordano/

Scala Programming A comprehensive beginner s guide to


Scala 2nd Edition Claudia Alves

https://textbookfull.com/product/scala-programming-a-
comprehensive-beginner-s-guide-to-scala-2nd-edition-claudia-
alves/

The Economics of Education: A Comprehensive Overview


2nd Edition Steve Bradley (Editor)

https://textbookfull.com/product/the-economics-of-education-a-
comprehensive-overview-2nd-edition-steve-bradley-editor/

Quick Boot A Guide for Embedded Firmware Developers 2nd


Edition Pete Dice

https://textbookfull.com/product/quick-boot-a-guide-for-embedded-
firmware-developers-2nd-edition-pete-dice/
The Rust Programming Language 1st Edition Steve Klabnik

https://textbookfull.com/product/the-rust-programming-
language-1st-edition-steve-klabnik/

Building Generative AI Powered Apps A Hands on Guide


for Developers 1st Edition Kansal

https://textbookfull.com/product/building-generative-ai-powered-
apps-a-hands-on-guide-for-developers-1st-edition-kansal/

Reservoir Modelling A Practical Guide 1st Edition Steve


Cannon

https://textbookfull.com/product/reservoir-modelling-a-practical-
guide-1st-edition-steve-cannon/

Linux for Developers Jumpstart Your Linux Programming


Skills William "Bo" Rothwell

https://textbookfull.com/product/linux-for-developers-jumpstart-
your-linux-programming-skills-william-bo-rothwell/

Executive Functioning: A Comprehensive Guide for


Clinical Practice 1st Edition Yana Suchy

https://textbookfull.com/product/executive-functioning-a-
comprehensive-guide-for-clinical-practice-1st-edition-yana-suchy/
Programming
Persistent
Memory
A Comprehensive Guide for Developers

Steve Scargall
Programming Persistent
Memory
A Comprehensive Guide for
Developers

Steve Scargall
Programming Persistent Memory: A Comprehensive Guide for Developers

Steve Scargall
Santa Clara, CA, USA

ISBN-13 (pbk): 978-1-4842-4931-4 ISBN-13 (electronic): 978-1-4842-4932-1


https://doi.org/10.1007/978-1-4842-4932-1
Copyright © 2020 by Intel
This work is subject to copyright. All rights are reserved by the Publisher, whether the whole or part of the material is
concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction
on microfilms or in any other physical way, and transmission or information storage and retrieval, electronic
adaptation, computer software, or by similar or dissimilar methodology now known or hereafter developed.
Open Access This book is licensed under the terms of the Creative Commons Attribution 4.0
International License (http://creativecommons.org/licenses/by/4.0/), which permits use,
sharing, adaptation, distribution and reproduction in any medium or format, as long as you give
appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and
indicate if changes were made.
The images or other third party material in this book are included in the book’s Creative Commons license, unless
indicated otherwise in a credit line to the material. If material is not included in the book’s Creative Commons
license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need
to obtain permission directly from the copyright holder.
Trademarked names, logos, and images may appear in this book. Rather than use a trademark symbol with every
occurrence of a trademarked name, logo, or image we use the names, logos, and images only in an editorial fashion
and to the benefit of the trademark owner, with no intention of infringement of the trademark.
The use in this publication of trade names, trademarks, service marks, and similar terms, even if they are not identified
as such, is not to be taken as an expression of opinion as to whether or not they are subject to proprietary rights.
While the advice and information in this book are believed to be true and accurate at the date of publication, neither
the authors nor the editors nor the publisher can accept any legal responsibility for any errors or omissions that may
be made. The publisher makes no warranty, express or implied, with respect to the material contained herein.
Managing Director, Apress Media LLC: Welmoed Spahr
Acquisitions Editor: Susan McDermott
Development Editor: Laura Berendson
Coordinating Editor: Jessica Vakili
Distributed to the book trade worldwide by Springer Science+Business Media New York, 233 Spring Street,
6th Floor, New York, NY 10013. Phone 1-800-SPRINGER, fax (201) 348-4505, e-mail orders-ny@springer-sbm.com,
or visit www.springeronline.com. Apress Media, LLC is a California LLC and the sole member (owner) is Springer
Science + Business Media Finance Inc (SSBM Finance Inc). SSBM Finance Inc is a Delaware corporation.
For information on translations, please e-mail rights@apress.com, or visit http://www.apress.com/
rights-permissions.
Apress titles may be purchased in bulk for academic, corporate, or promotional use. eBook versions and licenses
are also available for most titles. For more information, reference our Print and eBook Bulk Sales web page at
http://www.apress.com/bulk-sales.
Any source code or other supplementary material referenced by the author in this book is available to readers on
GitHub via the book's product page, located at www.apress.com/978-1-4842-4931-4. For more detailed
information, please visit http://www.apress.com/source-code.
Printed on acid-free paper
Table of Contents
About the Author��������������������������������������������������������������������������������������������������� xiii
About the Technical Reviewer���������������������������������������������������������������������������������xv
About the Contributors������������������������������������������������������������������������������������������xvii
Acknowledgments��������������������������������������������������������������������������������������������������xxi
Preface�����������������������������������������������������������������������������������������������������������������xxiii

Chapter 1: Introduction to Persistent Memory Programming���������������������������������� 1


A High-Level Example Program���������������������������������������������������������������������������������������������������� 2
What’s Different?��������������������������������������������������������������������������������������������������������������������� 5
The Performance Difference��������������������������������������������������������������������������������������������������� 6
Program Complexity���������������������������������������������������������������������������������������������������������������� 7
How Does libpmemkv Work?��������������������������������������������������������������������������������������������������� 8
What’s Next?��������������������������������������������������������������������������������������������������������������������������������� 9
Summary�������������������������������������������������������������������������������������������������������������������������������������� 9

Chapter 2: Persistent Memory Architecture����������������������������������������������������������� 11


Persistent Memory Characteristics��������������������������������������������������������������������������������������������� 12
Platform Support for Persistent Memory������������������������������������������������������������������������������������ 13
Cache Hierarchy�������������������������������������������������������������������������������������������������������������������������� 14
Power-Fail Protected Domains���������������������������������������������������������������������������������������������������� 16
The Need for Flushing, Ordering, and Fencing���������������������������������������������������������������������������� 19
Data Visibility������������������������������������������������������������������������������������������������������������������������������ 23
Intel Machine Instructions for Persistent Memory���������������������������������������������������������������������� 24
Detecting Platform Capabilities�������������������������������������������������������������������������������������������������� 25

iii
Table of Contents

Application Startup and Recovery����������������������������������������������������������������������������������������������� 27


What’s Next?������������������������������������������������������������������������������������������������������������������������������� 29
Summary������������������������������������������������������������������������������������������������������������������������������������ 29

Chapter 3: Operating System Support for Persistent Memory������������������������������� 31


Operating System Support for Memory and Storage������������������������������������������������������������������ 31
Persistent Memory As Block Storage������������������������������������������������������������������������������������������ 33
Persistent Memory-Aware File Systems������������������������������������������������������������������������������������� 34
Memory-Mapped Files���������������������������������������������������������������������������������������������������������������� 35
Persistent Memory Direct Access (DAX)������������������������������������������������������������������������������������� 43
Summary������������������������������������������������������������������������������������������������������������������������������������ 53

Chapter 4: Fundamental Concepts of Persistent Memory Programming��������������� 55


What’s Different?������������������������������������������������������������������������������������������������������������������������ 55
Atomic Updates��������������������������������������������������������������������������������������������������������������������������� 56
Transactions�������������������������������������������������������������������������������������������������������������������������������� 57
Atomicity������������������������������������������������������������������������������������������������������������������������������� 57
Consistency��������������������������������������������������������������������������������������������������������������������������� 58
Isolation��������������������������������������������������������������������������������������������������������������������������������� 58
Durability������������������������������������������������������������������������������������������������������������������������������� 58
Flushing Is Not Transactional������������������������������������������������������������������������������������������������������ 59
Start-Time Responsibilities��������������������������������������������������������������������������������������������������������� 59
Tuning for Hardware Configurations������������������������������������������������������������������������������������������� 60
Summary������������������������������������������������������������������������������������������������������������������������������������ 60

Chapter 5: Introducing the Persistent Memory Development Kit��������������������������� 63


Background��������������������������������������������������������������������������������������������������������������������������������� 63
Choosing the Right Semantics���������������������������������������������������������������������������������������������������� 64
Volatile Libraries������������������������������������������������������������������������������������������������������������������������� 65
libmemkind���������������������������������������������������������������������������������������������������������������������������� 65
libvmemcache����������������������������������������������������������������������������������������������������������������������� 66
libvmem��������������������������������������������������������������������������������������������������������������������������������� 67

iv
Table of Contents

Persistent Libraries��������������������������������������������������������������������������������������������������������������������� 67
libpmem�������������������������������������������������������������������������������������������������������������������������������� 67
libpmemobj���������������������������������������������������������������������������������������������������������������������������� 68
libpmemobj-cpp�������������������������������������������������������������������������������������������������������������������� 68
libpmemkv����������������������������������������������������������������������������������������������������������������������������� 69
libpmemlog���������������������������������������������������������������������������������������������������������������������������� 69
libpmemblk���������������������������������������������������������������������������������������������������������������������������� 69
Tools and Command Utilities������������������������������������������������������������������������������������������������������� 70
pmempool������������������������������������������������������������������������������������������������������������������������������ 70
pmemcheck��������������������������������������������������������������������������������������������������������������������������� 70
pmreorder������������������������������������������������������������������������������������������������������������������������������ 71
Summary������������������������������������������������������������������������������������������������������������������������������������ 71

Chapter 6: libpmem: Low-Level Persistent Memory Support��������������������������������� 73


Using the Library������������������������������������������������������������������������������������������������������������������������� 74
Mapping a File���������������������������������������������������������������������������������������������������������������������������� 75
Copying to Persistent Memory���������������������������������������������������������������������������������������������������� 76
Separating the Flush Steps��������������������������������������������������������������������������������������������������������� 77
Summary������������������������������������������������������������������������������������������������������������������������������������ 79

Chapter 7: libpmemobj: A Native Transactional Object Store��������������������������������� 81


What is libpmemobj?������������������������������������������������������������������������������������������������������������������ 81
Why not malloc( )?����������������������������������������������������������������������������������������������������������������������� 82
Grouping Operations������������������������������������������������������������������������������������������������������������������� 83
Memory Pools����������������������������������������������������������������������������������������������������������������������������� 83
Creating Memory Pools��������������������������������������������������������������������������������������������������������� 83
Pool Object Pointer (POP) and the Root Object���������������������������������������������������������������������� 87
Opening and Reading from Memory Pools���������������������������������������������������������������������������� 88
Memory Poolsets������������������������������������������������������������������������������������������������������������������������ 90
Concatenated Poolsets���������������������������������������������������������������������������������������������������������� 90
Replica Poolsets�������������������������������������������������������������������������������������������������������������������� 91
Managing Memory Pools and Poolsets��������������������������������������������������������������������������������������� 92

v
Table of Contents

Typed Object Identifiers (TOIDs)�������������������������������������������������������������������������������������������������� 92


Allocating Memory���������������������������������������������������������������������������������������������������������������������� 93
Persisting Data���������������������������������������������������������������������������������������������������������������������������� 94
Atomic Operations����������������������������������������������������������������������������������������������������������������� 94
Reserve/Publish API�������������������������������������������������������������������������������������������������������������� 97
Transactional API����������������������������������������������������������������������������������������������������������������� 100
Optional Flags���������������������������������������������������������������������������������������������������������������������� 104
Persisting Data Summary���������������������������������������������������������������������������������������������������� 104
Guarantees of libpmemobj’s APIs��������������������������������������������������������������������������������������������� 105
Managing Library Behavior������������������������������������������������������������������������������������������������������� 106
Debugging and Error Handling�������������������������������������������������������������������������������������������������� 106
Summary���������������������������������������������������������������������������������������������������������������������������������� 108

Chapter 8: libpmemobj-cpp: The Adaptable Language - C++ and


Persistent Memory����������������������������������������������������������������������������������������������� 111
Introduction������������������������������������������������������������������������������������������������������������������������������� 111
Metaprogramming to the Rescue��������������������������������������������������������������������������������������������� 112
Persistent Pointers�������������������������������������������������������������������������������������������������������������� 112
Transactions������������������������������������������������������������������������������������������������������������������������ 113
Snapshotting����������������������������������������������������������������������������������������������������������������������� 115
Allocating����������������������������������������������������������������������������������������������������������������������������� 116
C++ Standard limitations���������������������������������������������������������������������������������������������������������� 118
An Object’s Lifetime������������������������������������������������������������������������������������������������������������ 119
Trivial Types������������������������������������������������������������������������������������������������������������������������� 120
Object Layout����������������������������������������������������������������������������������������������������������������������� 122
Pointers������������������������������������������������������������������������������������������������������������������������������� 123
Limitations Summary���������������������������������������������������������������������������������������������������������� 125
Persistence Simplified�������������������������������������������������������������������������������������������������������������� 126
The Ecosystem�������������������������������������������������������������������������������������������������������������������������� 133
Persistent Containers���������������������������������������������������������������������������������������������������������� 134
Examples of Persistent Containers�������������������������������������������������������������������������������������� 135
Summary���������������������������������������������������������������������������������������������������������������������������������� 138

vi
Table of Contents

Chapter 9: pmemkv: A Persistent In-­Memory Key-Value Store���������������������������� 141


pmemkv Architecture���������������������������������������������������������������������������������������������������������������� 143
A Phonebook Example�������������������������������������������������������������������������������������������������������������� 147
Bringing Persistent Memory Closer to the Cloud���������������������������������������������������������������������� 151
Summary���������������������������������������������������������������������������������������������������������������������������������� 152

Chapter 10: Volatile Use of Persistent Memory���������������������������������������������������� 155


Introduction������������������������������������������������������������������������������������������������������������������������������� 155
Background������������������������������������������������������������������������������������������������������������������������������� 156
Memory Allocation��������������������������������������������������������������������������������������������������������������� 156
How it Works����������������������������������������������������������������������������������������������������������������������� 156
Supported “Kinds” of Memory��������������������������������������������������������������������������������������������� 157
The memkind API���������������������������������������������������������������������������������������������������������������������� 159
Kind Management API��������������������������������������������������������������������������������������������������������� 159
Heap Management API�������������������������������������������������������������������������������������������������������� 164
Kind Configuration Management����������������������������������������������������������������������������������������� 167
Additional memkind Code Examples����������������������������������������������������������������������������������� 168
C++ Allocator for PMEM Kind��������������������������������������������������������������������������������������������������� 168
pmem::allocator methods���������������������������������������������������������������������������������������������������� 169
Nested Containers��������������������������������������������������������������������������������������������������������������� 169
C++ Examples�������������������������������������������������������������������������������������������������������������������������� 170
Using the pmem::allocator�������������������������������������������������������������������������������������������������� 170
Creating a Vector of Strings������������������������������������������������������������������������������������������������� 171
Expanding Volatile Memory Using Persistent Memory�������������������������������������������������������������� 173
libvmemcache: An Efficient Volatile Key-Value Cache for Large-Capacity
Persistent Memory�������������������������������������������������������������������������������������������������������������������� 177
libvmemcache Overview����������������������������������������������������������������������������������������������������� 178
libvmemcache Design��������������������������������������������������������������������������������������������������������� 180
Using libvmemcache����������������������������������������������������������������������������������������������������������� 183
Summary���������������������������������������������������������������������������������������������������������������������������������� 186

vii
Table of Contents

Chapter 11: Designing Data Structures for Persistent Memory��������������������������� 187


Contiguous Data Structures and Fragmentation����������������������������������������������������������������������� 187
Internal and External Fragmentation����������������������������������������������������������������������������������� 188
Atomicity and Consistency�������������������������������������������������������������������������������������������������� 189
Selective Persistence���������������������������������������������������������������������������������������������������������� 193
Example Data Structures����������������������������������������������������������������������������������������������������� 193
Summary���������������������������������������������������������������������������������������������������������������������������������� 206

Chapter 12: Debugging Persistent Memory Applications������������������������������������� 207


pmemcheck for Valgrind����������������������������������������������������������������������������������������������������������� 208
Stack Overflow Example������������������������������������������������������������������������������������������������������ 208
Memory Leak Example�������������������������������������������������������������������������������������������������������� 209
Intel Inspector – Persistence Inspector������������������������������������������������������������������������������������ 210
Stack Overflow Example������������������������������������������������������������������������������������������������������ 211
Memory Leak Example�������������������������������������������������������������������������������������������������������� 212
Common Persistent Memory Programming Problems�������������������������������������������������������������� 214
Nonpersistent Stores����������������������������������������������������������������������������������������������������������� 214
Stores Not Added into a Transaction������������������������������������������������������������������������������������ 228
Memory Added to Two Different Transactions��������������������������������������������������������������������� 233
Memory Overwrites������������������������������������������������������������������������������������������������������������� 240
Unnecessary Flushes���������������������������������������������������������������������������������������������������������� 242
Out-of-Order Writes������������������������������������������������������������������������������������������������������������� 247
Summary���������������������������������������������������������������������������������������������������������������������������������� 259

Chapter 13: Enabling Persistence Using a Real-World Application���������������������� 261


The Database Example������������������������������������������������������������������������������������������������������������� 262
Different Persistent Memory Enablement Approaches������������������������������������������������������������� 262
Developing a Persistent Memory-Aware MariaDB* Storage Engine����������������������������������������� 263
Understanding the Storage Layer���������������������������������������������������������������������������������������� 264
Creating a Storage Engine Class����������������������������������������������������������������������������������������� 265
Summary���������������������������������������������������������������������������������������������������������������������������������� 276

viii
Table of Contents

Chapter 14: Concurrency and Persistent Memory������������������������������������������������ 277


Transactions and Multithreading����������������������������������������������������������������������������������������������� 278
Mutexes on Persistent Memory������������������������������������������������������������������������������������������������ 282
Atomic Operations and Persistent Memory������������������������������������������������������������������������������ 285
Lock-Free Algorithms and Persistent Memory�������������������������������������������������������������������� 285
Concurrent Data Structures for Persistent Memory������������������������������������������������������������������ 286
Concurrent Ordered Map����������������������������������������������������������������������������������������������������� 287
Concurrent Hash Map���������������������������������������������������������������������������������������������������������� 291
Summary���������������������������������������������������������������������������������������������������������������������������������� 293

Chapter 15: Profiling and Performance���������������������������������������������������������������� 295


Introduction������������������������������������������������������������������������������������������������������������������������������� 295
Performance Analysis Concepts����������������������������������������������������������������������������������������������� 295
Compute-Bound vs. Memory-Bound����������������������������������������������������������������������������������� 295
Memory Latency vs. Memory Capacity������������������������������������������������������������������������������� 296
Read vs. Write Performance������������������������������������������������������������������������������������������������ 296
Memory Access Patterns����������������������������������������������������������������������������������������������������� 296
I/O Storage Bound Workloads���������������������������������������������������������������������������������������������� 297
Determining the Suitability of Workloads for Persistent Memory��������������������������������������������� 297
Volatile Use Cases��������������������������������������������������������������������������������������������������������������� 298
Use Cases Requiring Persistence���������������������������������������������������������������������������������������� 301
Performance Analysis of Workloads Using Persistent Memory������������������������������������������������ 302
Characterizing the Workload����������������������������������������������������������������������������������������������� 303
Memory Bandwidth and Latency����������������������������������������������������������������������������������������� 303
Persistent Memory Read-Write Ratio���������������������������������������������������������������������������������� 305
Working Set Size and Memory Footprint����������������������������������������������������������������������������� 305
Non-Uniform Memory Architecture (NUMA) Behavior��������������������������������������������������������� 305
Optimizing the Software for Persistent Memory����������������������������������������������������������������� 307
Summary���������������������������������������������������������������������������������������������������������������������������������� 311

ix
Table of Contents

Chapter 16: PMDK Internals: Important Algorithms and Data Structures������������ 313
A Pool of Persistent Memory: High-Level Architecture Overview��������������������������������������������� 313
The Uncertainty of Memory Mapping: Persistent Memory Object Identifier����������������������������� 315
Persistent Thread Local Storage: Using Lanes�������������������������������������������������������������������������� 318
Ensuring Power-Fail Atomicity: Redo and Undo Logging���������������������������������������������������������� 320
Transaction Redo Logging��������������������������������������������������������������������������������������������������� 320
Transaction Undo Logging��������������������������������������������������������������������������������������������������� 321
libpmemobj Unified Logging������������������������������������������������������������������������������������������������ 322
Persistent Allocations: The Interface of a Transactional Persistent Allocator���������������������������� 323
Persistent Memory Heap Management: Allocator Design for Persistent Memory�������������������� 324
ACID Transactions: Efficient Low-Level Persistent Transactions����������������������������������������������� 328
Lazy Reinitialization of Variables: Storing the Volatile State on Persistent Memory����������������� 330
Summary���������������������������������������������������������������������������������������������������������������������������������� 331

Chapter 17: Reliability, Availability, and Serviceability (RAS)������������������������������ 333


Dealing with Uncorrectable Errors�������������������������������������������������������������������������������������������� 333
Consumed Uncorrectable Error Handling���������������������������������������������������������������������������� 334
Unconsumed Uncorrectable Error Handling������������������������������������������������������������������������ 336
Clearing Uncorrectable Errors��������������������������������������������������������������������������������������������� 339
Device Health���������������������������������������������������������������������������������������������������������������������������� 339
ACPI-Defined Health Functions (_NCH, _NBS)��������������������������������������������������������������������� 342
Vendor-Specific Device Health (_DSMs)������������������������������������������������������������������������������ 342
ACPI NFIT Health Event Notification������������������������������������������������������������������������������������ 343
Unsafe/Dirty Shutdown������������������������������������������������������������������������������������������������������������� 343
Application Utilization of Data Loss Count (DLC)����������������������������������������������������������������� 344
Summary���������������������������������������������������������������������������������������������������������������������������������� 346

Chapter 18: Remote Persistent Memory��������������������������������������������������������������� 347


RDMA Networking Protocols����������������������������������������������������������������������������������������������������� 348
Goals of the Initial Remote Persistent Memory Architecture���������������������������������������������������� 351
Guaranteeing Remote Persistence�������������������������������������������������������������������������������������������� 351
General-Purpose Remote Replication Method��������������������������������������������������������������������� 353

x
Table of Contents

Appliance Remote Replication Method�������������������������������������������������������������������������������� 355


General Software Architecture�������������������������������������������������������������������������������������������������� 357
librpmem Architecture and Its Use in Replication��������������������������������������������������������������������� 358
Configuring Remote Replication Using Poolsets����������������������������������������������������������������� 362
Performance Considerations����������������������������������������������������������������������������������������������� 362
Remote Replication Error Handling������������������������������������������������������������������������������������� 364
Say Hello to the Replicated World��������������������������������������������������������������������������������������� 364
Summary���������������������������������������������������������������������������������������������������������������������������������� 370

Chapter 19: Advanced Topics������������������������������������������������������������������������������� 373


Nonuniform Memory Access (NUMA)���������������������������������������������������������������������������������������� 373
NUMACTL Linux Utility��������������������������������������������������������������������������������������������������������� 374
NDCTL Linux Utility�������������������������������������������������������������������������������������������������������������� 376
Intel Memory Latency Checker Utility���������������������������������������������������������������������������������� 378
NUMASTAT Utility����������������������������������������������������������������������������������������������������������������� 380
Intel VTune Profiler – Platform Profiler�������������������������������������������������������������������������������� 381
IPMCTL Utility���������������������������������������������������������������������������������������������������������������������� 381
BIOS Tuning Options������������������������������������������������������������������������������������������������������������ 382
Automatic NUMA Balancing������������������������������������������������������������������������������������������������� 382
Using Volume Managers with Persistent Memory�������������������������������������������������������������������� 383
The mmap( ) MAP_SYNC Flag��������������������������������������������������������������������������������������������������� 385
Summary���������������������������������������������������������������������������������������������������������������������������������� 386

Appendix A: How to Install NDCTL and DAXCTL on Linux������������������������������������� 389


Prerequisites����������������������������������������������������������������������������������������������������������������������������� 389
Installing NDCTL and DAXCTL Using the Linux Distribution Package Repository���������������������� 390
Searching for Packages Within a Package Repository�������������������������������������������������������� 391
Installing NDCTL and DAXCTL from the Package Repository���������������������������������������������� 392

xi
Table of Contents

Appendix B: How to Install the Persistent Memory Development Kit (PMDK)������ 395
PMDK Prerequisites������������������������������������������������������������������������������������������������������������������ 395
Installing PMDK Using the Linux Distribution Package Repository������������������������������������������� 395
Package Naming Convention����������������������������������������������������������������������������������������������� 396
Searching for Packages Within a Package Repository�������������������������������������������������������� 396
Installing PMDK Libraries from the Package Repository����������������������������������������������������� 398
Installing PMDK on Microsoft Windows������������������������������������������������������������������������������������ 402

Appendix C: How to Install IPMCTL on Linux and Windows���������������������������������� 403


IPMCTL Linux Prerequisites������������������������������������������������������������������������������������������������������ 404
libsafec�������������������������������������������������������������������������������������������������������������������������������� 404
IPMCTL Linux Packages������������������������������������������������������������������������������������������������������������ 404
IPMCTL for Microsoft Windows������������������������������������������������������������������������������������������������� 404
Using ipmctl������������������������������������������������������������������������������������������������������������������������������ 405

Appendix D: Java for Persistent Memory������������������������������������������������������������� 411


Volatile Use of Persistent Memory�������������������������������������������������������������������������������������������� 411
Heap Allocation on Alternative Memory Devices����������������������������������������������������������������� 412
Persistent Collections for Java (PCJ)���������������������������������������������������������������������������������������� 416
Using PCJ in Java Applications������������������������������������������������������������������������������������������������� 417
Low-Level Persistent Library (LLPL)����������������������������������������������������������������������������������������� 418
Using LLPL in Java Applications������������������������������������������������������������������������������������������������ 419
Summary���������������������������������������������������������������������������������������������������������������������������������� 419

Appendix E: The Future of Remote Persistent Memory Replication��������������������� 421

Glossary���������������������������������������������������������������������������������������������������������������� 425

Index��������������������������������������������������������������������������������������������������������������������� 429

xii
About the Author
Steve Scargall is a persistent memory software and cloud architect at Intel
Corporation. As a technology evangelist, he supports the enabling and development
effort to integrate persistent memory technology into software stacks, applications,
and hardware architectures. This includes working with independent software
vendors (ISVs) on both proprietary and open source development, original equipment
manufacturers (OEMs), and cloud service providers (CSPs).

Steve holds a Bachelor of Science in computer science and cybernetics from the
University of Reading, UK, where he studied neural networks, AI, and robotics. He
has over 19 years’ experience providing performance analysis on x86 architecture and
SPARC for Solaris Kernel, ZFS, and UFS file system. He performed DTrace debugging in
enterprise and cloud environments during his tenures at Sun Microsystems and Oracle.

xiii
About the Technical Reviewer
Andy Rudoff is a principal engineer at Intel Corporation, focusing on non-volatile
memory programming. He is a contributor to the SNIA NVM Programming Technical
Work Group. His more than 30 years’ industry experience includes design and
development work in operating systems, file systems, networking, and fault management
at companies large and small, including Sun Microsystems and VMware. Andy has
taught various operating systems classes over the years and is a coauthor of the popular
UNIX Network Programming textbook.

xv
About the Contributors
Piotr Balcer is a software engineer at Intel Corporation with many years’ experience
working on storage-related technologies. He holds a Bachelor of Science in engineering
from the Gdańsk University of Technology, Poland, where he studied system software
engineering. Piotr has been working on the software ecosystem for next-generation
persistent memory since 2014.

Eduardo Berrocal joined Intel Corporation as a cloud software engineer in 2017 after
receiving his PhD in computer science from the Illinois Institute of Technology. His
doctoral research focused on data analytics and fault tolerance for high-performance
computing. Past experience includes working as an intern at Bell Labs (Nokia), a research
aid at Argonne National Laboratory, a scientific programmer and web developer at the
University of Chicago, and an intern in the CESVIMA laboratory in Spain.

Adam Borowski is a software engineer at Intel Corporation, hailing from the


University of Warsaw, Poland. He is a Debian developer and has made many open
source contributions over the past two decades. Adam is currently working on
persistent memory stacks, both on upstream code and integrating it with downstream
distributions.

Igor Chorazewicz is a software engineer at Intel Corporation. His main focus is on


persistent memory data structures and enabling C++ applications for persistent
memory. Igor holds a Bachelor of Science in engineering from the Gdańsk University of
Technology, Poland.

Adam Czapski is a technical writer at Intel Corporation. He writes technical


documentation in the Data Center Group and is currently working in the persistent
memory department. Adam holds a Bachelor of Arts in English philology and a master’s
degree in natural language processing from the Gdańsk University of Technology, Poland.

Steve Dohrmann is a software engineer at Intel Corporation. He has worked on a variety


of projects over the past 20 years, including media frameworks, mobile agent software,
secure collaboration software, and parallel programming language implementation. He
is currently working on enabling the use of persistent memory in Java*.

xvii
About the Contributors

Chet Douglas is a principal software engineer at Intel Corporation and focuses on cloud
software architecture along with operating system and OEM enabling of non-volatile
memory technologies. He has over 14 years’ experience working on various enterprise
and client programs and 28 years of total storage experience. Chet has worked in all
aspects of storage, including storage controller hardware design, SCSI disk/tape/CD
writer firmware architecture, storage management software architecture, Microsoft
Windows* and Linux kernel-mode drivers, enterprise hardware RAID, and client/
workstation software RAID. He holds seven storage-related hardware and software
patents and has a dual Bachelor of Science in electrical engineering and computer
engineering from Clarkson University, New York.

Ken Gibson is the director of persistent memory software architecture within Intel
Corporation’s Data Center Group. Since 2012, Ken and his team have been working with
Intel’s server and software partners to create the open persistent memory programming
model.

Tomasz Gromadzki is a software architect in Intel Corporation’s Non-Volatile Memory


Solutions Group. His focus is on remote persistent memory access, which includes
proper integration of persistent memory with other (networking) technologies as well as
optimal persistent memory replication procedures and algorithms.

Kishor Kharbas is a software engineer on the Java runtime engineering team at Intel
Corporation. For the past eight years, he has been working to optimize Oracle’s OpenJDK
on Intel platforms. This involves Java garbage collection and compiler back-end
optimization.

Jackson Marusarz is a senior technical consulting engineer (TCE) in Intel Corporation's


Compute Performance and Developer Products Division. As the lead TCE for Intel
VTune Profiler, his main focus is on software performance analysis and tuning for both
serial and multithreaded applications. Jackson’s time is split between determining how
to analyze and tune software and creating tools that help others do the same.

Jan Michalski is a software engineer in Intel Corporation’s Non-­Volatile Memory


Solutions Group. His focus is on remote persistent memory access, which includes
proper integration of persistent memory with other technologies, as well as looking for
optimal persistent memory replication procedures and algorithms. He holds a master's
degree in computer engineering from the Gdańsk University of Technology, Poland,
where he studied system software engineering.

xviii
About the Contributors

Nicholas Moulin is a cloud software architect at Intel Corporation. Since joining Intel
in 2012, he has focused on enabling and developing persistent memory software for
operating systems and platform firmware and managing persistent memory hardware.
Nicholas is currently working with industry partners to define and improve RAS features
relevant to the persistent memory programming model.

Szymon Romik is a software engineer at Intel Corporation and is currently focused on


persistent memory programming. He previously worked as a lead software engineer
on 5G technologies at Ericsson. Szymon holds a master’s degree in mathematics from
Jagiellonian University, Poland.

Jakub Schmiegel is a software architect in Intel Corporation’s Non-Volatile Memory


Solutions Group where he has been focused on enabling existing applications to
persistent memory and analyzing their performance for more than four years. Jakub
holds a master’s degree in computer science from the Gdańsk University of Technology,
Poland.

Kevin Shalkowsky is a Telly Award–winning creative director, graphic designer, and


animator with more than a decade of experience. While his contributions are in
technology today, Kevin has spent time in broadcast journalism and selling numerous
products through 30-minute late-night infomercials. He resides in Oregon with his wife
and son. From time to time, you can find Kevin lost in the woods, lost in a parking lot,
or lost in his design process – but this somehow got him to where he is today, and he
wouldn’t have it any other way.

Vineet Singh is a memory and storage tools software engineer at Intel Corporation.
He develops techniques to help developers adapt to the latest memory technologies.
Vineet holds a PhD in philosophy from the University of California and has a Bachelor
of Technology degree from the Indian Institute of Information Technology, Design, and
Manufacturing in Jabalpur.

Pawel Skowron is a software engineering manager at Intel Corporation with 20 years'


experience in the software industry. Pawel has worked in various roles related to the
whole-software development life cycle. His software engineering background lies in the
areas of embedded systems, database systems, and applications. For the past few years,
Pawel has led the development and validation of the Persistent Memory Development
Kit (https://github.com/pmem/pmdk).

xix
About the Contributors

Usha Upadhyayula has been with Intel Corporation for 20 years serving in many
different roles. Usha holds a master’s degree in computer science from the University
of South Carolina, and she spent the first few years at Intel developing user-level
applications in C and C++. She later moved to customer-­enabling roles for Intel media
processors and support for Intel RAID software. Usha is currently part of the Data Center
Group where she is focused on enabling cloud service providers to fully utilize and
accelerate the adoption of Intel persistent memory products.

Sergey Vinogradov is a senior software development engineer at Intel Corporation


where he spent more than seven years working on performance profiling tools and
threading runtime libraries. During the past four years, Sergey has been working on C++
programming models and performance profiling methodologies for persistent memory.

xx
Acknowledgments
First and foremost, I would like to thank Ken Gibson for masterminding this book idea
and for gifting me the pleasure of writing and managing it. Your support, guidance, and
contributions have been instrumental in delivering a high-quality product.
If the Vulcan mind-meld or The Matrix Headjack were possible, I could have cloned
Andy Rudoff’s mind and allowed him to work on his daily activities. Instead, Andy’s
infinite knowledge of persistent memory had to be tapped through good old verbal
communication and e-mail. I sincerely thank you for devoting so much time to me and
this project. The results read for themselves.
Debbie Graham was instrumental in helping me manage this colossal project. Her
dedication and support helped drive the project to an on-time completion.
To my friends and colleagues at Intel who contributed content, supported
discussions, helped with decision-making, and reviewed drafts during the book-writing
process. These are the real heroes. Without your heavily invested time and support, this
book would have taken considerably longer to complete. It is a much better product as a
result of the collaborative effort. A huge thanks to all of you.
I'd like to express my sincerest gratitude and appreciation to the people at Apress,
without whom this book could not have been published. From the initial contact and
outline discussions through the entire publishing process to this final polished product,
the Apress team delivered continuous support and assistance. Many thanks to Susan,
Jessica, and Rita. It was a real pleasure working with you.

xxi
Preface
About This Book
Persistent memory is often referred to as non-volatile memory (NVM) or storage
class memory (SCM). In this book, we purposefully use persistent memory as an all-­
encompassing term to represent all the current and future memory technologies that
fall under this umbrella. This book introduces the persistent memory technology and
provides answers to key questions. For software developers, those questions include:
What is persistent memory? How do I use it? What APIs and libraries are available?
What benefits can it provide for my application? What new programming methods do I
need to learn? How do I design applications to use persistent memory? Where can I find
information, documentation, and help?
System and cloud architects will be provided with answers to questions such as:
What is persistent memory? How does it work? How is it different than DRAM or SSD/
NVMe storage devices? What are the hardware and operating system requirements?
What applications need or could benefit from persistent memory? Can my existing
applications use persistent memory without being modified?
Persistent memory is not a plug-and-play technology for software applications.
Although it may look and feel like traditional DRAM memory, applications need to be
modified to fully utilize the persistence feature of persistent memory. That is not to say
that applications cannot run unmodified on systems with persistent memory installed,
they can, but they will not see the full potential of what persistent memory offers without
code modification.
Thankfully, server and operating system vendors collaborated very early in the
design phase and already have products available on the market. Linux and Microsoft
Windows already provide native support for persistent memory technologies. Many
popular virtualization technologies also support persistent memory.
For ISVs and the developer community at large, the journey is just beginning. Some
software has already been modified and is available on the market. However, it will
take time for the enterprise and cloud computing industries to adopt and make the
hardware available to the general marketplace. ISVs and software developers need time
to understand what changes to existing applications are required and implement them.
xxiii
Preface

To make the required development work easier, Intel developed and open sourced
the Persistent Memory Development Kit (PMDK) available from https://pmem.io/
pmdk/. We introduce the PMDK in more detail in Chapter 5 and walk through most of
the available libraries in subsequent chapters. Each chapter provides an in-depth guide
so developers can understand what library or libraries to use. PMDK is a set of open
source libraries and tools based on the Storage Networking Industry Association (SNIA)
NVM programming model designed and implemented by over 50 industry partners. The
latest NVM programming model document can be found at https://www.snia.org/
tech_activities/standards/curr_standards/npm. The model describes how software
can utilize persistent memory features and enables designers to develop APIs that take
advantage of NVM features and performance.
Available for both Linux and Windows, PMDK facilitates persistent memory
programming adoption with higher-level language support. C and C++ support is fully
validated. Support for other languages such as Java and Python is work in progress
at the time this book was written. Other languages are expected to also adopt the
programming model and provide native persistent memory APIs for developers. The
PMDK development team welcomes and encourages new contributions to core code,
new language bindings, or new storage engines for the persistent memory key-value
store called pmemkv.
This book assumes no prior knowledge of persistent memory hardware devices
or software development. The book layout allows you to freely navigate the content in
the order you want. It is not required to read all chapters in order, though we do build
upon concepts and knowledge described in previous chapters. In such cases, we make
backward and forward references to relevant chapters and sections so you can learn or
refresh your memory.

B
 ook Structure
This book has 19 chapters, each one focusing on a different topic. The book has three
main sections. Chapters 1-4 provide an introduction to persistent memory architecture,
hardware, and operating system support. Chapters 5-16 allow developers to understand
the PMDK libraries and how to use them in applications. Finally, Chapters 17-19 provide
information on advanced topics such as RAS and replication of data using RDMA.

xxiv
Preface

• Chapter 1. Introduction to Persistent Memory – Introduces persistent


memory and dips our toes in the water with a simple persistent key-­
value store example using libpmemkv.

• Chapter 2. Persistent Memory Architecture – Describes the persistent


memory architecture and focuses on the hardware requirements
developers should know.

• Chapter 3. Operating System Support for Persistent Memory –


Provides information relating to operating system changes, new
features, and how persistent memory is seen by the OS.

• Chapter 4. Fundamental Concepts of Persistent Memory


Programming – Builds on the first three chapters and describes the
fundamental concepts of persistent memory programming.

• Chapter 5. Introducing the Persistent Memory Development Kit


(PMDK) – Introduces the Persistent Memory Development Kit
(PMDK), a suite of libraries to assist software developers.

• Chapter 6. libpmem: Low-Level Persistent Memory Support –


Describes and shows how to use libpmem from the PMDK, a low-­level
library providing persistent memory support.

• Chapter 7. libpmemobj: A Native Transactional Object Store –


Provides information and examples using libpmemobj, a C native
object store library from the PMDK.
• Chapter 8. libpmemobj-cpp: The Adaptable Language - C++ and
Persistent Memory – Demonstrates the C++ libpmemobj-cpp object
store from the PMDK, built using C++ headers on top of libpmemobj.

• Chapter 9. pmemkv: A Persistent In-Memory Key-Value Store –


Expands upon the introduction to libpmemkv from Chapter 1 with a
more in-depth discussion using examples.

• Chapter 10. Volatile Use of Persistent Memory – This chapter is


for those who want to take advantage of persistent memory but
do not require data to be stored persistently. libmemkind is a user-
extensible heap manager built on top of jemalloc which enables
control of memory characteristics and a partitioning of the heap

xxv
Preface

between different kinds of memory, including persistent memory.


libvmemcache is an embeddable and lightweight in-memory caching
solution. It is designed to fully take advantage of large-capacity
memory, such as persistent memory with DAX, through memory
mapping in an efficient and scalable way.

• Chapter 11. Designing Data Structures for Persistent Memory –


Provides a wealth of information for designing data structures for
persistent memory.
• Chapter 12. Debugging Persistent Memory Applications – Introduces
tools and walks through several examples for how software developers
can debug persistent memory–enabled applications.

• Chapter 13. Enabling Persistence using a Real-World Application –


Discusses how a real-world application was modified to enable
persistent memory features.

• Chapter 14. Concurrency and Persistent Memory – Describes how


concurrency in applications should be implemented for use with
persistent memory.

• Chapter 15. Profiling and Performance – Teaches performance


concepts and demonstrates how to use the Intel VTune suite of tools
to profile systems and applications before and after code changes are
made.

• Chapter 16. PMDK Internals: Important Algorithms and Data


Structures – Takes us on a deep dive of the PMDK design, architecture,
algorithms, and memory allocator implementation.

• Chapter 17. Reliability, Availability, and Serviceability (RAS) –


Describes the implementation of reliability, availability, and
serviceability (RAS) with the hardware and operating system layers.

• Chapter 18. Remote Persistent Memory – Discusses how applications


can scale out across multiple systems using local and remote persistent
memory.

• Chapter 19. Advanced Topics – Describes things such as NUMA, using


software volume managers, and the mmap() MAP_SYNC flag.

xxvi
Preface

The Appendixes have separate procedures for installing the PMDK and utilities
required for managing persistent memory. We also included an update for Java and the
future of the RDMA protocols. All of this content is considered temporal, so we did not
want to include it in the main body of the book.

Intended Audience
This book has been written for experienced application developers in mind. We
intend the content to be useful to a wider readership such as system administrators
and architects, students, lecturers, and academic research fellows to name but a few.
System designers, kernel developers, and anyone with a vested or passing interest in this
emerging technology will find something useful within this book.
Every reader will learn what persistent memory is, how it works, and how operating
systems and applications can utilize it. Provisioning and managing persistent memory
are vendor specific, so we include some resources in the Appendix sections to avoid
overcomplicating the main chapter content.
Application developers will learn, by example, how to integrate persistent memory
in to existing or new applications. We use examples extensively throughout this book
using a variety of libraries available within the Persistent Memory Development Kit
(PMDK). Example code is provided in a variety of programming languages such as C,
C++, JavaScript, and others. We want developers to feel comfortable using these libraries
in their own projects. The book provides extensive links to resources where you can find
help and information.
System administrators and architects of Cloud, high-performance computing,
and enterprise environments can use most of the content of this book to
understand persistent memory features and benefits to support applications and
developers. Imagine being able to deploy more virtual machines per physical server or
provide applications with this new memory/storage tier such that they can keep more
data closer to the CPU or restart in a fraction of the time they could before while keeping
a warm cache of data.
Students, lecturers, and academic research fellows will also benefit from many
chapters within this book. Computer science classes can learn about the hardware,
operating system features, and programming techniques. Lecturers are free use the
content in student classes or to form the basis of research projects such as new persistent
memory file systems, algorithms, or caching implementations.

xxvii
Preface

We introduce tools that profile the server and applications to better understand CPU,
memory, and disk IO access patterns. Using this knowledge, we show how applications
can be modified to take full advantage of persistence using the Persistent Memory
Development Kit (PMDK).

A Future Reference
The book content has been written to provide value for many years. Industry
specification such as ACPI, UEFI, and the SNIA non-volatile programming model will,
unless otherwise stated by the specification, remain backward compatible as new
versions are released. If new form factors are introduced, the approach to programming
remains the same. We do not limit ourselves to one specific persistent memory vendor
or implementation. In places where it is necessary to describe vendor-specific features
or implementations, we specifically call this out as it may change between vendors or
between product generations. We encourage you to read the vendor documentation for
the persistent memory product to learn more.
Developers using the Persistent Memory Development Kit (PMDK) will retain a stable
API interface. PMDK will deliver new features and performance improvements with each
major release. It will evolve with new persistent memory products, CPU instructions,
platform designs, industry specifications, and operating system feature support.

Source Code Examples


Concepts and source code samples within this book adhere to the vendor neutral
SNIA non-volatile memory programming model. SNIA which is the Storage
Networking Industry Association is a non-profit global organization dedicated to
developing standards and education programs to advance storage and information
technology. The model was designed, developed, and is maintained by the SNIA NVM
Technical Working Group (TWG) which includes many leading operating system,
hardware, and server vendors. You can join this group or find information at https://
www.snia.org/forums/sssi/nvmp.

xxviii
Preface

The code examples provided with this book have been tested and validated using
Intel Optane DC persistent memory. Since the PMDK is vendor neutral, they will also
work on NVDIMM-N devices. PMDK will support any future persistent memory product
that enters the market.
The code examples used throughout this book are current at the time of
publication. All code examples have been validated and tested to ensure they compile
and execute without error. For brevity, some of the examples in this book use assert()
statements to indicate unexpected errors. Any production code would likely replace
these with the appropriate error handling actions which would include friendlier
error messages and appropriate error recovery actions. Additionally, some of the code
examples use different mount points to represent persistent memory aware file systems,
for example “/daxfs”, “/pmemfs”, and “/mnt/pmemfs”. This demonstrates persistent
memory file systems can be mounted and named appropriately for the application, just
like regular block-based file systems. Source code is from the repository that accompanies
this book – https://github.com/Apress/programming-persistent-memory.
Since this is a rapidly evolving technology, the software and APIs references
throughout this book may change over time. While every effort is made to be backward
compatible, sometimes software must evolve and invalidate previous versions. For this
reason, it is therefore expected that some of the code samples may not compile on newer
hardware or operating systems and may need to be changed accordingly.

B
 ook Conventions
This book uses several conventions to draw your attention to specific pieces of
information. The convention used depends on the type of information displayed.

C
 omputer Commands
Commands, programming library, and API function references may be presented in line
with the paragraph text using a monospaced font. For example:
To illustrate how persistent memory is used, let’s start with a sample program
demonstrating the key-value store provided by a library called libpmemkv.

xxix
Preface

Computer Terminal Output


Computer terminal output is usually taken directly from a computer terminal presented
in a monospaced font such as the following example demonstrating cloning the
Persistent Memory Development Kit (PMDK) from the GitHub project:

$ git clone https://github.com/pmem/pmdk


Cloning into 'pmdk'...
remote: Enumerating objects: 12, done.
remote: Counting objects: 100% (12/12), done.
remote: Compressing objects: 100% (10/10), done.
remote: Total 100169 (delta 2), reused 7 (delta 2), pack-reused 100157
Receiving objects: 100% (100169/100169), 34.71 MiB | 4.85 MiB/s, done.
Resolving deltas: 100% (83447/83447), done.

S
 ource Code
Source code examples taken from the accompanying GitHub repository are shown with
relevant line numbers in a monospaced font. Below each code listing is a reference to
the line number or line number range with a brief description. Code comments use
language native styling. Most languages use the same syntax. Single line comments
will use // and block/multiline comments should use /*..*/. An example is shown in
Listing 1.

Listing 1. A sample program using libpmemkv


    37  #include <iostream>
    38  #include "libpmemkv.h"
    39  
    40  using namespace pmemkv;
    41  
    42  /*
    43   * kvprint -- print a single key-value pair
    44   */
    45  void kvprint(const string& k, const string& v) {
    46      std::cout << "key: " << k << ", value: " << v << "\n";
    47  }

xxx
Preface

• Line 45: Here we define a small helper routine, kvprint(), which prints
a key-value pair when called.

Notes
We use a standard format for notes, cautions, and tips when we want to direct your
attention to an important point, for example.

Note Notes are tips, shortcuts, or alternative approaches to the current


discussion topic. Ignoring a note should have no negative consequences, but you
might miss out on a nugget of information that makes your life easier.

xxxi
Discovering Diverse Content Through
Random Scribd Documents
lawfully engaged in commerce on the high seas, and in waters
of the United States:
“And Whereas, An Executive Proclamation has already been
issued, requiring the persons engaged in these disorderly
proceedings to desist therefrom, calling out a militia force for
the purpose of repressing the same, and convening Congress in
extraordinary session to deliberate and determine thereon:
“Now, therefore, I, Abraham Lincoln, President of the United
States, with a view to the same purposes before mentioned,
and to the protection of the public peace, and the lives and
property of quiet and orderly citizens pursuing their lawful
occupations, until Congress shall have assembled and
deliberated on the said unlawful proceedings, or until the same
shall have ceased, have further deemed it advisable to set on
foot a blockade of the ports within the States aforesaid, in
pursuance of the laws of the United States, and of the laws of
nations in such cases provided. For this purpose a competent
force will be posted so as to prevent entrance and exit of
vessels from the ports aforesaid. If, therefore, with a view to
violate such blockade, a vessel shall approach, or shall attempt
to leave any of the said ports, she will be duly warned by the
commander of one of the blockading vessels, who will indorse
on her register the fact and date of such warning; and if the
same vessel shall again attempt to enter or leave the blockaded
port, she will be captured and sent to the nearest convenient
port, for such proceedings against her and her cargo as prize,
as may be deemed advisable.
“And I hereby proclaim and declare, that if any person,
under the pretended authority of said States, or under any other
pretence, shall molest a vessel of the United States, or the
persons or cargo on board of her, such person will be held
amenable to the laws of the United States for the prevention
and punishment of piracy.
“By the President: Abraham Lincoln.
“William H. Seward, Secretary of State.”

On the 27th of April, the following additional proclamation was


issued:

“Whereas, For the reasons assigned in my proclamation of


the 19th instant, a blockade of the ports of the States of South
Carolina, Georgia, Florida, Alabama, Louisiana, Mississippi, and
Texas was ordered to be established; And whereas, since that
date public property of the United States has been seized, the
collection of the revenue obstructed, and duly commissioned
officers of the United States, while engaged in executing the
orders of their superiors, have been arrested and held in
custody as prisoners, or have been impeded in the discharge of
their official duties, without due legal process, by persons
claiming to act under authority of the States of Virginia and
North Carolina, an efficient blockade of the ports of these States
will therefore also be established.
“In witness whereof, I have hereunto set my hand, and
caused the seal of the United States to be affixed.
“Done at the City of Washington, this 27th day of April, in
the year of our Lord one thousand eight hundred and sixty-one,
and of the independence of the United States the eighty-fifth.
“By the President: Abraham Lincoln.
“William H. Seward, Secretary of State.”

This greatly affected the commercial interests of the European


powers, who made haste to announce that the blockade must be an
effectual one, in order to be respected; supposing, in common with
the rebels, that they were demanding what would prove to be an
impossibility. To say that they erred decidedly in this opinion, is but
stating a matter of general notoriety, and simply adds another to the
list of serious mistakes made, during the progress of the war, by the
two European nations most deeply interested in its issue.
It was soon perceived that more men would be needed in the
field, Davis, in a message to his Congress, having proposed “to
organize and hold in readiness for instant action, in view of the
exigencies of the country, an army of six hundred thousand men.”
On the 3d of May, accordingly, another call was made, in anticipation
of its ratification at the extra session of Congress, which ratification
took place, without opposition.

“Whereas, Existing exigencies demand immediate and


adequate measures for the protection of the national
Constitution and the preservation of the national Union by the
suppression of the insurrectionary combinations now existing in
several States for opposing the laws of the Union and
obstructing the execution thereof, to which end a military force,
in addition to that called forth by my Proclamation of the
fifteenth day of April, in the present year, appears to be
indispensably necessary, now, therefore, I, Abraham Lincoln,
President of the United States, and Commander-in-chief of the
Army and Navy thereof, and of the militia of the several States,
when called into actual service, do hereby call into the service
of the United States forty-two thousand and thirty-four
volunteers, to serve for a period of three years, unless sooner
discharged, and to be mustered into service as infantry and
cavalry. The proportions of each arm, and the details of
enrolment and organization will be made known through the
Department of War; and I also direct that the regular army of
the United States be increased by the addition of eight
regiments of infantry, one regiment of cavalry and one regiment
of artillery, making altogether a maximum aggregate increase of
twenty-two thousand seven hundred and fourteen officers and
enlisted men, the details of which increase will also be made
known through the Department of War; and I further direct the
enlistment, for not less than one nor more than three years, of
eighteen thousand seamen, in addition to the present force, for
the naval service of the United States. The details of the
enlistment and organization will be made known through the
Department of the Navy. The call for volunteers, hereby made,
and the direction of the increase of the regular army, and for
the enlistment of seamen hereby given, together with the plan
of organization adopted for the volunteers and for the regular
forces hereby authorized, will be submitted to Congress as soon
as assembled.
“In the meantime, I earnestly invoke the coöperation of all
good citizens in the measures hereby adopted for the effectual
suppression of unlawful violence, for the impartial enforcement
of constitutional laws, and for the speediest possible restoration
of peace and order, and with those of happiness and prosperity
throughout our country.
“In testimony whereof, I have hereunto set my hand, and
caused the seal of the United States to be affixed.
“Done at the City of Washington, this third day of May, in
the year of our Lord one thousand eight hundred and sixty-one,
and of the Independence of the United States the eighty-fifth.
“By the President: Abraham Lincoln.
“William H. Seward, Secretary of State.”

On the 10th of May, 1861, the following proclamation was


promulgated:

“Whereas, An insurrection exists in the State of Florida, by


which the lives, liberty, and property of loyal citizens of the
United States are endangered.
“And Whereas, It is deemed proper that all needful
measures should be taken for the protection of such citizens
and all officers of the United States in the discharge of their
public duties in the State aforesaid:
“Now, therefore, be it known that I, Abraham Lincoln,
President of the United States, do hereby direct the commander
of the forces of the United States on the Florida coast to permit
no person to exercise any office or authority upon the islands of
Key West, the Tortugas, and Santa Rosa, which may be
inconsistent with the laws and Constitution of the United States,
authorizing him at the same time, if he shall find it necessary, to
suspend there the writ of habeas corpus, and to remove from
the vicinity of the United States fortresses all dangerous or
suspected persons.
“In witness whereof, I have hereunto set my hand, and
caused the seal of the United States to be affixed.
“Done at the City of Washington, this tenth day of May, in
the year of our Lord, one thousand eight hundred and sixty-one,
and of the Independence of the United States the eighty-fifth.
“By the President: Abraham Lincoln.
“William H. Seward, Secretary of State.”

Volunteers meanwhile presented themselves for the defence of


the country in numbers greater than could be accepted, and the
strife was who should secure the coveted distinction of a citizen
soldier. An early movement upon the rebel army in Virginia was
contemplated, and it was confidently anticipated that to advance
was to put the enemies of the Government to flight.
CHAPTER VIII.
THE FIRST SESSION OF CONGRESS.

Opening of Congress​—​President’s First Message​—I​ ts Nature​—​


Action of Congress​—​Resolution Declaring the Object of the War​
—​Bull Run​—I​ ts Effect.

The first session of Congress during Mr. Lincoln’s Administration


commenced on the 4th of July, 1861, in pursuance of his call to that
effect. The following message was transmitted from the Executive:

“Fellow-Citizens of the Senate and House of Representatives:​—​


Having been convened on an extraordinary occasion, as
authorized by the Constitution, your attention is not called to
any ordinary subject of legislation. At the beginning of the
present Presidential term, four months ago, the functions of the
Federal Government were found to be generally suspended
within the several States of South Carolina, Georgia, Alabama,
Mississippi, Louisiana and Florida, excepting only those of the
Post-office Department.
“Within these States, all the Forts, Arsenals, Dock-Yards,
Custom-Houses, and the like, including the movable and
stationary property in and about them, had been seized, and
were held in open hostility to this Government, excepting only
Forts Pickens, Taylor and Jefferson, on and near the Florida
coast, and Fort Sumter in Charleston harbor, South Carolina.
The forts thus seized had been put in improved condition, new
ones had been built, and armed forces had been organized, and
were organizing, all avowedly with the same hostile purpose.
“The forts remaining in possession of the Federal
Government in and near these States were either besieged or
menaced by warlike preparations, and especially Fort Sumter
was nearly surrounded by well-protected hostile batteries, with
guns equal in quality to the best of its own, and outnumbering
the latter as, perhaps, ten to one​—a​ disproportionate share of
the Federal muskets and rifles had somehow found their way
into these States, and had been seized to be used against the
Government.
“Accumulations of the public revenue lying within them had
been seized for the same object. The navy was scattered in
distant seas, leaving but a very small part of it within the
immediate reach of the Government.
“Officers of the Federal Army had resigned in great
numbers, and of those resigning a large proportion had taken
up arms against the Government.
“Simultaneously, and in connection with all this, the purpose
to sever the Federal Union was openly avowed. In accordance
with this purpose an ordinance had been adopted in each of
these States, declaring the States respectively to be separated
from the National Union. A formula for instituting a combined
Government of those States had been promulgated, and this
illegal organization, in the character of the ‘Confederate States,’
was already invoking recognition, aid and intervention from
foreign powers.
“Finding this condition of things, and believing it to be an
imperative duty upon the incoming Executive to prevent, if
possible, the consummation of such attempt to destroy the
Federal Union, a choice of means to that end became
indispensable. This choice was made and was declared in the
Inaugural Address.
“The policy chosen looked to the exhaustion of all peaceful
measures before a resort to any stronger ones. It sought only to
hold the public places and property not already wrested from
the Government, and to collect the revenue, relying for the rest
on time, discussion, and the ballot-box. It promised a
continuance of the mails, at Government expense, to the very
people who were resisting the Government, and it gave
repeated pledges against any disturbances to any of the people,
or any of their rights, of all that which a President might
constitutionally and justifiably do in such a case; every thing
was forborne, without which it was believed possible to keep
the Government on foot.
“On the 5th of March, the present incumbent’s first full day
in office, a letter from Major Anderson, commanding at Fort
Sumter, written on the 28th of February, and received at the
War Department on the 4th of March, was by that Department
placed in his hands. This letter expressed the professional
opinion of the writer, that reinforcements could not be thrown
into that fort within the time for its relief rendered necessary by
the limited supply of provisions, and with a view of holding
possession of the same, with a force less than twenty thousand
good and well-disciplined men. This opinion was concurred in by
all the officers of his command, and their memoranda on the
subject were made inclosures of Major Anderson’s letter. The
whole was immediately laid before Lieutenant-General Scott,
who at once concurred with Major Anderson in his opinion. On
reflection, however, he took full time, consulting with other
officers, both of the Army and Navy, and at the end of four days
came reluctantly but decidedly to the same conclusion as
before. He also stated at the same time that no such sufficient
force was then at the control of the Government, or could be
raised and brought to the ground, within the time when the
provisions in the fort would be exhausted. In a purely military
point of view, this reduced the duty of the Administration in the
case to the mere matter of getting the garrison safely out of the
fort.
“It was believed, however, that to so abandon that position,
under the circumstances, would be utterly ruinous; that the
necessity under which it was to be done would not be fully
understood; that by many it would be construed as a part of a
voluntary policy; that at home it would discourage the friends of
the Union, embolden its adversaries, and go far to insure to the
latter a recognition abroad; that, in fact, it would be our
national destruction consummated. This could not be allowed.
Starvation was not yet upon the garrison, and ere it would be
reached, Fort Pickens might be reinforced. This last would be a
clear indication of policy, and would better enable the country to
accept the evacuation of Fort Sumter as a military necessity. An
order was at once directed to be sent for the landing of the
troops from the steamship Brooklyn into Fort Pickens. This order
could not go by land, but must take the longer and slower route
by sea. The first return news from the order was received just
one week before the fall of Sumter. The news itself was that the
officer commanding the Sabine, to which vessel the troops had
been transferred from the Brooklyn, acting upon some quasi
armistice of the late Administration, and of the existence of
which the present Administration, up to the time the order was
dispatched, had only too vague and uncertain rumors to fix
attention, had refused to land the troops. To now reinforce Fort
Pickens before a crisis would be reached at Fort Sumter, was
impossible, rendered so by the near exhaustion of provisions at
the latter named fort. In precaution against such a conjuncture
the Government had a few days before commenced preparing
an expedition, as well adapted as might be, to relieve Fort
Sumter, which expedition was intended to be ultimately used or
not, according to circumstances. The strongest anticipated case
for using it was now presented, and it was resolved to send it
forward as had been intended. In this contingency it was also
resolved to notify the Governor of South Carolina that he might
expect an attempt would be made to provision the fort, and that
if the attempt should not be resisted there would be no attempt
to throw in men, arms or ammunition, without further notice, or
in case of an attack upon the fort. This notice was accordingly
given, whereupon the fort was attacked and bombarded to its
fall, without even awaiting the arrival of the provisioning
expedition.
“It is thus seen that the assault upon and reduction of Fort
Sumter, was in no sense, a matter of self-defense on the part of
the assailants. They well knew that the garrison in the fort could
by no possibility commit aggression upon them; they knew they
were expressly notified that the giving of bread to the few brave
and hungry men of the garrison was all which would, on that
occasion, be attempted, unless themselves, by resisting so
much, should provoke more. They knew that this Government
desired to keep the garrison in the fort, not to assail them, but
merely to maintain visible possession, and thus to preserve the
Union from actual and immediate dissolution; trusting, as
hereinbefore stated, to time, discussion, and the ballot-box for
final adjustment, and they assailed and reduced the fort, for
precisely the reverse object, to drive out the visible authority of
the Federal Union, and thus force it to immediate dissolution;
that this was their object the Executive well understood, having
said to them in the Inaugural Address, ‘You can have no conflict
without being yourselves the aggressors.’ He took pains not only
to keep this declaration good, but also to keep the case so far
from ingenious sophistry as that the world should not
misunderstand it. By the affair at Fort Sumter, with its
surrounding circumstances, that point was reached. Then and
thereby the assailants of the Government began the conflict of
arms​—w ​ ithout a gun in sight, or in expectancy, to return their
fire, save only the few in the fort sent to that harbor years
before, for their own protection, and still ready to give that
protection in whatever was lawful. In this act, discarding all
else, they have forced upon the country the distinct issue,
immediate dissolution or blood, and this issue embraces more
than the fate of these United States. It presents to the whole
family of man the question whether a Constitutional Republic or
Democracy, a Government of the people, by the same people,
can or can not maintain its territorial integrity against its own
domestic foes. It presents the question whether discontented
individuals, too few in numbers to control the Administration
according to the organic law in any case, can always, upon the
pretenses made in this case, or any other pretenses, or
arbitrarily without any pretense, break up their Government,
and thus practically put an end to free government upon the
earth. It forces us to ask, ‘Is there in all republics this inherent
and fatal weakness?’ ‘Must a Government of necessity be too
strong for the liberties of its own people, or too weak to
maintain its own existence?’ So viewing the issue, no choice was
left but to call out the war power of the Government, and so to
resist the force employed for its destruction by force for its
preservation. The call was made, and the response of the
country was most gratifying, surpassing, in unanimity and spirit,
the most sanguine expectation. Yet none of the States,
commonly called Slave States, except Delaware, gave a
regiment through the regular State organization. A few
regiments have been organized within some others of those
States by individual enterprise, and received into the
Government service. Of course the seceded States so called,
and to which Texas had been joined about the time of the
inauguration, gave no troops to the cause of the Union. The
Border States, so called, were not uniform in their action, some
of them being almost for the Union, while in others, as in
Virginia, North Carolina, Tennessee, and Arkansas, the Union
sentiment was nearly repressed and silenced. The course taken
in Virginia was the most remarkable, perhaps the most
important. A Convention, elected by the people of that State to
consider this very question of disrupting the Federal Union, was
in session at the capitol of Virginia when Fort Sumter fell.
“To this body the people had chosen a large majority of
professed Union men. Almost immediately after the fall of
Sumter many members of that majority went over to the
original disunion minority, and with them adopted an ordinance
for withdrawing the State from the Union. Whether this change
was wrought by their great approval of the assault upon Sumter,
or their great resentment at the Government’s resistance to that
assault, is not definitely known. Although they submitted the
ordinance for ratification to a vote of the people, to be taken on
a day then somewhat more than a month distant, the
Convention, and the Legislature, which was also in session at
the same time and place, with leading men of the State, not
members of either, immediately commenced acting as if the
State was already out of the Union. They pushed military
preparations vigorously forward all over the State. They seized
the United States Armory at Harper’s Ferry, and the Navy Yard
at Gosport, near Norfolk. They received, perhaps invited into
their State, large bodies of troops, with their warlike
appointments, from the so-called seceded States.
“They formally entered into a treaty of temporary alliance
with the so-called Confederate States, and sent members to
their Congress at Montgomery, and finally they permitted the
insurrectionary Government to be transferred to their capitol at
Richmond. The people of Virginia have thus allowed this giant
insurrection to make its nest within her borders, and this
Government has no choice left but to deal with it where it finds
it, and it has the less to regret as the loyal citizens have, in due
form, claimed its protection. Those loyal citizens this
Government is bound to recognize and protect as being in
Virginia. In the Border States, so called, in fact the Middle
States, there are those who favor a policy which they call armed
neutrality, that is, an arming of those States to prevent the
Union forces passing one way or the disunion forces the other,
over their soil. This would be disunion completed. Figuratively
speaking, it would be the building of an impassable wall along
the line of separation, and yet not quite an impassable one, for
under the guise of neutrality it would tie the hands of the Union
men, and freely pass supplies from among them, to the
insurrectionists, which it could not do as an open enemy. At a
stroke it would take all the trouble off the hands of secession,
except only what proceeds from the external blockade. It would
do for the disunionists that which of all things they most desire,
feed them well, and give them disunion, without a struggle of
their own. It recognizes no fidelity to the Constitution, no
obligation to maintain the Union, and while very many who have
favored it are doubtless loyal citizens, it is, nevertheless, very
injurious in effect.
“Recurring to the action of the Government, it may be
stated that at first a call was made for seventy-five thousand
militia, and rapidly following this, a proclamation was issued for
closing the ports of the insurrectionary districts by proceedings
in the nature of a blockade. So far all was believed to be strictly
legal.
“At this point the insurrectionists announced their purpose
to enter upon the practice of privateering.
“Other calls were made for volunteers, to serve three years,
unless sooner discharged, and also for large additions to the
regular army and navy. These measures, whether strictly legal
or not, were ventured upon under what appeared to be a
popular demand and a public necessity, trusting then, as now,
that Congress would ratify them.
“It is believed that nothing has been done beyond the
constitutional competency of Congress. Soon after the first call
for militia it was considered a duty to authorize the commanding
general, in proper cases, according to his discretion, to suspend
the privilege of the writ of habeas corpus; or, in other words, to
arrest and detain, without resort to the ordinary processes and
forms of law, such individuals as he might deem dangerous to
the public safety. This authority has purposely been exercised,
but very sparingly. Nevertheless, the legality and propriety of
what has been done under it are questioned, and the attention
of the country has been called to the proposition, that one who
is sworn to take care that the laws be faithfully executed should
not himself violate them. Of course some consideration was
given to the questions of power and propriety before this matter
was acted upon. The whole of the laws, which were required to
be faithfully executed, were being resisted, and failing of
execution in nearly one-third of the States. Must they be
allowed to finally fail of execution, even had it been perfectly
clear that, by use of the means necessary to their execution,
some single law, made in such extreme tenderness of the
citizen’s liberty that practically it relieves more of the guilty than
the innocent, should, to a very great extent, be violated? To
state the question more directly, are all the laws but one to go
unexecuted, and the Government itself to go to pieces, lest that
one be violated? Even in such a case would not the official oath
be broken, if the Government should be overthrown when it was
believed that disregarding the single law would tend to preserve
it?
“But it was not believed that this question was presented. It
was not believed that any law was violated. The provision of the
Constitution, that the privilege of the writ of habeas corpus shall
not be suspended, unless when, in cases of rebellion or
invasion, the public safety may require it, is equivalent to a
provision that such privilege may be suspended when, in cases
of rebellion or invasion, the public safety does require it. It was
decided that we have a case of rebellion, and that the public
safety does require the qualified suspension of the privilege of
the writ, which was authorized to be made. Now, it is insisted
that Congress, and not the Executive, is vested with this power.
But the Constitution itself is silent as to which or who is to
exercise the power; and as the provision was plainly made for a
dangerous emergency, it cannot be believed that the framers of
the instrument intended that in every case the danger should
run its course until Congress could be called together, the very
assembling of which might be prevented, as was intended in
this case by the rebellion. No more extended argument is now
afforded, as an opinion at some length will probably be
presented by the Attorney-General. Whether there shall be any
legislation on the subject, and if so, what, is subject entirely to
the better judgment of Congress. The forbearance of this
Government had been so extraordinary, and so long continued,
as to lead some foreign nations to shape their action as if they
supposed the early destruction of our National Union was
probable. While this, on discovery, gave the Executive some
concern, he is now happy to say that the sovereignty and rights
of the United States are now everywhere practically respected
by foreign powers, and a general sympathy with the country is
manifested throughout the world.
“The reports of the Secretaries of the Treasury, War, and the
Navy, will give the information, in detail, deemed necessary and
convenient for your deliberation and action, while the Executive
and all the Departments will stand ready to supply omissions or
to communicate new facts considered important for you to
know.
“It is now recommended that you give the legal means for
making this contest a short and decisive one; that you place at
the control of the Government for the work, at least 400,000
men and $400,000,000; that number of men is about one-tenth
of those of proper ages within the regions where apparently all
are willing to engage, and the sum is less than a twenty-third
part of the money value owned by the men who seem ready to
devote the whole. A debt of $600,000,000 now is a less sum per
head than was the debt of our Revolution when we came out of
that struggle, and the money value in the country bears even a
greater proportion to what it was then than does the
population. Surely each man has as strong a motive now to
preserve our liberties, as each had then to establish them.
“A right result at this time will be worth more to the world
than ten times the men and ten times the money. The evidence
reaching us from the country, leaves no doubt that the material
for the work is abundant, and that it needs only the hand of
legislation to give it legal sanction, and the hand of the
Executive to give it practical shape and efficiency. One of the
greatest perplexities of the Government is to avoid receiving
troops faster than it can provide for them; in a word, the people
will save their Government if the Government will do its part
only indifferently well. It might seem at first thought to be of
little difference whether the present movement at the South be
called secession or rebellion. The movers, however, well
understand the difference. At the beginning they knew that they
could never raise their treason to any respectable magnitude by
any name which implies violation of law; they knew their people
possessed as much of moral sense, as much of devotion to law
and order, and as much pride in its reverence for the history and
government of their common country, as any other civilized and
patriotic people. They knew they could make no advancement
directly in the teeth of these strong and noble sentiments.
Accordingly they commenced by an insidious debauching of the
public mind; they invented an ingenious sophism, which, if
conceded, was followed by perfectly logical steps through all the
incidents of the complete destruction of the Union. The sophism
itself is that any State of the Union may, consistently with the
nation’s Constitution, and therefore lawfully and peacefully,
withdraw from the Union without the consent of the Union or of
any other State.
“The little disguise that the supposed right, is to be
exercised only for just cause, themselves to be the sole judge of
its justice, is too thin to merit any notice with rebellion. Thus
sugar-coated, they have been drugging the public mind of their
section for more than thirty years, and until at length they have
brought many good men to a willingness to take up arms
against the Government the day after some assemblage of men
have enacted the farcical pretence of taking their State out of
the Union, who could have been brought to no such thing the
day before. This sophism derives much, perhaps the whole of its
currency, from the assumption that there is some omnipotent
and sacred supremacy pertaining to a State, to each State of
our Federal Union. Our States have neither more nor less power
than that reserved to them in the Union by the Constitution, no
one of them ever having been a State out of the Union. The
original ones passed into the Union before they cast off their
British Colonial dependence, and the new ones came into the
Union directly from a condition of dependence, excepting Texas,
and even Texas, in its temporary independence, was never
designated as a State. The new ones only took the designation
of States on coming into the Union, while that name was first
adopted for the old ones in and by the Declaration of
Independence. Therein the United Colonies were declared to be
free and independent States. But even then the object plainly
was not to declare their independence of one another of the
Union, but directly the contrary, as their mutual pledge and their
mutual action before, at the time, and afterward, abundantly
show. The express plight of faith by each and all of the original
thirteen States in the Articles of Confederation two years later
that the Union shall be perpetual, is most conclusive. Having
never been States either in substance or in name outside of the
Union, whence this magical omnipotence of State rights,
asserting a claim of power to lawfully destroy the Union itself?
Much is said about the sovereignty of the States, but the word
even is not in the National Constitution, nor, as is believed, in
any of the State constitutions. What is sovereignty in the
political sense of the word? Would it be far wrong to define it a
political community without a political superior? Tested by this,
no one of our States, except Texas, was a sovereignty, and even
Texas gave up the character on coming into the Union, by which
act she acknowledged the Constitution of the United States; and
the laws and treaties of the United States, made in pursuance of
States, have their status in the Union, made in pursuance of the
Constitution, to be for her the supreme law. The States have
their status in the Union, and they have no other legal status. If
they break from this, they can only do so against law and by
revolution. The Union and not themselves, separately procured
their independence and their liberty by conquest or purchase.
The Union gave each of them whatever of independence and
liberty it has. The Union is older than any of the States, and, in
fact, it created them as States. Originally, some dependent
Colonies made the Union, and in turn the Union threw off their
old dependence for them, and made them States, such as they
are. Not one of them ever had a State constitution independent
of the Union. Of course it is not forgotten that all the new
States formed their constitutions before they entered the Union;
nevertheless, dependent upon, and preparatory to coming into
the Union. Unquestionably the States have the powers and
rights reserved to them in and by the National Constitution.
“But among these surely are not included all conceivable
powers, however mischievous or destructive, but at most such
only as were known in the world at the time as governmental
powers, and certainly a power to destroy the Government itself
had never been known as a governmental, as a merely
administrative power. This relative matter of National power and
State rights as a principle, is no other than the principle of
generality and locality. Whatever concerns the whole should be
conferred on the whole General Government, while whatever
concerns only the State should be left exclusively to the State.
This is all there is of original principle about it. Whether the
National Constitution, in defining boundaries between the two,
has applied the principle with exact accuracy, is not to be
questioned. We are all bound by that defining without question.
What is now combatted is the position that secession is
consistent with the Constitution, is lawful and peaceful. It is not
contended that there is any express law for it, and nothing
should ever be implied as law which leads to unjust or absurd
consequences. The nation purchased with money the countries
out of which several of these States were formed. Is it just that
they shall go off without leave and without refunding? The
nation paid very large sums in the aggregate, I believe nearly a
hundred millions, to relieve Florida of the aboriginal tribes. Is it
just that she shall now be off without consent, or without any
return? The nation is now in debt for money applied to the
benefit of these so-called seceding States, in common with the
rest. Is it just, either that creditors shall go unpaid, or the
remaining States pay the whole? A part of the present National
debt was contracted to pay the old debt of Texas. Is it just that
she shall leave and pay no part of this herself? Again, if one
State may secede, so may another, and when all shall have
seceded none is left to pay the debts. Is this quite just to
creditors? Did we notify them of this sage view of ours when we
borrowed their money? If we now recognize this doctrine by
allowing the seceders to go in peace, it is difficult to see what
we can do if others choose to go, or to extort terms upon which
they will promise to remain. The seceders insist that our
Constitution admits of secession. They have assumed to make a
National Constitution of their own, in which, of necessity, they
have either discarded or retained the right of secession, as they
insist exists in ours. If they have discarded it, they thereby
admit that on principle it ought not to exist in ours; if they have
retained it, by their own construction of ours that shows that to
be consistent, they must secede from one another whenever
they shall find it the easiest way of settling their debts, or
effecting any other selfish or unjust object. The principle itself is
one of disintegration, and upon which no Government can
possibly endure. If all the States save one should assert the
power to drive that one out of the Union, it is presumed the
whole class of seceder politicians would at once deny the power,
and denounce the act as the greatest outrage upon State rights.
But suppose that precisely the same act, instead of being called
driving the one out, should be called the seceding of the others
from that one, it would be exactly what the seceders claim to
do, unless, indeed, they made the point that the one, because it
is a minority, may rightfully do what the others, because they
are a majority, may not rightfully do. These politicians are
subtle, and profound in the rights of minorities. They are not
partial to that power which made the Constitution, and speaks
from the preamble, calling itself, ‘We, the people.’ It may be well
questioned whether there is to-day a majority of the legally
qualified voters of any State, except, perhaps, South Carolina, in
favor of disunion. There is much reason to believe that the
Union men are the majority in many, if not in every one of the
so-called seceded States. The contrary has not been
demonstrated in any one of them. It is ventured to affirm this,
even of Virginia and Tennessee, for the result of an election held
in military camps, where the bayonets are all on one side of the
question voted upon, can scarcely be considered as
demonstrating popular sentiment. At such an election all that
large class who are at once for the Union and against coercion
would be coerced to vote against the Union. It may be affirmed,
without extravagance, that the free institutions we enjoy have
developed the powers and improved the condition of our whole
people beyond any example in the world. Of this we now have a
striking and impressive illustration. So large an army as the
Government has now on foot was never before known, without
a soldier in it but who has taken his place there of his own free
choice. But more than this, there are many single regiments
whose members, one and another, possess full practical
knowledge of all the arts, sciences, professions, and whatever
else, whether useful or elegant, is known in the whole world,
and there is scarcely one from which there could not be
selected a President, a Cabinet, a Congress, and perhaps a
Court, abundantly competent to administer the Government
itself. Nor do I say this is not true also in the army of our late
friends, now adversaries, in this contest. But it is so much better
the reason why the Government which has conferred such
benefits on both them and us should not be broken up.
Whoever in any section proposes to abandon such a
Government, would do well to consider in deference to what
principle it is that he does it. What better he is likely to get in its
stead, whether the substitute will give, or be intended to give so
much of good to the people. There are some foreshadowings on
this subject. Our adversaries have adopted some declarations of
independence in which, unlike our good old one penned by
Jefferson, they omit the words, ‘all men are created equal.’
Why? They have adopted a temporary National Constitution, in
the preamble of which, unlike our good old one signed by
Washington, they omit, ‘We, the people,’ and substitute, ‘We,
the deputies of the sovereign and independent States.’ Why?
Why this deliberate pressing out of view the rights of men and
the authority of the people? This is essentially a people’s
contest. On the side of the Union it is a struggle for maintaining
in the world that form and substance of Government whose
leading object is to elevate the condition of men, to lift artificial
weights from all shoulders, to clear the paths of laudable pursuit
for all, to afford all an unfettered start and a fair chance in the
race of life, yielding to partial and temporary departures from
necessity. This is the leading object of the Government for
whose existence we contend.
“I am most happy to believe that the plain people
understand and appreciate this. It is worthy of note that while
in this, the Government’s hour of trial, large numbers of those in
the army and navy who have been favored with the offices,
have resigned and proved false to the hand which pampered
them, not one common soldier or common sailor is known to
have deserted his flag. Great honor is due to those officers who
remained true despite the example of their treacherous
associates, but the greatest honor and the most important fact
of all, is the unanimous firmness of the common soldiers and
common sailors. To the last man, so far as known, they have
successfully resisted the traitorous efforts of those whose
commands but an hour before they obeyed as absolute law.
This is the patriotic instinct of plain people. They understand
without an argument that the destroying the Government which
was made by Washington means no good to them. Our popular
Government has often been called an experiment. Two points in
it our people have settled: the successful establishing and the
successful administering of it. One still remains. Its successful
maintenance against a formidable internal attempt to overthrow
it. It is now for them to demonstrate to the world that those
who can fairly carry an election, can also suppress a rebellion;
that ballots are the rightful and peaceful successors of bullets,
and that when ballots have fairly and constitutionally decided,
there can be no successful appeal except to ballots themselves
at succeeding elections. Such will be a great lesson of peace,
teaching men that what they cannot take by an election, neither
can they take by a war, teaching all the folly of being the
beginners of a war.
“Lest there should be some uneasiness in the minds of
candid men as to what is to be the course of the Government
toward the Southern States after the rebellion shall have been
suppressed, the Executive deems it proper to say it will be his
purpose then, as ever, to be guided by the Constitution and the
laws, and that he probably will have no different understanding
of the powers and duties of the Federal Government relatively
to the rights of the United States and the people under the
Constitution than that expressed in the Inaugural Address. He
desires to preserve the Government that it may be administered
for all, as it was administered by the men who made it. Loyal
citizens everywhere have a right to claim this of their
Government, and the Government has no right to withhold or
neglect it. It is not perceived that in giving it there is any
coercion, conquest or subjugation in any sense of these terms.
“The Constitution provided, and all the States have accepted
the provision, ‘that the United States shall guarantee to every
State in this Union a Republican form of government,’ but if a
State may lawfully go out of the Union, having done so, it may
also discard the Republican form of Government. So that to
prevent its going out is an indispensable means to the end of
maintaining the guaranty mentioned; and when an end is lawful
and obligatory, the indispensable means to it are also lawful and
obligatory.
“It was with the deepest regret that the Executive found the
duty of employing the war power. In defence of the Government
forced upon him, he could but perform this duty or surrender
the existence of the Government. No compromise by public
servants could in this case be a cure, not that compromises are
not often proper, but that no popular government can long
survive a marked precedent, that those who carry an election
can only save the Government from immediate destruction by
giving up the main point upon which the people gave the
election. The people themselves and not their servants can
safely reverse their own deliberate decisions.
“As a private citizen the Executive could not have consented
that these institutions shall perish, much less could he, in
betrayal of so vast and so sacred a trust as these free people
had confided to him. He felt that he had no moral right to
shrink, nor even to count the chances of his own life in what
might follow.
“In full view of his great responsibility, he has so far done
what he has deemed his duty. You will now, according to your
own judgment, perform yours. He sincerely hopes that your
views and your actions may so accord with his as to assure all
faithful citizens who have been disturbed in their rights, of a
certain and speedy restoration to them, under the Constitution
and laws; and having thus chosen our cause without guile, and
with pure purpose, let us renew our trust in God, and go
forward without fear and with manly hearts.
“July 4, 1861. Abraham Lincoln.”

This document, it will be observed, sets forth in temperate


language the facts bearing upon the rebellion in its then stage​—​facts
so stated that the common people could readily comprehend the
exact situation of affairs. Such a message, always in place, was
never more needed than at a juncture when​—a ​ s seemed not
altogether impossible to many​—a ​ n appeal might yet have to be
made again and again to the great mass of the people for men and
money to maintain the unity of the nation. It may be safely asserted,
that the messages of none of our Presidents have been so generally
read and so thoroughly mastered by the average mind, as those of
Mr. Lincoln, himself the tribune of the people.
Congress granted five hundred millions in money, and directed a
call for five hundred thousand volunteers for the army; made
provisions for a popular national loan; revised the tariff; passed a
direct tax bill; adopted measures, moderate in their scope, for the
confiscation of rebel property; legalized the official acts of the
President during the emergency in which the country had been
placed; and the House of Representatives, with but two dissentients,
passed the following resolution:

“Resolved, By the House of Representatives of the Congress


of the United States, That the present deplorable civil war has
been forced upon the country by the disunionists of the
Southern States, now in revolt against the Constitutional
Government, and in arms around the capital; that in this
national emergency Congress, banishing all feeling of mere
passion or resentment, will recollect only its duty to the whole
country; that this war is not waged on our part in any spirit of
oppression, nor for any purpose of conquest or subjugation, nor
purpose of authorizing or interfering with the rights or
established institutions of the States, but to defend and
maintain the supremacy of the Constitution, and to preserve the
Union, with all the dignities, equality, and rights of the several
States unimpaired, and that as soon as these objects are
accomplished the war ought to cease.”
On the 21st of July, the Army of the Union, under the direct
command of General McDowell, and the general supervision of the
veteran Scott​—​from whose onward movement against the rebels in
Virginia so much had been expected​—​met with a serious reverse at
Bull Run. They went forth, exulting in victory as certain; they came
back a panic-stricken mob. For an instant, despondency took
possession of every loyal heart; all manner of vague fears seized the
people; Washington would be captured; the cause was lost.
It was but for an instant, however. The rebound came.
Washington which might easily have been captured and sacked, had
the rebels known how to improve their success, was securely
fortified and amply garrisoned. One did not then comprehend what
now the most concede​—​that Bull Run was a necessary discipline​—a ​
school in which all learned somewhat​—​though, unfortunately, not all
of us as much as we should. That came later.
CHAPTER IX.
CLOSE OF 1861.

Elation of the Rebels​—​Davis’s boast​—​McClellan appointed


Commander of Potomac Army​—​Proclamation of a
National Fast​—I​ ntercourse with rebels forbidden​—​
Fugitive slaves​—​Gen. Butler’s views​—​Gen. McClellan’s
letter from Secretary Cameron​—A​ ct of August 6th, 1861​
—​Gen. Fremont’s order​—​Letter of the President
modifying the same​—I​ nstructions to Gen. Sherman​—​
Ball’s Bluff​—​Gen. Scott’s retirement​—A ​ rmy of the
Potomac.

The victory of the conspirators at Bull Run, as was to have been


expected, elated them no little. Their President in his message was
supercilious and confident. Lauding the prowess and determination
of his confederates, he said:
“To speak of subjugating such a people, so united and
determined, is to speak in a language incomprehensible to them: to
resist attack on their rights or their liberties is with them an instinct.
Whether this war shall last one, or three, or five years, is a problem
they leave to be solved by the enemy alone. It will last till the enemy
shall have withdrawn from their borders; till their political rights,
their altars, and their homes are freed from invasion. Then, and then
only, will they rest from this struggle to enjoy in peace, the blessings
which, with the favor of Providence, they have secured by the aid of
their own strong hearts and steady arms.”
On the 25th of July, a new commander was assigned to the
Army of the Potomac, upon the warm recommendation of Gen.
Scott; George B. McClellan, who had already become favorably
known from his conducting a successful campaign in Western
Virginia. With the extravagance so characteristic of the American
people, this commander​—w ​ hose laurels were yet to be won​—w ​ as
hailed as a young Napoleon, lauded to the skies, and failure under
him regarded as an utter impossibility.
And the General betook himself to the organizing, disciplining,
and supplying his army, to which large accessions were continually
making from week to week.
On the 12th day of August was issued the following
proclamation:

“Whereas, A joint committee of both Houses of Congress has


waited on the President of the United States, and requested him
to ‘recommend a day of public humiliation, prayer, and fasting,
to be observed by the people of the United States with religious
solemnities, and the offering of fervent supplications to Almighty
God for the safety and welfare of these States, His blessings on
their arms, and a speedy restoration of peace.’
“And whereas, It is fit and becoming in all people, at all
times, to acknowledge and revere the Supreme Government of
God; to bow in humble submission to his chastisements; to
confess and deplore their sins and transgressions, in the full
conviction that the fear of the Lord is the beginning of wisdom,
and to pray, with all fervency and contrition, for the pardon of
their past offences, and for a blessing upon their present and
prospective action.
“And whereas, When our own beloved country, once, by the
blessing of God, united, prosperous, and happy, is now afflicted
with faction and civil war, it is peculiarly fit for us to recognize
the hand of God in this terrible visitation, and, in sorrowful
remembrance of our own faults and crimes as a nation, and as
individuals, to humble ourselves before Him, and to pray for His
mercy​—​to pray that we may be spared further punishment,
though most justly deserved; that our arms may be blessed and
made effectual for the re-establishment of law, order, and peace
throughout the wide extent of our country; and that the
inestimable boon of civil and religious liberty, earned under His
guidance and blessing by the labors and sufferings of our
fathers, may be restored in all its original excellence;
“Therefore, I, Abraham Lincoln, President of the United
States, do appoint the last Thursday in September next as a day
of humiliation, prayer, and fasting for all the people of the
nation. And I do earnestly recommend to all the people, and
especially to all ministers and teachers of religion, of all
denominations, and to all heads of families, to observe and keep
that day, according to their several creeds and modes of
worship, in all humility, and with all religious solemnity, to the
end that the united prayer of the nation may ascend to the
Throne of Grace, and bring down plentiful blessings upon our
country.
“In testimony whereof, I have hereunto set my hand, and
caused the seal of the United States to be affixed, this 12th day
of August, A. D. 1861, and of the Independence of the United
States of America the eighty-sixth.
“By the President: Abraham Lincoln.
“William H. Seward, Secretary of State.”

And four days later the following:

“Whereas, On the 15th day of April, the President of the


United States, in view of an insurrection against the laws,
Constitution, and Government of the United States, which had
broken out within the States of South Carolina, Georgia,
Alabama, Florida, Mississippi, Louisiana, and Texas, and in
pursuance of the provisions of an act entitled an act to provide
for calling forth the militia to execute the laws of the Union,
suppress insurrections and repel invasions, and to repeal the act
now in force for that purpose, approved February 28th, 1795,
did call forth the militia to suppress said insurrection and cause
the laws of the Union to be duly executed​—a ​ nd the insurgents
have failed to disperse by the time directed by the President;
AND WHEREAS, such insurrection has since broken out and yet
exists within the States of Virginia, North Carolina, Tennessee,
and Arkansas; AND WHEREAS, the insurgents in all the said States
claim to act under authority thereof, and such claim is not
discarded or repudiated by the persons exercising the functions
of government in such State or States, or in the part or parts
thereof, in which such combinations exist, nor has such
insurrection been suppressed by said States.
“Now, therefore, I, Abraham Lincoln, President of the United
States, in pursuance of the Act of Congress approved July 13th,
1861, do hereby declare that the inhabitants of the said States
of Georgia, South Carolina, Tennessee, Alabama, Louisiana,
Texas, Arkansas, Mississippi, and Florida, except the inhabitants
of that part of the State of Virginia lying west of the Allegheny
Mountains, and of such other parts of that State and the other
States hereinbefore named as may maintain a loyal adhesion to
the Union and the Constitution, or may be, from time to time
occupied and controlled by the forces of the United States
engaged in the dispersion of said insurgents, are in a state of
insurrection against the United States, and that all commercial
intercourse between the same and the inhabitants thereof, with
the exception aforesaid, and the citizens of other States and
other parts of the United States, is unlawful, and will remain
unlawful until such insurrection shall cease or has been
suppressed; that all goods and chattels, wares and
merchandise, coming from any of the said States, with the
exceptions aforesaid, into other parts of the United States,
without the special license and permission of the President,
through the Secretary of the Treasury, or proceeding to any of
the said States, with the exception aforesaid, by land or water,
together with the vessel or vehicle conveying the same, or
conveying persons to and from the said States, with the said
exceptions, will be forfeited to the United States; and that, from
and after fifteen days from the issuing of this proclamation, all
ships and vessels belonging, in whole or in part, to any citizen
or inhabitant of any of the said States, with the said exceptions,
found at sea in any part of the United States, will be forfeited to
the United States; and I hereby enjoin upon all District
Attorneys, Marshals, and officers of the revenue of the military
and naval forces of the United States, to be vigilant in the
execution of the said act, and in the enforcement of the
penalties and forfeitures imposed or declared by it, leaving any
party who may think himself aggrieved thereby, to his
application to the Secretary of the Treasury for the remission of
any penalty or forfeiture, which the said Secretary is authorized
by law to grant, if in his judgment, the special circumstances of
any case shall require such a remission.
“In witness whereof, I have hereunto set my hand, and
caused the seal of the United States to be affixed.
“Done in the City of Washington, this, the 16th day of
August, in the year of our Lord one thousand eight hundred and
sixty-one, and of the Independence of the United States of
America the eighty-sixth.
“By the President: Abraham Lincoln.
“William H. Seward, Secretary of State.”

The question as to the disposition to be made of the slaves of


rebel masters presented itself early in the contest, and it was at
once perceived that its settlement would be attended with no little
embarrassment.
Welcome to our website – the ideal destination for book lovers and
knowledge seekers. With a mission to inspire endlessly, we offer a
vast collection of books, ranging from classic literary works to
specialized publications, self-development books, and children's
literature. Each book is a new journey of discovery, expanding
knowledge and enriching the soul of the reade

Our website is not just a platform for buying books, but a bridge
connecting readers to the timeless values of culture and wisdom. With
an elegant, user-friendly interface and an intelligent search system,
we are committed to providing a quick and convenient shopping
experience. Additionally, our special promotions and home delivery
services ensure that you save time and fully enjoy the joy of reading.

Let us accompany you on the journey of exploring knowledge and


personal growth!

textbookfull.com

You might also like