Download the full version of the ebook at

https://ebookultra.com

Puppet Best Practices 1 (Early Release)

Edition Chris Barbour

https://ebookultra.com/download/puppet-best-
practices-1-early-release-edition-chris-barbour/

Explore and download more ebook at https://ebookultra.com

 Recommended digital products (PDF, EPUB, MOBI) that
you can download immediately if you are interested.

GitHub 1 (Early Release) Edition Chris Dawson

https://ebookultra.com/download/github-1-early-release-edition-chris-
dawson/

ebookultra.com

Two Scoops of Django Best Practices For Django 1 5 1st

Edition Daniel Greenfeld

https://ebookultra.com/download/two-scoops-of-django-best-practices-
for-django-1-5-1st-edition-daniel-greenfeld/

ebookultra.com

Android Best Practices 1st Edition Godfrey Nolan

https://ebookultra.com/download/android-best-practices-1st-edition-
godfrey-nolan/

ebookultra.com

The Regulation of Unfair Commercial Practices under EC

Directive 2005 29 Stephen Weatherill

https://ebookultra.com/download/the-regulation-of-unfair-commercial-
practices-under-ec-directive-2005-29-stephen-weatherill/

ebookultra.com
Accounting best practices 3rd ed Edition Steven M Bragg

https://ebookultra.com/download/accounting-best-practices-3rd-ed-
edition-steven-m-bragg/

ebookultra.com

Healthcare Information Systems Second Edition Best

Practices Kevin Beaver

https://ebookultra.com/download/healthcare-information-systems-second-
edition-best-practices-kevin-beaver/

ebookultra.com

Ruby best practices 1st ed Edition Gregory T Brown

https://ebookultra.com/download/ruby-best-practices-1st-ed-edition-
gregory-t-brown/

ebookultra.com

Best Practices for Effective Boards 1st Edition E. Lebron

Fairbanks

https://ebookultra.com/download/best-practices-for-effective-
boards-1st-edition-e-lebron-fairbanks/

ebookultra.com

Puppet Cookbook 3rd Edition Thomas Uphill

https://ebookultra.com/download/puppet-cookbook-3rd-edition-thomas-
uphill/

ebookultra.com
Puppet Best Practices 1 (Early Release) Edition Chris
Barbour Digital Instant Download
Author(s): Chris Barbour
ISBN(s): 9781491923009, 1491923008
Edition: 1 (Early Release)
File Details: PDF, 1.56 MB
Year: 2015
Language: english
Puppet Best Practices

Chris Barbour
Puppet Best Practices
by Chris Barbour
Copyright © 2010 Chris Barbour. All rights reserved.
Printed in the United States of America.
Published by O’Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopol, CA 95472.
O’Reilly books may be purchased for educational, business, or sales promotional use. Online editions are
also available for most titles (http://safaribooksonline.com). For more information, contact our corporate/
institutional sales department: 800-998-9938 or corporate@oreilly.com.
Editor: Brian Anderson Indexer: FIX ME!
Production Editor: FIX ME! Cover Designer: Karen Montgomery
Copyeditor: FIX ME! Interior Designer: David Futato
Proofreader: FIX ME! Illustrator: Rebecca Demarest

January -4712: First Edition

Revision History for the First Edition:

2015-03-13: First early release

2015-06-04: Early release revision 2

See http://oreilly.com/catalog/errata.csp?isbn=0636920038528 for release details.

Nutshell Handbook, the Nutshell Handbook logo, and the O’Reilly logo are registered trademarks of O’Reilly
Media, Inc. !!FILL THIS IN!! and related trade dress are trademarks of O’Reilly Media, Inc.
Many of the designations used by manufacturers and sellers to distinguish their products are claimed as
trademarks. Where those designations appear in this book, and O’Reilly Media, Inc. was aware of a trademark
claim, the designations have been printed in caps or initial caps.
While every precaution has been taken in the preparation of this book, the publisher and authors assume
no responsibility for errors or omissions, or for damages resulting from the use of the information contained
herein.

ISBN: 063-6-920-03852-8
[?]
 Table of Contents

1. Forward. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
What’s new? 1
About best practices 2
About this pre-release 2
Preface. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iii

2. The Puppet Design Philosophy. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

Declarative Code 9
What is declarative code anyway? 10
A practical example 12
Non-Declarative Code with Puppet 16
Idempotency 18
Side Effects 19
Resource Level Idempotence 20
Run Level Idempotence 22
Non-deterministic code 23
Stateless 24
Sources of State 26
Summary 28

3. Code and Data; High Level Design. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

Code and Data 29
Application Logic 30
Business Logic 31
Site Specific Data 31
Node Specific Data 31
Service Data 32
Breaking it down 32
Application Logic & Puppet Modules 33

iii
 Business logic with Roles and Profiles 36
Hiera & Site Specific Data 39
Node Classification 40
Exported Resources & Service Discovery 41
Summary 42

4. Coding Practices. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
The Style Guide 45
Coding Principles 46
KISS 46
The Single Responsibility Principle 48
Seperation of Concerns 49
Interface Driven Design 51
Don’t Repeat Yourself (the DRY principle) 52
General Coding Recommendations 55
The balance of Code and Resources 55
Conditional Logic 57
Selectors 59
Variables 61
Variable Naming 61
Referencing Variables 61
Other Variable Use Cases 66
Function Calls 66
Functions for logging and Debugging 67
String manipulation functions 68
Path Manipulation 68
Input validation functions 68
Catalog tests 70
Iteration 72
Iteration with Puppet 3 72
Iteration with Puppet 4 and the Future Parser 73
Generating lists 75
Data Transformation 76
Templates 78
ERB Templates 78
EPP Templates 79
EPP vs. ERB 80
Template abuse 80
The puppet::file defined type 80
Other Language Features 81
Summary 81

iv | Table of Contents
5. Puppet Module Design. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83
Design modules for public consumption 84
Using public modules 84
Picking good modules 84
Module checklist 85
Module Applicability 85
Contributing Modules 86
Planning and scoping your module 87
Basic Module Layout 87
manifests/init.pp; the module entry point 88
An example init class 89
Parameterizing your module 91
Input validation 95
params.pp pattern 96
Module data sources; the alternative to params.pp 98
Subclasses 99
Subclass relationships 99
Subclass containment 101
Interfacing with subclasses. 105
Defined Types 108
Iteration and DRY with defined types 108
Module Interfaces with defined types 108
Providing services with defined types 109
Defined types for simplifying complex tasks 111
Interacting with the rest of the module 112
Documentation 113
Markdown 113
In-line documentation 115
Rake tasks 116
Testing 116
Rspec 117
Acceptance testing 120
Module testing best practices 121
Continuous Integration 122
Dependencies 122
Summary 123

6. Built-in Types. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125

Table of Contents | v
 CHAPTER 1
Forward

First of all, I’d like to thank you for taking the time to read this early preview of Puppet
Best Practices. The reception to the book has been amazing, and we’ve received great
feedback about our initial update.

What’s new?
This months update includes two new chapters. Chapter 4 focusing on coding best
practices and the features of the Puppet DSL. This chapter explores iterators, variables,
and conditional logic among other language features. It also discusses coding principles
as they apply to Puppet.
Chapter 5 focuses on the development of Puppet modules. In this chapter we explore
module structure, scoping, the use of defined types, and module testing.
Puppet 4.0 was officially launched following our previous pre-release. We had already
been discussing practices relating to Puppet 4. This update significantly extends our
Puppet 4 coverage.
Puppet 4 is our target platform for this book, however we expect that a lot of sites will
continue to use Puppet 3 for the foreseeable future. We attempt to address practices for
both major releases of Puppet, directly exploring differences between the two releases
where applicable. This should both help you handle your current requirements, and
plan your approach for the future.
The introduction of the future parser and the release of Hiera 2 are both fairly significant
changes for Puppet 4. We are excited about the introduction of iterators to the Puppet
DSL, as well as module data sources, and the improvements for data handling. We have
however, found that most best practices apply to both releases of Puppet.

1
Best practices will become even more important to this release; the new features of
Puppet 4 are powerful, but carry a lot of potential for mis-use. This book should prove
to be invaluable as you upgrade, grow, and improve your environment.

About best practices

Best practices books are the result of years of professional experience. Unlike purely
technical works, there often isn’t always objectively correct best practices answer for
every conceivable situation. By exposing this early release to your criticism, I hope to
identify weaknesses in this book; sections that can be improved upon. If you find any‐
thing in this book confusing or incomplete, I encourage you to reach out for clarifica‐
tion. My goal is to produce the best work I possibly can. Your feedback is invaluable to
this process, and very much appreciated.
My objective for this book is to share professional experience, help other IT professio‐
nals solve problems, and to improve the overall quality of code and infrastructure de‐
ployed in the real world.

About this pre-release

This work has not yet been professionally edited, and I hope you will forgive the gram‐
mar, syntax, and formatting errors that are present in this initial draft. Our goal is to
document the core concepts of this book first, and then correct the flaws iteratively.
What you currently see here is the product of passion and many sleepless nights fueled
only by caffeine and determination.
While all feedback is sincerely appreciated, the most valuable feedback you can provide
are your thoughts regarding the concepts and recommendations provided by this book.
Do you find the discussion of programming principles useful? Do you disagree with
any of the recommendations, and if so why? Is the material easy to understand, or
confusing?
If you have any feedback, I request that you please reach out. Please feel free to email
me at puppet.best.practices@gmail.com
Sincerely,
Chris Barbour, Author, Puppet Best Practices

2 | Chapter 1: Forward
 Preface

This book is a work in progress – new chapters will be added as they are written. We
welcome feedback – if you spot any errors or would like to suggest improvements, please
email the author at puppet.best.practices@gmail.com.

In this book, we discuss on how to build and deploy highly maintainable Puppet code,
with a focus on avoiding and eliminating technical debt.
Puppet is by far the most popular configuration management and automation platform
available today, and is quickly being adopted by companies of all sizes, from single
platform startups to established enterprises that may be using it to manage half a dozen
different operating systems.
This book does not attempt to explain the basics of using Puppet; instead we will be
looking at how to use Puppet most effectively. This will include discussions about the
design decisions behind Puppet, an exploration of how to organize code and data, a
look at many common features of Puppet, and discussions about common pitfalls and
traps when deploying Puppet infrastructure.

Who Should Read This Book

This book is intended for readers who have some basic familiarity with Puppet and are
interested in improving their understanding of Puppet and the surrounding ecosystem
of tools. The concepts described in this book are appropirate for Puppet novices and
experts alike, however readers who are new to Puppet should consider using this as a
suppliment to Puppet Labs training or other introductory work that can provide a
foundational understanding of Puppet.
This book is appropriate for all professionals working with Puppet, regardless of wheth‐
er you are responsible for architecting Puppet infrastructure or responsible for writing
a module or two to support a single application.

iii
The information contained in this book will be invaluable to both green and brown field
deployments of Puppet. If you are building a new environment, we will discuss how to
lay a solid foundation that provides flexibility to grow and change. If you already have
an exiting environment or are inheriting an environment, we will explore useful strate‐
gies to eliminate many pain points in your code base and improve upon what you have.

Why I Wrote This Book

This book draws heavily from experiences as a Puppet consultant. The folks I’ve met
while deploying Puppet have been very bright and talented individuals who were quick
to learn the features of Puppet, and could often find very innovative ways of using those
features to solve immediate problems, and to produce code that is impossible to un‐
derstand.
Almost universlly, sites deploying Puppet have grown both quickly and organically. The
consequences of design decisions made early in development often had reprocussions
that were not obvious when they were originally made. As code became established and
moved to production, it became harder and harder to correct those problems without
risking the stability of the site that code manages.
None of these problems are specific to Puppet. As with any CFM solution, the amount
of code being maintained will grow with the size of the site. It will often dwarf any other
single codebase maintained by your operations team.
This book highlights design patterns, both good and bad, that can be used when building
Puppet environments and discusses the impact of each decision. The coding patterns
contained in this book will help you design code that can be extended, maintained, and
supported not only by yourself, but by the people who may inherit your work down the
road.

A Word on Puppet Today

Puppet Best Practices have changed significantly since Pupet’s early releases. In some
ways, Puppet is much easier to work with and support. For example, parameterized
modules and automatic data bindings have made it much simpler to re-use 3rd party
modules. On the other hand, Puppet has added many new features and design patterns
since then, and the demarcation points between various systems aren’t always clear.
Best practices will continue to evolve. Puppet 4 will introduce new syntax and features.
Common complaints will be resolved. New issues will be discovered.
The goal of this book is not just to help you solve the problems you may be experiencing
today, but also to help prepare you for the future.

iv | Preface
Navigating This Book
This book is organized roughly as follows:

• Chapters 1 and 2 discuss design concepts that drive the recommendations made
throughout this book.
• Chapters 3 through 8 explore major features of Puppet such as Hiera in depth and
provide many concrete recommendations for each component.
• Chapter 9 and 10 discusses release management practices and development tools.
• Chapter 11 and 13 discuss advanced topics relating to extending Puppet and other
infrastructure management tools that may be useful to your site.

This book is organized so that it can be read front to back, however most of the chapters
in this book are fairly self contained and will provide references to other topics where
appropriate.
I strongly encourage you to start with chapters 2 and 3. From there, feel free to skip
around if you’re the impatient type, or continue reading through to gain a wholistic
understanding of the Puppet infrastructure. Once you’ve read through this book, it is
my sincere hope that you will return to individual sections when needed to address a
difficult problem, to refresh your knowledge, or to pickup strategies that may have been
missed the first time through.

Online Resources
• https://docs.puppetlabs.com/
• https://ask.puppetlabs.com

Conventions Used in This Book

The following typographical conventions are used in this book:
Italic
Indicates new terms, URLs, email addresses, filenames, and file extensions.
Constant width
Used for program listings, as well as within paragraphs to refer to program elements
such as variable or function names, databases, data types, environment variables,
statements, and keywords.
Constant width bold
Shows commands or other text that should be typed literally by the user.

Preface | v
Constant width italic
Shows text that should be replaced with user-supplied values or by values deter‐
mined by context.

This icon signifies a tip, suggestion, or general note.

This icon indicates a warning or caution.

Using Code Examples

Supplemental material (code examples, exercises, etc.) is available for download at
https://github.com/oreillymedia/title_title.
This book is here to help you get your job done. In general, if example code is offered
with this book, you may use it in your programs and documentation. You do not need
to contact us for permission unless you’re reproducing a significant portion of the code.
For example, writing a program that uses several chunks of code from this book does
not require permission. Selling or distributing a CD-ROM of examples from O’Reilly
books does require permission. Answering a question by citing this book and quoting
example code does not require permission. Incorporating a significant amount of ex‐
ample code from this book into your product’s documentation does require permission.
We appreciate, but do not require, attribution. An attribution usually includes the title,
author, publisher, and ISBN. For example: “Book Title by Some Author (O’Reilly).
Copyright 2012 Some Copyright Holder, 978-0-596-xxxx-x.”
If you feel your use of code examples falls outside fair use or the permission given above,
feel free to contact us at permissions@oreilly.com.

Safari® Books Online

Safari Books Online is an on-demand digital library that
delivers expert content in both book and video form from
the world’s leading authors in technology and business.

vi | Preface
Technology professionals, software developers, web designers, and business and crea‐
tive professionals use Safari Books Online as their primary resource for research, prob‐
lem solving, learning, and certification training.
Safari Books Online offers a range of product mixes and pricing programs for organi‐
zations, government agencies, and individuals. Subscribers have access to thousands of
books, training videos, and prepublication manuscripts in one fully searchable database
from publishers like O’Reilly Media, Prentice Hall Professional, Addison-Wesley Pro‐
fessional, Microsoft Press, Sams, Que, Peachpit Press, Focal Press, Cisco Press, John
Wiley & Sons, Syngress, Morgan Kaufmann, IBM Redbooks, Packt, Adobe Press, FT
Press, Apress, Manning, New Riders, McGraw-Hill, Jones & Bartlett, Course Technol‐
ogy, and dozens more. For more information about Safari Books Online, please visit us
online.

How to Contact Us
Please address comments and questions concerning this book to the publisher:

O’Reilly Media, Inc.

1005 Gravenstein Highway North
Sebastopol, CA 95472
800-998-9938 (in the United States or Canada)
707-829-0515 (international or local)
707-829-0104 (fax)

We have a web page for this book, where we list errata, examples, and any additional
information. You can access this page at http://www.oreilly.com/catalog/<catalog page>.
To comment or ask technical questions about this book, send email to bookques
tions@oreilly.com.
For more information about our books, courses, conferences, and news, see our website
at http://www.oreilly.com.
Find us on Facebook: http://facebook.com/oreilly
Follow us on Twitter: http://twitter.com/oreillymedia
Watch us on YouTube: http://www.youtube.com/oreillymedia

Acknowledgments
This book would not have been possible without the patience and support of my wife
and son.

Preface | vii
 CHAPTER 2
The Puppet Design Philosophy

Before we begin to explore practical best practices with Puppet, it’s valuable to under‐
stand the reasoning behind these recommendations.
Puppet can be somewhat alien to technologists who have a background in automation
scripting. Where most of our scripts scripts are procedural, Puppet is declarative. While
a declarative language has many major advantages for configuration management, it
does impose some interesting restrictions on the approaches we use to solve common
problems.
Although Puppet’s design philosophy may not be the most exciting topic to begin this
book, it drives many of the practices in the coming chapters. Understanding that phi‐
losophy will help contextualize many of the recommendations covered in this book.

Declarative Code
The Puppet Domain Specific Language (DSL) is a declarative language, as opposed to
the imperative or procedural languages that system administrators tend to be most
comfortable and familiar with.

In an imperative language, we describe how to accomplish a task. In

a declarative language, we describe what we want to accomplish. Im‐
perative languages focus on actions to reach a result, and declara‐
tive languages focus on the result we wish to achieve. We will see
examples of the difference below.

Puppet’s language is (mostly) verb-less. Understanding and internalizing this paradigm

is critical when working with Puppet; attempting to force Puppet to use a procedural
or imperative process can quickly become an exercise in frustration and will tend to
produce fragile code.

9
In theory, a declarative language ideal for configuration base-lining tasks. With the
Puppet DSL, we describe the desired state of our systems, and Puppet handles all re‐
sponsibility for making sure the system conforms to this desired state. Unfortunately,
most of us are used to a procedural approach to system administration. The vast majority
of the bad Puppet code I’ve seen has been the result of trying to write procedural code
in Puppet, rather than adapting existing procedures to Puppet’s declarative model.

Procedural Code with Puppet

In some cases writing procedural code in Puppet is unavoidable. However, such code is
rarely elegant, often creates unexpected bugs, and can be difficult to maintain. We will
see practical examples and best practices for writing procedural code when we look at
the exec resource type in Chapter 6

Of course, it’s easy to simply say “be declarative.” In the real world, we are often tasked
to deploy software that isn’t designed for a declarative installation process. A large part
of this book will attempt to address how to handle many uncommon tasks in a declar‐
ative way. As a general rule, if your infrastructure is based around packaged open source
software, writing declarative Puppet code will be relatively straight forward. Puppet’s
built in types and providers will provide a declarative way to handle most of your op‐
erational tasks. If you’re infrastructure includes Windows clients and a lot of Enterprise
software writing declarative Puppet code may be significantly more challenging.
Another major challenge system administrators face when working within the con‐
straints of a declarative model is that we tend to operate using an imperative work-flow.
How often have you manipulated files using regular expression substitution? How often
do we massage data using a series of temp files and piped commands? While Puppet
offers many ways to accomplish the same tasks, most of our procedural approaches do
not map well into Puppet’s declarative language. We will explore some examples of this
common problem, and discuss alternative approaches to solving it.

What is declarative code anyway?

As mentioned earlier, declarative code tends not to have verbs. We don’t create users
and we don’t remove them; we ensure that the users are present or absent. We don’t
install or remove software; we ensure that software is present or absent. Where create
and install are verbs, present and absent are adjectives. The difference seems trivial at
first, but proves to be very important in practice.
A real world example:
Imagine that I’m giving you directions to the Palace of Fine Arts in San Francisco.
Procedural instructions:

10 | Chapter 2: The Puppet Design Philosophy

 • Head North on 19th Avenue
• Get on US-101S
• Take Marina Blvd. to Palace Dr.
• Park at the Palace of Fine Arts Theater

These instructions make a few major assumptions:

• You aren’t already at the Palace of Fine Arts

• You are driving a car
• You are currently in San Francisco
• You are currently on 19th avenue or know how to get there.
• You are heading North on 19th avenue.
• There are no road closures or other traffic disruptions that would force you to a
different route.

Compare this to the declarative instructions:

• Be at 3301 Lyon Street, San Francisco, CA 94123 at 7:00PM

The declarative approach has a few major advantages in this case:

• It makes no assumptions about your mode of transportation. These instructions

are still valid if your plans involve public transit or parachuting into the city.
• The directions are valid even if you’re currently at the Palace of Fine Arts
• These instructions empower you to route around road closures and traffic

The declarative approach allows you to chose the best way to reach the destination based
on your current situation, and it relies on your ability to find your way to the destination
given.
Declarative languages aren’t magic. Much like an address relies on you understanding
how to read a map or use a GPS device, Puppet’s declarative model relies on it’s own
procedural code to turn your declarative request into a set of instructions that can ach‐
ieve the declared state. Puppet’s model uses a Resource type to model an object, and a
provider implementing procedural code to produce the state the model describes.
The major limitation imposed by Puppet’s declarative model might be somewhat obvi‐
ous. If a native resource type doesn’t exist for the resource you’re trying to model, you
can’t manage that resource in a declarative way. Declaring that I want a red two story
house with 4 bedrooms might empower you to build the house out of straw or wood or

Declarative Code | 11
brick, but it probably won’t actually accomplish anything if you don’t happen to be a
general contractor.
There is some good news on this front, however. Puppet already includes native types
and providers for most common objects, the Puppet community has supplied additional
native models, and if you absolutely have to accomplish something procedurally you
can almost always fall back to the exec resource type.

A practical example
Let’s examine a practical example of procedural code for user management. We will
discuss how to make the code can be made robust, it’s declarative equivalent in Puppet,
and the benefits of using Puppet rather than a shell script for this task.

Imperitive / Procedural Code

Here’s an example of an imperative process using BASH. In this case, we are going to
create a user with a home directory and an authorized SSH key on a CentOS 6 Host.
Example 2-1. Imperative user creation with BASH
groupadd examplegroup
useradd -g examplegroup alice
mkdir ~alice/.ssh/
chown alice.examplegroup ~alice/.ssh
echo "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAIEAm3TAgMF/2RY+r7KIeUoNbQb1TP6ApOtgJPNV\
0TY6teCjbxm7fjzBxDrHXBS1vr+fe6xa67G5ef4sRLl0kkTZisnIguXqXOaeQTJ4Idy4LZEVVbngkd\
2R9rA0vQ7Qx/XrZ0hgGpBA99AkxEnMSuFrD/E5TunvRHIczaI9Hy0IMXc= \
alice@localhost" > ~alice/.ssh/authorized_keys

What if we decide this user should also be a member of the wheel group?
Example 2-2. Imperative user modification with BASH
useradd -g examplegroup alice
usermod -G wheel alice

And if we want to remove that user and that user’s group?

Example 2-3. Imperative user removal with BASH
userdel alice
groupdel examplegroup

Notice a few things about this example:

• Each process is completely different

• The correct process to use depends on the current state of the user
• Each of these processes will produce errors if invoked more than one time

12 | Chapter 2: The Puppet Design Philosophy

Imagine for a second that we have several systems. On some systems, example user is
absent. On other systems, Alice is present, but not a member of the wheel group. On
some systems, Alice is present and a member of the wheel group. Imagine that we need
to write a script to ensure that Alice exists, and is a member of the wheel group on every
system, and has the correct authorized key. What would such a script look like?
Example 2-4. Robust user management with BASH
#!/bin/bash

if ! getent group examplegroup; then

groupadd examplegroup
fi

if ! getent passwd alice; then

useradd -g examplegroup -G wheel alice
fi

if ! id -nG alice | grep -q 'examplegroup wheel'; then

usermod -g examplegroup -G wheel alice
fi

if ! test -d ~alice/.ssh; then

mkdir -p ~alice/.ssh
fi

chown alice.examplegroup ~alice/.ssh

if ! grep -q alice@localhost ~alice/.ssh/authorized_keys; then

echo "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAIEAm3TAgMF/2RY+r7KIeUoNbQb1TP6ApOtg\
JPNV0TY6teCjbxm7fjzBxDrHXBS1vr+fe6xa67G5ef4sRLl0kkTZisnIguXqXOaeQTJ4Idy4LZEVVb\
ngkd2R9rA0vQ7Qx/XrZ0hgGpBA99AkxEnMSuFrD/E5TunvRHIczaI9Hy0IMXc= \
alice@localhost" >> ~alice/.ssh/authorized_keys
fi

chmod 600 ~alice/.ssh/authorized_keys

Of course, this example only covers the use case of creating and managing a few basic
properties about a user. If our policy changed, we would need to write a completely
different script to manage this user. Even fairly simple changes, such as revoking this
user’s wheel access could require somewhat significant changes to this script.
This approach has one other major disadvantage; it will only work on platforms that
implement the same commands and arguments of our reference platform. This example
will fail on FreeBSD (implements adduser, not useradd) Mac OSX, and Windows.

Declarative Code
Let’s look at our user management example using Puppet’s declarative DSL.
Creating a user and group:

Declarative Code | 13
Example 2-5. Declarative user creation with Puppet
$ssh_key = "AAAAB3NzaC1yc2EAAAABIwAAAIEAm3TAgMF/2RY+r7KIeUoNbQb1TP6ApOtgJPNV0T\
Y6teCjbxm7fjzBxDrHXBS1vr+fe6xa67G5ef4sRLl0kkTZisnIguXqXOaeQTJ4Idy4LZEVVbngkd2R\
9rA0vQ7Qx/XrZ0hgGpBA99AkxEnMSuFrD/E5TunvRHIczaI9Hy0IMXc="

group { 'examplegroup':
ensure => 'present',
}

user { 'alice':
ensure => 'present',
gid => 'examplegroup',
managehome => true,
}

ssh_authorized_key { 'alice@localhost':
ensure => 'present',
user => 'alice',
type => 'ssh-rsa',
key => $ssh_key,
}

Adding alice to the wheel group:

Example 2-6. Declarative group membership with puppet
$ssh_key = "AAAAB3NzaC1yc2EAAAABIwAAAIEAm3TAgMF/2RY+r7KIeUoNbQb1TP6ApOtgJPNV0T\
Y6teCjbxm7fjzBxDrHXBS1vr+fe6xa67G5ef4sRLl0kkTZisnIguXqXOaeQTJ4Idy4LZEVVbngkd2R\
9rA0vQ7Qx/XrZ0hgGpBA99AkxEnMSuFrD/E5TunvRHIczaI9Hy0IMXc="

group { 'examplegroup':
ensure => 'present',
}

user { 'alice':
ensure => 'present',
gid => 'examplegroup',
groups => 'wheel', #
managehome => true,
}

ssh_authorized_key { 'alice@localhost':
ensure => 'present',
user => 'alice',
type => 'ssh-rsa',
key => $ssh_key,
}
Note that the only change between this example and the previous example is the
addition of the groups parameter for the alice resource.

14 | Chapter 2: The Puppet Design Philosophy

Remove alice:
Example 2-7. Ensure that a user is absent using Puppet
$ssh_key = "AAAAB3NzaC1yc2EAAAABIwAAAIEAm3TAgMF/2RY+r7KIeUoNbQb1TP6ApOtgJPNV0T\
Y6teCjbxm7fjzBxDrHXBS1vr+fe6xa67G5ef4sRLl0kkTZisnIguXqXOaeQTJ4Idy4LZEVVbngkd2R\
9rA0vQ7Qx/XrZ0hgGpBA99AkxEnMSuFrD/E5TunvRHIczaI9Hy0IMXc="

group { 'examplegroup':
ensure => 'absent', #
}

user { 'alice':
ensure => 'absent', #
gid => 'examplegroup',
groups => 'wheel',
managehome => true,
}

ssh_authorized_key { 'alice@localhost':
ensure => 'absent', #
user => 'alice',
type => 'ssh-rsa',
key => $ssh_key,
}

Ssh_authorized_key['alice@localhost'] -> #
User['alice'] -> #
Group['examplegroup']
Ensure values are changed from Present to Absent.

Resource ordering is added to ensure groups are removed after users. Normally,
the correct order is implied due to the Autorequire feature discussed in Chapter 6

You may notice the addition of resource ordering in this example

when it wasn required in previous examples. This is a byproduct of
Puppet’s Autorequire feature. PUP-2451 explains the issue in greater
depth.

In practice, rather than managing Alice as 3 individual resources, we

would abstract this into a defined type that has its own ensure pa‐
rameter, and conditional logic to enforce the correct resource de‐
pendency ordering.

Declarative Code | 15
In this example, we are able to remove the user by changing the ensure state from present
to absent on the user’s resources. Although we could remove other parameters such as
gid, groups, and the users key, in most cases it’s better to simply leave the values in place,
just in case we ever decide to restore this user.

It’s usually best to disable accounts rather than remove them. This
helps preserve file ownership information and helps avoid UID reuse.

In our procedural examples, we saw a script that would bring several divergent systems
into conformity. For each step of that example script, we had to analyze the current state
of the system, and perform an action based on state. With a declarative model, all of
that work is abstracted away. If we wanted to have a user who was a member of 2 groups,
we would simply declare that user as such, as in Example 2-6.

Non-Declarative Code with Puppet

It is possible to write non-declarative code with Puppet. Please don’t do this:
$app_source = 'http://www.example.com/application.tar.gz'
$app_target = '/tmp/application.tar.gz'

exec { 'download application':

command => "/usr/bin/wget -q ${app_source} -O ${app_target}",
creates => '/usr/local/application/',
notify => exec['extract application'],
}

exec { 'extract application':

command => "/bin/tar -zxf ${app_target} -C /usr/local",
refreshonly => true,
creates => '/usr/local/application/',
}
This specific example has a few major problems:

1. Exec resources have a set timeout. This example may work well over a relatively fast
corporate netowkr connection, and then fail completely from a home DSL line. The
solution would be to set the timeout parameter of the exec resources to a reasonably
high value.
2. This example does not validate the checksum of the downloaded file, which could
produce some odd results upon extraction. An additional exec resource might be
used to test and correct for this case automatically.

16 | Chapter 2: The Puppet Design Philosophy

 3. In some cases, a partial or corrupted download may wedge this process. We attempt
to work around this problem by overwriting the archive each time it’s downloaded.
4. This example makes several assumptions about the contents of application.tar.gz.
If any of those assumptions are wrong, these commands will repeat every time
Puppet is invoked.
5. This example is not particularly portable, and would require a platform specific
implementation for each supported OS.
6. This example would not be particularly useful for upgrading the application.

This is a relatively clean example of non-declarative Puppet code, and tends to be seen
when working with software that is not available in a native packaging format. Had this
application been distributed as an RPM, dpkg, or MSI, we could have simply used a
package resource for improved portability, flexibility, and reporting. While this example
is not best practices, there are situations where is unavoidable, often for business or
support reasons.

This example could be made declarative using the nanliu/staging

module.

Another common pattern is the use of conditional logic and custom facts to test for the
presence of software. Please don’t do this unless it’s absolutely unavoidable:
Facter.add(:example_app_version) do
confine :kernel => 'Linux'
setcode do
Facter::Core::Execution.exec('/usr/local/app/example_app --version')
end
end
$app_source = 'http://www.example.com/app-1.2.3.tar.gz'
$app_target = '/tmp/app-1.2.3.tar.gz'

if $example_app_version != '1.2.3' {
exec { 'download application':
command => "/usr/bin/wget -q ${app_source} -O ${app_target}",
before => exec['extract application'],
}

exec { 'extract application':

command => "/bin/tar -zxf ${app_target} -C /usr/local",
}
}

Declarative Code | 17
This particular example has many of the same problems of the previous example, and
introduces one new problem: it breaks Puppet’s reporting and auditing model. The
conditional logic in this example causes the download and extraction resources not to
appear in the catalog sent to the client following initial installation. We won’t be able to
audit our run reports to see whether or not the download and extraction commands
are in a consistent state. Of course, we could check the example_application_ver
sion fact if it happens to be available, but this approach becomes increasingly useless
as more resources are embedded in conditional logic.
This approach is also sensitive to facter and pluginsync related issues, and would defi‐
nitely produce some unwanted results with cached catalogs.
Using facts to exclude parts of the catalog does have one benefit: it can be used to
obfuscate parts of the catalog so that sensitive resources do not exist in future Puppet
runs. This can be handy if, for example, your wget command embeds a pass-phrase,
and you wish to limit how often it appears in your catalogs and reports. Obviously, there
are better solutions to that particular problem, but in some cases there is also benefit to
security in depth.

Idempotency
In computer science, an idempotent function is a function that will return the same
value each time it’s called, whether it’s only called once, or called 100 times. For example:
X = 1 is an idempotent operation. X = X + 1 is a non-idempotent,recursive operation.
Puppet as a language is designed to be inherently idempotent. As a system, Puppet
designed to be used in an idempotent way. A large part of this idempotency owed to it’s
declarative resource management model, however Puppet also enforces a number of
rules on it’s variable handling, iterators, and conditional logic to maintain it’s idempo‐
tency.
Idempotence has major benefits for a configuration management language:

• The configuration is inherently self healing

• State does not need to be maintained between invocations
• Configurations can be safely re-applied

For example, if for some reason Puppet fails part way through a configuration run, re-
invoking Puppet will complete the run and repair any configurations that were left in
an inconsistent state by the previous run.

18 | Chapter 2: The Puppet Design Philosophy

 Convergence vs Idempotence
Configuration management languages are often discussed in terms of their convergence
model. Some tools are designed to be eventually convergent; others immediately con‐
vergent and/or idempotent.
With an eventually convergent system, the configuration management tool is invoked
over and over; each time the tool is invoked, the system approaches a converged state,
where all changes defined in the configuration language have been applied, and no more
changes can take place. During the process of convergence, the system is said to be in a
partially converged, or inconsistent state.
For Puppet to be idempotent, it cannot by definition also be eventually convergent. It
must reach a convergent state in a single run, and remain in the same state for any
subsequent invocations. Puppet can still be described as an immediately convergent
system, since it is designed to reach a convergent state after a single invocation.
Convergence of course also implies the existence of a diverged state. Divergence is the
act of moving the system away from the desired converged state. This typically happens
when someone attempts to manually alter a resource that is under configuration man‐
agement control.
There are many practices that can break Puppet’s idempotence model. In most cases,
breaking Puppet’s idempotence model would be considered a bug, and would be against
best practices. There are however some cases where a level of eventual convergence is
unavoidable. One such example is handling the numerous post-installation software
reboots that are common when managing Windows nodes.

Side Effects
In computer science, a side effect is a change of system or program state that is outside
the defined scope of the original operation. Declarative and idempotent languages usu‐
ally attempt to manage, reduce, and eliminate side effects. With that said, it is entirely
possible for an idempotent operation to have side effects.
Puppet attempts to limit side effects, but does not eliminate them by any means; doing
so would be nearly impossible given Puppet’s role as a system management tool.
Some side effects are designed into the system. For example, every resource will generate
a notification upon changing a resource state that may be consumed by other resources.
The notification is used to restart services in order to ensure that the running state of
the system reflects the configured state. File bucketing is another obvious intended side-
effect designed into Puppet.

Idempotency | 19
Some side effects are unavoidable. Every access to a file on disk will cause that file’s atime
to be incremented unless the entire file-system is mounted with the noatime attribute.
This is of course true whether or not Puppet is being invoked in noop mode.

Resource Level Idempotence

Many common tasks are not idempotent by nature, and will either throw an error or
produce undesirable results if invoked multiple times. For example, the following code
is not idempotent because it will set a state the first time, and throw an error each time
it’s subsequently invoked.
Example 2-8. A non-idempotent operation that will throw an error
useradd alice

The following code is not idempotent, because it will add undesirable duplicate host
entries each time it’s invoked:
Example 2-9. A non-idempotent operation that will create duplicate records
echo '127.0.0.1 example.localdomin' >> /etc/hosts

The following code is idempotent, but will probably have undesirable results:
Example 2-10. An idempotent operation that will destroy data
echo '127.0.0.1 example.localdomin' > /etc/hosts

To make our example idempotent without clobbering /etc/hosts, we can add a simple
check before modifying the file:
Example 2-11. An imperative idempotent operation
grep -q '^127.0.0.1 example.localdomin$' /etc/hosts \
|| echo '127.0.0.1 example.localdomin' >> /etc/hosts

The same example is simple to write in a declarative and idempotent way using the
native Puppet host resource type:
Example 2-12. Declarative Idempotence with Puppet
host { 'example.localdomain':
ip => '127.0.0.1',
}

Alternatively, we could implement this example using the file_line resource type from
the optional stdlib Puppet module:
Example 2-13. Idempotent host entry using the File_line resource type
file_line { 'example.localdomin host':
path => '/etc/hosts',

20 | Chapter 2: The Puppet Design Philosophy

 line => '127.0.0.1 example.localdomain',
}

In both cases, the resource is modeled in a declarative way and is idempotent by it’s very
nature. Under the hood, Puppet handles the complexity of determining whether the
line already exists, and how it should be inserted into the underlying file. Using the
native host resource type, Puppet also determines what file should be modified and
where that file is located. Example 2-12 will work on Windows.
The idempotent examples are safe to run as many times as you like. This is a huge benefit
across large environments; when trying to apply a change to thousands of hosts, it’s
relatively common for failures to occur on a small subset of the hosts being managed.
Perhaps the host is down during deployment? Perhaps you experienced some sort of
transmission loss or timeout when deploying a change? If you are using an idempotent
language or process to manage your systems, it’s possible to handle these exceptional
cases simply by performing a second configuration run against the affected hosts (or
even against the entire infrastructure.)
When working with native resource types, you typically don’t have to worry about
idempotence; most resources handle idempotence natively. A couple of notable excep‐
tions to this statement are the exec and augeas resource types. We’ll explore those in
depth in Chapter 6.
Puppet does however attempt to track whether or not a resource has changed state. This
is used as part of Puppet’s reporting mechanism and used to determine whether or not
a signal should be send to resources with a notify relationship. Because Puppet tracks
whether or not a resource has made a change, it’s entirely possible to write code that is
functionally idempotent, without meeting the criteria of idempotent from Puppet’s re‐
source model.
For example, the following code is functionally idempotent, but will report as having
changed state with every Puppet run.
Example 2-14. Puppet code that will report as non-idempotent
exec { 'grep -q /bin/bash /etc/shells || echo /bin/bash >> /etc/shells':
path => '/bin',
provider => 'shell',
}

Puppet’s idempotence model relies on a special aspect of it’s resource model. For every
resource, Puppet first determines that resource’s current state. If the current state does
not match the defined state of that resource, Puppet invokes the appropriate methods
on the resources native provider to bring the resource into conformity with the desired
state. In most cases, this is handled transparently, however there are a few exceptions
that we will discuss in their respective chapters. Understanding these cases will be critical
in order to avoid breaking Puppet’s simulation and reporting models.

Idempotency | 21
This example will report correctly:
Example 2-15. Improved code that will report as Idempotent
exec { 'echo /bin/bash >> /etc/shells':
path => '/bin',
unless => 'grep -q /bin/bash /etc/shells',
}

In this case, unless provides a condition Puppet can use to determine whether or not a
change actually needs to take place.

Using condition such as unless and onlyif properly will help pro‐
duce safe and robust exec resources. We will explore this in depth in
Chapter 6

A final surprising example is the notify resource, which is often used to produce de‐
bugging information and log entries.
Example 2-16. The Notify resource type
notify { 'example':
message => 'Danger, Will Robinson!'
}

The notify resource generates an alert every time its invoked, and will always report as
a change in system state.

Run Level Idempotence

Puppet is designed to be idempotent both at the resource level and at the run level. Much
like resource idempotence means that a resource applied twice produces the same result,
run level idempotence means that invoking Puppet multiple times on a host should be
safe, even on live production environment.

You don’t have to run Puppet in enforcing mode in production.

Run level idempotence is a place where Puppet’s idea of change is just as important as
whether or not the resources are functionally idempotent. Remember that before per‐
forming any configuration change, Puppet must first determine whether or not the
resource currently conforms to policy. Puppet will only make a change if resources are

22 | Chapter 2: The Puppet Design Philosophy

in an inconstant state. The practical implication is that if Puppet does not report having
made any changes, you can trust this is actually the case.
In practice, determining whether or not your Puppet runs are truly idempotent is fairly
simple: If Puppet reports no changes upon it’s second invocation on a fresh system, your
Puppet code base is idempotent.
Because Puppet’s resources tend to have side effects, it’s much possible (easy) to break
Puppet’s idempotence model if we don’t carefully handle resource dependencies.
Example 2-17. Ordering is critical for run-level idempotence
package { 'httpd':
ensure => 'installed',
}

file { '/etc/httpd/conf/httpd.conf':
ensure => 'file',
content => template('apache/httpd.conf.erb'),
}

Package['httpd'] ->
File['/etc/httpd/conf/httpd.conf']

The file resource will not create paths recursively. In Example 2-17, the httpd package
must be installed before the httpd.conf file resource is enforced; and it depends on the
existence of /etc/httpd/conf/httpd.conf, which is only present after the httpd pack‐
age has been installed. If this dependency is not managed, the file resource becomes
non-idempotent; upon first invocation of Puppet it may throw an error, and only enforce
the state of httpd.conf upon subsequent invocations of Puppet.
Such issues will render Puppet convergent. Because Puppet typically runs on a 30 minute
interval, convergent infrastructures can take a very long time to reach a converged state.
There are a few other issues that can render Puppet non-idempotent

Non-deterministic code
As a general rule, the Puppet DSL is deterministic, meaning that a given set of inputs
(manifests, facts, exported resources, etc) will always produce the same output with no
variance.
For example, the language does not implement a random() function; instead a
fqdn_rand() function is provided that returns random values based on a static seed
(the host’s fully qualified domain name.) This function is by it’s very nature not cryp‐
tographically secure, and not actually random at all. It is however useful for in cases
where true randomness is not needed, such as distributing the start times of load in‐
tensive tasks across the infrastructure.

Idempotency | 23
Non-deterministic code can pop up in strange places with Puppet. A notorious example
is Ruby 1.8.7’s handling of hash iteration. The following code is non-deterministic with
Ruby 1.8.7; the output will not preserve the original order and will change between runs:
Example 2-18. Non-deterministic hash ordering with Ruby 1.8.x
$example = {
'a' => '1',
'b' => '2',
'c' => '3',
}

alert(inline_template("<%= @example.to_a.join(' ') %>\n"))

Another common cause of non-deterministic code pops up when our code is dependent
on a transient state.
Environment induced non-determinism.
file { '/tmp/example.txt':
ensure => 'file',
content => "${::servername}\n",
}
Environment induced non-determinism will not be idempotent if you have a load bal‐
anced cluster of Puppet Masters. The value of $::servername changes depending on
which master compiles the catalog for a particular run.
With non-deterministic code, Puppet loses run level idempotence. For each invocation
of Puppet, some resources will change shape. Puppet will converge, but it will always
report your systems as having been brought into conformity with it’s policy, rather than
being conformant. As a result, it’s virtually impossible to determine whether or not
changes are actually pending for a host. It’s also more difficult to track what changes
were made to the configuration, and when they were made.
Non deterministic code also has the side effect that it can cause services to restart due
to Puppet’s notify behavior. This can cause unintended service disruption.

Stateless
Puppet’s client / server API is stateless, and with a few major (but optional) exceptions,
catalog compilation is a completely stateless process.
A stateless system is a system that does not preserve state between requests; each request
is completely independent from previous request, and the compiler does not need to
consult data from previous request in order to produce a new catalog for a node.
Puppet uses a RESTful API over HTTPS for client server communications.

24 | Chapter 2: The Puppet Design Philosophy

With master/agent Puppet, the Puppetmaster need only have a copy of the facts supplied
by the agent in order to compile a catalog. Natively, Puppet doesn’t care whether or not
this is the first time it’s generated a catalog for this particular node, nor whether or not
the last run was successful, or if any change occurred on the client node during the last
run. The nodes catalog is compiled in it’s entirety every time the node issues a catalog
request. The responsibility for modeling the current state of the system then rests en‐
tirely on the client, as implemented by the native resource providers.
IF you don’t use a puppetmaster or have a small site with a single master, statelessness
may not be a huge benefit to you. For medium to large sites however, keeping Puppet
stateless is tremendously useful. In a stateless system, all Puppetmasters are equal. There
is no need to synchronize data or resolve conflicts between masters. There is no locking
to worry about. There is no need to design a partition tolerant system in case you lose
a data-center or data-link, and no need to worry about clustering strategies. Load can
easily be distributed across a pool of masters using a load balancer or DNS SRV record,
and fault tolerance is as simple as ensuring nodes avoid failed masters.
It is entirely possible to submit state to the master using custom facts or other other
techniques. It’s also entirely possible to compile a catalog conditionally based on that
state. There are cases where security requirements or particularly idiosyncratic software
will necessitate such an approach. Of course, this approach is most often used when
attempting to write non-declarative code in Puppet’s DSL. Fortunately, even in these
situations, the Server doesn’t have to actually store the node’s state between runs; the
client simply re-submits it’s state as part of it’s catalog request.
If you keep your code declarative, it’s very easy to work with Puppet’s stateless client/
server configuration model. IF a manifest declares that a resource such as a user should
exist, the compiler doesn’t have to be concerned with the current state of that resource
when compiling a catalog. The catalog simply has to declare a desired state, and the
Puppet agent simply has to enforce that state.
Puppet’s stateless model has several major advantages over a stateful model:

• Puppet scales horizontally

• Catalogs can be compared
• Catalogs can be cached locally to reduce server load

It is worth noting that there are a few stateful features of Puppet. It’s important to weigh
the value of these features against the cost of making your Puppet infrastructure stateful,
and to design your infrastructure to provide an acceptable level of availability and fault
tolerance. We will discuss how to approach each of these technologies in upcoming
chapters, but a quick overview is provided here.

Stateless | 25
Sources of State
In the beginning of this section, I mentioned that there are a few features and design
patterns that can impose state on Puppet catalog compilation. Let’s look at some of these
features in a bit more depth.

Filebucketing
File-bucketing is an interesting and perhaps under-appreciated feature of the File re‐
source type. If a file-bucket is configured, the file provider will create a backup copy of
any file before overwriting the original file on disk. The backup may be bucketed locally,
or it can be submitted to the Puppetmaster.
Bucketing your files is useful for keeping backups, auditing, reporting, and disaster
recovery. It’s immensely useful if you happen to blast away a configuration you needed
to keep, or if you discover a bug and would like to see how the file is changed. The
Puppet enterprise console can use file-bucketing to display the contents of managed
files.
File-buckets can also be used for content distribution, however using a file-bucket this
way creates state. Files are only present in a bucket when placed there; either as a backup
from a previous run, or by the static_compiler terminus. Placing a file in the bucket only
happens during a Puppet run, and Puppet has no internal facility to synchronize buckets
between masters. Reliance upon file buckets for content distribution can create prob‐
lems if not applied cautiously. It can create problems when migrating hosts between
data-centers, when rebuilding masters. Use of file bucketing in your modules can also
create problems during local testing with puppet apply.

Exported Resources
Exported resources provide a simple service discovery mechanism for Puppet. When a
puppetmaster or agent compiles a catalog, resources can be marked as exported by the
compiler. Once the resources are marked as exported, they are recorded in a SQL da‐
tabase. Other nodes may then collect the exported resources, and apply those resources
locally. Exported resources persist until they are overwritten or purged.
As you might imagine, exported resources are, by definition stateful and will affect your
catalog if used.
We will take an in depth look at PuppetDB and exported resources in (to come). For
the time being, just be aware that exported resources introduce a source of state into
your infrastructure.
In this example, a pool of web-servers export their pool membership information to a
haproxy load balancer, using the puppetlabs/haproxy module and exported resources.

26 | Chapter 2: The Puppet Design Philosophy

Example 2-19. Declaring state with an exported resource
include haproxy

haproxy::listen { 'web':
ipaddress => $::ipaddress,
ports => '80',
}

Haproxy::Balancermember <<| listening_service == 'web' |>>

Example 2-20. Applying state with an exported resource

@@haproxy::balancermember { $::fqdn:
listening_service => 'web',
server_names => $::hostname,
ipaddresses => $::ipaddress,
ports => '80',
options => 'check',
}

This particular example is a relatively safe use of exported resour‐

ces; if PuppetDB for some reason became unavailable the pool would
continue to work; new nodes would not be added to the pool until
PuppetDB was restored. TODO: Validate what I just said is true giv‐
en the internal use of concat on this module…

Exported resources rely on PuppetDB, and are typically stored in a PostgreSQL database.
While the PuppetDB service is fault tolerant and can scale horizontally, the PostgreSQL
itself scales Vertically and introduces a potential single point of failure into the infra‐
structure. We will discuss approaches to scale and mitigate risk in (to come)

Hiera
Hiera is by design a pluggable system. By default is provides JSON and YAML back-
ends, both of which are completely stateless. However, it is possible to attach Hiera to
a database or inventory service, including PuppetDB. If you use this approach, it can
introduce a source of state into your Puppet Infrastructure. We will explore Hiera in
depth in [Link to Come].

Inventory and Reporting

The Puppet Infrastructure maintains a considerable amount of reporting information
pertaining to the state of each node. This information includes facts about each node,
detailed information about the catalogs sent to the node, and the reports produced at
the end of each Puppet run. While this information is stateful, this information is not
typically consumed when compiling catalogs. We’ll be taking a close look at inventory
and reporting services in [Link to Come]

Stateless | 27
There are plug-ins to Puppet that allow inventory information to be used during catalog
compilation, however these are not core to Puppet.

Custom Facts
Facts themselves do not inherently add state to your Puppet manifests, however they
can be used to communicate state to the Puppetmaster, which can then be used to
compile conditional catalogs. We saw an example of this in [Link to Come] when dis‐
cussing non-declarative code. Using facts in this way does not create the scaling and
availability problems inherent in server site state, but it does create problems if you
intend to use cached catalogs, and it does reduce the effectiveness of your reporting
infrastructure.

Summary
In this chapter, we reviewed the major design features of Puppet’s language, both in
terms of the benefits provided by Puppet’s language, and the restrictions it’s design places
on us. Future chapters will provide more concrete recommendations for the usage of
Puppet’s language, overall architecture of Puppet, and usage of Puppet’s native types
and providers. Building code that leverages Puppet’s design will be a major driving force
behind may of the considerations in future chapters.
Takeaways from this chapter:

• Puppet is declarative, idempotent, and stateless

• In some cases violation of these design ideals is unavoidable
• Write declarative, idempotent, and stateless code whenever possible

28 | Chapter 2: The Puppet Design Philosophy

 CHAPTER 3
Code and Data; High Level Design

In this chapter, we will look at the primary structures of a Puppet code-base, and discuss
how the design of your site will impact module maintenance, code reuse, debugging,
and scaling.
Here, we introduce the major systems of Puppet, including Hiera and Puppet Modules
as well as several common design patterns. We also attempt to broadly classify the com‐
mon kinds of code and data-sources seen with puppet into categories, so that we can
discuss the most appropriate way of handling each concern.
Organization of your Puppet infrastructure becomes critical as your environment
grows; correct placement of your data will improve the flexibility of your code-base and
control the disruption caused by environmental changes. Correct separation of your
code will help to create small problem domains that are much easier to debug and
improve. Correct organization and design will help promote code re-use as you bring
up new applications, and will reduce the impact of changes to business logic and ap‐
plication stack design driven by ever changing business requirements.
As with Chapter 2 this document focuses primarily on “why” rather than “how,” and
will help lay a foundation for many of the recommendations in coming chapters.

Code and Data

When I discuss Puppet architecture with new clients, I generally define 5 major cate‐
gories of code and data typically seen in the infrastructure.

1. Application logic
2. Business logic (site specific logic)
3. Site specific data
4. Node specific data (node classification)

29
 5. Service discovery data (optional)

These terms are not universally accepted, but are useful for discussing the organization
of your code, and how it maps to the various features and design patterns of Puppet.
These concepts loosely map to a number of Puppet features and design patterns:

• The logic to manage an application, service, or subsystem can usually be contained

in a single Puppet module.
• Business logic is usually contained within roles and profiles
• Site specific data is usually stored in Hiera
• Node specific data is usually managed using some sort of console and/or stored in
a database
• Service data is usually managed with Hiera, PuppetDB, or a service discovery sys‐
tem

Application Logic
Application logic is the logic to manage a single service, application, or subsystem. As
a general rule, application logic will manage a single named product and some of it’s
dependencies. E.g. Apache, Nginx, and IIS are applications. A web-server is a higher
level abstraction that will probably contain multiple applications, services, and their
dependencies.
The application logic for Apache will contain resources to deploy Apache’s packages,
manage Apache’s configuration files, and ensure that the Apache service is running. The
application logic may also contain code to setup virtual hosts that can be declared by
other modules or as part of an application stack.
Application logic may also contain conditional logic and data to handle platform specific
implementation details. For example, an Apache module should contain platform spe‐
cific logic to ensure that the correct package is installed (E.g. httpd on RedHat and
apache2 on Debian platforms) that the configuration file is installed to the correct lo‐
cation and contains the correct platform specific configurations, and that the correct
service name is used when managing the apache service state.
The Apache module contains a lot of data about Apache; package names, configuration
details, file locations, default port numbers, docroot, and file ownership information. It
may even include generic usernames, passwords, hostnames, and contact information
for the cases where the user simply wants to test the module or isn’t concerned about
such details.

30 | Chapter 3: Code and Data; High Level Design

In some cases, the application may include some data that tends to be universal but may
be overridden in a site specific way. For example, the NTP module includes a set of
publicly available NTP servers that will be used if site specific servers are not configured.

Business Logic
Business logic also manages applications, but typically at a higher level. At this level, we
aren’t managing Apache, we are managing a web-server. At this level, we aren’t con‐
cerned with package names or the platform specific implementation details of our serv‐
ices; instead we’re concerned with what components are deployed to our web-server.
The implementation details of each components are abstracted away as application code.
Business logic contains code that ensures our web-servers have the appropriate HTTP
server installed with the necessary extensions, and that the module deploying our web‐
site is applied. Business logic may still contain conditional code; e.g. we might declare
that Windows hosts get the IIS and Linux hosts get the Apache Module. The database
names used by your applications are site specific data.
Business logic may pass higher level data on to your applications. For example, your
application may require that web-server listen on a nonstandard port.

Site Specific Data

Site specific data is the data that is proprietary to your site, and isn’t fundamental to any
of your applications. The usernames and passwords your applications use to authenti‐
cate to their respective database servers are site specific data. Your internal YUM repo‐
sitories and NTP servers are site specific data.
In some cases, Puppet resources may also be handled as site specific data. It’s very easy
to store users, packages, or yum repositories as a data structure and convert that data
into resources inside a profile. The key here is that these resources need to be site specific;
the fact that mlocate and tree should be installed on all your hosts is site specific. The
dependencies for your Apache module are business logic, and the name of the package
that contains the Apache2 daemon is application logic.

Node Specific Data

Node specific data is a special case of site specific data. Node specific data is the list of
hosts managed by Puppet, basic properties about those systems such as their tier, owner,
or location, and a list of what Puppet should apply to that host.
While this list can be maintained with the same system that manages your site specific
data (see Hiera below) it’s usually separated. Node classification is incredibly transient;
nodes are constantly brought up, shut down, modified, and re-assigned.

Code and Data | 31

In the olden days, Nodes were typically managed using node statements within the site
wide Puppet manifest. Nodes classification could be a mix of code and data with In‐
heritance thrown on top. In modern sites, nodes definition is handled entirely as data,
and any logic that needs to be applied on top of the node classification is handled using
a hierarchical data lookup if site specific data, or at a higher level as business or appli‐
cation specific logic.

Service Data
Service data is another subset of site specific data, and is related to node classification.
Service data is the record of what hosts provide a specific service as part of a specific
application stack. This kind of data is prevalent with horizontally scaled and multi-tier
applications.
For example, the classic 3-tier application stack consisting of Apache, Tomcat, and
MySQL would typically have 2 pools of service data that would need to be maintained
to configure every host in the stack.
One pool of application data would consist of service data for the Tomcat servers. This
service data would be consumed by the Apache servers in order to configure reverse
proxies. The Apache servers could use this data to load balance requests to the appli‐
cation servers. Service data about the Tomcat servers could also be consumed by the
MySQL servers in order to define database grants. Doing so would provide much more
granular control than authorizing access by user-name and password alone or by au‐
thorizing an entire subnet of hosts.
The other pool of application data would contain information about the MySQL servers.
This could allow the Tomcat servers to load balance read requests across a set of read-
only slaves. It might also identify the write master in a pool of database servers.
Service discovery is often handled as site-specific data and updated manually, however
this approach does not facilitate auto-scaling, or automatic fail-over, and adds extra
manual steps to the process of bringing up new nodes in a pool of hosts. For these and
other reasons, it’s typical to handle service data as a special class of data, managed using
a special set of tools.
More on service discovery tools below, and in (to come)

Breaking it down
Let’s take a look at how these concerns map to the features and common design patterns
of Puppet.

32 | Chapter 3: Code and Data; High Level Design

Application Logic & Puppet Modules
The Puppet Labs documentation describes modules as “self-contained bundles of code
and data.” Modules serve many purposes in Puppet, and it’s difficult to provide a more
specific description than that while remaining concise and accurate.
For this chapter, when we discuss Modules and module design we are almost always
looking at modules from the perspective of service or application management using
the Puppet DSL.
With this scope in mind, Puppet modules map fairly well to the concept of application
logic. A single module can easily manage a single application, service, or component of
a larger system. The prerequisites, and dependencies of the service can be managed by
other modules, and multiple modules can be combined into application stacks at a
higher level of abstraction.
Modules are most effective when the serve a single purpose, limit dependencies, and
concern themselves only with managing system state relating to their named purpose.
Puppet modules are intended to be portable and reusable. A good module will usually
manage a single service, and accept just enough input to be re-used with multiple ap‐
plication stacks. In an ideal world, our modules are entirely self contained. They provide
complete functionality without creating dependencies on any other modules, and can
be combined as needed to build different application stacks.
Modules need not contain or even declare all of their own dependencies; doing so almost
invariably begins to embed business logic into our modules. Let’s consider the example
of a Java application that depends on the availability of a Java Runtime Environment
(JRE.) On a modern Linux distribution, installing the default JRE is usually just a simple
package resource declaration, and at first it makes sense to install a JRE as part of our
application. However, there are several different JRE offerings available, and usually
different releases and feature sets for each offering. If we bundle OpenJDK into our
module, we would prevent our users installing Oracle JRE instead. We would force our
users to use our preferred JRE and our own installation parameters, but that could create
a conflict with any other module that attempts to install it’s own JRE. In this case, the
cleanest approach would be to document our module’s Java dependency, and provide a
test manifest demonstrating how to install a JRE and other dependencies along with
our module. This approach allows our users to handle the JRE dependency themselves
with their own business logic. It also allows our users to manage conflicts and interde‐
pendencies using their own site-specific logic.
As mentioned earlier, a good module has no interdependencies. This should not be
taken as advise to eliminate module interdependencies altogether; instead this is a rec‐
ommendation to create interdependencies with care. There are many cases where mod‐
ules extended the features to Puppet and provide useful functionality to be used in your
own code. For example, it’s typical for a module to depend on puppetlabs/stdlib due to

Breaking it down | 33
the number of useful functions it offers. As another example, it’s usually better to create
a dependency on nanliu/staging than to re-implement your own archive retrieval and
extraction code on a per-module basis.
Tightly scoped modules provide a lot of flexibility, but their greatest benefit is their ease
of maintenance. Writing test cases for small simple modules is much easier than writing
test cases for big complex modules. Debugging a problem is much simpler when you
can test each module in isolation and confirm that each module’s behavior is reasonable
and its test cases are correct.
Modules may contain Puppet resources, Puppet code, and data relating to the module.
Let’s take a look at the Puppetlabs/ntp module as an example.

• Package, File, and Service resources manage the basic components of the NTP ser‐
vice.
• Conditional logic provides platform specific defaults for NTP servers, service
names, and file locations
• A list of default, platform specific public NTP servers
• Parameters to change the behavior of the module or set your own NTP servers
• Input validation to ensure that data supplied to the module is sane (if not correct)
• Documentation describing the features use of the module
• Test cases for ensuring correctness and stability of the module
• Meta-data that can be consumed by the Puppet Forge and Puppet Module utility

The NTP module is fairly simple as far as Puppet modules are concerned. It contains
few resources, and a lot of data. However, all of this data is specific to the NTP module.
You can easily use this module as is in your site either by applying it with default pa‐
rameters, or you could overriding the the default parameters with your own site specific
data using site-specific logic or automatic parameter lookups.
From a business logic perspective, this module would probably be part of your baseline
system configuration, and would most likely be applied to every node in your infra‐
structure.
Although this module is concerned with service data, the list of NTP servers in your
site will tend to remain fairly static. In this case, service discovery can be handled as
generic site-specific data provided by Hiera.

Identifying business logic in Puppet modules

As a general rule, you are writing business logic rather than application logic when:

34 | Chapter 3: Code and Data; High Level Design

 • Your code could create a conflict with another module that otherwise does not
overlap with your module
• The thing you’re including is outside the explicit scope/concern of your module
• The thing you’re including could be a standalone application or service
• The thing you’re including has usefulness outside your module
• The dependency you’re resolving could be handled using other solutions

When you start to notice that your code meets one or more of the above criteria, consider
how your module might be split into multiple modules and managed using the Roles
and Profiles design pattern.

Site specific data vs application data in Puppet Modules

Modules will tend to contain a lot of data. This data takes the form of variable defaults,
package and file names, hostnames, ports, files, and templates. As a guideline, the ideal
module contains no data specific to your site, but does contain the necessary data to
bring this application or service up in a generic way with appropriate values for sup‐
ported platforms.
An example of site-specific data is a URL to download a file from an internal server. An
example of application specific data is the name of a package resource that should be
deployed, and the general layout of its configuration file.
There are several important reasons to keep site specific data out of your module, even
if your module is completely proprietary and would never be released to the public.
One issue is that embedding data in a module module tends to create module interde‐
pendencies. When a module contains data about your site, it’s tempting to de-duplicate
the data by referencing data stored in other modules rather than redefining that data in
each module. Doing so creates explicit interdependencies. This pattern also tends to
violate the principles of interface driven design; Puppet’s language doesn’t offer getter
methods, or the concept of private vs public variables.
Data changes. Constantly. This is by far the greatest issue created by embedding site-
data in modules. If your site specific data is spread across multiple modules, simple
changes to your site’s configuration can suddenly require a massive effort to hunt down
and update each module that might be affected by the change.
Data stored inside your modules is rarely going to be formed in a consistent way, and
it will tend to be spread across a large set of files. If you are using R10k and the module-
per-repo pattern, all this data may not even be stored in the same RCS repository, and
you may even have to be concerned about situations where the data has to be updated
in more than one revision of the module, and the module revisions need to be updated
in multiple control repositories.

Breaking it down | 35
By designing your module to be portable, you force your site specific data to be cen‐
tralized and stored outside of the module. By adhering to this simple pattern, you en‐
courage use of node classification and Hiera.
Conversely, it is a good idea to contain application specific data entirely inside your
module. If application data, such as the name of the packages your module installs or
the template used to configure a service are externalized, the module becomes unusable
in isolation; it can only be tested in the greater context of your infrastructure. It can’t be
reused without also supplying the data that lives outside the module, and it isn’t going
to be easily portable since that data is often stored or formulated in such a way to become
non portable. Storing application data outside the module also tends to increase de‐
pendencies; if your module depends on data stored in Hiera in order to perform a basic
smoke test, you must have hiera configured and your data deployed in order to simply
test the module.
One caution about this recommendation: Always try to provide sane defaults for site
specific data, even if those defaults aren’t terribly useful in the real world. This provides
significant benefits for anyone attempting to debug, test, or simply play with your mod‐
ule.

Business logic with Roles and Profiles

Roles and Profiles are not a design feature of Puppet, instead they are a desgin patern
originally described in Craig Dunn’s blog post Designing Puppet - Roles and Profiles.
We will take a closer look at the Roles and Profiles design pattern in [Link to Come].
Roles and profiles have a few important design properties:

1. They describe application stacks and business logic

2. They are typically site specific

A major feature of Puppet is code reuse. Modules provide a way to model applications
and services in a portable and reusable way. Application stacks can be created by com‐
bining multiple modules together in interesting ways using the roles and profiles pat‐
tern.
For example, if you wanted to setup an instance of WordPress, we would need an ap‐
plication stack containing the following components:

• A database server (MySQL)

• A web-server (such as Apache)
• PHP
• WordPress

36 | Chapter 3: Code and Data; High Level Design

While you could create a monolithic Puppet Module to configure Wordpress and all of
its dependencies, such a module would be quite complex and inflexible. We could build
a module that concerns itself only with the deployment of Wordpress, and relies on the
official puppetlabs/apache, puppetlabs/mysql Puppet modules. Using this approach al‐
lows us to rely on the development efforts and support provided by the authors of the
Apache and PHP modules (in this case, Puppet Labs) and provides the benefits of com‐
munity experience and bugfixes.
By writing a module that only concerned it’s self with WordPress, the consumer of your
module could easily swap in arbitrary web-servers and database servers. Each module
could be self contained, which would make it simpler to test each system in isolation,
and help reduce the amount of code that would need to be reviewed to identify and
isolate bugs.
The modular approach also facilitates code re-use. Other users will consume public
Apache and MySQL modules even if they aren’t interested in WordPress. By using those
modules, you can leverage their shared experience, improvements, and bug fixes.
Leveraging public modules creates a new design problem: if we have two modules as‐
signed to a host that both depend on Apache, which of those modules should be the one
to own and declare it?
The answer is neither. Instead, we abstract our application stacks into a profile. The
profile simply defines the modules needed to build our application stack, their ordering
dependencies, and if absolutely necessary any parameters that should be passed to the
modules. Because a profile is site specific, we can design our profiles to avoid conflicts.
For example, we could have the profiles both include apache, and only use resource style
declaration for our application vhost, via defined types. With this approach, the one
node could potentially apply multiple profiles that declare Apache without issue.

Roles and Profiles vs. Node Classifiers

Roles and profiles are a design pattern. Strictly speaking, both roles and profiles are
simply modules containing Puppet Manifests. What makes roles and profiles special
are their purpose as an abstraction layer between your modules and your node classifier.
In the old days, this sort of thing was common:
Example 3-1. Node classification with Inheritance
node base {
include ::accounts
include ::security
include ::repos
class { '::ntp':
servers => [ 'ntp1.corp.com', 'ntp2.corp.com' ],
}
}

Breaking it down | 37
node web inherits base {
include ::apache
include ::php
}

node db inherits base {

include ::mysql
}

node www1.example.com www2.example.com inherits web { }

node www-dev1.example.com inherits web {

include ::mysql
}

# etc.

This approach tended to be difficult to maintain, and had a few major limitations; node
definitions do not support multiple inheritance, so it’s impossible to build a development
box that contains both the db and web node classifications. Node inheritance also has
some very surprising behavior relating to variable inheritance and scope.
Trying to migrate this approach to an external node classifier creates some issues as well.
Many web console ENCs support the concept of groups and group inheritance, however
they often don’t support parameterized classes, and when they do they may only support
strings as a data-type.
Of course, conditional logic isn’t really an option with an off-the-shelf console; if you
want to install IIS on your Windows web nodes and Apache on your RHEL nodes, you
need to create two groups and assign them manually.
With roles and profiles, you move node classification logic to a set of conventional
puppet manifests/classes.
Example 3-2. Node Classification with Roles & Profiles
class profiles::base {
include ::accounts
include ::security
include ::repos
class { '::ntp':
servers => [ 'ntp1.corp.com', 'ntp2.corp.com' ],
}
}

class profiles:web {
include ::apache
include ::php
}

38 | Chapter 3: Code and Data; High Level Design

Exploring the Variety of Random
Documents with Different Content
The Project Gutenberg eBook of Quaker
Strongholds
This ebook is for the use of anyone anywhere in the United States
and most other parts of the world at no cost and with almost no
restrictions whatsoever. You may copy it, give it away or re-use it
under the terms of the Project Gutenberg License included with this
ebook or online at www.gutenberg.org. If you are not located in the
United States, you will have to check the laws of the country where
you are located before using this eBook.

Title: Quaker Strongholds

Author: Caroline Emelia Stephen

Release date: April 10, 2020 [eBook #61802]

Language: English

Credits: Produced by WebRover, QuakerHeron, Stephen Hutcheson,

and
the Online Distributed Proofreading Team at
https://www.pgdp.net (This file was produced from images
generously made available by The Internet Archive)

*** START OF THE PROJECT GUTENBERG EBOOK QUAKER

STRONGHOLDS ***
QUAKER STRONGHOLDS
BY
CAROLINE EMELIA STEPHEN
AUTHOR OF “THE SERVICE OF THE POOR”

PHILADELPHIA
HENRY LONGSTRETH, 740 SANSOM STREET
1891
 CONTENTS.
CHAPTER PAGE
Introduction 1
I. Organization 6
II. The Inner Light 20
III. Worship 51
IV. Free Ministry 91
V. Special Testimonies 118
VI. Our Calling 157
Appendix 199

QUAKER STRONGHOLDS.
 INTRODUCTION.
Whether Quakerism be, as some Friends believe, destined to any
considerable revival or not, it seems at least certain that any
important revival of religion must be the result of a fresh recognition
and acceptance of the very principles upon which the Society of
Friends is built. What these principles and the practices resulting
from them really are, is a subject on which there is a surprising
amount of ignorance amongst us, considering how widely spread is
the connection with and interest about Friends amongst the
members of other persuasions. One seldom meets any one who has
not some link with the Society, and yet it is rare to find any one not
belonging to it at all accurately informed as to its point of view or its
organization. The notorious disinclination of Friends to any attempts
at proselytizing, and perhaps some lingering effects of persecution,
probably account for the very common impression that Friends’ 2
meetings are essentially private—mysterious gatherings into
which it would be intrusive to seek admission. Many people, indeed,
probably suppose (if they think about it at all) that such meetings
are no longer held; that the Society is fast dying out, and the “silent
worship” of tradition is a thing of the past—impracticable, and hardly
to be seriously mentioned in these days of talk and of breathless
activity.

Some such vague impression floated, I believe, over my own mind,

when, some seventeen years ago, I first found myself within reach
of a Friends’ meeting, and, somewhat to my surprise, cordially made
welcome to attend it. The invitation came at a moment of need, for I
was beginning to feel with dismay that I might not much longer be
able conscientiously to continue to join in the Church of England
service; not for want of appreciation of its unrivalled richness and
beauty, but from doubts of the truth of its doctrines, combined with
a growing recognition that to me it was as the armour of Saul in its
elaboration and in the sustained pitch of religious fervour for which it
was meant to provide an utterance. Whether true or not in its
speculative and theoretical assumptions, it was clear to me that it
was far from true as a periodical expression of my own 3
experience, belief, or aspiration. The more vividly one feels the
force of its eloquence, the more, it seems to me, one must hesitate
to adopt it as the language of one’s own soul, and the more unlikely
it is that such heights and depths of feeling as it demands should be
ready to fill its magnificent channels every Sunday morning at a
given hour. The questionings with which at that period I was
painfully struggling were stirred into redoubled activity by the
dogmatic statements and assumptions with which the Liturgy
abounds, and its unbroken flow left no loophole for the utterance of
my own less disciplined, but to myself far more urgent, cries for
help. Thus the hour of public worship, which should have been a
time of spiritual strengthening and calming, became to me a time of
renewed conflict, and of occasional exaltation and excitement of
emotion, leading but too surely to reaction and apathy.

I do not attempt to pass any judgment on this mental condition. I

have described it at some length because I cannot believe it to be
altogether exceptional, or without significance. At any rate, it was
fast leading me to dread the moment when I should be unable
either to find the help I needed, or to offer my tribute of devotion in
any place of worship amongst my fellow-Christians. When lo, on 4
one never-to-be-forgotten Sunday morning, I found myself one
of a small company of silent worshippers, who were content to sit
down together without words, that each one might feel after and
draw near to the Divine Presence, unhindered at least, if not helped,
by any human utterance. Utterance I knew was free, should the
words be given; and before the meeting was over, a sentence or two
were uttered in great simplicity by an old and apparently untaught
man, rising in his place amongst the rest of us. I did not pay much
attention to the words he spoke, and I have no recollection of their
purport. My whole soul was filled with the unutterable peace of the
undisturbed opportunity for communion with God, with the sense
that at last I had found a place where I might, without the faintest
suspicion of insincerity, join with others in simply seeking His
presence. To sit down in silence could at the least pledge me to
nothing; it might open to me (as it did that morning) the very gate
of heaven. And since that day, now more than seventeen years ago,
Friends’ meetings have indeed been to me the greatest of outward
helps to a fuller and fuller entrance into the spirit from which they
have sprung; the place of the most soul-subduing, faith-restoring,
strengthening, and peaceful communion, in feeding upon the 5
bread of life, that I have ever known. I cannot but believe that
what has helped me so unspeakably might be helpful to multitudes
in this day of shaking of all that can be shaken, and of restless
inquiry after spiritual good. It is in the hope of making more widely
known the true source and nature of such spiritual help that I am
about to attempt to describe what I have called our strongholds—
those principles which cannot fail, whatever may be the future of the
Society which for more than two hundred years has taken its stand
upon them. I wish to trace, as far as my experience as a “convinced
Friend” enables me to do so, what is the true life and strength of our
Society; and the manner in which its principles, as actually embodied
in its practice, its organization, and, above all, its manner of worship,
are fitted to meet the special needs of an important class in our own
day.

Mount Pleasant,
West Malvern, 1890.

6
 CHAPTER I.
ORGANIZATION.

The actual organization of the Society of Friends is, I believe, by no

means familiarly known outside its own borders, and a slight sketch
of it may be neither uninteresting in itself, nor out of place as a
preliminary to the endeavour to explain our general position. I
propose, therefore, to give such an outline of our constitution as a
Society, so far as I have become acquainted with it. The fullest
details respecting it are to be found in the “Book of Discipline,” which
is the authorized exponent of all such matters.
[1]
This book has been recently revised, and the edition of 1883 (a
large octavo volume) contains the latest regulations on all points of
internal government. The Yearly Meeting also publishes 7
annually a volume of Extracts from its proceedings, a full
statement of accounts and statistics, and a summary of the reports
received from the subordinate meetings all over the country.

Every “particular meeting,” that is, every congregation meeting

habitually for worship on the first (and generally also on one other)
day of the week, is one of a group of meetings for worship (usually
about five or six), which meet together once a month, for the
transaction of business and of discipline, and which together form
what is therefore called a Monthly Meeting. Each Monthly Meeting,
again, is one of a group of probably four or five Monthly Meetings,
which in like manner unite to form a Quarterly Meeting, at whose
quarterly sittings matters of larger importance are considered, and
the eighteen Quarterly Meetings of Great Britain form in their turn
the London Yearly Meeting, which is the supreme authority in the
Society. It may in a certain sense be said, indeed, that it is the
Society of Friends of Great Britain, for every Friend is a member of
the Monthly, Quarterly, and Yearly Meetings to which he or she
belongs, and is entitled to a voice in all their deliberations. The
Yearly Meeting assembles in May, and its sittings, which are held, as
they have been from the first, in Devonshire House, 8
Bishopsgate Street, last generally about a fortnight. The actual
attendance is, of course, small in comparison with the number of
members. At the present time the Society in Great Britain consists of
about fifteen thousand members, and the annual gatherings in
Bishopsgate Street number perhaps from twelve to fifteen hundred.

The men and women sit separately, or it would perhaps be more

correct to say that the men and the women Friends have each a
separate Yearly Meeting; the women’s Yearly Meeting being of
considerably later date than the men’s. It was established in 1790,
and it deals in general with matters of less importance, or at any
rate of more restricted scope, than the men’s meeting. It is,
however, not unusual for men Friends, “under religious concern,” to
visit the women’s meeting, nor for women Friends on a similar
ground to visit that of the men.

“Joint sittings”—meetings, that is, of men and women Friends in one

body—are also held occasionally, when any question of special
interest to all the members is to be considered, and on these
occasions the women are free to take their full share in the
discussions. These occasional combinations are the more easily
practicable, because, strange as it may seem to most people, 9
no question is ever put to the vote. From the earliest times, all
decisions have been arrived at by what may be called a practical
unanimity. The Yearly Meeting, like every other meeting for
“business” or “discipline,” has its clerk, who, with one or more
assistants, performs the combined functions of chairman and
secretary. When any question has been fully considered, it is the
duty of the clerk to interpret the sense of the meeting, and to
prepare a minute accordingly; which minute, being read to the
meeting, often receives a certain amount of verbal, or even of
substantial modification, in accordance with the suggestions of
individual Friends; but, when entered upon the books, is accepted as
embodying the decision of the meeting. Should there be any
considerable division of judgment upon any important question, it is
usually, if possible, adjourned till the next Yearly Meeting; and this
plan has, I believe, been almost invariably found sufficient to bring
about the practical unanimity required for a final settlement of the
question. It is certainly a very remarkable fact that so large a body
should transact all its affairs without ever voting, to the full
satisfaction of the great majority of those concerned.

The Quarterly and Monthly Meetings are, in most respects, 10

repetitions on a smaller scale of the Yearly Meeting. The
business of all these subordinate meetings is transacted, like that of
the Yearly Meeting, without voting, and settled similarly through the
action of the clerk when a practical unanimity is arrived at. Each
Monthly Meeting appoints “representatives” to the next Quarterly
Meeting, and the Quarterly Meetings in like manner appoint
“representatives” to the Yearly Meeting. These Friends have no very
definite function to perform, but their names are called over, and
their presence or absence noted at the opening of each meeting to
which they are sent; and they are expected to serve in a general
way as a special medium of communication between the larger and
the smaller meetings to which they belong.

In like manner, upon any subject affecting the Society at large, the
Yearly Meeting communicates with the Quarterly Meetings, who in
their turn diffuse the impulse through their own Monthly and
particular meetings, till it reaches every individual member; and, in
return, information respecting every meeting for worship is from
time to time given to the Monthly Meetings, to be by them in a
condensed form reported to the Quarterly Meetings, and so
eventually presented to the Yearly Meeting in London. All these
ascending and descending processes are carried on with 11
minute accuracy and regularity, and are duly recorded at every
stage in the books of each meeting. There is thus a complete system
of circulation, as of veins and arteries, by which every individual
member is brought within reach of the Society at large, and through
which information, influence, and discipline are carried to and from
the centre and the extremities.

The “discipline” of the Society is a matter of extreme interest, as to

which I cannot venture to say with any confidence how far our
recognized ideal is actually carried out in practice. There is no doubt
that of late years considerable changes have taken place, mainly in
the direction of a relaxation of discipline with regard to
comparatively trivial matters. Certain “queries” have from the earliest
times been appointed by the authority of the Yearly Meeting, to be
read and considered at certain seasons in the subordinate meetings,
and to most of these queries (some relating to various branches of
Christian morality, and some to regularity in attendance at meetings
and conformity to established standards of simplicity in dress and
language) it was formerly the practice to require detailed answers
from each particular meeting, to be in due course transmitted in a
summarized form to the Yearly Meeting itself. In 1861, 12
however, the Yearly Meeting issued directions that a certain
number of these queries should be merely “considered,” but not
answered. In 1875 this method was adopted with regard to nearly
all the queries, and at present those only which relate to the
regularity of attendance at meetings for worship and business are
[2]
answered. This change has a very obvious significance, and I
believe that its effect is even more marked than would be
understood by any one not accustomed to the extreme care and
gravity with which these matters were formerly pondered and
reported upon in each “preparative meeting” (i.e. each particular
meeting sitting specially with a view to preparing the business to be
transacted at any approaching Monthly Meeting), and again at each
stage of the progress of the report towards its final presentation by
the Quarterly to the Yearly Meeting. Dress and language and other
external matters are now practically left entirely to the individual
conscience, as is surely wisest. With regard to weightier matters,
such as strict integrity in business, sobriety, and correctness of moral
conduct, etc., there is still, I hope and believe, a considerable reality
of watchful care exercised through specially appointed 13
members. In every Monthly Meeting there are Friends holding
the offices of elder and overseer. The business of the elders is to
watch over the ministers in the exercise of their gift; that of the
overseers to see to the relief of the poorer members, the care of the
sick, and other such matters; to watch over the members generally
with regard to their Christian conduct, to warn privately any who
may be giving cause of offence or scandal, and in case of need to
bring the matter before the Monthly Meeting, to be dealt with as it
may require. Should the Monthly Meeting think it necessary to
disown a member for persisting in conduct not consistent with our
Christian profession, or for any other reason, the member in
question may appeal to the Quarterly Meeting, and from its decision
to that of the Yearly Meeting, which is in all cases final.

The London Yearly Meeting has two standing committees for the
transaction of such of its affairs as need attention more frequently
than once a year. One of these represents the Yearly Meeting at
large, and has charge of its money matters and other general
business; it bears the curious and suggestive title of the “Meeting for
Sufferings,” from having been originally occupied mainly in relieving
Friends under persecution. The other is a committee of the 14
Yearly Meeting on Ministry and Oversight, and is called the
“Morning Meeting.”

Meetings on Ministry and Oversight are held in every Quarterly and

Monthly Meeting as well as at the Yearly Meeting. They are
composed of all the recorded ministers, the elders and overseers of
each meeting, together with (in some Quarterly Meetings) some
Friends described as associate members, who attend them as it
were not officially, but by a standing invitation. These meetings are
concerned, of course, with questions relating to the special offices
exercised by their members.
The ministers are, as is well known, not appointed or set apart by
any human ordination, nor are any of them ever paid, or liable to be
called upon by any human authority, for any ministerial services. By
the word “ministers” we mean simply those, be they men or women,
who have received a gift and call to minister, that is to offer vocal
service, in meetings for worship. When any Friend has exercised
such a gift for a considerable time, in a manner which is recognized
by the other members as evincing a true vocation, the Monthly
Meeting proceeds to record the fact on the books of the meeting.
This acknowledgment is made merely for the sake of “good order,”
and is not supposed to confer any additional power or 15
authority on the minister “recorded.” The ministers are
perfectly free to continue their ordinary occupations, and many of
them are, in fact, engaged in earning their own living in trades,
business, or professions.

When a minister, in the exercise of his or her gift, feels called to

travel to any distant place, it is thought right that the “concern”
should be laid before the Monthly Meeting, and, should it be an
important or distant concern, before the Quarterly and, in some
cases, even the Yearly Meeting also; when the meetings in question
will, if they feel “unity” with it, give the minister a minute or
certificate to that effect, which serves as an introduction and
guarantee in whatever meetings the minister may visit during that
“service.” In such cases the ministers’ travelling expenses are paid
from one Monthly or Quarterly Meeting to another, and it is usual for
them to be welcomed into the houses of some of the Friends
belonging to the meetings visited. The extent to which Friends do
thus travel, both in England and abroad, “in the service of Truth,” is
something of which few people outside the Society have any idea.
Between England and America there is a continual interchange of
such visits, and the very copious biographical literature of the
Society teems with the records of journeys undertaken “under 16
an impression of religious duty,” and lasting sometimes for
months, or even years, before the Friend could “feel clear” of the
work. No limit is ever set beforehand to such work. It is felt to be
work in which the daily unfolding of the Divine ordering must be
watched and waited for.

Such is a general outline of what may be called the machinery of the

Society. It remains to state briefly its distinguishing tenets before
proceeding to consider the spirit and inner spring from which these
outward developments have arisen, and from which they derive all
their significance and value.

I have already referred to the peculiarity which lies at the root of all
the rest; namely, our views as to the nature of the true gospel
ministry, as a call bestowed on men and women, on old and young,
learned and unlearned; bestowed directly from above, and not to be
conferred by any human authority, or hired for money; to be
exercised under the sole and immediate direction of the one Master,
the only Head of the Church, Christ the Lord. As a consequence of
this view, Friends have, as is well known, refused as a matter of
conscience to pay tithes, or in any way to contribute to the
maintenance of a paid ministry, and of the services prescribed by the
Established Church.

Closely connected with these views on ministry, is our 17

testimony against the observance of any religious rites or
ceremonies whatever. Neither baptizing with water, nor the breaking
of bread and drinking of wine, are recognized by us as Divinely
ordained institutions of permanent obligation, and neither of these
ceremonies is practised by us. We believe that the coming of Christ
put an end to the old dispensation of outward observances, and that
the whole drift of His teaching was against the attaching of
importance to such things. The passages relating to His last supper
with His disciples, and those in which He speaks of His permanent
influence upon them under the images of bread, blood, etc., seem to
us much more intelligible and impressive when understood without
reference to the sacramental theories which have been engrafted
upon them. The one baptism “with the Holy Ghost and with fire,”
and the continual spiritual communion to be enjoyed in feeding on
the bread of life, are felt by us to be of the very essence of true and
spiritual worship; but we believe them to be entirely independent of
any outward observances. We therefore feel that no other condition
is needed for the highest acts of worship than the presence and the
right spiritual disposition of the worshippers.

The rejection of any separate priesthood, and of all outward 18

observances, is the main divergence between us and other
Christians. We have always maintained a testimony against war as
inconsistent with the full acceptance of the spirit of Christ, and
against oaths as distinctly forbidden by Him. We have also been led
to abandon the pursuit of changing fashions, and to cherish a
plainness in dress and language of a marked character, now fast
changing its type, but not, we trust, really disappearing. These
minor testimonies are probably more widely known than the more
fundamental ones; and though concerned with comparatively trivial
matters, they also spring from a deep root of principle. It is a
remarkable fact that from time to time religious bodies have sprung
up in various parts of the world who, without any communication
with us, have adopted similar views on many, if not all, of these
points. This fact, as well as the continuance and the widely spread
influence of our own Society, seems to show that its roots lie deep in
some fundamental principles of truth.

I am now about to attempt to deal with those principles, not in the

way of analysis or with any attempt at precision of language, but as
a record of their practical working, as gathered mainly from personal
experience. It is not, I confess, without some anxiety that I, as 19
a new-comer, enter upon this task. In the preceding sketch of
matters of fact, it has of course been easy to guard against any
serious misstatements; but in the following chapters I must deal
with matters less easily verifiable. It seems to me in some respects
hardly possible that any one not born and bred in the Society should
be fully qualified to unfold its principles and practices. There is, on
the other hand, in the very fact of having entered it from without, a
special qualification for the office of interpreting them to outsiders. It
will, I hope, be remembered that I have no kind of claim to speak in
any sense in the name of the Society. My object is to explain (so far
as the experience of ten years’ membership may enable me) the
secret of its strength and of its attraction for others; and for this
attempt one brought up outside its pale, and speaking in a purely
individual capacity, may well feel a special freedom. If I cannot
pretend to possess the entirely correct accent of a born Friend, I
may be none the less intelligible to those amongst whom my own
Christian principles were imbibed and nourished until the years of
maturity.

20
 CHAPTER II.
THE INNER LIGHT.

The one corner-stone of belief upon which the Society of Friends is

built is the conviction that God does indeed communicate with each
one of the spirits He has made, in a direct and living inbreathing of
some measure of the breath of His own life; that He never leaves
Himself without a witness in the heart as well as in the surroundings
of man; and that in order clearly to hear the Divine voice thus
speaking to us we need to be still; to be alone with Him in the secret
place of His presence; that all flesh should keep silence before Him.

This belief may be more precisely stated, explained, and as we think

justified, by those who are competent to deal with it in a
philosophical manner. The founders of our Society were not
philosophers, but spoke of these things from an intense and
abundant personal experience, which led them with confidence to
appeal to the experience of all sorts and conditions of men for 21
confirmation of their doctrine as to the light within. And they
were not disappointed. The history of the sudden gathering of the
Society, of its rapid formation into a strongly organized body, and of
the extraordinary constancy, zeal, and integrity displayed by its
original members, is a most impressive proof of the trueness of their
[3]
aim.

I have no ambition to clothe the fundamental doctrine of our Society

in any less popular language than that in which it was originally
preached. I would rather, even did necessity not compel me, be
content to appeal, as did the early Friends, to common experience.
My aim is to explain for practical purposes, and in modern as well as
simple language, the way in which our whole constitution as a
Society, and our various special testimonies, have resulted from this
one main principle.

When questioned as to the reality and nature of the inner light, the
early Friends were accustomed in return to ask the 22
questioners whether they did not sometimes feel something
within them that showed them their sins; and to assure them that
this same power, which made manifest, and therefore was truly
light, would also, if yielded to, lead them out of sin. This assurance,
that the light which revealed was also the power which would heal
sin, was George Fox’s gospel. The power itself was described by him
in many ways. Christ within, the hope of glory; the light, life, Spirit,
and grace of Christ; the seed, the new birth, the power of God unto
salvation, and many other such expressions, flow forth in abundant
streams of heartfelt eloquence. To “turn people to the light within,”
to “direct them to Christ, their free Teacher,” was his daily business.

For this purpose he and his friends travelled continually up and down
the country, holding meetings everywhere, and finding a never-
failing response to their appeal, as is proved by the bare numbers of
those who, within a very few years, were ready to encounter
persecution, and to maintain their testimony through long years of
imprisonment and sufferings. In the earlier days of the Society the
doctrine of the inner light was clearly one readily understood and
accepted by the ordinary English mind. In our own day it is 23
usually spoken of as a mysterious tenet, springing up now and
again in the minds of isolated enthusiasts, but indigenous only in
Oriental countries, and naturally abhorrent to the practical common
sense of our own people.

The difference arises, I think, from the fact that there are circles
within circles, or spheres within spheres, and that the light to which
the early Friends bore witness was not confined to that innermost
sanctuary of whose very existence, perhaps, none but a few
“mystics” are conscious; but that, while proceeding from those
deepest depths, it was recognized as also lighting up conscience,
and conduct, and all the tangible outer framework of life; and that it
was called “within” not alone in the sense of lying nearer the centre
of our being than anything else, but also in the (to ordinary minds)
more intelligible sense of beginning at home—of being the reward of
each man’s own faithfulness, of being independent of priests and
ordinances. The religion they preached was one which enforced the
individual responsibility of each one for his own soul; it was a
portable and verifiable religion—a religion which required truth in
word and deed, plain dealing and kindness and self-control, and
which did not require ceremonial observances or priestly 24
guarantees; a religion in which practice went for more than
theory, and all were expected to take their stand on one level, and
their share in the worship and the business of “the Church.” It is
easy to see how such preaching as this would commend itself to
English independence. It surely commends itself to the unchanging
sense of truth in the human heart, and will be welcomed whenever it
is preached from first-hand experience of its power.

“That which you seek without you have already within you.” The
words which changed the life of Madame Guyon will never lose their
power while human nature is occupied with the struggle for a state
of stable equilibrium. The perennial justification of Quakerism lies in
its energetic assertion that the kingdom of heaven is within us; that
we are not made dependent upon any outward organization for our
spiritual welfare. Its perennial difficulty lies in the inveterate
disposition of human beings to look to each other for spiritual help,
in the feebleness of their perception of that Divine Voice which
speaks to each one in a language no other ear can hear, and in the
apathy which is content to go through life without the attempt at
any true individual communion with God.

“The kingdom of heaven is within us.” No Christian, surely, can 25

dispute the truth of this deep word of Christ Himself. But its
interpretation has a wide range. In his own lips it was used in
opposition to the “Lo here! and lo there!” for which he was preparing
His disciples. They were not to be hurried away into a search for
Christ in all directions, but were to remember that His kingdom
(surely implying His living presence) is in the hearts of His people.
He Himself makes none of those abstruse distinctions between
consciousness and being, accident and essence, subject and object,
or even superficial and profound, and so forth, which it has been the
delight of many of His most devoted followers to interweave with
this simple expression “within you.”

I think it is inevitable that the more deeply we penetrate into the

recesses of the human mind, the more we should have a sense of
approaching an inner sanctuary, and that there is a very real and
deep sense in which this word “within you” may be understood as
meaning “above all in your inmost depths.” But this is not its original
or its obvious meaning. In the teaching of our Lord there is a
frequent reference to the distinction of inward or outward, but the
distinction is drawn in a broad and simple manner. It is oftenest a
demand upon our sincerity and thoroughness, not upon our 26
powers of introspection—an appeal on behalf of the weightier
matters of the Law as compared with trivial and ceremonial
observances. It would scarcely, I think, be true to say that the
doctrine of an “inner light,” as we understand it, is explicitly laid
down in the Gospels, although, to my own mind, that doctrine
appears to be an almost inevitable inference from their teaching. I
am not, however, attempting to deal with the question on its merits.
I only wish to draw attention to the wide range of meaning covered
by such expressions as “the light within,” and “the inner light.”

Both by our Master Himself, and by the Friends who originally

preached Him as the Light, the figure of light was used in a broad
and popular sense. Light is the most obvious and the most eternally
satisfying figure for Divine truth. It is, however, hardly more obvious
or more satisfying than the other figure so commonly, and almost
interchangeably, used by the same teachers, of breath—inspiration. I
scarcely know whether it would convey most truth to say that the
cornerstone of our Society was a belief in “the light within,” or in
“immediate inspiration.” I doubt whether the two ideas are in all
respects altogether distinguishable. Belief in the fact to which they
both refer, of an actual Divine influence communicated to 27
[4]
every human spirit, is our real corner-stone.

The fact of inspiration is denied by no Christian—the full recognition

of its present and constant operation is in some degree a peculiarity
of Friends. It is not uncommon outside the Society to hear
expressions implying that Divine inspiration is a thing of the past; a
quite exceptional gift, familiar only in apostolic times. It seems to me
that this limitation of its range amounts almost to a denial of its
reality. I can hardly understand the idea that God did occasionally
long ago speak to human beings, but that He never does so now. It
seems, at any rate, inconsistent with any worthy sense of His
unchangeableness.

Many of us have come to believe that one of the greatest 28

hindrances to a real belief in or recognition of inspiration has
been the exceedingly crude and mechanical conception of it as
attributed to the letter of Scripture. From this hard and shallow way
of thinking about inspiration, Friends have generally been preserved
in proportion as they have held firmly the old Quaker doctrine of the
inner light. Some, no doubt, have gone too far in the direction of
transferring the idea of infallibility from the Bible to themselves. But,
on the whole, I believe the doctrine of Fox and Barclay (i.e., briefly,
that the “Word of God” is Christ, not the Bible, and that the
Scriptures are profitable in proportion as they are read in the same
spirit which gave them forth) to have been a most valuable
equipoise to the tendency of other Protestant sects to transfer the
idea of infallibility from the Church to the Bible. Nothing, I believe,
can really teach us the nature and meaning of inspiration but
personal experience of it. That we may all have such experience if
we will but attend to the Divine influences in our own hearts, is the
cardinal doctrine of Quakerism. Whether this belief, honestly acted
on, will manifest itself in the homespun and solid, but only too sober
morality of the typical everyday Quaker, or whether it will land us in
the mystical fervours of an Isaac Penington, or the apostolic 29
labours of a John Woolman or a Stephen Grellet, must depend
chiefly upon our natural temperament and special gifts. The range of
the different forms taken by the doctrine is as wide as the range of
human endowment and experience. A belief which is the common
property of the prophet and the babe will, of course, yield every
variety of practical result.

It is a belief which it is hardly possible to inculcate by anything more

or less than a direct appeal to experience, to the witness within; and
there is the further difficulty, that the experience to which we can
appeal only as sharers in it, must be expressed in language very
often and very naturally misunderstood. The assertion, however
guarded, that one has actual experience of Divine inspiration in one’s
own person, is very apt to sound like a claim to personal infallibility.
Yet in reality nothing can be further from the mark. The first effect
of the shining of light within is to show what is amiss—to “convince
of sin.” It is not claiming any superiority to ordinary human
conditions to say, in response to such an appeal as that of the
Friends just referred to, “Yes, I have indeed been conscious of a
power within making manifest to me my sins and errors, and I have
indeed experienced its healing and emancipating power as 30
well as its fiery purgings and bitter condemnations. That which
has shown me my fault has healed me; the light has led and is
leading me onwards and upwards out of the abyss, nearer and
nearer to its own eternal Source; and I know that, in so far as I am
obedient to it, I am safe.” What is such a reply but an
acknowledgment that “the light, the Spirit, and grace of Christ” have
indeed been an indwelling, inbreathing power in one’s own heart? If
it be a claim to inspiration, it is a claim which implies no merit and
no eminence in him who makes it; it is made on ground common to
the publican, the prodigal, and the sinner, to Magdalen and to Paul.
It is the history of every child returning to the Father’s house.

But it is not every one to whom it would be natural to describe this

experience in language so mystical as this, nor would the mystic’s
experience be likely to stop short at anything so simple and
elementary as the process just described. And here we are
confronted with the real “peculiarity” of Quakerism—its relation to
mysticism. There is no doubt that George Fox himself and the other
fathers of the Society were of a strongly mystical turn of mind,
though not in the sense in which the word is often used by the
worshippers of “common sense,” as a mild term of reproach, 31
to convey a general vague dreaminess. Nothing, certainly,
could be less applicable to the early Friends than any such reproach
as this. They were fiery, dogmatic, pugnacious, and intensely
practical and sober-minded. But they were assuredly mystics in what
I take to be the more accurate sense of that word—people, that is,
with a vivid consciousness of the inwardness of the light of truth.

Mysticism in this sense is a well-known phenomenon, of which a

multitude of examples may be found in all religions. It is, indeed,
rather a personal peculiarity than a form of belief; and therefore,
although from time to time associations (our own, for one) have
been based upon what are called mystical tenets, there can scarcely
be anything like a real school of mysticism—at any rate, in Europe.
Mysticism, as we know it, is essentially individual. It refuses to be
formulated or summed up. In one sense it is common to all religious
persuasions; in another, it equally eludes them all. We can easily
understand what constitutes a mystic, but the peculiarity itself is
incommunicable. Their belief is an open secret. They themselves
have ever desired to communicate it, though continually feeling the
impossibility of doing so by words alone. It is the secret of 32
light—an inward light clothing itself in life, and living to bring
all things to the light.

Mystics, as I understand the matter, are those whose minds, to their

own consciousness, are lighted from within; who feel themselves to
be in immediate communication with the central Fountain of light
and life. They have naturally a vivid sense both of the distinction and
of the harmony between the inward and the outward—a sense so
vivid that it is impossible for them to believe it to be unshared by
others. A true mystic believes that all men have, as he himself is
conscious of having, an inward life, into which, as into a secret
[5]
chamber, he can retreat at will. In this inner chamber he finds a
refuge from the ever-changing aspects of outward existence; from
the multitude of cares and pleasures and agitations which belong to
the life of the senses and the affections; from human judgments;
from all change, and chance, and turmoil, and distraction. He finds
there, first repose, then an awful guidance; a light which burns and
purifies; a voice which subdues; he finds himself in the 33
presence of his God. It is here, in this holy of holies, that
“deep calleth unto deep;” here that the imperishable, unfathomable,
unchanging elements of humanity meet and are one with the Divine
Fountain of life from whence they flow; here that the well of living
waters springeth up unto eternal life.

“The kingdom of heaven is within you.” Personal religion is a real

and a living thing only in proportion as it springs from this deep
inward root. The root itself is common to all true believers. The
consciousness of its “inwardness” is that which distinguishes the
mystic. How it should be that to some minds the words “inward and
outward” express the most vivid and continuous fact of
consciousness, while to others they appear to have no meaning at
all; how it comes that some are born mystics, while to others the
report of the mystic concerning the inner life is a thing impossible to
be believed and hardly to be understood;—these are psychological
problems I cannot attempt to unravel. If, however, a certain
correspondence between the inward and the outward do really exist
(and this, I suppose, will hardly be denied, whatever may be the
most philosophically accurate way of expressing it), the faculty of
discerning it must needs be a gift. I believe, indeed, that the 34
power in this direction which distinguishes such mystics as,
e.g., Thomas à Kempis, Jacob Boehme, Tauler, Fénélon, Madame
Guyon, George Fox, William Law, St. Theresa, Molinos, and others, is
essentially the same gift which in a different form, or in combination
with a different temperament and gifts of another order, makes
poets. It is the gift of seeing truth at first-hand, the faculty of
receiving a direct revelation. To have it is to be assured that it is the
common inheritance, the “light which lighteth every man that
cometh into the world.” Preachers like those I have just mentioned
always appeal to it with confidence as to a witness to be found in
every heart. And surely experience confirms this conviction of theirs.
It is in degree only that their gift is exceptional. They may have the
sight of the eagle, but they see by the same light as the bat.

Now, the obvious tendency of a vivid first-hand perception of truth,

or light, is to render the possessor of it so far independent of
external teachers. And we all know that in point of fact such
illuminati always have shown a disposition to go their own way, and
to disregard, if not to denounce, traditional teaching, which has
brought them into frequent collisions with ecclesiastical and other
authorities. Those of the Church of Rome have, with their 35
wonted sagacity, as much as possible sought to turn this
strange power to account, while providing safety-valves for the
unmanageable residue.

It is the easier to do this because of the two marked characteristics

of mystics—quietness and independence. Mystics are naturally
independent, not only of ecclesiastical authority, but of each other.
This is necessarily implied in the very idea of first-hand reception of
light. While it must always constitute a strong bond of sympathy
between those who recognize it in themselves and in each other, it
naturally indisposes them to discipleship. They sit habitually at no
man’s feet, and do not as a rule greatly care to have any one sit at
theirs. Mysticism in this sense seems naturally opposed to tradition.
No true mystic would hold himself bound by the thoughts of others.
He does not feel the need of them, being assured of the sufficiency
and conscious of the possession of that inward guidance, whether
called light, or voice, or inspiration, which must be seen, heard, felt,
by each one in his own heart, or not at all. But the duty of looking
for and of obeying this guidance is a principle which may be
inculcated and transmitted from generation to generation like any
other principle. Its hereditary influence is very perceptible in 36
old Quaker families, where a unique type of Christian
character resulting from it is still to be met with.

Quietness naturally accompanies the belief in this inward guidance,

not only because in the Divine presence all that is merely human
necessarily sinks into silent insignificance, but also because it is
instinctively felt that it is only in stillness that any perfect reflection
from above can be formed in the mirror of the human spirit. The
natural fruit of mysticism is quietism.

I have no means of estimating the actual prevalence of mystical and

quietist principles in the Society of Friends at the present time. But I
am sure that our Society is the natural home for the spirits of all
those who hold them, for it is the one successful embodiment of
these principles in a system of “Church government.” Every
arrangement is made to favour and to maintain the practice of
looking for individual inward guidance, and to give the freest scope
to its results. Everything which tends to hinder obedience to it is
abandoned and discouraged. I shall endeavour to trace the working
of this aim in various special directions hereafter. I must now
endeavour to explain as well as I can what it is precisely that I
understand by that inward light, voice, or Divine guidance 37
which we Friends believe it our duty and our highest privilege
in all things to watch for.

I do not, indeed, claim that my own share in this deepest region of

human experience amounts to more than a faint and intermittent
glimmering of what I know to be possible. I earnestly desire to
explain to others what to myself has been especially blessed and
helpful in the deepest unfoldings, whether by word or in life, of
Quaker principles; but I feel that the task would demand for its full
accomplishment not only greater powers than mine, but also the
assistance which can be given only by something more than candour
in the reader—by a real desire to help out the stammering utterance,
and to supply the gaps left by individual shortcomings. To such a
helpful auditor, therefore, I will in imagination address myself.

Faithfulness to the light is the watchword of all who hunger and

thirst after righteousness—of all seekers after the kingdom of
heaven. Is this merely an equivalent for the more commonplace
expression, “obedience to conscience”? Surely not. Conscience, as
we all know, is liable to perversion, to morbid exaggerations, to
partial insensibility, to twists and crotchets of all sorts, and itself
needs correction by various external standards. Conscience,
therefore, can never be our supreme and absolute guide. 38
Whether it can ever be right to disobey it, must depend on the
precise meaning we attach to the words “conscience” and “right,”
and into this puzzle I have no intention of entering. In a broad and
practical sense, we all know that if there were nothing above
conscience, conscience would assuredly lead many of us into the
ditch; nay, that, for want of enlightenment from above, it actually
has led many there. The light by which our consciences must be
enlightened, the light in obedience to which is our supreme good,
must be something purer than this fallible faculty itself. It must be
that power within us, if any such power there be, which is one with
all the wisdom, all the goodness, all the order and harmony, without
us; one with “the power, not ourselves, which makes for
righteousness;” one with “the eternal will towards all goodness.” It
must be a power as all-pervading and immanent in the spirit of man
as is the power of gravity (or whatever yet more elementary force
gravity may be resolved into) in the outer world he inhabits. It must
be the power in which we live and move and have our being—the
power and the presence of God.

I do not attempt—idle indeed would be the attempt in such hands as

mine—to contribute anything towards the arguments in favour of
Theism. To those who do not believe in the existence of the 39
living God, the whole subject upon which I am engaged must
be without interest or significance. And I leave it to others to
reconcile, or to show that we need not attempt to reconcile, the