Building a Monitoring Infrastructure with Nagios

1st Edition David Josephsen pdf download

https://ebookname.com/product/building-a-monitoring-
infrastructure-with-nagios-1st-edition-david-josephsen/

Get Instant Ebook Downloads – Browse at https://ebookname.com

Instant digital products (PDF, ePub, MOBI) available
Download now and explore formats that suit you...

Building a Monitoring Infrastructure with Nagios 1st

Edition David Josephsen

https://ebookname.com/product/building-a-monitoring-
infrastructure-with-nagios-1st-edition-david-josephsen/

Building a Windows IT Infrastructure in the Cloud

Distributed Hosted Environments with AWS 1st Edition
David K. Rensin

https://ebookname.com/product/building-a-windows-it-
infrastructure-in-the-cloud-distributed-hosted-environments-with-
aws-1st-edition-david-k-rensin/

Nagios System and Network Monitoring 2nd Ed Second

Edition Wolfgang Barth

https://ebookname.com/product/nagios-system-and-network-
monitoring-2nd-ed-second-edition-wolfgang-barth/

Wait How Do I Write This Email 2nd Edition Danny Rubin

https://ebookname.com/product/wait-how-do-i-write-this-email-2nd-
edition-danny-rubin/
Host Identity Protocol HIP Towards the Secure Mobile
Internet Wiley Series on Communications Networking
Distributed Systems 1st Edition Andrei Gurtov

https://ebookname.com/product/host-identity-protocol-hip-towards-
the-secure-mobile-internet-wiley-series-on-communications-
networking-distributed-systems-1st-edition-andrei-gurtov/

Computer Security and Penetration Testing 2nd Edition

Alfred Basta

https://ebookname.com/product/computer-security-and-penetration-
testing-2nd-edition-alfred-basta/

Decision Making in Pain Management 2nd Edition Somayaji

Ramamurthy Md

https://ebookname.com/product/decision-making-in-pain-
management-2nd-edition-somayaji-ramamurthy-md/

The Encyclopedia of Philosophy 2nd Edition Donald M.

Borchert

https://ebookname.com/product/the-encyclopedia-of-philosophy-2nd-
edition-donald-m-borchert/

Lean Sigma A Practitioner s Guide 2nd Edition Ian

Wedgwood Phd

https://ebookname.com/product/lean-sigma-a-practitioner-s-
guide-2nd-edition-ian-wedgwood-phd/
Current Clinical Evidence in Ophthalmology 1st Edition
H.W. Roberts

https://ebookname.com/product/current-clinical-evidence-in-
ophthalmology-1st-edition-h-w-roberts/
www.dbeBooks.com - An Ebook Library
B UILDING A M ONITORING
INFRASTRUCTURE WITH
N AGIOS
This page intentionally left blank
B UILDING A M ONITORING
INFRASTRUCTURE WITH
N AGIOS

David Josephsen

Upper Saddle River, NJ • Boston • Indianapolis • San Francisco

New York • Toronto • Montreal • London • Munich • Paris
Madrid • Cape Town • Sydney • Tokyo • Singapore • Mexico City
Many of the designations used by manufacturers and sellers to distinguish their products are claimed as trade-
marks. Where those designations appear in this book, and the publisher was aware of a trademark claim, the des-
ignations have been printed with initial capital letters or in all capitals.
The author and publisher have taken care in the preparation of this book, but make no expressed or implied war-
ranty of any kind and assume no responsibility for errors or omissions. No liability is assumed for incidental or
consequential damages in connection with or arising out of the use of the information or programs contained
herein.
The publisher offers excellent discounts on this book when ordered in quantity for bulk purchases or special sales,
which may include electronic versions and/or custom covers and content particular to your business, training
goals, marketing focus, and branding interests. For more information, please contact:
U.S. Corporate and Government Sales
(800) 382-3419
corpsales@pearsontechgroup.com

For sales outside the United States please contact:

International Sales
international@pearsoned.com

This Book Is Safari Enabled

The Safari® Enabled icon on the cover of your favorite technology book means the book is available
through Safari Bookshelf. When you buy this book, you get free access to the online edition for 45 days.
Safari Bookshelf is an electronic reference library that lets you easily search thousands of technical books,
find code samples, download chapters, and access technical information whenever and wherever you need
it.

If you have difficulty registering on Safari Bookshelf or accessing the online edition, please e-mail customer-
service@safaribooksonline.com.

Visit us on the Web: www.prenhallprofessional.com

Library of Congress Cataloging-in-Publication Data
Josephsen, David.
Building a Monitoring Infrastructure with Nagios / David Josephsen, 1st ed.
p. cm.
Includes bibliographical references.
ISBN 0-13-223693-1 (pbk. : alk. paper)
1. Computer systems—Evaluation. 2. Computer systems—Reliability.
3. Computer networks—Monitoring. I. Title.

QA76.9.E94J69 2007
004.2’4--dc22
2006037765
Copyright © 2007 Pearson Education, Inc.
All rights reserved. Printed in the United States of America. This publication is protected by copyright, and permis-
sion must be obtained from the publisher prior to any prohibited reproduction, storage in a retrieval system, or
transmission in any form or by any means, electronic, mechanical, photocopying, recording, or likewise. For infor-
mation regarding permissions, write to:
Pearson Education, Inc.
Rights and Contracts Department
75 Arlington Street, Suite 300
Boston, MA 02116
Fax: (617) 848-7047

ISBN 0-132-23693-1
Text printed in the United States on recycled paper at R.R. Donnelley & Sons in Crawfordsville, IN 60#
in Williamsburg
First printing, February 2007
For Gu, for enduring and encouraging my incessant curiosity.
And for Tito, the cat with the biggest heart.
This page intentionally left blank
 CONTENTS

Acknowledgments xiii

About the Author xv

About the Technical Reviewers xvii

Introduction xix
Do it Right the First Time xix
Why Nagios? xx
What’s in This Book? xxii
Who Should Read This Book? xxiv

CHAPTER 1 Best Practices 1

A Procedural Approach to Systems Monitoring 1
Processing and Overhead 4
Remote Versus Local Processing 4
Bandwidth Considerations 4
Network Location and Dependencies 6
Security 7
Silence Is Golden 9
Watching Ports Versus Watching Applications 11
Who’s Watching the Watchers? 11

CHAPTER 2 Theory of Operations 13

The Host and Service Paradigm 14
Starting from Scratch 14
Hosts and Services 15

vii
viii Contents

Interdependence 16
The Down Side of Hosts and Services 17
Plugins 18
Exit Codes 18
Remote Execution 20
Scheduling 23
Check Interval and States 23
Distributing the Load 26
Reapers and Parallel Execution 27
Notification 28
Global Gotchas 28
Notification Options 29
Templates 30
Time Periods 30
Scheduled Downtime, Acknowledgments, and Escalations 31
I/O Interfaces Summarized 32
The Web Interface 32
Monitoring 33
Reporting 36
The External Command File 37
Performance Data 37
The Event Broker 38

CHAPTER 3 Installing Nagios 39

OS Support and the FHS 39
Installation Steps and Prerequisites 41
Installing Nagios 41
Configure 42
Make 44
Make Install 44
Patches 45
Secondary IP Patch 46
SNMP Community String Patch 46
Colored Statusmap Patch 46
Installing the Plugins 47
Installing NRPE 48
Contents ix

CHAPTER 4 Configuring Nagios 51

Objects and Definitions 52
nagios.cfg 54
The CGI Config 57
Templates 58
Timeperiods 60
Commands 61
Contacts 62
Contactgroup 64
Hosts 64
Services 66
Hostgroups 68
Servicegroups 69
Escalations 69
Dependencies 70
Extended Information 72
Apache Configuration 72
GO! 73

CHAPTER 5 Bootstrapping the Configs 75

Scripting Templates 76
Auto-Discovery 79
Nmap and NACE 79
Namespace 81
GUI Configuration Tools 82
Fruity 82
Monarch 83

CHAPTER 6 Watching 85
Local Queries 85
Pings 86
Port Queries 88
Querying Multiple Ports 90
(More) Complex Service Checks 92
E2E Monitoring with WebInject 94
x Contents

Watching Windows 98
The Windows Scripting Environment 98
COM and OLE 100
WMI 101
To WSH or not to WSH 105
To VB or Not to VB 106
The Future of Windows Scripting 107
Getting Down to Business 109
NRPE 109
NSClient/NSCPlus 111
Watching UNIX 112
NRPE 113
CPU 113
Memory 116
Disk 118
Watching “Other Stuff ” 119
SNMP 119
Working with SNMP 122
Environmental Sensors 126
Stand-alone Sensors 127
LMSensors 128
IPMI 129

CHAPTER 7 Visualization 131

Foundations, MRTG, and RRDTool 132
MRTG 135
RRDTool 135
RRD Data Types 136
Heartbeat and Step 137
Min and Max 139
Round Robin Archives 139
RRDTool Create Syntax 140
Data Collection and Polling 145
Shopping for Glue 145
NagiosGraph 146
Front-Ends and Dashboards 149
RRDTool Graph Mode 149
Contents xi

RPN 152
Shopping for Front-Ends 154
drraw 155
Management Interfaces 158
Know What You’re Doing 159
RRDTool Fetch Mode 162
The GD Graphics Library 164
NagVis 166
GraphViz 167
Sparklines 169
Force Directed Graphs with jsvis 171

CHAPTER 8 Nagios Event Broker Interface 173

Function References and Callbacks in C 173
The NEB Architecture 175
Implementing a Filesystem Interface Using NEB 178

APPENDIX A Configure Options 193

APPENDIX B nagios.cfg and cgi.cfg 197

APPENDIX C Command-Line Options 207

Nagios 207
Nagios Binary 207
Plugins 208
check_ping 208
check_tcp 209
check_http 211
check_load 213
check_disk 213
check_procs 215

Index 217
This page intentionally left blank
 ACKNOWLEDGMENTS

I’m terrified to think of the wrong I might do by leaving someone out of this section. Though
I’m tempted to give in to that fear and abandon the endeavor entirely, it wouldn’t be fair
because the wisdom, encouragement, and time I’ve received from those around me are gifts
that cannot go unacknowledged.
First in line is my wife Cynthia, who has put on hold a year’s worth of projects to make
time for me to write. She is patient and encouraging and pretty, and I love her.
Thanks to my parents for being so thrilled and supportive throughout the duration.
To my surrogate family: Jose, Elodia, Claudia, and Ana, for their warmth and well
wishes.
Tito, Chuck, Lucas, Gillie, Thunder, Daemos, Phobos, and Gus, who brighten my
days and have had to make due with a small house until I could get the time to fix up
the new one.
Jer, the best co-author, mentor, and friend a guy could ask for.
I owe my boss at DBG, Todd Rowan, more than a little time and attention.
The tech reviewers on this project were outstanding. In no particular order: Russell
Adams, Kate Harris, Chris Burgess, and Taylor Dondich. I want you guys to know that I’m
working on figuring out the difference between “weather” and “whether” and shortly there-
after, I plan to tackle the intricacies of the apostrophe.
Lastly, my editors at Prentice Hall have been great. They aren’t at all like the editors in
Spiderman or Fletch, which is what I was expecting. Catherine Nolan, Raina Chrobak, and
Mark Taub are an amazingly hardworking, on-the-ball, and clued-in group of professionals.
They’ve been patient and helpful, and I appreciate their time and attention.
Thanks.

xiii
This page intentionally left blank
 ABOUT THE AUTHOR

Dave Josephsen is the senior systems administrator at DBG, where he maintains a geographi-
cally dispersed collection of server farms and occasionally puts paper in the printer. Winner
of LISA 2004’s Best Paper Award and author of numerous articles, he enjoys writing about
technology, but admittedly, has more fun solving interesting problems and getting his hands
dirty with routers, firewalls, load balancers, and UNIX systems. His interests are too numer-
ous to list; he is quite uncomfortable writing about himself in the third person, and he’s hav-
ing so much fun he can’t believe he’s being paid. (But he’d prefer that you not mention that
last bit to his boss or publishers.)

xv
This page intentionally left blank
 ABOUT THE TECHNICAL REVIEWERS

Russell Adams
Russell Adams (rladams@adamsinfoserv.com) is an enterprise systems consultant. He has
been working with Linux since the mid 1990s. Russell specializes in high-availability cluster-
ing, automated systems management, and network monitoring solutions. He is a member of
the Houston Area Linux Users Group and contributes software to the OSS community.

Chris Burgess
Chris Burgess has more than ten years of experience in information technology and cur-
rently works as a system administrator for a nonprofit organization in Melbourne, Australia.
He also works as a freelance Red Hat Certified Engineer on a broad range of UNIX and
Windows systems. Furthermore, Chris has worked as an independent security consultant
for several organizations specializing in Internet security. He enjoys making presentations at
conferences and providing training sessions on a variety of technologies to IT professionals
and end-users. Chris is also the Victorian President of the System Administrators Guild of
Australia and regularly coordinates and attends industry and user group meetings.

Taylor Dondich
Previously working in the enterprise network engineering market, Taylor Dondich is
now a senior developer at Groundwork Open Source Solutions, a provider of open source
IT operations management software and services. Taylor has extensive experience in imple-
menting open source tools to provide the solution to IT management tasks. Taylor is also the
author of Fruity, one of the leading Nagios configuration tools available as open source.

Kate Harris
Kate Harris (kate@totkat.org) has been playing with computers since 1980, and despite
a master’s degree and very nearly a Ph.D. in materials science, she has had the pleasure of
being paid to do so for the last ten years. She has brought Nagios into organizations that
were paying vast sums of money for less effective solutions. Kate also specializes in herding
cats or, in other words, managing system administrators.

xvii
This page intentionally left blank
 INTRODUCTION

This is a book about untrustworthy machines; machines, in fact, that are every bit as
untrustworthy as they are critical to our well-being. But I don’t need to bore you with a
laundry list of how prevalent computer systems have become or with horror stories about
what can happen when they fail. If you picked up this book, then I’m sure you’re aware of
the problems: layer upon layer of interdependent libraries hiding bugs in their abstraction,
script kiddies, viruses, DDOS attacks, hardware failures, end-user errors, back-hoes, hur-
ricanes, and on and on. It doesn’t matter whether the root cause is malicious or accidental,
your systems will fail. When they do fail, only two things will save you from the downtime:
redundancy and monitoring systems.

Do It Right the First Time

In concept, monitoring systems are simple: an extra system or collection of systems whose
job is to watch the other systems for problems. For example, the monitoring system can peri-
odically connect to a Web server to make sure it responds and, if not, send notifications to
the administrators. Although it sounds straightforward, monitoring systems have grown into
expensive, complex pieces of software. Many now have agents larger than 500 MB, include
proprietary scripting languages, and sport price tags above $60,000.
When implemented correctly, a monitoring system can be your best friend. It can notify
administrators of glitches before they become crises, help architects tease out patterns cor-
responding to chronic interoperability issues, and give engineers detailed capacity planning
information. A good monitoring system can help the security guys correlate interesting
events, show the network operations center personnel where the bandwidth bottlenecks are,
and provide management with much needed high-level visibility into the critical systems that
they bet their business on. A good monitoring system can help you uphold your service level
agreement (SLA) and even take steps to solve problems without waking anyone up. Good
monitoring systems save money, bring stability to complex environments, and make every-
one happy.
When done poorly, however, the same system can wreak havoc. Bad monitoring systems
cry wolf at all hours of the night so often that nobody pays attention anymore; they install
backdoors into your otherwise secure infrastructure, leech time and resources away from

xix
xx Introduction

other projects, and congest network links with megabyte upon megabyte of health checks.
Bad monitoring systems can really suck.
Unfortunately, getting it right the first time isn’t as easy as you might think, and in my
experience, a bad monitoring system doesn’t usually survive long enough to be fixed. Bad
monitoring systems are too much of a burden on everyone involved, including the systems
being monitored. In this context, it’s easy to see why large corporations and governments
employ full-time monitoring specialists and purchase software with six-figure price tags.
They know how important it is to get it right the first time.
Small- to medium-sized businesses and universities can have environments as complex
or even more complex then large companies, but they obviously don’t have the luxury of
high-priced tools and specialized expertise. Getting a well-built monitoring infrastructure in
these environments, with their geographically dispersed campuses and satellite offices, can
be a challenge. But having spent the better part of the last 7 years building and maintaining
monitoring systems, I’m here to tell you that not only is it possible to get it done right the
first time, but you can also do it for free, with a bit of elbow grease, some open source tools,
and a pinch of imagination.

Why Nagios?
Nagios is, in my opinion, the best system and network-monitoring tool available, open
source or otherwise. Its modularity and straightforward approach to monitoring make it
easy to work with and highly scalable. Further, Nagios’s open source license makes it freely
available and easy to extend to meet your specific needs. Instead of trying to do everything
for you, Nagios excels at interoperability with other open source tools, which makes it
flexible. If you’re looking for a monolithic piece of software with check boxes that solve all
your problems, this probably isn’t the book for you. But before you stop reading, give me
another paragraph or two to convince you that the check boxes aren’t really what you’re
looking for.
The commercial offerings get it wrong because their approach to the problem assumes
that everyone wants the same solution. To a certain extent, this is true. Everyone has a large
glob of computers and network equipment and wants to be notified if some subset of it fails.
So, if you want to sell monitoring software, the obvious way to go about it is to create a piece
of software that knows how to monitor every conceivable piece of computer software and
networking gear in existence. The more gadgets your system can monitor, the more people
you can sell it to. To someone who wants to sell monitoring software, it’s easy to believe that
monitoring systems are turnkey solutions and whoever’s software can monitor the largest
number of gadgets wins.
The commercial packages I’ve worked with all seem to follow this logic. Not unlike the
Borg, they are methodically locating new computer gizmos and adding the requisite moni-
toring code to their solution, or worse: acquiring other companies that already know how
to monitor lots of computer gadgetry and bolting those companies’ codes onto their own.
Introduction xxi

They quickly become obsessed with features, creating enormous spreadsheets of supported
gizmos. Their software engineers exist so that the presales engineers can come to your office
and say to your managers, through seemingly layers of white gleaming teeth, “Yes, our soft-
ware can monitor that.”
The problem is that monitoring systems are not turnkey solutions. They require a large
amount of customization before they start solving problems, and herein lies the difference
between people selling monitoring software and those designing and implementing monitor-
ing systems. When you’re trying to build a monitoring system, a piece of software that can
monitor every gadget in the world by clicking a check box is not as useful to you as the one
that makes it easy to monitor what you need, in exactly the manner that you want. By focus-
ing on what to monitor, the proprietary solutions neglect the how, which limits the context
in which they may be used.
Take ping, for example. Every monitoring system I’ve ever dealt with uses ICMP echo
requests, also known as pings, to check host availability in one way or another. But if you
want to control how a proprietary monitoring system uses ping, architectural limitations
become quickly apparent. Let’s say I want to specify the number of ICMP packets to send or
want to send notifications based on the round-trip time of the packet in microseconds instead
of simple pass/fail. More complex environments may necessitate that I use IPv6 pings, or that
I portknock before I ping. The problem with the monolithic, feature-full approach is that
these changes represent changes to the core application logic and are, therefore, nontrivial
to implement.
In the commercial-monitoring applications I’ve worked with, if these ping examples can
be performed at all, they require re-implementing the ping logic in the monitoring system’s
proprietary scripting language. In other words, you would have to toss out the built-in ping
functionality altogether. Perhaps controlling the specifics of ping checks is of questionable
value to you, but if you don’t actually have any control over something as basic as ping, what
are the odds that you’ll have finite enough control over the most important checks in your
environment? They’ve made the assumption that they know how you want to ping things,
and from then on it was game over; they never thought about it again. And why would they?
The ping feature is already in the spreadsheet, after all.
When it comes to gizmos, Nagios’s focus is on modularity. Single-purpose monitoring
applets called plugins provide support for specific devices and services. Rather than par-
ticipating in the feature arms race, hardware support is community-driven. As community
members have a need to monitor new devices or services, new plugins are written and usually
more quickly than the commercial applications can add the same support. In practice, Nag-
ios always supports everything you need it to and without ever needing to upgrade Nagios
itself. Nagios also provides the best of both worlds when it comes to support, with several
commercial options, as well as a thriving and helpful community that provides free support
through various forums and mailing lists.
Choosing Nagios as your monitoring platform means that your monitoring effort will be
limited only by your own imagination, technical prowess, and political savvy. Nagios can go
anywhere you want it to, and the trip there is usually simple. Although Nagios can do every-
xxii Introduction

thing the commercial applications can and more, without the bulky insecure agent install, it
usually doesn’t compare favorably to commercial-monitoring systems because when spread-
sheets are parsed, Nagios doesn’t have as many checks. If they’re counting correctly, Nagios
has no checks at all, because technically it doesn’t know how to monitor anything; it prefers
that you tell it how. How, in fact, is exactly the variable that the aforementioned check box
cannot encompass. Check boxes cannot ask how; therefore, you don’t want them.

What’s in This Book?

Although Nagios is the biggest piece of the puzzle, it’s only one of the myriad of tools that
make up a world-class open source monitoring system. With several books, superb online
documentation, and lively and informative mailing lists, it’s also the best documented piece
of the puzzle. So my intention in writing this book is to pick up where the documentation
leaves off. This is not a book about Nagios as much as it is a book about the construction
of monitoring systems using Nagios, and there is much more to building monitoring systems
than configuring a monitoring tool.
I cover the usual configuration boilerplate, but configuring and installing Nagios is not
my primary focus. Instead, to help you build great monitoring systems, I need to introduce
you to the protocols and tools that enhance Nagios’s functionality and simplify its con-
figuration. I need to give you an in-depth understanding of the inner workings of Nagios
itself, so you can extend it to do whatever you might need. I need to spend some time in
this book exploring possibilities because Nagios is limited only by what you feel it can do.
Finally, I need to write about things only loosely related to Nagios, such as best practices,
SNMP, visualizing time-series data, and various Microsoft scripting technologies, such as
WMI and WSH.
Most importantly, I need to document Nagios itself in a different way than normal. By
introducing it in terms of a task-efficient scheduling and notification engine, I can keep things
simple while talking about the internals upfront. Rather than relegating important informa-
tion to the seldom-read advanced section, I empower you early on by covering topics such as
plugin customization and scheduling as core concepts.
Although the chapters stand on their own and I’ve tried to make the book as reference-
friendly as possible, I think it reads better as a progression from start to end. I encourage
you to read from cover to cover, skipping over anything you are already familiar with. The
text is not large, but I think you’ll find it dense with information and even the most-seasoned
monitoring veterans should find more than a few nuggets of wisdom.
The chapters tend to build on each other and casually introduce Nagios-specific details
in the context of more general monitoring concepts. Because there are many important deci-
sions that need to be made before any software is installed, I begin with “Best Practices” in
Introduction xxiii

Chapter 1. This should get you thinking in terms of what needs to take place for your moni-
toring initiative to be successful, such as how to go about implementing, who to involve, and
what pitfalls to avoid.
Chapter 2, “Theory of Operations,” builds on Chapter 1’s general design guidance by
providing a theoretical overview of Nagios from the ground up. Rather than inundating you
with configuration minutiae, Chapter 2 gives you a detailed understanding of how Nagios
works without being overly specific about configuration directives. This knowledge will go a
long way toward making configuration more transparent later.
Before we can configure Nagios to monitor our environment, we need to install it. Chap-
ter 3, “Installing Nagios,” should help you install Nagios, either from source or via a pack-
age manager.
Chapter 4, “Configuring Nagios,” is the dreaded configuration chapter. Configuring
Nagios for the first time is not something most people consider to be fun, but I hope I’ve kept
it as painless as possible by taking a bottom-up approach, only documenting the most-used
and required directives, providing up-front examples, and specifying exactly what objects
refer to what other objects and how.
Most people who try Nagios become attached to it and are loathe to use anything else.
But if there is a universal complaint, it is certainly configuration. Chapter 5, “Bootstrapping
the Configs,” takes a bit of a digression to document some of the tools available to make
configuration easier to stomach. These include automated discovery tools, as well as graphi-
cal user interfaces.
In Chapter 6, “Watching,” you are finally ready to get into the nitty-gritty of watching
systems, which includes specific examples of Nagios plugin configuration syntax and how
to solve real-world problems. I begin with a section on watching Microsoft Windows boxes,
followed by a section on UNIX, and finally the “Other Stuff” section, which encompasses
networking gear and environmental sensors.
Chapter 7, “Visualization,” covers one of my favorite topics: data visualization. Good
data visualization solves problems that cannot be solved otherwise, and I’m excited about
the options that exist now, as well as what’s on the horizon. With fantastic visualization tools
such as RRDTool and no fewer than 12 different glue layers to choose from, graphing time
series data from Nagios is getting easier every day, but this chapter doesn’t stop at mere line
graphs.
And finally, now that you know the rules, it’s time to teach you how to break them. At
the time of writing Chapter 8, “The Nagios Event Broker Interface,” it was the only docu-
mentation I’m aware of to cover the new Nagios Event Broker interface. The Event Broker
is the most powerful Nagios interface available. Mastering it rewards you with nothing less
than the ability to rewrite Chapter 2 for yourself by fundamentally changing any aspect of
how Nagios operates or extending it to meet any need you might have. I describe how the
Event Broker works and walk you through building an NEB module.
xxiv Introduction

Who Should Read This Book?

If you are a systems administrator with a closet full of UNIX systems, Windows systems, and
assorted network gadgetry, and you need a world-class monitoring system on the cheap, this
book is for you. Contrary to what you might expect, building monitoring systems is not a
trivial undertaking. Constructing the system that potentially interacts with every TCP-based
device in your environment requires a bit of knowledge on your part. But don’t let that give
you pause; systems monitoring has taught me more than anything else I’ve done in my career
and, in my experience, no matter what your level of knowledge, working with monitoring
systems has a tendency to constantly challenge your assumptions, deepen your understand-
ing, and keep you right on the edge of what you know.
To get the most out of this book, you should have a good handle on the text-based Inter-
net protocols that you use regularly, such as SMTP and HTTP. Although it interacts with
Windows servers very well, Nagios is meant to run on Linux, which makes the text Linux-
heavy, so a passing familiarity with Linux or UNIX-like systems is helpful. Although not
strictly required, you should also have some programming skills. The book has a fair number
of code listings, but I’ve tried to keep them as straightforward and easy-to-follow as possible.
With the exception of Chapter 8, which is exclusively C, the code listings are written in either
UNIX shell or Perl.
Perhaps the only strict requirement is that you approach the subject matter with a healthy
dose of open curiosity. If something seems unclear, don’t be discouraged; check out the online
documentation, ask on the lists, or even shoot me an email; I’d be glad to help if I can.
For more information, as well as full-color diagrams and code listings, visit http://www.
skeptech.org/nagiosbook.
Exploring the Variety of Random
Documents with Different Content
men seen running out of the timber were only chasing some of the
animals that had not been killed by the initial volley. Meanwhile, the
whole expeditionary force had been halted, and with loaded guns
put in readiness for the attack. Some members, unable to control
themselves, did not wait for the command, but broke ranks and ran
toward the imagined Indians with guns ready for firing. After some
little time the expedition was again restored to a state of order and
the march resumed.
Upon reaching Granger’s Point that evening, they were very
inhospitably received by a man and boy who were occupying the
cabin. Little information and absolutely no assistance could be
secured from them. They reported that they had no food, withdrew
into the cabin, and barred the door. Within a brief time, however, a
horseman arrived, who proved to be a United States regular from
Captain Bee’s command which had but lately arrived at Springfield.
He brought the information of Bee’s arrival, of the flight of the
Indians westward, and of Bee’s sending a detail to Spirit Lake to
bury the dead. He said, however, that the detail had visited only one
cabin on Spirit Lake and had there found one body which they
buried. They had made no attempt to reach the lower lakes on
account of bad weather and roads and the shortage of provisions.
That night Major Williams called a council, and upon a review of the
facts it was decided to abandon the chase. But since the bodies of
the massacred were yet unburied, it was thought that a detail of
volunteers should proceed to the lakes on that mission.[259]
 XXIV
THE BURIAL DETAIL

When morning came the conclusions of the council were reported to

the command, and volunteers, not over twenty-five in number, were
called for to serve on the burial detail. The report met with a most
cordial response and the full quota of volunteers was obtained at
once. Those who signified their willingness to serve were: Captain J.
C. Johnson and Captain Charles B. Richards, Lieutenant John N.
Maxwell, and privates Henry Carse, William E. Burkholder, William
Ford, H. E. Dalley, Orlando C. Howe, George P. Smith, Owen S.
Spencer, Carl Stebbins, Silas Van Cleave, R. U. Wheelock, R. A.
Smith, William A. De Foe, B. F. Parmenter, Jesse Addington, R.
McCormick, J. M. Thatcher, William R. Wilson, William K. Laughlin,
Elias D. Kellogg, and another whose name is not known.[260]
These men were placed by Major Williams under the immediate
command of Captain Johnson of Company C; and on the morning of
April second the detail, supplied with two days’ rations, took up its
march for the lakes. From the outset their undertaking was
precarious; with limited rations the men had no assurance that they
would be able to secure any more supplies. Nevertheless, they
courageously undertook the humanitarian task with the hope that
somehow the future would care for itself.
The burial detail was to proceed to the lakes, perform the sad task
of burying the dead, and rejoin the main command at the Irish
settlement on Medium Lake. Accompanied by two mounted men—
Captain Richards and another whose name is now lost—the detail
set out upon its journey; but at the crossing of the Des Moines, the
first stream reached, the horsemen were unable to force a passage.
The men crossed safely on a log; but the horses could not be forced
to swim the channel, and after an hour’s work Captain Richards, and
his companion gave up the effort and returned to the main
command.[261]
Without incident the members of the party reached the southeastern
shore of the east lake about two o’clock in the afternoon. Making
their way to the Noble and Thatcher cabin, they found the bodies of
Enoch Ryan and Alvin Noble at the rear of the house. Each body had
been riddled with bullets. The yard and adjacent prairie were thickly
sprinkled with feathers which had come from the destroyed feather
ticks for which the Indians had had no use. The bodies were buried
at the foot of a large oak tree near the house. While some of the
party were interring the dead at this cabin, others walked on to the
Howe cabin where seven bodies were found lying about the cabin
doorstep. Among the mangled remains found in the yard Thatcher
identified his infant child. The burials at the Howe cabin were
completed late in the afternoon; but darkness prevented the men
from proceeding to the other cabins. Returning to the Thatcher cabin
they there planned to pass the night. The body of the Thatcher child
was interred near the head of a ravine not far from the Thatcher
cabin. This was in keeping with the desire of the father that his child
should be buried upon his own property. Returning to the Howe
cabin the following morning, they found the body of a boy of about
thirteen years of age lying at the side of a fallen tree in the
dooryard. This apparently was Jacob, the brother of Mrs. Noble,
whom she vainly tried to get into the house. The burial detail
reported the interment of eight bodies at the Howe cabin.
From Howe’s cabin they proceeded to the settlements on the west
lake. At this juncture the party was divided, and one section under
Captain Johnson took the lake shore trail, while a second under
Lieutenant Maxwell crossed the lake directly in line with the Mattock
cabin. The Johnson party is said to have found the body of Joel
Howe near the trail and to have buried it near the spot where it was
found—a place which was lost sight of until its alleged discovery in
August, 1914, by a young man, Lee Goodenough of Knoxville, Iowa,
while attending a Young Men’s Christian Association camp.[262] At
the Mattock cabin the dead were found widely scattered through the
clearing and along the trail toward the Granger home across the
strait. Every evidence of a desperate resistance was noted. Dr.
Harriott was found with his broken rifle still grasped in his hand.
Eleven bodies were collected and buried at this place.
Across the strait at the Granger cabin they found the body of Carl
Granger horribly mutilated, as by cutting or slashing with some
sharp instrument about the face. Near him lay his dog which had
evidently remained faithfully by him to the last. The dog’s body was
also terribly mangled.
The Gardner home was the last place to be visited. Here six bodies
were found and buried about fifty yards to the southeast of the
cabin on a spot said to have been designated by Eliza Gardner when
she met the rescue party. As yet the bodies of Luce and Clark had
not been found; indeed they were not found until the following June
when they were discovered near the outlet of the east lake. Their
burial place is not known.[263]
By the time the work of interment was completed at the Gardner
cabin, it was late in the afternoon. The rations of the party were all
but gone; but the night was coming on, and so the party decided to
remain and camp to the north of the Gardner cabin. Fortunately
Wilson’s memory came to the rescue of the party in their stress for
food: he now recalled that in the fall when a visitor at the Gardner
cabin he had seen Gardner bury a box of potatoes beneath the stove
to insure them against being frozen during the winter. Upon
investigation there was discovered nearly a bushel of the potatoes
which satisfied the hunger of the men that evening and on the
following morning.
After this potato breakfast on the morning of April fourth, sixteen of
the twenty-three men composing the detail began the return trip;
while seven of the party having interests to look after at the lakes,
decided to remain a few days longer. Those who decided to remain
were R. A. Smith, Orlando C. Howe, R. U. Wheelock, B. F. Parmenter,
Asa Burtch, J. M. Thatcher, and William R. Wilson. Howe and
Wheelock remained to make sure of their load of supplies which
Parmenter had been compelled to abandon when his two
companions started ahead of him to Fort Dodge with the news of
the massacre.[264]
It appears, however, that the split in the party is to be attributed to
something besides business demands. There was a disagreement
over the best route to be taken on the return trip. While breakfasting
that morning the discussion had arisen. The majority favored as
direct a route as possible across the open prairie to the Irish Colony.
Others of the party did not consider such a route to be safe, arguing
that it would be better to retrace the route by which they had come
—which route would lead them to Granger’s Point and thence to the
Irish Colony. Meanwhile, a storm was gathering which seemed to
add force to the arguments of those in favor of a known road.
The matter could not be settled by argument; and so, after breakfast
Captain Johnson, gave the command to fall in. “After the men had
fallen in he gave the further order, ‘All who favor starting at once
across the prairie, step three paces to the front; the rest stand
fast’.... What little provision was left in camp was speedily packed
and the party made ready to depart at once.”[265] Captain Johnson
and Burkholder urged united action upon the seven who stood fast;
but the appeal was unavailing, for the seven men remained
steadfast in their conviction that the course as planned was wrong.
They offered to join the party if they would take the Granger route;
but Johnson and Burkholder stood as firmly against that proposition
as the seven were opposed to their plans. Thus the two groups
parted company—good friends but each firmly convinced that the
other was in the wrong. The members of the party that left took all
the food, and were allowed to do so because those who remained
behind counted upon securing their store from the wagonload of
supplies which had been left somewhere out on the prairie.
The men who remained set out at once to locate the wagon and
bring in the needed food. It appears that there was no difficulty in
finding the wagon with its cargo of supplies. When each man had
loaded himself with a supply, they returned as rapidly as possible for
the gathering storm had broken and snow was falling heavily. In a
short time, it became a blinding, driving whirlwind of snow. Reaching
the cabin, they laid in a supply of fuel. Being well armed, they felt no
alarm at the prospect of an Indian attack. All that could be done
while the storm raged was to await patiently its abatement. Only
after two days did the fury of the storm abate sufficiently to permit
the men to leave the cabin in safety.
The morning of the second day after the beginning of the blizzard
dawned clear and intensely cold, although the weather had
moderated somewhat since the previous evening. The snow was
frozen with a hard crust and upon it the party from the Gardner
cabin made their way rapidly in the direction of Granger’s Point.
When they arrived at the Des Moines they found the river completely
frozen, which made the crossing easy. Thus with little trouble they
were again at Granger’s Point where they had left the main body five
days previously. They now procured a team and wagon, loaded their
baggage, and, after resting a day, started for the Irish settlement. At
this point they found some of the wounded from the Springfield
settlement who had not been able to proceed with the main
command. Here also was Henry Carse who, as will be seen, suffered
so terribly on the night out from the Gardner cabin. Resting a day at
the Irish settlement, they resumed their journey to Fort Dodge.
What had been a small party on leaving the Gardner cabin had more
than doubled in number when the Irish colonists were bidden
goodbye.
When Cylinder Creek was reached the party succeeded through
great effort in effecting a crossing. The undertaking required the
whole of an afternoon, but by nightfall the men succeeded in
reaching Shippey’s Point two miles beyond. “From here the party
proceeded on their way to Fort Dodge, which they reached without
further adventures than such as are incident to swimming swollen
streams and living on short rations, which, in some instances,
consisted of a handful of flour and a little salt, which they mixed up
with water and baked over a campfire. A few of the party shot,
dressed and broiled some muskrats and tried to make the rest
believe they considered them good eating, but that diet did not
become popular.”[266]

The early part of the day upon which Captain Johnson and party left
the Gardner cabin, after the disagreement of the morning, was quite
warm, and the rapidly melting snow added greatly to the difficulties
of traveling. Being forced to wade through sloughs several feet deep
in slush the men were soon wet to the shoulders. But they plodded
on cheerfully for they were on the way home after the completion of
an arduous duty. While they were in this cheery frame of mind, the
blizzard broke upon them in all its fury about four in the afternoon.
With the storm came a rapid fall in temperature, and it was not long
before the clothes of the members of the party were frozen stiff
from feet to shoulders—rendering progress next to impossible.
With the oncoming of the storm began the first disagreement among
the men after leaving the Gardner cabin in the morning. Again, it
was a matter of the best route to be taken. Jonas Murray, a trapper
who had volunteered as guide, claimed to be thoroughly familiar
with the country. Not all, however, were willing to accept his
guidance. Spencer and McCormick were the first to break away from
his leadership. This they did when Mud Creek was reached only
about eight or nine miles from the point of starting. Crossing far to
the north of where Murray maintained was the proper place, these
men struck directly east for the settlement which they reached
within a short time after the storm broke upon them.[267]
The other members of the party lost much valuable time in
wandering southward along the course of Mud Creek. Finally a
crossing was effected, but much farther to the south than several
thought it should have been. Against the protests of a number,
Murray continued to lead the party still farther south. Near sunset
Maxwell and Laughlin found a township corner pit, at which they
proposed to camp for the night since they feared the loss of
direction in the oncoming darkness. But Murray, Johnson, and
Burkholder, thought it best to continue and so the party pressed on.
[268] Ahead of them was a lake to the east of which was a great
stretch of uncommonly high grass which seemed to afford good
shelter. Maxwell, Laughlin, and seven others started to walk around
this lake to the east; but Johnson, Burkholder, Addington, G. P.
Smith, and Murray went around in the opposite direction. Finding a
shelter Laughlin called to Johnson’s party which could then only be
dimly seen through the sedge. Apparently he was not heard, for the
men struck out toward the southeast and were not again seen
before the Irish settlement was reached. Laughlin’s party decided to
remain where it was rather than attempt to follow.
As soon as the halt was made the men tumbled down in a shivering
heap and huddled closely together to keep from freezing. In crossing
sloughs several men had removed their boots to keep them dry,
while others had cut holes in the leather in order to let the water
out. Carse had removed his boots, but found it impossible to replace
them for they were frozen stiff. He then tore his blanket into pieces
and wrapped his feet as well as he could, but even then he suffered
fearfully from the cold. Maxwell and Laughlin, realizing the danger of
freezing to death, did not permit themselves to sleep the whole
night through: they kept constantly on the move and compelled the
others to do the same. Whenever any man fell asleep the others
would pick him up, arouse him, and force him to remain awake and
on the move regardless of his objections. Some of the men begged
that they be allowed to sleep, protesting that moving about in their
ice stiffened garments was worse punishment than they could bear.
Thus all night long the awful vigil was kept. It was largely due to the
tireless watching of Maxwell and Laughlin that no one froze to death,
although the temperature that night was said to have been thirty-
four degrees below zero at points in Iowa much farther south.[269]
The next day opened clear and cold. About eight miles to the east
was seen a grove of timber. Every man expressed himself as willing
and able to travel; and so without breakfast (for they had no food)
the party started in that direction, believing that the timber bordered
the Des Moines. Maxwell was the last to leave camp, and when
about three miles from the timber he found Carse sitting on the
sunny side of a small mound trying to pull on his frozen boots. The
blanket wrappings of his feet had already become so worn in
traveling over the ice and snow that he could go no further. Maxwell
endeavored to take Carse along with him, but every time he tried to
guide him toward the timber Carse obstinately insisted on taking the
opposite direction. It soon became evident that the man had grown
delirious and that nothing could be done with him on the open
prairie. Henry E. Dalley, seeing the difficulty, came to Maxwell’s
assistance. The two were able to get Carse to the timber, by which
time he was unconscious and blood was streaming from his mouth.
[270]

Laughlin and Kellogg, who had reached the timber first, had set
about the building of a fire when it was discovered that not a
member of the party had matches. Laughlin’s ingenuity, however,
came to the rescue. He had a gun and powder, and was wearing a
vest with a heavy, quilted cotton lining. Removing some of the
cotton from his vest he loaded the gun with a powder charge and
rammed it down tight with cotton. He then discharged the gun into a
piece of rotten wood which, after some attention, began blazing.
Dalley soon arrived with the helpless Carse. When the blanket
wrappings were removed from Carse’s feet the skin of the soles
came with them. Dalley finally succeeded in stopping the bleeding
and in reviving him. It was only a few nights before that Carse had
befriended Dalley by taking him under his own blanket. The boy—for
such he was, being less than twenty years of age—was poorly clad
and had suffered much from the trials of the expedition. His youthful
strength and courage, however, carried him safely through to the
end. Meanwhile, Kellogg had seated himself at the base of a tree
and before anyone had observed his need for attention he too had
become unconscious from exposure. Before he could be revived it
was necessary to cut his icy clothing away from his body as the only
practicable means of removing it. When this had been done he
gradually regained consciousness and seemed but little the worse for
his experience.[271]
Laughlin and Maxwell, having attended those who were needing
help and noting that all were as comfortable as conditions would
permit, started out to cross the river with a view to locating the Irish
settlement. They found the river frozen thick enough to support
them, with the exception of a few spots over which they improvised
a bridge of poles. Making their way to the margin of the timber, they
saw the settlement in plain sight not over three miles away. Help
was at once secured which enabled them to get the disabled
members of the party across the river and to safety in the homes of
the settlement. Here they found Major Williams awaiting their
coming.
Without delay Major Williams sent men down the Des Moines to look
for Johnson and his companions. They remained out during the
whole of the day; and when they returned near dark reported that
they had discovered no trace of the men, but had found a cabin in
which a good fire was burning. The Major concluded that the men
had been at the cabin and had then gone southward, following the
course of the river. Three of the five men in the party—Smith,
Addington, and Murray—came to the settlement the following
morning but could give little information concerning Johnson and
Burkholder. Smith had been the last to see them; and his story left
no doubt in the minds of most of his hearers that the two men had
perished somewhere to the west of the Des Moines River.
The two unfortunate men having become completely exhausted by
wading streams and sloughs had finally sat down declaring that they
were unable to go any farther. They were sheeted with ice from
head to feet. Their feet were badly frozen and, unable to walk, they
insisted, against Smith’s advice, upon removing their boots. Realizing
that they could not replace the boots they cut their blankets in strips
with which to wrap their feet. At this time they were in sight of the
timber along the Des Moines River, which they were urged to exert
every effort to reach. But they were unable to rise from the ground.
“After vainly trying for a long time to get them to make another
effort to reach the timber, Smith at last realized that to save his own
life he must leave them. After going some distance he looked back
and saw them still on their knees in the snow, apparently unable to
rise. It is not likely they ever left the spot where Smith left them,
but, overcome with cold, they finally sank down and perished side by
side.”[272] Nearly eleven years later two skeletons were found near
the place where Smith said he left his companions. By the guns and
powder flasks lying near them the skeletons were identified as being
those of Johnson and Burkholder.[273]
 XXV
RETURN OF THE RELIEF EXPEDITION

From Granger’s Point the return of the main body of the command
was uneventful until the Irish settlement was reached and passed. It
will be recalled that when the burial detail was outfitted nearly all of
the scanty rations then remaining were turned over to them because
of the probable hardships which would be encountered in venturing
into the hostile lake region. Thus the main command was hard
pressed in the matter of providing itself with adequate supplies. By
the end of the first day the command had reached the cabin of an
old trapper near the shore of Mud Lake. The experiences of the first
night out are illustrative of the extremity to which members of the
expedition were driven upon their homeward journey.
At the trapper’s cabin were found the frozen carcasses of some
beaver, which it was thought could be utilized as food. But frozen
beaver even when roasted failed to satisfy the hunger of the men.
Captain Richards tells of one member of his company, George W.
Brizee, who, as a result of exposure was suffering from a severe
case of toothache and very sore feet. Finally, the pain in his feet
grew easier. But “his tooth reminded him that it needed his
attention; and after lying down and trying to sleep, frequently
reiterating that he knew he should die, he got up and went out and
returned with a hind-quarter of beaver and began to roast it over
the coals; and in a half-reclining position he spent the entire night
roasting and trying to eat the tough, leathery meat, first consigning
his feet to a warmer climate, and then as his toothache for a time
attracted most of his attention, giving us a lecture on dentistry;
when his tooth was relieved for a short time he would, with both
hands holding on to the partially roasted quarter of beaver, get hold
with his teeth and try to tear off a piece! The picture by the weird
light of the fire was a striking one”.[274]
The party did not tarry long at the Irish settlement, which was
reached on the evening of the next day, since it was evident that the
settlers had barely sufficient food to keep themselves alive and
would surely suffer if the command remained for any length of time.
[275] The day of leaving Medium Lake was a cloudy one and rather
warm—just such a day as is sure to start the water running from
rapidly melting snow. Only a short distance had been traveled when
rain began falling—first as a drizzle, but by the time Cylinder Creek
was reached it was a downpour. The prairies were flooded, while
Cylinder Creek was about half a mile wide, completely covering its
rather narrow bottom, which was under from two to five feet of
water, while the main channel had a depth of fifteen to twenty feet
and was from sixty to eighty feet wide. Obviously the problem of
crossing would be a serious one. Arriving at the border of the valley
about two o’clock in the afternoon the command vainly sought a
passage. Then suddenly the wind veered sharply to the northwest
and became a gale—the rain changing into a blinding fall of snow.
This was the fearful blizzard of April fourth that overtook the
Johnson party on its return from the Gardner cabin.
Captains Richards and Duncombe, not despairing of being able to
effect a crossing of the main channel, undertook to improvise a boat
out of a nearly new wagon box. With very little effort this wagon box
was caulked water tight with bedquilt cotton. Solon Mason and
Guernsey Smith were the men chosen to assist in getting the boat
across the channel. But the wind blew so hard that, although
Richards and Duncombe bailed water as rapidly as they could, the
party scarcely reached the opposite side of the channel before the
make-believe boat sank—the men barely saving themselves from
drowning. Thus the attempt to take all across in that manner failed.
[276] Having no blankets and unable to assist their comrades on the
opposite side, there was nothing to do but hasten on to Shippey’s
Point which was two or three miles distant.[277] This point they
reached about nine o’clock at night. Here they were liberally fed, and
by sitting around the fire all night were able to dry their clothes by
exposing first one side and then the other to the fire.
When morning came the storm had abated somewhat, and so it was
decided to return to the creek in an effort to locate the command.
Mason had not gone far when he succumbed to the cold and had to
be taken back. It seems that in crossing the Cylinder he had lost
both overcoat and cap. Upon their arrival at the east side of the
bottom the men could see nothing on the other side to indicate the
presence of their comrades. After spending some time in trying to
accomplish a crossing, they gave up the attempt and returned to
Shippey’s. There they remained until about the middle of the
afternoon when they again returned to the creek. This time they
were no more successful than before. Resigned to the thought that
the remainder of the command had either perished or returned to
Medium Lake, they wandered back to Shippey’s. Shortly after their
return, Hoover and Howland came in and reported that when they
left the command all were safe on the west side, though suffering
considerably while waiting for the channel to freeze.
Early on Monday morning, while the blizzard was yet raging and the
cold was still intense, the little group at Shippey’s once more started
for the creek in an effort to locate their companions. Reaching the
creek, the little group saw the men on the opposite side making
preparations to cross—the storm by this time having abated so that
a crossing might be attempted. The creek was now solidly frozen so
that the task of crossing was easy. The way to Shippey’s was soon
made. Here they told the story of how they had saved themselves
from the terrors of the awful storm.
From this story it appears that no thought of returning to the Irish
settlement had been entertained by those who had been left behind.
Major Williams and two or three others had, indeed, returned,
hoping that they would there find the burial detail and guide them to
the Cylinder Creek camp. Those at the latter place resolved to
remain and await the dying down of the storm before making any
further attempt at crossing; and they set to work to improvise a
shelter. Again the tarpaulin wagon covers were brought into use and
supplemented with blankets, which when fastened together were
stretched around and over the wagon frames and then staked down
to the frozen ground. This improvised shelter was completely closed
excepting a small flap opening on the south or lea side which served
the purpose of a door. Then with blankets and other covers a
common bed was made; and into this the party crowded, wet from
head to feet. Here they remained from Saturday night until Monday
morning when a few ventured out to examine the state of the
weather.[278] Finding conditions satisfactory they began the crossing
after having tarried “over forty hours, without food or fire, on the
open prairie, with the mercury at 32° below zero.”[279]
It is little wonder that when they started to make the crossing the
men had scarcely “strength enough to reach the opposite shore....
Every man’s mouth was open wide, his tongue hanging out, and in
some instances blood running from nose or mouth.”[280] Governor
Carpenter, in commenting upon this terrific test of endurance notes
that “since that experience upon Cylinder Creek, I have marched
with armies engaged in actual war. During three and a half years’
service, the army with which I was connected, marched from Cairo
to Chattanooga, from Chattanooga to Atlanta, from Atlanta to the
Sea, and from the Sea through the Carolinas to Richmond.... But I
never in those weary years experienced a conflict with the elements
that could be compared with the two nights and one day on Cylinder
Creek.”[281]
After refreshing themselves at Shippey’s the men held a consultation
and reached the decision that henceforth the command should break
up into small details—a plan that seemed necessary on account of
the increasing difficulty of securing food. Each group was to find its
way home in the best manner it might be able to devise. Every man
was ordered to rid himself of all surplus baggage, retaining only his
blanket. Thus the expedition really came to an end with the crossing
of Cylinder Creek. But the hardships of the men were not ended;
before a number of the squads reached home they endured trials
almost as severe as those encountered before crossing the Cylinder.
The experience of the little group which Frank R. Mason undertook
to guide is perhaps typical of the hardships of the journey south
from Shippey’s. Mason had frequently been north of Fort Dodge
hunting in the timber along Lott’s Creek, and for that reason he was
selected by a Webster City group to pilot them home. With his party
he struck out boldly across the prairie in a line which he thought
would lead to a clearing in the timber where he knew they would
receive a hearty welcome. As darkness came on the men began to
show exhaustion; but the looked-for timber along Lott’s Creek did
not appear. One of the men, Hathaway by name, became wholly
exhausted and had to be carried. Within a short time he became
delirious; and then the united efforts of three of the party were
needed to keep him under control, with only indifferent success.
Finally passing into a stupor he was more easily managed.
When Mason and his companions reached the timber at about
eleven o’clock the expected cabin could not be found. The men grew
impatient and at times were inclined to criticize Mason as an
incompetent guide. Having reached a slight elevation or ridge, and
despairing of locating the cabin, they prepared to spend the night.
Snow was cleared away until the bare ground was reached and upon
this they threw themselves. They had had no food since the start;
indeed they had not brought any with them, for they had expected
to reach the cabin before nightfall. When they had lain sleepless for
nearly an hour, voices were heard and out of the darkness appeared
human forms.[282] The newcomers were Mr. and Mrs. Elwood Collins
who were returning from an evening spent at a neighbor’s home.
The finding of the men is thus described by Mrs. Collins. “Husband
and I, after having stayed later than usual at a neighbor’s, started
for home.... All at once the outline of dark objects appeared before
us.... I at first thought we might be upon a company of Indians! We
were too near to retreat.... I then heard groans of distress, and I
thought sobs.... We had a lantern, and as the light shone upon the
place my pity was truly stirred. There, with the snow crushed
beneath them, were eight men; some sitting, some reclining, and
others lying flat upon their backs!”[283]
Having been piloted to the clearing the men slept that night in the
cabin loft. In the morning they breakfasted hastily and resumed their
journey to Webster City. Hathaway and Gates had to be left at the
cabin as they were not able to proceed. This day’s experience was
but a repetition of the previous one. As darkness fell the men were
again exhausted, but by crawling on hands and knees they managed
to reach the cabin of a Mr. Corsau where they were taken in for the
night. On the following day they were taken by Corsau to Webster
City. Thus ended, for this Webster City group, the fearful experience
of attempting to relieve the settlers of the lake region from Indian
attacks.
For the Fort Dodge men the task of making their way home was
easier, as it did not necessitate the crossing of as many streams—
which at this time were in flood condition. At the same time their trip
was not lacking in incidents of trial. They arranged the march from
cabin to cabin so that they might have no difficulty in procuring
food, for they, too, made no attempt to carry supplies. More than
once the men experienced trials similar to those encountered by the
Mason party, and like them they too found the place searched for
before hope was gone. Within three or four days after leaving
Cylinder Creek, all parties had straggled in—weary, worn, and
wasted. They were met with a hearty welcome from friends who had
thought them in all probability lost on the northwestern prairies. All
who had volunteered in the expedition returned home in safety,
except Johnson and Burkholder who perished in the snow.
 XXVI
THE DEATH OF MRS. THATCHER

From March twenty-sixth to April tenth, while the relief expedition

from Fort Dodge and Webster City was making its way painfully to
and from the scene of the massacre at the lakes, Inkpaduta and his
band continued their flight. When Lieutenant Murry’s men had been
sighted by the look-out, warning of their approach was
communicated through the Indian camp. The warriors crouched
among the willows along the creek ready to spring out upon their
pursuers, while the squaws and children made hurried preparations
for a hasty retreat if need be. Meanwhile, a warrior stood guard over
the helpless white captives with orders to shoot them the moment
the soldiers should attack. But Coursalle and La Framboise, who
were guiding Murry’s men, declared that the signs were so old that
pursuit would be hopeless; and so the soldiers returned to the main
command. No sooner had they started on their return than
Inkpaduta fled from his temporary camp and began the long journey
to the Big Sioux, the James, and the region beyond.
The Indians were now thoroughly alarmed at the nearness of
danger, and for two days and nights they kept up a continuous
flight. No stops were made to prepare food: if they ate at all it was
while they were on the move. Such a sustained flight would have
been arduous enough for untrained marchers under the most
favorable conditions, but for the women captives it was terrible. Not
only were they compelled to wade through snow and slush but they
were burdened with loads which might well have been regarded as
too heavy for men to bear.
Mrs. Marble states that upon leaving Heron Lake she and her
associates “were forced to carry heavy packs, and perform the
degrading and menial services in the camp ... that the pack ...
consisted of two bags of shot, each weighing twenty-five pounds,
and a lot of camp furniture, increasing the weight of the pack to 100
pounds. On top of this heavy load ... was placed the additional
weight of an Indian urchin of some three or four years of age.”[284]
The papoose which she was supposed to carry seemed to consider
that it was entitled to as many liberties and as much attention when
carried by her as it would have enjoyed if in the care of its mother.
Mrs. Marble objected to making friends with the baby, and watching
her opportunity would scratch it in the face until the Indians, hearing
its cries, finally concluded it didn’t like her and took it away.
Abbie Gardner, though but a girl, was also burdened with a pack—
though its weight was somewhat less than that carried by Mrs.
Marble. It was made up of “eight bars of lead, one pint of leadballs,
one tepee cover made of the heaviest, thickest cloth, one blanket,
one bed-comforter, one iron bar, three feet long and half an inch
thick ... one gun, and one piece of wood several inches wide and
four feet long, to keep the pack in shape.”[285]
This burdening of the captives was the more objectionable to them
since the Indian men were encumbered with nothing but a gun. As a
matter of course the squaws carried packs, but they were
accustomed to such burden-bearing and knew how to save
themselves from its ill effects. Moreover, the squaws were frequently
equipped with a sort of crude snowshoe which greatly aided them in
walking. The white captives sank deep into the snow at every step.
They dared not stop to rest, for whenever they slackened their pace
the Indians would level guns at them and resort to various other
devices to keep them moving.
The food which the Indians had secured at Okoboji and Springfield
supplied them for about four weeks. Following this they made little
or no effort to secure food by hunting. If game crossed their path
they would kill it—if they could do so without much effort. But there
was no organization of hunting parties. After the confiscated
supplies were exhausted, they contented themselves with muskrat
and skunk; and as a luxury, Mrs. Sharp relates, they indulged in dog.
As spring opened they were able to secure a few ducks and geese,
which seemed very plentiful, but of which the Indians obtained only
a few. Such delicacies, however, were never shared with the
captives: they were not even allowed to assist in their preparation.
The treatment of the horses secured at Okoboji and Springfield was
still worse. There was neither hay nor grass—little or nothing upon
which the horses might feed. Even so they were given but slight
opportunity to feed. Before the Big Sioux had been reached nearly
all of the horses taken in the raids at the lakes had died of
starvation.[286]
Continued pursuit and ultimate capture by the soldiers seem to have
soon lost their terrors for the Indians. Although they kept constantly
on the move, progress was not very rapid—largely owing to the
huge drifts of snow over and through which they were compelled to
travel. Their first stopping-place, after nearly two weeks of
uninterrupted marching, was at the great red pipestone quarry in
southwestern Minnesota. This was but little more than one hundred
miles northwest of Heron Lake. Here they remained for a day
quarrying pipestone and fashioning pipes. A further cause for delay
was the fact that the snow was rapidly melting and travel, even for
the Indians, was very difficult.
The Indians were now in a sacred region to which all the Sioux were
wont to make frequent journeys—a region closely associated with
the superstitions of their race. Here the footprints made by the Great
Spirit when he alighted upon the earth could be seen. It was while
he stood here that a stream of water burst forth from beneath his
feet and flowed away to nourish the plain. Here it was that the Great
Spirit fashioned a pipe and smoked: huge volumes of smoke issued
forth serving as a signal for all the tribes to assemble from far and
near. When so assembled, the Great Spirit, blowing the smoke over
all, bade them meet here always in peace even though they might
be at war elsewhere. Moreover, if they wished to receive his favor,
the calumet must be fashioned from the rock upon which he stood.
Having thus enjoined his people, the Great Spirit disappeared in a
cloud. It is said that ever afterward when the Indians met at the
pipestone quarry, they met in peace though elsewhere they might be
at war.[287]
After leaving the pipestone region so much time was consumed by
the Indians in camping that it might be said they camped more than
they marched. This is explained by the fact that they felt themselves
now wholly free from the danger of pursuit. Spring was rapidly
approaching and the smaller game was becoming more plentiful;
and so they did not feel the need of hastening to the buffalo ranges
in Dakota.
The burdens of the captives grew increasingly more difficult.
Although snow no longer impeded their march, the rains were
frequent and the rivers and creeks were flowing wide over the
valleys. When it rained they were without shelter. The streams were
crossed by the Indians on the backs of the few ponies that yet
survived. But the captives had to wade at the risk of losing their
lives: they could not swim.
Notwithstanding the hardships through which they were compelled
to pass, all but Mrs. Thatcher were faring much better than might
have been expected. Mrs. Marble, Mrs. Noble, and Abbie Gardner
were willing to appear resigned to their lot and did all that was
requested of them: they even appeared ready and willing to perform
the many menial duties which fell to their lot. With Mrs. Thatcher,
however, it was different. She had from the first rebelled at the
service imposed by her Indian captors; nor did she hesitate to show
them very plainly her frame of mind. This attitude on her part
proved to be most unfortunate.
From the beginning of her captivity Mrs. Thatcher had been ill with
phlebitis, which before the end of two weeks had developed into
virulent blood poisoning.[288] Indeed, so serious was her condition
that for a large portion of the march she had been relieved of much
of her pack. At the pipestone quarry and on the march after leaving
that region the medicine man of the band had undertaken to treat
her—and the treatment seemed to help her. To such an extent had
she been relieved that the Indians considered her again able to bear
a pack. Thus it happened that when they arrived at the crossing of
the Big Sioux near the present village of Flandrau, Mrs. Thatcher
was laden as heavily as were the other three captives.
This crossing had been for generations the fording place of the red
peoples in their pilgrimages to the pipestone quarry. Normally the
river at this point is wide but shallow. But “the vast amount of snow
which covered the ground that memorable winter had nearly gone,
by reason of the rapid thawing during the last few weeks, causing
the river to rise beyond all ordinary bounds, and assume majestic
proportions.”[289] Throughout the greater portion of the upper
course of the Big Sioux it flows between perpendicular and
continuous cliffs of red jasper rocks peculiar to the region, but at or
near this traditional crossing place the stone cliffs were neither high
nor continuous. Moreover, at this particular time so many tree trunks
had become lodged by the spring freshets that at one point a bridge
crossing was formed. Upon this the Indians proposed to cross,
instead of attempting the more dangerous method of fording. At the
prospect of crossing the swollen stream, the captives were terrified,
believing that they would again be compelled to wade. They
despaired of being able to get across. The situation seemed quite
hopeless.
As soon as the determination to cross had been reached, an Indian
warrior—the one who had seized the box of caps from Gardner—
removed the pack from Mrs. Thatcher’s back and transferred it to his
own.[290] This in itself was ominous, and Mrs. Thatcher was not slow
to perceive that some unusual disposition was to be made of her. As
she was ordered forward to the driftwood bridge she spoke to her
companions, bidding them goodbye and saying as she did so: “If any
of you escape, tell my dear husband that I wanted to live for his
sake.”[291] When she had made the middle of the stream, the Indian
carrying her pack suddenly tripped her into the river. Retaining her
presence of mind she was able by desperate efforts to keep herself
afloat. A number of times she succeeded in making her way to the
banks of the stream where, grasping the roots of trees, she strove to
pull herself out of the water. But each time she was met by an
Indian who clubbed her loose and with a long pole pushed her into
the main current. Finally, as she came to shore and grasped the
roots of a tree for what proved to be the last time, an Indian who
had always been peculiarly brutal in his treatment of the captive
raised his gun and shot her through the head, killing her instantly.
[292]

Mrs. Marble relates that the death of Mrs. Thatcher “was hailed by
the Indian women with loud shouts of joy and exultation.—The
feelings of the surviving prisoners at this horrid murder, cannot be
imagined. They beheld in Mrs. Thatcher’s death, the fate reserved
for them, when overpowered by fatigue, they would be unable to
proceed.”[293]
The death of Mrs. Thatcher was a sad blow to the remaining
captives: it was particularly distressing to Mrs. Noble. These two
women had been lifelong friends and had married cousins. The
families had come to the frontier together, had lived in the same
cabin, and had planned to build homes as nearly together as
possible. Mrs. Noble was so depressed and so bereft of any hope
that in the evening she proposed to the other captives that they
steal away to the Big Sioux and drown themselves. Mrs Marble,
however, succeeded in convincing her that such an act would be
useless. But from this time Mrs. Noble seemed to be wholly
indifferent as to her treatment or possible fate at the hands of her
captors. The captives were now made to realize as never before the
heartlessness of their captors: they lived in the expectation that any
day might see for them the end of life.
Welcome to our website – the ideal destination for book lovers and
knowledge seekers. With a mission to inspire endlessly, we offer a
vast collection of books, ranging from classic literary works to
specialized publications, self-development books, and children's
literature. Each book is a new journey of discovery, expanding
knowledge and enriching the soul of the reade

Our website is not just a platform for buying books, but a bridge
connecting readers to the timeless values of culture and wisdom. With
an elegant, user-friendly interface and an intelligent search system,
we are committed to providing a quick and convenient shopping
experience. Additionally, our special promotions and home delivery
services ensure that you save time and fully enjoy the joy of reading.

Let us accompany you on the journey of exploring knowledge and

personal growth!

ebookname.com