Download the full version of the ebook at

https://ebookultra.com

Automated Machine Learning in Action 1st

Edition Qingquan Song

https://ebookultra.com/download/automated-machine-
learning-in-action-1st-edition-qingquan-song/

Explore and download more ebook at https://ebookultra.com

 Recommended digital products (PDF, EPUB, MOBI) that
you can download immediately if you are interested.

A First Course in Machine Learning Chapman Hall CRC

Machine Learning Pattern Recognition 2nd Edition Simon
Rogers
https://ebookultra.com/download/a-first-course-in-machine-learning-
chapman-hall-crc-machine-learning-pattern-recognition-2nd-edition-
simon-rogers/
ebookultra.com

A First Course in Machine Learning 1st Edition Simon

Rogers

https://ebookultra.com/download/a-first-course-in-machine-
learning-1st-edition-simon-rogers/

ebookultra.com

Python Machine Learning By Example The Easiest Way to Get

into Machine Learning 1st Edition Yuxi (Hayden) Liu

https://ebookultra.com/download/python-machine-learning-by-example-
the-easiest-way-to-get-into-machine-learning-1st-edition-yuxi-hayden-
liu/
ebookultra.com

How Assessment Supports Learning Learning Oriented

Assessment in Action 1st Edition David Carless

https://ebookultra.com/download/how-assessment-supports-learning-
learning-oriented-assessment-in-action-1st-edition-david-carless/

ebookultra.com
Action Learning in Practice 4th Edition Mike Pedler

https://ebookultra.com/download/action-learning-in-practice-4th-
edition-mike-pedler/

ebookultra.com

Python Machine Learning 1st Edition Wei-Meng Lee

https://ebookultra.com/download/python-machine-learning-1st-edition-
wei-meng-lee/

ebookultra.com

Statistical Machine Learning 1st Edition Richard M. Golden

https://ebookultra.com/download/statistical-machine-learning-1st-
edition-richard-m-golden/

ebookultra.com

Bayesian reasoning and machine learning Barber D.

https://ebookultra.com/download/bayesian-reasoning-and-machine-
learning-barber-d/

ebookultra.com

Informed systems organizational design for learning in

action 1st Edition Somerville

https://ebookultra.com/download/informed-systems-organizational-
design-for-learning-in-action-1st-edition-somerville/

ebookultra.com
Automated Machine Learning in Action 1st Edition
Qingquan Song Digital Instant Download
Author(s): Qingquan Song, Haifeng Jin, Xia Hu
ISBN(s): 9781617298059, 1617298050
Edition: 1
File Details: PDF, 23.95 MB
Year: 2022
Language: english
Qingquan Song
Haifeng Jin
Xia Hu

MANNING
 The life cycle of a machine learning (ML) model

Hyperparameter tuning

Problem framing and Data preprocessing and ML algorithm Model training Service deployment and
dataset collection feature engineering selection and evaluation model monitoring

Continuous maintenance and feedback integration

The hyperparameter-tuning workflow

Manually adjust
hyperparameter. T esting
Training

Training ML Learned Final

dataset algorithm model model

Validation T est
dataset dataset
Tuning
Automated Machine Learning in Action
Automated Machine
Learning in Action
QINGQUAN SONG
HAIFENG JIN
XIA HU

MANNING
SHELTER ISLAND
For online information and ordering of this and other Manning books, please visit
www.manning.com. The publisher offers discounts on this book when ordered in quantity.
For more information, please contact
Special Sales Department
Manning Publications Co.
20 Baldwin Road
PO Box 761
Shelter Island, NY 11964
Email: orders@manning.com

©2022 by Manning Publications Co. All rights reserved.

No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in

any form or by means electronic, mechanical, photocopying, or otherwise, without prior written
permission of the publisher.

Many of the designations used by manufacturers and sellers to distinguish their products are
claimed as trademarks. Where those designations appear in the book, and Manning Publications
was aware of a trademark claim, the designations have been printed in initial caps or all caps.

Recognizing the importance of preserving what has been written, it is Manning’s policy to have
the books we publish printed on acid-free paper, and we exert our best efforts to that end.
Recognizing also our responsibility to conserve the resources of our planet, Manning books
are printed on paper that is at least 15 percent recycled and processed without the use of
elemental chlorine.

The author and publisher have made every effort to ensure that the information in this book
was correct at press time. The author and publisher do not assume and hereby disclaim any
liability to any party for any loss, damage, or disruption caused by errors or omissions, whether
such errors or omissions result from negligence, accident, or any other cause, or from any usage
of the information herein.

Manning Publications Co. Development editor: Toni Arritola

20 Baldwin Road Technical development editor: Kostas Passadis
PO Box 761 Review editor: Aleksandar Dragosavljević
Shelter Island, NY 11964 Production editor: Andy Marinkovich
Copy editor: Pamela Hunt
Proofreader: Keri Hales
Technical proofreaders: Karsten Strøbaek, Ninoslav Čerkez
Typesetter: Dennis Dalinnik
Cover designer: Marija Tudor

ISBN: 9781617298059
Printed in the United States of America
 brief contents
PART 1 FUNDAMENTALS OF AUTOML ........................................1
1 ■ From machine learning to automated
machine learning 3
2 ■ The end-to-end pipeline of an ML project 17
3 ■ Deep learning in a nutshell 41

PART 2 AUTOML IN PRACTICE.................................................71

4 ■ Automated generation of end-to-end ML solutions 73
5 ■ Customizing the search space by creating
AutoML pipelines 99
6 ■ AutoML with a fully customized search space 138

PART 3 ADVANCED TOPICS IN AUTOML . ................................185

7 ■ Customizing the search method of AutoML 187
8 ■ Scaling up AutoML 223
9 ■ Wrapping up 249

v
 contents
preface xiii
acknowledgments xv
about this book xvii
about the authors xx
about the cover illustration xxii

PART 1 FUNDAMENTALS OF AUTOML ..............................1

1 From machine learning to automated machine learning 3

1.1 A glimpse of automated machine learning 4
1.2 Getting started with machine learning 6
What is machine learning? 6 The machine learning process 7
■

Hyperparameter tuning 9 The obstacles to applying machine

■

learning 11
1.3 AutoML: The automation of automation 12
Three key components of AutoML 12 ■
Are we able to achieve full
automation? 15

2 The end-to-end pipeline of an ML project

2.1 An overview of the end-to-end pipeline 18
17

2.2 Framing the problem and assembling the dataset 19

vii
viii CONTENTS

2.3 Data preprocessing 22

2.4 Feature engineering 25
2.5 ML algorithm selection 28
Building the linear regression model 29 ■
Building the decision
tree model 31
2.6 Fine-tuning the ML model: Introduction to grid
search 34

3 Deep learning in a nutshell

3.1 What is deep learning? 42
41

3.2 TensorFlow and Keras 43

3.3 California housing price prediction with a multilayer
perceptron 43
Assembling and preparing the data 44 Building up the
■

multilayer perceptron 45 Training and testing the neural

■

network 49 Tuning the number of epochs 52

■

3.4 Classifying handwritten digits with convolutional neural

networks 55
Assembling and preparing the dataset 55 Addressing the problem
■

with an MLP 57 Addressing the problem with a CNN 59

■

3.5 IMDB review classification with recurrent neural

networks 64
Preparing the data 65 Building up the RNN
■
67
Training and validating the RNN 69

PART 2 AUTOML IN PRACTICE .......................................71

4 Automated generation of end-to-end ML solutions

4.1 Preparing the AutoML toolkit: AutoKeras 73
73

4.2 Automated image classification 76

Attacking the problem with five lines of code 76 Dealing with
■

different data formats 80 Configuring the tuning process 81

■

4.3 End-to-end AutoML solutions for four supervised learning

problems 83
Text classification with the 20 newsgroups dataset 83 Structured ■

data classification with the Titanic dataset 85 Structured data

■

regression with the California housing dataset 88 Multilabel ■

image classification 89
 CONTENTS ix

4.4 Addressing tasks with multiple inputs or outputs 91

Automated image classification with the AutoKeras IO API 91
Automated multi-input learning 93 Automated multi-output
■

learning 94

5 Customizing the search space by creating AutoML pipelines

5.1 Working with sequential AutoML pipelines 100
99

5.2 Creating a sequential AutoML pipeline for automated

hyperparameter tuning 102
Tuning MLPs for structured data regression 103 ■ Tuning CNNs
for image classification 109
5.3 Automated pipeline search with hyperblocks 111
Automated model selection for image classification 112
Automated selection of image preprocessing methods 117
5.4 Designing a graph-structured AutoML pipeline 121
5.5 Designing custom AutoML blocks 125
Tuning MLPs with a custom MLP block 125 ■
Designing a
hyperblock for model selection 132

6 AutoML with a fully customized search space

6.1 Customizing the search space in a layerwise fashion
138
139
Tuning an MLP for regression with KerasTuner 139 ■
Tuning
an autoencoder model for unsupervised learning 147
6.2 Tuning the autoencoder model 151
6.3 Tuning shallow models with different search
methods 154
Selecting and tuning shallow models 154 Tuning a shallow
■

model pipeline 157 Trying out different search methods 158

■

Automated feature engineering 159

6.4 Controlling the AutoML process by customizing
tuners 169
Creating a tuner for tuning scikit-learn models 170 Creating a
■

tuner for tuning Keras models 174 Jointly tuning and selection
■

among deep learning and shallow models 176 Hyperparameter

■

tuning beyond Keras and scikit-learn models 179

x CONTENTS

PART 3 ADVANCED TOPICS IN AUTOML .......................185

7 Customizing the search method of AutoML

7.1 Sequential search methods 188
187

7.2 Getting started with a random search method 189

7.3 Customizing a Bayesian optimization search method 193
Vectorizing the hyperparameters 194 Updating the surrogate
■

function based on historical model evaluations 198 Designing ■

the acquisition function 201 Sampling the new hyperparameters

■

via the acquisition function 204 Tuning the GBDT model with
■

the Bayesian optimization method 206 Resuming the search ■

process and recovering the search method 208

7.4 Customizing an evolutionary search method 210
Selection strategies in the evolutionary search method 210 The ■

aging evolutionary search method 212 Implementing a simple

■

mutation operation 215 Evaluating the aging evolutionary

■

search method 219

8 Scaling up AutoML
8.1
223
Handling large-scale datasets 224
Loading an image-classification dataset 225 Splitting the ■

loaded dataset 226 Loading a text-classification dataset 229

■

Handling large datasets in general 231

8.2 Parallelization on multiple GPUs 234
Data parallelism 236 ■ Model parallelism 237 ■ Parallel
tuning 238
8.3 Search speedup strategies 240
Model scheduling with Hyperband 241 Faster convergence with ■

pretrained weights in the search space 244 Warm-starting the ■

search space 247

9 Wrapping up
9.1
249
Key concepts in review 250
The AutoML process and its key components 250 The machine ■

learning pipeline 251 The taxonomy of AutoML 252

■

Applications of AutoML 252 Automated deep learning

■

with AutoKeras 253 Fully personalized AutoML with

■

KerasTuner 255 Implementing search techniques 257

■

Scaling up the AutoML process 258

 CONTENTS xi

9.2 AutoML tools and platforms 259

Open source AutoML tools 259 ■
Commercial AutoML
platforms 261
9.3 The challenges and future of AutoML 262
Measuring the performance of AutoML 262 Resource
■

complexity 263 Interpretability and transparency 263

■

Reproducibility and robustness 263 Generalizability and

■

transferability 264 Democratization and

■

productionization 264
9.4 Staying up-to-date in a fast-moving field 264

appendix A Setting up an environment for running code 266

appendix B Three examples: Classification of image, text, and
tabular data 278
index 305
 preface
The goal of automated machine learning (AutoML) is to make machine learning
(ML) accessible to everyone, including physicians, civil engineers, material scientists,
and small business owners, as well as statisticians and computer scientists. This long-
term vision is very similar to that of Microsoft Office—enabling normal users to easily
create documents and prepare reports—and cameras in smartphones, facilitating
convenient photos taken from anywhere at any time. Although the ML community
has devoted a lot of R&D efforts to pursuing this goal, through our collaboration
with domain experts and data scientists, we determined that there is a high demand
to reveal the magic behind AutoML, including fundamental concepts, algorithms,
and tools.
To begin, we would like to share several steps that got us here. (Okay, now you can
skip to our main content if you want, but hey, who doesn’t like a good story?)
We started our data science and ML journey many years ago and have been
researching and developing ML algorithms and systems from scratch ever since. In
the early days, we were tortured, like many of you, by complicated equations, unstable
results, and hard-to-understand combinations of hyperparameters. Later, more and
more advanced algorithms were developed, and open source implementations became
available. Unfortunately, training an effective machine learning/deep learning model
is still very much like alchemy, and it takes years of training to become a capable alche-
mist . . . yes, we are certified alchemists.
Over the years, we were approached by many domain experts who wanted to try
out the magical tool called machine learning because of its premier performance on

xiii
xiv PREFACE

many tasks (or simply because everyone was talking about it). Not surprisingly, it
worked well on many datasets and improved traditional rule-based or heuristic-based
methods. After working with many people with similar tasks again and again (classifi-
cation, clustering, and prediction), we were not only tired of applying ML tools but
also felt strongly that we could do something to democratize ML for all. AutoML, here
we go!
Since then, we have worked on a project called Data-Driven Discovery of Models
(D3M), supported by DARPA, and initiated the open source project, AutoKeras. We
were happy to see many people interested in our developed software, and they pro-
vided copious positive and harsh feedback on the tools we developed. At the same
time, we got the chance to know and collaborate with great researchers and engineers
working on similar problems. Everything was going in the right direction!
Our vision evolved as we worked with more and more data scientists and ML
engineers. Initially, we wanted to just help people quickly make use of ML with a few
lines of code, but we gradually realized, as we were facing too many downstream tasks
and problems, that we had a long way to go to achieve this goal. What was most urgent
was that many practitioners were working on their own AutoML systems, which could
run well with their own internal, small-scale problems, such as automated outlier detec-
tion, automated recommender systems, and automated feature engineering. Our
goal then became making ML accessible to everyone. Oops! This seemed to be the
same as our original plan! To better achieve the goal, we decided to spend a big
chunk of our time writing this book to help you make better use of, and easily develop,
AutoML tools.
We hope you enjoy the book and look forward to your feedback!
 acknowledgments
We would like to thank everyone who helped us while we wrote this book, without
whom it would not have been possible. The first person on this list is François Chollet.
He not only provided valuable guidance and feedback on the content of our book,
but also made major contributions to the design and implementation of KerasTuner
and AutoKeras, which made these libraries so delightful to use. We also really appreci-
ate his amazing work with Keras, which laid a solid foundation for the hyperparameter
tuning and AutoML work to build upon.
Thank you to all the open source contributors to KerasTuner and AutoKeras who
provided valuable feedback, and even code contributions, to these open source librar-
ies. Although we have not met all of you, your code became an indispensable part of
this large ecosystem, which has helped thousands (or maybe even millions) of people.
We send a heartfelt thank you to our lab mates from DATA Lab at Texas A&M
University, who helped us during the writing of this book. We are especially grateful
for Yi-Wei Chen, who helped us write the examples in chapter 9, which made this
book even better.
To all the reviewers: Alain Couniot, Amaresh Rajasekharan, Andrei Paleyes, David
Cronkite, Dewayne Cushman, Didier Garcia, Dimitris Polychronopoulos, Dipkumar
Patel, Gaurav Kumar Leekha, Harsh Raval, Howard Bandy, Ignacio Ruiz, Ioannis
Atsonios, Lucian Mircea Sasu, Manish Jain, Marco Carnini, Nick Vazquez, Omar El
Malak, Pablo Roccatagliata, Richard Tobias, Richard Vaughan, Romit Singhai, Satej
Kumar Sahu, Sean Settle, Sergio Govoni, Sheik Uduman Ali M, Shreesha Jagadeesh,
Stanley Anozie, Steve D Sussman, Thomas Joseph Heiman, Venkatesh Rajagopal,

xv
xvi ACKNOWLEDGMENTS

Viton Vitanis, Vivek Krishnan, Walter Alexander Mata López, Xiangbo Mao, and
Zachery Beyel, your careful reviews and suggestions gave us the incentive to keep pol-
ishing the book.
Finally, this book would not be possible without the amazing people at Manning
Publications. We send our special thanks to our editors, Toni Arritola and Rachel
Head, who provided valuable comments and worked diligently in revising our manu-
scripts. They made the book so smooth to read and taught us how to write a good
book. We also would like to thank Paul Wells, Andy Marinkovich, and Keri Hales on
the production staff and our technical proofreaders, Karsten Strøbaek and Ninoslav
Čerkez. This book could not have been written without you.
 about this book
Automated Machine Learning in Action was written to help you learn the basic concepts
of AutoML and adopt AutoML techniques to address machine learning tasks and
improve the machine learning pipeline in practice, with the help of advanced
AutoML toolkits such as AutoKeras and KerasTuner. It begins by focusing on the ele-
ments of AutoML and its connection with machine learning, then gradually leads
you to the intangibles of working with AutoML problems—from those that require
the least experience with machine learning to the ones that allow the most flexible
customization.

Who should read this book

The book aims at providing systematic guidance for learning, using, and designing
handy AutoML pipelines to students, instructors, practitioners, and researchers who
want to learn AutoML basics and adopt AutoML techniques. It’s our intention to
avoid onerous mathematical formulations and notations and instead introduce the
AutoML concepts and techniques with specific usage examples and code design snip-
pets from both users’ and developers’ perspectives.

How this book is organized: A road map

The book has three main sections that cover nine chapters. Part 1 introduces the
core concepts and some popular models of machine learning to help readers under-
stand the basic machine learning building blocks and gain the knowledge for learn-
ing AutoML. Readers who do not have much experience solving ML problems

xvii
xviii ABOUT THIS BOOK

should be sure to read this part of the book so that they can be prepared for learn-
ing AutoML.
 Chapter 1 introduces the definition, core ideas, and concepts of automated
machine learning.
 Chapter 2 walks through several concrete examples of solving an ML problem
to help you understand the machine learning building blocks and gain the
knowledge for learning AutoML.
 Chapter 3 presents the basic building blocks of deep learning and serves as a
stepping-stone to help you better understand the AutoML methods for generat-
ing and tuning deep learning methods, introduced in the second part of the
book.
Part 2 explains the ways to adopt AutoML to address ML problems and improve ML
solutions in practice.
 Chapter 4 teaches you how to use AutoML to create an end-to-end deep learn-
ing solution specifically for supervised learning problems.
 Chapter 5 discusses how to customize the AutoML search space based on your
requirements and automatically discover certain kinds of deep learning solu-
tions for different types of tasks.
 Chapter 6 goes deep into the customization of the AutoML search space. A lay-
erwise design gives you more flexibility for tuning unsupervised learning mod-
els and optimizing algorithms.
Part 3 explores some advanced AutoML design and setup from the perspectives of
search methods and acceleration strategies.
 Chapter 7 discusses how to implement a sequential search method for explor-
ing the AutoML search space.
 Chapter 8 introduces various techniques to accelerate the search process with
limited computing resources.
 Chapter 9 reviews the core concepts we’ve covered and provides you with a
short list of resources and strategies for expanding your AutoML horizons and
staying up to date with the state of the art.

About the code

This book contains many examples of source code both in numbered listings and in
line with normal text. In both cases, source code is formatted in a fixed-width font
like this to separate it from ordinary text. Sometimes code is also in bold to high-
light code that has changed from previous steps in the chapter, such as when a new
feature adds to an existing line of code.
In many cases, the original source code has been reformatted; we’ve added line
breaks and reworked indentation to accommodate the available page space in the
book. In rare cases, even this was not enough, and listings include line-continuation
 ABOUT THIS BOOK xix

markers (➥). Additionally, comments in the source code have often been removed
from the listings when the code is described in the text. Code annotations accompany
many of the listings, highlighting important concepts.
You can get executable snippets of code from the liveBook (online) version of this
book at https://livebook.manning.com/book/automated-machine-learning-in-action.
The complete code for the examples in the book is available for download from the
Manning website at http://mng.bz/y48p.
As the technologies and the open source libraries used in this book continue to
develop and evolve, the source code examples in this book are subject to changes in
the future. Please refer to our GitHub repository (http://mng.bz/M2ZQ) as the latest
source of truth for the code examples.

liveBook discussion forum

Purchase of Automated Machine Learning in Action includes free access to liveBook,
Manning’s online reading platform. Using liveBook’s exclusive discussion features,
you can attach comments to the book globally or to specific sections or paragraphs.
It’s a snap to make notes for yourself, ask and answer technical questions, and receive
help from the author and other users. To access the forum, go to https://livebook
.manning.com/book/automated-machine-learning-in-action/discussion. You can also
learn more about Manning’s forums and the rules of conduct at https://livebook
.manning.com/discussion.
Manning’s commitment to our readers is to provide a venue where a meaningful
dialogue between individual readers and between readers and the author can take
place. It is not a commitment to any specific amount of participation on the part of
the author, whose contribution to the forum remains voluntary (and unpaid). We sug-
gest you try asking the author some challenging questions lest their interest stray! The
forum and the archives of previous discussions will be accessible from the publisher’s
website as long as the book is in print.

Other online resources

Check out these resources for additional help on the topics covered in this book:
 The GitHub page (http://mng.bz/aDEj) provides a great place to submit issues
or provide comments related to our books.
 The GitHub discussion of AutoKeras (http://mng.bz/g4ve) is also a great place
to both ask questions and help others. Helping someone else is a great way to
learn!
 about the authors
DR. QINGQUAN SONG is a machine learning and relevance engi-
neer in the AI Foundation team at LinkedIn. He received his
PhD in computer science from Texas A&M University. His
research interests are automated machine learning, dynamic
data analysis, tensor decomposition, and their applications in
recommender systems and social networks. He is one of the
authors of AutoKeras. His papers have been published at major
data mining and machine learning venues, including KDD,
NeurIPS, Transactions on Knowledge Discovery from Data
(TKDD), and others.

DR. HAIFENG JIN is a software engineer on the Keras team at

Google. He is the creator of AutoKeras and the project lead of
KerasTuner. He is also a contributor to Keras and TensorFlow.
He received his PhD in computer science from Texas A&M Uni-
versity. His research interests focus on machine learning and
AutoML.

xx
 ABOUT THE AUTHORS xxi

DR. XIA “BEN” HU is an associate professor at Rice University in

the department of computer science. Dr. Hu has published
more than 100 papers in several major academic venues, includ-
ing NeurIPS, ICLR, KDD, WWW, IJCAI, and AAAI. An open
source package developed by his group, namely AutoKeras, has
become the most-used automated deep learning system on
GitHub (with over 8,000 stars and 1,000 forks). Also, his work
on deep collaborative filtering, anomaly detection, and knowl-
edge graphs has been included in the TensorFlow package,
Apple production system, and Bing production system, respec-
tively. His papers have received several Best Paper (Candidate) awards from venues
such as WWW, WSDM, and ICDM. He is the recipient of an NSF CAREER Award and
the ACM SIGKDD Rising Star Award. His work has been cited more than 10,000 times
with an h-index of 43. He was the conference general co-chair for WSDM 2020.
 about the cover illustration
The figure on the cover of Automated Machine Learning in Action is “Homme de l’ Ara-
gon,” or man from Aragon, taken from a book by Jacques Grasset de Saint-Sauveur,
published in 1797. Each illustration is finely drawn and colored by hand.
In those days, it was easy to identify where people lived and what their trade or sta-
tion in life was just by their dress. Manning celebrates the inventiveness and initiative
of today’s computer business with book covers based on the rich diversity of regional
culture centuries ago, brought back to life by pictures from collections such as this
one.

xxii
 Part 1

Fundamentals of AutoML

T he first three chapters of the book provide you an introduction of some

basic concepts and models of machine learning to help you understand the
basic machine learning building blocks and lay the groundwork for learning
AutoML. You’ll start to get a glimpse of AutoML, its concepts, and its connection
with general machine learning in chapter 1. You’ll learn the research value and
practical benefits of AutoML. Chapter 2 introduces the classic machine learning
pipeline for addressing a machine learning problem. Considering the popular-
ity of deep learning models in the AI community and beyond, in chapter 3, we
cover the basic knowledge of deep learning with the examples of three popular
types of models. We do not touch the complicated deep learning concepts but
just introduce the basic building blocks and three classical models applied on
different data formats. If you don’t have much experience with machine learn-
ing, deep learning, and how to apply them with Python, you should definitely
begin by reading part 1 in full before moving on to the practical applications of
AutoML in part 2. You can also find additional examples in appendix B to
become more familiar with the basic machine learning pipeline after reading
this part of the book.
 From machine
learning to automated
machine learning

This chapter covers

 Defining and introducing the fundamental
concepts of machine learning
 Describing the motivation for and high-level
concepts of automated machine learning

Artificial intelligence (AI), which reaches into many aspects of everyday life, has been
extensively explored in recent years. It attempts to use computational devices to
automate tasks by allowing them to perceive the environment as humans do. As a
branch of AI, machine learning (ML) enables a computer to perform a task through
self-exploration of data. It allows the computer to learn, so it can do things that go
beyond what we know how to order it to do. But the barriers to entry are high: the
cost of learning the techniques involved and accumulating the necessary experi-
ence with applications means practitioners without much expertise cannot easily
use ML. Taking ML techniques from their ivory tower and making them accessible
to more people is becoming a key focus of research and industry. Toward this end,
automated machine learning (AutoML) has emerged as a prevailing research field. Its
aim is to simulate how human experts solve ML problems and discover the optimal
ML solutions for a given problem automatically, thereby granting practitioners
without extensive experience access to off-the-shelf ML techniques. As well as being

3
4 CHAPTER 1 From machine learning to automated machine learning

beneficial for newcomers, AutoML will relieve experts and data scientists of the bur-
den of designing and configuring ML models. Being a cutting-edge topic, it is new to
most people, and its current capabilities are often exaggerated by mass media. To give
you a glimpse of what AutoML is, this chapter provides some background and an
introduction to the fundamental concepts and orients you to its research value and
practical benefits. Let’s start with a toy example.

1.1 A glimpse of automated machine learning

Suppose you want to design an ML model to recognize handwritten digits in images.
The ML model will take the images as inputs and output the corresponding digits in
each of the images (see figure 1.1).

ML model

Figure 1.1 Recognizing handwritten digits with an ML model

In case you’re not experienced with ML, let’s use a programmatic illustration with
Pythonic style to show how we usually achieve this goal in practice. We take an ML
model as an object instantiated from a class, as shown in listing 1.1. This class corre-
sponds to a specific type of ML algorithm (a set of procedures) that we would like to
use in our model.1 To instantiate a model, besides selecting the algorithm class to be
used, we also need to feed the algorithm some historical data and arguments (arg1
and arg2). The historical data used here consists of images of handwritten digits,
whose labels (corresponding numbers) are already known. This helps the machine
(or the ML algorithm) to conduct the learning process—that is, to learn how to recog-
nize the digits in images, similar to how a child is trained to recognize objects from
pictures. (You’ll see the details of this process in later sections.) The arguments here
are used to control the algorithm, instructing it how to do this process. The resulting
ML model will be able to predict the digits in previously unseen images (see figure 1.1)
with the second line of code in the next listing.

Listing 1.1 A simplified ML process

ml_model = MachineLearningAlgorithm1( Creates an ML model

arg1=..., arg2=..., data=historical_images)
digits=[model.predict_image_digit(image) for image in new_images]
Makes predictions with the ML model

1
Many well-known ML packages provide these kinds of classes corresponding to ML algorithms, such as scikit-
learn.
 A glimpse of automated machine learning 5

As you can see from the code, besides the dataset, which we may need to prepare our-
selves, we need to provide the following two things based on our prior knowledge to
address the task:
 The ML algorithm (or method) to be used; that is, MachineLearningAlgorithm1
 The arguments of the algorithm

Selecting the algorithm and configuring its arguments can be difficult in practice.
Let’s use algorithm selection as an example. As a beginner, a typical approach is to col-
lect some learning materials, explore the code for some related tasks, and identify a
pool of ML algorithms you might be able to use for the task at hand. You can then try
them out one by one on your historical data (as we do in listing 1.1) and pick the best
one based on their performance at recognizing the digits in the images. This repeti-
tive process is summarized in the next code sample.

Listing 1.2 A naive way of selecting ML algorithms

ml_algorithm_pool = [
MachineLearningAlgorithm1,
MachineLearningAlgorithm2, A pool of ML
..., algorithms to
MachineLearningAlgorithmN,
be tested
]
for ml_algorithm in ml_algorithm_pool:
Loops over all the
candidate ML algorithms
model = ml_algorithm(
arg1=..., arg2=..., Instantiates and evaluates the ML
data=historical_images) model based on each ML algorithm
result = evaluate(model)
push result into the result_pool
push model into the model_pool
best_ml_model = pick_the_best(result_pool, Selects the best ML model
ml_model_pool) based on the performance
return best_ml_model

The process looks intuitive but may take you hours or days if you do not have much
ML knowledge or experience for a few reasons. First, collecting a pool of feasible
ML algorithms could be challenging. You may need to explore the literature, iden-
tify the state-of-the-art algorithms, and learn how to implement them. Second, the
number of feasible ML algorithms could be huge. Trying them out one by one may
not be a good choice and may even be prohibitive. Third, each algorithm has its
own arguments. Configuring them correctly requires expertise, experience, and
even some luck.
Might there be a better way of doing this? Is it possible to let the machine perform
automatically for you? If you have faced similar problems and want to adopt ML in a
more labor-saving way, AutoML could be the tool you are looking for. Loosely speak-
ing, AutoML mimics the manual process described in the preceding pseudocode. It
tries to automate the repetitive and tedious process of selecting and configuring ML
algorithms and can allow you access to many advanced algorithms without even
6 CHAPTER 1 From machine learning to automated machine learning

knowing they exist. The following two lines of pseudocode illustrate how to use an
AutoML algorithm to generate the ML solution:

automl_model = AutoMLAlgorithm()
best_ml_model = automl_model.generate_model(data=historical_images)

Creating an AutoML model object from an AutoML algorithm means you don’t even
need to provide the pool of ML algorithms to test, and you can generate the desired
model simply by feeding data into it.
But how do you select an AutoML algorithm? What are the ML algorithms it will
choose from? How does it evaluate them and choose a model? Before going any fur-
ther, I’ll give you some background on ML so you can better understand what
AutoML automates and how to use it in practice to save yourself time and effort. The
focus here will be on what you need to know to learn and use AutoML. If you want to
learn more about these algorithms, I recommend referring to other ML books, such
as Machine Learning in Action by Peter Harrington (Manning, 2012) and Deep Learning
with Python, 2nd ed., by François Chollet (Manning, 2021). For readers who are
already familiar with the basics of ML, this next section will serve as a recap, make sure
we’re all on the same page with some terminology, and better motivate the following
introduction to AutoML.

1.2 Getting started with machine learning

This section provides a brief introduction to ML—what it is, the critical components
in an ML algorithm, and how an ML model is created based on a selected algorithm
and data input. Learning these basics is essential to understanding the concepts of
AutoML introduced in the next sections.

1.2.1 What is machine learning?

Before the emergence of ML, the dominant paradigm in AI research was symbolic AI,
where the computer could process data only based on predefined rules explicitly
input by humans. The advent of ML revolutionized the programming paradigm by
enabling knowledge to be learned from the data implicitly. For example, suppose you
want a machine to recognize images of apples and bananas automatically. With sym-
bolic AI, you would need to provide human-readable rules associated with the reason-
ing process, perhaps specifying features like color and shape, to the AI method. In
contrast, an ML algorithm takes a bunch of images and their corresponding labels
(“banana” or “apple”) and outputs the learned rules, which can be used to predict
unlabeled images (see figure 1.2).
The essential goals of ML are automation and generalization. Automation means an
ML algorithm is trained on the data provided to automatically extract rules (or pat-
terns) from the data. It mimics human thinking and allows the machine to improve
itself by interacting with the historical data fed to it, which we call training or learning.
The rules are then used to perform repetitive predictions on new data without human
 Getting started with machine learning 7

Data Rules (color) Data Answers

Apple Apple

Banana Banana

Symbolic AI ML

Apple Apple

Answers Rules
Banana Banana

Figure 1.2 Comparison of symbolic AI and ML

intervention. For example, in figure 1.2, the ML algorithm interacts with the apple
and banana images provided and extracts a color rule that enables it to recognize
them through the training process. These rules can help the machine classify new
images without human supervision, which we call generalizing to new data. The ability
to generalize is an important criterion in evaluating whether an ML algorithm is
good. In this case, suppose an image of a yellow apple is fed to the ML algorithm—the
color rule will not enable it to correctly discern whether it’s an apple or a banana. An
ML algorithm that learns and applies a shape feature for prediction may provide bet-
ter predictions.

1.2.2 The machine learning process

An ML algorithm learns rules through exposure to examples with known outputs.
The rules are expected to enable it to transform inputs into meaningful outputs, such
as transforming images of handwritten digits to the corresponding numbers. So, the
goal of learning can also be thought of as enabling data transformation. The learning
process generally requires the following two components:
 Data inputs—Data instances of the target task to be fed into the ML algorithm,
for example, in the image recognition problem (see figure 1.2), a set of apple
and banana images and their corresponding labels
 Learning algorithm—A mathematical procedure to derive a model based on the
data inputs, which contains the following four elements:
– An ML model with a set of parameters to be learned from the data
8 CHAPTER 1 From machine learning to automated machine learning

– A measurement to measure the model’s performance (such as prediction accu-

racy) with the current parameters
– A way to update the model, which we call an optimization method
– A stop criterion to determine when the learning process should stop
After the model parameters are intialized,2 the learning algorithm can update the
model iteratively by modifying the parameters based on the measurement until the
stop criterion is reached. This measurement is called a loss function (or objective func-
tion) in the training phase; it measures the difference between the model’s predic-
tions and the ground-truth targets. This process is illustrated in figure 1.3.

Data
inputs

Satisfy the Yes Learned

Measure the
+ current model criterion? model

ML model with
initial parameters Update model No
parameters

Figure 1.3 The process of training an ML model

Let’s look at an example to help you better understand the learning process. Imagine
we have a bunch of data points in two-dimensional space (see figure 1.4). Each point
is either black or white. We want to build an ML model that, whenever a new point
arrives, can decide whether this is a black point or a white point based on the point’s
position. A straightforward way to achieve this goal is to draw a horizontal line to sepa-
rate the two-dimensional space into two parts based on the data points in hand. This
line could be regarded as an ML model. Its parameter is the horizontal position,
which can be updated and learned from the provided data points. Coupled with the
learning process introduced in figure 1.3, the required components could be chosen
and summarized as follows:
 The data inputs are a bunch of white and black points described by their loca-
tion in the two-dimensional space.
 The learning algorithm consists of the following four selected components:
– ML model—A horizontal line that can be formulated as y = a, where a is the
parameter that can be updated by the algorithm.
– Accuracy measurement—The percentage of points that are labeled correctly
based on the model.

2
The parameter values may be intialized randomly or assigned following a strategy such as a warm start, where
you begin with some existing parameters learned by similar models.
 Getting started with machine learning 9

y y y

x x x
Accuracy
8 / 10 9 / 10 10 / 10
measure:
Stop
criterion:

Figure 1.4 An example of the learning process: Learning a horizontal line to split
white and black points

– Optimization method—Move the line up or down by a certain distance. The

distance can be related to the value of the measurement in each iteration. It
will not stop until the stop criterion is satisfied.
– Stop criterion—Stop when the measurement is 100%, which means all the
points in hand are labeled correctly based on the current line.
In the example shown in figure 1.4, the learning algorithm takes two iterations to
achieve the desired line, which separates all the input points correctly. But in practice,
this criterion may not always be satisfied. It depends on the distribution of the input
data, the selected model type, and how the model is measured and updated. We often
need to choose different components and try different combinations to adjust the
learning process to get the expected ML solution. Also, even if the learned model is
able to label all the training inputs correctly, it is not guaranteed to work well on
unseen data. In other words, the model’s ability to generalize may not be good (we’ll
discuss this further in the next section). It’s important to select the components and
adjust the learning process carefully.

1.2.3 Hyperparameter tuning

How do we select the proper components to adjust the learning process so that we can
derive the expected model? To answer this question, we need to introduce a concept
called hyperparameters and clarify the relationship between these and the parameters
we’ve been discussing as follows:
 Parameters are variables that can be updated by the ML algorithm during the
learning process. They are used to capture the rules from the data. For exam-
ple, the position of the horizontal line is the only parameter in our previous
example (figure 1.4) to help classify the points. It is adjusted during the train-
ing process by the optimization method to capture the position rule for split-
ting the points with different colors. By adjusting the parameters, we can derive
an ML model that can accurately predict the outputs of the given input data.
10 CHAPTER 1 From machine learning to automated machine learning

 Hyperparameters are also parameters, but they’re ones we predefine for the algo-
rithm before the learning process begins, and their values remain fixed during
the learning process. These include the measurement, the optimization
method, the speed of learning, the stop criterion, and so on. An ML algorithm
usually has multiple hyperparameters. Different combinations of them have dif-
ferent effects on the learning process, resulting in ML models with different
performances. We can also consider the algorithm type (or the ML model type)
as a hyperparameter, because we select it ourselves, and it is fixed during the
learning process.
The selection of an optimal combination of hyperparameters for an ML algorithm is
called hyperparameter tuning and is often done through an iterative process. In each
iteration, we select a set of hyperparameters to use to learn an ML model with the
training dataset. The ML algorithm block in figure 1.5 denotes the learning process
described in figure 1.3. By evaluating each learned model on a separate dataset called
the validation set, we can then pick the best one as the final model. We can evaluate the
generalizability of that model using another dataset called the test set, which concludes
the whole ML workflow.

Manually adjust
hyperparameter. T esting
Training

Training ML Learned Final

dataset algorithm model model

Validation T est
dataset dataset Figure 1.5 The classic
Tuning ML workflow

In general, we will have three datasets in the ML workflow. Each dataset is distinct
from the other two, as described next:
 The training set is used during the learning process to train a model given a
fixed combination of hyperparameters.
 The validation set is used during the tuning process to evaluate the trained
models to select the best hyperparameters.
 The test set is used for the final testing, after the tuning process. It is used only
once, after the final model is selected, and should not be used for training or
tuning the ML algorithm.
The training and test sets are straightforward to understand. The reason we want to
have an additional validation dataset is to avoid exposing the algorithm to all the
 Getting started with machine learning 11

training data during the tuning stages—this enhances the generalizability of the final
model to unseen data. If we don’t have a validation set, the best model selected in the
tuning stage would be the one that focuses on extracting any subtle features in the
training data to ceaselessly increase the training accuracy without caring about any
unseen dataset. This situation will likely lead to bad performance on the final test
set, which contains different data. When the model performs worse on the test set
(or validation set) than the training set, this is called overfitting. It’s a well-known prob-
lem in ML and often happens when the model’s learning capacity is too strong and
the size of the training dataset is limited. For example, suppose you want to predict
the fourth number of a series, given the first three numbers as training data: a1 = 1,
a2 = 2, a3 = 3, a4 = ? (a4 is the validation set here; a5 onward are the test sets.) If the
right solution is a4 = 4, a naive model, ai = i, would provide the correct answer. If you
use a third-degree polynomial to fit the series, a perfect solution for the training data
would be ai = i 3 – 6i 2 + 12i – 6, which will predict a4 as 10. The validation process
enables a model’s generalization ability to be better reflected during evaluation so
that better models can be selected.

NOTE Overfitting is one of the most important problems studied in ML.

Besides doing validation during the tuning process, we have many other ways
to address the problem, such as augmenting the dataset, adding regulariza-
tion to the model to constrain its learning capacity during training, and so
on. We won’t go into this in more depth here. To learn more about this topic,
see Chollet’s Deep Learning with Python.

1.2.4 The obstacles to applying machine learning

At this point, you should have a basic understanding of what ML is and how it pro-
ceeds. Although you can make use of many mature ML toolkits, you may still face dif-
ficulties in practice. This section describes some of these challenges—the aim is not to
scare you off, but to provide context for the AutoML techniques that are described
afterward. Obstacles you may meet include the following:
 The cost of learning ML techniques—We’ve covered the basics, but more knowl-
edge is required when applying ML to a real problem. For example, you’ll need
to think about how to formulate your problem as an ML problem, which ML
algorithms you could use for your problem and how they work, how to clean
and preprocess the data into the expected format to input into your ML algo-
rithm, which evaluation criteria should be selected for model training and
hyperparameter tuning, and so on. All these questions need to be answered in
advance, and doing so may require a large time commitment.
 Implementation complexity—Even with the necessary knowledge and experience,
implementing the workflow after selecting an ML algorithm is a complex task.
The time required for implementation and debugging will grow as more
advanced algorithms are adopted.
12 CHAPTER 1 From machine learning to automated machine learning

 The gap between theory and practice—The learning process can be hard to inter-
pret, and the performance is highly data driven. Furthermore, the datasets used
in ML are often complex and noisy and can be difficult to interpret, clean, and
control. This means the tuning process is often more empirical than analytical.
Even ML experts sometimes cannot achieve the desired results.
These difficulties significantly impede the democratization of ML to people with lim-
ited experience and correspondingly increase the burden on ML experts. This has
motivated ML researchers and practitioners to pursue a solution to lower the barriers,
circumvent the unnecessary procedures, and alleviate the burden of manual algo-
rithm design and tuning—AutoML.

1.3 AutoML: The automation of automation

The goal of AutoML is to allow a machine to mimic how humans design, tune, and
apply ML algorithms so that we can adopt ML more easily (see figure 1.6). Because a key
property of ML is automation, AutoML can be regarded as automating automation.

Automated algorithm
selection and tuning
Move human
out of the loop.

Manually adjust
hyperparameter. T esting
Training

Training ML Learned Final

dataset algorithm model model

Validation T est
dataset dataset
Tuning

Figure 1.6 The main goal of AutoML: taking humans out of the loop of
ML algorithm design and tuning

To help you understand how AutoML works, let’s first go over the key components.

1.3.1 Three key components of AutoML

Here’s a recap of the pseudocode introduced in section 1.1:

ml_algorithm_pool = [
MachineLearningAlgorithm1,
MachineLearningAlgorithm2,
...,
 AutoML: The automation of automation 13

MachineLearningAlgorithmN,
]
for ml_algorithm in ml_algorithm_pool:
model = ml_algorithm(arg1=..., arg2=..., data=historical_images)
result = evaluate(model)
push result into the result_pool
push model into the model_pool
best_ml_model = pick_the_best(result_pool, ml_model_pool)
return best_ml_model

This pseudocode can be regarded as a simple AutoML algorithm that takes a pool of
ML algorithms as input, evaluates them one by one, and outputs a model learned
from the best algorithm. Each AutoML algorithm consists of the following three core
components (see figure 1.7):
 Search space—A set of hyperparameters, and the ranges of each hyperparameter
from which to select. The range of each hyperparameter can be defined based
on the user’s requirements and knowledge. For example, the search space can
be a pool of ML algorithms, as shown in the pseudocode. In this case, we treat
the type of ML algorithm as a hyperparameter to be selected. The search space
can also be the hyperparameters of a specific ML algorithm, such as the struc-
ture of the ML model. The design of the search space is highly task-dependent,
because we may need to adopt different ML algorithms for various tasks. It is
also quite personalized and ad hoc, depending on the user’s interests, exper-
tise, and level of experience. There is always a tradeoff between the conve-
nience you’ll enjoy by defining a large search space and the time you’ll spend
identifying a good model (or the performance of the model you can achieve in
a limited amount of time). For beginners, it can be tempting to define a broad
search space that is general enough to apply to any task or situation, such as a
search space containing all the ML algorithms—but the time and computa-
tional cost involved make this a poor solution. We’ll discuss these consider-

Search space Search Performance

strategy evaluation strategy

Training ML Learned Final

dataset algorithm model model

Validation Test
dataset dataset
AutoML algorithm

Figure 1.7 The AutoML process

14 CHAPTER 1 From machine learning to automated machine learning

ations more in the second part of the book, where you’ll learn how to customize
your search space in different scenarios based on additional requirements.
 Search strategy—A strategy to select the optimal set of hyperparameters from the
search space. Because AutoML is often an iterative trial-and-error process, the
strategy often sequentially selects the hyperparameters in the search space and
evaluates their performance. It may loop through all the hyperparameters in
the search space (as in the pseudocode), or the search strategy may be adapted
based on the hyperparameters that have been evaluated so far to increase the
efficiency of the later trials. A better search strategy can help you achieve a bet-
ter ML solution within the same amount of time. It may also allow you to use a
larger search space by reducing the search time and computational cost. How
to adopt, compare, and implement different search algorithms will be intro-
duced in the third part of the book.
 Performance evaluation strategy—A way to evaluate the performance of a specific
ML algorithm instantiated by the selected hyperparameters. The evaluation cri-
teria are often the same as the ones used in manual tuning, such as the valida-
tion performance of the model learned from the selected ML algorithm. In this
book, we discuss different evaluation strategies in the context of adopting
AutoML to solve different types of ML tasks.
To facilitate the adoption of AutoML algorithms, an AutoML toolkit often wraps up
these three components and provides some general application programming inter-
faces (APIs) with a default search space and search algorithm so that you don’t need
to worry about selecting them yourself. For end users, in the simplest case, all you
need to do to obtain the final model is provide the data, as shown here—you don’t
even need to split the data into training and validation sets:
automl_model = AutoMLAlgorithm()
best_ml_model = automl_model.generate_model(data=...)

But because different users may have different use cases and levels of ML expertise, they
may need to design their own search spaces, evaluation strategies, and even search strat-
egies. Existing AutoML systems, therefore, often also provide APIs with configurable
arguments to allow you to customize different components. A broad spectrum of solu-
tions are available, from the simplest to the most configurable (figure 1.8).

Simple Configurable

Less ML More ML Figure 1.8 The spectrum

experience experience of AutoML APIs

The range of APIs available allows you to pick the most suitable one for your use case.
This book will teach you how to select the right API in an advanced AutoML toolkit,
AutoKeras, for different AutoML applications. You’ll also learn how to create your own
AutoML algorithm with the help of KerasTuner.
 AutoML: The automation of automation 15

1.3.2 Are we able to achieve full automation?

The field of AutoML has been evolving for three decades, with the involvement of
industry and the open source community. Many successful implementations and
promising developments have been seen, as described here:
 Many company internal tools and open source platforms have been developed
to help with hyperparameter tuning of ML models and model selection (Goo-
gle Vizier, Facebook Ax, and so on).
 AutoML solutions performing at near human levels have been observed in
many Kaggle data science competitions.
 Vast open source ML packages for improved hyperparameter tuning and ML pipe-
line creation have been developed, such as Auto-sklearn, AutoKeras, and so on.
 Commercial AutoML products are helping many companies, big and small, to
adopt ML in production. For example, Disney has successfully used Google
Cloud AutoML to develop ML solutions for its online store without hiring a team
of ML engineers (https://blog.google/products/google-cloud/cloud-automl-
making-ai-accessible-every-business/).
 Researchers in fields other than computer science, such as medicine, neuro-
biology, and economics, are also leveraging the power of AutoML. They can now
bring new ML solutions to domain-specific problems such as medical image seg-
mentation,3 genomic research,4 and animal recognition and protection,5 with-
out going through the long learning curve of ML and programming.
We are still exploring the full capabilities of AutoML to democratize ML techniques
and make them accessible to more people in different domains. Despite the many suc-
cessful applications of AutoML that have been seen so far, we still have a lot of chal-
lenges and limitations to further explore and address, including the following:
 The difficulty of building AutoML systems—Compared to building an ML system,
building an AutoML system from scratch is a more complex and involved process.
 The automation of collecting and cleaning data—AutoML still requires people to
collect, clean, and label data. These processes are often more complicated in
practice than the design of ML algorithms, and, for now at least, they cannot be
automated by AutoML. For AutoML to work today, it has to be given a clear task
and objective with a high-quality dataset.
 The costs of selecting and tuning the AutoML algorithm—The “no free lunch” theo-
rem tells us that there is no omnipotent AutoML algorithm that fits any hyper-
parameter tuning problem. The effort you save on selecting and tuning an ML

3
Weng, Yu, et al., “NAS-Unet: Neural Architecture Search for Medical Image Segmentation,” IEEE Access 7
(2019): 44247–44257.
4
Liu, Denghui, et al., “AutoGenome: An AutoML Tool for Genomic Research,” bioRxiv (2019): 842526.
5
Liu, Yao, and Ze Luo, “Species Recognition of Protected Area Based on AutoML,” Computer Systems and Appli-
cations 28 (2019): 147–153.
16 CHAPTER 1 From machine learning to automated machine learning

algorithm may be amortized or even outweighed by the effort you need to put
into selecting and tuning the AutoML algorthm.
 Resource costs—AutoML is a relatively costly process, in terms of both time and
computational resources. Existing AutoML systems often need to try more
hyperparameters than human experts to achieve comparable results.
 The cost of human–computer interaction —Interpreting the solution and the tuning
process of AutoML may not be easy. As these systems become more complex, it
will become harder and harder for humans to get involved in the tuning pro-
cess and understand how the final model is achieved.
AutoML is still in its early stages of development, and its continuing progress will rely
heavily on the participation of researchers, developers, and practitioners from differ-
ent domains. Although you may contribute to that effort one day, the goal of this book
is more modest. It mainly targets practitioners who have limited expertise in machine
learning, or who have some experience but want to save themselves some effort in cre-
ating ML solutions. The book will teach you how to address an ML problem automati-
cally with as few as five lines of code. It will gradually approach more sophisticated
AutoML solutions for more complicated scenarios and data types, such as images,
text, and so on. To get you started, in the next chapter, we’ll dig more deeply into the
fundamentals of ML and explore the end-to-end pipeline of an ML project. It will
help you better understand and make use of AutoML techniques in the later chapters.

Summary
 Machine learning refers to the capacity of a computer to modify its processing
by interacting with data automatically, without being explicitly programmed.
 The ML process can be described as an iterative algorithmic process to adjust
the parameters of an ML model based on the data inputs and certain measure-
ments. It stops when the model is able to provide the expected outputs, or
when some particular criterion defined by the user is reached.
 Tuning the hyperparameters in an ML algorithm allows you to adjust the learn-
ing process and select components tailored to the ML problem at hand.
 AutoML aims to learn from the experience of designing and applying ML mod-
els and automate the tuning process, thereby relieving data scientists of this
burden and making off-the-shelf ML techniques accessible to practitioners with-
out extensive experience.
 An AutoML algorithm consists of three key components: the search space,
search strategy, and evaluation strategy. Different AutoML systems provide dif-
ferent levels of APIs that either configure these for you or allow you to custom-
ize them based on your use case.
 AutoML contains many unaddressed challenges, preventing it from living up to
the highest expectations. Achieving true automatic machine learning will be diffi-
cult. We should be optimistic but also take care to avoid exaggerating AutoML’s
current capabilities.
 The end-to-end pipeline
of an ML project

This chapter covers

 Getting familiar with the end-to-end pipeline
for conducting an ML project
 Preparing data for ML models (data collection
and preprocessing)
 Generating and selecting features to enhance
the performance of the ML algorithm
 Building up linear regression and decision tree
models
 Fine-tuning an ML model with grid search

Now that the first chapter has set the scene, it’s time to get familiar with the basic
concepts of ML and AutoML. Because AutoML is grounded in ML, learning the
fundamentals of ML will help you better understand and make use of AutoML
techniques. This is especially the case when it comes to designing the search space
in an AutoML algorithm, which characterizes the ML components to be used and
the ranges of their hyperparameters. In this chapter, we will walk through a con-
crete example of solving an ML problem. This will help you gain a deeper under-
standing of the overall process of building up an ML pipeline, especially if you have
little experience working on ML projects. You will also learn a naive way of tuning

17
18 CHAPTER 2 The end-to-end pipeline of an ML project

the hyperparameters of an ML model. This can be thought of as one of the simplest

applications of AutoML, showing how it can help you find a better ML solution. More
advanced AutoML tasks and solutions will be introduced in the second part of the book.
NOTE All of the code snippets included in this and later chapters are written
in Python, in the form of Jupyter notebooks. They are all generated by Jupy-
ter Notebook (https://jupyter.org), an open source web application that
offers features such as interactive code design, data processing and visualiza-
tion, narrative text, and so on. It is widely popular in the machine learning
and data science communities. If you’re not familiar with the environmental
setup or do not have sufficient hardware resources, you can also run the code
in Google Colaboratory (http://colab.research.google.com/), a free Jupyter
notebook environment where anyone can run ML experiments. Detailed
instructions for setting up the environment in Google Colaboratory, or Colab
for short, are provided in appendix A. The notebooks are available at https://
github.com/datamllab/automl-in-action-notebooks.

2.1 An overview of the end-to-end pipeline

An ML pipeline is a sequence of steps for conducting an ML project. Those steps follow:
 Problem framing and data collection—Frame the problem as an ML problem and
collect the data you need.
 Data preprocessing and feature engineering—Process the data into a suitable format that
can be input into the ML algorithms. Select or generate features that are related
to the target output to improve the performance of the algorithms. This step is
usually done by first exploring the dataset to get a sense of its characteristics. The
operations should accommodate the specific ML algorithms you are considering.
 ML algorithm selection —Select ML algorithms appropriate for the task that you
would like to test, based on your prior knowledge of the problem and your
experience.
 Model training and evaluation—Apply the selected ML algorithm (or algorithms)
to train an ML model with your training data, and evaluate its performance on
the validation dataset.
 Hyperparameter tuning—Attempt to achieve better performance by iteratively
tuning the model’s hyperparameters.
 Service deployment and model monitoring—Deploy the final ML solution, and moni-
tor its performance so you can maintain and improve the pipeline continuously.
As you can see, an ML project is a human-in-the-loop process. Starting from problem
framing and data collection, the pipeline involves multiple data processing steps,
which typically happen asynchronously (see figure 2.1). We will focus on the steps
before service deployment and monitoring in the rest of the book. To learn more
about deploying and serving models, please refer to a reference such as Machine Learning
Systems by Jeff Smith (Manning, 2018) or Machine Learning for Business by Doug Hud-
geon and Richard Nichol (Manning, 2019).
 Framing the problem and assembling the dataset 19

Hyperparameter tuning

Problem framing and Data preprocessing and ML algorithm Model training Service deployment and
dataset collection feature engineering selection and evaluation model monitoring

Continuous maintenance and feedback integration

Figure 2.1 The end-to-end ML project pipeline

Let’s start working on a real problem to get you familiar with each component in the
pipeline. The problem we explore here is predicting the average house price in a
housing block, given features of the houses such as their locations and number of
rooms. The data we use is the California housing dataset featured in R. Kelley Pace
and Ronald Barry’s 1997 article “Sparse Spatial Autoregressions,” collected via the
1990 census. This is a representative problem used in many practical ML books as a
starter, due to the small scale of the data and the simplicity of the data preparation.

NOTE Selecting the right problem to work on can be difficult. It depends on

multiple factors, such as your business need and research objectives. Before
really engaging in a problem, ask yourself what solutions you expect to
achieve and how they will benefit your downstream applications and whether
any existing work has already fulfilled the need. This will help you decide
whether the problem is worth investing in.

2.2 Framing the problem and assembling the dataset

The first thing you need to do in any ML project is frame the problem and collect the
corresponding data. Framing the problem requires you to specify the inputs and out-
puts of the ML model. In the California housing problem, the inputs are the set of fea-
tures describing the housing blocks.
In this dataset, a housing block is a group of, on average, 1,425 individuals living in
a geographically compact area. The features are the average number of rooms per
house in the housing block, the latitude and longitude of the center of the block, and
so on. The outputs should be the average housing prices of the blocks. We are trying
to train an ML model given housing blocks with known median prices and predict the
unknown prices of housing blocks based on their features. The returned predicted
values are also called the targets (or annotations) of the model. Generally, any problem
aiming to learn the relationship between data inputs and targets based on existing
annotated examples is called a supervised learning problem. This is the most widely stud-
ied branch of ML and will be our main focus in the rest of this book.
20 CHAPTER 2 The end-to-end pipeline of an ML project

We can further classify supervised learning problems into different categories

based on the type of target value. For example, any supervised learning problem with
continuous targets can be categorized as a regression problem. Because price is a contin-
uous variable, predicting California housing prices is, therefore, essentially a regres-
sion problem. If instead the target values in a supervised learning problem are
discrete values with limited categories, we call the problem a classification problem.
You can find some examples of classification problems in appendix B, and we will also
explore them in the next chapter.
After framing the problem, the next step is to collect the data. Because the Califor-
nia housing dataset is one of the most-used ML datasets, you can easily access it with
scikit-learn, a popular ML library. It should be noted, however, that in real life, discov-
ering and acquiring datasets is a nontrivial activity and might require additional skills,
such as knowledge of Structured Query Language (SQL), which is beyond the scope of
this book. (To learn more about this, see Jeff Smith’s book Machine Learning Systems.)
The following code will load the dataset for our problem.

Listing 2.1 Loading the California housing dataset

Imports the dataset-
from sklearn.datasets import fetch_california_housing loading function from
the scikit-learn library
house_dataset = fetch_california_housing()
Loads the California
housing dataset
The original data is a dictionary containing the data points formatted as an instance-
feature matrix. Each data point is a housing block described by the features in a row
of the matrix. Their targets are formatted as a vector. The dictionary also contains the
feature names and descriptions indicating the features’ meanings and creation infor-
mation for the dataset, as shown next:

>>> house_dataset.keys()
dict_keys(['data', 'target', 'feature_names', 'DESCR'])

After loading the original dataset, we extract the data points and convert them into a
DataFrame, which is a primary structure of the pandas library. pandas is a potent tool
for data analysis and manipulation in Python. As shown in listing 2.2, the targets are
formatted as a Series object; it’s a vector with the label “MedPrice,” standing for the
median price of the housing block in millions of dollars.

Listing 2.2 Extracting the data samples and the targets

Imports the pandas package Extracts the features with

import pandas as pd their names into a DataFrame
data = pd.DataFrame(house_dataset.data, columns=house_dataset.feature_names)

target = pd.Series(house_dataset.target, name = 'MedPrice')

Extracts the targets into a Series
object with the name "MedPrice"
 Framing the problem and assembling the dataset 21

Let’s print out the first five samples of the data (shown in figure 2.2). The first row
indicates the feature names, details of which can be found at https://scikit-learn.org/
stable/datasets.html. For example, the “AveRooms” feature indicates the average
number of rooms within a housing block. We can also check the values of the target
data the same way, shown here:

>>> data.head(5)

Figure 2.2 Features of the first five samples in the California housing dataset

Before moving on to the data-preprocessing step, let’s first do a data split to separate
the training data and test set. As you learned in the previous chapter, the primary pur-
pose of doing this is to avoid testing your model on the same data that you use to con-
duct your analysis and train your model. The code for splitting the data into training
and test sets is shown in the following listing.

Listing 2.3 Splitting the data into training and test sets

from sklearn.model_selection import train_test_split

Imports the data
split function from
X_train, X_test, y_train, y_test = train_test_split( scikit-learn
data, target,
test_size=0.2, Randomly splits 20% of
random_state=42) the data into the test set

We split a random 20% of the data into the test set. Now let’s do a quick check of the
split. Looking at the full dataset, you’ll see that it contains 20,640 data points. The
number of features for each housing block is eight. The training set contains 16,512
samples, and the test set contains 4,128 samples, as depicted in the next code snippet:

>>> (data.shape, target.shape), (X_train.shape, y_train.shape), (X_test.shape,

y_test.shape)
(((20640, 8), (20640,)), ((16512, 8), (16512,)), ((4128, 8), (4128,)))

You should not touch the target data in the test set until you have your final ML solu-
tion. Otherwise, all your analysis, including the data preparation and model training,
Exploring the Variety of Random
Documents with Different Content
 CHAPTER IV.
PROMINENT REGIMENTS AND OTHER SUBORDINATE
COMMANDS ENGAGED IN THE BATTLE.

Confederate Commands.—The 3d Louisiana Infantry—1st Arkansas

Mounted Rifles—2d Arkansas Mounted Rifles—McRea’s Arkansas
Battalion—3d Arkansas Infantry—4th Arkansas Infantry—6th
Arkansas Infantry—1st Arkansas Cavalry. Federal Commands.—The
1st Iowa Infantry—1st Missouri Infantry—1st Kansas Infantry—2d
Kansas Infantry—Totten’s Battery—Dubois’ Battery—Steele’s
Battalion—Plummer’s Battalion—The Home Guards.

The particular part taken in the battle of Wilson’s creek by some of

the leading regiments of each side may be of interest, and is here
described, the facts being obtained from actual participants—the
commanding officers when possible.

CONFEDERATE COMMANDS.

THIRD LOUISIANA INFANTRY.

A considerable portion of the services of this regiment have already

been narrated. Aroused by Gen. McCulloch himself, the colonel of
the regiment, Louis Hebert, formed the 3d Louisiana and followed
the road to Springfield, a short distance to a narrow by-road running
north and leading to Ray’s corn-field, then held by Plummer’s
regulars. In front of the corn-field was a dense thicket, through
which the regiment advanced, and here it instantly became
engaged. At the first fire Sergeant Major Renwick, of the regiment,
and Private Placide Bossier, of the “Pelican Rangers No. 1,” of
Natchitoches, were killed.
The 3d Louisiana jumped the fence, charged, and soon drove
Plummer’s men from the corn-field. Still advancing, they reached an
oat field, and here Dubois’ battery opened on them, as did some of
the Federal infantry, and the regiment was driven back in some
confusion. Col. Hebert ordered it to fall back to the woods higher to
the right, but the regiment became separated, and the greater
portion—the right wing and some of the left—were formed outside
of the field by Lieut. Col. S. M. Hyams, and, by orders of Gen. 77
McCulloch, went down the creek valley to attack Sigel. Col.
Hebert succeeded in forming two companies into a detachment of
about 100 men and marched in an opposite direction, toward the
force under Gen. Lyon. Col. Hebert advanced within about 500 yards
of Totten’s battery, where he remained in front of the Federal line for
nearly half an hour under a severe fire, when the detachment was
forced to retire. Again it formed and then marched down and joined
the right wing under Lt.-Col. Hyams, which had just returned from
defeating Sigel. The entire regiment then moved against the Federal
position on Bloody Hill.

The companies led by Lieut. Col. Hyams against Sigel were the
“Pelican Rifles,” Capt. John P. Vigilini; the “Eberville Grays,” Lieut.
Verbois; the “Morehouse Guards,” Capt. Hinson; the “Pelican
Rangers No. 1,” Capt. Breazeale; the “Pelican Rangers No. 2,” Capt.
Blair; the “Winn Rifles,” Capt. Pierson; the “Morehouse Fencibles,”
Capt. Harris; the “Shreveport Rangers,” Capt. Gilmore; a few of the
“Monticello Rifles,” under Sergeant Walcott, and a detachment of
Missourians, 75 in number, commanded by a Capt. Johnson. The
regiment was conducted across the ford of Wilson’s creek and down
the valley in front of Sigel’s position by Col. James McIntosh.

Arriving in front of Sigel’s battery, the regiment formed, and by order

of Lieut. Col. Hyams advanced up the steep hill to the charge. Near
the brow of the hill Lieut. Lacey, of the “Shreveport Rangers,” sprang
on a log, waved his sword, and called out to his company, “Come on,
Caddo!” Shreveport is in the parish of Caddo, Louisiana. The whole
command rushed forward, carried the position, captured the guns,
and drove the already panic-stricken Federal Germans in terror from
the field. The captured cannon were rolled down the hill, and one
piece, with its horses, was taken to the main command.

The “Pelican Rifles” and the “Iberville Grays” were under command
of Capt. Vigilini, of the former company. When within thirty or forty
yards of Sigel’s battery, Gen. McCulloch being in person with the 3d
Louisiana, a Federal soldier appeared in plain view on the hill. Gen.
McCulloch himself called out, “What troops are those?” The 78
man replied, “Sigel’s regiment,” at the same time raising his
rifle to shoot the general; but Corporal Henry Gentles, of Vigilini’s
company, had his Mississippi rifle in position and shot the Federal
dead in an instant, thus saving Gen. McCulloch’s life.

In the assault on Sigel Capt. Hinson and his brother-in-law, Private

Whetstone, both of the “Morehouse Guards,” were killed, it is said,
by the same shot. Among the members of the regiment who
distinguished themselves were Color-bearer Felix Chaler, Corporal
Hicock (killed), Drum-Major Patterson, Orderly Sergeant Alphonse
Prudhomme, Private I. P. Hyams, Corporal Gentles, and Sergt. W. H.
Tunnard. The regiment was especially complimented by Gen.
McCulloch in his official report.

The 3d Louisiana lost in killed one commissioned officer, one non-

commissioned officer and seven privates; total killed nine; wounded,
three commissioned officers, six non-commissioned officers and
thirty-nine privates; total wounded, forty-eight; missing, three
privates. Total casualties, sixty.

FIRST ARKANSAS MOUNTED RIFLES.

The 1st Arkansas Mounted rifles was at the time of the battle of
Wilson’s creek in the Confederate service, and was commanded by
Col. T. J. Churchill. It belonged to McCulloch’s division and was
encamped at the lower end of the Confederate position. At about
breakfast time Sigel’s battery and his infantry opened on the
regiment, which lay in an open field. Being exposed to a raking fire
from cannon and musketry, the regiment fell back into the woods on
the north, and there formed under direction of Col. Churchill himself,
who, as soon as his alignment had been made, moved down on the
wire road in the direction of Springfield.

Having reached the little valley of Wilson’s creek, Col. Churchill was
met by an aid de camp of Gen. Price, asking for a reinforcement to
go to the assistance of Gen. Slack, then being hard pressed. Col.
Churchill immediately moved his regiment rapidly forward, under a
heavy fire, took position on Gen. Slack’s left, and ordered his men to
commence firing. Here the 1st Arkansas fought for about four 79
hours, being in front of Totten’s battery and the 1st Iowa the
greater portion of the time. At times it would advance, then fall
back, but its conduct was most admirable, there being but little if
any straggling or disorder. It continued to fight until the Federals
retreated from the field.

During the engagement Col. Churchill had two horses shot under
him. The lieutenant colonel, Matlock, and the major, Harper, of the
regiment evinced great bravery and gallantry. Adjutant James
Harper, Capt. M. E. Alexander, and Lieuts. Dawson, Chambers, and
Johnson were killed; Capts. Ramsaur and Porter, and Lieuts. King,
Raney, Adams, Hardester and McIvor were severely wounded, and
Capts. Pearson and Gibbs and Lieuts. Saddler, Wair and Head were
slightly wounded. The 1st Arkansas suffered more than any other
regiment of Southern troops engaged in the battle. Its loss was
forty-two killed and one hundred and fifty-five wounded.

In acknowledgment of the conspicuous services rendered by the 1st

Arkansas to his army, Gen. Price, a few days after the battle caused
the following letter to be sent to Col. Churchill:—

Headquarters Missouri State Guard,

Springfield, Mo., Aug. 15, 1861.
 Colonel:—I am directed by Maj.-Gen. Price to thank you, in the
name of this army and of the State of Missouri, for the very
important services which you and your fine regiment of Mounted
Riflemen have rendered during the campaign in this State, and do
particularly acknowledge, in the most grateful manner, the eager
bravery with which your men met the enemy on the 10th inst.—
the constancy with which they fought, and the spirit with which
they rushed upon and drove back his disciplined soldiers. Your
own gallantry and skill were so conspicuous on that memorable
day that every Missourian will always cherish the remembrance of
you with pride and gratitude.

I have the honor to be, Colonel,

Your obedient servant,

Thos. L. Snead,
Acting Adjt. Gen.

Col. Thos. J. Churchill, 1st Regt. Ark. Mounted Rifles.

SECOND ARKANSAS MOUNTED RIFLES.

This regiment was led by its commander, Col. James McIntosh, 80

into the battle, and took part with the 3d Louisiana in the fight
with Plummer’s regulars, in Ray’s cornfield, early in the morning.
When first attacked it was at breakfast, but, instead of retreating in
confusion, rallied at the call of the bugle, mounted, and was
marched by Lieut. Col. Benj. T. Embry to the timber on the east side
of Wilson’s Creek, north of Woodruff’s battery. Here it dismounted
and stripped for the fight, and Col. McIntosh then appeared and
took it into the engagement.

After the fight in Ray’s cornfield, Col. McIntosh was sent for by Gen.
McCulloch, and Lieut. Col. Embry took command of the 2d Arkansas.
The regiment moved across the creek to the west and became
engaged with Lyon’s men on Bloody Hill. At one time it repulsed a
desperate charge, losing heavily in so doing, however. After this,
under Col. Embry, the regiment fell back to the creek and rested a
short time, preparing to receive a cavalry charge, which it was
expected would be made. It then moved up the hill again, but did
not become engaged, and soon after the Federals retreated.

The loss of the 2d Arkansas was, in killed, one non-commissioned

officer and 9 privates; total, 10. Wounded, one captain, two second
lieutenants, eight non-commissioned officers, and 33 privates; total
44. Total casualties, 54.

M’RAE’S ARKANSAS BATTALION.

Upon the opening of the battle in good earnest, or about 6 A. M.,

this battalion, led by its commander, Lieut. Col. D. H. McRae, moved
up the wire road toward Springfield, and formed to the left of the 3d
Louisiana, and in front of Woodruff’s battery. In a short time, by
Gen. McCulloch’s orders, it countermarched and moved off across
the valley toward the southwest to take and hold an eminence in
that quarter threatened by Sigel’s men. While on its way the
battalion was broken up by a large body of mounted Missourians,
who, panic-stricken and demoralized, were riding rapidly away from
Totten’s battery and the Federals on Bloody Hill. These mounted
warriors rode wildly through the battalion, threatening to trample
down the men and forcing them to scatter to save themselves. Col.
McRae was able to take but one entire company and a few 81
files of another into the fight proper.

On arriving at the summit of the hill the battalion was fired on by a

battery reported as being Bledsoe’s, of the Missouri troops, but in
reality was Sigel’s. Thick brush intervening, Col. McRae was unable
to distinguish for himself, but at last charged at a “trail arms.” Within
twenty paces of the Fayetteville road a body of men were observed
moving rapidly away, and these were fired on. Here Col. McRae
halted and formed his men so as to sweep the road. In a short time
another body came up, and being dressed like the Confederates, and
some of them calling out, “We are from the South,” deceived Col.
McRae until nearly all of them had passed, when he opened fire on
their rear. He then led his men to the hill where Sigel’s battery had
been captured, and here he found the three companies of the
battalion that had been cut off by the Missouri horsemen. The united
battalion then marched to the Fayetteville road to the north, it
having been reported that the Federals were reforming there, but
this report was found to be untrue, and Col. McRae returned to
camp.

The loss of the battalion was two men killed, one mortally wounded,
one severely wounded, and five slightly wounded.

THIRD ARKANSAS INFANTRY.

This regiment was commanded by Col. John R. Gratiot, and

belonged to the Arkansas State troops, Gen. Pearce’s division. Its
lieutenant-colonel, David Provence, and its major, Ward, were
present at the battle.

In the first part of the action the regiment was moved to the support
of Woodruff’s battery, and here it remained for some hours under a
heavy fire of shot and shell. At about 11 o’clock Gen. Pearce ordered
the regiment to cross the creek and move to the help of Price’s
division. Col. Gratiot marched the men over the stream and up the
ridge by a flank movement and in column of fours. When near the
Federal position the line was fronted and faced the enemy, and
moved forward, but just then a heavy fire was opened in front, two
guns of Totten’s battery turned loose on the regiment with 82
grape, and canister and shell, and so terrific was the ordeal,
that the regiment was obliged to lie down and return the fire in that
position. This was the last fight of Lyon’s men, and they kept it up
only about thirty minutes, when they retreated. The 3d Arkansas
remained on the field in position long after the firing had ceased.
After Maj. Sturgis had retreated with the remnant of Lyon’s division
from Bloody Hill, it was feared that he would cross the creek, move
round to the east and come upon Woodruff’s battery, still in position
on the ground it had occupied during the day. Col. Gratiot’s regiment
was again ordered to the support of this battery, and here it
remained until ordered into camp by Gen. McCulloch.

The regiment suffered most during the thirty minutes it was

engaged with the Federal infantry (Second Kansas and part of First
Iowa), and Totten’s battery on Bloody Hill, but it stood well and gave
back blow for blow. Capts. Brown and Bell were killed and about
twenty-five other brave men and true met their fate in this battle.

Capt. Woodruff’s battery, the “Pulaski artillery,” was attached to the

Third Arkansas during the battle. This battery did more execution
and service than any other Confederate battery that took part in the
engagement. The damage it inflicted on the enemy was prodigious.
Officers and men behaved with great coolness, courage and
judgment.

The casualties in the Third Arkansas and Woodruff’s battery were:—

killed, twenty-five; wounded, eighty-four; missing, one; total, one
hundred and ten.

FOURTH ARKANSAS INFANTRY.

The Fourth Arkansas infantry on the morning of the battle was

placed under Adjutant-General Rector, who remained in command
during the day. The regiment was not brought into immediate
action, being stationed on the hill for the protection of Reid’s battery,
and, although exposed to the trial of having to submit to a severe
fire from the enemy, which it was unable to return, all the officers
and men behaved with great coolness during the day. There were
none killed or wounded in this regiment. [The colonel of the 83
First Arkansas, J. D. Walker, is at present (1883) one of the
United States Senators from Arkansas.]
 FIFTH ARKANSAS INFANTRY.

This, another regiment of Gen. Pearce’s division, was commanded by

Col. Tom P. Dockery, and for about two hours after the battle
commenced was posted on the height southeast of McCulloch’s
encampment, and occupied a hill east of Wilson’s creek as a guard
for Reid’s battery.

When the Third Louisiana and Third Arkansas moved up against the
Federals on Bloody Hill Col. Dockery sent to their support Capts.
Titsworth’s, Dismuke’s, Neal’s, Dowd’s, Whaling’s and Lawrence’s
companies, all under Lieut. Col. Neal. While gallantly leading his men
Col. Neal fell severely wounded, and Col. Dockery then assumed
command. Only the companies named were actively engaged
against the enemy on Bloody Hill, the companies of Capts. Hartzig,
Arnold, McKeon and Hutchinson having been detailed to serve as
skirmishers at one time after Reid’s battery had changed position.
The Fifth Arkansas did its duty well in this battle, and its conduct
was commented on by Gen. Pearce in the warmest terms. It never
wavered or showed the least sign of demoralization. The loss of the
regiment was three killed and eleven wounded.

FIRST ARKANSAS CAVALRY.

Mention has already been made of the services performed by this

regiment. For a while it supported the Missourians of Price’s division;
then it charged by the flank on Totten’s battery; then charged again
on the position held by the Second Kansas, and all the time during
the engagement was under fire. While thousands of other cavalry
were demoralized and fleeing hither and thither, the First Arkansas
kept on the field and sought more than once to charge as cavalry
over ground almost Alpine in character—rugged, rough, precipitous
and broken. Its commander, Col. DeRosey Carroll, was complimented
more than once for the gallant conduct shown by himself and his
regiment.
The loss of the First Arkansas cavalry was five killed; two 84
mortally wounded; twenty-six severely wounded, and nineteen
missing, as follows:—

Capt. Lewis’ company—two killed, two mortally and five severely

wounded.

Capt. Park’s company—one killed, three wounded, one missing.

Capt. Walker’s company—four wounded, including Capt. Walker

himself, and three missing.

Capt. Withers’ company—two killed, four wounded, two missing.

Capt. Perkins’ company—four wounded, four missing.

Capt. Kelly’s company—one missing.

Capt. Armstrong’s company—one wounded, eight missing.

FEDERAL COMMANDS.

THE FIRST IOWA INFANTRY.

The 1st Iowa Infantry was a three months regiment whose time had
expired several days before the battle, but it had remained on duty
with Lyon to aid him in his emergency. At the time of the fight its
colonel, J. F. Bates, lay sick in Springfield, and Lieut. Col. Wm. H.
Merritt, led the regiment. As has been stated the 1st Iowa was in
Lyon’s column. At the beginning of the fight it was in the reserve,
but advanced when the 1st Kansas gave way and received the first
fire of the enemy while in a state of some confusion, the result of
the retreat of the Kansans through the Iowa’s ranks. The regiment
fired on a body of Confederate cavalry advancing to charge Totten’s
battery, and dispersed it, or drove it back. Soon after the regiment
became engaged generally, and bore its full share of the conflict.
Four companies of the 1st Iowa, and a company of regular infantry
under Capt. Lothrop, supported Totten’s battery at the close of the
engagement and covered the retreat, receiving and returning the
last fire of the enemy. The regiment lost 13 killed (including 85
Capt. A. L. Mason) 138 wounded and 4 missing. Total 155.

FIRST MISSOURI INFANTRY.

The 1st Missouri Infantry was led into the battle of Wilson’s Creek by
its lieutenant colonel, George L. Andrews, its colonel, Frank P. Blair,
being in his seat in Congress at the time. When Gen. Lyon’s column
had reached the immediate proximity of Rains’ division, the regiment
was brought forward to the head of the column and directed to
march parallel with the advance—Gilbert’s regulars—and about 60
yards distant to the right. In a few moments orders were received to
throw one company forward as skirmishers, and Company H, Capt.
Yates, was sent forward, followed by the regiment in column of
companies.

It is claimed that the action was begun by shots from Capt. Yates’
skirmishers. At any rate, soon after they opened fire Company B was
sent up as a reinforcement, and the regiment wheeled into line and
immediately became engaged, at first returning a fire directed
against its left flank. Very soon after Woodruff’s and Guibor’s
batteries opened on the Federal position and their shells fell
uncomfortably plenty among the 1st Missouri. One or two of the
shells which did not explode were examined and pronounced to be
those furnished Sigel’s batteries, leading to the conclusion that Sigel
was firing by mistake against Lyon’s column.

The regiment stood well in line and fought bravely. Capt. Nelson
Cole was severely wounded in the jaw, but remained on the field,
and, though unable to speak, from the nature of his wound, he
continued to encourage his men by signs to stand their ground.
Capt. Cary Gratz, of St. Louis, a native of Lexington, Ky., while
advancing at the head of his company, discovered a body of
Confederates advancing, led on by a mounted officer. Capt. Gratz
fired with his revolver and the Confederate officer fell from his horse,
but rose and rushed toward his lines, when the captain fired again
and the other officer pitched headlong to the ground. Almost
immediately Capt. Gratz fell dead, being pierced by four or five
shots.

Capt. John S. Cavender, with his company, G, was in an 86

advanced position and several times prevented the left flank
from being turned. Col. Andrews, while with the left wing was
severely wounded, but he procured a big drink of whiskey, and soon
returned to his post. In a few minutes his horse was killed and fell
upon him. Going to Dubois’ battery during a lull in the fighting, Col.
Andrews was sent to the rear by Surgeon P. M. Carnyn.

The Confederates now pressed the 1st Missouri so vigorously that

the regiment in all probability would have given way had not Maj.
Schofield and Gen. Lyon opportunely brought up the 1st Iowa and
Maj. Osterhaus (assisted by Lieut. David Murphy, of the 1st Missouri)
come forward with his battalion of the 2d Missouri Infantry. The
regiment then remained on the field and did nobly during the
remainder of the engagement, and when ordered to fall back with
the main column and leave the field it did so in good order.

Out of 27 officers who went into the fight, 13 were either killed or
wounded. Capt. Madison Miller discovered a movement of the
Confederate cavalry to his rear and stopped it by the stout fight he
made with his company, assisted by the artillery. Capt. Cavender,
though severely wounded, refused to leave his post, mounted his
horse and remained until completely exhausted. Surgeon Carnyn, on
more than one occasion, took up a musket and fought in the ranks.
Lieut. David Murphy, although severely wounded in the leg, went to
the rear and assisted in bringing up Osterhaus’ battalion of the 2d
Missouri. Adjt. Hiscock and other officers bore themselves so well
that they received especial mention.
Among the men Corporal Kane, of Company K, when the color
sergeant was killed and nearly all the color guard either killed or
wounded, brought the colors safely off the field. Sergt. Chas. M.
Callahan, Company K, Sergt. Chris. Conrad, of Company G, and
Private Elworthy, of Company F, were noted for their valuable
services and for their coolness and bravery. The part borne by the
1st Missouri may be imagined when it is remembered that its loss
was 76 men killed, 208 wounded and 11 missing.

87
FIRST KANSAS INFANTRY.

On coming upon the battle field in the early morning, the 1st Kansas
Infantry—led by its colonel, Geo. W. Deitzler, and its major, John A.
Halderman—was posted in the rear of the 1st Missouri and 1st Iowa.
Very soon Gen. Rains’ skirmishers or outposts were driven in,
Totten’s battery took position and opened fire, while the 1st Missouri
was sent up and soon became engaged.

At this time, under an order from Gen. Lyon, the 1st Kansas moved
to the front in “double quick,” while the right wing and one company
from the left, under command respectively of Capts. Chenoweth,
Walker, Swift, Zesch, McFarland, and Lieut. McGonigle—all under Col.
Deitzler—advanced to a position beyond that occupied by the 1st
Missouri, and here, forming in the very face of the enemy, engaged
the Confederates and held their ground steadfastly under an
uninterrupted and murderous fire of artillery and infantry.

The four remaining companies of Capts. Clayton, Roberts, Stockton,

and Lieut. Angell, under Maj. Halderman, having been posted on the
right of Totten’s battery as support, where they suffered severely
from a constant fire, were now brought up by Maj. Halderman, who
called out, “Forward boys, for Kansas and the old flag!” Aligning with
remarkable coolness upon the remnant of the six right companies
the four left companies settled down to work. With but slight and
immaterial changes of position the 1st Kansas occupied this ground
for over two hours, holding its ground and dealing and receiving
severe punishment.

While thus engaged, Capt. Chenoweth, Capt. Clayton and a portion

of Capt. McFarland’s company under Lieut. Malone, were ordered to
charge the enemy with their commands, which order they executed
and drove back the Confederates a considerable distance, although
soon after they were themselves compelled to retire. While leading
this charge Col. Deitzler had his horse killed under him, and was
himself severely wounded. The command then devolved on Maj.
Halderman. The regiment now had a very exposed position, lying in
plain view, obliquely across a ridge, but, though it suffered 88
severely, it bore itself well.

When the 2d Kansas fell back the 1st formed on its left, three
companies remaining on the brow of the hill, and on the right of the
battery. After the severe volley firing had ceased for a few minutes—
the Confederates having retired—it was recommenced by them
again as they advanced, and kept up for nearly a quarter of an hour,
when the whole Federal line, apparently, opened on them and they
again retired down the hill. After this Maj. Sturgis ordered the
retreat.

With scarcely any material change of its position the 1st Kansas
stood under fire and returned it, maintained every ground assigned
it, without turning its back on the foe, for the five long hours during
which the battle raged. Its loss was the heaviest in killed of the
Federal regiments engaged—77, one more than the 1st Missouri. It
had 187 men wounded and 20 missing; total, 284. It went into
action with nearly 800 men, and left the field in good order with
about 500.

THE SECOND KANSAS INFANTRY.

This regiment, as stated in the general description of the battle, was

at first stationed in reserve on a hill on the right of and overlooking
Ray’s cornfield, where Plummer’s battalion fought. After Plummer
had been driven back and the pursuing Confederates checked by
Dubois’ battery, Lieut-Col. Chas. W. Blair, of the 2d Kansas, rode to
Gen. Lyon and requested that the regiment be given a place in the
front. Gen. Lyon gave the order and Col. Mitchell brought the
regiment forward, in time to take part in the last grand charge. Prior
to this, and early in the action, before the regiment, as a regiment,
was fairly under fire, a force of Missouri cavalry (presumably of
Rains’ division) attempted a flank movement, and Maj. W. F. Cloud,
of the 2d, taking Capt. McClure’s company and deploying it, drove
them back after a few volleys.

As the regiment went up to the forefront, Gen. Lyon put himself at

its head and assisted the field officers in bringing it into action. Just
as the regiment raised the crest of the hill, and while it was still in
column, a terrific fire was opened on it, and it was under this 89
fire that Gen. Lyon fell dead and Col. Mitchell was severely
wounded. Gen. Lyon was leading the 2d when he was killed. After
Col. Mitchell was wounded, command of the regiment was assumed
by Lieut-Col. Blair and Maj. Cloud, who threw the men into line, and
after a hard fight of fifteen minutes the Confederates were driven
down the hill, and a lull in the conflict resulted.

About this time Capt. Powell Clayton’s company, of the 1st Kansas,
was attached to the left of the 2d, and the companies of Capts.
Roberts, Walker and Zesch, also of the 1st, were formed on the
right. On the right of this position a ravine stretched down to the
enemy, and up this ravine the Confederates (of John B. Clark’s
division) attempted to flank Col. Blair. Some men sent down it from
Capt. Cracklin’s company did not return, and then Col. Blair himself
rode out to see what was the matter. He had not gone twenty yards
when he “found what darkened de hole!” A sharp fire was opened
on him and his horse killed, but the colonel himself was unhurt, and
mounting a horse brought him by his orderly, Alex. H. Lamb, he was
soon again directing the movements of his men.
Meantime, aware of the danger in front, Maj. Cloud had gone back
to Sturgis for reinforcements and obtained two guns of Totten’s
battery, under Lieut. Sokalski. These came up in good time. As they
stopped, Capt. Chenoweth, of the 1st Kansas, rode out to the head
of the ravine, and saw the Confederates coming up it in considerable
numbers. Cloud and Sokalski got the guns in position and opened on
the ravine. As the Confederates approached nearer Col. Blair ordered
the men to lie down and load and fire in that position and to be
careful of their ammunition. Here the men received a most terrific
fire, which they seemed to relish. Artillery and musketry were
playing on them, but the shot and shell went too high and only the
grape, the muskets, and the rifles of the enemy did execution. Yet
not a man broke ranks or left his place in the line. At last the
Confederates fell back or slackened their fire and the artillery
limbered up and retired to the rear to join in the general retreat,
which bad been ordered some minutes before.

Maj. Sturgis, on assuming command after Lyon had fallen, 90

sent Col. Blair word to retreat as soon as he could do so with
safety, and after the Confederates had fallen back the last time he
did so. The men were brought off in good order and in slow time,
without a panic or confusion. After crossing the first ravine in the
rear the line was reformed and marched by the right flank to the
main command and off the battle field.

The loss of the 2d Kansas in the battle was 5 killed, 59 wounded,

and 6 missing—total, 70. Both officers and men behaved splendidly.
When Col. Mitchell fell he turned over the command to Col. Blair,
saying: “Colonel, take the regiment and maintain the honor of
Kansas.” As he was being carried from the field he called out to
Gordon Granger, of Sturgis’ staff, “For God’s sake, support my
regiment.” Of Lieut. Col. Blair, it was said by Gen. Sturgis that “he
attracted the attention of all who saw him.” Col. Mitchell, Lieut. Col.
Blair and Maj. Cloud were all highly complimented by Gen. Sturgis
and Gen. Fremont and recommended for promotion. Maj. Cloud,
Adjutant Lines and Capt. Ayres were mentioned in Col. Blair’s report
as conspicuous for their gallantry.

TOTTEN’S BATTERY.

The share of fighting done by this organization (Light Company F, 2d

Artillery), in the battle of Wilson’s Creek was large and important.
Soon after the skirmishers of Lyon’s advance fired on the Southern
pickets, the line of march, as directed by Gen. Lyon in person, lay
through a small valley which debouched into that through which
Wilson’s Creek runs at the point immediately occupied by the front of
Price’s troops and just where a road to Springfield entered the valley,
keeping along the foot of the hills, and soon after the battery
opened. The left section, under Lieut. Sokalski, was first brought to
bear upon the enemy in the woods in front, and shortly afterward
the other four pieces were thrown forward into battery to the right
on higher ground. A few rounds from the artillery assisted the
infantry in driving the secession troops back toward the crests of the
hills, nearer and immediately over their own camp.

Capt. Totten now conducted the battery up the hill to the left and
front, and soon found a position where he brought it into 91
battery directly over the northern position of the enemy’s
camp. The camp of Gen. Rains’ division lay directly beneath the front
and to the left of, though very close to, the position of the battery,
while a battery of the secessionists (Woodruff’s Arkansas) was in
front and within easy range. Of course Rains’ camp was entirely
deserted, and therefore Totten’s first efforts were directed against
the Arkansas battery in his front and right. The right half of Totten’s
guns were principally directed against Woodruff, although the entire
six pieces as opportunity offered, played upon him. The two
batteries pounded away on each other for some time, neither
seeming to get much the advantage of the other. As the position of
the Arkansas battery was somewhat masked by the timber, Totten’s
gunners were obliged to give direction to their pieces by the flash
and smoke of the opposing artillery.
In the meantime, while this fight between Totten and Woodruff was
in progress, the battle was raging in the thick woods and underbrush
to the front and right of the position of the Federal battery, and the
1st Missouri was being hard pressed.

Gen. Lyon ordered Totten to move a section of his battery forward to

the support of the Missourians, which was done, the guns coming up
on a run and unlimbering in front of the right company of the
regiment. A Confederate regiment with a Confederate flag, which at
that distance seemed to be the stars and stripes, was two hundred
yards away, and fearing they might be friends Totten hesitated
before opening. Their fire soon undeceived him and he turned loose
his guns upon them with canister from both pieces.

The next important step in the progress of the battle was when a
portion of Clark’s (?) division tried to force its way up the
neighborhood road passing along toward Springfield in order to turn
the Federal right. For a time the situation was critical, for the
Missourians were plucky and were fighting hard. Four pieces of
Totten’s battery were still in position commanding that point, and
Dubois’ four guns were on the left also near the road and
commanding it. As Slack’s men came in good view and range, both
artillery and infantry opened on them and drove them back.

Just after this had been done Gen. Lyon came up to the 92
battery and complimented the men who were working it. Capt.
Totten saw blood trickling from Lyon’s heel and the general said he
had been wounded in the leg, but not seriously. The captain offered
his commander some brandy in a canteen, but the general refused
it, and rode away, and that was the last time Totten ever saw Lyon
alive. Soon after leaving Totten, Lyon sent him word to support the
Kansas men on the extreme right, who were being hard pressed.
Lieut. Sokalski took up his section immediately and saved the
Kansans from being overthrown and driven back.
After this came an attempted charge on the Federal position by
some Missouri, Texas and Arkansas cavalry, of which there was a
great abundance. Some 800 of them, including a battalion or so of
Greer’s Texans and Carroll’s Arkansans, fresh from the southern end
of the little valley where Sigel had been so easily whipped, formed at
the foot of the hill on which four of Totten’s guns stood and were
getting ready for a charge, when the artillery and the infantry
opened on them and they were driven away so rapidly that they
were out of sight in a moment.

The last point where the battery was engaged was on the right of
the left wing of the 1st Iowa and somewhat in front. Lyon was then
killed and Sturgis was preparing to retreat. Totten’s battery was still
in action, two pieces in advance on the right so hot that the water
thrown on them almost hissed, and yet pounding away. The left
wing of the 1st Iowa came up and supported the guns from the
field, and they were brought away off the field and to Springfield,
without the loss of a sponge-stick. The battery lost 4 killed and 7
wounded; no prisoners.

DUBOIS’ BATTERY.

This battery so named consisted of four pieces of light artillery, three

six-pounders, and one twelve-pounder, commanded by second
Lieutenant John V. Dubois, of the U. S. Mounted Rifles, a semi-
cavalry regiment. Lieut. Dubois had been detailed from his regiment
to command this battery, newly organized and manned by recruits.

Entering the fight Dubois selected his own position, some 80 93

yards to the left and rear of Totten’s battery, where his men
were partially and his horses entirely protected from the enemy’s
fire. He assisted Totten in clearing the ground of the enemy at the
start, and under direction of Capt. Gordon Granger (afterward a
major general), opened on McCulloch’s men down in Ray’s cornfield,
who had just driven back Plummer, and drove them in disorder,
Capt. Granger directing one of the guns. The Confederates rallied
behind a house (J. A. Ray’s), on the right of their line. Dubois struck
this house twice with a twelve-pound shot, when a hospital flag was
displayed and he ceased firing. Using small charges of powder,
Dubois’ guns now shelled the thicket in the ravine, a short distance
in front, and forced some of Price’s Missourians to retire.

Bledsoe’s battery now opened on Dubois from the crest of the hill
opposite and “Old Hi’s” fire did great execution. “Old Sacramento,”
as Bledsoe’s twelve-pounder was called, got in her work very
disadvantageously to the Federals. The shots passing over Dubois’
gun fell among the Federal wounded that had been carried to the
rear and did considerable execution. Dubois could not entirely
silence Capt. Bledsoe’s guns, but he made it very uncomfortable for
him. One shot from Dubois’ gun, killed two of Bledsoe’s battery
horses, tore off one arm and all of the hand of the other arm of the
man who held them, Judge James Young, now of Lexington, and
killed another man far in the rear. While engaged with Bledsoe and
his Lafayette county battery, Lieut. Dubois assisted in driving back
the cavalry that formed to charge on Totten.

During the last charge of Price and McCulloch on Sturgis, two of

Dubois’ guns were limbered up to be sent to Totten, but before a
road could be opened through the brush and through the wounded,
orders came for Dubois to fall back to a hill in the rear and protect a
retreat. This he did, remaining until all the troops had passed when
he turned and marched toward the rear. Shortly after starting back
the twelve-pounder broke down. While it was being repaired Maj.
Osterhaus’s two companies remained with it to protect it, and then
followed in its rear until the main portion of the command on the
prairie was reached. Here the battery joined Steele’s battalion and
formed the rear guard the rest of the way into Springfield, 94
neither firing or receiving a shot on the way and not being
molested in anywise—never seeing an enemy. Being well protected
during the entire engagement, the loss in this battery was very slight
—none killed and only two severely wounded. Several of the men
and Dubois himself received slight wounds.
 STEELE’S BATTALION.

The battalion of regulars commanded by Capt. Fred Steele was

composed of two companies of the 2d regular infantry—company B,
commanded by Sergt. Griffin, and company C, commanded by Sergt.
McLaughlin, a company of “general service” recruits under Lieut. M.
L. Lothrop, and a company of mounted rifle recruits commanded by
Lance-Sergeant Morine. It will be seen that Capt. Steele had but one
commissioned officer under him.

During the early part of the action the battalion was in position to
support Dubois’ battery, but had no opportunity of engaging the
enemy except to assist in dispersing a body of cavalry that
threatened the rear. Soon after the fall of Gen. Lyon, Capt. Gilbert’s
company joined the battalion and Maj. Sturgis ordered Capt. Steele
to form in line of battle and advance against the enemy’s front.
Heavy firing on both sides followed, without any apparent
permanent advantage to either, until the suspension of hostilities
mentioned before. During this suspension Lieut. Lothrop took his
company forward as skirmishers, but they were driven back in very
short order and without much ceremony.

A Confederate field piece (probably one of Guibor’s), was run up

under the hill and threw grape and occasionally a shell over Steele’s
battalion, but with no serious effect, as the shots passed too high.
Two other pieces were added and worked vigorously but not
carefully, and with no other effect than to cause Steele’s men to lie
close to the ground.

In the last grand charge on Totten’s battery and the main Federal
position, Steele’s battalion did good work, the men firing away nearly
all their cartridges. Just before the retreat began Capt. Gordon, with
his hastily collected detachment from different regiments, and 95
Capt. Steele repulsed another attack, and enabled Totten’s
battery and other commands to retire in good order. On the retreat
to Springfield after reaching the prairie Capt. Steele commanded the
rear guard, and states that he was not molested at all, “never seeing
an enemy.” The loss of Steele’s battalion was 15 killed, 44 wounded,
and two prisoners. Sergt. Morine, commanding the rifle recruits, was
killed on the field.

PLUMMER’S BATTALION.

This battalion performed brave service at the battle of Wilson’s

Creek. It belonged to the 1st U. S. Regular Infantry, and most of the
men had been some time in the service. Frequent reference has
already been made to the part it performed in this battle. Gilbert’s
company had the advance upon reaching the battlefield and was the
first thrown forward on the skirmish line. The principal portion of the
battalion, commanded by Capt. J. N. Plummer himself, made the
fight in Ray’s cornfield against the 3d Louisiana and 2d Arkansas
Mounted Rifles, and afterward engaged in the terrible conflict on and
along Bloody Hill.

The battalion was remarked as much for its coolness as for its
bravery. Upon the retreat of the Federals to Springfield it entered the
town in perfect order, the flag flying, the drums beating, and the
men keeping perfect step as if they were on parade or drill, and as
collected and unexcited as if nothing of consequence had taken
place that day.

The battalion was composed of Cos. B, C and D, of the 1st Regulars,

commanded by, respectively, Capts. Gilbert, Plummer, and Huston,
and Lieut. Wood’s squad of Rifle Recruits. Capts. Plummer and
Gilbert were severely wounded, and Capt. D. Huston then took
command. Out of 230 men engaged, the battalion lost 19 killed, 52
wounded and 9 missing—a total of 80, or a little more than one-third
of the entire number in the fight.

THE HOME GUARDS.

Two companies of mounted Union home guards—one called the
Dade county squadron, commanded by Capt. Clark W. Wright, 96
and the other under Capt. T. A. Switzler—were present at the
battle, but took no very important part. They made some charges on
scattered squads of secessionists, driving them under cover and out
of all danger to the Federal line, but for the greater portion of the
time they were stationed to the right and rear of Lyon’s position as a
post of observation and to prevent the line from being flanked by the
enemy’s cavalry. In one of the charges Capts. Wright and Switzler
ascended a hill in plain view of Gen. Rains’ camp, and counted a
number of tents.

DISPOSITION OF THE BODY OF GEN. LYON.

Ah, Sir Launcelot. Thou there liest that never wert matched of
earthly hands. Thou wert the fairest person and the goodliest of
any that rode in the press of knights. Thou wert the truest to thy
sworn brother of any that buckled on the spur; * * * and thou
wert the sternest knight to thy mortal foe that ever laid spear in
rest.

For the purpose of ascertaining the truth concerning the death and
burial of the body of Gen. Lyon, the writer hereof caused certain
newspaper publications to be made in the St. Louis Republican and
other journals, making inquiries pertinent to the case. Many and
varied were the replies, some of which, perhaps, ought to be given,
as illustrating the different lights in which men see the same object,
and the morbid desire for notoriety on the part of others, which
leads them to lie like book-agents, in order that their names may be
published in connection with some notable event. No less than ten
newspaper articles were published and thirty-two written
communications were received by the compiler relating to the death
and burial of Gen. Lyon. The result was forty-two different versions
thereof.
The work of ascertaining the truth was thereby complicated instead
of being facilitated. A dozen or more claimants for the distinction of
having first discovered the body on the battlefield appeared. Half a
score bore the corpse to Gen. Price’s tent. Twenty saw the body,
noted its appearance carefully, etc. Knowing from incontrovertible
proof how the general was dressed when he was killed, the writer
inserted a test question asking that his garb be described. Two ex-
officers, one Union, the other Confederate, answered that he was “in
full general’s uniform.” A minister of the gospel, who was also 97
the “first to discover the body,” promptly replied that he was
“dressed in a complete suit of black broadcloth, white shirt, gold
studs, fine boots and kid gloves!” The majority of the answers,
however, were to the same effect, that he was dressed in his old
fatigue uniform of his former rank—that of captain in the regular
army—without epaulettes or shoulder-straps. After much labored
investigation the writer has ascertained the following facts, which he
can easily substantiate:—

Gen. Lyon was killed while placing the 2d Kansas Infantry in position,
by a rifle or navy revolver ball through the region of the heart. He
was borne to the rear by Lieut. Schreyer, of Capt. Tholen’s company,
2d Kansas, two other members of the same regiment, and Ed.
Lehman, of Co. B, 1st U. S. Cavalry, the latter the soldier who caught
the general’s body as it fell from the horse. As the body was borne
to the rear, Lieut. Wm. Wherry, one of the general’s aides, had the
face covered, and ordered Lehman, who was crying like a child, to
“stop his noise,” and tried in other ways to suppress the news that
the general had been killed. The body was placed in the shade of a
small black-jack, the face covered with half of a soldier’s blanket, the
limbs composed, and in a few minutes there were present Surgeon
F. M. Cornyn, Maj. Sturgis, Maj. Schofield, Gen. Sweeney, and
Gordon Granger, and perhaps other officers. Cornyn examined the
body, and from the side of the face wiped the blood made by the
wound in the head. He also opened the vest and split the general’s
shirts, which were soaked with blood, and examined the wound,
which was found to be in the heart, the aorta having been pierced.
The minister’s story, which he has the effrontery to give a
newspaper publication, relates that two or three hours afterward the
body was neatly dressed, with its smooth white shirt and studs, kid
gloves, etc.! Those best acquainted with the personal habits of Gen.
Lyon say he never wore a pair of kid gloves during his term of
service.

Maj. Sturgis ordered the body to be carried back to a place selected

as a sort of field hospital and there to be placed in an ambulance
and taken to Springfield. While the body was here lying a few
Federal officers examined it and one of them reports that the face
had again become bloody, from the wound in the head, and 98
that the shirt front was gory from the death wound. About
twenty minutes after the body had been brought back, Lieut. David
Murphy, of the 1st Missouri, who was already badly wounded in the
leg, and Lehman placed the body in an army wagon, being used as
an ambulance, and belonging to Co. B, 1st U. S. Cavalry. This wagon
was about to start to Springfield, and contained already some
wounded men. A few minutes later a sergeant of the regular army
came up and ordered the body taken out, saying, “There will be an
ambulance here in a minute for it.” The corpse was then carried
beneath the shade tree where it had before reposed.

The Federal army now retreated, and the ambulance ordered never
came up. Before the Confederates came on to the ground where the
body lay, which location was 200 yards northeast of “Bloody Hill,”
half a dozen slightly wounded Federal soldiers had gathered about
the dead hero, and an hour after the Federal retreat a party of
Arkansas skirmishers came upon them and discovering the occasion
of the crowd instantly spread the news that Gen. Lyon had been
killed. Immediately there was a great tumult and the report was
borne to Price and McCulloch by half a dozen. Many were
incredulous and did not believe that a body so plainly dressed—in an
old, faded captain’s uniform, with but three U. S. buttons on the coat
and a blue (or red) cord down the legs of the trousers to indicate
that he was in the military service—was that of Gen. Lyon.
The body was then placed in a small covered wagon, used as an
ambulance, to be conveyed to Gen. McCulloch’s headquarters (not
Gen. Price’s) when an order arrived that it should be taken to Price’s
and delivered to Dr. S. H. Melcher, of the 5th Missouri, who, as
before stated, had come upon the field in company with Dr. Smith,
Gen. Rains’ division surgeon. Dr. Melcher had been informed by Col.
Emmett McDonald that Lyon had been killed, and at once asked for
his body. When the little covered wagon containing the corpse had
driven up and Gen. Price and Gen. Rains and other officers had
viewed the body, it was turned over to Dr. Melcher. A number of
Southern soldiers standing by drew knives and made attempts to cut
off some buttons or pieces of the uniform as relics, and one or two
expressed a wish to “cut his d——d heart out;” but Gen. Rains drew
his sword (or revolver) and swore he would kill the first man 99
that touched the corpse, and Emmett McDonald denounced
the ruffianly would-be violators in the harshest terms—and
McDonald could be harsh when he wanted to be!

Beside the body of Gen. Lyon was a wounded man, who was now
taken out, and then Gen. Rains himself and some of his cavalry
escorted the wagon to the house of Mr. Ray, on or near the
battlefield. It is proper now to give the testimony of Dr. Melcher
himself, as given to the writer and furnished the press for
publication. Speaking of the courtesy of Gen. Rains in escorting the
body to Ray’s house, Dr. Melcher goes on to say:

Arriving there the body was carried into the house and placed on
a bed; then I carefully washed his face and hands, which were
much discolored by dust and blood, and examined for wounds.
There was a wound on the right side of the head, another in the
right leg below the knee, and another, which caused his death,
was by a small rifle ball, which entered about the fourth rib on the
left side, passing entirely through the body, making its exit from
the right side, evidently passing through both lungs and heart.
From the character of this wound it is my opinion that Gen. Lyon
was holding the bridle rein in his left hand, and had turned in the
saddle to give a command, or words of encouragement, thus
exposing his left side to the fire of the enemy.

At this time he had on a dark blue, single breasted captain’s coat,

with the buttons used by the regular army of the United States. It
was the same uniform coat I had frequently seen him wear in the
arsenal at St. Louis, and was considerably worn and faded. He had
no shoulder-straps; his pants were dark blue; the wide-brim felt
hat he had worn during the campaign was not with him. After
arranging the body as well as circumstances permitted, it was
carried to the wagon and covered with a spread or sheet furnished
me by Mrs. Ray.

When I was ready to start Gen. Rains said: “I will not order any to
go with you, but volunteers may go;” and five Confederate
soldiers offered their service of escort. One drove the team; the
others, being mounted, rode with me in rear of wagon. The only
name I can give is that of Orderly Sergt. Brackett of a company in
Churchill’s Arkansas regiment. Another of the escort was a German
who in 1863 was clerking in Springfield, and during the defence of
Springfield against the attack of Marmaduke, January 8, 1863, did
service in the citizens’ company of 42 men which was attached to
my “Quinine Brigade” from the hospitals.

The following is a copy of a paper written at Mr. Ray’s 100

[14]
house. The original I now have:—

Gen. James S. Rains, commanding Missouri State Guards,

having learned that Gen. Lyon, commanding United States
forces during action near Springfield, Mo., Aug. 10, 1861, had
fallen, kindly afforded military escort and transportation subject
to my order. I have also his assurance that all the wounded shall
be well taken care of and may be removed under the hospital
flag, and that the dead shall be buried as rapidly as possible.

[Signed]
 S. H. Melcher,
Asst. Surg. 5th Reg. Mo. Vols.

Wilson Creek, Aug. 10, 1861.

The above fully approved and indorsed.

[Signed]
James S. Rains,
Brig.-Gen. 8th M. D., M. S. G.

About half way to Springfield I saw a party under flag of truce

going toward the battlefield. Arriving at Springfield, the first officer
I reported to was the ever faithful Col. Nelson Cole, then captain
of company E, 1st Missouri Volunteer Infantry, who, with what
remained of his gallant company, was guarding the outposts. I
passed on to the camps of Gen. James Totten and T. W. Sweeney.
Here Gen. Totten relieved my escort and sent them back to their
command, a new driver was furnished, and I delivered the body of
Gen. Lyon to Maj. J. M. Schofield, 1st Missouri Volunteer Infantry—
now Maj.-Gen. Schofield, U. S. A.—at the house that had been
used previous to the battle by Gen. Lyon for his headquarters.

It is proper to state that Dr. Melcher’s testimony is corroborated in

part by two survivors of the 1st Arkansas, and by Mrs. Livonia
Green, now of Lane county, Oregon, and also by Mrs. Jerome
Yarbrough, of this county, both of the latter being daughters of the
Mr. and Mrs. Ray mentioned. (Mr. and Mrs. Ray are now dead.)

After Sturgis’ army had gotten well on the road to Springfield, it was
discovered that Gen. Lyon’s body had been left behind. Sturgis
immediately started back a flag of truce party under Lieut. Canfield,
of the regular army, with orders to go to Gens. Price and McCulloch,
and, if possible, procure the remains and bring them on to
Springfield. Lieut. Canfield and party went to the battle field, saw
Gen. McCulloch, obtained his order for the body (the general
remarking that he wished he had a thousand other dead Yankee
bodies to send off) and there ascertained that the body had 101
already started for the Federal lines.

When the corpse was deposited in the former headquarters of the

general, on the north side of College street, west of Main, in
Springfield, word was sent to Sturgis. He and Schofield and other
officers held a consultation, and decided that the body should be
taken with the army to Rolla, if possible. There not being a metallic
coffin in the place, it was determined to embalm it, or preserve it by
some artificial process. Accordingly, the chief surgeon, Dr. E. C.
Franklin, was sent for. Responding to the inquiries of the writer, Dr.
Franklin says:—

About ten o’clock p. m., on the night when it arrived at head

quarters, I was summoned there and then first saw the body of
Gen. Lyon lying upon a table, covered with a white spread, in a
room adjoining the one where two or three of the Union officers
were seated. Gens. Schofield, Sturgis, and others consulted me as
to the possibility of injecting the body with such materials that
would prevent decay during its transit to St. Louis. I prepared the
fluid for injection into the body, but discovered that instead of
being retained within the vessels it passed out into the cavity of
the chest. This led me to suspect a laceration either of one of the
large arteries near the heart, or, possibly, a wound of the heart
itself. This hypothesis, coupled with the fact that there was an
external wound in the region of the heart, confirmed my opinion
of the utter uselessness of attempting the preservation of the
body during its passage to St. Louis. These facts I reported to the
commanding officer, who then gave me verbal orders to attend to
the disposal of the body in the best possible manner. At this time
preparations were being made and the orders given for the troops
to retreat and fall back upon Rolla, some fifty or more miles nearer
St. Louis. Returning to the general hospital, of which I was in
charge, I detailed a squad of nurses to watch by the body of Gen.
Lyon till morning, which order was faithfully carried out. I then
 disposed of my time for the best interests of the wounded and
sick under my charge.

Dr. Franklin was furnished with money and directed to have the
general’s remains well cared for, and he ordered an undertaker, Mr.
Presley Beal, to make a good, substantial coffin at once. Early the
following morning, in some way, word was sent to Mrs. Mary Phelps,
wife of Hon. John S. Phelps, that the body of the great Union leader
was lying stiff, and bloody, and neglected in the temporary 102
charnel house on College street. Soon she and the wife of Mr.
Beal were by his side, and watching him. Not long thereafter came
the wife of Col. Marcus Boyd and her two daughters (one of whom,
now Mrs. Lula Kennedy, still resides in Springfield), and kept them
company. And so it was that women, “last at the cross and first at
the tomb,” were those who kept vigil over the corse of the dead
warrior, who, although he died the earliest, was one of the greatest
Union generals the war produced.

The body had now lain about twenty-four hours in very hot weather.
It was changing fast, and its condition made it necessary that it
should be buried as soon as possible. Mrs. Phelps left Mrs. Boyd and
her daughters and went to see about the coffin. Dr. Franklin came in
and sprinkled the corpse with bay rum and alcohol. Mr. Beal brought
the coffin, and soon a wagon—a butcher’s wagon—was on its way to
Col. Phelps’ farm with all that was mortal of the dead hero, and with
no escort save the driver, Mrs. Phelps, Mr. Beal and one or two
soldiers.

Col. Emmett McDonald, than whom the war produced no more

knightly a soldier, had been made a prisoner by Gen. Lyon, at the
capture of Camp Jackson. When Lyon was killed, Col. McDonald not
only assisted Dr. Melcher in recovering the body, but Dr. Franklin
says of him:—

Here let me do justice to Col. Emmett McDonald, who called upon

me at the general hospital and after some conversation in regard