100% found this document useful (3 votes)
18 views

Instant Download (Ebook) Beginning Anomaly Detection Using Python-Based Deep Learning: Implement Anomaly Detection Applications with Keras and PyTorch by Suman Kalyan Adari and Sridhar Alla ISBN 9798868800078, 8868800071 PDF All Chapters

The document promotes the ebook 'Beginning Anomaly Detection Using Python-Based Deep Learning' by Suman Kalyan Adari and Sridhar Alla, available for download at ebooknice.com. It includes various editions and related titles on anomaly detection and machine learning, providing links for access. The ebook covers foundational concepts in anomaly detection, data science, and machine learning techniques using Python frameworks like Keras and PyTorch.

Uploaded by

dnadieanig
Copyright
© © All Rights Reserved
Available Formats
Download as PDF, TXT or read online on Scribd
100% found this document useful (3 votes)
18 views

Instant Download (Ebook) Beginning Anomaly Detection Using Python-Based Deep Learning: Implement Anomaly Detection Applications with Keras and PyTorch by Suman Kalyan Adari and Sridhar Alla ISBN 9798868800078, 8868800071 PDF All Chapters

The document promotes the ebook 'Beginning Anomaly Detection Using Python-Based Deep Learning' by Suman Kalyan Adari and Sridhar Alla, available for download at ebooknice.com. It includes various editions and related titles on anomaly detection and machine learning, providing links for access. The ebook covers foundational concepts in anomaly detection, data science, and machine learning techniques using Python frameworks like Keras and PyTorch.

Uploaded by

dnadieanig
Copyright
© © All Rights Reserved
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 81

Visit https://ebooknice.

com to download the full version and


explore more ebooks

(Ebook) Beginning Anomaly Detection Using Python-


Based Deep Learning: Implement Anomaly Detection
Applications with Keras and PyTorch by Suman Kalyan
Adari and Sridhar Alla ISBN 9798868800078,
_____ Click the link below to download _____
8868800071
https://ebooknice.com/product/beginning-anomaly-
detection-using-python-based-deep-learning-implement-
anomaly-detection-applications-with-keras-and-
pytorch-58356254

Explore and download more ebooks at ebooknice.com


Here are some recommended products that might interest you.
You can download now and explore!

(Ebook) Beginning Anomaly Detection Using Python-Based Deep Learning,


2nd Edition by Suman Kalyan Adari , Sridhar Alla ISBN 9798868800078,
8868800071

https://ebooknice.com/product/beginning-anomaly-detection-using-
python-based-deep-learning-2nd-edition-54693456

ebooknice.com

(Ebook) Beginning Anomaly Detection Using Python-Based Deep Learning:


With Keras and PyTorch by Sridhar Alla, Suman Kalyan Adari ISBN
9781484251768, 9781484251775, 1484251768, 1484251776

https://ebooknice.com/product/beginning-anomaly-detection-using-
python-based-deep-learning-with-keras-and-pytorch-10795944

ebooknice.com

(Ebook) Beginning Anomaly Detection Using Python-Based Deep Learning


by Suman Kalyan Adari and Sridhar Alla

https://ebooknice.com/product/beginning-anomaly-detection-using-
python-based-deep-learning-54818762

ebooknice.com

(Ebook) Beginning MLOps with MLFlow : Deploy Models in AWS SageMaker,


Google Cloud, and Microsoft Azure by Sridhar Alla; Suman Kalyan Adari
ISBN 9781484265499, 1484265491

https://ebooknice.com/product/beginning-mlops-with-mlflow-deploy-
models-in-aws-sagemaker-google-cloud-and-microsoft-azure-21976300

ebooknice.com
(Ebook) Beginning MLOps with MLFlow: Deploy Models in AWS SageMaker,
Google Cloud, and Microsoft Azure by Sridhar Alla, Suman Kalyan Adari
ISBN 9781484265482, 9781484265499, 1484265483, 1484265491

https://ebooknice.com/product/beginning-mlops-with-mlflow-deploy-
models-in-aws-sagemaker-google-cloud-and-microsoft-azure-22417650

ebooknice.com

(Ebook) Finding Ghosts in Your Data: Anomaly Detection Techniques with


Examples in Python by Kevin Feasel ISBN 9781484288696, 1484288696

https://ebooknice.com/product/finding-ghosts-in-your-data-anomaly-
detection-techniques-with-examples-in-python-47213874

ebooknice.com

(Ebook) Finding Ghosts in Your Data: Anomaly Detection Techniques with


Examples in Python by Kevin Feasel ISBN 9781484288702, 148428870X

https://ebooknice.com/product/finding-ghosts-in-your-data-anomaly-
detection-techniques-with-examples-in-python-47435122

ebooknice.com

(Ebook) Deep Learning with Python: Develop Deep Learning Models on


Theano and TensorFLow Using Keras by Jason Brownlee

https://ebooknice.com/product/deep-learning-with-python-develop-deep-
learning-models-on-theano-and-tensorflow-using-keras-7295338

ebooknice.com

(Ebook) Control Charts and Machine Learning for Anomaly Detection in


Manufacturing by Kim Phuc Tran ISBN 9783030838188, 3030838188

https://ebooknice.com/product/control-charts-and-machine-learning-for-
anomaly-detection-in-manufacturing-34603960

ebooknice.com
Beginning Anomaly
Detection Using
Python-Based Deep
Learning
Implement Anomaly Detection
Applications with Keras and PyTorch
Second Edition

Suman Kalyan Adari


Sridhar Alla
Beginning Anomaly Detection Using Python-Based Deep Learning: Implement
Anomaly Detection Applications with Keras and PyTorch, Second Edition
Suman Kalyan Adari Sridhar Alla
Tampa, FL, USA Delran, NJ, USA

ISBN-13 (pbk): 979-8-8688-0007-8 ISBN-13 (electronic): 979-8-8688-0008-5


https://doi.org/10.1007/979-8-8688-0008-5

Copyright © 2024 by Suman Kalyan Adari, Sridhar Alla


This work is subject to copyright. All rights are reserved by the Publisher, whether the whole or part of the
material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation,
broadcasting, reproduction on microfilms or in any other physical way, and transmission or information
storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now
known or hereafter developed.
Trademarked names, logos, and images may appear in this book. Rather than use a trademark symbol with
every occurrence of a trademarked name, logo, or image we use the names, logos, and images only in an
editorial fashion and to the benefit of the trademark owner, with no intention of infringement of the
trademark.
The use in this publication of trade names, trademarks, service marks, and similar terms, even if they are not
identified as such, is not to be taken as an expression of opinion as to whether or not they are subject to
proprietary rights.
While the advice and information in this book are believed to be true and accurate at the date of publication,
neither the authors nor the editors nor the publisher can accept any legal responsibility for any errors or
omissions that may be made. The publisher makes no warranty, express or implied, with respect to the
material contained herein.
Managing Director, Apress Media LLC: Welmoed Spahr
Acquisitions Editor: Celestin Suresh John
Development Editor: James Markham
Coordinating Editor: Gryffin Winkler
Cover designed by eStudioCalamar
Cover image by Tony Litvyak@Unsplash.com
Distributed to the book trade worldwide by Apress Media, LLC, 1 New York Plaza, New York, NY 10004,
U.S.A. Phone 1-800-SPRINGER, fax (201) 348-4505, e-mail orders-ny@springer-sbm.com, or visit
www.springeronline.com. Apress Media, LLC is a California LLC and the sole member (owner) is Springer
Science + Business Media Finance Inc (SSBM Finance Inc). SSBM Finance Inc is a Delaware corporation.
For information on translations, please e-mail booktranslations@springernature.com; for reprint,
paperback, or audio rights, please e-mail bookpermissions@springernature.com.
Apress titles may be purchased in bulk for academic, corporate, or promotional use. eBook versions and
licenses are also available for most titles. For more information, reference our Print and eBook Bulk Sales
web page at http://www.apress.com/bulk-sales.
Any source code or other supplementary material referenced by the author in this book is available to
readers on GitHub (https://github.com/Apress). For more detailed information, please visit
https://www.apress.com/gp/services/source-code.
Paper in this product is recyclable
Table of Contents
About the Authors���������������������������������������������������������������������������������������������������� ix

About the Technical Reviewers������������������������������������������������������������������������������� xi


Acknowledgments������������������������������������������������������������������������������������������������� xiii

Introduction�������������������������������������������������������������������������������������������������������������xv

Chapter 1: Introduction to Anomaly Detection��������������������������������������������������������� 1


What Is an Anomaly?��������������������������������������������������������������������������������������������������������������������� 1
Anomalous Swans������������������������������������������������������������������������������������������������������������������� 1
Anomalies as Data Points�������������������������������������������������������������������������������������������������������� 4
Anomalies in a Time Series����������������������������������������������������������������������������������������������������� 6
Categories of Anomalies������������������������������������������������������������������������������������������������������������� 12
Data Point–Based Anomalies������������������������������������������������������������������������������������������������� 12
Context-Based Anomalies����������������������������������������������������������������������������������������������������� 13
Pattern-Based Anomalies������������������������������������������������������������������������������������������������������ 14
Anomaly Detection���������������������������������������������������������������������������������������������������������������������� 14
Outlier Detection�������������������������������������������������������������������������������������������������������������������� 15
Noise Removal����������������������������������������������������������������������������������������������������������������������� 15
Novelty Detection������������������������������������������������������������������������������������������������������������������ 16
Event Detection��������������������������������������������������������������������������������������������������������������������� 16
Change Point Detection��������������������������������������������������������������������������������������������������������� 16
Anomaly Score Calculation���������������������������������������������������������������������������������������������������� 17
The Three Styles of Anomaly Detection�������������������������������������������������������������������������������������� 17
Where Is Anomaly Detection Used?�������������������������������������������������������������������������������������������� 18
Data Breaches����������������������������������������������������������������������������������������������������������������������� 18
Identity Theft�������������������������������������������������������������������������������������������������������������������������� 19
Manufacturing����������������������������������������������������������������������������������������������������������������������� 20

iii
Table of Contents

Networking���������������������������������������������������������������������������������������������������������������������������� 21
Medicine�������������������������������������������������������������������������������������������������������������������������������� 21
Video Surveillance����������������������������������������������������������������������������������������������������������������� 22
Environment�������������������������������������������������������������������������������������������������������������������������� 22
Summary������������������������������������������������������������������������������������������������������������������������������������ 22

Chapter 2: Introduction to Data Science����������������������������������������������������������������� 23


Data Science������������������������������������������������������������������������������������������������������������������������������� 24
Dataset���������������������������������������������������������������������������������������������������������������������������������� 24
Pandas, Scikit-Learn, and Matplotlib������������������������������������������������������������������������������������� 27
Data I/O���������������������������������������������������������������������������������������������������������������������������������� 28
Data Manipulation����������������������������������������������������������������������������������������������������������������� 33
Data Analysis������������������������������������������������������������������������������������������������������������������������� 75
Visualization�������������������������������������������������������������������������������������������������������������������������� 79
Data Processing�������������������������������������������������������������������������������������������������������������������� 85
Feature Engineering and Selection��������������������������������������������������������������������������������������� 96
Summary���������������������������������������������������������������������������������������������������������������������������������� 103

Chapter 3: Introduction to Machine Learning������������������������������������������������������� 105


Machine Learning��������������������������������������������������������������������������������������������������������������������� 106
Introduction to Machine Learning��������������������������������������������������������������������������������������� 106
Data Splitting����������������������������������������������������������������������������������������������������������������������� 111
Modeling and Evaluation����������������������������������������������������������������������������������������������������� 112
Overfitting and Bias-Variance Tradeoff�������������������������������������������������������������������������������� 121
Hyperparameter Tuning������������������������������������������������������������������������������������������������������� 130
Validation���������������������������������������������������������������������������������������������������������������������������������� 132
Summary���������������������������������������������������������������������������������������������������������������������������������� 134

Chapter 4: Traditional Machine Learning Algorithms������������������������������������������� 135


Traditional Machine Learning Algorithms���������������������������������������������������������������������������������� 136
Isolation Forest�������������������������������������������������������������������������������������������������������������������� 136
One-Class Support Vector Machine������������������������������������������������������������������������������������� 159
Summary���������������������������������������������������������������������������������������������������������������������������������� 182

iv
Table of Contents

Chapter 5: Introduction to Deep Learning������������������������������������������������������������ 183


Introduction to Deep Learning�������������������������������������������������������������������������������������������������� 185
What Is Deep Learning?������������������������������������������������������������������������������������������������������ 185
The Neuron�������������������������������������������������������������������������������������������������������������������������� 187
Activation Functions������������������������������������������������������������������������������������������������������������ 189
Neural Networks������������������������������������������������������������������������������������������������������������������ 203
Loss Functions�������������������������������������������������������������������������������������������������������������������� 210
Gradient Descent and Backpropagation������������������������������������������������������������������������������ 213
Loss Curve��������������������������������������������������������������������������������������������������������������������������� 224
Regularization��������������������������������������������������������������������������������������������������������������������� 227
Optimizers��������������������������������������������������������������������������������������������������������������������������� 228
Multilayer Perceptron Supervised Anomaly Detection�������������������������������������������������������� 238
Simple Neural Network: Keras��������������������������������������������������������������������������������������������� 243
Simple Neural Network: PyTorch����������������������������������������������������������������������������������������� 251
Summary���������������������������������������������������������������������������������������������������������������������������������� 260

Chapter 6: Autoencoders�������������������������������������������������������������������������������������� 261


What Are Autoencoders?����������������������������������������������������������������������������������������������������������� 262
Simple Autoencoders���������������������������������������������������������������������������������������������������������������� 264
Sparse Autoencoders���������������������������������������������������������������������������������������������������������������� 281
Deep Autoencoders������������������������������������������������������������������������������������������������������������������� 284
Convolutional Autoencoders����������������������������������������������������������������������������������������������������� 286
Denoising Autoencoders����������������������������������������������������������������������������������������������������������� 294
Variational Autoencoders���������������������������������������������������������������������������������������������������������� 304
Summary���������������������������������������������������������������������������������������������������������������������������������� 320

Chapter 7: Generative Adversarial Networks������������������������������������������������������� 321


What Is a Generative Adversarial Network?������������������������������������������������������������������������������ 322
Generative Adversarial Network Architecture���������������������������������������������������������������������� 325
Wasserstein GAN����������������������������������������������������������������������������������������������������������������� 327
WGAN-GP����������������������������������������������������������������������������������������������������������������������������� 329
Anomaly Detection with a GAN�������������������������������������������������������������������������������������������� 330
Summary���������������������������������������������������������������������������������������������������������������������������������� 343
v
Table of Contents

Chapter 8: Long Short-Term Memory Models������������������������������������������������������� 345


Sequences and Time Series Analysis���������������������������������������������������������������������������������������� 346
What Is an RNN?����������������������������������������������������������������������������������������������������������������������� 349
What Is an LSTM?��������������������������������������������������������������������������������������������������������������������� 350
LSTM for Anomaly Detection����������������������������������������������������������������������������������������������������� 355
Examples of Time Series����������������������������������������������������������������������������������������������������������� 377
art_daily_no_noise.csv������������������������������������������������������������������������������������������������������� 378
art_daily_nojump.csv���������������������������������������������������������������������������������������������������������� 379
art_daily_jumpsdown.csv��������������������������������������������������������������������������������������������������� 381
art_daily_perfect_square_wave.csv����������������������������������������������������������������������������������� 384
art_load_balancer_spikes.csv�������������������������������������������������������������������������������������������� 386
ambient_temperature_system_failure.csv�������������������������������������������������������������������������� 387
ec2_cpu_utilization.csv������������������������������������������������������������������������������������������������������� 389
rds_cpu_utilization.csv������������������������������������������������������������������������������������������������������� 390
Summary���������������������������������������������������������������������������������������������������������������������������������� 392

Chapter 9: Temporal Convolutional Networks������������������������������������������������������ 393


What Is a Temporal Convolutional Network?����������������������������������������������������������������������������� 394
Dilated Temporal Convolutional Network���������������������������������������������������������������������������������� 399
Anomaly Detection with the Dilated TCN����������������������������������������������������������������������������� 404
Encoder-Decoder Temporal Convolutional Network������������������������������������������������������������������ 421
Anomaly Detection with the ED-TCN����������������������������������������������������������������������������������� 424
Summary���������������������������������������������������������������������������������������������������������������������������������� 441

Chapter 10: Transformers������������������������������������������������������������������������������������� 443


What Is a Transformer?������������������������������������������������������������������������������������������������������������� 443
Transformer Architecture����������������������������������������������������������������������������������������������������� 446
Transformer Encoder����������������������������������������������������������������������������������������������������������� 447
Transformer Decoder����������������������������������������������������������������������������������������������������������� 452
Transformer Inference��������������������������������������������������������������������������������������������������������� 455
Anomaly Detection with the Transformer���������������������������������������������������������������������������� 455
Summary���������������������������������������������������������������������������������������������������������������������������������� 480

vi
Table of Contents

Chapter 11: Practical Use Cases and Future Trends of Anomaly Detection ��������� 481
Anomaly Detection�������������������������������������������������������������������������������������������������������������������� 482
Real-World Use Cases of Anomaly Detection���������������������������������������������������������������������������� 485
Telecom������������������������������������������������������������������������������������������������������������������������������� 485
Banking������������������������������������������������������������������������������������������������������������������������������� 487
Environmental��������������������������������������������������������������������������������������������������������������������� 488
Health Care�������������������������������������������������������������������������������������������������������������������������� 490
Transportation��������������������������������������������������������������������������������������������������������������������� 493
Social Media������������������������������������������������������������������������������������������������������������������������ 494
Finance and Insurance�������������������������������������������������������������������������������������������������������� 495
Cybersecurity���������������������������������������������������������������������������������������������������������������������� 496
Video Surveillance��������������������������������������������������������������������������������������������������������������� 499
Manufacturing��������������������������������������������������������������������������������������������������������������������� 500
Smart Home������������������������������������������������������������������������������������������������������������������������� 503
Retail����������������������������������������������������������������������������������������������������������������������������������� 504
Implementation of Deep Learning–Based Anomaly Detection�������������������������������������������������� 504
Future Trends���������������������������������������������������������������������������������������������������������������������������� 506
Summary���������������������������������������������������������������������������������������������������������������������������������� 508

Index��������������������������������������������������������������������������������������������������������������������� 511

vii
About the Authors
Suman Kalyan Adari is currently a machine learning research engineer. He obtained a
B.S. in computer science at the University of Florida and an M.S. in computer science,
specializing in machine learning, at Columbia University. He has been conducting
deep learning research in adversarial machine learning since his freshman year at the
University of Florida and has presented at the IEEE Dependable Systems and Networks
workshop on Dependable and Secure Machine Learning held in Portland, Oregon,
USA in June 2019. Currently, he works on various anomaly detection tasks spanning
behavioral tracking and geospatial trajectory modeling.
He is quite passionate about deep learning, and specializes in various fields ranging
from video processing to generative modeling, object tracking, time-series modeling,
and more.

Sridhar Alla is the co-founder and CTO of Bluewhale, which helps organizations big
and small in building AI-driven big data solutions and analytics, as well as SAS2PY, a
powerful tool to automate migration of SAS workloads to Python-based environments
using Pandas or PySpark. He is a published author of books and an avid presenter
at numerous Strata, Hadoop World, Spark Summit, and other conferences. He also
has several patents filed with the US PTO on large-scale computing and distributed
systems. He has extensive hands-on experience in several technologies, including Spark,
Flink, Hadoop, AWS, Azure, TensorFlow, Cassandra, and others. He spoke on anomaly
detection using deep learning at Strata SFO in March 2019 and at Strata London in
October 2019. He was born in Hyderabad, India, and now lives in New Jersey with his
wife, Rosie, his daughters, Evelyn and Madelyn, and his son, Jayson. When he is not busy
writing code, he loves to spend time with his family and also training, coaching, and
organizing meetups.

ix
About the Technical Reviewers
Puneet Sinha has accumulated more than 12 years of work
experience in developing and deploying end-to-end models
in credit risk, multiple marketing optimization, A/B testing,
demand forecasting and brand evaluation, profit and price
analyses, anomaly and fraud detection, propensity modeling,
recommender systems, upsell/cross-sell models, modeling
response to incentives, price optimization, natural language
processing, and OCR using ML/deep learning algorithms.

Shubho Mohanty is a product thinker and creator, bringing


two decades of experience in the “concept-to-­market”
life cycle of some of the unique, innovative, and highly
successful industry-first products and platforms in the data
and security spaces.
Shubho holds 12+ US patents in data, analytics, and
cloud security. He has also been awarded IDG CIO100, 2020
for strategizing and developing a technology innovation
ecosystem.
He currently serves as the Chief Product Officer at Calibo, where he leads the
product vision, strategy, innovation, and development of Calibo’s enterprise PaaS. Prior
to Calibo, Shubho was the Global VP of Product & Engineering at CDK Global (formerly,
ADP Inc). He has also served in various product leadership roles in organizations like
Symantec and Microsoft. He also co-founded Ganos, a B2B data start-up.
He received his B.Tech. in Electrical Engineering from National Institute of
Technology (NIT), India. He is a mentor to many high-repute start-up programs where
he guides young entrepreneurs to solve some of the most pressing challenges. He is also
an influential speaker at leading technology and industry forums.

xi
Acknowledgments
Suman Kalyan Adari
I would like to thank my parents, Krishna and Jyothi, my sister, Niha, and my loving dog,
Pinky, for supporting me throughout the entire process of writing this book as well as my
various other endeavors.

Sridhar Alla
I would like to thank my wonderful, loving wife, Rosie Sarkaria, and my beautiful,
loving children, Evelyn, Madelyn, and Jayson, for all their love and patience during the
many months I spent writing this book. I would also like to thank my parents, Ravi and
Lakshmi Alla, for their blessings and all the support and encouragement they continue
to bestow upon me.

xiii
Introduction
Congratulations on your decision to explore the exciting world of anomaly detection
using deep learning!
Anomaly detection involves finding patterns that do not adhere to what is
considered as normal or expected behavior. Businesses could lose millions of dollars
due to abnormal events. Consumers could also lose millions of dollars. In fact, there are
many situations every day where people’s lives are at risk and where their property is at
risk. If your bank account gets cleaned out, that’s a problem. If your water line breaks,
flooding your basement, that’s a problem. If all flights at an airport get delayed due to a
technical glitch in the traffic control system, that’s a problem. If you have a health issue
that is misdiagnosed or not diagnosed, that’s a very big problem that directly impacts
your well-being.
In this book, you will learn how anomaly detection can be used to solve business
problems. You will explore how anomaly detection techniques can be used to address
practical use cases and address real-life problems in the business landscape. Every
business and use case is different, so while we cannot copy and paste code and build a
successful model to detect anomalies in any dataset, this book will cover many use cases
with hands-on coding exercises to give you an idea of the possibilities and concepts
behind the thought process. All the code examples in the book are presented in Python
3.8. We choose Python because it is truly the best language for data science, with a
plethora of packages and integrations with scikit-learn, deep learning libraries, etc.
We will start by introducing anomaly detection, and then we will look at legacy
methods of detecting anomalies that have been used for decades. Then we will look
at deep learning to get a taste of it. Then we will explore autoencoders and variational
autoencoders, which are paving the way for the next generation of generative models.
Following that, we will explore generative adversarial networks (GANs) as a way to detect
anomalies, delving directly into generative AI.
Then we’ll look at long short-term memory (LSTM) models to see how temporal data
can be processed. We will cover temporal convolutional networks (TCNs), which are
excellent for temporal data anomaly detection. We will also touch upon the transformer

xv
Introduction

architecture, which has revolutionized the field of natural language processing as


another means for temporal anomaly detection. Finally, we will look at several examples
of anomaly detection in various business use cases.
In addition, all coding examples will be provided in TensorFlow 2/Keras, with
accompanying PyTorch equivalents, on the GitHub repository for this book. You will
combine all this extensive knowledge with hands-on coding using Jupyter notebook-
based exercises to experience the knowledge firsthand and see where you can use these
algorithms and frameworks. Best of luck, and welcome to the world of deep learning!

xvi
CHAPTER 1

Introduction to Anomaly
Detection
In this chapter, you will learn about anomalies in general, the categories of anomalies,
and anomaly detection. You will also learn why anomaly detection is important, how
anomalies can be detected, and the use case for such a mechanism.
In a nutshell, this chapter covers the following topics:

• What is an anomaly?

• Categories of different anomalies

• What is anomaly detection?

• Where is anomaly detection used?

What Is an Anomaly?
Before you get started with learning about anomaly detection, you must first understand
what exactly you are targeting. Generally, an anomaly is an outcome or value that
deviates from what is expected, but the exact criteria for what determines an anomaly
can vary from situation to situation.

Anomalous Swans
To get a better understanding of what an anomaly is, let’s take a look at some swans
sitting by a lake (Figure 1-1).

1
© Suman Kalyan Adari, Sridhar Alla 2024
S. K. Adari and S. Alla, Beginning Anomaly Detection Using Python-Based Deep Learning,
https://doi.org/10.1007/979-8-8688-0008-5_1
Chapter 1 Introduction to Anomaly Detection

Figure 1-1. A couple swans by a lake

Let’s say that we want to observe these swans and make assumptions about the
color of the swans at this particular lake. Our goal is to determine what the normal
color of swans is and to see if there are any swans that are of a different color than this
(Figure 1-2).

Figure 1-2. More swans show up, all of which are white

2
Chapter 1 Introduction to Anomaly Detection

We continue to observe swans for a few years and all of them have been white. Given
these observations, we can reasonably conclude that every swan at this lake should be
white. The very next day, we are observing swans at the lake again. But wait! What’s this?
A black swan has just flown in (Figure 1-3).

Figure 1-3. A black swan appears

Considering our previous observations, we thought that we had seen enough swans
to assume that the next swan would also be white. However, the black swan defies that
assumption entirely, making it an anomaly. It’s not really an outlier, which would be, for
example, a really big white swan or a really small white swan; it’s a swan that’s entirely
a different color, making it an anomaly. In our scenario, the overwhelming majority of
swans are white, making the black swan extremely rare.
In other words, given a swan by the lake, the probability of it being black is very
small. We can explain our reasoning for labeling the black swan as an anomaly with one
of two approaches (though we aren’t limited to only these two approaches).
First, given that a vast majority of swans observed at this particular lake are white, we
can assume that, through a process similar to inductive reasoning, the normal color for a
swan here is white. Naturally, we would label the black swan as an anomaly purely based
on our prior assumptions that all swans are white, considering that we’d only seen white
swans before the black swan arrived.

3
Chapter 1 Introduction to Anomaly Detection

Another way to look at why the black swan is an anomaly is through probability.
Now assume that there is a total of 1000 swans at this lake and only two are black swans;
the probability of a swan being black is 2 / 1000, or 0.002. Depending on the probability
threshold, meaning the lowest probability for an outcome or event that will be accepted
as normal, the black swan could be labeled as anomalous or normal. In our case, we will
consider it an anomaly because of its extreme rarity at this lake.

Anomalies as Data Points


We can extend this same concept to a real-world application. In the following example,
we will take a look at a factory that produces screws and attempt to determine what an
anomaly could be in this context and individual screws are sampled from each batch
and are tested to ensure a certain level of quality is maintained. For each sampled screw,
assume that the density and tensile strength (how resistant the screw is to breaking
under stress) are measured.
Figure 1-4 is an example graph of various sampled screws with the dotted lines
representing the range of densities and tensile strengths allowed. The solid lines form a
bounding box where any value of tensile strength and density inside it is considered good.

Figure 1-4. Density and tensile strength in a batch of screw samples

The intersections of the dotted lines have created several different regions containing
data points. Of interest is the bounding box (solid lines) created from the intersection of
both sets of dotted lines since it contains the data points for samples deemed acceptable
(Figure 1-5). Any data point outside of that specific box will be considered anomalous.
4
Chapter 1 Introduction to Anomaly Detection

Figure 1-5. Data points are identified as “good” or “anomaly” based on their
location

Now that we know which points are and aren’t acceptable, let’s pick out a sample from
a new batch of screws and check its data to see where it falls on the graph (Figure 1-6).

Figure 1-6. A new data point representing the new sample screw is generated,
with the data falling within the bounding box

The data for this sample screw falls within the acceptable range. That means that this
batch of screws is good to use since its density as well as tensile strength is appropriate
for use by the consumer. Now let’s look at a sample from the next batch of screws and
check its data (Figure 1-7).

5
Chapter 1 Introduction to Anomaly Detection

Figure 1-7. A new data point is generated for another sample, but this falls
outside the bounding box

The data falls far outside the acceptable range. For its density, the screw has abysmal
tensile strength and is unfit for use. Since it has been flagged as an anomaly, the
factory can investigate why this batch of screws turned out to be brittle. For a factory of
considerable size, it is important to hold a high standard of quality as well as maintain
a high volume of steady output to keep up with consumer demands. For a monumental
task like that, automation to detect any anomalies to avoid sending out faulty screws is
essential and has the benefit of being extremely scalable.
So far, we have explored anomalies as data points that are either out of place, in the
case of the black swan, or unwanted, in the case of faulty screws. So what happens when
we introduce time as a new variable?

Anomalies in a Time Series


With the introduction of time as a variable, we are now dealing with a notion of
temporality associated with the data sets. What this means is that certain patterns are
dependent on time. For example, daily, monthly, or yearly occurrences are time-series
patterns as they present on regular time intervals.
To better understand time series–based anomalies, let’s look at a few examples.

6
Chapter 1 Introduction to Anomaly Detection

Personal Spending Pattern


Figure 1-8 depicts a random person’s spending habits over the course of a month.

Figure 1-8. Spending habits of a person over the course of a month

Assume the initial spike in expenditures at the start of the month is due to the
payment of bills such as rent and insurance. During the weekdays, our example person
occasionally eats out, and on the weekends goes shopping for groceries, clothes, and
various other items. Also assume that this month does not include any major holidays.
These expenditures can vary from month to month, especially in months with major
holidays. Assume that our person lives in the United States, in which the holiday of
Thanksgiving falls on the last Thursday of the month of November. Many U.S. employers
also include the Friday after Thanksgiving as a day off for employees. U.S. retailers have
leveraged that fact to entice people to begin their Christmas shopping by offering special
deals on what has colloquially become known as “Black Friday.” With that in mind, let’s
take a look at our person’s spending pattern in November (Figure 1-9). As expected, a
massive spike in purchases occurred on Black Friday, some of them quite expensive.

7
Chapter 1 Introduction to Anomaly Detection

Figure 1-9. Spending habits for the same person during the month of November

Now assume that, unfortunately, our person has had their credit card information
stolen, and the criminals responsible for it have decided to purchase various items of
interest to them. Using the same month as in the first example (Figure 1-8; no major
holidays), the graph in Figure 1-10 depicts what could happen.

Figure 1-10. Purchases in the person’s name during the same month as in
Figure 1-8

Let’s assume we have a record of purchases for this user going back many years.
Thanks to this established prior history, this sudden influx in purchases would be
flagged as anomalies. Such a cluster of purchases might be normal for Black Friday or
in the weeks before Christmas, but in any other month without a major holiday, it looks

8
Chapter 1 Introduction to Anomaly Detection

out of place. In this case, our person might be contacted by the credit card company to
confirm whether or not they made the purchases.
Some companies might even flag purchases that follow normal societal trends. What
if that TV wasn’t really bought by our person on Black Friday? In that case, the credit
card company’s software can ask the client directly through a phone app, for example,
whether or not they actually bought the item in question, allowing for some additional
protection against fraudulent purchases.

Taxi Cabs
As another example of anomalies in a time series, let’s look at some sample data for taxi
cab pickups and drop-offs over time for a random city and an arbitrary taxi company and
see if we can detect any anomalies.
On an average day, the total number of pickups can look somewhat like the pattern
shown in Figure 1-11.

Figure 1-11. Number of pickups for a taxi company throughout the day

From the graph, we see that there’s a bit of post-midnight activity that drops off
to near zero during the late-night hours. However, customer traffic picks up suddenly
around morning rush hour and remains high until the early evening, when it peaks
during evening rush hour. This is essentially what an average day looks like.

9
Chapter 1 Introduction to Anomaly Detection

Let’s expand the scope out a bit more to gain some perspective of passenger traffic
throughout the week (Figure 1-12).

Figure 1-12. Number of pickups for a taxi company throughout the week

As expected, most of the pickups occur during the weekday when commuters
must get to and from work. On the weekends, a fair amount of people still go out to get
groceries or just go out somewhere for the weekend.
On a small scale like this, causes for anomalies would be anything that prevents
taxis from operating or incentivizes customers not to use a taxi. For example, say that a
terrible thunderstorm hits on Friday. Figure 1-13 shows that graph.

Figure 1-13. Number of pickups for a taxi company throughout the week, with a
heavy thunderstorm on Friday

10
Chapter 1 Introduction to Anomaly Detection

The thunderstorm likely influenced some people to stay indoors, resulting in a lower
number of pickups than unusual for a weekday. However, these sorts of anomalies are
usually too small-scale to have any noticeable effect on the overall pattern.
Let’s take a look at the data over the entire year, as shown in Figure 1-14.

Figure 1-14. Number of pickups for a taxi company throughout the year

The largest dips occur during the winter months when snowstorms are expected.
These are regular patterns that can be observed at similar times every year, so they
are not an anomaly. But what happens to customer traffic levels when a relatively rare
polar vortex descends on the city in early April and unleashes several intense blizzards?
Figure 1-15 shows the graph.

11
Chapter 1 Introduction to Anomaly Detection

Figure 1-15. Number of pickups for a taxi company throughout the year, with a
polar vortex descending on the city in April

As you can see in Figure 1-15, the intense blizzards severely slowed down all traffic
in the first week of April and burdened the city in the following two weeks. Comparing
this graph to the graph shown in Figure 1-14, there’s a clearly defined anomaly in April
caused by the polar vortex. Since this pattern is extremely rare for the month of April, it
would be flagged as an anomaly.

Categories of Anomalies
Now that you have more perspective of what anomalies can be in various situations, you
can see that they generally fall into these broad categories:

• Data point–based anomalies

• Context-based anomalies

• Pattern-based anomalies

Data Point–Based Anomalies


Data point–based anomalies may seem comparable to outliers in a set of data points.
However, as previously mentioned, anomalies and outliers are not the same thing,
though these terms are sometimes used interchangeably. Outliers are data points that

12
Chapter 1 Introduction to Anomaly Detection

are expected to be present in the data set and can be caused by unavoidable random
errors or from systematic errors relating to how the data was sampled. Anomalies would
be outliers or other values that one doesn’t expect to exist. These data anomalies might
be present wherever a data set of values exist.
As an example of a data set in which data point–based anomalies may exist,
consider a data set of thyroid diagnostic values, where the majority of the data points are
indicative of normal thyroid functionality. In this case, anomalous values represent sick
thyroids. While they are not necessarily outliers, they have a low probability of existing
when taking into account all the normal data.
We can also detect individual purchases totaling excessive amounts and label
them as anomalies since, by definition, they are not expected to occur or have a very
low probability of occurrence. In this case, they are labeled as potentially fraudulent
transactions, and the card holder is contacted to ensure the validity of the purchase.
Basically, we can say this about the difference between anomalies and outliers:
we should expect a data set to include outliers, but we should not expect it to include
anomalies. Though the terms “anomaly” and “outlier” are sometimes interchanged,
anomalies are not always outliers, and not all outliers are anomalies.

Context-Based Anomalies
Context-based anomalies consist of data points that might seem normal at first but
are considered anomalies in their respective contexts. Returning to our earlier personal
spending example, we might expect a sudden surge in purchases near certain holidays,
but these purchases could seem unusual in the middle of August. The person’s high
volume of purchases on Black Friday was not flagged because it is typical spending
behavior for people on Black Friday. However, if the purchases were made in a month
where it is out of place given previous purchase history, it would be flagged as an
anomaly. This might seem similar to the example presented for data point–based
anomalies, but the distinction for context-based anomalies is that the individual purchase
does not have to be expensive. If a person never buys gasoline because they own an
electric car, sudden purchases of gasoline would be out of place given the context. Buying
gasoline is normal behavior for many people, but in this context, it is an anomaly.

13
Chapter 1 Introduction to Anomaly Detection

Pattern-Based Anomalies
Pattern-based anomalies are patterns and trends that deviate from their historical
counterparts, and they often occur in time-series or other sequence-based data. In the
earlier taxi cab company example, the customer pickup counts for the month of April
were pretty consistent with the rest of the year. However, once the polar vortex hit, the
numbers tanked visibly, resulting in a huge drop in the graph, labeled as an anomaly.
Similarly, when monitoring network traffic in the workplace, expected patterns of
network traffic are formed from constant monitoring of data over several months or
even years for some companies. If an employee attempts to download or upload large
volumes of data, it generates a certain pattern in the overall network traffic flow that
could be considered anomalous if it deviates from the employee’s usual behavior.
As another example of pattern-based anomalies, if an external hacker decided to hit
a company’s website with a distributed denial-of-service (DDoS) attack—an attempt
to overwhelm the server that handles network flow to a certain website in an attempt
to bring the entire website down or stop its functionality—every single attempt would
register as an unusual spike in network traffic. All of these spikes are clearly deviants
from normal traffic patterns and would be considered anomalous.

Anomaly Detection
Now that you have a better understanding of the different types of anomalies, we can
proceed to discuss approaches to creating models to detect anomalies. This section presents
a few approaches we can take, but keep in mind we are not limited to just these methods.
Recall our reasoning for labeling the black swan as an anomaly. One reason was
that since all the swans we have seen thus far were white, the single black swan is an
obvious anomaly. A statistical way to explain this reasoning is that as of the most recent
set of observations, we have one black swan and tens of thousands of white swans.
Thus, the probability of occurrence of the black swan is one out of tens of thousands
of all observed swans. Since this probability is so low, it would make the black swan an
anomaly just because we do not expect to see it at all.
The anomaly detection models we will explore in this book follow these approaches
either by training on unlabeled data, training on normal, nonanomalous data, or training
on labeled data for normal and anomalous data. In the context of identifying swans, we
would be told which swans are normal and which swans are anomalies.

14
Chapter 1 Introduction to Anomaly Detection

So, what is anomaly detection? Quite simply, anomaly detection is the process in
which an advanced algorithm identifies certain data or data patterns to be anomalous.
Falling under anomaly detection are the tasks of outlier detection, noise removal, novelty
detection, event detection, change point detection, and anomaly score calculation.
In this book, we will explore all of these as they are all basically anomaly detection
methods. The following tasks of anomaly detection are not exhaustive, but are some of
the more common anomaly detection tasks today.

Outlier Detection
Outlier detection is a technique that aims to detect anomalous outliers within a given
data set. As previously discussed, three methods that can be applied to this situation are
to train a model only on normal data to identify anomalies (by a high reconstruction error,
described next), to model a probability distribution in which anomalies would be labeled
based on their association with really low probabilities, or to train a model to recognize
anomalies by teaching it what an anomaly looks like and what a normal point looks like.
Regarding the high reconstruction error, think of it this way: the model trains on a
set of normal data and learns the patterns corresponding to normal data. When exposed
to an anomalous data point, the patterns do not line up with what the model learned to
associate with normal data. The reconstruction error can be analogous to the deviance in
learned patterns between the anomalous point and the normal points the model trained
on. We will formally go over reconstruction error in Chapter 6. Going back to the example
of the swans, the black swan is different based on the patterns that we learned by observing
swans at the lake, and was thus anomalous because it did not follow the color pattern.

Noise Removal
Noise removal involves filtering out any constant background noise in the data set.
Imagine that you are at a party and you are talking with a friend. There is a lot of
background noise, but your brain focuses on your friend’s voice and isolates it because
that’s what you want to hear. Similarly, a model learns to efficiently encode the original
sound to represent only the essential information. For example, encoding the pitch of
your friend’s voice, the tempo, the vocal inflections, etc. Then, it is able to reconstruct the
original sound without the anomalous interference noise.

15
Chapter 1 Introduction to Anomaly Detection

This can also be a case where an image has been altered in some form, such as by
having perturbations, loss of detail, fog, etc. The model learns an accurate representation
of the original image and outputs a reconstruction without any of the anomalous
elements in the image.

Novelty Detection
Novelty detection is very similar to outlier detection. In this case, a novelty is a data
point outside of the training set the model was exposed to, that was shown to the model
to determine if it is an anomaly or not. The key difference between novelty detection and
outlier detection is that in outlier detection, the job of the model is to determine what is
an anomaly within the training data set. In novelty detection, the model learns what is a
normal data point and what isn’t and tries to classify anomalies in a new data set that it
has never seen before.
Examples can include quality assurance in factories to make sure new batches of
created products are up to par, such as with the example of screws from earlier. Another
case is network security. Incoming traffic data can be monitored to ensure there is no
anomalous behavior going on. Both of these situations involve novelties (new data) to
constantly predict on.

Event Detection
Event detection involves the detection of points in a time-series dataset that deviate
anomalously from the norm. For example, in the taxi cab company example earlier in the
chapter, the polar vortex reduced the customer pickup counts for April. These deviations
from the norm for April were all associated with an anomalous event that occurred
in the dataset, which an event detector algorithm would identify. Another example of
event detection would be the tracking of sea-ice levels at the poles over time, forming a
time series. An event detector algorithm could flag exactly when sea-ice levels deviate
anomalously from the usual norm, such as occurred recently in July 2023, when the sea-
ice levels were detected to be six standard deviations below the established average.

Change Point Detection


Change point detection involves the detection of points in the dataset where their
statistical properties start to consistently deviate from the established norm given by
the rest of the data. In other words, change point detection algorithms measure shifts
16
Chapter 1 Introduction to Anomaly Detection

in statistical trends. A good example is global average temperatures over time. A change
point detection algorithm could identify periods of sustained warming where the
statistical properties start to shift over time. A change point detection algorithm could
identify accelerated warming periods that differ from the normal rate of warming.

Anomaly Score Calculation


Anomaly score calculation involves assigning a score of how anomalous a given data
sample is. Rather than an outright determination of whether or not something is an
anomaly, this is for when we want to assign some type of measure to how deviant from
the normal a data sample may be. In some cases, anomaly score calculation is a direct
prelude to anomaly detection, since you could assign a threshold that determines what
are anomalies and what aren’t by the anomaly score. In other cases, anomaly score
calculation can help with understanding a data set more deeply and aid with data
analysis by providing a more nuanced measure of the normalcy of a data point or a
measure of how much a data sample deviates from the norm.

The Three Styles of Anomaly Detection


There are three overarching “styles” of anomaly detection:

• Supervised anomaly detection

• Semi-supervised anomaly detection


• Unsupervised anomaly detection

Supervised anomaly detection is a technique in which the training data has labels
for both anomalous data points and normal data points. Basically, we tell the model
during the training process if a data point is an anomaly or not. Unfortunately, this
isn’t the most practical method of training, especially because the entire data set needs
to be processed and each data point needs to be labeled. Since supervised anomaly
detection is basically a type of binary classification task, meaning the job of the model is
to categorize data under one of two labels, any classification model can be used for the
task, though not every model can attain a high level of performance. Chapter 9 provides
an example in the context of a temporal convolutional network.

17
Chapter 1 Introduction to Anomaly Detection

Semi-supervised anomaly detection involves partially labeling the training data set.
Exact implementations and definitions for what “semi-supervised” entails may differ, but
the gist of it is that you are working with partially labeled data. In the context of anomaly
detection, this can be a case where only the normal data is labeled. Ideally, the model will
learn what normal data points look like so that it can flag as anomalous data points that
differ from normal data points. Examples of models that can use semi-supervised learning
for anomaly detection include autoencoders, which you will learn about in Chapter 6.
Unsupervised anomaly detection, as the name implies, involves training the model
on unlabeled data. After the training process, the model is expected to know what data
points are normal and what points are anomalous within the data set. Isolation forest, a
model we will explore in Chapter 4, is one such model that can be used for unsupervised
anomaly detection.

Where Is Anomaly Detection Used?


Whether we realize it or not, anomaly detection is being utilized in nearly every facet
of our lives today. Pretty much any task involving data collection of any sort could have
anomaly detection applied to it. Let’s look at some of the most prevalent fields and topics
to which anomaly detection can be applied.

Data Breaches
In today’s age of big data, where huge volumes of information are stored about users
in various companies, information security is vital. Any information breaches must be
reported and flagged immediately, but it is hard to do so manually at such a scale. Data
leaks can range from simple accidents, such as an employee losing a USB stick that
contains sensitive company information that someone picks up and accesses the data
on, to intentional actions, such as an employee intentionally sending data to an outside
party, or an attacker gaining access to a database via an intrusion attack. Several high-
profile data leaks have been widely reported in news media, from Facebook / Meta,
iCloud, and Google security breaches where millions of passwords or photos were
leaked. All of those companies operate on an international scale, requiring automation
to monitor everything in order to ensure the fastest response time to any data breach.
The data breaches might not even need network access. For example, an employee
could email an outside party or another employee with connections to rival companies
about travel plans to meet up and exchange confidential information. Of course, these
18
Chapter 1 Introduction to Anomaly Detection

emails would not be so obvious as to state such intentions directly. However, monitoring
these emails could be helpful as a post-breach analysis to find out anyone suspicious
from within the company, or as part of a real-time monitoring software to ensure data
confidentiality compliance across teams for example. Anomaly detection models can sift
through and process employee emails to flag any suspicious activity by employees. The
software can pick up key words and process them to understand the context and decide
whether or not to flag an employee’s email for review.
The following are a few more examples of how anomaly detection software can
detect an internal data breach:

• Employees may be assigned a specific connection to upload data


to. For example, a mass storage drive that employees should not
frequently access.

• An employee may regularly be accessing data as part of their work


responsibilities. However, if they suddenly start downloading a lot of
data that they shouldn’t, then the anomaly detector may flag this.

• The detector would be looking at many variables, including who


the user is, what data store is being accessed, what volume of data
was transferred, how frequently did the user interact with this data
store, etc. to find out if this recent interaction was a deviation from
established patterns.

In this case, something won’t add up, which the software will detect and then flag
the employee. It could either turn out to be a one-off sanctioned event, which great, the
model did its job but it was ok this time, or it could turn out that the employee somehow
accessed data they shouldn’t have, which would mean there was a data breach.
The key benefit to using anomaly detection in the workspace is how easy it is to scale
up. These models can be used for small companies as well as large-scale international
companies.

Identity Theft
Identity theft is another common problem in today’s society. Thanks to development of
online services allowing for ease of access when purchasing items, the volume of credit
and debit card transactions that take place every day has grown immensely. However,
this development also makes it easier to steal credit and debit card information or bank
account information, allowing the criminals to purchase anything they want if the card
19
Chapter 1 Introduction to Anomaly Detection

isn’t deactivated or if the account isn’t secured again. Because of the huge volume
of transactions, monitoring everything is difficult. However, this is where anomaly
detection can step in and help, since it is highly scalable and can help detect fraud
transactions the moment the request is sent.
As we discussed earlier, context matters. When a payment card transaction is made, the
payment card company’s anomaly detection software takes into account the card holder’s
previous history to determine if the new transaction should be flagged or not. Obviously, a
series of high value purchases made suddenly would raise alarms immediately, but what
if the criminals were smart enough to realize that and just make a series of purchases over
time that won’t put a noticeable hole in the card holder’s account? Again, depending on
the context, the software would pick up on these transactions and flag them again.
For example, let’s say that someone’s grandmother was recently introduced to
Amazon and to the concept of buying things online. One day, unfortunately, she
stumbles upon an Amazon lookalike website and enters her credit card information.
On the other side, some criminal takes it and starts buying random things, but not all
at once so as not to raise suspicion—or so he thought. The grandmother’s identity theft
insurance company starts noticing some recent purchases of batteries, hard drives, flash
drives, and other electronic items. While these purchases might not be that expensive,
they certainly stand out when all the prior purchases made by the grandmother
consisted of groceries, pet food, and various decorative items. Based on this previous
history, the detection software would flag the new purchases and the grandmother
would be contacted to verify these purchases. These transactions might even be flagged
as soon as an attempt to purchase is made. In this case, either the location of the
purchaser or the nature of the transactions could raise alarms and stop the transaction
from being successful.

Manufacturing
We have explored a use case of anomaly detection in manufacturing. Manufacturing
plants usually have a certain level of quality that they must ensure their products meet
before shipping them out. When factories are configured to produce massive quantities
of output at a near constant rate, it becomes necessary to automate the process of
checking the quality of various samples. Similar to the fictitious example of the screw
manufacturer, manufacturing plants in real life might test and use anomaly detection
software to ensure the quality of various metal parts, tools, engines, food, clothes, etc.

20
Chapter 1 Introduction to Anomaly Detection

Networking
Perhaps one of the most important use cases for anomaly detection is in networking.
The Internet is host to a vast array of various websites located on servers all around the
world. Unfortunately, due to the ease of access to the Internet, many individuals access
the Internet for nefarious purposes. Similar to the data leaks that were discussed earlier
in the context of protecting company data, hackers can launch attacks on websites as
well to leak their information.
One such example would be hackers attempting to leak government secrets
through a network attack. With such sensitive information as well as the high volumes
of expected attacks every day, automation is a necessary tool to help cybersecurity
professionals deal with the attacks and preserve state secrets. On a smaller scale, hackers
might attempt to breach a cloud network or a local area network and try to leak data.
Even in smaller cases like this, anomaly detection can help detect network intrusion
attacks as they happen and notify the proper officials. An example data set for network
intrusion anomaly detection is the KDD Cup 1999 data set. This data set contains a
large amount of entries that detail various types of network intrusion attacks as well as a
detailed list of variables for each attack that can help a model identify each type of attack.

Medicine
Anomaly detection also has a massive role to play in the field of medicine. For example,
models can detect subtle irregularities in a patient’s heartbeat in order to classify
diseases, or they can measure brainwave activity to help doctors diagnose certain
conditions. Beyond that, they can help analyze raw diagnostic data for a patient’s organ
and process it in order to quickly diagnose any possible problems, similarly to the
thyroid example discussed earlier.
Anomaly detection can even be used in medical imagery to determine whether a given
image contains anomalous objects. For example, suppose an anomaly detection model
was trained by exposing it only to MRI imagery of normal bones. When shown an image of
a broken bone, it would flag the new image as an anomaly. Similarly, anomaly detection
can even be extended to tumor detection, allowing for the model to analyze every image in
a full-body MRI scan and look for the presence of abnormal growth or patterns.

21
Chapter 1 Introduction to Anomaly Detection

Video Surveillance
Anomaly detection also has uses in video surveillance. Anomaly detection software can
monitor video feeds and flag any videos that capture anomalous action. While this might
seem dystopian, it can certainly help catch criminals and maintain public safety on
busy streets and other transportation systems. For example, this type of software could
potentially identify a mugging in a street at night as an anomalous event and alert the
nearest police department. Additionally, this type of software can detect unusual events
at crossroads, such as an accident or some unusual obstruction, and immediately call
attention to the footage.

Environment
Anomaly detection can be used to monitor environmental conditions as well. For
example, anomaly detection systems are used to monitor heavy-metal levels in rivers
to pick up potential spills or leaks into the water supply. Another example is air quality
monitoring, which can detect anything from seasonal pollen to wildfire smoke coming in
from far away. Additionally, anomaly detection can be utilized to monitor soil health in
agricultural or environmental survey cases to gauge the health of an ecosystem. Any dips
in soil moisture or specific nutrient levels could indicate some kind of problem.

Summary
Generally, anomaly detection is utilized heavily in medicine, finance, cybersecurity,
banking, networking, transportation, and manufacturing, but it is not limited to those fields.
For nearly every case imaginable involving data collection, anomaly detection can be put to
use to help users automate the process of detecting anomalies and possibly removing them.
Many fields in science can benefit from anomaly detection because of the large volume of
raw data collection that goes on. Anomalies that would interfere with the interpretation
of results or otherwise introduce some sort of bias into the data could be detected and
removed, provided that the anomalies are caused by systematic or random errors.
In this chapter, we discussed what anomalies are and why detecting anomalies can
be very important to the data processing we have at our organizations.
Next, Chapter 2 introduces you to core data science concepts that you need to know
to follow along for the rest of the book.

22
CHAPTER 2

Introduction to Data
Science
This chapter introduces you to the basic principles of the data science workflow. These
concepts will help you prepare your data as necessary to be fed into a machine learning
model as well as understand its underlying structure through analysis.
You will learn how to use the libraries Pandas, Numpy, Scikit-Learn, and Matplotlib
to load, manipulate, and analyze data. Furthermore, you will learn how to perform data
I/O; manipulate, transform, and process the data to your liking; analyze and plot the
data; and select/create relevant features for the machine learning modeling task.
In a nutshell, this chapter covers the following topics:

• Data I/O

• Data manipulation

• Data analysis

• Data visualization
• Data processing

• Feature engineering/selection

Note Code examples are provided in Python 3.8. Package versions of all
frameworks used are provided. You will need some type of program to run
Python code, so make sure to set this up beforehand. The code repository for this
book is available at ­https://github.com/apress/beginning-anomaly-­
detection-python-deep-learning-2e/tree/master.

23
© Suman Kalyan Adari, Sridhar Alla 2024
S. K. Adari and S. Alla, Beginning Anomaly Detection Using Python-Based Deep Learning,
https://doi.org/10.1007/979-8-8688-0008-5_2
Chapter 2 Introduction to Data Science

The repository also includes a requirements.txt file to check your packages and their
versions.
Code examples for this chapter are available at https://github.com/apress/
beginning-anomaly-detection-python-deep-learning-2e/blob/master/Chapter%20
2%20Introduction%20to%20Data%20Science/chapter2_datascience.ipynb. Navigate
to “Chapter 2 Introduction to Data Science” and then click chapter2_datascience.
ipynb. The code is provided as a .py file as well, though it is the exported version of the
notebook.
We will be using JupyterLab (https://jupyter.org) to present all of the code
examples.

Data Science
“Data science” is quite the popular term and buzzword nowadays, so what exactly is it?
In recent times, the term “data science” has come to represent a wide range of roles and
responsibilities. Depending on the company, a data scientist can be expected to perform
anything from data processing (often at scale, dipping into “big data” territory), to
statistical analysis and visualization, to training and deploying machine learning models.
In fact, data scientists often perform two or more of these roles at once.
This chapter focuses on concepts from all three roles, walking you through the
process of preparing and analyzing the dataset before you explore the modeling aspect
in Chapter 3.
Be advised that this will be a brief, high-level walkthrough over the most relevant
functionality that these various data science packages offer. Each package is so
comprehensive that a full book would be required to cover it in depth. Therefore, you are
encouraged to explore each package’s online documentation, as well as other guides and
walkthroughs, to learn as much as you can.

Dataset
A popular introductory dataset for budding data scientists is the Titanic Dataset,
available at Kaggle: https://www.kaggle.com/c/titanic. You can also find this dataset
hosted on this book’s repository at ­https://github.com/apress/beginning-anomaly-­
detection-python-deep-learning-2e/blob/master/data/train.csv.

24
Chapter 2 Introduction to Data Science

Kaggle is a great place to find datasets. It also hosts machine learning modeling
competitions, some of which even have prize money. Kaggle is an excellent resource
for practicing your skills, and you are encouraged to do so! If you would like to practice
the concepts that you learn in this book, search Kaggle for various anomaly detection
datasets.
To download the Titanic dataset from the Kaggle web site, follow the instructions
provided next. If you prefer, Kaggle offers an API that you can install through PIP,
available at https://github.com/Kaggle/kaggle-api.

1. Go to https://www.kaggle.com/c/titanic, where you are


greeted by something that looks like Figure 2-1.

Figure 2-1. Overview page for the Titanic dataset on Kaggle (as it looks as of
April, 2023)

2. Click the Data tab. You should see a brief description of the
dataset as well as a Data Explorer, as shown in Figure 2-2.

3. Click Download All. You will be prompted to sign in. Create an


account or log in with an alternate option. After logging in, you
will be returned to the Overview page.

25
Chapter 2 Introduction to Data Science

Figure 2-2. Click Download All to download the data as a zip file

4. Return to the Data tab, scroll down, and click Download All. A zip
file will download.

5. Extract this zip file anywhere that you’d like and make a note of
the directory path.

After you have extracted the zip file, you are almost ready to start processing this
dataset in Python. First, make sure you have an IDE to develop Python code with. To
easily follow the examples in this book, you are recommended to use Jupyter Notebook
or JupyterLab.
Next, make sure you have the following libraries and versions installed, though you
might also want to check the requirements.txt file available on the GitHub repository or
use it to prepare your environment:

• pandas==2.0.0

• numpy==1.22.2

• scikit-learn==1.2.2

• matplotlib==3.7.1

26
Chapter 2 Introduction to Data Science

It is not necessary to have the exact same versions, but keep in mind that older
versions may not contain features we explore later in the book. Newer versions should be
fine unless they are significantly more recent, which might result in reworked syntax or
features and thus introduce incompatibility.
You can easily check the version in Python. Figure 2-3 introduces code to import
these packages and print their versions.

import pandas
import numpy
import sklearn
import matplotlib

print(f"Pandas version: ", pandas.__version__)


print(f"Numpy version: ", numpy.__version__)
print(f"Scikit-learn version: ", sklearn.__version__)
print(f"Matplotlib version: ", matplotlib.__version__)

Figure 2-3. Code to import pandas, numpy, sklearn, and matplotlib and print
their versions

You should see output similar to that displayed in Figure 2-4.

Pandas version: 2.0.0


Numpy version: 1.22.2
Scikit-learn version: 1.2.2
Matplotlib version: 3.7.1

Figure 2-4. The text output of running the code in Figure 2-3

Pandas, Scikit-Learn, and Matplotlib


Pandas is a data science framework for Python that lets you freely manipulate and
analyze data to your liking. It is an expansive, comprehensive framework offering a lot of
functionality, and should suit your data processing needs. Until you start to scale into the
gigabyte realm with your datasets, Pandas can be quick and efficient.

27
Chapter 2 Introduction to Data Science

Scikit-Learn is a machine learning library that covers a wide range of functionality


related to machine learning modeling, ranging from data processing functions to various
machine learning methods
Matplotlib is a data visualization library that lets you build customized plots and
graphs. It integrates well with pandas and scikit-learn, and even lets you display images.
NumPy is a numerical computation library with highly efficient implementations for
various computational tasks. You can represent your data as numpy arrays and perform
quick, vectorized computations, linear algebra operations, and more. NumPy integrates
very well with many popular Python packages, including pandas, scikit-learn, and
matplotlib.
With all of these packages, you should be able to perform comprehensive data
analysis. However, keep in mind that there are other packages available for Python that
you may want to check out to complement your data analysis, such as statsmodels
(statistical modeling library), seaborn (another plotting library, which we will use in
subsequent chapters), plotly (yet another plotting library but for interactive graphs), and
scipy (scientific computing library, which lets you perform various statistical tests)

Data I/O
With our environment set up, let’s jump straight into the content. Before we conduct any
type of data analysis, we need to actually have data. There are a myriad of ways to load
data in Pandas, but we will keep it simple and load from a csv file.
For the sake of convenience, let’s reimport our libraries with aliases, as shown in
Figure 2-5.

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt

Figure 2-5. Importing pandas, numpy, and matplotlib’s pyplot as aliases, for
convenience

Once you have executed this code, let’s move on to loading our dataset.

28
Chapter 2 Introduction to Data Science

Data Loading
First, make sure that the path to your dataset is defined, like in Figure 2-6.

# Path to the training data


data_path = '../data/train.csv'

Figure 2-6. Path to the Titanic training dataset defined

You may optionally define this to be the full, absolute path, just to make sure pandas
will find this file if you are having problems. Here, the data folder resides in a directory
level above our notebook, as it contains data files common to every chapter.
To load the data, we will use pd.read_csv(data_path), as shown in Figure 2-7. The
method read_csv() reads a csv file and all the data contained with it. It also can read
from other input formats, including JSON, SAS7BDAT, and Excel files. You can find more
details in the Pandas documentation: https://pandas.pydata.org/docs/reference/
index.html.

# pd.read_csv() returns a Pandas DataFrame object


df = pd.read_csv(data_path)

Figure 2-7. pd.read_csv() takes in the path as a parameter and returns a


pandas DataFrame

If this runs without producing any errors, you now have a pandas dataframe loaded.
An easy way to visualize this dataframe is to use df.head(N), where N is an optional
parameter to define how many rows you want to see. If you don’t pass N, the default is
five displayed rows. Simply run the line of code shown in Figure 2-8.

df.head(2)

Figure 2-8. Calling .head(2) on df to display two rows of df

You should see output that looks like Table 2-1.

29
30
Chapter 2

Table 2-1. Output of Executing the Code in Figure 2-8


PassengerId Survived Pclass Name Sex Age SibSp Parch Ticket Fare Cabin Embarked
Introduction to Data Science

0 1 0 3 Braund, Mr. Owen male 22.0 1 0 A/5 21171 7.2500 NaN S


Harris
1 2 1 1 Cumings, Mrs. female 38.0 1 0 PC 17599 71.2833 C85 C
John Bradley
(Florence Briggs
Th...
Chapter 2 Introduction to Data Science

To get the table’s dimensions, run the following:

df.shape

This returns a tuple (M, N) with the table’s dimensions. It shows M rows and N
columns. For this example, you should see the following printed output:

(891, 12)

Data Saving
To save your dataset, call df.to_csv(save_path), where save_path is a variable that
contains a string path to where you want to save the dataframe. (There are many other
output formats available besides csv.) As an example, run the code shown in Figure 2-9.

df2 = df.head(2)
df2.to_csv('two_rows.csv', index=False)

Figure 2-9. df.head(2) returns two rows of df, which is saved as df2. This is then
saved as a csv to two_rows.csv, with the parameter index=False passed. This
parameter tells pandas not to save the index as part of the csv, so an extra column
is not introduced into the data where there was none before

The parameter index=False tells pandas to not save the dataframe index to the csv.
Pandas creates an index when you load in data, which you can override with a custom
index if it is relevant. You can change this to index=True (which is the default) and see
how that changes the csv output.

DataFrame Creation
Besides loading data from a specific source, you can create a dataframe from
scratch given a list or a dictionary. This is very useful to do when you are conducting
experiments in an iterative manner over several different variable settings and you want
to save the data into a nice table.
The code shown in Figure 2-10 creates a dataframe of arbitrary metrics.

31
Chapter 2 Introduction to Data Science

metric_rows = [ [0.9, 0.2, 0.3], [0.8, 0.3, 0.2] ]


metrics = pd.DataFrame(metric_rows, columns=['Model1', ↵
'Model2', 'Model3'])
metrics

Figure 2-10. Creating a dataframe from a list of lists (two rows, three columns)

Note The ↵ symbol in Figure 2-10 (and subsequent code displays) indicates
that the code has been truncated and that it’s still the same line. So 'Model1',
'Model2', 'Model3']) is the actual ending of this line.

The output should look like Table 2-2.

Table 2-2. Output of Executing the Code in Figure 2-10


Model1 Model2 Model3

0 0.9 0.2 0.3


1 0.8 0.3 0.2

To create a dataframe from a dictionary, execute the code shown in Figure 2-11. In
this format, the keys of the dictionary are the columns themselves, and the values are
lists with the data corresponding to the keys.

metric_dict = {'Model1': [0.9, 0.8], 'Model2': [0.2, 0.3], ↵


'Model3':[0.3, 0.2]}
metrics = pd.DataFrame(metric_dict)
metrics

Figure 2-11. Creating a dataframe from dictionaries

After executing this code, you should once again see the dataframe in Table 2-2, as
this code creates the same output as the code shown in Figure 2-10.
Now that you know the very basics of data I/O in pandas, let’s move on to the many
ways we can manipulate this data to our liking.

32
Random documents with unrelated
content Scribd suggests to you:
Mr Cecil J. Sharp, published 1905.)
—— A Garland of Country Songs. 1895.

Kidson, Frank. Old English Country Dances gathered from scarce


printed collections, and from manuscript. 1890.
(Contains several hitherto unprinted folk-airs used
as dance tunes.)
—— Traditional Tunes: a collection of ballad airs, chiefly
obtained in Yorkshire and the South of Scotland. 1891.

Barrett, W. Alex. English Folk-Songs. Collected and arranged by


W. A. Barrett. Novello. 1891.

Broadwood, Lucy E., and J. A. Fuller Maitland. English County


Songs. Collected and edited by ——. Leadenhall Press,
and Cramer. 1893.
—— English Traditional Songs and Carols. Boosey. 1908.

Tozer, F. Sailors’ Songs and Chanties. Words by F. Davis, music


arranged from traditional sailors’ airs. Boosey.

Bradford, J., and A. Fagge. Old Sea Chanties. Collected and


arranged by ——. Metzler.

Graham, John. Dialect Songs of the North. Collected and edited


by ——. Curwen.

Sharp, Cecil J. Folk-Songs from Somerset, gathered and edited


with pianoforte accompaniments by Cecil J. Sharp (and
Rev. Charles L. Marson). 1905.
(Four more parts afterwards added. In the fourth
and fifth parts Mr Sharp’s name alone appears as
editor.)
—— Folk-Song Airs. Collected and arranged for the pianoforte
by ——. Books I. and II. Novello.
—— A School Series of Folk-Songs, in numbers. Novello.

Baring-Gould, Rev. S., and Cecil J. Sharp. English Folk-Songs for


Schools. 1906. Curwen.
Folk-Songs of England. Novello.
Book I. Folk-Songs from Dorset. Collected by H. E. D.
Hammond.
Book II. Folk-Songs from the Eastern Counties. Collected by
R. Vaughan Williams.
Book III. Folk-Songs from Hampshire. Collected by George B.
Gardiner.
Folk-Song Society, Journals of. 18 parts have been issued. (Hon.
Sec., 19 Berners Street.)

CAROLS
Gilbert, Davies. Some Ancient Christmas Carols with the tunes to
which they were formerly sung. 1822. Second edition,
1823.
(The tunes are eight in number.)

Sandys, William. Christmas Carols, Ancient and Modern, including


the most popular in the West of England. 1833.
—— Christmastide: its history, festivities and carols. c. 1852.

Rimbault, E. F. Christmas Carols. Edited and arranged by ——.


Chappell.

Husk, W. H. Songs of the Nativity, being Christmas Carols old


and new. c. 1865.
Helmore, Thomas. Christmas Carols. Novello. 1853.

Bramley, Rev. H. R., and Stainer, John. Christmas Carols old and
new. c. 1868. Reprinted in two small editions with a third
series added. Novello.
Fuller Maitland, J. A. English Carols of the Fifteenth Century,
from a MS. roll in the library of Trinity College, Cambridge,
with added parts by W. S. Rockstro.

Duncombe, W. D. V. A Collection of Old English Carols as sung at


Hereford Cathedral. Weekes.

Hill, Rev. Geoffry. Wiltshire Folk-Songs and Carols. 1904.

Broadwood, Lucy E. English Traditional Songs and Carols. Boosey.


1908.

Sharp, Cecil J. English Folk-Carols. Novello. 1911.

Gillington, Alice. Old Christmas Carols of the Southern Counties.


Curwen.

Shaw, Martin, and Percy Dearmer. The English Carol Book. 1913.

SINGING-GAMES
Gomme, Alice B. Traditional Games of England, Scotland, and
Ireland. 2 vols. 1894-1898.
—— Children’s Singing-Games, with the tunes to which they
are sung. Oblong. 2 series. 1894.
—— Old English Singing-Games.
Kidson, Frank, and Alfred Moffat. Eighty Singing-Games, old,
new, and adapted. Bayley & Ferguson.

Gillington, Alice E. Old Hampshire Singing-Games.


—— Old Isle of Wight Singing-Games.
—— Old Surrey Singing-Games. Curwen.

LITERATURE
Eckenstein, Lina. Comparative Studies in Nursery Rhymes. 1906.

Engel, Carl. The Study of National Music. 1866.


—— The Literature of National Music. 1879.

Glyn, Margaret H. Analysis of the Evolution of Musical Form.


1909.

Halliwell, James Orchard. The Nursery Rhymes of England,


obtained principally from Oral Tradition. 1843.
(A reprint from the volume contributed to the Percy
Society’s publications by Halliwell in 1841. A most
valuable work, several times, with the addition of
“Nursery Tales,” re-issued.)

Musical Association, Proceedings of. For 1904-1905-1907-1908.


(These years contain different papers on folk-
music.)

Sharp, Cecil J. English Folk-Songs: Some Conclusions. 1907.


ENGLISH FOLK-DANCE
BY MARY NEAL

ACKNOWLEDGMENTS
Acknowledgment is given with many thanks to—
Miss Lucy Broadwood, who directed attention to several works
of reference.

Miss A. G. Gilchrist, for permission to use the music of the


Moston rush-bearing Morris dance and for pointing out
its connection with “To-morrow shall be my dancing day.”

Miss Nellie Chaplin, for allowing me to use her copy of


Playford’s Dancing Master.
Mr Clive Carey, for reading proofs and other help.
Mr Walter Dodgson, for translating passages from Mysterium
und Mimus im Rigveda by Leopold von Schroeder.
Mr Brassington, of the Stratford-on-Avon Memorial Library, for
help in looking up works of reference.
Mr John Graham, for suggesting works of reference.
Mr John Beck, for lending Kemp’s Morris Dance Wonder.
Sir J. G. Frazer, whose work, The Golden Bough, is
indispensable to the study of folk-dance.
The Daily Chronicle, for permission to reproduce the pictures
of the Bampton dancers.
Messrs Curwen & Sons, for permission to reproduce two
pictures from the Espérance Morris Books.
Morris Dancers at Bampton-in-the-Bush, Oxon.

INTRODUCTION
Before the year 1905 few people knew that England possessed a traditional folk-
dance of her own, and fewer still realised that the national dances were still
practised on certain festival occasions in several villages and country towns, for the
most part in the Midland and Northern counties. The word “folk” when used to
describe a dance may be interpreted in two ways. It may be used to signify a dance
(either traditional or not) at one time popular amongst the people, or its meaning
may be limited to those dances whose origin is lost in antiquity and which have
been passed on from generation to generation by unlettered folk without the aid of
written music or written instruction as to steps or evolutions. That this book may be
of more use to those who wish either to study the available English dances or to
pass them on to the present generation, the wider meaning of the word “folk” will
be understood.
But a very marked distinction must be drawn between the two classes of folk-
dance, between those recorded in books and still danced by peasant folk as a
merely social dance, with no special significance beyond being an occasion for the
display of gallantry, coquetry, and the courtesies of social intercourse; and those
dances, until lately unrecorded, which are religious in origin and are the expression
in rhythm of primitive beliefs and magical ceremonial.
In the former are included the country dances and certain popular court dances.
The latter include the Sword dances, Morris dances, the Furry dance, and Horn
dance. As the origin of all dancing may be directly or indirectly traced to the
ceremonial of primitive religions, it will be well first of all to give some account of
those traditional dances still lingering in English villages which give unmistakable
signs of their origin.
The primitive forms of the traditional dance can only be guessed at by the
student of to-day, for probably every epoch, every generation, and every individual
dancer has added to, modified, or taken from the original dance, but enough is still
with us to make the study an intensely interesting one both from the archæological
and from the social point of view.
The most important surviving traditional dance in England to-day is undoubtedly
the Morris dance, both because of the far greater number of Morris dances still in
existence and because of the greater differences between the individual dances.
But very closely allied to, if not identical with, the Morris is the Sword dance, and
again allied to both is the Mummers’ play.
My own experience in talking to country dancers coincides with that of Mr Cecil
Sharp, who says that if you ask a sword dancer of Grenoside or Earsdon, he will
insist that he is a Morris dancer, and that one is often sent after a Morris dance,
only to find traces of a Mummers’ play. He adds, “In due course it will dawn upon
him (the collector) that the sword dancer of Northern England, the Morris dancer of
the Midlands and the South, and the Mummer of all England and Scotland are in
the popular view as one and pass under the same name.” And it is the word
“Morris” which gives the clue to the origin and nature of the dance, whatever the
precise form which it takes.
With one exception the dictionaries and glossaries I have consulted derive the
word “Morris” from “Moorish.” Mr Cecil Sharp says that the weight of testimony
must be held to show Morocco as the fount and origin of the dance,[1] but Mr John
Graham and Mr Kidson throw doubt on its Moorish origin. The fact that the Morisco,
supposed to be the counterpart of the English Morris, was a solo dance performed
with castanets, and the fact that amongst Orientals only women danced, whilst the
Morris is essentially a man’s dance, seems to me to put the Morris definitely into an
entirely different category from the Morisco, although both words are used for the
same dance.
Eventually I found what I believe to be the true derivation of the word in the
Glossary of C. Mackay, LL.D., published in 1887. He says that the word “Morris” is
probably of Keltic origin, and comes from “Mor,” great, and “uasal,” noble and
dignified. The final syllable was dropped in the course of ages, when Mor-uasal and
Mor-uiseil became Moruis, great, noble, stately, dignified, solemn. Dr Mackay
connects the dance with the Druidical Festival of Beltane (from Bal or Bael, the Sun-
god), and he says that all traditions of the Druids show the solemn importance
which they attached to May Day, or Beltane. Multitudes of devotees preceded by
three orders of the priesthood—priests, bards, and prophets—marched in solemn
procession to the top of a high hill to watch the kindling of a fire on May the first by
the direct agency of the sun. The solemn and mysterious dance around the fire
thus kindled appears to have been the origin of the Morris or Mor-uiseil dance.
Many writers have pointed out the curious resemblance between the dances of
the Salii and the English Morris and sword dances, and this resemblance adds to
the evidence in favour of our traditional dance having originated in sun worship and
nature worship generally.
The following description of the Salii from The Golden Bough will illustrate this
point:—“As priests of Mars, the god of Agriculture, the Salii probably had also
certain agricultural functions. They were named from the remarkable leaps which
they made. Now we have seen that dancing and leaping high are common
sympathetic charms to make the crops grow high. Was it one of the functions of
the Salii to dance and leap on the fields at the spring or autumn sowing, or at both?
“The dancing procession of the Salii took place in October as well as in March,
and the Romans sowed both in spring and autumn. The weapons borne by the Salii,
while effective against demons in general, may have been especially directed
against the demons who steal the seed corn in the ripe grain. In Western Africa the
field labours of tilling and sowing are sometimes accompanied by dances of armed
men in the fields.”
In a footnote the author also throws out a suggestion that, as the Salii were said
to have been founded by Morrius, King of Veii, and this seems to be etymologically
the same as Mamurius and Mars, the word “Morris” may be the same. In answer to
a query, however, he does not appear to take this as a really serious suggestion.[2]
The following are some of the reasons for connecting the Morris dance with
primitive religious customs:—
(1) The characteristic of the processional form of the dance as performed
by the living dancers to-day is a slow, dignified rhythmic movement,
which is very marked in the Bampton (Oxon.) dancers, who have an
unbroken tradition going back some hundreds of years.
The set dances display a much more lively character and are
characterised by wild leaps, twirlings round, hand-clapping, stick
clashing, and the waving of handkerchiefs, so that we can easily
imagine the present Morris as a descendant of the solemn
processional up the mountain-side to greet the morning sun, and
the scenes of wild joy on the summit at the appearance of the
source of light and life to his waiting worshippers.
(2) Many of the Morris and Sword dancers have evolutions which are
characteristic of ceremonial used by savage people in the worship
of the sun. The Abingdon (Berks) dances, a very old tradition, end
with a complete circle. Bean-setting, one of the Headington (Oxon.)
dances, begins with two half-circles danced in opposite directions.
The Bampton dances have circles, half-circles, and gypsies, another
form of circle. The Gloucestershire dances have the same. In some
dances the dancers advance and retire into the centre, forming a
widening and narrowing circle alternately, all of which illustrate by
mimetic action the supposed movements of the sun and the sun’s
rays. All were probably actuated by mimetic magic, primitive man
believing that by imitating the rising and setting of the sun and by
lighting fires he actually caused the return of the sun to the earth.
(3) The appearance in different forms of the King and Queen, the Lord
and Lady, the Mayor and Squire in the ceremonial of the dance.
These figures indubitably link up the dances with those ceremonies
attending the crowning of the King of the Wood, who, representing
the life of the earth’s vegetation, was yearly slain lest his vigour
might wane and all the green life of earth perish with it. The slaying
of this King and the revels which preceded it and the crowning of
the fresh and younger Monarch were all still dimly to be traced in
many revels and dances in English villages within quite recent
years.
At Abingdon, the story runs, two hundred years ago a great fight took place
between the dwellers in Ock Street and the rest of the burghers of the town. The
Ock Street people outnumbering the rest of the whole town thought they had the
right to appoint a Mayor. A beast was slain in the market-place and roasted whole,
a fight took place for the horns, and the winning side then carried the horns in the
Morris dancing round the town. The horns are in existence to-day and are carried
by the Mayor of the Morris accompanied by the Squire carrying a Sword. There are
traces left still of the gold used to tip the horns which are mounted on a bull’s head,
with flaming red nostrils, thus making it evident that the beast was regarded as
sacred. These ceremonies took place on St John the Baptist’s Eve, celebrating the
summer solstice.
Abingdon Dancers, whose tradition goes back to 1700
(“The Squire” holding the sword, wooden cup, and collection box, also the pole on
which is mounted the bull’s head and horns formerly carried by the “Mayor of the
Morris”)
At Kirtlington it was customary for the dancers to conduct ceremonially a young
maiden from her father’s house early in the morning. She must be of spotless
reputation, and dressed in white with floating blue ribbons. She stayed with the
dancers until night fell, when she was taken back to her father’s house. During the
time she was with the dancers she was regarded as sacred, and anyone who so
much as jostled her in the crowd must pay a fine of half a crown. Later a lamb
substituted for the maiden was decorated with flowers and ribbons, carried round
by the dancers, and at intervals put down while they danced round about it in a
circle.
At Kidlington (Oxon.) Blount describes a similar ceremony. “The Monday after
Whitsun week a fat lamb was provided, and the maidens of the town having their
thumbs tied together were permitted to run after it, and she who with her mouth
took hold of the lamb was declared the Lady of the Lamb, which was killed,
cleaned, and with the skin hanging on it was carried on a pole before the lady and
her companions to the green attended with music and a Morisco dance of men and
another of women. The rest of the day was spent in mirth and merry glee. Next day
the lamb, partly baked, partly boiled and partly roasted, was served up for the
lady’s feast, when she sat majestically at the upper end of the table and her
companions, with the music playing during the repast, which having finished the
solemnity ended.”
In most places where there are still lingering traces of the Morris there also
linger these traces of the ancient sacrifice of the King of the Wood, and of the
Worship of the Sun.
Another link with the festivals of ancient religions seems to be the constant use
of a mask in the traditional dance, or the disguising of the face with black, white, or
red paint. In The Golden Bough Sir James Frazer gives an account of a pagan
festival which may possibly account for this survival.
“In Mexico a Woman who represented the Mother of the Gods, the Earth
Goddess, after being feasted and entertained by sham fights for some days was
beheaded on the shoulders of a man. The body was flayed, and one of the men
clothing himself in the skin became the representative of the goddess Toci. The skin
of the thigh was removed separately, and the young man who represented the
Maize god, the son of the goddess Toci, wrapt it round his face as a mask. Various
ceremonies then followed in which the two men clad in the woman’s skin played the
parts respectively of god and goddess.”
To-day in England curious hints still survive which show that the simple country
folk never altogether lost the feeling that these dances were not quite ordinary, but
represented some sort of magic charm with which it would be unsafe to interfere.
Mr Sidney O. Addy, in his Household Tales, says:—“At Curbar, in Derbyshire, it is
said that Morris dancing is really fairy dancing, and that ‘Morris dancing’ means
‘fairy dancing.’ Morris dancers of the present day (1895), it is said, go through the
same form of dancing that the fairies go through, except that of course they cannot
perform such intricate figures as the fairies can. The figures which the Morris
dancers of the present day go through are very elaborate and very difficult to learn.
A man said to me ‘that Morris dancing had been taken away from the fairies.’ There
is something beautiful and strange in the music to which the Morris dancers dance.
If ever music was not of this world it is this. To hear it is to believe that Morris
dancing was a religious rite.“
The following extract seems to link up our English Morris dance with the Moorish
dance, so that whether we choose to derive the word “Morris” from the Keltic Mor-
uiseil, or from the Moorish, or whether we think that the similarity of the two words
made a confusion in the popular mind, and so the two kinds of dances came to be
known by one name, we can still hold the belief that the English traditional dance
which has come to us down the ages was originally a religious dance celebrating
the return of the Sun-god and the sowing and the gathering of the crops on which
man’s life depended.
Mr Addy asks:—“Has it (the Morris dance) descended to us from a dusky Iberian
people, once a distinct caste in England, in whose magical powers and religion the
dominant races believed? In his dictionary, Professor Skeat has concluded that a
Morris dancer was a Moorish dancer. Assuming that such is the case, we may ask
ourselves why these dances were so called. Are we to suppose that English
peasants borrowed the dance from the Moors in historical times? Or are we to
believe that it was handed down in England from an early period by the remnants
of a dark-coloured Iberian people who, according to Tacitus, crossed over from
Spain and were, in fact, Moors? In Yorkshire, a rude Christmas play known as the
Peace Egg is performed. In that play the chief act is the slaughter by St George of
England of a Black Prince of Paladine whom St George stigmatises as a ‘Black
Morocco Dog.’ The play seems to represent an old feud between a light-haired and
a dark-haired people once inhabiting England: and it may be that in popular speech
the dark-haired people were once known as Moors. If this dramatised contest
between St George of England and the ‘Black Morocco Dog’ does not point back to
a time when conflicts existed in this country between a dusky race of Iberian or
Moorish origin and a light-haired people which conquered and enslaved them, to
what can we ascribe its origin? We can only say that this play is of historical or
literary and not of traditional origin. But the form of the play renders an historical or
literary origin impossible, and the whole performance seems to be nothing else but
a rude and popular reminiscence of an ancient national feud.
“It seems relevant to mention here an old earthwork, extending for some miles
in length near Sheffield, known in one part of its course as Barber Balk. The
direction of the earthwork is from south-west to north-east, and the ditch is
uniformly on the southern side, as if it had been intended as a defence against
attack from that side. Some modern scholars identify the Barbars or Berbers, a
people inhabiting the Saracen countries along the north coast of Africa, with the
Iberians. Can it be that an invading Celtic people threw up this earthwork as a
defence against a dusky Iberian foe coming from the south, and that the ancient
name of the earthwork has been handed down from a remote time, thereby
preserving its true history? And is it not possible that the Iberians, the Morris or
Moor, the ‘Black Morocco Dog’ of the traditional play and the Barber are identical?
“A great authority on early Britain ‘has accepted and employed
the theory advanced by ethnologists that the early inhabitants of
this country were of Iberian origin.’”
The fact that the Morris dancers sometimes blackened their faces need not
necessarily mean that they wish to represent the Moors, but that they were thought
to represent Moors because their faces were blackened.
The Morris dance was called in some places the Northern lights and the Aurora
Borealis because of its desultory movements, and it may have been this which
inspired Milton to write—

“The sounds and seas, with all their finny drove,


Now to the Moon in wavering Morrice move.”

If, as I have tried to show, the traditional dance is part of an ancient religious
ceremonial dating from pre-Christian days, we shall not be surprised to find that in
Early Christian times the dance still found some place in the ceremonial of worship.
Sir Hubert Parry, in a chapter on dance rhythm in Grove’s Dictionary of Music,
says:—“Dance rhythm and dance gestures have exerted the most powerful
influence on music from prehistoric times until to-day. The analogy of a similar state
of things among uncultivated races still existing confirms the inherent probability of
the view that definiteness of any kind of music, whether of figure or phrase, was
first arrived at through connection with dancing. The beating of some kind of noisy
instrument as an accompaniment to gestures in the excitement of actual war or
victory or other such exciting cause was the first type of rhythmic music, and the
telling of tribal or national stories, of deeds of heroes in the indefinite chant
consisting of a monotone slightly varying with occasional cadences which is met
with among so many barbarous peoples, was the first type of vocal music.
“This vague approach to musical recitation must have received its first rhythmic
arrangement when it came to be accompanied by rhythmic gestures and the two
processes were thereby combined, while song and dance went on together as in
mediæval times in Europe.
“In Oratorio the importance of dance rhythm is shown by negative as well as
positive evidence. In the parts in which composers arrived at pure declamatory
music, the result, though often expressive, is hopelessly and inextricably indefinite
in form. But in most cases they submitted either openly or covertly to dance rhythm
in some part or other of their works.
“In Oratorio the dance influence maintained its place, but not so openly as in
Opera.”
In actual Church worship we find that rhythmic ball was played by bishop and
priests round the altar, and at the present day on Corpus Christi Day and other
festivals in the Cathedral at Seville the choir boys perform a dance.
The fact that to-day the Christian Festival of Whitsuntide is the most usual time
for Morris dancing in those places where it still survives is also an indication that the
pagan ceremonial dance was transferred to the Christian Church ceremonial in early
Christian times.
Several Churchwarden accounts giving items paid for Morris dancers’ clothes,
decorations, and regalia also point the same way. One at Kingston-on-Thames reads
thus—

For payneing of the Mores garments and for sarten gret


1508. £0 2 4
leveres
” For plyts and ¼ of laun for the Mores garments 0 2 11
” For Orsden for the same 0 0 10
” For bellys for the daunsars 0 0 12
1509- For silver paper for the Mores daunsars the frere,
10.
and Mayde Maryan at 1d. a peyne £0 5 4
1521- Eight yards of fustyan for the Mores daunsars’ coats
0 16 0
22.
” A dozyn of gold skynnes for the Mores 0 0 10
1536- Five hats and 4 porses for the daunsars
0 0 4½
37.

But a carol collected in 1833 from a peasant in West Cornwall and included in
William Sandys’ collection is the most interesting proof I have yet found of the
association between dancing and the Christian religion. Nothing more is known of
the Carol in spite of many inquiries which are still being pursued. This is the carol—

TO-MORROW SHALL BE MY DANCING DAY

To - mor - row shall be my danc - ing day, I

would my true love did so chance To see the

le - gend of my play, To call my true love


Chorus
to my dance. Sing, oh! my love, Oh, my

love, my love, my love, This have I done for my true love.

[Listen]
2. “Then was I born of a Virgin pure,
Of her I took fleshly substance:
Then was I knit to man’s nature,
To call my true love to my dance.
Sing oh! etc.

3. “In a manger laid and wrapp’d I was,


So very poor this was my chance,
Betwixt an ox and a silly poor ass,
To call my true love to my dance.
Sing oh! etc.

4. “Then afterwards baptised I was,


The Holy Ghost on me did glance,
My Father’s voice heard from above,
To call my true love to my dance.
Sing oh! etc.

5. “Into the desert I was led,


Where I fasted without substance:
The Devil bade me make stones my bread,
To have me break my true love’s dance.
Sing oh! etc.

6. “The Jews on me they made great suit,


And with me made great variance,
Because they loved darkness rather than light,
To call my true love to the dance.
Sing oh! etc.

7. “For thirty pence Judas me sold,


His covetousness for to advance;
Mark, where I kiss, the same do hold,
The same is he shall lead the dance,
Sing oh! etc.

8. “Before Pilate the Jews me brought,


When Barabbas had deliverance;
They scourg’d me and set me at nought,
Judged me to die to lead the dance.
Sing oh! etc.

9. “When on the cross hanged I was;


When a spear to my heart did glance,
There issued forth both water and blood,
To call my true love to the dance.
Sing oh! etc.

10. “Then down to Hell I took my way,


For my true love’s deliverance,
And rose again on the third day,
Up to my true love and the dance.
Sing oh! etc.

11. “Then up to Heaven I did ascend,


Where now I dwell in sure substance,
On the right hand of God, that man
May come into the general dance.
Sing oh! etc.”

Mr G. R. S. Mead thinks that this carol was originally sung by the mediæval
minstrels, jongleurs, and troubadours, who are said to have invented the word
carol, meaning a dance in which the performers moved slowly in a circle, singing as
they went. The Troubadours are responsible for the preservation of many fragments
of old mystery plays, and this carol is probably one such fragment, and as such is a
link between the definitely pagan folk-dance and through the Christian Church to
those alive in England to-day.
The following tune is taken from Miss A. G. Gilchrist’s Manuscript Collection, and
was noted and sent to her by Mr Smith Williamson, bandmaster of Moston, W.
Manchester, in 1907. Miss Gilchrist thinks it interesting in connection with the tune
of this Carol, as it is called “My love, my love,” and was played as a Morris dance at
the Rush-Cart ceremony at Moston up to forty-five or fifty years ago.

MOSTON RUSH-CART MORRIS TUNE,


“MY LOVE, MY LOVE”
[Listen]

The following account of the sacred all-night dance written by Philo (about a.d.
26) is quoted by Mr Mead in Quest, October 1910, and is interesting because the
dance described is curiously like the surviving processional dances which have
intervals in the processional when a set dance is performed.
“After the banquet they kept the sacred all-night festival. And this is how they
keep it. They all stand up in a body, and in the middle of the banqueting-place they
first form the Choroi, one of men and the other of women, and a leader and
conductor is chosen for each, the one whose reputation is greatest for a knowledge
of music: they then chant hymns composed in God’s honour in many metres and
melodies, sometimes singing together, sometimes one chorus beating the measure
with their hands for the antiphonal chanting of the other, now dancing to the
measure and now inspiring it, at times dancing in procession, at times set dances,
and then circle-dances right and left.” The latter part of this description might
almost have been taken down from some of the Morris dances danced to-day.
In Anglo-Saxon times the sword dance was in great repute, and Saxon nobles
kept dancers to amuse their guests. There is mention of Morris dancing in Edward
III.’s reign, when John of Gaunt returned from Spain, but few, if any, vestiges of it
can be traced in writers beyond the reign of Henry VII.: about which time, and
particularly in that of Henry VIII., the Churchwarden’s accounts in several parishes
show that the Morris dance made a very considerable figure in the parochial
festivals. Some of the accounts of the May Games of Robin Hood include a Morris
dance, but it is doubtful if the Morris was an intrinsic part of the Robin Hood
pageant, as it was very often danced on separate occasions altogether. I am
inclined to think that both are fragments of much older dances and dramas, and
that it is almost impossible to say what was their exact relation to one another. By
the time of Henry VIII. and Elizabeth, when references to the Morris dance are very
frequent, all idea of its religious significance had disappeared, and it represented
the characteristics of the English peasant in a holiday mood in the days when life
was a big adventure, and revelry and sport were rude and boisterous. It is a little
difficult to realise, as one watches the few remaining traditional dancers to-day,
either that their dancing has represented all that mankind knew of primitive
religious aspiration and ceremonial, or later that it embodied all the frolic and revel
of the rollicking days of Queen Elizabeth.
Although there is only one written record of steps and figures, there are so
many general descriptions of the dances in the writers of that time that it is a little
difficult, in the short space at my disposal, to choose which will give the best idea
of the dance as then performed. I have chosen a few descriptions which seem best
to fulfil this purpose.
But first there are four pictures of Morris dancing which may be here described.
The first is a painted glass window at Betley, in Staffordshire. It exhibits in all
probability the most curious as well as the oldest representation of an English May
game and Morris dance that is anywhere to be found. The dresses look as if they
belong to the reign of Edward IV., but the owner, Tollet, thought they were in the
time of Henry VIII.
Another early representation of a Morris dance is a copy of a very scarce
engraving on copper by Israhel Van Meckenem (died 1503), so named from the
place of his nativity, a German village in the confines of Flanders, in which latter
country this artist appears chiefly to have resided, and therefore in most of his
prints we may observe the Flemish costume of his time. From the pointed shoes
that we see in one of the figures it must have been executed between the years
1460 and 1470, about which latter period the broad-toed shoes came into fashion
in France and Flanders. It seems to have been intended as a pattern for goldsmith’s
work, probably a cup or tankard.
And thirdly, there is in the old Town Hall at Munich a series of ten figures of
Morris dancers, carved in wood by Erasmus Schnitznar in 1480. All these figures
have bells, and one has long streamers to his sleeves.
There is a fourth described by Walpole in his Catalogue of English Engravers,
under the name of Peter Stent. It is a painting at Lord Fitzwilliam’s on Richmond
Green, which came out of the old neighbouring palace. It was executed by
Vinckenkroom about the end of the reign of James I., and exhibits a view of the
above palace. A Morris dance is introduced, consisting of seven figures, viz.:—A
fool, hobby-horse, piper, Maid Marian, and three dancers, the rest of the figures
being spectators.
In a tract entitled Plaine Percevall the Peace Maker of England, 1590, mention is
made of a “stranger, which, seeing a quintessence (besides the Foole and Maid
Marian) of all the picked youth strained out of a whole endship, footing the Morris
about a May-pole, and he, not hearing the ministrelsie for the noise of the tabors,
bluntly demanded if they were not all beside themselves that they so lip’d and
skip’d without an occasion.”
Stubbes, in his Anatomie of Abuses, thus describes a Morris dance under the
title of the “Devil’s Daunce.”
Morris Dancers in the time of James I.

Description of the Lord of Misrule,


and attendant Morris
“First, all the wilde heads of the parish, flocking together, chuse them a graund
Captaine (of Mischiefe) whome they innoble with the title of my Lord of Misrule,
and him they crowne with great solemnitie, and adopt for their King. This king
annoynted, chooseth forth twentie, fourtie, three score or a hundred lustie guttes
like to himselfe to wait upon his Lordly Majesty, and to guard his noble person.
Then everyone of these his men he investeth with his liveries of greene, yellow, or
some other light wanton colour. And as though that were not (bawdy) gawdy
ynough, I should say, they bedecke themselves with scarffes, ribbons and laces
hanged all over with golde ringes, precious stones and other jewels: this done, they
tie about either legge twentie or fourtie belles with rich handkerchiefe in their
handes and sometimes laide a cross over their shoulders and neckes, borrowed for
the most part of their pretie Mopsies and loving Bessies, for bussing them in the
darke. Thus all things set in order, then have they their hobby-horses, their dragons
and other antiques together with their baudie pipers and thundering drummers, to
strike up the Devil’s Daunce, withall: then martch this heathen company towards
the Church and Church-yarde, their pypers pyping, their drummers thundering,
their stumpes dauncing, their belles iyngling, their handkercheefes fluttering about
their heades like madde men, their hobbie horses and other monsters skirmishing
amongst the throng: and in this sorte they goe to the Church (though the Minister
be at prayer or preaching), dauncing and swinging their handkercheefes over their
heads in the Church like Devils incarnate, with such a confused noise, that no man
can heare his owne voyce.”
To come to later times; in a curious story of a Country Squire who turned
Methodist and went about the country preaching, called “The Spiritual Quixote or
the Summer’s Ramble” of Mr Geoffry Wildgoose, a comic romance (1773), there is
an amusing account of a Morris dance.
“In the afternoon when they were got within a few miles of Gloucester at a
genteel house near the end of the village they saw almost the whole parish
assembled in the Court to see a set of Morrice dancers who (this holiday time),
dressed up in bells and ribbands, were performing for the entertainment of the
family of some company that had dined there.”

“Those who are acquainted with this sort of Morrice dance (which is still
practised in several parts of England) must know that they are usually attended
with one character called the Tom fool, who like the clown in the pantomime seems
to be a burlesque upon all the rest. His fool’s cap has a fox’s tail depending like a
ramillie whip: and instead of the small bells which the others wear on their legs he
has a great sheep-bell hung on his back side. Whilst the company therefore were all
attention to the preacher this buffoon contrived to slip the fool’s cap upon Tugwell’s
head, and to fix the sheep’s bell to his rump. Which Jerry no sooner perceived than
his choler arose, and spitting into his hands and clenching his fists he gave the Tom
fool a swinging blow in the face. Tugwell pursued with the sheep-bell at his tail.
Ended the preaching.”
At Abingdon-on-Thames the date on the regalia of the Morris dances still in
existence is 1700, and the Bampton Morris “side” claims an unbroken tradition, so
that in these places at any rate we are in touch with the dance as it has come to us
from the days when it was an inherent part of country life, and it is from these and
other isolated “sides” and individuals that the steps, figures, and tunes have been
taken down at the present day. A complete reconstruction of the dance is of course
impossible, so is an exact lesson of the way in which it should be danced, but with
the general descriptions and the remaining dancers enough can be ascertained to
justify the contention that England has a real folk-dance of her own which
compares very favourably with that of other nations.
[Listen]
[Listen]

Morris Dance and Music from the


Orchesographie of Thoinot-Arbeau

I. THE MORRIS DANCE TO-DAY


Before the revival of Morris dancing in 1905, there was only one description of
the steps and evolutions of the dance, and that was in Orchesographie et Traicte en
Forme de Dialogue, by Thoinot-Arbeau, published in 1588.
This is so interesting that I have had a photograph of it taken from the copy in
the British Museum.
The Morris dance may be roughly divided into five kinds—processional, corner,
handkerchief, stick dances, and solo jigs.
The corner dances are danced with handkerchiefs and so are the processional
and jigs, but there are also others where handkerchiefs take the place of sticks.
Since 1905 Mr Cecil Sharp has published instructions for the Morris dance in a
series called the Morris Book, with the tunes as played at the present day. The first
two volumes of the series were written in collaboration with Mr H. C. MacIlwaine,
and with the help of Miss Florence Warren, of the Espérance Guild of Morris
dancers, from whom the steps of most of the dances were taken down as she had
learned them from William Kimber, of Headington.
I have also edited, with the help of Mr Clive Carey, Mr Geoffrey Toye, and Miss
Florence Warren, two volumes called The Espérance Morris Book, and Mr John
Graham has collected Midland, Lancashire, and Cheshire dances in two volumes.[3]
It is probable that these books contain complete instructions in the steps and
figures of the dance and are a fairly complete collection of the existing dances, and
that others still in the Collector’s books, but not yet published, may only be variants.
There is a varied and extensive terminology used by the old dancers, and it is often
difficult to arrive at the exact meaning of certain expressions, for each set of
dancers has its own phraseology, which varies considerably from that of other sets
even when they are not many miles apart in locality. The following are some of the
terms used:—“Shake up” and “foot up” for the first figure of a dance; “hey up” or
“hey sides up,” “back to back,” “hands across,” and “capers.” “Gipsies” must be seen
to be understood, and “galley” is a turning round on your own axis with a single or
double shake of the leg, which seems to be better done the older and more shaky
the dancer is.
Each village has its own steps and its own evolutions, and the evolutions
generally follow the same order in each dance, the particular steps of that dance
being done between the evolutions.
This is a very usual order in which the dance is done—
Foot up or Shake up. Special step.
Special step. Back to back.
Hey up. Special step.
Special step. Hey up and All in.
Hands across.
But it would take a whole volume to describe each step done by each “side” of
dancers, and by the time this book was in print other variants would have arisen, if
any of the “sides” had danced in the meantime.
For practical purposes one has to decide on the most typical step one has seen
and adopt it for those to whom one is responsible for teaching the dances.
But the characteristics of all the dances are vigour and virility, and there is
nothing in the least like the posturing with pointed toe which characterises the
ordinary ball-room and stage dances.
The following is a complete list as far as I know of all the Morris dances
collected and published since the revival in 1905:—
Bean-setting. The Maid of the Mill.
Laudnum Bunches. Bobbing Joe.
Country Gardens. Glorisheers.
Constant Billy. The Gallant Hussar.
Trunkles. Leap Frog.
Rigs o’ Marlow. Shooting.
Bluff King Hal. Brighton Camp.
How d’ye do, Sir? Green Garters.
Shepherds’ Hey. Princess Royal.
Blue-eyed Stranger. Lumps of Plum Pudding.
Hey diddle Dis. The Fool’s Dance.
Hunting the Squirrel. Derbyshire Morris.
Getting up Stairs. Derbyshire Morris Reel.
Double set Back. The Cuckoo’s Nest.
Haste to the Wedding. The Monk’s March.
Rodney. Longborough Morris.
Processional Morris. Heel and Toe.
Jockie to the Fair. Bobby’s Joan.
Old Mother Oxford. Banks of the Dee.
Old Woman tossed up Dearest Dicky.
in a blanket. London Pride.
Bacca Pipes jig. Swaggering Boney.
Flowers of Edinburgh. Young Collins.
The Rose. All’s for the Best
Field Town Morris. and Richmond Hill.

Step Back.[4] The Boatman’s Song.


I’ll go and enlist for a Sailor. The Tight Little Island.
Sherborne Jig. The Girl I left behind Me.
None so Pretty. The Rose Tree and
Cross Caper, or The British Grenadiers.
Prince’s Royal. Garryowen.
We won’t go home With a Hundred Pipers.
till Morning. Ninety-five.
Abraham Brown. Draw Back.
Morris Off. Bumpus o’ Stretton.
Long Morris. Lively Jig.
Cross Morris. Morris On.
Three Cans Morris. Sally Luker.
Nancy Dawson. A Nutting we will go.
In addition to these Mr F. Kidson has also published a set of Country and Morris
dance tunes, but without instructions as to the dances.
Although this list gives a very fair idea of the traditional Morris dances still
lingering in country places, two things must be borne in mind—first, that many of
these dances with different names are practically the same dances; another tune
and a very slight alteration in the step is quite enough for a Morris dancer to say he
has another dance to show; and secondly, that the collectors have not yet finished
their work. I have in my possession quite a long list of people and places as yet
unvisited which may yield dances yet unrecorded, and Mr Cecil Sharp has
announced many dances and variants collected but not published.
The Folk-dance has been found in the following counties:—
Gloucestershire. Monmouthshire.
Oxfordshire. Yorkshire.
Berkshire. Lancashire.
Northamptonshire. Cheshire.
Lincolnshire. Northumberland.
Derbyshire. Warwickshire.
Nottinghamshire. Worcestershire.
Sussex. Surrey.
Cornwall.

II. TUNES
A word must be said about the tunes played for the dances by country
musicians to-day.
These tunes are, of course, of much later date than the Morris and Sword
dances, and probably contemporary with the original country dances. The musicians
took any tune which was popular at the time and adapted it to the dances, so that
the tunes are not by any means all traditional. As an instance of this, I remember
that old Mr Trafford, of Headington, told me that one day when he heard a military
band playing, he went and listened at the door of the barracks, and that he was so
attracted by the tune that he at once hummed it to the Morris dance fiddler and
adapted it to a Morris dance. To this day he likes the tune, which he calls “Buffalo
Gals,”[5] so much that he wanted Mr Carey to take it down and use it. There is no
doubt that at any given time the musicians used to adapt to the dances any popular
tune that took their fancy, and I think that probably the name of the dance was
altered to fit the tune. Anyway the tune which Mr Trafford liked, called “The Buffalo
Girls,” had certainly been taken for the name of a dance. The only dance tune that I
have been able to discover which has its dance steps attached to it is the one
before mentioned in Arbeau’s book. There is no doubt either that the nature of the
tunes changed considerably as the whittle and dub went out of fashion and were
superseded by the fiddle and later by the concertina, from which latter instrument
the first revived tunes were taken by Mr Sharp from William Kimber. The tunes
taken from the violin were more likely to be played in a modal scale; for instance,
Kimber played “The Rigs of Marlow” in the modern scale, but Mark Cox, who gave it
to us from the fiddle, played it in a modal form.
In the summer of 1912 the fiddler who played for the Morris dancers at
“Shakspeare’s England” in a few days played in a modal form a tune which had
been given him in the modern scale and was quite unconscious that he had altered
it.
Two Morris dance tunes, “Bean-setting” and “Laudnum Bunches,” do not seem
to be allied to any other forms of the airs.
In insisting on the traditional nature of the dances it is necessary to admit that
the same cannot be said of all, or even most, of the tunes played to-day.

III. MUSICAL INSTRUMENTS


In the earliest records of Morris dancing, the pipe and tabor, or whittle and dub,
were the musical instruments in use, and the oldest dancers to-day are never tired
of lamenting that the pipe and tabor to which they danced in their youth have gone
out of fashion.
A Morris dancer in Fleet Street, London, is described in a seventeenth century
manuscript in the British Museum (Harleian MS. 3910):

“In Fleet Strete then I heard a shoote:


I putt of my hatt, and I made no staye,
And when I came unto the roote,
Good Lord! I heard a taber play,
For so, God save me, a morrys-daunce.”

In the old play of Jacke Drums Entertainment (1601)—


The taber and pipe strike up a morrice.
A shoute within.
Ed. Oh, a morrice is come, observe our country sports,
’Tis Whitsun-tyde and we must frolick it.

Enter the Morrice.

The Song

“Skip it and trip it, nimbly, nimbly,


Tickle it, tickle it lustily,
Strike up the taber, for the wenches favour,
Tickle it, tickle it lustily.

Let us be seen in Hygate Greene,


To dance for the honour of Holloway.
Since we are come hither, let’s spare no leather,
To dance for the honour of Holloway.”

Later the fiddle took the place of pipe and tabor, and still more recently the
concertina.
The present-day fiddler at Bampton, Mr Wells, and Mr Mark Cox of Headington
are well worth a visit from musicians interested in the actual form in which the
tunes are played to-day by the musically unlettered.
Mr William Kimber, jun., of Headington, is also in possession of the old tunes,
which he plays skilfully on the concertina. Patience will be needed should the tunes
be noted, for very few musicians can repeat a phrase, even if it is the very last bar,
without going right back to the beginning of the tune. When the phrase is intricate,
and has to be often repeated, this means that a considerable amount of time is
taken up. The same applies to the dance; the traditional dancer is quite
unselfconscious, and if he is pulled up and asked for a repetition of a step, he
cannot give it, as a rule, without going back to the beginning of the dance; so that
in writing down the steps and evolutions of the dance much patience is needed and
understanding of the way in which the minds of simple folk work.
Within the memory of some of the oldest dancers the dancing was always
accompanied by singing, and old Master Druce, of Ducklington, told us that the
Morris could not be properly danced without singing. He could, however, only
remember a few of the words of one dance—“The Lollypop Man.”
The Bampton men gave us a few odd verses of one or two songs, but I am
afraid the real song will never be recovered, for, as one old man put it to a friend of
mine, “the words are too clumsy for girls.”
Miss Gilchrist gave me the words of a Lancashire Morris which we have often
used with very good effect—

“Morris dance is a very pretty tune,


Lads and lassies plenty,
Every lad shall have his lass,
And I’ll have four and twenty.

My new shoone they are so good,


I could dance Morris if I would,
And if hat and coat be dressed,
I will dance Morris with the best.

This is it and that is it,


And this is Morris dancing,
My poor father broke his leg,
And so it was a’chancing.

Bread and cheese and the old cow’s head


Roasted in a lantern,
A bit for me and a bit for you,
And a bit for the Morris dancer.”

Even in this doggerel there are traces of the old sacrificial rite of animal sacrifice
in which the head was considered as the most sacred part of the animal and much
coveted. It was generally awarded to the victor or victorious side in a fight for its
possession.

IV. THE DRESS


The Morris dance dress had two characteristics; it was the holiday attire of the
dancers and it had added to that certain special ceremonial features. These were
bells, ribbons, sticks or swords, and handkerchiefs.
In the Kingston-on-Thames Churchwardens’ Accounts (1536-37) the dresses of
the Morris dancers are thus described:—They consisted of four coats of white
fustian spangled, and two green satin coats with garters on which small bells were
fastened. In an old tract called “Old Meg of Herefordshire for a Mayde Marian, and
Hereford Town for a Morris Daunce” (1609), the musicians and the twelve dancers
have “long coats of the old fashion, high sleeves gathered at the elbows, and
hanging sleeves behind: the stuff red buffin striped with white, girdles with white
stockings, white and red roses to their shoes: the one Six, a white jew’s cap with a
jewel and a long red feather: the other a scarlet jew’s cap with a jewel and a white
feather.” Scarves, ribbands, and laces hung all over with gold rings, and even
precious stones are also mentioned in the time of Elizabeth. Miles, the Miller of
Ruddington, in Sampson’s play, “The Vow Breaker, or the Fayre Maid of Clifton”
(1636), says he is come to borrow “a few ribbands, bracelets, eare-rings, wyasyters
and silke girdle and hand-kerchers for a morris.”
Joe Miller, writing in 1874, gives the following description of the preparations of
a Morris dance:—“One Molly o’ Cheetham’s sent specially to London for bows and
flowerets to dress her Robin’s hat with, and Jenny of the Warden House Cottage
kept her thumb nail strapped up for a month to crimp her Billy’s ruffled shirt. She
was so feared of spoiling the edge of the nail: and Phœbe of the Dean Farm, took
Billy’s breeches to St Ann’s Square (Manchester) to have them laced with blue
ribbons and bows down the side. All the lasses of the village were as busy as bees,
making bows, getting up fine shirts, and tying white handkerchiefs with ribbons to
dance with.”
The late Mr Alfred Burton, writing in 1891, says:—“The costume worn now and
for many years past (colour being left to individual taste, except in the case of the
breeches, which are generally of the same colour and material in each band of
dancers) consists of shoes with buckles, white stockings, knee breeches tied with
ribbons, a brightly coloured scarf or sash round the waist, white shirt trimmed with
ribbons and fastened with brooches, and white straw hats decorated with ribbons
and rosettes. White handkerchiefs or streamers are tied to the wrist.”
Strutt, in his Sports and Pastimes of the People of England (1810), observes that
the garments of the Morris dancers were adorned with bells, which were not placed
there merely for the sake of ornament, but were to be sounded as they danced.
These bells were of unequal sizes and differently denominated, as the fore-bell, the
second-bell, the treble, and the tenor or great bell, and mention is also made of
double bells. Sometimes they used trebles only, but these refinements were of later
times. At first, these bells were small and numerous and affixed to all parts of the
body—the neck, shoulders, elbows, wrists, waist, knee, and ankle: the wrist, knee,
and ankle being, however, the principal places. The number of bells round each leg
sometimes amounted to from twenty to forty. They were occasionally jingled by the
hands.
The following description of a Morris dancer taken from Recreations for
Ingenious Head Pieces (1667) gives a very good idea of his appearance at that
date:—
Welcome to our website – the ideal destination for book lovers and
knowledge seekers. With a mission to inspire endlessly, we offer a
vast collection of books, ranging from classic literary works to
specialized publications, self-development books, and children's
literature. Each book is a new journey of discovery, expanding
knowledge and enriching the soul of the reade

Our website is not just a platform for buying books, but a bridge
connecting readers to the timeless values of culture and wisdom. With
an elegant, user-friendly interface and an intelligent search system,
we are committed to providing a quick and convenient shopping
experience. Additionally, our special promotions and home delivery
services ensure that you save time and fully enjoy the joy of reading.

Let us accompany you on the journey of exploring knowledge and


personal growth!

ebooknice.com

You might also like