Instant Download (Ebook) Beginning Anomaly Detection Using Python-Based Deep Learning: Implement Anomaly Detection Applications with Keras and PyTorch by Suman Kalyan Adari and Sridhar Alla ISBN 9798868800078, 8868800071 PDF All Chapters
Instant Download (Ebook) Beginning Anomaly Detection Using Python-Based Deep Learning: Implement Anomaly Detection Applications with Keras and PyTorch by Suman Kalyan Adari and Sridhar Alla ISBN 9798868800078, 8868800071 PDF All Chapters
https://ebooknice.com/product/beginning-anomaly-detection-using-
python-based-deep-learning-2nd-edition-54693456
ebooknice.com
https://ebooknice.com/product/beginning-anomaly-detection-using-
python-based-deep-learning-with-keras-and-pytorch-10795944
ebooknice.com
https://ebooknice.com/product/beginning-anomaly-detection-using-
python-based-deep-learning-54818762
ebooknice.com
https://ebooknice.com/product/beginning-mlops-with-mlflow-deploy-
models-in-aws-sagemaker-google-cloud-and-microsoft-azure-21976300
ebooknice.com
(Ebook) Beginning MLOps with MLFlow: Deploy Models in AWS SageMaker,
Google Cloud, and Microsoft Azure by Sridhar Alla, Suman Kalyan Adari
ISBN 9781484265482, 9781484265499, 1484265483, 1484265491
https://ebooknice.com/product/beginning-mlops-with-mlflow-deploy-
models-in-aws-sagemaker-google-cloud-and-microsoft-azure-22417650
ebooknice.com
https://ebooknice.com/product/finding-ghosts-in-your-data-anomaly-
detection-techniques-with-examples-in-python-47213874
ebooknice.com
https://ebooknice.com/product/finding-ghosts-in-your-data-anomaly-
detection-techniques-with-examples-in-python-47435122
ebooknice.com
https://ebooknice.com/product/deep-learning-with-python-develop-deep-
learning-models-on-theano-and-tensorflow-using-keras-7295338
ebooknice.com
https://ebooknice.com/product/control-charts-and-machine-learning-for-
anomaly-detection-in-manufacturing-34603960
ebooknice.com
Beginning Anomaly
Detection Using
Python-Based Deep
Learning
Implement Anomaly Detection
Applications with Keras and PyTorch
Second Edition
Introduction�������������������������������������������������������������������������������������������������������������xv
iii
Table of Contents
Networking���������������������������������������������������������������������������������������������������������������������������� 21
Medicine�������������������������������������������������������������������������������������������������������������������������������� 21
Video Surveillance����������������������������������������������������������������������������������������������������������������� 22
Environment�������������������������������������������������������������������������������������������������������������������������� 22
Summary������������������������������������������������������������������������������������������������������������������������������������ 22
iv
Table of Contents
vi
Table of Contents
Chapter 11: Practical Use Cases and Future Trends of Anomaly Detection ��������� 481
Anomaly Detection�������������������������������������������������������������������������������������������������������������������� 482
Real-World Use Cases of Anomaly Detection���������������������������������������������������������������������������� 485
Telecom������������������������������������������������������������������������������������������������������������������������������� 485
Banking������������������������������������������������������������������������������������������������������������������������������� 487
Environmental��������������������������������������������������������������������������������������������������������������������� 488
Health Care�������������������������������������������������������������������������������������������������������������������������� 490
Transportation��������������������������������������������������������������������������������������������������������������������� 493
Social Media������������������������������������������������������������������������������������������������������������������������ 494
Finance and Insurance�������������������������������������������������������������������������������������������������������� 495
Cybersecurity���������������������������������������������������������������������������������������������������������������������� 496
Video Surveillance��������������������������������������������������������������������������������������������������������������� 499
Manufacturing��������������������������������������������������������������������������������������������������������������������� 500
Smart Home������������������������������������������������������������������������������������������������������������������������� 503
Retail����������������������������������������������������������������������������������������������������������������������������������� 504
Implementation of Deep Learning–Based Anomaly Detection�������������������������������������������������� 504
Future Trends���������������������������������������������������������������������������������������������������������������������������� 506
Summary���������������������������������������������������������������������������������������������������������������������������������� 508
Index��������������������������������������������������������������������������������������������������������������������� 511
vii
About the Authors
Suman Kalyan Adari is currently a machine learning research engineer. He obtained a
B.S. in computer science at the University of Florida and an M.S. in computer science,
specializing in machine learning, at Columbia University. He has been conducting
deep learning research in adversarial machine learning since his freshman year at the
University of Florida and has presented at the IEEE Dependable Systems and Networks
workshop on Dependable and Secure Machine Learning held in Portland, Oregon,
USA in June 2019. Currently, he works on various anomaly detection tasks spanning
behavioral tracking and geospatial trajectory modeling.
He is quite passionate about deep learning, and specializes in various fields ranging
from video processing to generative modeling, object tracking, time-series modeling,
and more.
Sridhar Alla is the co-founder and CTO of Bluewhale, which helps organizations big
and small in building AI-driven big data solutions and analytics, as well as SAS2PY, a
powerful tool to automate migration of SAS workloads to Python-based environments
using Pandas or PySpark. He is a published author of books and an avid presenter
at numerous Strata, Hadoop World, Spark Summit, and other conferences. He also
has several patents filed with the US PTO on large-scale computing and distributed
systems. He has extensive hands-on experience in several technologies, including Spark,
Flink, Hadoop, AWS, Azure, TensorFlow, Cassandra, and others. He spoke on anomaly
detection using deep learning at Strata SFO in March 2019 and at Strata London in
October 2019. He was born in Hyderabad, India, and now lives in New Jersey with his
wife, Rosie, his daughters, Evelyn and Madelyn, and his son, Jayson. When he is not busy
writing code, he loves to spend time with his family and also training, coaching, and
organizing meetups.
ix
About the Technical Reviewers
Puneet Sinha has accumulated more than 12 years of work
experience in developing and deploying end-to-end models
in credit risk, multiple marketing optimization, A/B testing,
demand forecasting and brand evaluation, profit and price
analyses, anomaly and fraud detection, propensity modeling,
recommender systems, upsell/cross-sell models, modeling
response to incentives, price optimization, natural language
processing, and OCR using ML/deep learning algorithms.
xi
Acknowledgments
Suman Kalyan Adari
I would like to thank my parents, Krishna and Jyothi, my sister, Niha, and my loving dog,
Pinky, for supporting me throughout the entire process of writing this book as well as my
various other endeavors.
Sridhar Alla
I would like to thank my wonderful, loving wife, Rosie Sarkaria, and my beautiful,
loving children, Evelyn, Madelyn, and Jayson, for all their love and patience during the
many months I spent writing this book. I would also like to thank my parents, Ravi and
Lakshmi Alla, for their blessings and all the support and encouragement they continue
to bestow upon me.
xiii
Introduction
Congratulations on your decision to explore the exciting world of anomaly detection
using deep learning!
Anomaly detection involves finding patterns that do not adhere to what is
considered as normal or expected behavior. Businesses could lose millions of dollars
due to abnormal events. Consumers could also lose millions of dollars. In fact, there are
many situations every day where people’s lives are at risk and where their property is at
risk. If your bank account gets cleaned out, that’s a problem. If your water line breaks,
flooding your basement, that’s a problem. If all flights at an airport get delayed due to a
technical glitch in the traffic control system, that’s a problem. If you have a health issue
that is misdiagnosed or not diagnosed, that’s a very big problem that directly impacts
your well-being.
In this book, you will learn how anomaly detection can be used to solve business
problems. You will explore how anomaly detection techniques can be used to address
practical use cases and address real-life problems in the business landscape. Every
business and use case is different, so while we cannot copy and paste code and build a
successful model to detect anomalies in any dataset, this book will cover many use cases
with hands-on coding exercises to give you an idea of the possibilities and concepts
behind the thought process. All the code examples in the book are presented in Python
3.8. We choose Python because it is truly the best language for data science, with a
plethora of packages and integrations with scikit-learn, deep learning libraries, etc.
We will start by introducing anomaly detection, and then we will look at legacy
methods of detecting anomalies that have been used for decades. Then we will look
at deep learning to get a taste of it. Then we will explore autoencoders and variational
autoencoders, which are paving the way for the next generation of generative models.
Following that, we will explore generative adversarial networks (GANs) as a way to detect
anomalies, delving directly into generative AI.
Then we’ll look at long short-term memory (LSTM) models to see how temporal data
can be processed. We will cover temporal convolutional networks (TCNs), which are
excellent for temporal data anomaly detection. We will also touch upon the transformer
xv
Introduction
xvi
CHAPTER 1
Introduction to Anomaly
Detection
In this chapter, you will learn about anomalies in general, the categories of anomalies,
and anomaly detection. You will also learn why anomaly detection is important, how
anomalies can be detected, and the use case for such a mechanism.
In a nutshell, this chapter covers the following topics:
• What is an anomaly?
What Is an Anomaly?
Before you get started with learning about anomaly detection, you must first understand
what exactly you are targeting. Generally, an anomaly is an outcome or value that
deviates from what is expected, but the exact criteria for what determines an anomaly
can vary from situation to situation.
Anomalous Swans
To get a better understanding of what an anomaly is, let’s take a look at some swans
sitting by a lake (Figure 1-1).
1
© Suman Kalyan Adari, Sridhar Alla 2024
S. K. Adari and S. Alla, Beginning Anomaly Detection Using Python-Based Deep Learning,
https://doi.org/10.1007/979-8-8688-0008-5_1
Chapter 1 Introduction to Anomaly Detection
Let’s say that we want to observe these swans and make assumptions about the
color of the swans at this particular lake. Our goal is to determine what the normal
color of swans is and to see if there are any swans that are of a different color than this
(Figure 1-2).
Figure 1-2. More swans show up, all of which are white
2
Chapter 1 Introduction to Anomaly Detection
We continue to observe swans for a few years and all of them have been white. Given
these observations, we can reasonably conclude that every swan at this lake should be
white. The very next day, we are observing swans at the lake again. But wait! What’s this?
A black swan has just flown in (Figure 1-3).
Considering our previous observations, we thought that we had seen enough swans
to assume that the next swan would also be white. However, the black swan defies that
assumption entirely, making it an anomaly. It’s not really an outlier, which would be, for
example, a really big white swan or a really small white swan; it’s a swan that’s entirely
a different color, making it an anomaly. In our scenario, the overwhelming majority of
swans are white, making the black swan extremely rare.
In other words, given a swan by the lake, the probability of it being black is very
small. We can explain our reasoning for labeling the black swan as an anomaly with one
of two approaches (though we aren’t limited to only these two approaches).
First, given that a vast majority of swans observed at this particular lake are white, we
can assume that, through a process similar to inductive reasoning, the normal color for a
swan here is white. Naturally, we would label the black swan as an anomaly purely based
on our prior assumptions that all swans are white, considering that we’d only seen white
swans before the black swan arrived.
3
Chapter 1 Introduction to Anomaly Detection
Another way to look at why the black swan is an anomaly is through probability.
Now assume that there is a total of 1000 swans at this lake and only two are black swans;
the probability of a swan being black is 2 / 1000, or 0.002. Depending on the probability
threshold, meaning the lowest probability for an outcome or event that will be accepted
as normal, the black swan could be labeled as anomalous or normal. In our case, we will
consider it an anomaly because of its extreme rarity at this lake.
The intersections of the dotted lines have created several different regions containing
data points. Of interest is the bounding box (solid lines) created from the intersection of
both sets of dotted lines since it contains the data points for samples deemed acceptable
(Figure 1-5). Any data point outside of that specific box will be considered anomalous.
4
Chapter 1 Introduction to Anomaly Detection
Figure 1-5. Data points are identified as “good” or “anomaly” based on their
location
Now that we know which points are and aren’t acceptable, let’s pick out a sample from
a new batch of screws and check its data to see where it falls on the graph (Figure 1-6).
Figure 1-6. A new data point representing the new sample screw is generated,
with the data falling within the bounding box
The data for this sample screw falls within the acceptable range. That means that this
batch of screws is good to use since its density as well as tensile strength is appropriate
for use by the consumer. Now let’s look at a sample from the next batch of screws and
check its data (Figure 1-7).
5
Chapter 1 Introduction to Anomaly Detection
Figure 1-7. A new data point is generated for another sample, but this falls
outside the bounding box
The data falls far outside the acceptable range. For its density, the screw has abysmal
tensile strength and is unfit for use. Since it has been flagged as an anomaly, the
factory can investigate why this batch of screws turned out to be brittle. For a factory of
considerable size, it is important to hold a high standard of quality as well as maintain
a high volume of steady output to keep up with consumer demands. For a monumental
task like that, automation to detect any anomalies to avoid sending out faulty screws is
essential and has the benefit of being extremely scalable.
So far, we have explored anomalies as data points that are either out of place, in the
case of the black swan, or unwanted, in the case of faulty screws. So what happens when
we introduce time as a new variable?
6
Chapter 1 Introduction to Anomaly Detection
Assume the initial spike in expenditures at the start of the month is due to the
payment of bills such as rent and insurance. During the weekdays, our example person
occasionally eats out, and on the weekends goes shopping for groceries, clothes, and
various other items. Also assume that this month does not include any major holidays.
These expenditures can vary from month to month, especially in months with major
holidays. Assume that our person lives in the United States, in which the holiday of
Thanksgiving falls on the last Thursday of the month of November. Many U.S. employers
also include the Friday after Thanksgiving as a day off for employees. U.S. retailers have
leveraged that fact to entice people to begin their Christmas shopping by offering special
deals on what has colloquially become known as “Black Friday.” With that in mind, let’s
take a look at our person’s spending pattern in November (Figure 1-9). As expected, a
massive spike in purchases occurred on Black Friday, some of them quite expensive.
7
Chapter 1 Introduction to Anomaly Detection
Figure 1-9. Spending habits for the same person during the month of November
Now assume that, unfortunately, our person has had their credit card information
stolen, and the criminals responsible for it have decided to purchase various items of
interest to them. Using the same month as in the first example (Figure 1-8; no major
holidays), the graph in Figure 1-10 depicts what could happen.
Figure 1-10. Purchases in the person’s name during the same month as in
Figure 1-8
Let’s assume we have a record of purchases for this user going back many years.
Thanks to this established prior history, this sudden influx in purchases would be
flagged as anomalies. Such a cluster of purchases might be normal for Black Friday or
in the weeks before Christmas, but in any other month without a major holiday, it looks
8
Chapter 1 Introduction to Anomaly Detection
out of place. In this case, our person might be contacted by the credit card company to
confirm whether or not they made the purchases.
Some companies might even flag purchases that follow normal societal trends. What
if that TV wasn’t really bought by our person on Black Friday? In that case, the credit
card company’s software can ask the client directly through a phone app, for example,
whether or not they actually bought the item in question, allowing for some additional
protection against fraudulent purchases.
Taxi Cabs
As another example of anomalies in a time series, let’s look at some sample data for taxi
cab pickups and drop-offs over time for a random city and an arbitrary taxi company and
see if we can detect any anomalies.
On an average day, the total number of pickups can look somewhat like the pattern
shown in Figure 1-11.
Figure 1-11. Number of pickups for a taxi company throughout the day
From the graph, we see that there’s a bit of post-midnight activity that drops off
to near zero during the late-night hours. However, customer traffic picks up suddenly
around morning rush hour and remains high until the early evening, when it peaks
during evening rush hour. This is essentially what an average day looks like.
9
Chapter 1 Introduction to Anomaly Detection
Let’s expand the scope out a bit more to gain some perspective of passenger traffic
throughout the week (Figure 1-12).
Figure 1-12. Number of pickups for a taxi company throughout the week
As expected, most of the pickups occur during the weekday when commuters
must get to and from work. On the weekends, a fair amount of people still go out to get
groceries or just go out somewhere for the weekend.
On a small scale like this, causes for anomalies would be anything that prevents
taxis from operating or incentivizes customers not to use a taxi. For example, say that a
terrible thunderstorm hits on Friday. Figure 1-13 shows that graph.
Figure 1-13. Number of pickups for a taxi company throughout the week, with a
heavy thunderstorm on Friday
10
Chapter 1 Introduction to Anomaly Detection
The thunderstorm likely influenced some people to stay indoors, resulting in a lower
number of pickups than unusual for a weekday. However, these sorts of anomalies are
usually too small-scale to have any noticeable effect on the overall pattern.
Let’s take a look at the data over the entire year, as shown in Figure 1-14.
Figure 1-14. Number of pickups for a taxi company throughout the year
The largest dips occur during the winter months when snowstorms are expected.
These are regular patterns that can be observed at similar times every year, so they
are not an anomaly. But what happens to customer traffic levels when a relatively rare
polar vortex descends on the city in early April and unleashes several intense blizzards?
Figure 1-15 shows the graph.
11
Chapter 1 Introduction to Anomaly Detection
Figure 1-15. Number of pickups for a taxi company throughout the year, with a
polar vortex descending on the city in April
As you can see in Figure 1-15, the intense blizzards severely slowed down all traffic
in the first week of April and burdened the city in the following two weeks. Comparing
this graph to the graph shown in Figure 1-14, there’s a clearly defined anomaly in April
caused by the polar vortex. Since this pattern is extremely rare for the month of April, it
would be flagged as an anomaly.
Categories of Anomalies
Now that you have more perspective of what anomalies can be in various situations, you
can see that they generally fall into these broad categories:
• Context-based anomalies
• Pattern-based anomalies
12
Chapter 1 Introduction to Anomaly Detection
are expected to be present in the data set and can be caused by unavoidable random
errors or from systematic errors relating to how the data was sampled. Anomalies would
be outliers or other values that one doesn’t expect to exist. These data anomalies might
be present wherever a data set of values exist.
As an example of a data set in which data point–based anomalies may exist,
consider a data set of thyroid diagnostic values, where the majority of the data points are
indicative of normal thyroid functionality. In this case, anomalous values represent sick
thyroids. While they are not necessarily outliers, they have a low probability of existing
when taking into account all the normal data.
We can also detect individual purchases totaling excessive amounts and label
them as anomalies since, by definition, they are not expected to occur or have a very
low probability of occurrence. In this case, they are labeled as potentially fraudulent
transactions, and the card holder is contacted to ensure the validity of the purchase.
Basically, we can say this about the difference between anomalies and outliers:
we should expect a data set to include outliers, but we should not expect it to include
anomalies. Though the terms “anomaly” and “outlier” are sometimes interchanged,
anomalies are not always outliers, and not all outliers are anomalies.
Context-Based Anomalies
Context-based anomalies consist of data points that might seem normal at first but
are considered anomalies in their respective contexts. Returning to our earlier personal
spending example, we might expect a sudden surge in purchases near certain holidays,
but these purchases could seem unusual in the middle of August. The person’s high
volume of purchases on Black Friday was not flagged because it is typical spending
behavior for people on Black Friday. However, if the purchases were made in a month
where it is out of place given previous purchase history, it would be flagged as an
anomaly. This might seem similar to the example presented for data point–based
anomalies, but the distinction for context-based anomalies is that the individual purchase
does not have to be expensive. If a person never buys gasoline because they own an
electric car, sudden purchases of gasoline would be out of place given the context. Buying
gasoline is normal behavior for many people, but in this context, it is an anomaly.
13
Chapter 1 Introduction to Anomaly Detection
Pattern-Based Anomalies
Pattern-based anomalies are patterns and trends that deviate from their historical
counterparts, and they often occur in time-series or other sequence-based data. In the
earlier taxi cab company example, the customer pickup counts for the month of April
were pretty consistent with the rest of the year. However, once the polar vortex hit, the
numbers tanked visibly, resulting in a huge drop in the graph, labeled as an anomaly.
Similarly, when monitoring network traffic in the workplace, expected patterns of
network traffic are formed from constant monitoring of data over several months or
even years for some companies. If an employee attempts to download or upload large
volumes of data, it generates a certain pattern in the overall network traffic flow that
could be considered anomalous if it deviates from the employee’s usual behavior.
As another example of pattern-based anomalies, if an external hacker decided to hit
a company’s website with a distributed denial-of-service (DDoS) attack—an attempt
to overwhelm the server that handles network flow to a certain website in an attempt
to bring the entire website down or stop its functionality—every single attempt would
register as an unusual spike in network traffic. All of these spikes are clearly deviants
from normal traffic patterns and would be considered anomalous.
Anomaly Detection
Now that you have a better understanding of the different types of anomalies, we can
proceed to discuss approaches to creating models to detect anomalies. This section presents
a few approaches we can take, but keep in mind we are not limited to just these methods.
Recall our reasoning for labeling the black swan as an anomaly. One reason was
that since all the swans we have seen thus far were white, the single black swan is an
obvious anomaly. A statistical way to explain this reasoning is that as of the most recent
set of observations, we have one black swan and tens of thousands of white swans.
Thus, the probability of occurrence of the black swan is one out of tens of thousands
of all observed swans. Since this probability is so low, it would make the black swan an
anomaly just because we do not expect to see it at all.
The anomaly detection models we will explore in this book follow these approaches
either by training on unlabeled data, training on normal, nonanomalous data, or training
on labeled data for normal and anomalous data. In the context of identifying swans, we
would be told which swans are normal and which swans are anomalies.
14
Chapter 1 Introduction to Anomaly Detection
So, what is anomaly detection? Quite simply, anomaly detection is the process in
which an advanced algorithm identifies certain data or data patterns to be anomalous.
Falling under anomaly detection are the tasks of outlier detection, noise removal, novelty
detection, event detection, change point detection, and anomaly score calculation.
In this book, we will explore all of these as they are all basically anomaly detection
methods. The following tasks of anomaly detection are not exhaustive, but are some of
the more common anomaly detection tasks today.
Outlier Detection
Outlier detection is a technique that aims to detect anomalous outliers within a given
data set. As previously discussed, three methods that can be applied to this situation are
to train a model only on normal data to identify anomalies (by a high reconstruction error,
described next), to model a probability distribution in which anomalies would be labeled
based on their association with really low probabilities, or to train a model to recognize
anomalies by teaching it what an anomaly looks like and what a normal point looks like.
Regarding the high reconstruction error, think of it this way: the model trains on a
set of normal data and learns the patterns corresponding to normal data. When exposed
to an anomalous data point, the patterns do not line up with what the model learned to
associate with normal data. The reconstruction error can be analogous to the deviance in
learned patterns between the anomalous point and the normal points the model trained
on. We will formally go over reconstruction error in Chapter 6. Going back to the example
of the swans, the black swan is different based on the patterns that we learned by observing
swans at the lake, and was thus anomalous because it did not follow the color pattern.
Noise Removal
Noise removal involves filtering out any constant background noise in the data set.
Imagine that you are at a party and you are talking with a friend. There is a lot of
background noise, but your brain focuses on your friend’s voice and isolates it because
that’s what you want to hear. Similarly, a model learns to efficiently encode the original
sound to represent only the essential information. For example, encoding the pitch of
your friend’s voice, the tempo, the vocal inflections, etc. Then, it is able to reconstruct the
original sound without the anomalous interference noise.
15
Chapter 1 Introduction to Anomaly Detection
This can also be a case where an image has been altered in some form, such as by
having perturbations, loss of detail, fog, etc. The model learns an accurate representation
of the original image and outputs a reconstruction without any of the anomalous
elements in the image.
Novelty Detection
Novelty detection is very similar to outlier detection. In this case, a novelty is a data
point outside of the training set the model was exposed to, that was shown to the model
to determine if it is an anomaly or not. The key difference between novelty detection and
outlier detection is that in outlier detection, the job of the model is to determine what is
an anomaly within the training data set. In novelty detection, the model learns what is a
normal data point and what isn’t and tries to classify anomalies in a new data set that it
has never seen before.
Examples can include quality assurance in factories to make sure new batches of
created products are up to par, such as with the example of screws from earlier. Another
case is network security. Incoming traffic data can be monitored to ensure there is no
anomalous behavior going on. Both of these situations involve novelties (new data) to
constantly predict on.
Event Detection
Event detection involves the detection of points in a time-series dataset that deviate
anomalously from the norm. For example, in the taxi cab company example earlier in the
chapter, the polar vortex reduced the customer pickup counts for April. These deviations
from the norm for April were all associated with an anomalous event that occurred
in the dataset, which an event detector algorithm would identify. Another example of
event detection would be the tracking of sea-ice levels at the poles over time, forming a
time series. An event detector algorithm could flag exactly when sea-ice levels deviate
anomalously from the usual norm, such as occurred recently in July 2023, when the sea-
ice levels were detected to be six standard deviations below the established average.
in statistical trends. A good example is global average temperatures over time. A change
point detection algorithm could identify periods of sustained warming where the
statistical properties start to shift over time. A change point detection algorithm could
identify accelerated warming periods that differ from the normal rate of warming.
Supervised anomaly detection is a technique in which the training data has labels
for both anomalous data points and normal data points. Basically, we tell the model
during the training process if a data point is an anomaly or not. Unfortunately, this
isn’t the most practical method of training, especially because the entire data set needs
to be processed and each data point needs to be labeled. Since supervised anomaly
detection is basically a type of binary classification task, meaning the job of the model is
to categorize data under one of two labels, any classification model can be used for the
task, though not every model can attain a high level of performance. Chapter 9 provides
an example in the context of a temporal convolutional network.
17
Chapter 1 Introduction to Anomaly Detection
Semi-supervised anomaly detection involves partially labeling the training data set.
Exact implementations and definitions for what “semi-supervised” entails may differ, but
the gist of it is that you are working with partially labeled data. In the context of anomaly
detection, this can be a case where only the normal data is labeled. Ideally, the model will
learn what normal data points look like so that it can flag as anomalous data points that
differ from normal data points. Examples of models that can use semi-supervised learning
for anomaly detection include autoencoders, which you will learn about in Chapter 6.
Unsupervised anomaly detection, as the name implies, involves training the model
on unlabeled data. After the training process, the model is expected to know what data
points are normal and what points are anomalous within the data set. Isolation forest, a
model we will explore in Chapter 4, is one such model that can be used for unsupervised
anomaly detection.
Data Breaches
In today’s age of big data, where huge volumes of information are stored about users
in various companies, information security is vital. Any information breaches must be
reported and flagged immediately, but it is hard to do so manually at such a scale. Data
leaks can range from simple accidents, such as an employee losing a USB stick that
contains sensitive company information that someone picks up and accesses the data
on, to intentional actions, such as an employee intentionally sending data to an outside
party, or an attacker gaining access to a database via an intrusion attack. Several high-
profile data leaks have been widely reported in news media, from Facebook / Meta,
iCloud, and Google security breaches where millions of passwords or photos were
leaked. All of those companies operate on an international scale, requiring automation
to monitor everything in order to ensure the fastest response time to any data breach.
The data breaches might not even need network access. For example, an employee
could email an outside party or another employee with connections to rival companies
about travel plans to meet up and exchange confidential information. Of course, these
18
Chapter 1 Introduction to Anomaly Detection
emails would not be so obvious as to state such intentions directly. However, monitoring
these emails could be helpful as a post-breach analysis to find out anyone suspicious
from within the company, or as part of a real-time monitoring software to ensure data
confidentiality compliance across teams for example. Anomaly detection models can sift
through and process employee emails to flag any suspicious activity by employees. The
software can pick up key words and process them to understand the context and decide
whether or not to flag an employee’s email for review.
The following are a few more examples of how anomaly detection software can
detect an internal data breach:
In this case, something won’t add up, which the software will detect and then flag
the employee. It could either turn out to be a one-off sanctioned event, which great, the
model did its job but it was ok this time, or it could turn out that the employee somehow
accessed data they shouldn’t have, which would mean there was a data breach.
The key benefit to using anomaly detection in the workspace is how easy it is to scale
up. These models can be used for small companies as well as large-scale international
companies.
Identity Theft
Identity theft is another common problem in today’s society. Thanks to development of
online services allowing for ease of access when purchasing items, the volume of credit
and debit card transactions that take place every day has grown immensely. However,
this development also makes it easier to steal credit and debit card information or bank
account information, allowing the criminals to purchase anything they want if the card
19
Chapter 1 Introduction to Anomaly Detection
isn’t deactivated or if the account isn’t secured again. Because of the huge volume
of transactions, monitoring everything is difficult. However, this is where anomaly
detection can step in and help, since it is highly scalable and can help detect fraud
transactions the moment the request is sent.
As we discussed earlier, context matters. When a payment card transaction is made, the
payment card company’s anomaly detection software takes into account the card holder’s
previous history to determine if the new transaction should be flagged or not. Obviously, a
series of high value purchases made suddenly would raise alarms immediately, but what
if the criminals were smart enough to realize that and just make a series of purchases over
time that won’t put a noticeable hole in the card holder’s account? Again, depending on
the context, the software would pick up on these transactions and flag them again.
For example, let’s say that someone’s grandmother was recently introduced to
Amazon and to the concept of buying things online. One day, unfortunately, she
stumbles upon an Amazon lookalike website and enters her credit card information.
On the other side, some criminal takes it and starts buying random things, but not all
at once so as not to raise suspicion—or so he thought. The grandmother’s identity theft
insurance company starts noticing some recent purchases of batteries, hard drives, flash
drives, and other electronic items. While these purchases might not be that expensive,
they certainly stand out when all the prior purchases made by the grandmother
consisted of groceries, pet food, and various decorative items. Based on this previous
history, the detection software would flag the new purchases and the grandmother
would be contacted to verify these purchases. These transactions might even be flagged
as soon as an attempt to purchase is made. In this case, either the location of the
purchaser or the nature of the transactions could raise alarms and stop the transaction
from being successful.
Manufacturing
We have explored a use case of anomaly detection in manufacturing. Manufacturing
plants usually have a certain level of quality that they must ensure their products meet
before shipping them out. When factories are configured to produce massive quantities
of output at a near constant rate, it becomes necessary to automate the process of
checking the quality of various samples. Similar to the fictitious example of the screw
manufacturer, manufacturing plants in real life might test and use anomaly detection
software to ensure the quality of various metal parts, tools, engines, food, clothes, etc.
20
Chapter 1 Introduction to Anomaly Detection
Networking
Perhaps one of the most important use cases for anomaly detection is in networking.
The Internet is host to a vast array of various websites located on servers all around the
world. Unfortunately, due to the ease of access to the Internet, many individuals access
the Internet for nefarious purposes. Similar to the data leaks that were discussed earlier
in the context of protecting company data, hackers can launch attacks on websites as
well to leak their information.
One such example would be hackers attempting to leak government secrets
through a network attack. With such sensitive information as well as the high volumes
of expected attacks every day, automation is a necessary tool to help cybersecurity
professionals deal with the attacks and preserve state secrets. On a smaller scale, hackers
might attempt to breach a cloud network or a local area network and try to leak data.
Even in smaller cases like this, anomaly detection can help detect network intrusion
attacks as they happen and notify the proper officials. An example data set for network
intrusion anomaly detection is the KDD Cup 1999 data set. This data set contains a
large amount of entries that detail various types of network intrusion attacks as well as a
detailed list of variables for each attack that can help a model identify each type of attack.
Medicine
Anomaly detection also has a massive role to play in the field of medicine. For example,
models can detect subtle irregularities in a patient’s heartbeat in order to classify
diseases, or they can measure brainwave activity to help doctors diagnose certain
conditions. Beyond that, they can help analyze raw diagnostic data for a patient’s organ
and process it in order to quickly diagnose any possible problems, similarly to the
thyroid example discussed earlier.
Anomaly detection can even be used in medical imagery to determine whether a given
image contains anomalous objects. For example, suppose an anomaly detection model
was trained by exposing it only to MRI imagery of normal bones. When shown an image of
a broken bone, it would flag the new image as an anomaly. Similarly, anomaly detection
can even be extended to tumor detection, allowing for the model to analyze every image in
a full-body MRI scan and look for the presence of abnormal growth or patterns.
21
Chapter 1 Introduction to Anomaly Detection
Video Surveillance
Anomaly detection also has uses in video surveillance. Anomaly detection software can
monitor video feeds and flag any videos that capture anomalous action. While this might
seem dystopian, it can certainly help catch criminals and maintain public safety on
busy streets and other transportation systems. For example, this type of software could
potentially identify a mugging in a street at night as an anomalous event and alert the
nearest police department. Additionally, this type of software can detect unusual events
at crossroads, such as an accident or some unusual obstruction, and immediately call
attention to the footage.
Environment
Anomaly detection can be used to monitor environmental conditions as well. For
example, anomaly detection systems are used to monitor heavy-metal levels in rivers
to pick up potential spills or leaks into the water supply. Another example is air quality
monitoring, which can detect anything from seasonal pollen to wildfire smoke coming in
from far away. Additionally, anomaly detection can be utilized to monitor soil health in
agricultural or environmental survey cases to gauge the health of an ecosystem. Any dips
in soil moisture or specific nutrient levels could indicate some kind of problem.
Summary
Generally, anomaly detection is utilized heavily in medicine, finance, cybersecurity,
banking, networking, transportation, and manufacturing, but it is not limited to those fields.
For nearly every case imaginable involving data collection, anomaly detection can be put to
use to help users automate the process of detecting anomalies and possibly removing them.
Many fields in science can benefit from anomaly detection because of the large volume of
raw data collection that goes on. Anomalies that would interfere with the interpretation
of results or otherwise introduce some sort of bias into the data could be detected and
removed, provided that the anomalies are caused by systematic or random errors.
In this chapter, we discussed what anomalies are and why detecting anomalies can
be very important to the data processing we have at our organizations.
Next, Chapter 2 introduces you to core data science concepts that you need to know
to follow along for the rest of the book.
22
CHAPTER 2
Introduction to Data
Science
This chapter introduces you to the basic principles of the data science workflow. These
concepts will help you prepare your data as necessary to be fed into a machine learning
model as well as understand its underlying structure through analysis.
You will learn how to use the libraries Pandas, Numpy, Scikit-Learn, and Matplotlib
to load, manipulate, and analyze data. Furthermore, you will learn how to perform data
I/O; manipulate, transform, and process the data to your liking; analyze and plot the
data; and select/create relevant features for the machine learning modeling task.
In a nutshell, this chapter covers the following topics:
• Data I/O
• Data manipulation
• Data analysis
• Data visualization
• Data processing
• Feature engineering/selection
Note Code examples are provided in Python 3.8. Package versions of all
frameworks used are provided. You will need some type of program to run
Python code, so make sure to set this up beforehand. The code repository for this
book is available at https://github.com/apress/beginning-anomaly-
detection-python-deep-learning-2e/tree/master.
23
© Suman Kalyan Adari, Sridhar Alla 2024
S. K. Adari and S. Alla, Beginning Anomaly Detection Using Python-Based Deep Learning,
https://doi.org/10.1007/979-8-8688-0008-5_2
Chapter 2 Introduction to Data Science
The repository also includes a requirements.txt file to check your packages and their
versions.
Code examples for this chapter are available at https://github.com/apress/
beginning-anomaly-detection-python-deep-learning-2e/blob/master/Chapter%20
2%20Introduction%20to%20Data%20Science/chapter2_datascience.ipynb. Navigate
to “Chapter 2 Introduction to Data Science” and then click chapter2_datascience.
ipynb. The code is provided as a .py file as well, though it is the exported version of the
notebook.
We will be using JupyterLab (https://jupyter.org) to present all of the code
examples.
Data Science
“Data science” is quite the popular term and buzzword nowadays, so what exactly is it?
In recent times, the term “data science” has come to represent a wide range of roles and
responsibilities. Depending on the company, a data scientist can be expected to perform
anything from data processing (often at scale, dipping into “big data” territory), to
statistical analysis and visualization, to training and deploying machine learning models.
In fact, data scientists often perform two or more of these roles at once.
This chapter focuses on concepts from all three roles, walking you through the
process of preparing and analyzing the dataset before you explore the modeling aspect
in Chapter 3.
Be advised that this will be a brief, high-level walkthrough over the most relevant
functionality that these various data science packages offer. Each package is so
comprehensive that a full book would be required to cover it in depth. Therefore, you are
encouraged to explore each package’s online documentation, as well as other guides and
walkthroughs, to learn as much as you can.
Dataset
A popular introductory dataset for budding data scientists is the Titanic Dataset,
available at Kaggle: https://www.kaggle.com/c/titanic. You can also find this dataset
hosted on this book’s repository at https://github.com/apress/beginning-anomaly-
detection-python-deep-learning-2e/blob/master/data/train.csv.
24
Chapter 2 Introduction to Data Science
Kaggle is a great place to find datasets. It also hosts machine learning modeling
competitions, some of which even have prize money. Kaggle is an excellent resource
for practicing your skills, and you are encouraged to do so! If you would like to practice
the concepts that you learn in this book, search Kaggle for various anomaly detection
datasets.
To download the Titanic dataset from the Kaggle web site, follow the instructions
provided next. If you prefer, Kaggle offers an API that you can install through PIP,
available at https://github.com/Kaggle/kaggle-api.
Figure 2-1. Overview page for the Titanic dataset on Kaggle (as it looks as of
April, 2023)
2. Click the Data tab. You should see a brief description of the
dataset as well as a Data Explorer, as shown in Figure 2-2.
25
Chapter 2 Introduction to Data Science
Figure 2-2. Click Download All to download the data as a zip file
4. Return to the Data tab, scroll down, and click Download All. A zip
file will download.
5. Extract this zip file anywhere that you’d like and make a note of
the directory path.
After you have extracted the zip file, you are almost ready to start processing this
dataset in Python. First, make sure you have an IDE to develop Python code with. To
easily follow the examples in this book, you are recommended to use Jupyter Notebook
or JupyterLab.
Next, make sure you have the following libraries and versions installed, though you
might also want to check the requirements.txt file available on the GitHub repository or
use it to prepare your environment:
• pandas==2.0.0
• numpy==1.22.2
• scikit-learn==1.2.2
• matplotlib==3.7.1
26
Chapter 2 Introduction to Data Science
It is not necessary to have the exact same versions, but keep in mind that older
versions may not contain features we explore later in the book. Newer versions should be
fine unless they are significantly more recent, which might result in reworked syntax or
features and thus introduce incompatibility.
You can easily check the version in Python. Figure 2-3 introduces code to import
these packages and print their versions.
import pandas
import numpy
import sklearn
import matplotlib
Figure 2-3. Code to import pandas, numpy, sklearn, and matplotlib and print
their versions
Figure 2-4. The text output of running the code in Figure 2-3
27
Chapter 2 Introduction to Data Science
Data I/O
With our environment set up, let’s jump straight into the content. Before we conduct any
type of data analysis, we need to actually have data. There are a myriad of ways to load
data in Pandas, but we will keep it simple and load from a csv file.
For the sake of convenience, let’s reimport our libraries with aliases, as shown in
Figure 2-5.
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
Figure 2-5. Importing pandas, numpy, and matplotlib’s pyplot as aliases, for
convenience
Once you have executed this code, let’s move on to loading our dataset.
28
Chapter 2 Introduction to Data Science
Data Loading
First, make sure that the path to your dataset is defined, like in Figure 2-6.
You may optionally define this to be the full, absolute path, just to make sure pandas
will find this file if you are having problems. Here, the data folder resides in a directory
level above our notebook, as it contains data files common to every chapter.
To load the data, we will use pd.read_csv(data_path), as shown in Figure 2-7. The
method read_csv() reads a csv file and all the data contained with it. It also can read
from other input formats, including JSON, SAS7BDAT, and Excel files. You can find more
details in the Pandas documentation: https://pandas.pydata.org/docs/reference/
index.html.
If this runs without producing any errors, you now have a pandas dataframe loaded.
An easy way to visualize this dataframe is to use df.head(N), where N is an optional
parameter to define how many rows you want to see. If you don’t pass N, the default is
five displayed rows. Simply run the line of code shown in Figure 2-8.
df.head(2)
29
30
Chapter 2
df.shape
This returns a tuple (M, N) with the table’s dimensions. It shows M rows and N
columns. For this example, you should see the following printed output:
(891, 12)
Data Saving
To save your dataset, call df.to_csv(save_path), where save_path is a variable that
contains a string path to where you want to save the dataframe. (There are many other
output formats available besides csv.) As an example, run the code shown in Figure 2-9.
df2 = df.head(2)
df2.to_csv('two_rows.csv', index=False)
Figure 2-9. df.head(2) returns two rows of df, which is saved as df2. This is then
saved as a csv to two_rows.csv, with the parameter index=False passed. This
parameter tells pandas not to save the index as part of the csv, so an extra column
is not introduced into the data where there was none before
The parameter index=False tells pandas to not save the dataframe index to the csv.
Pandas creates an index when you load in data, which you can override with a custom
index if it is relevant. You can change this to index=True (which is the default) and see
how that changes the csv output.
DataFrame Creation
Besides loading data from a specific source, you can create a dataframe from
scratch given a list or a dictionary. This is very useful to do when you are conducting
experiments in an iterative manner over several different variable settings and you want
to save the data into a nice table.
The code shown in Figure 2-10 creates a dataframe of arbitrary metrics.
31
Chapter 2 Introduction to Data Science
Figure 2-10. Creating a dataframe from a list of lists (two rows, three columns)
Note The ↵ symbol in Figure 2-10 (and subsequent code displays) indicates
that the code has been truncated and that it’s still the same line. So 'Model1',
'Model2', 'Model3']) is the actual ending of this line.
To create a dataframe from a dictionary, execute the code shown in Figure 2-11. In
this format, the keys of the dictionary are the columns themselves, and the values are
lists with the data corresponding to the keys.
After executing this code, you should once again see the dataframe in Table 2-2, as
this code creates the same output as the code shown in Figure 2-10.
Now that you know the very basics of data I/O in pandas, let’s move on to the many
ways we can manipulate this data to our liking.
32
Random documents with unrelated
content Scribd suggests to you:
Mr Cecil J. Sharp, published 1905.)
—— A Garland of Country Songs. 1895.
CAROLS
Gilbert, Davies. Some Ancient Christmas Carols with the tunes to
which they were formerly sung. 1822. Second edition,
1823.
(The tunes are eight in number.)
Bramley, Rev. H. R., and Stainer, John. Christmas Carols old and
new. c. 1868. Reprinted in two small editions with a third
series added. Novello.
Fuller Maitland, J. A. English Carols of the Fifteenth Century,
from a MS. roll in the library of Trinity College, Cambridge,
with added parts by W. S. Rockstro.
Shaw, Martin, and Percy Dearmer. The English Carol Book. 1913.
SINGING-GAMES
Gomme, Alice B. Traditional Games of England, Scotland, and
Ireland. 2 vols. 1894-1898.
—— Children’s Singing-Games, with the tunes to which they
are sung. Oblong. 2 series. 1894.
—— Old English Singing-Games.
Kidson, Frank, and Alfred Moffat. Eighty Singing-Games, old,
new, and adapted. Bayley & Ferguson.
LITERATURE
Eckenstein, Lina. Comparative Studies in Nursery Rhymes. 1906.
ACKNOWLEDGMENTS
Acknowledgment is given with many thanks to—
Miss Lucy Broadwood, who directed attention to several works
of reference.
INTRODUCTION
Before the year 1905 few people knew that England possessed a traditional folk-
dance of her own, and fewer still realised that the national dances were still
practised on certain festival occasions in several villages and country towns, for the
most part in the Midland and Northern counties. The word “folk” when used to
describe a dance may be interpreted in two ways. It may be used to signify a dance
(either traditional or not) at one time popular amongst the people, or its meaning
may be limited to those dances whose origin is lost in antiquity and which have
been passed on from generation to generation by unlettered folk without the aid of
written music or written instruction as to steps or evolutions. That this book may be
of more use to those who wish either to study the available English dances or to
pass them on to the present generation, the wider meaning of the word “folk” will
be understood.
But a very marked distinction must be drawn between the two classes of folk-
dance, between those recorded in books and still danced by peasant folk as a
merely social dance, with no special significance beyond being an occasion for the
display of gallantry, coquetry, and the courtesies of social intercourse; and those
dances, until lately unrecorded, which are religious in origin and are the expression
in rhythm of primitive beliefs and magical ceremonial.
In the former are included the country dances and certain popular court dances.
The latter include the Sword dances, Morris dances, the Furry dance, and Horn
dance. As the origin of all dancing may be directly or indirectly traced to the
ceremonial of primitive religions, it will be well first of all to give some account of
those traditional dances still lingering in English villages which give unmistakable
signs of their origin.
The primitive forms of the traditional dance can only be guessed at by the
student of to-day, for probably every epoch, every generation, and every individual
dancer has added to, modified, or taken from the original dance, but enough is still
with us to make the study an intensely interesting one both from the archæological
and from the social point of view.
The most important surviving traditional dance in England to-day is undoubtedly
the Morris dance, both because of the far greater number of Morris dances still in
existence and because of the greater differences between the individual dances.
But very closely allied to, if not identical with, the Morris is the Sword dance, and
again allied to both is the Mummers’ play.
My own experience in talking to country dancers coincides with that of Mr Cecil
Sharp, who says that if you ask a sword dancer of Grenoside or Earsdon, he will
insist that he is a Morris dancer, and that one is often sent after a Morris dance,
only to find traces of a Mummers’ play. He adds, “In due course it will dawn upon
him (the collector) that the sword dancer of Northern England, the Morris dancer of
the Midlands and the South, and the Mummer of all England and Scotland are in
the popular view as one and pass under the same name.” And it is the word
“Morris” which gives the clue to the origin and nature of the dance, whatever the
precise form which it takes.
With one exception the dictionaries and glossaries I have consulted derive the
word “Morris” from “Moorish.” Mr Cecil Sharp says that the weight of testimony
must be held to show Morocco as the fount and origin of the dance,[1] but Mr John
Graham and Mr Kidson throw doubt on its Moorish origin. The fact that the Morisco,
supposed to be the counterpart of the English Morris, was a solo dance performed
with castanets, and the fact that amongst Orientals only women danced, whilst the
Morris is essentially a man’s dance, seems to me to put the Morris definitely into an
entirely different category from the Morisco, although both words are used for the
same dance.
Eventually I found what I believe to be the true derivation of the word in the
Glossary of C. Mackay, LL.D., published in 1887. He says that the word “Morris” is
probably of Keltic origin, and comes from “Mor,” great, and “uasal,” noble and
dignified. The final syllable was dropped in the course of ages, when Mor-uasal and
Mor-uiseil became Moruis, great, noble, stately, dignified, solemn. Dr Mackay
connects the dance with the Druidical Festival of Beltane (from Bal or Bael, the Sun-
god), and he says that all traditions of the Druids show the solemn importance
which they attached to May Day, or Beltane. Multitudes of devotees preceded by
three orders of the priesthood—priests, bards, and prophets—marched in solemn
procession to the top of a high hill to watch the kindling of a fire on May the first by
the direct agency of the sun. The solemn and mysterious dance around the fire
thus kindled appears to have been the origin of the Morris or Mor-uiseil dance.
Many writers have pointed out the curious resemblance between the dances of
the Salii and the English Morris and sword dances, and this resemblance adds to
the evidence in favour of our traditional dance having originated in sun worship and
nature worship generally.
The following description of the Salii from The Golden Bough will illustrate this
point:—“As priests of Mars, the god of Agriculture, the Salii probably had also
certain agricultural functions. They were named from the remarkable leaps which
they made. Now we have seen that dancing and leaping high are common
sympathetic charms to make the crops grow high. Was it one of the functions of
the Salii to dance and leap on the fields at the spring or autumn sowing, or at both?
“The dancing procession of the Salii took place in October as well as in March,
and the Romans sowed both in spring and autumn. The weapons borne by the Salii,
while effective against demons in general, may have been especially directed
against the demons who steal the seed corn in the ripe grain. In Western Africa the
field labours of tilling and sowing are sometimes accompanied by dances of armed
men in the fields.”
In a footnote the author also throws out a suggestion that, as the Salii were said
to have been founded by Morrius, King of Veii, and this seems to be etymologically
the same as Mamurius and Mars, the word “Morris” may be the same. In answer to
a query, however, he does not appear to take this as a really serious suggestion.[2]
The following are some of the reasons for connecting the Morris dance with
primitive religious customs:—
(1) The characteristic of the processional form of the dance as performed
by the living dancers to-day is a slow, dignified rhythmic movement,
which is very marked in the Bampton (Oxon.) dancers, who have an
unbroken tradition going back some hundreds of years.
The set dances display a much more lively character and are
characterised by wild leaps, twirlings round, hand-clapping, stick
clashing, and the waving of handkerchiefs, so that we can easily
imagine the present Morris as a descendant of the solemn
processional up the mountain-side to greet the morning sun, and
the scenes of wild joy on the summit at the appearance of the
source of light and life to his waiting worshippers.
(2) Many of the Morris and Sword dancers have evolutions which are
characteristic of ceremonial used by savage people in the worship
of the sun. The Abingdon (Berks) dances, a very old tradition, end
with a complete circle. Bean-setting, one of the Headington (Oxon.)
dances, begins with two half-circles danced in opposite directions.
The Bampton dances have circles, half-circles, and gypsies, another
form of circle. The Gloucestershire dances have the same. In some
dances the dancers advance and retire into the centre, forming a
widening and narrowing circle alternately, all of which illustrate by
mimetic action the supposed movements of the sun and the sun’s
rays. All were probably actuated by mimetic magic, primitive man
believing that by imitating the rising and setting of the sun and by
lighting fires he actually caused the return of the sun to the earth.
(3) The appearance in different forms of the King and Queen, the Lord
and Lady, the Mayor and Squire in the ceremonial of the dance.
These figures indubitably link up the dances with those ceremonies
attending the crowning of the King of the Wood, who, representing
the life of the earth’s vegetation, was yearly slain lest his vigour
might wane and all the green life of earth perish with it. The slaying
of this King and the revels which preceded it and the crowning of
the fresh and younger Monarch were all still dimly to be traced in
many revels and dances in English villages within quite recent
years.
At Abingdon, the story runs, two hundred years ago a great fight took place
between the dwellers in Ock Street and the rest of the burghers of the town. The
Ock Street people outnumbering the rest of the whole town thought they had the
right to appoint a Mayor. A beast was slain in the market-place and roasted whole,
a fight took place for the horns, and the winning side then carried the horns in the
Morris dancing round the town. The horns are in existence to-day and are carried
by the Mayor of the Morris accompanied by the Squire carrying a Sword. There are
traces left still of the gold used to tip the horns which are mounted on a bull’s head,
with flaming red nostrils, thus making it evident that the beast was regarded as
sacred. These ceremonies took place on St John the Baptist’s Eve, celebrating the
summer solstice.
Abingdon Dancers, whose tradition goes back to 1700
(“The Squire” holding the sword, wooden cup, and collection box, also the pole on
which is mounted the bull’s head and horns formerly carried by the “Mayor of the
Morris”)
At Kirtlington it was customary for the dancers to conduct ceremonially a young
maiden from her father’s house early in the morning. She must be of spotless
reputation, and dressed in white with floating blue ribbons. She stayed with the
dancers until night fell, when she was taken back to her father’s house. During the
time she was with the dancers she was regarded as sacred, and anyone who so
much as jostled her in the crowd must pay a fine of half a crown. Later a lamb
substituted for the maiden was decorated with flowers and ribbons, carried round
by the dancers, and at intervals put down while they danced round about it in a
circle.
At Kidlington (Oxon.) Blount describes a similar ceremony. “The Monday after
Whitsun week a fat lamb was provided, and the maidens of the town having their
thumbs tied together were permitted to run after it, and she who with her mouth
took hold of the lamb was declared the Lady of the Lamb, which was killed,
cleaned, and with the skin hanging on it was carried on a pole before the lady and
her companions to the green attended with music and a Morisco dance of men and
another of women. The rest of the day was spent in mirth and merry glee. Next day
the lamb, partly baked, partly boiled and partly roasted, was served up for the
lady’s feast, when she sat majestically at the upper end of the table and her
companions, with the music playing during the repast, which having finished the
solemnity ended.”
In most places where there are still lingering traces of the Morris there also
linger these traces of the ancient sacrifice of the King of the Wood, and of the
Worship of the Sun.
Another link with the festivals of ancient religions seems to be the constant use
of a mask in the traditional dance, or the disguising of the face with black, white, or
red paint. In The Golden Bough Sir James Frazer gives an account of a pagan
festival which may possibly account for this survival.
“In Mexico a Woman who represented the Mother of the Gods, the Earth
Goddess, after being feasted and entertained by sham fights for some days was
beheaded on the shoulders of a man. The body was flayed, and one of the men
clothing himself in the skin became the representative of the goddess Toci. The skin
of the thigh was removed separately, and the young man who represented the
Maize god, the son of the goddess Toci, wrapt it round his face as a mask. Various
ceremonies then followed in which the two men clad in the woman’s skin played the
parts respectively of god and goddess.”
To-day in England curious hints still survive which show that the simple country
folk never altogether lost the feeling that these dances were not quite ordinary, but
represented some sort of magic charm with which it would be unsafe to interfere.
Mr Sidney O. Addy, in his Household Tales, says:—“At Curbar, in Derbyshire, it is
said that Morris dancing is really fairy dancing, and that ‘Morris dancing’ means
‘fairy dancing.’ Morris dancers of the present day (1895), it is said, go through the
same form of dancing that the fairies go through, except that of course they cannot
perform such intricate figures as the fairies can. The figures which the Morris
dancers of the present day go through are very elaborate and very difficult to learn.
A man said to me ‘that Morris dancing had been taken away from the fairies.’ There
is something beautiful and strange in the music to which the Morris dancers dance.
If ever music was not of this world it is this. To hear it is to believe that Morris
dancing was a religious rite.“
The following extract seems to link up our English Morris dance with the Moorish
dance, so that whether we choose to derive the word “Morris” from the Keltic Mor-
uiseil, or from the Moorish, or whether we think that the similarity of the two words
made a confusion in the popular mind, and so the two kinds of dances came to be
known by one name, we can still hold the belief that the English traditional dance
which has come to us down the ages was originally a religious dance celebrating
the return of the Sun-god and the sowing and the gathering of the crops on which
man’s life depended.
Mr Addy asks:—“Has it (the Morris dance) descended to us from a dusky Iberian
people, once a distinct caste in England, in whose magical powers and religion the
dominant races believed? In his dictionary, Professor Skeat has concluded that a
Morris dancer was a Moorish dancer. Assuming that such is the case, we may ask
ourselves why these dances were so called. Are we to suppose that English
peasants borrowed the dance from the Moors in historical times? Or are we to
believe that it was handed down in England from an early period by the remnants
of a dark-coloured Iberian people who, according to Tacitus, crossed over from
Spain and were, in fact, Moors? In Yorkshire, a rude Christmas play known as the
Peace Egg is performed. In that play the chief act is the slaughter by St George of
England of a Black Prince of Paladine whom St George stigmatises as a ‘Black
Morocco Dog.’ The play seems to represent an old feud between a light-haired and
a dark-haired people once inhabiting England: and it may be that in popular speech
the dark-haired people were once known as Moors. If this dramatised contest
between St George of England and the ‘Black Morocco Dog’ does not point back to
a time when conflicts existed in this country between a dusky race of Iberian or
Moorish origin and a light-haired people which conquered and enslaved them, to
what can we ascribe its origin? We can only say that this play is of historical or
literary and not of traditional origin. But the form of the play renders an historical or
literary origin impossible, and the whole performance seems to be nothing else but
a rude and popular reminiscence of an ancient national feud.
“It seems relevant to mention here an old earthwork, extending for some miles
in length near Sheffield, known in one part of its course as Barber Balk. The
direction of the earthwork is from south-west to north-east, and the ditch is
uniformly on the southern side, as if it had been intended as a defence against
attack from that side. Some modern scholars identify the Barbars or Berbers, a
people inhabiting the Saracen countries along the north coast of Africa, with the
Iberians. Can it be that an invading Celtic people threw up this earthwork as a
defence against a dusky Iberian foe coming from the south, and that the ancient
name of the earthwork has been handed down from a remote time, thereby
preserving its true history? And is it not possible that the Iberians, the Morris or
Moor, the ‘Black Morocco Dog’ of the traditional play and the Barber are identical?
“A great authority on early Britain ‘has accepted and employed
the theory advanced by ethnologists that the early inhabitants of
this country were of Iberian origin.’”
The fact that the Morris dancers sometimes blackened their faces need not
necessarily mean that they wish to represent the Moors, but that they were thought
to represent Moors because their faces were blackened.
The Morris dance was called in some places the Northern lights and the Aurora
Borealis because of its desultory movements, and it may have been this which
inspired Milton to write—
If, as I have tried to show, the traditional dance is part of an ancient religious
ceremonial dating from pre-Christian days, we shall not be surprised to find that in
Early Christian times the dance still found some place in the ceremonial of worship.
Sir Hubert Parry, in a chapter on dance rhythm in Grove’s Dictionary of Music,
says:—“Dance rhythm and dance gestures have exerted the most powerful
influence on music from prehistoric times until to-day. The analogy of a similar state
of things among uncultivated races still existing confirms the inherent probability of
the view that definiteness of any kind of music, whether of figure or phrase, was
first arrived at through connection with dancing. The beating of some kind of noisy
instrument as an accompaniment to gestures in the excitement of actual war or
victory or other such exciting cause was the first type of rhythmic music, and the
telling of tribal or national stories, of deeds of heroes in the indefinite chant
consisting of a monotone slightly varying with occasional cadences which is met
with among so many barbarous peoples, was the first type of vocal music.
“This vague approach to musical recitation must have received its first rhythmic
arrangement when it came to be accompanied by rhythmic gestures and the two
processes were thereby combined, while song and dance went on together as in
mediæval times in Europe.
“In Oratorio the importance of dance rhythm is shown by negative as well as
positive evidence. In the parts in which composers arrived at pure declamatory
music, the result, though often expressive, is hopelessly and inextricably indefinite
in form. But in most cases they submitted either openly or covertly to dance rhythm
in some part or other of their works.
“In Oratorio the dance influence maintained its place, but not so openly as in
Opera.”
In actual Church worship we find that rhythmic ball was played by bishop and
priests round the altar, and at the present day on Corpus Christi Day and other
festivals in the Cathedral at Seville the choir boys perform a dance.
The fact that to-day the Christian Festival of Whitsuntide is the most usual time
for Morris dancing in those places where it still survives is also an indication that the
pagan ceremonial dance was transferred to the Christian Church ceremonial in early
Christian times.
Several Churchwarden accounts giving items paid for Morris dancers’ clothes,
decorations, and regalia also point the same way. One at Kingston-on-Thames reads
thus—
But a carol collected in 1833 from a peasant in West Cornwall and included in
William Sandys’ collection is the most interesting proof I have yet found of the
association between dancing and the Christian religion. Nothing more is known of
the Carol in spite of many inquiries which are still being pursued. This is the carol—
[Listen]
2. “Then was I born of a Virgin pure,
Of her I took fleshly substance:
Then was I knit to man’s nature,
To call my true love to my dance.
Sing oh! etc.
Mr G. R. S. Mead thinks that this carol was originally sung by the mediæval
minstrels, jongleurs, and troubadours, who are said to have invented the word
carol, meaning a dance in which the performers moved slowly in a circle, singing as
they went. The Troubadours are responsible for the preservation of many fragments
of old mystery plays, and this carol is probably one such fragment, and as such is a
link between the definitely pagan folk-dance and through the Christian Church to
those alive in England to-day.
The following tune is taken from Miss A. G. Gilchrist’s Manuscript Collection, and
was noted and sent to her by Mr Smith Williamson, bandmaster of Moston, W.
Manchester, in 1907. Miss Gilchrist thinks it interesting in connection with the tune
of this Carol, as it is called “My love, my love,” and was played as a Morris dance at
the Rush-Cart ceremony at Moston up to forty-five or fifty years ago.
The following account of the sacred all-night dance written by Philo (about a.d.
26) is quoted by Mr Mead in Quest, October 1910, and is interesting because the
dance described is curiously like the surviving processional dances which have
intervals in the processional when a set dance is performed.
“After the banquet they kept the sacred all-night festival. And this is how they
keep it. They all stand up in a body, and in the middle of the banqueting-place they
first form the Choroi, one of men and the other of women, and a leader and
conductor is chosen for each, the one whose reputation is greatest for a knowledge
of music: they then chant hymns composed in God’s honour in many metres and
melodies, sometimes singing together, sometimes one chorus beating the measure
with their hands for the antiphonal chanting of the other, now dancing to the
measure and now inspiring it, at times dancing in procession, at times set dances,
and then circle-dances right and left.” The latter part of this description might
almost have been taken down from some of the Morris dances danced to-day.
In Anglo-Saxon times the sword dance was in great repute, and Saxon nobles
kept dancers to amuse their guests. There is mention of Morris dancing in Edward
III.’s reign, when John of Gaunt returned from Spain, but few, if any, vestiges of it
can be traced in writers beyond the reign of Henry VII.: about which time, and
particularly in that of Henry VIII., the Churchwarden’s accounts in several parishes
show that the Morris dance made a very considerable figure in the parochial
festivals. Some of the accounts of the May Games of Robin Hood include a Morris
dance, but it is doubtful if the Morris was an intrinsic part of the Robin Hood
pageant, as it was very often danced on separate occasions altogether. I am
inclined to think that both are fragments of much older dances and dramas, and
that it is almost impossible to say what was their exact relation to one another. By
the time of Henry VIII. and Elizabeth, when references to the Morris dance are very
frequent, all idea of its religious significance had disappeared, and it represented
the characteristics of the English peasant in a holiday mood in the days when life
was a big adventure, and revelry and sport were rude and boisterous. It is a little
difficult to realise, as one watches the few remaining traditional dancers to-day,
either that their dancing has represented all that mankind knew of primitive
religious aspiration and ceremonial, or later that it embodied all the frolic and revel
of the rollicking days of Queen Elizabeth.
Although there is only one written record of steps and figures, there are so
many general descriptions of the dances in the writers of that time that it is a little
difficult, in the short space at my disposal, to choose which will give the best idea
of the dance as then performed. I have chosen a few descriptions which seem best
to fulfil this purpose.
But first there are four pictures of Morris dancing which may be here described.
The first is a painted glass window at Betley, in Staffordshire. It exhibits in all
probability the most curious as well as the oldest representation of an English May
game and Morris dance that is anywhere to be found. The dresses look as if they
belong to the reign of Edward IV., but the owner, Tollet, thought they were in the
time of Henry VIII.
Another early representation of a Morris dance is a copy of a very scarce
engraving on copper by Israhel Van Meckenem (died 1503), so named from the
place of his nativity, a German village in the confines of Flanders, in which latter
country this artist appears chiefly to have resided, and therefore in most of his
prints we may observe the Flemish costume of his time. From the pointed shoes
that we see in one of the figures it must have been executed between the years
1460 and 1470, about which latter period the broad-toed shoes came into fashion
in France and Flanders. It seems to have been intended as a pattern for goldsmith’s
work, probably a cup or tankard.
And thirdly, there is in the old Town Hall at Munich a series of ten figures of
Morris dancers, carved in wood by Erasmus Schnitznar in 1480. All these figures
have bells, and one has long streamers to his sleeves.
There is a fourth described by Walpole in his Catalogue of English Engravers,
under the name of Peter Stent. It is a painting at Lord Fitzwilliam’s on Richmond
Green, which came out of the old neighbouring palace. It was executed by
Vinckenkroom about the end of the reign of James I., and exhibits a view of the
above palace. A Morris dance is introduced, consisting of seven figures, viz.:—A
fool, hobby-horse, piper, Maid Marian, and three dancers, the rest of the figures
being spectators.
In a tract entitled Plaine Percevall the Peace Maker of England, 1590, mention is
made of a “stranger, which, seeing a quintessence (besides the Foole and Maid
Marian) of all the picked youth strained out of a whole endship, footing the Morris
about a May-pole, and he, not hearing the ministrelsie for the noise of the tabors,
bluntly demanded if they were not all beside themselves that they so lip’d and
skip’d without an occasion.”
Stubbes, in his Anatomie of Abuses, thus describes a Morris dance under the
title of the “Devil’s Daunce.”
Morris Dancers in the time of James I.
“Those who are acquainted with this sort of Morrice dance (which is still
practised in several parts of England) must know that they are usually attended
with one character called the Tom fool, who like the clown in the pantomime seems
to be a burlesque upon all the rest. His fool’s cap has a fox’s tail depending like a
ramillie whip: and instead of the small bells which the others wear on their legs he
has a great sheep-bell hung on his back side. Whilst the company therefore were all
attention to the preacher this buffoon contrived to slip the fool’s cap upon Tugwell’s
head, and to fix the sheep’s bell to his rump. Which Jerry no sooner perceived than
his choler arose, and spitting into his hands and clenching his fists he gave the Tom
fool a swinging blow in the face. Tugwell pursued with the sheep-bell at his tail.
Ended the preaching.”
At Abingdon-on-Thames the date on the regalia of the Morris dances still in
existence is 1700, and the Bampton Morris “side” claims an unbroken tradition, so
that in these places at any rate we are in touch with the dance as it has come to us
from the days when it was an inherent part of country life, and it is from these and
other isolated “sides” and individuals that the steps, figures, and tunes have been
taken down at the present day. A complete reconstruction of the dance is of course
impossible, so is an exact lesson of the way in which it should be danced, but with
the general descriptions and the remaining dancers enough can be ascertained to
justify the contention that England has a real folk-dance of her own which
compares very favourably with that of other nations.
[Listen]
[Listen]
II. TUNES
A word must be said about the tunes played for the dances by country
musicians to-day.
These tunes are, of course, of much later date than the Morris and Sword
dances, and probably contemporary with the original country dances. The musicians
took any tune which was popular at the time and adapted it to the dances, so that
the tunes are not by any means all traditional. As an instance of this, I remember
that old Mr Trafford, of Headington, told me that one day when he heard a military
band playing, he went and listened at the door of the barracks, and that he was so
attracted by the tune that he at once hummed it to the Morris dance fiddler and
adapted it to a Morris dance. To this day he likes the tune, which he calls “Buffalo
Gals,”[5] so much that he wanted Mr Carey to take it down and use it. There is no
doubt that at any given time the musicians used to adapt to the dances any popular
tune that took their fancy, and I think that probably the name of the dance was
altered to fit the tune. Anyway the tune which Mr Trafford liked, called “The Buffalo
Girls,” had certainly been taken for the name of a dance. The only dance tune that I
have been able to discover which has its dance steps attached to it is the one
before mentioned in Arbeau’s book. There is no doubt either that the nature of the
tunes changed considerably as the whittle and dub went out of fashion and were
superseded by the fiddle and later by the concertina, from which latter instrument
the first revived tunes were taken by Mr Sharp from William Kimber. The tunes
taken from the violin were more likely to be played in a modal scale; for instance,
Kimber played “The Rigs of Marlow” in the modern scale, but Mark Cox, who gave it
to us from the fiddle, played it in a modal form.
In the summer of 1912 the fiddler who played for the Morris dancers at
“Shakspeare’s England” in a few days played in a modal form a tune which had
been given him in the modern scale and was quite unconscious that he had altered
it.
Two Morris dance tunes, “Bean-setting” and “Laudnum Bunches,” do not seem
to be allied to any other forms of the airs.
In insisting on the traditional nature of the dances it is necessary to admit that
the same cannot be said of all, or even most, of the tunes played to-day.
The Song
Later the fiddle took the place of pipe and tabor, and still more recently the
concertina.
The present-day fiddler at Bampton, Mr Wells, and Mr Mark Cox of Headington
are well worth a visit from musicians interested in the actual form in which the
tunes are played to-day by the musically unlettered.
Mr William Kimber, jun., of Headington, is also in possession of the old tunes,
which he plays skilfully on the concertina. Patience will be needed should the tunes
be noted, for very few musicians can repeat a phrase, even if it is the very last bar,
without going right back to the beginning of the tune. When the phrase is intricate,
and has to be often repeated, this means that a considerable amount of time is
taken up. The same applies to the dance; the traditional dancer is quite
unselfconscious, and if he is pulled up and asked for a repetition of a step, he
cannot give it, as a rule, without going back to the beginning of the dance; so that
in writing down the steps and evolutions of the dance much patience is needed and
understanding of the way in which the minds of simple folk work.
Within the memory of some of the oldest dancers the dancing was always
accompanied by singing, and old Master Druce, of Ducklington, told us that the
Morris could not be properly danced without singing. He could, however, only
remember a few of the words of one dance—“The Lollypop Man.”
The Bampton men gave us a few odd verses of one or two songs, but I am
afraid the real song will never be recovered, for, as one old man put it to a friend of
mine, “the words are too clumsy for girls.”
Miss Gilchrist gave me the words of a Lancashire Morris which we have often
used with very good effect—
Even in this doggerel there are traces of the old sacrificial rite of animal sacrifice
in which the head was considered as the most sacred part of the animal and much
coveted. It was generally awarded to the victor or victorious side in a fight for its
possession.
Our website is not just a platform for buying books, but a bridge
connecting readers to the timeless values of culture and wisdom. With
an elegant, user-friendly interface and an intelligent search system,
we are committed to providing a quick and convenient shopping
experience. Additionally, our special promotions and home delivery
services ensure that you save time and fully enjoy the joy of reading.
ebooknice.com