Inside Deep Learning Math Algorithms Models Meap Edward Raff download
Inside Deep Learning Math Algorithms Models Meap Edward Raff download
https://ebookbell.com/product/inside-deep-learning-math-
algorithms-models-meap-edward-raff-22488472
Inside Deep Learning Math Algorithms Models Final Release Edward Raff
https://ebookbell.com/product/inside-deep-learning-math-algorithms-
models-final-release-edward-raff-42697134
https://ebookbell.com/product/inside-deep-learning-1st-edition-edward-
raff-169538870
https://ebookbell.com/product/deep-inside-the-underground-economy-how-
millions-of-americans-are-practising-free-enterprise-in-an-unfree-
economy-adam-cash-49163506
https://ebookbell.com/product/deep-inside-oscommerce-the-cookbook-
monika-math-979506
Deep Inside Virna Depaul
https://ebookbell.com/product/deep-inside-virna-depaul-47181650
https://ebookbell.com/product/the-wilderness-deep-inside-the-rlican-
partys-combative-contentious-chaotic-quest-to-take-back-the-white-
house-coppins-mckay-9338754
https://ebookbell.com/product/same-day-delivery-deep-inside-you-
minamino-sazan-44126082
The Boys In The Cave Deep Inside The Impossible Rescue In Thailand
Matt Gutman
https://ebookbell.com/product/the-boys-in-the-cave-deep-inside-the-
impossible-rescue-in-thailand-matt-gutman-7247342
Comfort Eating With Nick Cave Vegan Recipes To Get Deep Inside Of You
Vegan Cookbooks Joshua Ploeg
https://ebookbell.com/product/comfort-eating-with-nick-cave-vegan-
recipes-to-get-deep-inside-of-you-vegan-cookbooks-joshua-
ploeg-37513044
MEAP Edition
Manning Early Access Program
Inside Deep Learning
Math, Algorithms, Models
Version 1
To achieve this there are two overall parts to the book. In each chapter we will not only show
how to implement the code for these techniques but we'll show and explain the math behind
them too.
Part 1 will cover the basics. The framework we will use to implement our deep learning
models (PyTorch), and the three oldest and most common types of neural networks (they
come from the 1980s and earlier!). These are the foundations from which all the other
chapters will build. Once we have these foundations underfoot, we will learn the approaches
invented in the past ten years that helped revive neural networks. Combined, these give us a
set of “building blocks” that we can put together.
Part 2 will start building new kinds of approaches that move past the standard
classification and regression problems from normal machine learning. We'll do unsupervised
learning, object detection, generative modeling, and transfer learning as some major
concepts.
After almost a decade as a consultant I’ve had the opportunity to work on or assist with a
large number of topics in a variety of domains. The topics of these chapters have been chosen
based on what I’ve found useful to solving problems over my career thus far. It's all the things
—Edward Raff
1
The Mechanics of Learning
Deep Learning, also called neural networks, or artificial neural networks, has lead to dramatic
advances in the quality, accuracy, and usability machine learning. Technology that was considered
impossible just 10 years ago is now widely deployed or considered technically possible, even if
work remains. Digital assistants like Cortana, Google, Alexa and Siri are ubiquitous and can react
to natural spoken language. Self driving cars have been racking up millions of miles on the road
as they are refined for eventual deployment. We can finally catalog and calculate just how much
of the Internet is made of cat photos. Deep Learning has been instrumental to the success of all
of these use cases, and many more.
In this book I will give you a wide breadth of exposure to some of the most common and useful
techniques in deep learning today. A major focus will be not just how to use and code these
networks, but to understand how and why they work at a deep level. With a deeper
understanding you’ll be better equipped to select the best approach for your own problems.
Beyond being able to wield these tools effectively, you’ll also have the understanding to keep up
with advances in this rapidly progressing field. To make best use of this book, you should be
familiar with programming in python, and have some passing memory of a calculus, statistics,
and linear algebra course. You should also have some prior experience with ML, and it is OK if you
aren’t an expert. Topics from ML broadly will be quickly re-introduced, but our goal is to move
into new and interesting details about deep learning.
Let’s get a clearer idea of what Deep Learning is, and how this book will teach it. Deep Learning is
itself a sub-domain of Machine Learning, which is also a sub-domain of Artificial Intelligence.
Broadly 1, we could describe AI as getting computers to make decisions that look smart. I say
look because it is very hard to define what “smart” or “intelligence” truly is. These are deep
philosophical questions that we are not going to tackle. But the point is that AI should be making
decisions that we think are reasonable and what a “smart” person would do. Your GPS telling you
how to get home actually uses some old2 AI techniques to work, and taking the fastest route
home is a “smart” decision. Getting computers to play many video games has been done with
purely AI based approaches, where only the rules of the game are encoded - and one does not
need to actually show the AI how to play a single game of Chess. Figure 1.1 shows AI as the
outer most layer of these fields.
Figure 1.1: A (simplified) hierarchy of AI, ML, and Deep Learning. Each layer has a short description of how we might
characterized the added item that makes it different from the preceding layer. So a deep learning approach will
generally have the properties of ML and AI.
Machine Learning (ML) is when we start to give our AI examples of previous smart, and not so
smart, decisions. Telling it explicitly “this is what should have happened”. For example, we can
improve our chess playing AI by giving it example games from chess grand masters (which has a
winner and looser, so both a smart and not-as-smart set of decisions) to learn from as well.
Deep Learning in turn is not one “algorithm” in particular, but a whole slew of hundreds of small
“algorithms” that act like building blocks. Part of being a good practitioner is known the building
blocks you have available, and which ones to stick them together to create one larger model for
your problem. Each building block is designed to work well for certain kinds of problems, giving
the model valuable information. Figure 1.2 shows how we might compose different blocks
together to tackle three different situations. One of the goals in this book is to cover a wide
breadth of the different kinds of building blocks, so that you know and understand how they can
be used for different kinds of problems. Some of the blocks a very generic (“Data is a Sequence”
could be used for literally any kind of sequence) while others are more specific (“Data is an
Image” applies to only images), which will impact when and how you want to use them.
Figure 1.2: One of the defining characteristics of deep learning is the use of re-usable “blocks” that we can build
models from. Different blocks are useful for different kinds of data, and can be mixed-and-matched to deal with
different problems. Here we can see how five different blocks can be re-used to describe models for different types
of problems.
The first row shows two “image” blocks, in order to create a deep model. Applying blocks
repeatedly is where the “deep” in “deep learning” comes from. Adding depth makes a model
capable of solving more complex problems. This depth is often obtained by repeatedly stacking
the same kind of block multiple times. The second row in the figure shows a case for a sequence
problem. For example, text can be represented as a sequence of words. But not all words are
meaningful, and so we may want to give the model a block that helps it learn to ignore certain
words. The third row shows how we can describe new problems using the blocks we know about.
If we want our AI to watch a video and predict what is happening (e.g., “running”, “tennis”, or
“adorable puppy attack”) we can use the “image” and “sequence” block to create a “sequence of
images”, which is a verbose way of saying “a video”.
These different building blocks define our model, but like all of machine learning we also need
data and some mechanism for learning. When we say “learning”, we are not talking about the
way that humans learn. In machine (and deep) learning, the “learning” is the mechanical process
of getting the model to make the “smart looking” predictions on our data. This happens via a
process called “optimization” or “function minimization”. At the start, before we have seen any
data, our model is simply going to return random outputs because all of the parameters that
control the model are initialized to random values. By optimizing the blocks over the data, we
make our models learn. This gives us the larger picture in Figure 1.3.
Figure 1.3: The car of deep learning. The car is built from many different building blocks that we will learn about,
and we use different building locks to build cars for different tasks. But we need fuel and wheels to make the car go.
Our wheels are the task of learning which is done via a process called optimization, and the fuel is our data.
In most chapters of this book you will learn about new building blocks that you can use to build
deep learning models for different applications. You can think of each block as a kind of
(potentially very simple) algorithm. We will learn about the uses of each block, some explanation
about how or why they work, and how to combine them in code to create a new model. Thanks to
the nature of building blocks on top of each other, we will be able to ramp up from simple tasks
(e.g., simple prediction problems you could have tackled with your favorite non-deep ML
algorithm) to complex examples like machine translation (e.g., having a computer translate from
English to French). As we do this we will start with the basic approaches and methods that have
been in use since the 1960s for training and building neural networks, but using a modern
framework 3. As we progress through this book well build upon what we’ve learned in previous
chapters, introducing new blocks of using what we’ve learned to extend old blocks or build new
blocks from existing ones.
That said, this book is not a cook book of code snippets to just throw at any new problem. Rather,
the goal is to get you comfortable with the language that deep learning researchers use to
describe new and improved blocks they are creating, so that you can have some familiarity and
recognize when a new block may be useful to you. Math can often be used to express complex
changes very succinctly, so I will be sharing the math behind the building blocks.
We won’t do a lot of math, but we are going to show the math. When I say we won’t do math that
does not mean the book has none, but it means we are not going to derive or prove the math that
will be shown. Instead I’m going to present you with the final equations, explain what they mean
and do, and try to attach useful intuition to the different equations we will see through the book.
I’m calling it intuition because we will go through the bare minimum math needed to explain the
high level idea of what is happening, and showing exactly “why” a result is the way it is requires
more math than I’m asking you to have. As we show the math, we will interweave it with PyTorch
code whenever possible. This way you can start to build a mental map between equations and
deep learning code.
For for this chapter, we are first going to get introduced to our compute environment Google
Colab. Next, we can start talking about PyTorch and tensors, which is how we represent
information inside of PyTorch. After that well dive into the use of Graphics Processing Units
(GPUs) that make PyTorch fast, and automatic differentiation which is the “mechanics” PyTorch
leverages to make our neural network models learn. Finally, we will quickly implement a dataset
object which is what PyTorch will need to feed our data into the model for the learning process.
This gives us the fuel and wheels to get our deep learning car moving starting in chapter 2. From
there on we can focus on just the deep learning.
Unfortunately, GPUs are also decently expensive. The best option currently for most people who
want to get started with deep learning is to spend\$600-\$1200 on one of the higher end Nvidia
GTX or Titan GPUs. That is if you have a computer that can be expanded/upgraded with a high
end GPU. If not, you are probably looking at at least \$1500-\$2500 to build a nice workstation to
put those GPUs in. Thats a pretty steep cost just to learn about deep learning.
Google’s Colab (https://colab.research.google.com) provides a GPU for free, for a limited amount
of time. I’ve designed every example in this book to run within Colab’s time limits. Appendix A
contains the instructions for setting up Colab. Once you have it setup, the common data science
and machine learning tools like seaborn, matplotlib, tqdm, and pandas are all built in and ready to
go. Colab operates as a familiar Jupyter notebook, where you run code in “cells” which produce an
output directly below. This book is in fact a Jupyter notebook, so you can run the code blocks (like
the next one) to get the same results. I’ll let you know if a code cell isn’t meant to be run.
import seaborn as sns
import matplotlib.pyplot as plt
import numpy as np
from tqdm.autonotebook import tqdm
import pandas as pd
As we progress through this book, we are not going to repeatedly show all of the imports, as that
is mostly a waste of paper. Instead they will be available on-line as part of the downloadable
copies of the code, which can be found at https://github.com/EdwardRaff/Inside-Deep-Learning.
W E have much pleasure (said the Evening Moon, two days after
the fire) in presenting our readers with the last act of a drama
which, in plot, incident, and extraordinary development of character,
equals anything in the way of sensationalism which has ever graced
theatrical boards. The opportunity is an agreeable one to us, as it
enables us to do justice to a gentleman who has had reason to
complain of what has appeared in our columns concerning him.
What we have to say resolves itself into something more than the
last act of a drama; it is both that and the commencement of a
Sequel which, in all human probability, and because of the nature of
the persons engaged in it, will have a happier ending than that
which has been closed by the burning down of the house, No. 119,
Great Porter Square.
In our yesterday’s issues we gave the full particulars of that fire.
No one was injured except the two wretched beings who met their
just and awful fate in the grave they had prepared for themselves.
They have passed away from this world, but it will be long before
the memory of their crime and its involvements will be forgotten. It
has been determined to pull down the fatal house in which the
murder was committed, and to rebuild it anew. The house next to it,
No. 118, occupied by Mrs. Preedy, lodging-house keeper, received
some damage from the fire; but Mrs. Preedy is fully insured, and her
loss will be a gain to her—a paradox, but strictly accurate, for the
murder in the adjoining house had brought hers into disrepute, and
her business was languishing. It will revive now that the fire has
burnt out the terror of the crime; and the worthy Mrs. Preedy may
congratulate herself upon having gained friends in the persons of Mr.
Frederick Holdfast and the intrepid, noble-hearted lady who will
shortly bear his name.
In Mrs. Preedy’s house lived an old bedridden lady, Mrs. Bailey,
whose life was with some difficulty saved. She herself placed serious
obstacles in the way of her preservation, screaming out when they
attempted to remove her from her bed. She clung to this household
god with such tenacity that there was nothing for it but to humour
the old lady, and to remove it with her. As they carried it down
stairs, the covering was by an accident ripped, and there rolled out
of it between thirty and forty sovereigns, which Mrs. Bailey had
hoarded up since the death of her husband, an event which occurred
Heaven knows how many years ago. The distress of the old lady was
extreme, but the gold was picked up and returned to its owner,
minus a few sovereigns, which somehow had stuck to the fingers of
the searchers. She is, however, no loser by the accident, as Mr.
Frederick Holdfast made good the deficiency. It is satisfactory to
learn that a cherished tradition current in Great Porter Square, that
the old lady’s mattress was stuffed with gold, was verified by the
ripping of the sacking. Mrs. Bailey will no doubt find another safe for
her treasure in the future. The bedridden old lady sustained a loss in
the burning of a linnet without a note to its voice, and a very old
bull-finch, whose cage hung at the foot of her bed—a sacrifice of
life, in addition to the more terrible sacrifice of two human beings,
which we were almost forgetting to mention.
In another part of our paper will be found a full report of the
proceedings at the inquest upon the bodies of the man and woman,
which were found in the back room of No. 119, Great Porter Square.
The inquest was held this morning, and a verdict of accidental death
by burning was returned. As a rule such inquests are dull, miserable
affairs, and there is but little variety in the evidence presented to the
coroner and his panel, but in this special case were elements of
unexpected romance which raised it far above the ordinary level of a
simple death by misadventure.
Last evening a private note was sent to our office, signed by
Frederick Holdfast, requesting as an act of justice, that the Special
Reporter who wrote “The Romance of Real Life” from Mrs. Holdfast’s
account of her career and misfortunes, should attend and take
whatever notice of the proceedings he might deem fit and proper. In
accordance with the request our Special Reporter attended, and the
present report is written by him for our paper. The disclosures which
were made at the inquest were as interesting as they were
surprising, and our Reporter thanks Mr. Frederick Holdfast for the
opportunity afforded him of being present.
At the inquest our Reporter renewed his acquaintance with Mr.
Goldberry, solicitor, a gentleman whose name will be remembered as
having voluntarily come forward to defend Antony Cowlrick at the
Martin Street Police Court, when, upon the barest suspicion, without
a tittle of direct evidence, that person was accused by the police of
the murder of a man unknown in No. 119, Great Porter Square. Our
readers will remember how stoutly, and under what disadvantages,
Mr. Goldberry defended the man wrongfully accused of the crime;
how he protested against the numerous remands, and lifted up his
voice in the cause of justice against Scotland Yard officialism; and
how at length, to the manifest chagrin of the police, Antony Cowlrick
was discharged from custody. The particulars of the interview which
took place in Leicester Square, a few minutes after Antony Cowlrick’s
departure from the Police Court, between our Reporter, Mr.
Goldberry, and the accused man, was fully reported in our columns.
In that interview our Reporter lent Antony Cowlrick a sovereign,
which was faithfully repaid. We purpose reprinting in a pamphlet
that report and the “Romance in Real Life,” in addition to what
appears in our present issue relating to the case. They are worthy of
a record in a more permanent form than the columns of a
newspaper.
“Do you remember,” said Mr. Goldberry to our Reporter, referring
to that interview, “that Antony Cowlrick said to me that if at any time
he should need my services, he would call upon or send for me?”
“I do,” replied our Reporter, “and I remember, also, that Antony
Cowlrick asked you if you thought God would allow the guilty to
escape, or that He needed the assistance of a lawyer to punish the
man who shed another’s blood.”
“Yes,” said Mr. Goldberry, gravely, “he used those words, and in
this case they are justified by events. God has punished the
murderers without the assistance of a lawyer.”
“Why do you recall the name of Antony Cowlrick?” inquired our
Reporter.
“Because I am here to represent him. He has not only paid me for
my past services—forcing the money upon me—but he has thanked
me for them, which, in the bitterness of his heart, he declined to do,
although he was not asked, when he was finally discharged.”
“I had a suspicion,” remarked our Reporter, “at that time that he
was a gentleman; he spoke like one, and had the manner of one. It
was chiefly for that reason I took an interest in him.”
“No, no,” said Mr. Goldberry, jocosely; “you wanted copy. Every
man to his trade.”
“I could retort with good effect,” said our Reporter, good-
humouredly, “but I spare you. Will Antony Cowlrick be here this
morning?”
“Yes, and others whom you know.”
At this moment a lady and a gentleman entered the room in which
the inquest was held, and advancing to Mr. Goldberry shook hands
with him. The gentleman was Antony Cowlrick, who, after a few
words with his lawyer, turned, and offered his hand to our Reporter.
“I must apologise,” he said, “for not having kept the half-
appointment I made with you on the day you so generously lent me
the sovereign in Leicester Square, but I had my reasons, which you
will understand when I tell you as much of my story as I think it
proper for you to know.”
“I attend here,” said our Reporter, “on behalf of my paper, in
response to a letter sent to our editor by Mr. Frederick Holdfast.”
“I am Frederick Holdfast,” said the gentleman. “Antony Cowlrick
was an assumed name; I could not use my own when I was falsely
accused of the murder of my father.”
He turned aside with quivering lips, and our Reporter, holding his
grief in respect, did not intrude upon it. The face of the lady who
accompanied Frederick Holdfast appeared singularly familiar to our
Reporter, and his curiosity was presently appeased by Mr. Goldberry,
who informed him that she was the lady who, by the happiest of
chances, met Mr. Frederick Holdfast in Leicester Square after his
discharge.
“Were she willing to allow herself to be used in such a way,”
observed the lawyer, “her photograph to-morrow could be sold in
thousands all over England. But she does not belong to that class of
woman. She is a heroine, in the truest sense of the word. Mrs.
Holdfast, who supplied you with a Romance in Real Life fit for a
novel instead of the columns of a newspaper, would not, in such
circumstances as these, have withstood the temptation. But there
are women and women.”
“I grant you,” said our Reporter, “that I was deceived in the
character of Mrs. Holdfast. Am I the first who has been beguiled by
the soft speeches of a fair woman? And, my dear sir, if you want
novels and romances, take my word for it, you cannot do better than
go to the columns of a newspaper for them. What has become of
Mrs. Holdfast’s baby?”
“The child will be cared for,” replied Mr. Goldberry, “by Frederick
Holdfast, and will be brought up in ignorance of her mother’s
crimes.”
The proceedings at the inquest commenced languidly, but were
soon brightened by the extraordinary revelations made by the
witnesses. The bodies of the two persons burnt to death were
identified, and then evidence was given, in dramatic sequence, in
proof that, at the time of their death, the deceased were engaged in
unlawful proceedings, and that the male deceased had formed a
deliberate plan for setting fire to the house.
Mrs. Preedy, lodging-house keeper, deposed to the letting of a
furnished attic to a man who gave the name of Richard Manx, and
who spoke like a foreigner. The rent of this attic was three shillings a
week, but she had never seen the colour of Richard Manx’s money;
he “gave out” to her that he was very poor; she had no doubt he
was the man who was found dead in the next house; neither had
she any doubt that it was he who had spread the report that her
house was haunted, and that he did it to ruin her. This witness
rambled in her evidence, and caused great laughter by her irrelevant
replies to questions.
Mrs. Whittaker, lodging-house keeper in Buckingham Palace Road,
deposed to the letting of her first-floor to Mr. Pelham at a rental of
three guineas per week. He paid his rent regularly, and she believed
him to be a gentleman of considerable means. She recognised the
body of the male deceased as Mr. Pelham.
The principal detective employed by Mr. Frederick Holdfast
testified that the male body was that of Richard Manx, otherwise
Pelham, a notorious blackleg; that he had lodged at No. 118, Great
Porter Square as Richard Manx, and in Buckingham Palace Road as
Mr. Pelham; that he (the detective) was employed to watch the
deceased on suspicion that he was implicated in the murder of Mr.
Holdfast, senior; that on the night of the fire he saw a female enter
118, Great Porter Square, in the company of the deceased; and that
this female was Mrs. Holdfast, widow of the gentleman who had
been murdered some months ago.
A sensation was then caused by the appearance of Mr. Frederick
Holdfast as a witness. He recognised the bodies as those of Mr.
Pelham and Mrs. Holdfast, his father’s second wife. Before his father
contracted a second marriage he had an acquaintance with the
deceased persons in Oxford. Mr. Pelham was a blackleg, and had
been expelled from the company of gentlemen for cheating with
dice; and Mrs. Holdfast was a woman not entitled to respect. The
witness, in reply to questions put by his lawyer, Mr. Goldberry, said
he was the man who, under the name of Antony Cowlrick, had been
wrongfully charged at the Martin Street Police-court with the murder
of a gentleman, who, it was now known, was his father; and that he
had in his possession evidence in his father’s handwriting which
proved, beyond the possibility of doubt, that his father had been
murdered by one or both of the deceased. The other portions of this
witness’s evidence, relating to his taking possession of the house
No. 119 Great Porter Square, and to the watch he set upon Mr.
Pelham’s movements, are fully detailed in our verbatim report of the
inquest, and will be found most startling and dramatic.
Even more dramatic was the evidence of the next witness,
Blanche Daffarn, Mr. Frederick Holdfast’s fiancée, a young lady of
great personal attractions. For the purpose of clearing her lover from
the dreadful accusation brought against him, she had disguised
herself as a servant, and had taken service as a maid-of-all-work
with Mrs. Preedy. It was through her instrumentality that Pelham
and Richard Manx were discovered to be one and the same person,
and had it not been for her courage and devotion there is but little
doubt that the guilty ones would have escaped. She gave her
evidence with clearness and modesty, and she was frequently
interrupted by murmurs of applause, which the Coroner did not
attempt to suppress.
It might have been supposed that the climax of interest was
reached when the fair witness, towards whom every face in the
room was turned in admiration, took her seat; but it was not; a
higher point was attained upon the appearance of a little girl, a mere
child, whom our Reporter at once recognised as Fanny, a match girl,
with whom our readers have already made acquaintance. The
brightness, the vivacity, and the adventures of this little waif in
connection with the case, no less than her sensibility and gratitude
towards her guardian angel, Miss Blanche Daffarn, produced a
profound impression. It would be hard to say whether tears or
smiles predominated while this intelligent and grateful child stood
before the Coroner; both were freely produced by the wonderful
touches of nature which gleamed through little Fanny’s narrative,
which she was allowed to relate almost without interruption from
Coroner and jury. It is pleasant to be able to state that Fanny’s
future is made safe; Mr. Frederick Holdfast and his fiancée are her
protectors. The child is rescued from the gin shop and the gutter.
The inquest was over, and still the persons in the crowded room
lingered for a parting glance at those who had played their parts in
the strange and varied drama. The interest in the proceedings had
extended beyond the Court, and a large concourse of persons had
gathered outside, eager to see the brave young lady and the child,
whose names will be mentioned in terms of admiration in every
home in the kingdom. Such is the power of the newspaper. To
convey to remote distances, into village and city, to the firesides of
the poor and the rich, the records of ennobling deeds, and to cause
“God bless you little Fanny!” “May you live happy lives, Frederick and
Blanche!” to be breathed by the millions whose hearts shall be
stirred by this story of love and crime, of cunning which over-
reached itself and suffering which blossomed into sweetness, the
last scenes of which were enacted in a common lodging-house in
Great Porter Square.
THE END.
Transcriber’s note
ebookbell.com