Visit ebookfinal.

com to download the full version and

explore more ebooks or textbooks

Deep Learning with Python 2nd Edition François

Chollet

_____ Click the link below to download _____

https://ebookfinal.com/download/deep-learning-with-
python-2nd-edition-francois-chollet/

Explore and download more ebooks or textbook at ebookfinal.com

Here are some recommended products that we believe you will be
interested in. You can click the link to download.

Deep Learning for Finance Creating Machine Deep Learning

Models for Trading in Python 1st Edition Kaabar

https://ebookfinal.com/download/deep-learning-for-finance-creating-
machine-deep-learning-models-for-trading-in-python-1st-edition-kaabar/

Thoughtful Machine Learning with Python Early Release

Matthew Kirk

https://ebookfinal.com/download/thoughtful-machine-learning-with-
python-early-release-matthew-kirk/

Learning Unix for OS X Going Deep With the Terminal and

Shell 2nd Edition Dave Taylor

https://ebookfinal.com/download/learning-unix-for-os-x-going-deep-
with-the-terminal-and-shell-2nd-edition-dave-taylor/

Machine Learning with Spark and Python Essential

Techniques for Predictive Analytics 2nd Edition Michael
Bowles
https://ebookfinal.com/download/machine-learning-with-spark-and-
python-essential-techniques-for-predictive-analytics-2nd-edition-
michael-bowles/
Programming Python with CD 2nd Edition Mark Lutz

https://ebookfinal.com/download/programming-python-with-cd-2nd-
edition-mark-lutz/

Learning OpenCV 5 Computer Vision with Python Fourth

Edition Joseph Howse & Joe Minichino

https://ebookfinal.com/download/learning-opencv-5-computer-vision-
with-python-fourth-edition-joseph-howse-joe-minichino/

Tensorflow for Deep Learning Reza Bosagh Zadeh

https://ebookfinal.com/download/tensorflow-for-deep-learning-reza-
bosagh-zadeh/

Python Machine Learning Second Edition Sebastian Raschka

https://ebookfinal.com/download/python-machine-learning-second-
edition-sebastian-raschka/

Deep Learning for Coders with fastai PyTorch AI

Applications Without a PhD 1st Edition Jeremy Howard

https://ebookfinal.com/download/deep-learning-for-coders-with-fastai-
pytorch-ai-applications-without-a-phd-1st-edition-jeremy-howard/
Deep Learning with Python 2nd Edition François
Chollet Digital Instant Download
Author(s): François Chollet
ISBN(s): 9781617296864, 1617296864
Edition: 2
File Details: PDF, 7.69 MB
Year: 2021
Language: english
SECOND EDITION

François Chollet

MANNING
Deep Learning with Python
Deep Learning
with Python
SECOND EDITION

FRANÇOIS CHOLLET

MANNING
SHELTER ISLAND
For online information and ordering of this and other Manning books, please visit
www.manning.com. The publisher offers discounts on this book when ordered in quantity.
For more information, please contact
Special Sales Department
Manning Publications Co.
20 Baldwin Road
PO Box 761
Shelter Island, NY 11964
Email: orders@manning.com

©2021 by Manning Publications Co. All rights reserved.

No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in

any form or by means electronic, mechanical, photocopying, or otherwise, without prior written
permission of the publisher.

Many of the designations used by manufacturers and sellers to distinguish their products are
claimed as trademarks. Where those designations appear in the book, and Manning Publications
was aware of a trademark claim, the designations have been printed in initial caps or all caps.

Recognizing the importance of preserving what has been written, it is Manning’s policy to have
the books we publish printed on acid-free paper, and we exert our best efforts to that end.
Recognizing also our responsibility to conserve the resources of our planet, Manning books
are printed on paper that is at least 15 percent recycled and processed without the use of
elemental chlorine.

The author and publisher have made every effort to ensure that the information in this book
was correct at press time. The author and publisher do not assume and hereby disclaim any
liability to any party for any loss, damage, or disruption caused by errors or omissions, whether
such errors or omissions result from negligence, accident, or any other cause, or from any usage
of the information herein.

Development editor: Jennifer Stout

Technical development editor: Frances Buontempo
Manning Publications Co. Review editor: Aleksandar Dragosavljević
20 Baldwin Road Production editor: Keri Hales
PO Box 761 Copy editor: Andy Carroll
Shelter Island, NY 11964 Proofreaders: Katie Tennant and Melody Dolab
Technical proofreader: Karsten Strøbæk
Typesetter: Dennis Dalinnik
Cover designer: Marija Tudor

ISBN: 9781617296864
Printed in the United States of America
To my son Sylvain: I hope you’ll read this book someday!
 brief contents
1 ■ What is deep learning? 1
2 ■ The mathematical building blocks
of neural networks 26
3 ■ Introduction to Keras and TensorFlow 68
4 ■ Getting started with neural networks: Classification
and regression 95
5 ■ Fundamentals of machine learning 121
6 ■ The universal workflow of machine learning 153
7 ■ Working with Keras: A deep dive 172
8 ■ Introduction to deep learning
for computer vision 201
9 ■ Advanced deep learning for computer vision 238
10 ■ Deep learning for timeseries 280
11 ■ Deep learning for text 309
12 ■ Generative deep learning 364
13 ■ Best practices for the real world 412
14 ■ Conclusions 431

vii
 contents
preface xvii
acknowledgments xix
about this book xx
about the author xxiii
about the cover illustration xxiv

1 What is deep learning?

1.1
1
Artificial intelligence, machine learning, and deep
learning 2
Artificial intelligence 2 Machine learning 3 Learning
■ ■

rules and representations from data 4 The “deep” in “deep

■

learning” 7 Understanding how deep learning works, in

■

three figures 8 What deep learning has achieved so far 10

■

Don’t believe the short-term hype 11 The promise of AI 12

■

1.2 Before deep learning: A brief history of machine

learning 13
Probabilistic modeling 13 Early neural networks 14
■

Kernel methods 14 Decision trees, random forests, and

■

gradient boosting machines 15 Back to neural networks 16

■

What makes deep learning different 17 The modern machine

■

learning landscape 18

ix
x CONTENTS

1.3 Why deep learning? Why now? 20

Hardware 20 Data 21 Algorithms 22 A new wave of
■ ■ ■

investment 23 The democratization of deep learning 24

■

Will it last? 24

2 The mathematical building blocks of neural networks

2.1 A first look at a neural network 27
26

2.2 Data representations for neural networks 31

Scalars (rank-0 tensors) 31 Vectors (rank-1 tensors) 31
■

Matrices (rank-2 tensors) 32 Rank-3 and higher-rank

■

tensors 32 Key attributes 32 Manipulating tensors in

■ ■

NumPy 34 The notion of data batches 35 Real-world

■ ■

examples of data tensors 35 Vector data 35 Timeseries data

■ ■

or sequence data 36 Image data 37 Video data 37

■ ■

2.3 The gears of neural networks: Tensor operations 38

Element-wise operations 38 Broadcasting 40 Tensor product 41
■ ■

Tensor reshaping 43 Geometric interpretation of tensor operations 44

■

A geometric interpretation of deep learning 47

2.4 The engine of neural networks: Gradient-based
optimization 48
What’s a derivative? 49 Derivative of a tensor operation:
■

The gradient 51 Stochastic gradient descent 52 Chaining

■ ■

derivatives: The Backpropagation algorithm 55

2.5 Looking back at our first example 61
Reimplementing our first example from scratch in TensorFlow 63
Running one training step 64 The full training loop 65
■

Evaluating the model 66

3 Introduction to Keras and TensorFlow

3.1 What’s TensorFlow? 69
68

3.2 What’s Keras? 69

3.3 Keras and TensorFlow: A brief history 71
3.4 Setting up a deep learning workspace 71
Jupyter notebooks: The preferred way to run deep learning
experiments 72 Using Colaboratory 73
■

3.5 First steps with TensorFlow 75

Constant tensors and variables 76 Tensor operations: Doing
■

math in TensorFlow 78 A second look at the GradientTape

■

API 78 An end-to-end example: A linear classifier in pure

■

TensorFlow 79
 CONTENTS xi

3.6 Anatomy of a neural network: Understanding core

Keras APIs 84
Layers: The building blocks of deep learning 84 From layers ■

to models 87 The “compile” step: Configuring the learning

■

process 88 Picking a loss function 90 Understanding the

■ ■

fit() method 91 Monitoring loss and metrics on validation

■

data 91 Inference: Using a model after training 93

■

4 Getting started with neural networks: Classification and

regression 95
4.1 Classifying movie reviews: A binary classification
example 97
The IMDB dataset 97 Preparing the data 98 Building your
■ ■

model 99 Validating your approach 102 Using a trained

■ ■

model to generate predictions on new data 105 Further ■

experiments 105 Wrapping up 106 ■

4.2 Classifying newswires: A multiclass classification

example 106
The Reuters dataset 106 Preparing the data 107 Building
■ ■

your model 108 Validating your approach 109 Generating

■ ■

predictions on new data 111 A different way to handle the labels

■

and the loss 112 The importance of having sufficiently large

■

intermediate layers 112 Further experiments 113

■

Wrapping up 113
4.3 Predicting house prices: A regression example 113
The Boston housing price dataset 114 Preparing the data 114 ■

Building your model 115 Validating your approach using

■

K-fold validation 115 Generating predictions on new

■

data 119 Wrapping up 119

■

5 Fundamentals of machine learning

5.1 Generalization: The goal of machine learning
Underfitting and overfitting 122 ■
121

The nature of generalization

121

in deep learning 127

5.2 Evaluating machine learning models 133
Training, validation, and test sets 133 Beating a common-sense ■

baseline 136 Things to keep in mind about model

■

evaluation 137
5.3 Improving model fit 138
Tuning key gradient descent parameters 138 Leveraging better ■

architecture priors 139 Increasing model capacity 140

■
xii CONTENTS

5.4 Improving generalization 142

Dataset curation 142 Feature engineering 143
■ ■
Using early
stopping 144 Regularizing your model 145
■

6 The universal workflow of machine learning

6.1 Define the task 155
Frame the problem 155 Collect a dataset 156 Understand
■
153

your data 160 Choose a measure of success 160

■

6.2 Develop a model 161

Prepare the data 161 Choose an evaluation protocol
■
162
Beat a baseline 163 Scale up: Develop a model that
■

overfits 164 Regularize and tune your model 165

■

6.3 Deploy the model 165

Explain your work to stakeholders and set expectations 165
Ship an inference model 166 Monitor your model in the ■

wild 169 Maintain your model 170

■

7 Working with Keras: A deep dive

7.1 A spectrum of workflows 173
172

7.2 Different ways to build Keras models 173

The Sequential model 174 The Functional API 176 ■

Subclassing the Model class 182 Mixing and matching different ■

components 184 Remember: Use the right tool for the job 185
■

7.3 Using built-in training and evaluation loops 185

Writing your own metrics 186 Using callbacks 187 ■

Writing your own callbacks 189 Monitoring and visualization ■

with TensorBoard 190

7.4 Writing your own training and evaluation loops 192
Training versus inference 194 Low-level usage of metrics 195■

A complete training and evaluation loop 195 Make it fast with ■

tf.function 197 Leveraging fit() with a custom training loop 198

■

8 Introduction to deep learning for computer vision

8.1 Introduction to convnets
The convolution operation 204
202
■
The max-pooling
201

operation 209
8.2 Training a convnet from scratch on a small dataset 211
The relevance of deep learning for small-data problems 212
Downloading the data 212 Building the model 215 ■

Data preprocessing 217 Using data augmentation 221

■
 CONTENTS xiii

8.3 Leveraging a pretrained model 224

Feature extraction with a pretrained model 225 ■
Fine-tuning a
pretrained model 234

9 Advanced deep learning for computer vision

9.1 Three essential computer vision tasks 238
238

9.2 An image segmentation example 240

9.3 Modern convnet architecture patterns 248
Modularity, hierarchy, and reuse 249 Residual connections 251
■

Batch normalization 255 Depthwise separable convolutions 257

■

Putting it together: A mini Xception-like model 259

9.4 Interpreting what convnets learn 261
Visualizing intermediate activations 262 Visualizing convnet
■

filters 268 Visualizing heatmaps of class activation 273

■

10 Deep learning for timeseries

10.1
280
Different kinds of timeseries tasks 280
10.2 A temperature-forecasting example 281
Preparing the data 285 A common-sense, non-machine learning
■

baseline 288 Let’s try a basic machine learning model 289

■

Let’s try a 1D convolutional model 290 A first recurrent

■

baseline 292
10.3 Understanding recurrent neural networks 293
A recurrent layer in Keras 296
10.4 Advanced use of recurrent neural networks 300
Using recurrent dropout to fight overfitting 300 Stacking ■

recurrent layers 303 Using bidirectional RNNs 304

■

Going even further 307

11 Deep learning for text

11.1
309
Natural language processing: The bird’s eye view 309
11.2 Preparing text data 311
Text standardization 312 ■ Text splitting (tokenization) 313
Vocabulary indexing 314 ■ Using the TextVectorization
layer 316
11.3 Two approaches for representing groups of words:
Sets and sequences 319
Preparing the IMDB movie reviews data 320 Processing words ■

as a set: The bag-of-words approach 322 Processing words as a

■

sequence: The sequence model approach 327

xiv CONTENTS

11.4 The Transformer architecture 336

Understanding self-attention 337 Multi-head attention 341
■

The Transformer encoder 342 When to use sequence models over

■

bag-of-words models 349

11.5 Beyond text classification: Sequence-to-sequence
learning 350
A machine translation example 351 Sequence-to-sequence ■

learning with RNNs 354 Sequence-to-sequence learning with

■

Transformer 358

12 Generative deep learning

12.1 Text generation 366
364

A brief history of generative deep learning for sequence

generation 366 How do you generate sequence data? 367
■

The importance of the sampling strategy 368 Implementing ■

text generation with Keras 369 A text-generation callback

■

with variable-temperature sampling 372 Wrapping up 376 ■

12.2 DeepDream 376

Implementing DeepDream in Keras 377 ■ Wrapping up 383
12.3 Neural style transfer 383
The content loss 384 The style loss 384
■ ■ Neural style transfer
in Keras 385 ■ Wrapping up 391
12.4 Generating images with variational autoencoders 391
Sampling from latent spaces of images 391 Concept vectors ■

for image editing 393 Variational autoencoders 393

■

Implementing a VAE with Keras 396 Wrapping up 401 ■

12.5 Introduction to generative adversarial networks 401

A schematic GAN implementation 402 A bag of ■

tricks 403 Getting our hands on the CelebA dataset 404

■

The discriminator 405 The generator 407 The adversarial

■ ■

network 408 Wrapping up 410

■

13 Best practices for the real world

13.1 Getting the most out of your models
Hyperparameter optimization
412

413 ■
Model ensembling
413
420
13.2 Scaling-up model training 421
Speeding up training on GPU with mixed precision 422
Multi-GPU training 425 TPU training 428 ■
 CONTENTS xv

14 Conclusions
14.1
431
Key concepts in review 432
Various approaches to AI 432 What makes deep learning special
■

within the field of machine learning 432 How to think about

■

deep learning 433 Key enabling technologies 434 The

■ ■

universal machine learning workflow 435 Key network ■

architectures 436 The space of possibilities 440

■

14.2 The limitations of deep learning 442

The risk of anthropomorphizing machine learning models 443
Automatons vs. intelligent agents 445 Local generalization vs.
■

extreme generalization 446 The purpose of intelligence 448

■

Climbing the spectrum of generalization 449

14.3 Setting the course toward greater generality in AI 450
On the importance of setting the right objective: The shortcut
rule 450 A new target 452
■

14.4 Implementing intelligence: The missing ingredients 454

Intelligence as sensitivity to abstract analogies 454 The two poles
■

of abstraction 455 The missing half of the picture 458

■

14.5 The future of deep learning 459

Models as programs 460 Blending together deep learning and
■

program synthesis 461 Lifelong learning and modular

■

subroutine reuse 463 The long-term vision 465

■

14.6 Staying up to date in a fast-moving field 466

Practice on real-world problems using Kaggle 466 Read about ■

the latest developments on arXiv 466 Explore the Keras

■

ecosystem 467
14.7 Final words 467

index 469
 preface
If you’ve picked up this book, you’re probably aware of the extraordinary progress
that deep learning has represented for the field of artificial intelligence in the recent
past. We went from near-unusable computer vision and natural language processing
to highly performant systems deployed at scale in products you use every day. The
consequences of this sudden progress extend to almost every industry. We’re already
applying deep learning to an amazing range of important problems across domains as
different as medical imaging, agriculture, autonomous driving, education, disaster
prevention, and manufacturing.
Yet, I believe deep learning is still in its early days. It has only realized a small frac-
tion of its potential so far. Over time, it will make its way to every problem where it can
help—a transformation that will take place over multiple decades.
In order to begin deploying deep learning technology to every problem that it
could solve, we need to make it accessible to as many people as possible, including
non-experts—people who aren’t researchers or graduate students. For deep learning to
reach its full potential, we need to radically democratize it. And today, I believe that
we’re at the cusp of a historical transition, where deep learning is moving out of aca-
demic labs and the R&D departments of large tech companies to become a ubiquitous
part of the toolbox of every developer out there—not unlike the trajectory of web
development in the late 1990s. Almost anyone can now build a website or web app for
their business or community of a kind that would have required a small team of special-
ist engineers in 1998. In the not-so-distant future, anyone with an idea and basic coding
skills will be able to build smart applications that learn from data.

xvii
xviii PREFACE

When I released the first version of the Keras deep learning framework in March
2015, the democratization of AI wasn’t what I had in mind. I had been doing research
in machine learning for several years and had built Keras to help me with my own
experiments. But since 2015, hundreds of thousands of newcomers have entered the
field of deep learning; many of them picked up Keras as their tool of choice. As I
watched scores of smart people use Keras in unexpected, powerful ways, I came to
care deeply about the accessibility and democratization of AI. I realized that the fur-
ther we spread these technologies, the more useful and valuable they become. Accessi-
bility quickly became an explicit goal in the development of Keras, and over a few
short years, the Keras developer community has made fantastic achievements on this
front. We’ve put deep learning into the hands of hundreds of thousands of people,
who in turn are using it to solve problems that were until recently thought to be
unsolvable.
The book you’re holding is another step on the way to making deep learning avail-
able to as many people as possible. Keras had always needed a companion course to
simultaneously cover the fundamentals of deep learning, deep learning best practices,
and Keras usage patterns. In 2016 and 2017, I did my best to produce such a course,
which became the first edition of this book, released in December 2017. It quickly
became a machine learning best seller that sold over 50,000 copies and was translated
into 12 languages.
However, the field of deep learning advances fast. Since the release of the first edi-
tion, many important developments have taken place—the release of TensorFlow 2,
the growing popularity of the Transformer architecture, and more. And so, in late
2019, I set out to update my book. I originally thought, quite naively, that it would fea-
ture about 50% new content and would end up being roughly the same length as the
first edition. In practice, after two years of work, it turned out to be over a third lon-
ger, with about 75% novel content. More than a refresh, it is a whole new book.
I wrote it with a focus on making the concepts behind deep learning, and their
implementation, as approachable as possible. Doing so didn’t require me to dumb
down anything—I strongly believe that there are no difficult ideas in deep learning. I
hope you’ll find this book valuable and that it will enable you to begin building intelli-
gent applications and solve the problems that matter to you.
 acknowledgments
First of all, I’d like to thank the Keras community for making this book possible. Over
the past six years, Keras has grown to have hundreds of open source contributors and
more than one million users. Your contributions and feedback have turned Keras into
what it is today.
On a more personal note, I’d like to thank my wife for her endless support during
the development of Keras and the writing of this book.
I’d also like to thank Google for backing the Keras project. It has been fantastic to
see Keras adopted as TensorFlow’s high-level API. A smooth integration between
Keras and TensorFlow greatly benefits both TensorFlow users and Keras users, and
makes deep learning accessible to most.
I want to thank the people at Manning who made this book possible: publisher
Marjan Bace and everyone on the editorial and production teams, including Michael
Stephens, Jennifer Stout, Aleksandar Dragosavljević, and many others who worked
behind the scenes.
Many thanks go to the technical peer reviewers: Billy O’Callaghan, Christian
Weisstanner, Conrad Taylor, Daniela Zapata Riesco, David Jacobs, Edmon Begoli,
Edmund Ronald PhD, Hao Liu, Jared Duncan, Kee Nam, Ken Fricklas, Kjell Jansson,
Milan Šarenac, Nguyen Cao, Nikos Kanakaris, Oliver Korten, Raushan Jha, Sayak Paul,
Sergio Govoni, Shashank Polasa, Todd Cook, and Viton Vitanis—and all the other
people who sent us feedback on the draft on the book.
On the technical side, special thanks go to Frances Buontempo, who served as the
book’s technical editor, and Karsten Strøbæk, who served as the book’s technical
proofreader.

xix
Discovering Diverse Content Through
Random Scribd Documents
distributed Project Gutenberg™ eBooks with only a loose
network of volunteer support.

Project Gutenberg™ eBooks are often created from several

printed editions, all of which are confirmed as not protected by
copyright in the U.S. unless a copyright notice is included. Thus,
we do not necessarily keep eBooks in compliance with any
particular paper edition.

Most people start at our website which has the main PG search
facility: www.gutenberg.org.

This website includes information about Project Gutenberg™,

including how to make donations to the Project Gutenberg
Literary Archive Foundation, how to help produce our new
eBooks, and how to subscribe to our email newsletter to hear
about new eBooks.
Welcome to our website – the ideal destination for book lovers and
knowledge seekers. With a mission to inspire endlessly, we offer a
vast collection of books, ranging from classic literary works to
specialized publications, self-development books, and children's
literature. Each book is a new journey of discovery, expanding
knowledge and enriching the soul of the reade

Our website is not just a platform for buying books, but a bridge
connecting readers to the timeless values of culture and wisdom. With
an elegant, user-friendly interface and an intelligent search system,
we are committed to providing a quick and convenient shopping
experience. Additionally, our special promotions and home delivery
services ensure that you save time and fully enjoy the joy of reading.

Let us accompany you on the journey of exploring knowledge and

personal growth!

ebookfinal.com