Advanced Guide To Python 3 Programming 2nd 2nd

John Hunt download

https://ebookbell.com/product/advanced-guide-to-
python-3-programming-2nd-2nd-john-hunt-52673356

Explore and download more ebooks at ebookbell.com

Here are some recommended products that we believe you will be
interested in. You can click the link to download.

Advanced Guide To Python 3 Programming 2nd 2nd Edition John Hunt

https://ebookbell.com/product/advanced-guide-to-
python-3-programming-2nd-2nd-edition-john-hunt-52673496

Advanced Guide To Python 3 Programming 1st Edition 2019 Hunt

https://ebookbell.com/product/advanced-guide-to-
python-3-programming-1st-edition-2019-hunt-21896816

Python Programming For Advanced The Updated Advanced Guide To Master

Python Programming Step By Step In A Few Days With Machine Learning
And Data Science Resources Vol 3 Bill Steven

https://ebookbell.com/product/python-programming-for-advanced-the-
updated-advanced-guide-to-master-python-programming-step-by-step-in-a-
few-days-with-machine-learning-and-data-science-resources-vol-3-bill-
steven-232306752

Practical Guide To Python From Basics To Advanced Programming Arcadia

J Darell

https://ebookbell.com/product/practical-guide-to-python-from-basics-
to-advanced-programming-arcadia-j-darell-59301212
Python Advanced Guide To Artificial Intelligence Expert Machine
Learning Systems And Intelligent Agents Using Python Giuseppe
Bonaccorso

https://ebookbell.com/product/python-advanced-guide-to-artificial-
intelligence-expert-machine-learning-systems-and-intelligent-agents-
using-python-giuseppe-bonaccorso-48270252

Python Advanced Guide To Artificial Intelligence Rajalingappaa

Shanmugamani Armando Fandango Giuseppe Bonaccorso Rajalingappaa
Shanmugamani

https://ebookbell.com/product/python-advanced-guide-to-artificial-
intelligence-rajalingappaa-shanmugamani-armando-fandango-giuseppe-
bonaccorso-rajalingappaa-shanmugamani-23767668

Super Pythonista The Ultimate Guide To Python Programming From

Beginner To Advanced And Far Beyond Charles Kyriakou

https://ebookbell.com/product/super-pythonista-the-ultimate-guide-to-
python-programming-from-beginner-to-advanced-and-far-beyond-charles-
kyriakou-49111224

Python Programming Notes The Full Advanced Beginners Guide To Learn

Basic And Deep Core Concepts Of Python Easily Sudharshan

https://ebookbell.com/product/python-programming-notes-the-full-
advanced-beginners-guide-to-learn-basic-and-deep-core-concepts-of-
python-easily-sudharshan-231847900

Vba Integrating Python For Enhanced Automation A Comprehensive Guide

To Advanced Vba Techniques Van Der Post

https://ebookbell.com/product/vba-integrating-python-for-enhanced-
automation-a-comprehensive-guide-to-advanced-vba-techniques-van-der-
post-57665882
Undergraduate Topics in Computer Science

John Hunt

Advanced Guide
to Python 3
Programming
Second Edition
Undergraduate Topics in Computer Science

Series Editor
Ian Mackie, University of Sussex, Brighton, UK

Advisory Editors
Samson Abramsky , Department of Computer Science, University of Oxford,
Oxford, UK
Chris Hankin , Department of Computing, Imperial College London, London,
UK
Mike Hinchey , Lero – The Irish Software Research Centre, University of
Limerick, Limerick, Ireland
Dexter C. Kozen, Department of Computer Science, Cornell University, Ithaca,
NY, USA
Andrew Pitts , Department of Computer Science and Technology, University of
Cambridge, Cambridge, UK
Hanne Riis Nielson , Department of Applied Mathematics and Computer
Science, Technical University of Denmark, Kongens Lyngby, Denmark
Steven S. Skiena, Department of Computer Science, Stony Brook University,
Stony Brook, NY, USA
Iain Stewart , Department of Computer Science, Durham University, Durham,
UK
Joseph Migga Kizza, College of Engineering and Computer Science, The
University of Tennessee-Chattanooga, Chattanooga, TN, USA
‘Undergraduate Topics in Computer Science’ (UTiCS) delivers high-quality instruc-
tional content for undergraduates studying in all areas of computing and information
science. From core foundational and theoretical material to final-year topics and
applications, UTiCS books take a fresh, concise, and modern approach and are ideal
for self-study or for a one- or two-semester course. The texts are all authored by
established experts in their fields, reviewed by an international advisory board, and
contain numerous examples and problems, many of which include fully worked
solutions.
The UTiCS concept relies on high-quality, concise books in softback format, and
generally a maximum of 275–300 pages. For undergraduate textbooks that are likely
to be longer, more expository, Springer continues to offer the highly regarded Texts
in Computer Science series, to which we refer potential authors.
John Hunt

Advanced Guide to Python 3

Programming
Second Edition
John Hunt
Midmarsh Technology Ltd.
Chippenham, Wiltshire, UK

ISSN 1863-7310 ISSN 2197-1781 (electronic)

Undergraduate Topics in Computer Science
ISBN 978-3-031-40335-4 ISBN 978-3-031-40336-1 (eBook)
https://doi.org/10.1007/978-3-031-40336-1

© Springer Nature Switzerland AG 2019, 2023

This work is subject to copyright. All rights are reserved by the Publisher, whether the whole or part of
the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation,
broadcasting, reproduction on microfilms or in any other physical way, and transmission or information
storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology
now known or hereafter developed.
The use of general descriptive names, registered names, trademarks, service marks, etc. in this publication
does not imply, even in the absence of a specific statement, that such names are exempt from the relevant
protective laws and regulations and therefore free for general use.
The publisher, the authors, and the editors are safe to assume that the advice and information in this book
are believed to be true and accurate at the date of publication. Neither the publisher nor the authors or
the editors give a warranty, expressed or implied, with respect to the material contained herein or for any
errors or omissions that may have been made. The publisher remains neutral with regard to jurisdictional
claims in published maps and institutional affiliations.

This Springer imprint is published by the registered company Springer Nature Switzerland AG
The registered company address is: Gewerbestrasse 11, 6330 Cham, Switzerland
For Denise, my wife, my soulmate, my best
friend.
Preface to the Second Edition

This second edition represents a significant expansion of the material in the first
edition, as well as an update of that book from Python 3.7 to 3.12.
This book includes whole new sections on advanced language features, Reactive
Programming in Python and data analysts. New chapters on working with Tkinter,
on event handling with Tkinter and a simple drawing application using Tkinter have
been added. A new chapter on performance monitoring and profiling has also been
added. A chapter on pip and conda is included at the end of the book.
In all there are 18 completely new chapters that take you far further on your Python
journey. Enjoy the book and I hope you find it useful.

Chippenham, UK John Hunt

vii
Preface to the First Edition

Some of the key aspects of this book are

1. It assumes knowledge of Python 3 and of concepts such as functions, classes,
protocols, abstract base classes, decorators, iterables and collection types (such
as list and tuple).
2. However, the book assumes very little knowledge or experience of the topics
presented.
3. The book is divided into eleven topic areas: advanced language features,
Computer Graphics, games, testing, file input/output, database access, logging,
concurrency and parallelism, Reactive Programming, network programming and
data analytics.
4. Each topic in the book has an introductory chapter followed by chapters that
delve into that topic.
5. The book includes exercises at the end of most chapters.
6. All code examples (and exercise solutions) are provided on line in a GitHub
repository.

What You Need

You can of course just read this book; however following the examples in this book
will ensure that you get as much as possible out of the content. For this you will need
a computer.
Python is a cross-platform programming language, and as such you can use Python
on a Windows PC, a Linux box, an Apple Mac, etc. So you are not tied to a particular
type of operating system; you can use whatever you have available.
However you will need to install some software on that computer. At a minimum
you will need Python. The focus of this book is Python 3 so that is the version that
is assumed for all examples and exercises. As Python is available for a wide range

ix
x Preface to the First Edition

of platforms from Windows, to Mac OS and Linux, you will need to ensure that you
download the version for your operating system.
Python can be downloaded from the main Python website which can be found at
http://www.python.org/.

You will also need some form of editor to write your programs. There are numerous
generic programming editors available for different operating systems with VIM on
Linux, Notepad++ on Windows and Sublime Text on windows and Macs being
popular choices.
However, using an Integrated Development Environment (IDE) editor such as
PyCharm, Visual Studio Code or Spyder can make writing and running your programs
much easier.
However, this book does not assume any particular editor, IDE or environment
(other than Python 3 itself).

Conventions

Throughout this book you will find a number of conventions used for text styles.
These text styles distinguish between different kinds of information. Code words,
variable and Python values, used within the main body of the text, are shown using
a Courier font. A block of Python code is set out as shown here:
Preface to the First Edition xi

def draw_koch(size, depth):

if depth > 0:
for angle in ANGLES:
draw_koch(size / 3, depth - 1)
turtle.left(angle)
else:
turtle.forward(size)
# Draw three sides of snowflake
for_in range(3):
draw_koch(SIZE_OF_SNOWFLAKE, depth)
turtle.right(120)

Note that keywords and points of interest are shown in bold font.
Any command line or user input is shown in standard font as shown below, for
example:
Hello, world
Enter your name: John
Hello John

Example Code and Sample Solutions

The examples used in this book (along with sample solutions for the exercises at the
end of most chapters) are available in a GitHub repository. GitHub provides a web
interface to Git, as well as a server environment hosting Git.
Git is a version control system typically used to manage source code files (such
as those used to create systems in programming languages such as Python but also
Java, C#, C++ and Scala). Systems such as Git are very useful for collaborative
development as they allow multiple people to work on an implementation and to
merge their work together. They also provide a useful historical view of the code
(which also allows developers to roll back changes if modifications prove to be
unsuitable).
The GitHub repository for this book can be found at:
• https://github.com/johnehunt/advancedpython3_2nd
If you already have Git installed on your computer, then you can clone (obtain a
copy of) the repository locally using:
git clone https://github.com/johnehunt/advancedpyth
on3_2nd.git
If you do not have Git, then you can obtain a zip file of the examples using
https://github.com/johnehunt/advancedpython3_2nd/arc
hive/refs/heads/main.zip
xii Preface to the First Edition

You can of course install Git yourself if you wish. To do this, see https://git-scm.
com/downloads. Versions of the Git client for Mac OS, Windows and Linux/Unix
are available here.
However, many IDEs such as PyCharm come with Git support and so offer another
approach to obtaining a Git repository.
For more information on Git see http://git-scm.com/doc. This Git guide
provides a very good primer and is highly recommended.

Acknowledgement I would like to thank Phoebe Hunt for creating the pixel images used for the
Starship Meteors game in Chap. 22.
Contents

1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.2 Useful Python Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

Part I Advanced Language Features

2 Python Type Hints . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
2.2 Pythons Type System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
2.3 The Challenge for Python Developers . . . . . . . . . . . . . . . . . . . . . . 8
2.4 Static Typing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
2.5 Python Type Hints . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
2.6 Type Hint Layout . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
2.7 Type Hints for Multiple Types . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
2.8 The Self Type . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
2.9 The Benefits of Type Hints . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
2.10 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
2.11 Online Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
3 Class Slots . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
3.2 The Problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
3.3 Slots to the Rescue . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
3.4 Performance Benefits . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
3.5 Why Not Use Slots? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
3.6 Online Resource . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
4 Weak References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
4.2 How Garbage Collection Works: Reference Counting . . . . . . . . 23
4.3 Weak References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
4.4 When to Use Weak References . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25

xiii
xiv Contents

4.5 The Weakref Module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26

4.6 Creating Weak References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
4.7 Retrieving Objects from Weak References . . . . . . . . . . . . . . . . . . 28
4.8 The WeakValueDicitonary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
4.9 WeakKeyDictionary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
4.10 Proxy Objects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
4.11 Online Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
5 Data Classes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
5.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
5.2 A Traditional Class . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
5.3 Defining Data Classes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
5.4 Defining Additional Behaviour . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
5.5 The Dataclass Decorator . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
5.6 Custom Factory for Fields . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
5.7 Immutable Dataclasses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
5.8 Data Classes and Inheritance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
5.9 Post Initialisation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
5.10 Initialisation Variables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
5.11 Positional Attributes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
5.12 Exercise . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
6 Structural Pattern Matching . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
6.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
6.2 The Match Statement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
6.3 Matching Classes with Positional Attributes . . . . . . . . . . . . . . . . . 50
6.4 Matching Against Standard Classes . . . . . . . . . . . . . . . . . . . . . . . . 51
6.5 Online Resource . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
7 Working with pprint . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
7.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
7.2 The pprint Data Printer Module . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
7.3 Basic pprint Usage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
7.4 Changing the Width . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
7.5 Changing the Depth . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
7.6 Managing the Indentation Level . . . . . . . . . . . . . . . . . . . . . . . . . . . 58
7.7 Reducing Line Breaks Using Compact . . . . . . . . . . . . . . . . . . . . . 59
7.8 The pformat Function . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
7.9 The saferepr() Function . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
7.10 Using the PrettyPrinter Class . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
7.11 Online Resource . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63
8 Shallow v Deep Copy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65
8.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65
8.2 Copying a List of Lists . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65
8.3 The Problem with Copying . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66
Contents xv

8.4 The Copy Module to the Rescue . . . . . . . . . . . . . . . . . . . . . . . . . . . 68

8.5 Using the deepcopy() Function . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69
8.6 Online Resource . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70
9 The __init__ Versus __new__ and __call . . . . . . . . . . . . . . . . . . . . . . . . . 71
9.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71
9.2 The __new__ and __init__ Methods . . . . . . . . . . . . . . . . . . . . . . . 71
9.3 The __new__ Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72
9.4 When to Use the __new__ Method . . . . . . . . . . . . . . . . . . . . . . . . . 73
9.5 Using __new__ to Create a Singleton Object . . . . . . . . . . . . . . . . 74
9.6 The __init__ Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75
9.7 Can __new__ and __init__ Be Used Together? . . . . . . . . . . . . . . 76
9.8 The __call__ Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77
9.9 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77
10 Python Metaclasses and Meta Programming . . . . . . . . . . . . . . . . . . . . . 79
10.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79
10.2 Metaprogramming . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79
10.3 Decorators as a Form of Metaprogramming . . . . . . . . . . . . . . . . . 81
10.4 Metaclasses for Metaprogramming . . . . . . . . . . . . . . . . . . . . . . . . 83
10.4.1 Singleton Metaclass . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83
10.5 Exec and Eval for Metaprogramming . . . . . . . . . . . . . . . . . . . . . . 85
10.5.1 The exec() Function . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85
10.5.2 The eval() Function . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86
10.5.3 eval Versus exec() . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86

Part II Computer Graphics and GUIs

11 Introduction to Computer Graphics . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91
11.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91
11.2 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92
11.3 The Graphical Computer Era . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92
11.4 Interactive and Non Interactive Graphics . . . . . . . . . . . . . . . . . . . . 93
11.5 Pixels . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94
11.6 Bit Map Versus Vector Graphics . . . . . . . . . . . . . . . . . . . . . . . . . . . 96
11.7 Buffering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96
11.8 Python and Computer Graphics . . . . . . . . . . . . . . . . . . . . . . . . . . . 96
11.9 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97
11.10 Online Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97
12 Python Turtle Graphics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99
12.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99
12.2 The Turtle Graphics Library . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99
12.2.1 The Turtle Module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99
12.2.2 Basic Turtle Graphics . . . . . . . . . . . . . . . . . . . . . . . . . . . 100
12.2.3 Drawing Shapes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103
12.2.4 Filling Shapes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104
xvi Contents

12.3 Other Graphics Libraries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105

12.4 3D Graphics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106
12.4.1 PyOpenGL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106
12.5 Online Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107
12.6 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107
13 Computer Generated Art . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109
13.1 Creating Computer Art . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109
13.2 A Computer Art Generator . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111
13.3 Fractals in Python . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114
13.4 The Koch Snowflake . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114
13.5 Mandelbrot Set . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116
13.6 Online Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119
13.7 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119
14 Introduction to Matplotlib . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121
14.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121
14.2 Matplotlib . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122
14.3 Plot Components . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123
14.4 Matplotlib Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125
14.4.1 Backend Layer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125
14.4.2 The Artist Layer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126
14.4.3 The Scripting Layer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127
14.5 Online Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128
15 Graphing with Matplotlib Pyplot . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129
15.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129
15.2 The pyplot API . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129
15.3 Line Graphs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 130
15.3.1 Coded Format Strings . . . . . . . . . . . . . . . . . . . . . . . . . . . 132
15.4 Scatter Graph . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133
15.4.1 When to Use Scatter Graphs . . . . . . . . . . . . . . . . . . . . . 134
15.5 Pie Charts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 136
15.5.1 Expanding Segments . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137
15.5.2 When to Use Pie Charts . . . . . . . . . . . . . . . . . . . . . . . . . 138
15.6 Bar Charts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139
15.6.1 Horizontal Bar Charts . . . . . . . . . . . . . . . . . . . . . . . . . . . 140
15.6.2 Coloured Bars . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141
15.6.3 Stacked Bar Charts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141
15.6.4 Grouped Bar Charts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143
15.7 Figures and Subplots . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144
15.8 3D Graphs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 146
15.9 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 148
Contents xvii

16 Graphical User Interfaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 151

16.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 151
16.2 GUIs and WIMPS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 151
16.3 Windowing Frameworks for Python . . . . . . . . . . . . . . . . . . . . . . . . 153
16.3.1 Platform-Independent GUI Libraries . . . . . . . . . . . . . . 154
16.3.2 Platform-Specific GUI Libraries . . . . . . . . . . . . . . . . . . 154
16.4 Online Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 154
17 Tkinter GUI Library . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155
17.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155
17.2 Tkinter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155
17.3 Windows as Objects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 156
17.4 Key Concepts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 156
17.4.1 The Tk Class . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157
17.4.2 TK Widgets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157
17.4.3 The TopLevel Class . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 158
17.4.4 The Frame Class . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159
17.4.5 Dialogs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 160
17.4.6 The Canvas Class . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 161
17.5 The Class Inheritance Hierarchy . . . . . . . . . . . . . . . . . . . . . . . . . . . 162
17.5.1 Layout Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163
17.6 A Simple Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163
17.7 Tkinter Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164
17.7.1 Mac Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164
17.7.2 Windows Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165
17.8 GUI Builders for Tkinter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 166
17.9 Online Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 166
17.10 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 167
18 Events in Tkinter User Interfaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 169
18.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 169
18.2 Event Handling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 169
18.3 What is Event Handling? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 170
18.4 What Are Event Handlers? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 170
18.5 Event Binders . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 172
18.6 Virtual Events . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 173
18.7 Event Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 173
18.8 What Types of Event Are There? . . . . . . . . . . . . . . . . . . . . . . . . . . 174
18.9 Binding an Event to an Event Handler . . . . . . . . . . . . . . . . . . . . . . 175
18.10 Implementing Event Handling . . . . . . . . . . . . . . . . . . . . . . . . . . . . 175
18.11 An Interactive GUI Application . . . . . . . . . . . . . . . . . . . . . . . . . . . 178
18.12 Online Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 181
18.13 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 181
xviii Contents

19 PyDraw Tkinter Example Application . . . . . . . . . . . . . . . . . . . . . . . . . . 185

19.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 185
19.2 The PyDraw Application . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 185
19.3 The Structure of the Application . . . . . . . . . . . . . . . . . . . . . . . . . . . 186
19.3.1 Model, View and Controller Architecture . . . . . . . . . . 188
19.3.2 PyDraw MVC Architecture . . . . . . . . . . . . . . . . . . . . . . 189
19.3.3 Additional Classes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 190
19.3.4 Object Relationships . . . . . . . . . . . . . . . . . . . . . . . . . . . . 190
19.4 The Interactions Between Objects . . . . . . . . . . . . . . . . . . . . . . . . . 191
19.4.1 The PyDrawApp . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 192
19.5 The PyDrawView Constructor . . . . . . . . . . . . . . . . . . . . . . . . . . . . 192
19.5.1 Changing the Application Mode . . . . . . . . . . . . . . . . . . 193
19.5.2 Adding a Graphic Object . . . . . . . . . . . . . . . . . . . . . . . . 193
19.6 The Classes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 194
19.6.1 The PyDrawConstants Class . . . . . . . . . . . . . . . . . . . . . 194
19.6.2 The PyDrawView Class . . . . . . . . . . . . . . . . . . . . . . . . . 195
19.6.3 The PyDrawMenuBar Class . . . . . . . . . . . . . . . . . . . . . . 196
19.6.4 The PyDrawController Class . . . . . . . . . . . . . . . . . . . . . 196
19.6.5 The DrawingModel Class . . . . . . . . . . . . . . . . . . . . . . . . 197
19.6.6 The DrawingView Class . . . . . . . . . . . . . . . . . . . . . . . . . 198
19.6.7 The DrawingController Class . . . . . . . . . . . . . . . . . . . . . 198
19.6.8 The Figure Class . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 199
19.6.9 The Square Class . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 200
19.6.10 The Circle Class . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 200
19.6.11 The Line Class . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 200
19.6.12 The Text Class . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 201
19.7 Reference . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 201
19.8 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 201

Part III Computer Games

20 Introduction to Games Programming . . . . . . . . . . . . . . . . . . . . . . . . . . . 205
20.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 205
20.2 Games Frameworks and Libraries . . . . . . . . . . . . . . . . . . . . . . . . . 205
20.3 Python Games Development . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 206
20.4 Using Pygame . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 207
20.5 Online Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 207
21 Building Games with Pygame . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 209
21.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 209
21.2 The Display Surface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 210
21.3 Events . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 211
21.3.1 Event Types . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 211
21.3.2 Event Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 212
21.3.3 The Event Queue . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 213
Contents xix

21.4 A First pygame Application . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 214

21.5 Further Concepts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 217
21.6 A More Interactive pygame Application . . . . . . . . . . . . . . . . . . . . 219
21.7 Alternative Approach to Processing Input Devices . . . . . . . . . . . 221
21.8 pygame Modules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 222
21.9 Online Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 223
22 StarshipMeteors Pygame . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 225
22.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 225
22.2 Creating a Spaceship Game . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 225
22.3 The Main Game Class . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 226
22.4 The GameObject Class . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 228
22.5 Displaying the Starship . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 229
22.6 Moving the Spaceship . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 230
22.7 Adding a Meteor Class . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 233
22.8 Moving the Meteors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 235
22.9 Identifying a Collision . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 236
22.10 Identifying a Win . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 237
22.11 Increasing the Number of Meteors . . . . . . . . . . . . . . . . . . . . . . . . . 238
22.12 Pausing the Game . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 238
22.13 Displaying the Game Over Message . . . . . . . . . . . . . . . . . . . . . . . 239
22.14 The StarshipMeteors Game . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 240
22.15 Online Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 244
22.16 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 244

Part IV Testing
23 Introduction to Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 247
23.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 247
23.2 Types of Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 247
23.3 What Should Be Tested? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 248
23.4 Types of Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 249
23.4.1 Unit Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 250
23.4.2 Integration Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 251
23.4.3 System Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 251
23.4.4 Installation/Upgrade Testing . . . . . . . . . . . . . . . . . . . . . 252
23.4.5 Smoke Tests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 252
23.5 Automating Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 252
23.6 Test-Driven Development . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 253
23.6.1 The TDD Cycle . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 253
23.6.2 Test Complexity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 254
23.6.3 Refactoring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 255
23.7 Design for Testability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 255
23.7.1 Testability Rules of Thumb . . . . . . . . . . . . . . . . . . . . . . 255
23.8 Online Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 255
23.9 Book Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 256
xx Contents

24 PyTest Testing Framework . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 257

24.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 257
24.2 What is PyTest? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 257
24.3 Setting up PyTest . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 258
24.4 A Simple PyTest Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 258
24.5 Working with PyTest . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 261
24.6 Parameterised Tests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 265
24.7 Online Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 267
24.8 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 267
25 Mocking for Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 269
25.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 269
25.2 Why Mock? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 271
25.3 What is Mocking? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 272
25.4 Common Mocking Framework Concepts . . . . . . . . . . . . . . . . . . . 273
25.5 Mocking Frameworks for Python . . . . . . . . . . . . . . . . . . . . . . . . . . 274
25.6 The Unittest.Mock Library . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 274
25.6.1 Mock and Magic Mock Classes . . . . . . . . . . . . . . . . . . . 275
25.6.2 The Patchers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 276
25.6.3 Mocking Returned Objects . . . . . . . . . . . . . . . . . . . . . . . 277
25.6.4 Validating Mocks Have Been Called . . . . . . . . . . . . . . . 278
25.7 Mock and MagicMock Usage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 279
25.7.1 Naming Your Mocks . . . . . . . . . . . . . . . . . . . . . . . . . . . . 279
25.7.2 Mock Classes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 279
25.7.3 Attributes on Mock Classes . . . . . . . . . . . . . . . . . . . . . . 279
25.7.4 Mocking Constants . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 280
25.7.5 Mocking Properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 280
25.7.6 Raising Exceptions with Mocks . . . . . . . . . . . . . . . . . . . 281
25.7.7 Applying Patch to Every Test Method . . . . . . . . . . . . . 281
25.7.8 Using Patch as a Context Manager . . . . . . . . . . . . . . . . 281
25.8 Mock Where You Use It . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 282
25.9 Patch Order Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 282
25.10 How Many Mocks? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 283
25.11 Mocking Considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 283
25.12 Online Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 284
25.13 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 284

Part V File Input/Output

26 Introduction to Files, Paths and IO . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 289
26.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 289
26.2 File Attributes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 292
26.3 Paths . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 293
26.4 File Input/Output . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 295
Contents xxi

26.5 Sequential Access versus Random Access . . . . . . . . . . . . . . . . . . 296

26.6 Files and I/O in Python . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 297
26.7 Online Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 297
27 Reading and Writing Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 299
27.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 299
27.2 Obtaining References to Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 299
27.3 Reading Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 301
27.4 File Contents Iteration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 302
27.5 Writing Data to Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 302
27.6 Using Files and with Statements . . . . . . . . . . . . . . . . . . . . . . . . . . . 303
27.7 The Fileinput Module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 303
27.8 Renaming Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 304
27.9 Deleting Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 304
27.10 Random Access Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 305
27.11 Directories . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 306
27.12 Temporary Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 307
27.13 Working with Paths . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 309
27.14 Online Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 312
27.15 Exercise . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 313
28 Stream IO . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 315
28.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 315
28.2 What is a Stream? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 315
28.3 Python Streams . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 316
28.4 IOBase . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 317
28.5 Raw IO/UnBuffered IO Classes . . . . . . . . . . . . . . . . . . . . . . . . . . . 318
28.6 Binary IO/Buffered IO Classes . . . . . . . . . . . . . . . . . . . . . . . . . . . . 318
28.7 Text Stream Classes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 320
28.8 Stream Properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 321
28.9 Closing Streams . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 322
28.10 Returning to the Open() Function . . . . . . . . . . . . . . . . . . . . . . . . . . 322
28.11 Online Resource . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 323
28.12 Exercise . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 323
29 Working with CSV Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 325
29.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 325
29.2 CSV Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 325
29.2.1 The CSV Writer Class . . . . . . . . . . . . . . . . . . . . . . . . . . . 326
29.2.2 The CSV Reader Class . . . . . . . . . . . . . . . . . . . . . . . . . . 327
29.2.3 The CSV DictWriter Class . . . . . . . . . . . . . . . . . . . . . . . 328
29.2.4 The CSV DictReader Class . . . . . . . . . . . . . . . . . . . . . . 329
29.3 Online Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 330
29.4 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 330
xxii Contents

30 Working with Excel Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 333

30.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 333
30.2 Excel Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 333
30.3 The Openpyxl. Workbook Class . . . . . . . . . . . . . . . . . . . . . . . . . . . 334
30.4 The Openpyxl. WorkSheet Objects . . . . . . . . . . . . . . . . . . . . . . . . . 334
30.5 Working with Cells . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 334
30.6 Sample Excel File Creation Application . . . . . . . . . . . . . . . . . . . . 335
30.7 Loading a Workbook from an Excel File . . . . . . . . . . . . . . . . . . . . 336
30.8 Online Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 338
30.9 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 338
31 Regular Expressions in Python . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 341
31.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 341
31.2 What Are Regular Expressions? . . . . . . . . . . . . . . . . . . . . . . . . . . . 341
31.3 Regular Expression Patterns . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 342
31.3.1 Pattern Metacharacters . . . . . . . . . . . . . . . . . . . . . . . . . . 342
31.3.2 Special Sequences . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 343
31.3.3 Sets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 344
31.4 The Python re Module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 345
31.5 Working with Python Regular Expressions . . . . . . . . . . . . . . . . . . 345
31.5.1 Using Raw Strings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 345
31.5.2 Simple Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 346
31.5.3 The Match Object . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 346
31.5.4 The search() Function . . . . . . . . . . . . . . . . . . . . . . . . . . . 347
31.5.5 The match() Function . . . . . . . . . . . . . . . . . . . . . . . . . . . 348
31.5.6 The Difference Between Matching
and Searching . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 349
31.5.7 The finadall() Function . . . . . . . . . . . . . . . . . . . . . . . . . . 349
31.5.8 The finditer() Function . . . . . . . . . . . . . . . . . . . . . . . . . . 349
31.5.9 The split() Function . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 350
31.5.10 The sub() Function . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 350
31.5.11 The compile() Function . . . . . . . . . . . . . . . . . . . . . . . . . . 351
31.6 Online Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 353
31.7 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 354

Part VI Database Access

32 Introduction to Databases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 359
32.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 359
32.2 What Is a Database? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 359
32.2.1 Data Relationships . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 360
32.2.2 The Database Schema . . . . . . . . . . . . . . . . . . . . . . . . . . . 361
32.3 SQL and Databases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 363
32.4 Data Manipulation Language . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 364
32.5 Transactions in Databases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 365
32.6 Further Reading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 366
Contents xxiii

33 Python DB-API . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 369

33.1 Accessing a Database from Python . . . . . . . . . . . . . . . . . . . . . . . . 369
33.2 The DB-API . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 369
33.2.1 The Connect Function . . . . . . . . . . . . . . . . . . . . . . . . . . . 370
33.2.2 The Connection Object . . . . . . . . . . . . . . . . . . . . . . . . . . 370
33.2.3 The Cursor Object . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 371
33.2.4 Mappings from Database Types to Python Types . . . . 372
33.2.5 Generating Errors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 372
33.2.6 Row Descriptions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 373
33.3 Transactions in PyMySQL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 374
33.4 Online Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 375
34 PyMySQL Module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 377
34.1 The PyMySQL Module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 377
34.2 Working with the PyMySQL Module . . . . . . . . . . . . . . . . . . . . . . 377
34.2.1 Importing the Module . . . . . . . . . . . . . . . . . . . . . . . . . . . 378
34.2.2 Connect to the Database . . . . . . . . . . . . . . . . . . . . . . . . . 378
34.2.3 Obtaining the Cursor Object . . . . . . . . . . . . . . . . . . . . . . 379
34.2.4 Using the Cursor Object . . . . . . . . . . . . . . . . . . . . . . . . . 379
34.2.5 Obtaining Information About the Results . . . . . . . . . . . 380
34.2.6 Fetching Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 380
34.2.7 Close the Connection . . . . . . . . . . . . . . . . . . . . . . . . . . . 381
34.3 Complete PyMySQL Query Example . . . . . . . . . . . . . . . . . . . . . . 381
34.4 Inserting Data to the Database . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 382
34.5 Updating Data in the Database . . . . . . . . . . . . . . . . . . . . . . . . . . . . 384
34.6 Deleting Data in the Database . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 385
34.7 Creating Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 386
34.8 Online Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 386
34.9 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 387

Part VII Logging

35 Introduction to Logging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 391
35.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 391
35.2 Why Log? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 391
35.3 What is the Purpose of Logging? . . . . . . . . . . . . . . . . . . . . . . . . . . 392
35.4 What Should You Log? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 392
35.5 What not to Log . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 393
35.6 Why not Just Use Print? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 394
35.7 Online Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 395
36 Logging in Python . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 397
36.1 The Logging Module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 397
36.2 The Logger . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 398
36.3 Controlling the Amount of Information Logged . . . . . . . . . . . . . . 399
36.4 Logger Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 401
xxiv Contents

36.5 Default Logger . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 402

36.6 Module Level Loggers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 403
36.7 Logger Hierarchy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 404
36.8 Formatters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 405
36.8.1 Formatting Log Messages . . . . . . . . . . . . . . . . . . . . . . . . 406
36.8.2 Formatting Log Output . . . . . . . . . . . . . . . . . . . . . . . . . . 406
36.9 Online Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 408
36.10 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 408
37 Advanced Logging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 409
37.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 409
37.2 Handlers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 409
37.2.1 Setting the Root Output Handler . . . . . . . . . . . . . . . . . . 411
37.2.2 Programmatically Setting the Handler . . . . . . . . . . . . . 412
37.2.3 Multiple Handlers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 413
37.3 Filters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 415
37.4 Logger Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 416
37.5 Performance Considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 418
37.6 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 419

Part VIII Concurrency and Parallelism

38 Introduction to Concurrency and Parallelism . . . . . . . . . . . . . . . . . . . . 423
38.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 423
38.2 Concurrency . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 423
38.3 Parallelism . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 425
38.4 Distribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 426
38.5 Grid Computing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 427
38.6 Concurrency and Synchronisation . . . . . . . . . . . . . . . . . . . . . . . . . 428
38.7 Object Orientation and Concurrency . . . . . . . . . . . . . . . . . . . . . . . 428
38.8 Threads V Processes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 429
38.9 Some Terminology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 430
38.10 Online Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 430
39 Threading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 433
39.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 433
39.2 Threads . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 433
39.2.1 Thread States . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 433
39.2.2 Creating a Thread . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 434
39.2.3 Instantiating the Thread Class . . . . . . . . . . . . . . . . . . . . 435
39.3 The Thread Class . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 436
39.4 The Threading Module Functions . . . . . . . . . . . . . . . . . . . . . . . . . . 438
39.5 Passing Arguments to a Thread . . . . . . . . . . . . . . . . . . . . . . . . . . . . 438
39.6 Extending the Thread Class . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 440
39.7 Daemon Threads . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 441
Contents xxv

39.8 Naming Threads . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 442

39.9 Thread Local Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 442
39.10 Timers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 444
39.11 The Global Interpreter Lock . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 445
39.12 Online Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 446
39.13 Exercise . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 446
40 MultiProcessing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 449
40.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 449
40.2 The Process Class . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 449
40.3 Working with the Process Class . . . . . . . . . . . . . . . . . . . . . . . . . . . 451
40.4 Alternative Ways to Start a Process . . . . . . . . . . . . . . . . . . . . . . . . 453
40.5 Using a Pool . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 454
40.6 Exchanging Data Between Processes . . . . . . . . . . . . . . . . . . . . . . . 458
40.7 Sharing State Between Processes . . . . . . . . . . . . . . . . . . . . . . . . . . 460
40.7.1 Process Shared Memory . . . . . . . . . . . . . . . . . . . . . . . . . 460
40.8 Online Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 461
40.9 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 462
41 Inter Thread/Process Synchronisation . . . . . . . . . . . . . . . . . . . . . . . . . . . 463
41.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 463
41.2 Using a Barrier . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 463
41.3 Event Signalling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 467
41.4 Synchronising Concurrent Code . . . . . . . . . . . . . . . . . . . . . . . . . . . 469
41.5 Python Locks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 470
41.6 Python Conditions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 472
41.7 Python Semaphores . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 475
41.8 The Concurrent Queue Class . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 476
41.9 Online Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 478
41.10 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 478
42 Futures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 481
42.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 481
42.2 The Need for a Future . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 481
42.3 Futures in Python . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 482
42.3.1 Future Creation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 483
42.3.2 Simple Example Future . . . . . . . . . . . . . . . . . . . . . . . . . . 483
42.4 Running Multiple Futures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 485
42.4.1 Waiting for All Futures to Complete . . . . . . . . . . . . . . . 486
42.4.2 Processing Results as Completed . . . . . . . . . . . . . . . . . 488
42.5 Processing Future Results Using a Callback . . . . . . . . . . . . . . . . . 489
42.6 Online Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 490
42.7 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 490
xxvi Contents

43 Concurrency with AsyncIO . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 493

43.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 493
43.2 Asynchronous IO . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 493
43.3 Async IO Event Loop . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 494
43.4 The Async and Await Keywords . . . . . . . . . . . . . . . . . . . . . . . . . . . 495
43.4.1 Using Async and Await . . . . . . . . . . . . . . . . . . . . . . . . . . 496
43.5 Async IO Tasks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 498
43.6 Running Multiple Tasks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 500
43.6.1 Collating Results from Multiple Tasks . . . . . . . . . . . . . 500
43.6.2 Handling Task Results as They Are Made
Available . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 501
43.7 Online Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 502
43.8 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 502
44 Performance Monitoring and Profiling . . . . . . . . . . . . . . . . . . . . . . . . . . 505
44.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 505
44.2 Why Monitor Performance and Memory? . . . . . . . . . . . . . . . . . . . 505
44.3 Performance Monitoring and Profiling . . . . . . . . . . . . . . . . . . . . . 506
44.4 Performance Monitoring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 507
44.4.1 The Time Module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 507
44.4.2 The Timeit Module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 508
44.4.3 The Psutil Module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 510
44.5 Python Profiling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 511
44.5.1 The cProfile Module . . . . . . . . . . . . . . . . . . . . . . . . . . . . 511
44.5.2 The Line_Profiler Module . . . . . . . . . . . . . . . . . . . . . . . 511
44.5.3 The Memory_Profiler Module . . . . . . . . . . . . . . . . . . . . 512
44.5.4 Additional Third-Party Libraries . . . . . . . . . . . . . . . . . . 512
44.6 Profiling with cProfile . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 512
44.7 Memory Profiling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 515
44.8 Online Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 517

Part IX Reactive Programming

45 Reactive Programming Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . 521
45.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 521
45.2 What Is a Reactive Application? . . . . . . . . . . . . . . . . . . . . . . . . . . . 521
45.3 The ReactiveX Project . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 522
45.4 The Observer Pattern . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 522
45.5 Hot and Cold Observables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 523
45.6 Differences Between Event Driven Programming
and Reactive Programming . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 524
45.7 Advantages of Reactive Programming . . . . . . . . . . . . . . . . . . . . . . 525
45.8 Disadvantages of Reactive Programming . . . . . . . . . . . . . . . . . . . 525
45.9 The RxPy Reactive Programming Framework . . . . . . . . . . . . . . . 526
45.10 Online Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 526
Contents xxvii

46 RxPy Observables, Observers and Subjects . . . . . . . . . . . . . . . . . . . . . . 527

46.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 527
46.2 RxPy Library . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 527
46.3 Observables in RxPy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 527
46.4 Observers in RxPy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 528
46.5 Multiple Subscribers/Observers . . . . . . . . . . . . . . . . . . . . . . . . . . . 530
46.6 Subjects in RxPy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 531
46.7 Observer Concurrency . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 534
46.7.1 Available Schedulers . . . . . . . . . . . . . . . . . . . . . . . . . . . . 535
46.8 Online Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 536
46.9 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 536
47 RxPy Operators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 537
47.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 537
47.2 Reactive Programming Operators . . . . . . . . . . . . . . . . . . . . . . . . . . 537
47.3 Piping Operators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 538
47.4 Creational Operators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 539
47.5 Transformational Operators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 539
47.6 Combinatorial Operators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 541
47.7 Filtering Operators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 542
47.8 Mathematical Operators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 543
47.9 Chaining Operators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 544
47.10 Online Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 546
47.11 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 546

Part X Network Programming

48 Introduction to Sockets and Web Services . . . . . . . . . . . . . . . . . . . . . . . 551
48.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 551
48.2 Sockets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 551
48.3 Web Services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 552
48.4 Addressing Services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 552
48.5 Localhost . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 553
48.6 Port Numbers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 554
48.7 IPv4 Versus IPv6 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 555
48.8 Sockets and Web Services in Python . . . . . . . . . . . . . . . . . . . . . . . 555
48.9 Online Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 556
49 Sockets in Python . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 557
49.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 557
49.2 Socket to Socket Communication . . . . . . . . . . . . . . . . . . . . . . . . . . 557
49.3 Setting up a Connection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 558
49.4 An Example Client Server Application . . . . . . . . . . . . . . . . . . . . . 558
49.4.1 The System Structure . . . . . . . . . . . . . . . . . . . . . . . . . . . 558
49.4.2 Implementing the Server Application . . . . . . . . . . . . . . 559
xxviii Contents

49.4.3 Socket Types and Domains . . . . . . . . . . . . . . . . . . . . . . . 560

49.4.4 Implementing the Client Application . . . . . . . . . . . . . . 561
49.5 The Socketserver Module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 562
49.6 Http Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 565
49.7 Online Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 567
49.8 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 568
50 Web Services in Python . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 571
50.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 571
50.2 RESTful Services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 571
50.3 A RESTful API . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 572
50.4 Python Web Frameworks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 573
50.5 Online Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 574
51 Flask Web Services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 575
51.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 575
51.2 Flask . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 575
51.3 Hello World in Flask . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 576
51.3.1 Using JSON . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 576
51.4 Implementing a Flask Web Service . . . . . . . . . . . . . . . . . . . . . . . . 577
51.4.1 A Simple Service . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 577
51.4.2 Providing Routing Information . . . . . . . . . . . . . . . . . . . 577
51.5 Running the Service . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 578
51.6 Invoking the RESTFul Service . . . . . . . . . . . . . . . . . . . . . . . . . . . . 579
51.6.1 The Final Solution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 580
51.7 Online Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 580
52 Flask Bookshop Web Service . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 583
52.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 583
52.2 Building a Flask Bookshop Service . . . . . . . . . . . . . . . . . . . . . . . . 583
52.3 The Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 583
52.4 The Domain Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 585
52.5 Encoding Books into JSON . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 586
52.6 Setting Up the GET Services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 588
52.7 Deleting a Book . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 589
52.8 Adding a New Book . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 590
52.9 Updating a Book . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 592
52.10 What Happens if We Get It Wrong? . . . . . . . . . . . . . . . . . . . . . . . . 593
52.11 Bookshop Services Listing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 594
52.12 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 596

Part XI Data Science: Data Analytics and Machine Learning

53 Introduction to Data Science . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 601
53.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 601
53.2 Data Science . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 601
53.3 Data Science Tools and Techniques . . . . . . . . . . . . . . . . . . . . . . . . 602
Contents xxix

53.4 Data Analytics Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 604

53.5 Python and Data Science . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 606
53.6 Machine Learning for Data Science . . . . . . . . . . . . . . . . . . . . . . . . 607
53.7 Online Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 608
54 Pandas and Data Analytics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 611
54.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 611
54.2 The Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 611
54.2.1 The UK Government COVID Data Set . . . . . . . . . . . . . 611
54.2.2 The Google Mobility Data Set . . . . . . . . . . . . . . . . . . . . 613
54.3 Python Pandas . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 614
54.3.1 Pandas Series and DataFrames . . . . . . . . . . . . . . . . . . . . 615
54.4 Loading and Analysing UK COVID Data Set . . . . . . . . . . . . . . . . 616
54.5 Loading the Google Mobility Data Set . . . . . . . . . . . . . . . . . . . . . 621
54.6 Merging Two DataFrames . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 622
54.7 Analysing the Combined Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . 623
54.8 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 627
55 Alternatives to Pandas . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 629
55.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 629
55.2 Comparing Pandas 2.0.0 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 629
55.3 Pandas 1.x v 2.x . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 629
55.4 Pandas Versus Other Libraries and Tools . . . . . . . . . . . . . . . . . . . . 630
55.5 Online Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 632
56 Machine Learning in Python . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 633
56.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 633
56.2 The Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 633
56.3 SciKitLearn . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 634
56.4 The Problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 635
56.5 Using Regression Supervised Learning Systems . . . . . . . . . . . . . 636
56.6 K-Nearest Neighbour Regressor . . . . . . . . . . . . . . . . . . . . . . . . . . . 636
56.7 Decision Tree Regressor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 638
56.8 Random Forest Regressor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 639
56.9 Summary of Metrics Obtained . . . . . . . . . . . . . . . . . . . . . . . . . . . . 640
56.10 Creating the Regressor Object . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 640
56.11 Online Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 642
57 Pip and Conda Virtual Environments . . . . . . . . . . . . . . . . . . . . . . . . . . . 643
57.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 643
57.2 Virtual Environments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 643
57.3 Working with Pip . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 644
57.3.1 Activating a Pip Environment . . . . . . . . . . . . . . . . . . . . 644
57.3.2 Installing Modules Using Pip . . . . . . . . . . . . . . . . . . . . . 645
57.3.3 Deactivating a Pip Environment . . . . . . . . . . . . . . . . . . 646
57.3.4 Check Version of Pip . . . . . . . . . . . . . . . . . . . . . . . . . . . . 646
57.3.5 Installing Modules into a Pip Environment . . . . . . . . . 646
xxx Contents

57.3.6 Freezing Modules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 647

57.4 Conda . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 648
57.5 Anaconda . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 649
57.5.1 Installing Anaconda . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 649
57.6 Working with Anaconda . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 651
57.6.1 Checking the Conda Version . . . . . . . . . . . . . . . . . . . . . 651
57.6.2 Updating Conda . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 651
57.6.3 Creating a Conda Environment . . . . . . . . . . . . . . . . . . . 652
57.6.4 Listing Available Conda Environments . . . . . . . . . . . . 653
57.6.5 Activating a Conda Environment . . . . . . . . . . . . . . . . . . 654
57.6.6 Deactivating a Conda Environment . . . . . . . . . . . . . . . . 655
57.6.7 Listing the Modules Loaded into a Conda
Environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 655
57.6.8 Removing an Anaconda Environment . . . . . . . . . . . . . 656
57.6.9 Installing a Module into a Conda Environment . . . . . . 656
57.7 Anaconda in PyCharm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 658
57.8 Online Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 658
Chapter 1
Introduction

1.1 Introduction

I have heard many people over the years say that Python is an easy language to learn
and that Python is also a simple language.
To some extent both of these statements are true; but only to some extent.
While the core of the Python language is easy to lean and relatively simple (in
part thanks to its consistency), the sheer richness of the language constructs and
flexibility available can be overwhelming. In addition the Python environment, its
eco system, the range of libraries available, the often competing options available,
etc., can make moving to the next-level daunting.
Once you have learned the core elements of the language such as how classes
and inheritance work, how functions work, what are protocols and Abstract Base
Classes, etc. where do you go next?
The aim of this book is to delve into those next steps. The book is organised into
eleven different topics:
1. Advanced Language Features. The first section in the book covers topics
that are often missed out from introductory Python books such as slots, weak
references __init__() versus __new__() and metaclasses.
2. Computer Graphics. The book covers Computer Graphics and Computer
Generated Art in Python as well as graphical user interfaces and graphing/
charting via Matplotlib.
3. Games Programming. This topic is covered using the pygame library.
4. Testing and Mocking. Testing is an important aspect of any software develop-
ment; this book introduces testing in general and the PyTest module in detail.
It also considers mocking within testing including what and when to mock.

© Springer Nature Switzerland AG 2023 1

J. Hunt, Advanced Guide to Python 3 Programming,
Undergraduate Topics in Computer Science,
https://doi.org/10.1007/978-3-031-40336-1_1
2 1 Introduction

5. File Input/Output. The book covers text file reading and writing as well as
reading and writing CSV and Excel files. Although not strictly related to file
input, regulator expressions are included in this section as they can be used to
process textual data held in files.
6. Database Access. The book introduces databases and relational database in
particular. It then presents the Python DB-API database access standard and
one implementation of this standard, the PyMySQL module used to access a
MySQL database.
7. Logging. An often missed topic is that of logging. The book therefore introduces
logging the need for logging, what to log and what not to log as well as the Python
logging module.
8. Concurrency and Parallelism. The book provides extensive coverage of
concurrency topics including threads, processes and inter-thread or process
synchronisation. It also presents futures and AsyncIO.
9. Reactive Programming. This section of the book introduces Reactive Program-
ming using the PyRx Reactive Programming library.
10. Network Programming. The book introduces socket and web service commu-
nications in Python. It looks at both the Flask and the Django web service
libraries.
11. Data Analytics. A very hot topic for any potential Python programmer is data
analytics (and the related use of machine learning). The book concludes by
introducing these topics and there Pandas and scikit-learn (or SK-learn as it is
sometimes known) libraries.
Each section is introduced by a chapter providing the background and key concepts
of that topic. Subsequent chapters then cover various aspects of the topic.
For example, the second topic covered is on Computer Graphics. This section
has an introductory chapter on Computer Graphics in general. It then introduces the
Turtle Graphics Python library which can be used to generate a graphical display.
The following chapter considers the subject of Computer Generated Art and
uses the Turtle Graphics library to illustrate these ideas. Thus several examples
are presented that might be considered art. The chapter concludes by presenting the
well-known Koch Snowflake and the Mandelbrot Fractal set.
This is followed by a chapter presenting the Matplotlib library used for generating
2D and 3D charts and graphs (such as a line chart, bar chart or scatter graph).
The section concludes with a chapter on graphical user interfaces (or GUIs) using
the wxpython library. This chapter explores what we mean by a GUI and some of
the alternatives available in Python for creating a GUI.
Other topics follow a similar pattern.
Each programming or library-oriented chapter also includes numerous sample
programs that can be downloaded from the GitHub repository and executed. These
chapters also include one or more end of chapter exercises (with sample solutions
also in the GitHub repository).
1.2 Useful Python Resources 3

The topics within the book can be read mostly independently of each other. This
allows the reader to dip into subject areas as and when required. For example, the
File Input/Output section and the Database Access section can read independently
of each other (although in this case assessing both technologies may be useful in
selecting an appropriate approach to adopt for the long-term persistent storage of
data in a particular system).
Within each section there are usually dependencies; for example, it is neces-
sary to understand pygame library from the ‘Building Games with pygame’ intro-
ductory chapter, before exploring the worked case study presented by the chapter
on the StarshipMeteors game. Similarly it is necessary to have read the threading
and multiprocessing chapters before reading the inter-thread/process synchronisation
chapter.

1.2 Useful Python Resources

There are a wide range of resources on the web for Python; we will highlight a few
here that you should bookmark. We will not keep referring to these to avoid repetition
but you can refer back to this section whenever you need to:
• https://en.wikipedia.org/wiki/Python_Software_Foundation Python Software
Foundation.
• https://docs.python.org/3/ The main Python 3 documentation site. It contains
tutorials, library references, set up and installation guides as well as Python
how-tos.
• https://docs.python.org/3/library/index.html A list of all the built-in features for
the Python language—this is where you can find online documentation for the
various class and functions that we will be using throughout this book.
• https://pymotw.com/3/ the Python 3 Module of the week site. This site contains
many, many Python modules with short examples and explanations of what the
modules do. A Python module is a library of features that build on and expand
the core Python language. For example, if you are interested in building games
using Python then pygame is a module specifically designed to make this easier.
• https://www.fullstackpython.com/email.html is a monthly newsletter that
focusses on a single Python topic each month, such as a new library or module.
• http://www.pythonweekly.com/ is a free weekly summary of the latest Python
articles, projects, videos and upcoming events.
Each section of the book will provide additional online references relevant to the
topic being discussed.
 Part I
Advanced Language Features
Chapter 2
Python Type Hints

2.1 Introduction

Python is a dynamically typed language right—well yes it is however there is a

feature known as Type Hints that allows typing information to be provided when
functions and methods are defined. These Type Hints are extremely useful and can
help a developer understand what types are expected by a function or a method and
indeed what types are likely to be returned.
Recent versions of Python, including 3.10 and 3.11, have increased the support
for Type Hints so that they are now quite usable. These can be used by analysis tools
and IDEs to help developers create more stable and reliable applications.

2.2 Pythons Type System

Many people consider Python to be an untyped programming language. However,

that is not quite true. The type system in Python can be referred to as representing
a dynamically typed programming language. That is a variable holds a value and
the type of that value is known and understood by the language. At runtime Python
checks that what you are trying to do is valid given the types involved. For example,
you can use the type() function to find out what type of thing a variable holds at
any point in time:
a_variable = 42
print(f'a_variable type = {type(a_variable)} = {a_variable}')
a_variable = 1.345
print(f'a_variable type = {type(a_variable)} = {a_variable}')
a_variable = "Hello"
print(f'a_variable type = {type(a_variable)} = {a_variable}')
a_variable = True
print(f'a_variable type = {type(a_variable)} = {a_variable}')

© Springer Nature Switzerland AG 2023 7

J. Hunt, Advanced Guide to Python 3 Programming,
Undergraduate Topics in Computer Science,
https://doi.org/10.1007/978-3-031-40336-1_2
8 2 Python Type Hints

This produces as output:

a_variable type = <class 'int'> = 42
a_variable type = <class 'float'> = 1.345
a_variable type = <class 'str'> = Hello
a_variable type = <class 'bool'> = True

Of course, as the above shows, a variable in Python can hold different types of
things at different types, hence the term dynamically typed.
As Python knows what types variables hold it can check at runtime that your
programs are valid/correct given the types involved for example in a particular oper-
ation. Thus, it is valid to add two integers together and indeed two strings together
(as this is string concatenation) but attempting to add an integer to a string will result
in a TypeError:
print(1 + 1)
print(1.2 + 3.4)
print("Hello" + "world")
print("Hello" + 1)

This code produced the following output including the TypeError:

2
4.6
Helloworld
Traceback (most recent call last):
File "/Users/jeh/temp/pythonProjects/course/main.py", line 15,
in <module>
print("Hello" + 1)
TypeError: can only concatenate str (not "int") to str

2.3 The Challenge for Python Developers

The challenge for Python developers comes when they need to understand what types
are required by, or work with, some API. As a very simple example, consider the
following function:
def add(x, y):
return x + y

What types can be used with this function?

In essence any type can be used for the parameter x that supports the plus operator
(+) with the type in y. From the above we know that integers and strings can be used,
but we can also use floating point numbers, for example:
print(add(1, 2))
print(add(1.2, 3.4))
2.4 Static Typing 9

print(add(1, 3.4))
print(add(5.5, 1))
print(add("Hi", "There"))

All of the above are valid parameters, and the output produced from the above
code is:
3
4.6
4.4
6.5
HiThere

Even custom types can be used if they implement the special __add__(self,
other) operator method, for example:
class Quantity:
def __init__(self, amount):
self.amount = amount

def __add__(self, other):

return Quantity(self.amount + other.amount)

def __str__(self):
return f"Quantity({self.amount})"

q1 = Quantity(5)
q2 = Quantity(4)
print(add(q1, q2))

The __add__() method allows the custom type (class) being defined to be used
with the add operator (‘+’). Thus this program generates the following output:
Quantity(9)

However, what was the intent of the designer of this add() function? What did
they expect you to add together? The only option in traditional Python code is for
the developer to provide some form of documentation, for example in the form of a
docstring:
def add(x, y):
"""adds two integers together and
returns the resulting integer."""
return x + y

2.4 Static Typing

Languages such as Java, C# and C are statically typed languages. That is when a
variable, object attribute, parameter or return type is defined then the type of that
element is specified statically a compile time.
10 2 Python Type Hints

Thus, an add() method on a Java class Calculator might be written as

follows:
package com.jjh;

public class Calculator {

public int add(int x, int y) {

return x + y;
}

This makes it clear to a Java programmer and to the Java compiler that the add()
method will only handle integers and will return as a result an integer type. Thus,
there is no possibility that a developer might try to add a number to a Boolean value,
etc. In fact the compiler will not even allow it!
The Java Calculator class can be used as shown below, note that this code
will not even compile if the developer tries to add two strings together. In this case
we are adding two integers together, so all is fine:
package com.jjh;

public class App {

public static void main( String[] args ) {
System.out.println( "Starting" );
Calculator calc = new Calculator();
System.out.println(calc.add(4, 5));
System.out.println("Done");
}
}

As the above program uses valid integer types with the add() method, the output
from the compiled and executed program is:
Starting
9
Done

2.5 Python Type Hints

Python’s Type Hints are more like a half-way house between traditional Python’s
lack of typing information at all and the very strict string static typing approach of
languages such as Java.
A Type Hint is additional type information that can be used with a function
definition to indicate what types parameters should be and what type is returned.
This is illustrated below:
def add(x: int, y: int) -> int:
2.6 Type Hint Layout 11

return x + y

In this case it makes it clear that both x and y should be of type int (integer
types) and the returned result will be an int. However, adding Type Hints as shown
above has no effect on the runtime execution of the program; they are only hints and
are not enforced by Python per se. For example, it is still possible to pass a string
into the add() function as far as Python is concerned.
However, static analysis tools (such as MyPy) can be applied to the code to check
for such misuse. Some editors, such as the widely used PyCharm, already have such
tools integrated into their code checking behaviour.
If you want to use a tool such as mypy instead, or in addition to that available in
your IDE, then you can install it using
pip install mypy

Or if you want to use conda/Anaconda by using

conda install mypy

You can now analyse your code by applying MyPy to a Python file, for example:
% mypy main.py
main.py:3: error: Incompatible types in assignment (expression has
type "float", variable has type "int")
main.py:5: error: Incompatible types in assignment (expression has
type "str", variable has type "int")
main.py:24: error: Argument 1 to "add" has incompatible type "str";
expected "int"
main.py:24: error: Argument 2 to "add" has incompatible type "str";
expected "int"
main.py:44: error: Argument 1 to "add" has incompatible type
"Quantity"; expected "int"
main.py:44: error: Argument 2 to "add" has incompatible type
"Quantity"; expected "int"
Found 6 errors in 1 file (checked 1 source file)

2.6 Type Hint Layout

The Python Style Guide defined by Python Enhancement Proposal 8 (PEP 8) provides
some guidance for using Type Hints, for example:
• Use normal rules for colons, that is, no space before and one space after a colon:
text: str.
• Use spaces around the = sign when combining an argument annotation with a
default value: align: bool = True.
• Use spaces around the -> arrow: def headline(…) -> str.
12 2 Python Type Hints

2.7 Type Hints for Multiple Types

Of course our add() function could work with floating point numbers as well as it
works with integers. It would therefore be useful to be able to state this in terms of the
Type Hints. Prior to Python 3.10 this could be done using a Union type, for example
Union[int, float] which while it worked was a little unwieldy. Since Python
3.10 we can use the style syntax bar ‘|’ for example int | float as shown below:
def add(x: int | float, y: int | float) -> int:
return x + y

2.8 The Self Type

Python 3.11 introduced the Self type which is defined in PEP 673. This can be used
to indicate that a method returns a reference to itself, for example:
from typing import Self

class Shape:

def __init__(self):
self.scale = 0.0

def set_scale(self, scale: float) -> Self:

self.scale = scale
return self

2.9 The Benefits of Type Hints

There are a range of benefits to using Type Hints in Python, for example:
• They help catch some errors within programs. Obviously, the biggest benefit
is that Type Hints can help developers catch certain types of problems in their
code (assuming that some form of type checker is used).
• They provide documentation. Type Hints can also act as a level of document
that editors such as IDEs can pick up and display to other developers.
• They can be work with IDEs. They can help with code generation and IDE
auto-complete functionality.
• They can make developers stop and think. They can help ensure that developers
think about their code and what types should be supported.
• They can improve understanding of libraries. Although Type Hints may offer
little advantage in a single use script, or throw away program, they can be of
significant benefit when a library is being created. Such libraries will be used
2.11 Online Resources 13

by a range of different developers, and some may be released into the wild, for
example via PyPI, the Python Package Index. The use of Type Hints can greatly
enhance others understanding of the APIs provided by these libraries.

2.10 Summary

If you are just starting out with Python, or you are writing scripts that will only be used
once, then Type Hints may not be particularly useful. However, if you are creating
libraries or developing larger more complex applications with teams of developers,
then they can be very useful indeed.

2.11 Online Resources

• https://docs.python.org/3.10/ Python 3.10 documentation.

• https://docs.python.org/3.11/ Python 3.11 documentation.
• https://en.wikipedia.org/wiki/Type_system Wikipedia Type System page.
• https://en.wikipedia.org/wiki/Dynamic_programming_language Dynamically
typed languages.
• https://docs.python.org/3/library/exceptions.html Exception handling in Python.
• https://www.pythontutorial.net/python-basics/python-type-hints/ Tutorial on
Python Type Hints.
• https://pypi.org/project/mypy/ MyPy static type hint analysis tool.
• https://mypy.readthedocs.io/en/stable/ MyPy documentation.
• https://www.jetbrains.com/pycharm/ PyCharm IDE tool.
• https://peps.python.org/pep-0008/ Python (PEP 8) Style Guide including guid-
ance of how to layout Type Hints.
• https://peps.python.org/pep-0673/ Information on the Self type.
• https://pypi.org/ The Python package Index PyPi.
Chapter 3
Class Slots

3.1 Introduction

Python classes are very flexible, they allow data and behaviour to be defined when
the class is created, but also dynamically at any point in the lifetime of the class and
its instances. This technique is known as Monkey Patching and can be extremely
useful. However, in other situations, allowing the data or behaviour of a class to
change dynamically after the class has been defined, might be very confusing and
make the system harder to maintain. The issue is that a class’s attributes can be added
at any time, and there is no formal specification of the attributes—that is until we
look at slots. Slots allow us to specify what attributes a class will have and to ensure
that those attributes and only those attributes are used with the class and its instances.
This chapter introduces Python class slots.

3.2 The Problem

In many object-oriented languages it is necessary to define the attributes or fields

that a class will have explicitly within the class definition. For example, in Java we
might write the following:
public class Person {
private int age = 0;
private String name = "";

public Person (name String, int age) {

this.name = name;
this.age = age;
}
public int getAge () {
return age;

© Springer Nature Switzerland AG 2023 15

J. Hunt, Advanced Guide to Python 3 Programming,
Undergraduate Topics in Computer Science,
https://doi.org/10.1007/978-3-031-40336-1_3
16 3 Class Slots

}
public void setAge (int newAge) {
age = newAge;
}
public String getName () {
return name;
}
public void birthday () {
int oldAge, newAge;
oldAge = getAge();
System.out.println("Happy birthday " + getName());
System.out.print("You were " + oldAge);
System.out.print(" but now you are ");
this.age = this.age + 1;
System.out.println(age);
}
}

In the above class the two lines:

private int age = 0;
private String name = "";

declare that the Java class Person will have two attributes (also known as instance
variables in Java) called name and age. It cannot have any dynamically added
additional attributes, and it is not possible to create on the fly within a method.
The equivalent class definition in Python might look like:
class Person:
def __init__(self, name, age):
self.name = name
self.age = age

def birthday(self):
print(f'Happy birthday {self.name}',
f', you were {self.age}',
end = '')
self.age = self.age + 1
print(f' but now you are {self.age}')

This does essentially the same thing (although in a more concise format). Using
this class we can create instances of the class and print out the details associated with
the object, for example:
p1 = Person('Phoebe', 25)
print(f'p1: {p1.name} {p1.age}')

which produces:
p1: Phoebe 25
3.3 Slots to the Rescue 17

However there is nothing to stop us adding a new attribute address to the class,
for example:
p1.address = '10 High Street'
print(f'p1.address: {p1.address}')

When we run this we get:

p1.address: 10 High Street

This is not possible in Java as address was not defined within the scope of
the class and thus objects of class Person in Java can never have any additional
attributes such as address.
In Python not only is this legal, it is also sometimes quite useful.
However, how do you know that there is an attribute address on the object in
p1? Only be reading through the code using p1, you cannot see it by looking at the
Python class definition for Person.
Perhaps even more confusingly if we write:
p2 = Person('Gryff', 24)
print(f'p2: {p2.name} {p2.age}')
print(f'p2.address: {p2.address}')

That is we create a new instance of the class Person and try to access the
attributes name, age and address (which apparently all work for the instance in
p1), and we will raise a runtime AttributeError, for example:
p2: Gryff 24
Traceback (most recent call last):
File "/person.py", line 19, in <module>
print(f'p2.address: {p2.address}')
^^^^^^^^^^
AttributeError: 'Person' object has no attribute 'address'

This indicates that the Person object in p2 does not possess an attribute
address yet p1 did! Of course this is because we added the attribute only to
the object in p1 not to the class in general.
This can be very confusing and make maintaining code much more difficult!

3.3 Slots to the Rescue

There is a special class attribute called __slots__ which can be used to provide a
sequence of strings that define or specify the attributes that the class will hold. It is
a class attribute as it is part of the class not part of an instance of object of the class.
However, it defines the attributes any instance of the class can use.
Thus if an attribute is not included in the slots sequence then it cannot be defined
within the class. This means that to find out what attributes a class defines for its
objects all you have to do is look at the slots attribute and they will be listed there.
18 3 Class Slots

It also means that it is not possible to dynamically monkey patch a class with
additional attributes at runtime.
As an example, see the modified definition for the class Person below:
class Person:
__slots__ = ['name', 'age']

def __init__(self, name, age):

self.name = name
self.age = age

def __repr__(self):
return f'Person({self.name} is {self.age})'

This version of the class Person lists the attributes name and age in the _
_slots__ class attribute. Note that the names of the attributes are defined as
strings—so don’t forget the quotes around each attribute.
Then within the class an initialiser sets up the values for the attributes
self.name, self.age, etc.
We can now create an instance of this class and for example print out the age, the
name and use the __repr__() method to convert the object to a string for printing
purposes:
p1 = Person('Phoebe', 25)
print(p1)
print(f'p1: {p1.name} {p1.age}')

The output from this is:

Person(Phoebe is 25)
p1: Phoebe 25

So far so good, but what has this given us over the original version?
If we now try to dynamically add an attribute such as address to this version
of Person, for example:
p1.address = '10 High Street'

We will now generate a runtime AttributeError indicating that the object

does not have an attribute address, for example:
Traceback (most recent call last):
File "main.py", line 16, in <module>
p1.address = '10 High Street'
^^^^^^^^^^
AttributeError: 'Person' object has no attribute 'address'

We have now fixed the attributes defined within the class to be name and age
and only ever name and age.
3.4 Performance Benefits 19

This is actually true even if we tried to define an additional attribute within the
initialiser method, for example, in the following version of the Person class has
added the self.address attribute within the __init__() method:
class Person:
__slots__ = ['name', 'age']

def __init__(self, name, age):

self.name = name
self.age = age
self.address = None

def __repr__(self):
return f'Person({self.name} is {self.age})

When we try and use this class to create a new instance of the class Person we
again get an AttributeError raised:
Traceback (most recent call last):
File "main.py", line 13, in <module>
p1 = Person('Phoebe', 25)
^^^^^^^^^^^^^^^^^^^^
File "main.py", line 7, in __init__
self.address = None
^^^^^^^^^^^^
AttributeError: 'Person' object has no attribute 'address'

Thus we are guaranteed that the class Person and all instances of the class
Person, all work with just the name and age specified in the __slots__ class
attribute.

3.4 Performance Benefits

There are in fact additional benefits to be had from using the __slots__ class
attribute. These benefits relate to performance. This is because attributes defined
using slots are more efficient in terms of memory space and speed of access and a
bit safer than the default Python method of data access.
By default, when Python creates a new instance of a class, it creates a
__dict__ attribute for the class. The __dict__ attribute is a dictionary whose
keys are the variable names and whose values are the variable values. This allows
for dynamic variable creation but can also lead to uncaught errors.
The fact that under the hood a simple dict is used for attribute storage and lookup
as a few implications:
• Dictionaries are memory expensive objects. While this may not be a problem for
a small class or for a class with only a few instances it can become far more
significant with millions of objects as they will use a lot of memory.
20 3 Class Slots

• Dictionaries are based on a hash map. In the worst-case scenarios the time
complexity of the get and set options on a hash map is of O(n) that is of Order(n).
This means that when the time taken to access a value is measured as the number
of elements in the dictionary increases so does the access time and that it increases
linearly.
When using slots the attributes are created directly as properties of the class and
by pass the default dict-based implementation. This is both more efficient in terms
of access times and in terms of memory usage.
To illustrate this we can compare the size of the object created for the first version
and the second version of the Person class.
The size of the object created without using slots is:
print(f'sys.getsizeof(p1) noslots - {sys.getsizeof(p1)}')

which generates:
sys.getsizeof(p1) noslots - 56

And if we run the version using slots:

print(f'sys.getsizeof(p1) slots - {sys.getsizeof(p1)}')

This produces:
sys.getsizeof(p1) slots - 48

which is 8 bytes less. This may not sound like much but multiple this by a million
times and it starts becoming significant!

3.5 Why Not Use Slots?

So if slots are so good why doesn’t everyone use them for all classes? In short although
many people consider them a very useful features there as many who consider them
poorly understood, difficult to get right and restrictive.
The main issue comes down to flexibility—do you want or need to be able to
dynamically update the attributes in your class? If the answer is yes then slots are
not for you, if you don’t then slots may well be advantageous. Although the counter-
argument is that you are making this design decision which will impact anyone who
uses your class at any time in future!
Additionally by default slots cannot be used with weak references. Thus if you
want your class to use weak references you can’t use __slots__.
Although it should be noted that if you want to use slots with dynamic attributes
or indeed with weak references then you can do so by adding ‘__dict__’ or ‘__
weakref__’ as the last element in the __slots__ declaration.
3.6 Online Resource 21

The final issue might be that a library you are using might rely on the pres-
ence of a __dict__ for attributes for it to function correctly. For example, the
functools.cached_property() is an example of a function that requires an
instance __dict__ to function correctly.

3.6 Online Resource

• https://wiki.python.org/moin/UsingSlots Tutorial on using slots.

Chapter 4
Weak References

4.1 Introduction

In this chapter we will look at weak references. Most of the time an object has one or
more strong references associated with it. As long as at least one strong reference is
associated with an object it cannot be automatically garbage collected. However, in
some situations it may be useful to allow an object to be garbage collected (removed
from memory) while a program is still using it, particularly if the available memory
is becoming limited. To allow this to happen we can use weak references; weak
references do not stop an object being removed from memory and can be very useful
when used with a data cache, etc.

4.2 How Garbage Collection Works: Reference Counting

Before we go any further it is worth considering how the traditional Python garbage
collector determines if an object can be removed or not from memory.
To do this the Python runtime keeps a record of the number of references to an
object. If there is at least one (strong) reference to an object, then that object cannot
be garbage collected. However, if an object has no (strong) references to it then the
garbage collector can safely reclaim the memory used by the object.
In fact all Python objects include a reference count, which counts how many
things are referencing it. If an object is referenced by another object, then its counter
is incremented, if a variable references an object then the objects reference counter
will again be incremented. If a variable or another object dereferences the object then
the objects’ reference counter is decremented, etc.

© Springer Nature Switzerland AG 2023 23

J. Hunt, Advanced Guide to Python 3 Programming,
Undergraduate Topics in Computer Science,
https://doi.org/10.1007/978-3-031-40336-1_4
24 4 Weak References

By a reference we mean a variable or an attribute in another object is currently

referencing, pointing at, the object.
As an example, we will define a simple class Data that will hold some sort of
value:
class Data:
def __init__(self, value):
self.value = value

def __repr__(self):
return f'Data({self.value})'

We can create a simple program to create an instance of this class and store it into
a variable called data. Using the id of the object we can then find out how many
references there are to the data object:
import ctypes

print('Create data item')

data = Data(1)

print('Obtain the id of list object')

data_id = id(data)

print('Find the reference count of data')

ref_count = ctypes.c_long.from_address(data_id).value
print(f"Reference count for data is: {ref_count}")

The above code uses the ctypes.c_long.from_address value attribute

to find the number of (strong) references.
When we run this code we will see:
Create data item
Obtain the id of list object
Find the reference count of data
Reference count for data is: 1

That is, there is one reference to the Data object. This reference is currently held
in the variable data. If we now make another two variables reference the object
held in data we will increment the reference count:
print('Add some more references')
other_data1 = data
other_data2 = data

print('Find the reference count of data now')

ref_count = ctypes.c_long.from_address(data_id).value
print(f"Reference count for data is now: {ref_count}")

The output from this is:

Add some more references
Find the reference count of data now
Reference count for data is now: 3
4.4 When to Use Weak References 25

As you can see there are now 3 things referencing the Data object.
And finally if we reset the data variable to None and check the number of
references we will see that there are 2:
print('Reset data to None')
data = None
print('Find the reference count of data now')
ref_count = ctypes.c_long.from_address(data_id).value
print(f"Reference count for data is now: {ref_count}")

The output from this is:

Reset data to None
Find the reference count of data now
Reference count for data is now: 2

Notice that setting the data variable to None has no effect on the other_
data1 and other_data2 variables, all it does is reduce the number of references
to the Data instance from 3 down to 2.

4.3 Weak References

Unlike strong or normal references, a weak reference is a reference that does not
protect the object from being garbage collected.
A weak reference is not included in the main count for an objects’ reference.
Thus if an object has no (strong) references to it but only weak references, then if the
garbage collector needs to free up memory it can remove a weak reference object
from memory. The result is that the weak reference no long references an object and
any attempt to do so will return None.
That is to say that if the object is available in memory, calling the weak reference
returns it, otherwise if the object has been garbage collected, then None is returned.
In terms of terminology, we say that the object that is the target of a reference is
called the referent. Therefore, we can say that a weak reference does not prevent the
referent from being garbage collected.

4.4 When to Use Weak References

Why do we want weak references as a thing in the first place?

There are two main applications of weak references:
• Implement caches for large objects (weak dictionaries).
• Handling circular references.
To create weak references Python has provided us with a module named
weakref. We will explore the weakref module in the next section.
Random documents with unrelated
content Scribd suggests to you:
The Project Gutenberg eBook of In a Syrian
Saddle
 This ebook is for the use of anyone anywhere in the United
States and most other parts of the world at no cost and with
almost no restrictions whatsoever. You may copy it, give it away
or re-use it under the terms of the Project Gutenberg License
included with this ebook or online at www.gutenberg.org. If you
are not located in the United States, you will have to check the
laws of the country where you are located before using this
eBook.

Title: In a Syrian Saddle

Author: A. Goodrich-Freer

Release date: May 24, 2018 [eBook #57207]

Language: English

Credits: Produced by Chris Pinfield and the Online Distributed

Proofreading Team at http://www.pgdp.net (This file
was
produced from images generously made available by
The
Internet Archive/American Libraries.)

*** START OF THE PROJECT GUTENBERG EBOOK IN A SYRIAN

SADDLE ***
 Transcriber's Note.
Apparent typographical errors
have been corrected. The use
of hyphens has been
rationalised. Variations in the
use of accents have been
retained.
Two lines in blackletter font
have been bolded.
The musical score at the top of
page 97 is accompanied by a
link to a midi file.

IN A SYRIAN SADDLE
BY

A. GOODRICH-FREER

AUTHOR OF
"INNER JERUSALEM," "OUTER ISLES," ETC.

METHUEN & CO
36 ESSEX STREET W.C.
LONDON

First Published in 1905

 This record is dedicated

BY THE LADY
TO THE DOCTOR

On the eve of starting together upon a

longer journey
 CONTENTS

IN MOAB
CHAPTER PAGE
I. Going to Jericho 1
II. Stepping Eastward 20
III. Madaba 51
IV. Mshatta 64
V. Ammân 93
VI. Jerash, and the Fords of Jabbok 116
VII. Es-Salt 145
VIII. The Jordan Valley 161

IN GALILEE AND SAMARIA

I. To Nablus 178
II. To Samaria 194
III. To Taanak and Megiddo 217
IV. Haifa and Carmel 244
V. Nazareth and Tabor 258
VI. The Sea of Galilee 277
VII. Tiberias and Besan 302
VIII. West of the Jordan 323
INDEX 347
IN A SYRIAN SADDLE
 IN MOAB

CHAPTER I

GOING TO JERICHO
"A certain man went down from Jerusalem to Jericho"

L IFE is, in many respects, made very easy in the Holy Land. You
can return home in the afternoon with no anxious forebodings as
to how much waste of time is awaiting you in the shape of cards and
notes on the hall table; you may wear clothes for covering, you may
eat for nourishment; without taking thought for fashion in the one
case, or of competition with your neighbour's cook or gardener in
the other. But—according to our Occidental standards—you cannot
consistently indulge any taste you may happen to have for being
grand. Your attempts at a London, or shall we say a suburban,
drawing-room, your "At Home" days, your Europeanised service, the
dress of your womankind—distantly reminiscent of the ladies' papers
and of Answers to Correspondents—are certain to be complicated by
some contretemps provocative only of mirth. The Oriental himself
makes no attempt at being consistent. When you arrive at his house
he spreads a priceless carpet, but omits to remove last week's dust
from off the furniture; he gives you perfumed coffee, which is like a
dream of Olympus, and his servant brings you a piece of bread in his
fingers.
These reflections, and many more, were suggested during the
waiting which accompanied our start in the early sunrise at half-past
five on Saturday, 3rd October 1903. No one could have guessed how
grand we really were, and there were moments then, and later,
when the fact escaped even our own notice. We four, the Lady, the
Doctor (of various forms of scholarship) and the two Sportsmen,
were the chosen and proud companions of the Professor; and the
Professor, besides being the greatest epigraphist in Europe, was the
representative of a Royal Personage, and armed with all the permits
and safe-conducts and special privileges useful in a land of cholera,
quarantine, and backsheesh. Our eight horses were innocent of
grooming, and their equipment was fastened together mainly with
tin tacks, pieces of rope, and bits of string; but it would have been
difficult to find in England any animal to whom you could have
proposed, still less with whom you could have carried through, one
tithe of what our ragged regiment accomplished. Our two grooms,
mukaris, appealed to certain senses as vaguely horsey, though they
suggested nothing more distinguished than stable-helps; but their
management of eight animals, under conditions which seemed
especially designed for their destruction, when there was not a blade
of grass, perhaps for a whole day not a drop of water; when they
were ridden for ten, twelve, or even fourteen hours at a stretch with
merely an hour's rest—without forage—at noon, would have done
credit to any groom at Badminton or Berkeley. As we proposed to
ourselves both pleasure and profit we took no servants—still less a
dragoman. Our portable food had been very carefully selected, and
was the best obtainable. Bread, eggs, chickens, grapes, and lemons
we could count upon getting as we went along.
Each member of the party had clothing and a blanket in a pair of
saddle-bags—mostly of goats' hair or camels' hair, gaily decorated
with coloured tassels—and these, with an extra pair for the baskets
of food, spirit-lamps, plates, knives, and tin cups, were distributed
among the three baggage animals, who also carried, in turn, the two
mukaris, perched on the top of the pile, but capable of climbing up
and down with incredibly rapid agility.
At length the cavalcade was ready, and we turned our faces
towards Jericho. First came the Professor, on a tall, white Circassian
horse, with a tail which almost swept the ground, and was dyed with
henna for protection from the Evil One, who was further defied, by
each of us, by means of a large blue bead hanging round the neck
of every horse on a coloured worsted rope. The Professor himself
exhibited five foot of humanity, mostly brains; a personality which
consisted, to the eye, of a large scarlet and gold silk keffeeye (head
covering) with a goats' hair akal (rope to keep it in place) and an
elaborate silk fringe, below which emerged a pair of black leggings,
into one of which a whip was jauntily stuck. He was mounted on a
peaked, military saddle, and he alone of all the party refused to be
separated from his saddle-bags, which contained an assortment of
cigars, cigarettes, tobacco, and the long wooden pipe, for use in the
saddle, such as is in favour with the Bedu.
Next came the Lady, mounted on a long-legged Arab steed,
several sizes too large for her, but selected for her use mainly
because he could do the rahwân, the light canter special to the
desert horses, and which reduces fatigue to a minimum. It was
discovered, later in the day, that he was also capable, apparently, of
running for the Derby, an incident which may as well be recorded at
once, as it resulted in his banishment to the second class, and the
society of the mukaris.
The road from Jerusalem to Jericho still retains the character
recorded some two thousand years ago, but the thieves among
whom you inevitably fall are now licensed by the Government. There
is a whole village full of them, called Abu-dis, and they have the
privilege of protecting travellers from Bethany to Jericho—that is, of
enforcing payment for preventing anyone else from robbing you. It is
but some few years ago that an Englishman, suspected of seeking to
dispense with this advantage, had his donkey shot under him. At
Bethany, accordingly, we were joined by our escort, but, as became
our dignity, he was an officer, picturesquely attired, and mounted,
unfortunately, on a beautiful Arab mare. The misfortune lay in the
fact that all our horses, with one exception, were stallions, most of
whom became restless and uneasy, that of the Professor so
unmanageable that our escort was compelled to leave us, and to
take to bypaths from which he could, more or less, keep us in sight.
Nevertheless, even the temporary companionship had somewhat
excited the entire cavalcade. We were all in good spirits, and it must
be confessed that there was a certain amount of what may be called
"fooling"—-of what we would not for worlds describe as "showing
off," but, rather, as trying the paces of our steeds—an amusement
which the Professor saw reason, later, to forbid entirely.
The road to Jericho is a descent of over three thousand feet, but
at a point nearly half way, a long and steep climb brings you from
the transverse valley Sa'b-el-Meshak to the Khan of the Good
Samaritan. At this point it occurred to the Lady's horse to have a
private exhibition on his own account, and to set off at a truly
breakneck gallop, with which no other animal in the party could
possibly compete, even had it been wise to follow, except at a
considerable distance. Her strength was quite inadequate to check
him, but in the length and steepness of the hill lay promise of safety,
and it was with infinite relief that he was seen to pull up at last. He
had no vice, but the occasion was not one for a steeplechase, and it
was decided that, on the morrow, there should be a "general post"
of horses, the mukari being made responsible for his Derby winner,
and the Professor arranging, by exchange with one of the
Sportsmen, to ride an animal which would admit of conversation
with the officer, for such attainments as our leader's have not been
achieved by sitting in a library, or by confinement to the professorial
chair of his university, but rather by personal intercourse with the
Arabs in the various dialects of their own clans, by life in the desert,
and association with wandering tribes in the unexplored districts of
the Peræa Haurân and of Central Arabia.
The Sportsmen carried guns, the Doctor a notebook—though he
was more than suspected of yearning for a rifle,—the revolver which
he carried at his belt being better adapted for the murder of man
than of beast—not that the murder of man, to judge from the
experiences of earlier travellers, was a wholly improbable
contingency. Our road led us along almost the entire length of the
north and east wall of Jerusalem; we then crossed the bridge over
the Kedron valley—the brook, if any exist, is now far below the
surface; we passed the Garden of Gethsemane, skirted the southern
slope of the Mount of Olives, hastened past the filthy hovels of the
little village of Bethany, crowned by the so-called Castle of Lazarus,
probably the remains of a pre-crusading Benedictine convent, and
finally, about seven o'clock, pulled up at what is known as the Inn of
the Apostles' Fountain, just such a building as a child might draw
upon a slate. As this is the only well between Bethany and Jericho it
may be safely assumed that the apostles, coming up to Jerusalem,
would drink here, though it is to be hoped that it was less
contaminated than at present; for even the careless natives strain
the water through a sieve before allowing their animals to drink,
though, nevertheless, they still acquire leeches, as the bleeding
mouths of the camels and donkeys one meets along the road
frequently betray. The spot has been marked by a succession of
buildings; a little white dome over the well, and some hewn stones
and the ruins of an aqueduct in the hill across the road, being all
that now remains of its old dignity.
Passing the Khan of the Good Samaritan—a modern inn and
curiosity shop, at which you can, at your leisure, renew "a certain
man's" experiences—we paused at the top of the last hill before
descending towards the Jordan valley. Here the entire
neighbourhood was once commanded by a strong mediæval castle,
intended, like many all over the country, for the defence of the
district. The tribal marks of the Bedu to be found on its walls are of
extreme interest. The hill upon which it stands is known as Tel'at ed-
Dam, the hill of blood, probably from the red colour of the rock,
though some have sought to identify it, by reason of the sound of
the name, with the Adummim of Joshua xv. 7.
The view from this point is, in certain details, absolutely unique.
You look down at the lowest spot upon the earth's surface—the
hollow of the Dead Sea, blue as the sky in the morning sunshine,
flecked with cloudlike wavelets, beautiful, gay and smiling, but bitter,
treacherous, and the home only of mystery and death. The water
contains about twenty-five per cent. of solid substances; no
organism higher than such microbes as the bacilli of tetanus can live
in it; even swimming is almost impossible; neither shells nor coral
testify to any happier past. The water boils at 221 degrees
Fahrenheit, but the presence of chloride of magnesium makes it
incredibly nauseous, while the oily quality, which it derives from
chloride of calcium, makes any accidental splash upon the garments
very destructive. We gratefully take in long breaths of air which, hot
and dry as it is, are, we are well aware, more fresh and sweet than
any we are likely to obtain during the next twenty-four hours, for
only personal experience of the stifling heat of that unrivalled hollow
can make it possible to realise that six and a half million tons of
water which fall into the Dead Sea—a basin about the size of the
Lake of Geneva, but with no outlet—have to be daily evaporated. Far
away southward is the great salt district, where the salt deposit,
coated with chalky limestone and clay, takes many weird forms,
among which the Arabs point out Bint Shech Luth—the woman of
Shech Lot.
"Of whose wickedness, even to this day, the waste land that
smoketh is a testimony, and plants bearing fruit that never come to
ripeness, and a standing pillar of salt is a monument of an
unbelieving soul" (Wisdom x. 7).
"The waste land that smoketh" is a touch of autopic description
which one remembers when, towards sunset, great wreaths of white
mist lie low in the mountain hollows, as nowhere else in Judæa.
Eastward, the horizon is bounded by the long chain of the mountains
of Moab, which, ever since our arrival in the country, have seemed a
sort of mysterious dreamland, a limit of knowledge, the gate of
fairyland, the nightly stage of the great pageant gilded and painted
by the sunset. How often have we longed, like the youngest brother
of the fairy tale, to ride across the wide plain, and to wander forth
for a year and a day into that dim Unknown! We could hardly realise
that at last the time had come and we were stepping eastward.
Below us, in the great plain, a meandering green track shows
where the banks of the Jordan are offering shade and refreshment.
In the nearer foreground the scanty hovels and many hotels of
modern Jericho lie, embowered in tropical vegetation, and we
remember with an added interest that, within even the next day or
two, we shall pass through districts of three distinct flora and fauna,
and, leaving behind the oaks and pines, the familiar sparrows and
starlings of our Jerusalem environment, we shall rest to-night among
palms and bananas, we shall hear the cry of the jackal, and smell
the tracks of the hyæna, and again, in a couple of days, find
ourselves in desert surroundings, the nursery of the camel and home
of the gazelle, with the scanty herbage and stalkless flora of an
Alpine world.
Looking down upon the Jericho plain we note various points of
interest. We distinguish the sites of three Jerichos.
First we notice, to the south, the kraal-like village of to-day, on the
site of a castle and church of the Crusaders, afterwards a flourishing
Moslem town, plundered by Egyptian soldiers in 1840, and
subsequently destroyed by fire in 1871. It seems unlikely that it will
ever recover its former position; for, apart from apparent absence of
all ability for initiation on the part of the Arabs, the climate seems to
have a degenerating effect upon the inhabitants, Even German
perseverance, which has made habitable spots in the low maritime
plains of Judæa where all other colonists had failed, could not suffice
to render life here endurable, and an agricultural settlement
organised within the last few years has literally died out. The Latins
—or, as we say in England, the Roman Catholics—have also failed to
establish themselves, and the Russian and Greek settlements find
existence possible only under conditions of frequent change, and the
stimulus of the profits to be made out of the thousands of Russian
pilgrims who come, every year, at Epiphany, for baptism in the
Jordan. Last year, however, the Jordan was held to have acquired so
large a proportion of cholera bacilli on its way from Tiberias, where
there was a great outbreak of disease, that approach was justly
forbidden by the authorities. To the west lie the remains of the
Jericho of Bible history, of which, from earliest childhood, most of us
have had a mental picture—the great town enclosed by walls
enwreathed with vegetation, "that ancient city of palm-trees," few of
which still remain, though they were abundant as late as the
seventh century; and in Jewish amulets, and marriage or divorce
documents, which are commonly decorated with allegorical pictures,
Jericho is still represented by a group of palms. South of the
Israelitish town, and west of modern Jericho, are the remains of the
Roman Jericho, which, it is interesting to remember, was presented
by Anthony to Cleopatra, who, characteristically, promptly sold it to
Herod for a winter home. He made of it a beautiful city, adorned
with palms and gardens, and scented with the balsams for which it
was long famous as an article of commerce, but which are no longer
to be found in Syria, and where, in the time of Christ, the roadsides
were shaded with sycamores—not the pseudo-platanus with which
we are familiar, and which is not a sycomore at all, but the ficus
sycomorus, the mulberry fig, which, often attaining the proportions
of a handsome forest tree, still yields its wholesome and refreshing
fruit among the humbler surroundings of to-day. The remains of a
pool, five hundred and sixty-four feet long, part of an immense
system of conduits still visible, which was the immediate cause of
the fertility and beauty of the Roman Jericho, is said to indicate the
whereabouts of Herod's palace.
The Jericho of crusading times was, probably, supplied with water
from what is now locally known as the Ain es-Sultan (the Sultan's
Spring), although its more suggestive name of Elijah's Fountain is
still in use among the Christian population. Pilgrims of the fourth and
fifth centuries record the tradition that it was here that the prophet
healed the bitter water with salt. Salt is still thrown into a pool or
cistern which, toward the end of the dry season, is found to be
impregnated with noxious matter, animal or vegetable. As, before
the time of the Roman water system, there was no other means of
supply it is almost certain that the ancient town must have stood
near this, the only natural spring, and the site of the house of
Rahab, still shown, may quite well have been in the neighbourhood
indicated.
Rising almost perpendicularly a short distance beyond the
Fountain of Elijah, is the Quarantana Mountain, first so called by the
Crusaders, in memory of the forty days of the Temptation, although
it seems to have been held sacred from a much earlier period, as
there are remains of many hermitages, one of which is said to have
been occupied by S. Chariton about 400 A.D.
It is a panorama wonderful not only in extent but in the amount of
detail, which, in the cloudless air of the East, we are enabled to
distinguish, and we would willingly pause longer, but the sun is high
in the heavens, we have been six hours in the saddle, and, leaving
our horses to follow, we find a pleasant relief from the glare in
descending an almost perpendicular path into the Wady Kelt, the
deep gorge of the brook Cherith, where a monastery marks the site
of the alleged hiding-place of the prophet Elijah. It is perched on a
narrow shelf, high up on the perpendicular rock wall of the ravine,
and can only have acquired its present resemblance to domestic
architecture by slow and painful labour. The lower storey, of rough
massive stones, apparently designed for a fortress, is all that
remains of an ancient monastery, founded in 535, possibly upon the
site of an earlier habitation of the Essenes, an esoteric sect of Jews,
whose life somewhat resembled that of the religious Orders among
Christians. The cave, high up in the face of the rock, alleged to have
been occupied by Elijah, is now an oratory for the Greek monks,
who, in 1880, returned to an old foundation of Koziba, and built an
upper storey, with projecting balconies, from which one has a
wonderful view of the gorge below.
We left our horses upon a little bridge, which spans the bed of the
brook—where they found welcome shelter, after their giddy descent,
under a vine-covered pergola—and then, following a zigzag path, we
made our way within the doors of one of the many hospitable
monasteries which, all over the Holy Land, are ready to offer at least
shade and water, the two great boons of a hot country, to the weary
and thirsty traveller. No question was made as to creed, even as to
that of our officer, a Moslem, and we were allowed to spread the
meal we had with us, with kindly additions of water for drink and
ablution, coffee, liqueur, and fresh green lemons.
Ignoring all question as to whether the prophet were fed by Arabs
or by orabs (ravens) it is at least a pleasing sight to watch the
relations of the wild birds of the gorge with his modern
representatives. The old superior of the convent, silent, calm, with
an expression of infinite resignation to the poverty, in every sense,
of his ascetic life, seemed to recover some faint and passing interest
in the beautiful world about him as, bidding us be silent within the
window, he stepped out on to the balcony, and produced from his
pocket some dried figs. Scarcely raising his voice, he called gently,
Idoo sudar! Idoo sudar!—or such his cry sounded—Russian, as we
understood, for "Come along, sir!" The blue air was flecked with
gold, a morsel of the fruit was seized as it was thrown into the air,
there was another flash of golden wings, and on the head,
shoulders, and the extended arms of the old man there perched the
exquisite blackbirds of the district—the "Tristram's grakle" of the
Dead Sea. The sheen of the deep purple wing, with its orange lining,
was wonderfully rich, and the creatures themselves were, in every
movement, graceful as swallows. The dainties finished, there was an
instant flutter, and not a sign remained in all the clear, blue heaven
of our visitors of a moment ago; only a shimmer of silver on the
opposite cliff showed where a cloud of rock pigeons had descended
to inquire into the cause of excitement among their neighbours.
After a couple of hours' rest we went on our way, following the
narrow path which crept along the precipice, and looking with equal
wonderment at the rocky hermitages above our heads and at those
beneath our feet; some which seemed to be accessible only to birds,
while others were so deep down in the narrow gorge that the
necessaries of life have to be lowered to them from a roughly-
formed crane upon a narrow shelf of level ground above.
It was interesting to notice that, even among men of similar
religious impulses, and identical occupations and opportunities,
individual character nevertheless finds occasion for expression. While
some dwelt in holes in the rock, accessible only by a ladder,
sometimes of rope, and in one case, by a voluntary asceticism, only
by a pole, others showed a tendency to make the best of the
situation—two or three had constructed gardens, verandahs, or
porticos; one dwelling at least would have been described by an
auctioneer as a cottage orné, and some had even shown an
æsthetic realisation of what was befitting the situation, and had
sought after effects of colour and form as well as of convenience.
Not a human being was to be seen, but we wondered how many
pairs of eyes were watching our movements; whether it were
possible that the sound of our cheery voices, and the sight of our
enjoyment, may not have touched some heart to sense of loss, have
sounded some chord of regret, or even of remorse, have recalled
memories of other days, when friendship and anticipation and
sympathy and glad companionship were theirs, and life was other
than the awaiting of death, and the setting sun brought a sense of
something added to the days that were gone as well as of
something subtracted from such as might—in God's providence—
remain.
Our horses followed slowly down the glen, and the afterglow was
beautifying even the desolate village of Jericho as we finally
remounted and rode in among the groves of orange and banana.

CHAPTER II

STEPPING EASTWARD
"The dewy ground was dark and cold;
Behind all gloomy to behold;
And stepping eastward seemed to be
A kind of heavenly destiny."
W. Wordsworth

T HE Jericho hotels were closed for the season, but with the
connivance of the negro caretaker and of an Arab in charge of
the adjoining orange-gardens we obtained entrance at one, and
managed to provide ourselves with firing and an excellent supper,
and, subsequently, with beds. The Lady, who alone of the party
carried a watch, heard the negro awakening the Professor next
morning with the information that it was three o'clock, and added
greatly to her popularity by being in a position to call out an
assurance that it was only one, and that two hours' further repose
was permissible. The building, it should be mentioned, being
constructed mainly of wood and of mud bricks was well adapted for
distant conversation.
Three o'clock, however, duly arrived, all too soon, and by four
o'clock we had breakfasted and were on our way across the sandy
plain which stretches for about two hours between Jericho and the
Jordan. A few faint streaks in the east promised the coming day, but
it was still so dark that our horses required all our attention, as the
plain is full of holes; twice over a silver gleam ahead warned us of
fords to be crossed, and from time to time dark masses rose up
before us, and those riding in advance called to the others to avoid
the spreading branches of the jujube-tree, zizyphus lotus and
zizyphus spina Christi, called by the Arabs nebk and sidr, which are
the octopods of vegetable life, sending out long tentacles armed
with fierce thorns, capable of subtracting your head-gear, entering
your saddle, and imprisoning your horse.
The ride across the plain of Jordan, interesting at any time to
persons of imagination, was unspeakably weird and suggestive in
the morning twilight, but we differed as to whether the world in
which we found ourselves was one in course of construction or of
disintegration. Some of us were of opinion that the giant sand-hills—
a labyrinth of marl and salt deposit, worn by winds and washed by
winter torrents, an old sea bottom—which crumble at a touch, and
which resemble castles, churches, towers, domes, minarets, whole
towns of every variety of architecture, suggested an artist's dream of
a world to be; while others maintained that they were the images, in
the mind of a philosopher, dwelling upon the past. There was no
limit to the tricks which fancy might play in such surroundings, a
nearer fata morgana; a dream materialised as it created itself; a
poem precipitated as it was sung: castles in Spain in which each of
us saw some reminder of his personal aspirations: the land of By-
and-by; the ruins of Yesterday; the house of Never, according to our
individual temperament and faculty.
Welcome to our website – the perfect destination for book lovers and
knowledge seekers. We believe that every book holds a new world,
offering opportunities for learning, discovery, and personal growth.
That’s why we are dedicated to bringing you a diverse collection of
books, ranging from classic literature and specialized publications to
self-development guides and children's books.

More than just a book-buying platform, we strive to be a bridge

connecting you with timeless cultural and intellectual values. With an
elegant, user-friendly interface and a smart search system, you can
quickly find the books that best suit your interests. Additionally,
our special promotions and home delivery services help you save time
and fully enjoy the joy of reading.

Join us on a journey of knowledge exploration, passion nurturing, and

personal growth every day!

ebookbell.com