Read Anytime Anywhere Easy Ebook Downloads at ebookmeta.

com

Advanced Guide to Python 3 Programming, 2nd John

Hunt

https://ebookmeta.com/product/advanced-guide-to-
python-3-programming-2nd-john-hunt/

OR CLICK HERE

DOWLOAD EBOOK

Visit and Get More Ebook Downloads Instantly at https://ebookmeta.com

 Recommended digital products (PDF, EPUB, MOBI) that
you can download immediately if you are interested.

A Beginners Guide to Python 3 Programming 2nd Edition John

Hunt

https://ebookmeta.com/product/a-beginners-guide-to-
python-3-programming-2nd-edition-john-hunt/

ebookmeta.com

Beginner s Guide to Kotlin Programming John Hunt

https://ebookmeta.com/product/beginner-s-guide-to-kotlin-programming-
john-hunt/

ebookmeta.com

Learn to Program with Python 3: A Step-by-Step Guide to

Programming - 2nd ed 2nd Edition Irv Kalb

https://ebookmeta.com/product/learn-to-program-with-python-3-a-step-
by-step-guide-to-programming-2nd-ed-2nd-edition-irv-kalb/

ebookmeta.com

Social Capital Economic Growth and Well Being 1st Edition

Francesco Sarracino

https://ebookmeta.com/product/social-capital-economic-growth-and-well-
being-1st-edition-francesco-sarracino/

ebookmeta.com
My Government Means To Kill Me 1st Edition Rasheed Newson

https://ebookmeta.com/product/my-government-means-to-kill-me-1st-
edition-rasheed-newson/

ebookmeta.com

Traders and Tinkers Bazaars in the Global Economy 1st

Edition Maitrayee Deka

https://ebookmeta.com/product/traders-and-tinkers-bazaars-in-the-
global-economy-1st-edition-maitrayee-deka/

ebookmeta.com

Texas Law for the Social Worker A 2016 Sourcebook 4th

Edition Vicki Hansen

https://ebookmeta.com/product/texas-law-for-the-social-
worker-a-2016-sourcebook-4th-edition-vicki-hansen/

ebookmeta.com

Evil Roman Emperors The Shocking History of Ancient Rome s

Most Wicked Rulers from Caligula to Nero and More 1st
Edition Phillip Barlag
https://ebookmeta.com/product/evil-roman-emperors-the-shocking-
history-of-ancient-rome-s-most-wicked-rulers-from-caligula-to-nero-
and-more-1st-edition-phillip-barlag/
ebookmeta.com

Sex Lies Techy Guys Love on the Sunshine Coast Collection

1st Edition Campbell Barbra

https://ebookmeta.com/product/sex-lies-techy-guys-love-on-the-
sunshine-coast-collection-1st-edition-campbell-barbra/

ebookmeta.com
Like I Needed (Heather Bay 3) 1st Edition Charlie Novak

https://ebookmeta.com/product/like-i-needed-heather-bay-3-1st-edition-
charlie-novak/

ebookmeta.com
Undergraduate Topics in Computer Science

John Hunt

Advanced Guide
to Python 3
Programming
Second Edition
Undergraduate Topics in Computer Science

Series Editor
Ian Mackie, University of Sussex, Brighton, UK

Advisory Editors
Samson Abramsky , Department of Computer Science, University of Oxford,
Oxford, UK
Chris Hankin , Department of Computing, Imperial College London, London,
UK
Mike Hinchey , Lero – The Irish Software Research Centre, University of
Limerick, Limerick, Ireland
Dexter C. Kozen, Department of Computer Science, Cornell University, Ithaca,
NY, USA
Andrew Pitts , Department of Computer Science and Technology, University of
Cambridge, Cambridge, UK
Hanne Riis Nielson , Department of Applied Mathematics and Computer
Science, Technical University of Denmark, Kongens Lyngby, Denmark
Steven S. Skiena, Department of Computer Science, Stony Brook University,
Stony Brook, NY, USA
Iain Stewart , Department of Computer Science, Durham University, Durham,
UK
Joseph Migga Kizza, College of Engineering and Computer Science, The
University of Tennessee-Chattanooga, Chattanooga, TN, USA
‘Undergraduate Topics in Computer Science’ (UTiCS) delivers high-quality instruc-
tional content for undergraduates studying in all areas of computing and information
science. From core foundational and theoretical material to final-year topics and
applications, UTiCS books take a fresh, concise, and modern approach and are ideal
for self-study or for a one- or two-semester course. The texts are all authored by
established experts in their fields, reviewed by an international advisory board, and
contain numerous examples and problems, many of which include fully worked
solutions.
The UTiCS concept relies on high-quality, concise books in softback format, and
generally a maximum of 275–300 pages. For undergraduate textbooks that are likely
to be longer, more expository, Springer continues to offer the highly regarded Texts
in Computer Science series, to which we refer potential authors.
John Hunt

Advanced Guide to Python 3

Programming
Second Edition
John Hunt
Midmarsh Technology Ltd.
Chippenham, Wiltshire, UK

ISSN 1863-7310 ISSN 2197-1781 (electronic)

Undergraduate Topics in Computer Science
ISBN 978-3-031-40335-4 ISBN 978-3-031-40336-1 (eBook)
https://doi.org/10.1007/978-3-031-40336-1

© Springer Nature Switzerland AG 2019, 2023

This work is subject to copyright. All rights are reserved by the Publisher, whether the whole or part of
the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation,
broadcasting, reproduction on microfilms or in any other physical way, and transmission or information
storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology
now known or hereafter developed.
The use of general descriptive names, registered names, trademarks, service marks, etc. in this publication
does not imply, even in the absence of a specific statement, that such names are exempt from the relevant
protective laws and regulations and therefore free for general use.
The publisher, the authors, and the editors are safe to assume that the advice and information in this book
are believed to be true and accurate at the date of publication. Neither the publisher nor the authors or
the editors give a warranty, expressed or implied, with respect to the material contained herein or for any
errors or omissions that may have been made. The publisher remains neutral with regard to jurisdictional
claims in published maps and institutional affiliations.

This Springer imprint is published by the registered company Springer Nature Switzerland AG
The registered company address is: Gewerbestrasse 11, 6330 Cham, Switzerland
For Denise, my wife, my soulmate, my best
friend.
Preface to the Second Edition

This second edition represents a significant expansion of the material in the first
edition, as well as an update of that book from Python 3.7 to 3.12.
This book includes whole new sections on advanced language features, Reactive
Programming in Python and data analysts. New chapters on working with Tkinter,
on event handling with Tkinter and a simple drawing application using Tkinter have
been added. A new chapter on performance monitoring and profiling has also been
added. A chapter on pip and conda is included at the end of the book.
In all there are 18 completely new chapters that take you far further on your Python
journey. Enjoy the book and I hope you find it useful.

Chippenham, UK John Hunt

vii
Preface to the First Edition

Some of the key aspects of this book are

1. It assumes knowledge of Python 3 and of concepts such as functions, classes,
protocols, abstract base classes, decorators, iterables and collection types (such
as list and tuple).
2. However, the book assumes very little knowledge or experience of the topics
presented.
3. The book is divided into eleven topic areas: advanced language features,
Computer Graphics, games, testing, file input/output, database access, logging,
concurrency and parallelism, Reactive Programming, network programming and
data analytics.
4. Each topic in the book has an introductory chapter followed by chapters that
delve into that topic.
5. The book includes exercises at the end of most chapters.
6. All code examples (and exercise solutions) are provided on line in a GitHub
repository.

What You Need

You can of course just read this book; however following the examples in this book
will ensure that you get as much as possible out of the content. For this you will need
a computer.
Python is a cross-platform programming language, and as such you can use Python
on a Windows PC, a Linux box, an Apple Mac, etc. So you are not tied to a particular
type of operating system; you can use whatever you have available.
However you will need to install some software on that computer. At a minimum
you will need Python. The focus of this book is Python 3 so that is the version that
is assumed for all examples and exercises. As Python is available for a wide range

ix
x Preface to the First Edition

of platforms from Windows, to Mac OS and Linux, you will need to ensure that you
download the version for your operating system.
Python can be downloaded from the main Python website which can be found at
http://www.python.org/.

You will also need some form of editor to write your programs. There are numerous
generic programming editors available for different operating systems with VIM on
Linux, Notepad++ on Windows and Sublime Text on windows and Macs being
popular choices.
However, using an Integrated Development Environment (IDE) editor such as
PyCharm, Visual Studio Code or Spyder can make writing and running your programs
much easier.
However, this book does not assume any particular editor, IDE or environment
(other than Python 3 itself).

Conventions

Throughout this book you will find a number of conventions used for text styles.
These text styles distinguish between different kinds of information. Code words,
variable and Python values, used within the main body of the text, are shown using
a Courier font. A block of Python code is set out as shown here:
Preface to the First Edition xi

def draw_koch(size, depth):

if depth > 0:
for angle in ANGLES:
draw_koch(size / 3, depth - 1)
turtle.left(angle)
else:
turtle.forward(size)
# Draw three sides of snowflake
for_in range(3):
draw_koch(SIZE_OF_SNOWFLAKE, depth)
turtle.right(120)

Note that keywords and points of interest are shown in bold font.
Any command line or user input is shown in standard font as shown below, for
example:
Hello, world
Enter your name: John
Hello John

Example Code and Sample Solutions

The examples used in this book (along with sample solutions for the exercises at the
end of most chapters) are available in a GitHub repository. GitHub provides a web
interface to Git, as well as a server environment hosting Git.
Git is a version control system typically used to manage source code files (such
as those used to create systems in programming languages such as Python but also
Java, C#, C++ and Scala). Systems such as Git are very useful for collaborative
development as they allow multiple people to work on an implementation and to
merge their work together. They also provide a useful historical view of the code
(which also allows developers to roll back changes if modifications prove to be
unsuitable).
The GitHub repository for this book can be found at:
• https://github.com/johnehunt/advancedpython3_2nd
If you already have Git installed on your computer, then you can clone (obtain a
copy of) the repository locally using:
git clone https://github.com/johnehunt/advancedpyth
on3_2nd.git
If you do not have Git, then you can obtain a zip file of the examples using
https://github.com/johnehunt/advancedpython3_2nd/arc
hive/refs/heads/main.zip
xii Preface to the First Edition

You can of course install Git yourself if you wish. To do this, see https://git-scm.
com/downloads. Versions of the Git client for Mac OS, Windows and Linux/Unix
are available here.
However, many IDEs such as PyCharm come with Git support and so offer another
approach to obtaining a Git repository.
For more information on Git see http://git-scm.com/doc. This Git guide
provides a very good primer and is highly recommended.

Acknowledgement I would like to thank Phoebe Hunt for creating the pixel images used for the
Starship Meteors game in Chap. 22.
Contents

1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.2 Useful Python Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

Part I Advanced Language Features

2 Python Type Hints . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
2.2 Pythons Type System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
2.3 The Challenge for Python Developers . . . . . . . . . . . . . . . . . . . . . . 8
2.4 Static Typing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
2.5 Python Type Hints . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
2.6 Type Hint Layout . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
2.7 Type Hints for Multiple Types . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
2.8 The Self Type . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
2.9 The Benefits of Type Hints . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
2.10 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
2.11 Online Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
3 Class Slots . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
3.2 The Problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
3.3 Slots to the Rescue . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
3.4 Performance Benefits . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
3.5 Why Not Use Slots? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
3.6 Online Resource . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
4 Weak References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
4.2 How Garbage Collection Works: Reference Counting . . . . . . . . 23
4.3 Weak References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
4.4 When to Use Weak References . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25

xiii
xiv Contents

4.5 The Weakref Module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26

4.6 Creating Weak References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
4.7 Retrieving Objects from Weak References . . . . . . . . . . . . . . . . . . 28
4.8 The WeakValueDicitonary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
4.9 WeakKeyDictionary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
4.10 Proxy Objects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
4.11 Online Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
5 Data Classes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
5.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
5.2 A Traditional Class . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
5.3 Defining Data Classes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
5.4 Defining Additional Behaviour . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
5.5 The Dataclass Decorator . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
5.6 Custom Factory for Fields . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
5.7 Immutable Dataclasses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
5.8 Data Classes and Inheritance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
5.9 Post Initialisation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
5.10 Initialisation Variables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
5.11 Positional Attributes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
5.12 Exercise . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
6 Structural Pattern Matching . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
6.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
6.2 The Match Statement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
6.3 Matching Classes with Positional Attributes . . . . . . . . . . . . . . . . . 50
6.4 Matching Against Standard Classes . . . . . . . . . . . . . . . . . . . . . . . . 51
6.5 Online Resource . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
7 Working with pprint . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
7.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
7.2 The pprint Data Printer Module . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
7.3 Basic pprint Usage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
7.4 Changing the Width . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
7.5 Changing the Depth . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
7.6 Managing the Indentation Level . . . . . . . . . . . . . . . . . . . . . . . . . . . 58
7.7 Reducing Line Breaks Using Compact . . . . . . . . . . . . . . . . . . . . . 59
7.8 The pformat Function . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
7.9 The saferepr() Function . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
7.10 Using the PrettyPrinter Class . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
7.11 Online Resource . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63
8 Shallow v Deep Copy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65
8.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65
8.2 Copying a List of Lists . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65
8.3 The Problem with Copying . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66
Contents xv

8.4 The Copy Module to the Rescue . . . . . . . . . . . . . . . . . . . . . . . . . . . 68

8.5 Using the deepcopy() Function . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69
8.6 Online Resource . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70
9 The __init__ Versus __new__ and __call . . . . . . . . . . . . . . . . . . . . . . . . . 71
9.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71
9.2 The __new__ and __init__ Methods . . . . . . . . . . . . . . . . . . . . . . . 71
9.3 The __new__ Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72
9.4 When to Use the __new__ Method . . . . . . . . . . . . . . . . . . . . . . . . . 73
9.5 Using __new__ to Create a Singleton Object . . . . . . . . . . . . . . . . 74
9.6 The __init__ Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75
9.7 Can __new__ and __init__ Be Used Together? . . . . . . . . . . . . . . 76
9.8 The __call__ Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77
9.9 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77
10 Python Metaclasses and Meta Programming . . . . . . . . . . . . . . . . . . . . . 79
10.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79
10.2 Metaprogramming . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79
10.3 Decorators as a Form of Metaprogramming . . . . . . . . . . . . . . . . . 81
10.4 Metaclasses for Metaprogramming . . . . . . . . . . . . . . . . . . . . . . . . 83
10.4.1 Singleton Metaclass . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83
10.5 Exec and Eval for Metaprogramming . . . . . . . . . . . . . . . . . . . . . . 85
10.5.1 The exec() Function . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85
10.5.2 The eval() Function . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86
10.5.3 eval Versus exec() . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86

Part II Computer Graphics and GUIs

11 Introduction to Computer Graphics . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91
11.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91
11.2 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92
11.3 The Graphical Computer Era . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92
11.4 Interactive and Non Interactive Graphics . . . . . . . . . . . . . . . . . . . . 93
11.5 Pixels . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94
11.6 Bit Map Versus Vector Graphics . . . . . . . . . . . . . . . . . . . . . . . . . . . 96
11.7 Buffering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96
11.8 Python and Computer Graphics . . . . . . . . . . . . . . . . . . . . . . . . . . . 96
11.9 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97
11.10 Online Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97
12 Python Turtle Graphics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99
12.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99
12.2 The Turtle Graphics Library . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99
12.2.1 The Turtle Module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99
12.2.2 Basic Turtle Graphics . . . . . . . . . . . . . . . . . . . . . . . . . . . 100
12.2.3 Drawing Shapes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103
12.2.4 Filling Shapes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104
xvi Contents

12.3 Other Graphics Libraries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105

12.4 3D Graphics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106
12.4.1 PyOpenGL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106
12.5 Online Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107
12.6 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107
13 Computer Generated Art . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109
13.1 Creating Computer Art . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109
13.2 A Computer Art Generator . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111
13.3 Fractals in Python . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114
13.4 The Koch Snowflake . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114
13.5 Mandelbrot Set . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116
13.6 Online Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119
13.7 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119
14 Introduction to Matplotlib . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121
14.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121
14.2 Matplotlib . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122
14.3 Plot Components . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123
14.4 Matplotlib Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125
14.4.1 Backend Layer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125
14.4.2 The Artist Layer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126
14.4.3 The Scripting Layer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127
14.5 Online Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128
15 Graphing with Matplotlib Pyplot . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129
15.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129
15.2 The pyplot API . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129
15.3 Line Graphs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 130
15.3.1 Coded Format Strings . . . . . . . . . . . . . . . . . . . . . . . . . . . 132
15.4 Scatter Graph . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133
15.4.1 When to Use Scatter Graphs . . . . . . . . . . . . . . . . . . . . . 134
15.5 Pie Charts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 136
15.5.1 Expanding Segments . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137
15.5.2 When to Use Pie Charts . . . . . . . . . . . . . . . . . . . . . . . . . 138
15.6 Bar Charts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139
15.6.1 Horizontal Bar Charts . . . . . . . . . . . . . . . . . . . . . . . . . . . 140
15.6.2 Coloured Bars . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141
15.6.3 Stacked Bar Charts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141
15.6.4 Grouped Bar Charts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143
15.7 Figures and Subplots . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144
15.8 3D Graphs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 146
15.9 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 148
Contents xvii

16 Graphical User Interfaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 151

16.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 151
16.2 GUIs and WIMPS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 151
16.3 Windowing Frameworks for Python . . . . . . . . . . . . . . . . . . . . . . . . 153
16.3.1 Platform-Independent GUI Libraries . . . . . . . . . . . . . . 154
16.3.2 Platform-Specific GUI Libraries . . . . . . . . . . . . . . . . . . 154
16.4 Online Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 154
17 Tkinter GUI Library . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155
17.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155
17.2 Tkinter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155
17.3 Windows as Objects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 156
17.4 Key Concepts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 156
17.4.1 The Tk Class . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157
17.4.2 TK Widgets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157
17.4.3 The TopLevel Class . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 158
17.4.4 The Frame Class . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159
17.4.5 Dialogs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 160
17.4.6 The Canvas Class . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 161
17.5 The Class Inheritance Hierarchy . . . . . . . . . . . . . . . . . . . . . . . . . . . 162
17.5.1 Layout Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163
17.6 A Simple Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163
17.7 Tkinter Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164
17.7.1 Mac Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164
17.7.2 Windows Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165
17.8 GUI Builders for Tkinter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 166
17.9 Online Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 166
17.10 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 167
18 Events in Tkinter User Interfaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 169
18.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 169
18.2 Event Handling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 169
18.3 What is Event Handling? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 170
18.4 What Are Event Handlers? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 170
18.5 Event Binders . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 172
18.6 Virtual Events . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 173
18.7 Event Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 173
18.8 What Types of Event Are There? . . . . . . . . . . . . . . . . . . . . . . . . . . 174
18.9 Binding an Event to an Event Handler . . . . . . . . . . . . . . . . . . . . . . 175
18.10 Implementing Event Handling . . . . . . . . . . . . . . . . . . . . . . . . . . . . 175
18.11 An Interactive GUI Application . . . . . . . . . . . . . . . . . . . . . . . . . . . 178
18.12 Online Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 181
18.13 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 181
xviii Contents

19 PyDraw Tkinter Example Application . . . . . . . . . . . . . . . . . . . . . . . . . . 185

19.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 185
19.2 The PyDraw Application . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 185
19.3 The Structure of the Application . . . . . . . . . . . . . . . . . . . . . . . . . . . 186
19.3.1 Model, View and Controller Architecture . . . . . . . . . . 188
19.3.2 PyDraw MVC Architecture . . . . . . . . . . . . . . . . . . . . . . 189
19.3.3 Additional Classes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 190
19.3.4 Object Relationships . . . . . . . . . . . . . . . . . . . . . . . . . . . . 190
19.4 The Interactions Between Objects . . . . . . . . . . . . . . . . . . . . . . . . . 191
19.4.1 The PyDrawApp . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 192
19.5 The PyDrawView Constructor . . . . . . . . . . . . . . . . . . . . . . . . . . . . 192
19.5.1 Changing the Application Mode . . . . . . . . . . . . . . . . . . 193
19.5.2 Adding a Graphic Object . . . . . . . . . . . . . . . . . . . . . . . . 193
19.6 The Classes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 194
19.6.1 The PyDrawConstants Class . . . . . . . . . . . . . . . . . . . . . 194
19.6.2 The PyDrawView Class . . . . . . . . . . . . . . . . . . . . . . . . . 195
19.6.3 The PyDrawMenuBar Class . . . . . . . . . . . . . . . . . . . . . . 196
19.6.4 The PyDrawController Class . . . . . . . . . . . . . . . . . . . . . 196
19.6.5 The DrawingModel Class . . . . . . . . . . . . . . . . . . . . . . . . 197
19.6.6 The DrawingView Class . . . . . . . . . . . . . . . . . . . . . . . . . 198
19.6.7 The DrawingController Class . . . . . . . . . . . . . . . . . . . . . 198
19.6.8 The Figure Class . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 199
19.6.9 The Square Class . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 200
19.6.10 The Circle Class . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 200
19.6.11 The Line Class . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 200
19.6.12 The Text Class . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 201
19.7 Reference . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 201
19.8 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 201

Part III Computer Games

20 Introduction to Games Programming . . . . . . . . . . . . . . . . . . . . . . . . . . . 205
20.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 205
20.2 Games Frameworks and Libraries . . . . . . . . . . . . . . . . . . . . . . . . . 205
20.3 Python Games Development . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 206
20.4 Using Pygame . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 207
20.5 Online Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 207
21 Building Games with Pygame . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 209
21.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 209
21.2 The Display Surface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 210
21.3 Events . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 211
21.3.1 Event Types . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 211
21.3.2 Event Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 212
21.3.3 The Event Queue . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 213
Contents xix

21.4 A First pygame Application . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 214

21.5 Further Concepts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 217
21.6 A More Interactive pygame Application . . . . . . . . . . . . . . . . . . . . 219
21.7 Alternative Approach to Processing Input Devices . . . . . . . . . . . 221
21.8 pygame Modules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 222
21.9 Online Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 223
22 StarshipMeteors Pygame . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 225
22.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 225
22.2 Creating a Spaceship Game . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 225
22.3 The Main Game Class . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 226
22.4 The GameObject Class . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 228
22.5 Displaying the Starship . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 229
22.6 Moving the Spaceship . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 230
22.7 Adding a Meteor Class . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 233
22.8 Moving the Meteors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 235
22.9 Identifying a Collision . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 236
22.10 Identifying a Win . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 237
22.11 Increasing the Number of Meteors . . . . . . . . . . . . . . . . . . . . . . . . . 238
22.12 Pausing the Game . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 238
22.13 Displaying the Game Over Message . . . . . . . . . . . . . . . . . . . . . . . 239
22.14 The StarshipMeteors Game . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 240
22.15 Online Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 244
22.16 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 244

Part IV Testing
23 Introduction to Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 247
23.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 247
23.2 Types of Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 247
23.3 What Should Be Tested? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 248
23.4 Types of Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 249
23.4.1 Unit Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 250
23.4.2 Integration Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 251
23.4.3 System Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 251
23.4.4 Installation/Upgrade Testing . . . . . . . . . . . . . . . . . . . . . 252
23.4.5 Smoke Tests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 252
23.5 Automating Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 252
23.6 Test-Driven Development . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 253
23.6.1 The TDD Cycle . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 253
23.6.2 Test Complexity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 254
23.6.3 Refactoring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 255
23.7 Design for Testability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 255
23.7.1 Testability Rules of Thumb . . . . . . . . . . . . . . . . . . . . . . 255
23.8 Online Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 255
23.9 Book Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 256
xx Contents

24 PyTest Testing Framework . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 257

24.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 257
24.2 What is PyTest? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 257
24.3 Setting up PyTest . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 258
24.4 A Simple PyTest Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 258
24.5 Working with PyTest . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 261
24.6 Parameterised Tests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 265
24.7 Online Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 267
24.8 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 267
25 Mocking for Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 269
25.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 269
25.2 Why Mock? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 271
25.3 What is Mocking? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 272
25.4 Common Mocking Framework Concepts . . . . . . . . . . . . . . . . . . . 273
25.5 Mocking Frameworks for Python . . . . . . . . . . . . . . . . . . . . . . . . . . 274
25.6 The Unittest.Mock Library . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 274
25.6.1 Mock and Magic Mock Classes . . . . . . . . . . . . . . . . . . . 275
25.6.2 The Patchers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 276
25.6.3 Mocking Returned Objects . . . . . . . . . . . . . . . . . . . . . . . 277
25.6.4 Validating Mocks Have Been Called . . . . . . . . . . . . . . . 278
25.7 Mock and MagicMock Usage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 279
25.7.1 Naming Your Mocks . . . . . . . . . . . . . . . . . . . . . . . . . . . . 279
25.7.2 Mock Classes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 279
25.7.3 Attributes on Mock Classes . . . . . . . . . . . . . . . . . . . . . . 279
25.7.4 Mocking Constants . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 280
25.7.5 Mocking Properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 280
25.7.6 Raising Exceptions with Mocks . . . . . . . . . . . . . . . . . . . 281
25.7.7 Applying Patch to Every Test Method . . . . . . . . . . . . . 281
25.7.8 Using Patch as a Context Manager . . . . . . . . . . . . . . . . 281
25.8 Mock Where You Use It . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 282
25.9 Patch Order Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 282
25.10 How Many Mocks? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 283
25.11 Mocking Considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 283
25.12 Online Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 284
25.13 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 284

Part V File Input/Output

26 Introduction to Files, Paths and IO . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 289
26.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 289
26.2 File Attributes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 292
26.3 Paths . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 293
26.4 File Input/Output . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 295
Contents xxi

26.5 Sequential Access versus Random Access . . . . . . . . . . . . . . . . . . 296

26.6 Files and I/O in Python . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 297
26.7 Online Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 297
27 Reading and Writing Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 299
27.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 299
27.2 Obtaining References to Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 299
27.3 Reading Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 301
27.4 File Contents Iteration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 302
27.5 Writing Data to Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 302
27.6 Using Files and with Statements . . . . . . . . . . . . . . . . . . . . . . . . . . . 303
27.7 The Fileinput Module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 303
27.8 Renaming Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 304
27.9 Deleting Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 304
27.10 Random Access Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 305
27.11 Directories . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 306
27.12 Temporary Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 307
27.13 Working with Paths . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 309
27.14 Online Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 312
27.15 Exercise . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 313
28 Stream IO . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 315
28.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 315
28.2 What is a Stream? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 315
28.3 Python Streams . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 316
28.4 IOBase . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 317
28.5 Raw IO/UnBuffered IO Classes . . . . . . . . . . . . . . . . . . . . . . . . . . . 318
28.6 Binary IO/Buffered IO Classes . . . . . . . . . . . . . . . . . . . . . . . . . . . . 318
28.7 Text Stream Classes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 320
28.8 Stream Properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 321
28.9 Closing Streams . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 322
28.10 Returning to the Open() Function . . . . . . . . . . . . . . . . . . . . . . . . . . 322
28.11 Online Resource . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 323
28.12 Exercise . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 323
29 Working with CSV Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 325
29.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 325
29.2 CSV Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 325
29.2.1 The CSV Writer Class . . . . . . . . . . . . . . . . . . . . . . . . . . . 326
29.2.2 The CSV Reader Class . . . . . . . . . . . . . . . . . . . . . . . . . . 327
29.2.3 The CSV DictWriter Class . . . . . . . . . . . . . . . . . . . . . . . 328
29.2.4 The CSV DictReader Class . . . . . . . . . . . . . . . . . . . . . . 329
29.3 Online Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 330
29.4 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 330
xxii Contents

30 Working with Excel Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 333

30.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 333
30.2 Excel Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 333
30.3 The Openpyxl. Workbook Class . . . . . . . . . . . . . . . . . . . . . . . . . . . 334
30.4 The Openpyxl. WorkSheet Objects . . . . . . . . . . . . . . . . . . . . . . . . . 334
30.5 Working with Cells . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 334
30.6 Sample Excel File Creation Application . . . . . . . . . . . . . . . . . . . . 335
30.7 Loading a Workbook from an Excel File . . . . . . . . . . . . . . . . . . . . 336
30.8 Online Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 338
30.9 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 338
31 Regular Expressions in Python . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 341
31.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 341
31.2 What Are Regular Expressions? . . . . . . . . . . . . . . . . . . . . . . . . . . . 341
31.3 Regular Expression Patterns . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 342
31.3.1 Pattern Metacharacters . . . . . . . . . . . . . . . . . . . . . . . . . . 342
31.3.2 Special Sequences . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 343
31.3.3 Sets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 344
31.4 The Python re Module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 345
31.5 Working with Python Regular Expressions . . . . . . . . . . . . . . . . . . 345
31.5.1 Using Raw Strings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 345
31.5.2 Simple Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 346
31.5.3 The Match Object . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 346
31.5.4 The search() Function . . . . . . . . . . . . . . . . . . . . . . . . . . . 347
31.5.5 The match() Function . . . . . . . . . . . . . . . . . . . . . . . . . . . 348
31.5.6 The Difference Between Matching
and Searching . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 349
31.5.7 The finadall() Function . . . . . . . . . . . . . . . . . . . . . . . . . . 349
31.5.8 The finditer() Function . . . . . . . . . . . . . . . . . . . . . . . . . . 349
31.5.9 The split() Function . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 350
31.5.10 The sub() Function . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 350
31.5.11 The compile() Function . . . . . . . . . . . . . . . . . . . . . . . . . . 351
31.6 Online Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 353
31.7 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 354

Part VI Database Access

32 Introduction to Databases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 359
32.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 359
32.2 What Is a Database? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 359
32.2.1 Data Relationships . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 360
32.2.2 The Database Schema . . . . . . . . . . . . . . . . . . . . . . . . . . . 361
32.3 SQL and Databases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 363
32.4 Data Manipulation Language . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 364
32.5 Transactions in Databases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 365
32.6 Further Reading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 366
Contents xxiii

33 Python DB-API . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 369

33.1 Accessing a Database from Python . . . . . . . . . . . . . . . . . . . . . . . . 369
33.2 The DB-API . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 369
33.2.1 The Connect Function . . . . . . . . . . . . . . . . . . . . . . . . . . . 370
33.2.2 The Connection Object . . . . . . . . . . . . . . . . . . . . . . . . . . 370
33.2.3 The Cursor Object . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 371
33.2.4 Mappings from Database Types to Python Types . . . . 372
33.2.5 Generating Errors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 372
33.2.6 Row Descriptions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 373
33.3 Transactions in PyMySQL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 374
33.4 Online Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 375
34 PyMySQL Module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 377
34.1 The PyMySQL Module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 377
34.2 Working with the PyMySQL Module . . . . . . . . . . . . . . . . . . . . . . 377
34.2.1 Importing the Module . . . . . . . . . . . . . . . . . . . . . . . . . . . 378
34.2.2 Connect to the Database . . . . . . . . . . . . . . . . . . . . . . . . . 378
34.2.3 Obtaining the Cursor Object . . . . . . . . . . . . . . . . . . . . . . 379
34.2.4 Using the Cursor Object . . . . . . . . . . . . . . . . . . . . . . . . . 379
34.2.5 Obtaining Information About the Results . . . . . . . . . . . 380
34.2.6 Fetching Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 380
34.2.7 Close the Connection . . . . . . . . . . . . . . . . . . . . . . . . . . . 381
34.3 Complete PyMySQL Query Example . . . . . . . . . . . . . . . . . . . . . . 381
34.4 Inserting Data to the Database . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 382
34.5 Updating Data in the Database . . . . . . . . . . . . . . . . . . . . . . . . . . . . 384
34.6 Deleting Data in the Database . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 385
34.7 Creating Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 386
34.8 Online Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 386
34.9 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 387

Part VII Logging

35 Introduction to Logging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 391
35.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 391
35.2 Why Log? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 391
35.3 What is the Purpose of Logging? . . . . . . . . . . . . . . . . . . . . . . . . . . 392
35.4 What Should You Log? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 392
35.5 What not to Log . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 393
35.6 Why not Just Use Print? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 394
35.7 Online Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 395
36 Logging in Python . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 397
36.1 The Logging Module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 397
36.2 The Logger . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 398
36.3 Controlling the Amount of Information Logged . . . . . . . . . . . . . . 399
36.4 Logger Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 401
xxiv Contents

36.5 Default Logger . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 402

36.6 Module Level Loggers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 403
36.7 Logger Hierarchy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 404
36.8 Formatters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 405
36.8.1 Formatting Log Messages . . . . . . . . . . . . . . . . . . . . . . . . 406
36.8.2 Formatting Log Output . . . . . . . . . . . . . . . . . . . . . . . . . . 406
36.9 Online Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 408
36.10 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 408
37 Advanced Logging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 409
37.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 409
37.2 Handlers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 409
37.2.1 Setting the Root Output Handler . . . . . . . . . . . . . . . . . . 411
37.2.2 Programmatically Setting the Handler . . . . . . . . . . . . . 412
37.2.3 Multiple Handlers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 413
37.3 Filters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 415
37.4 Logger Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 416
37.5 Performance Considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 418
37.6 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 419

Part VIII Concurrency and Parallelism

38 Introduction to Concurrency and Parallelism . . . . . . . . . . . . . . . . . . . . 423
38.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 423
38.2 Concurrency . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 423
38.3 Parallelism . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 425
38.4 Distribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 426
38.5 Grid Computing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 427
38.6 Concurrency and Synchronisation . . . . . . . . . . . . . . . . . . . . . . . . . 428
38.7 Object Orientation and Concurrency . . . . . . . . . . . . . . . . . . . . . . . 428
38.8 Threads V Processes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 429
38.9 Some Terminology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 430
38.10 Online Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 430
39 Threading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 433
39.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 433
39.2 Threads . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 433
39.2.1 Thread States . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 433
39.2.2 Creating a Thread . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 434
39.2.3 Instantiating the Thread Class . . . . . . . . . . . . . . . . . . . . 435
39.3 The Thread Class . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 436
39.4 The Threading Module Functions . . . . . . . . . . . . . . . . . . . . . . . . . . 438
39.5 Passing Arguments to a Thread . . . . . . . . . . . . . . . . . . . . . . . . . . . . 438
39.6 Extending the Thread Class . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 440
39.7 Daemon Threads . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 441
Contents xxv

39.8 Naming Threads . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 442

39.9 Thread Local Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 442
39.10 Timers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 444
39.11 The Global Interpreter Lock . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 445
39.12 Online Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 446
39.13 Exercise . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 446
40 MultiProcessing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 449
40.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 449
40.2 The Process Class . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 449
40.3 Working with the Process Class . . . . . . . . . . . . . . . . . . . . . . . . . . . 451
40.4 Alternative Ways to Start a Process . . . . . . . . . . . . . . . . . . . . . . . . 453
40.5 Using a Pool . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 454
40.6 Exchanging Data Between Processes . . . . . . . . . . . . . . . . . . . . . . . 458
40.7 Sharing State Between Processes . . . . . . . . . . . . . . . . . . . . . . . . . . 460
40.7.1 Process Shared Memory . . . . . . . . . . . . . . . . . . . . . . . . . 460
40.8 Online Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 461
40.9 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 462
41 Inter Thread/Process Synchronisation . . . . . . . . . . . . . . . . . . . . . . . . . . . 463
41.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 463
41.2 Using a Barrier . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 463
41.3 Event Signalling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 467
41.4 Synchronising Concurrent Code . . . . . . . . . . . . . . . . . . . . . . . . . . . 469
41.5 Python Locks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 470
41.6 Python Conditions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 472
41.7 Python Semaphores . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 475
41.8 The Concurrent Queue Class . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 476
41.9 Online Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 478
41.10 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 478
42 Futures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 481
42.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 481
42.2 The Need for a Future . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 481
42.3 Futures in Python . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 482
42.3.1 Future Creation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 483
42.3.2 Simple Example Future . . . . . . . . . . . . . . . . . . . . . . . . . . 483
42.4 Running Multiple Futures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 485
42.4.1 Waiting for All Futures to Complete . . . . . . . . . . . . . . . 486
42.4.2 Processing Results as Completed . . . . . . . . . . . . . . . . . 488
42.5 Processing Future Results Using a Callback . . . . . . . . . . . . . . . . . 489
42.6 Online Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 490
42.7 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 490
xxvi Contents

43 Concurrency with AsyncIO . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 493

43.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 493
43.2 Asynchronous IO . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 493
43.3 Async IO Event Loop . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 494
43.4 The Async and Await Keywords . . . . . . . . . . . . . . . . . . . . . . . . . . . 495
43.4.1 Using Async and Await . . . . . . . . . . . . . . . . . . . . . . . . . . 496
43.5 Async IO Tasks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 498
43.6 Running Multiple Tasks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 500
43.6.1 Collating Results from Multiple Tasks . . . . . . . . . . . . . 500
43.6.2 Handling Task Results as They Are Made
Available . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 501
43.7 Online Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 502
43.8 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 502
44 Performance Monitoring and Profiling . . . . . . . . . . . . . . . . . . . . . . . . . . 505
44.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 505
44.2 Why Monitor Performance and Memory? . . . . . . . . . . . . . . . . . . . 505
44.3 Performance Monitoring and Profiling . . . . . . . . . . . . . . . . . . . . . 506
44.4 Performance Monitoring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 507
44.4.1 The Time Module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 507
44.4.2 The Timeit Module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 508
44.4.3 The Psutil Module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 510
44.5 Python Profiling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 511
44.5.1 The cProfile Module . . . . . . . . . . . . . . . . . . . . . . . . . . . . 511
44.5.2 The Line_Profiler Module . . . . . . . . . . . . . . . . . . . . . . . 511
44.5.3 The Memory_Profiler Module . . . . . . . . . . . . . . . . . . . . 512
44.5.4 Additional Third-Party Libraries . . . . . . . . . . . . . . . . . . 512
44.6 Profiling with cProfile . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 512
44.7 Memory Profiling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 515
44.8 Online Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 517

Part IX Reactive Programming

45 Reactive Programming Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . 521
45.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 521
45.2 What Is a Reactive Application? . . . . . . . . . . . . . . . . . . . . . . . . . . . 521
45.3 The ReactiveX Project . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 522
45.4 The Observer Pattern . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 522
45.5 Hot and Cold Observables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 523
45.6 Differences Between Event Driven Programming
and Reactive Programming . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 524
45.7 Advantages of Reactive Programming . . . . . . . . . . . . . . . . . . . . . . 525
45.8 Disadvantages of Reactive Programming . . . . . . . . . . . . . . . . . . . 525
45.9 The RxPy Reactive Programming Framework . . . . . . . . . . . . . . . 526
45.10 Online Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 526
Contents xxvii

46 RxPy Observables, Observers and Subjects . . . . . . . . . . . . . . . . . . . . . . 527

46.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 527
46.2 RxPy Library . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 527
46.3 Observables in RxPy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 527
46.4 Observers in RxPy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 528
46.5 Multiple Subscribers/Observers . . . . . . . . . . . . . . . . . . . . . . . . . . . 530
46.6 Subjects in RxPy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 531
46.7 Observer Concurrency . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 534
46.7.1 Available Schedulers . . . . . . . . . . . . . . . . . . . . . . . . . . . . 535
46.8 Online Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 536
46.9 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 536
47 RxPy Operators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 537
47.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 537
47.2 Reactive Programming Operators . . . . . . . . . . . . . . . . . . . . . . . . . . 537
47.3 Piping Operators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 538
47.4 Creational Operators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 539
47.5 Transformational Operators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 539
47.6 Combinatorial Operators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 541
47.7 Filtering Operators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 542
47.8 Mathematical Operators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 543
47.9 Chaining Operators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 544
47.10 Online Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 546
47.11 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 546

Part X Network Programming

48 Introduction to Sockets and Web Services . . . . . . . . . . . . . . . . . . . . . . . 551
48.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 551
48.2 Sockets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 551
48.3 Web Services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 552
48.4 Addressing Services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 552
48.5 Localhost . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 553
48.6 Port Numbers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 554
48.7 IPv4 Versus IPv6 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 555
48.8 Sockets and Web Services in Python . . . . . . . . . . . . . . . . . . . . . . . 555
48.9 Online Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 556
49 Sockets in Python . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 557
49.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 557
49.2 Socket to Socket Communication . . . . . . . . . . . . . . . . . . . . . . . . . . 557
49.3 Setting up a Connection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 558
49.4 An Example Client Server Application . . . . . . . . . . . . . . . . . . . . . 558
49.4.1 The System Structure . . . . . . . . . . . . . . . . . . . . . . . . . . . 558
49.4.2 Implementing the Server Application . . . . . . . . . . . . . . 559
xxviii Contents

49.4.3 Socket Types and Domains . . . . . . . . . . . . . . . . . . . . . . . 560

49.4.4 Implementing the Client Application . . . . . . . . . . . . . . 561
49.5 The Socketserver Module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 562
49.6 Http Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 565
49.7 Online Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 567
49.8 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 568
50 Web Services in Python . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 571
50.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 571
50.2 RESTful Services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 571
50.3 A RESTful API . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 572
50.4 Python Web Frameworks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 573
50.5 Online Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 574
51 Flask Web Services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 575
51.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 575
51.2 Flask . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 575
51.3 Hello World in Flask . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 576
51.3.1 Using JSON . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 576
51.4 Implementing a Flask Web Service . . . . . . . . . . . . . . . . . . . . . . . . 577
51.4.1 A Simple Service . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 577
51.4.2 Providing Routing Information . . . . . . . . . . . . . . . . . . . 577
51.5 Running the Service . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 578
51.6 Invoking the RESTFul Service . . . . . . . . . . . . . . . . . . . . . . . . . . . . 579
51.6.1 The Final Solution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 580
51.7 Online Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 580
52 Flask Bookshop Web Service . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 583
52.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 583
52.2 Building a Flask Bookshop Service . . . . . . . . . . . . . . . . . . . . . . . . 583
52.3 The Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 583
52.4 The Domain Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 585
52.5 Encoding Books into JSON . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 586
52.6 Setting Up the GET Services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 588
52.7 Deleting a Book . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 589
52.8 Adding a New Book . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 590
52.9 Updating a Book . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 592
52.10 What Happens if We Get It Wrong? . . . . . . . . . . . . . . . . . . . . . . . . 593
52.11 Bookshop Services Listing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 594
52.12 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 596

Part XI Data Science: Data Analytics and Machine Learning

53 Introduction to Data Science . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 601
53.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 601
53.2 Data Science . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 601
53.3 Data Science Tools and Techniques . . . . . . . . . . . . . . . . . . . . . . . . 602
Contents xxix

53.4 Data Analytics Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 604

53.5 Python and Data Science . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 606
53.6 Machine Learning for Data Science . . . . . . . . . . . . . . . . . . . . . . . . 607
53.7 Online Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 608
54 Pandas and Data Analytics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 611
54.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 611
54.2 The Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 611
54.2.1 The UK Government COVID Data Set . . . . . . . . . . . . . 611
54.2.2 The Google Mobility Data Set . . . . . . . . . . . . . . . . . . . . 613
54.3 Python Pandas . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 614
54.3.1 Pandas Series and DataFrames . . . . . . . . . . . . . . . . . . . . 615
54.4 Loading and Analysing UK COVID Data Set . . . . . . . . . . . . . . . . 616
54.5 Loading the Google Mobility Data Set . . . . . . . . . . . . . . . . . . . . . 621
54.6 Merging Two DataFrames . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 622
54.7 Analysing the Combined Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . 623
54.8 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 627
55 Alternatives to Pandas . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 629
55.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 629
55.2 Comparing Pandas 2.0.0 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 629
55.3 Pandas 1.x v 2.x . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 629
55.4 Pandas Versus Other Libraries and Tools . . . . . . . . . . . . . . . . . . . . 630
55.5 Online Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 632
56 Machine Learning in Python . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 633
56.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 633
56.2 The Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 633
56.3 SciKitLearn . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 634
56.4 The Problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 635
56.5 Using Regression Supervised Learning Systems . . . . . . . . . . . . . 636
56.6 K-Nearest Neighbour Regressor . . . . . . . . . . . . . . . . . . . . . . . . . . . 636
56.7 Decision Tree Regressor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 638
56.8 Random Forest Regressor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 639
56.9 Summary of Metrics Obtained . . . . . . . . . . . . . . . . . . . . . . . . . . . . 640
56.10 Creating the Regressor Object . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 640
56.11 Online Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 642
57 Pip and Conda Virtual Environments . . . . . . . . . . . . . . . . . . . . . . . . . . . 643
57.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 643
57.2 Virtual Environments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 643
57.3 Working with Pip . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 644
57.3.1 Activating a Pip Environment . . . . . . . . . . . . . . . . . . . . 644
57.3.2 Installing Modules Using Pip . . . . . . . . . . . . . . . . . . . . . 645
57.3.3 Deactivating a Pip Environment . . . . . . . . . . . . . . . . . . 646
57.3.4 Check Version of Pip . . . . . . . . . . . . . . . . . . . . . . . . . . . . 646
57.3.5 Installing Modules into a Pip Environment . . . . . . . . . 646
xxx Contents

57.3.6 Freezing Modules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 647

57.4 Conda . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 648
57.5 Anaconda . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 649
57.5.1 Installing Anaconda . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 649
57.6 Working with Anaconda . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 651
57.6.1 Checking the Conda Version . . . . . . . . . . . . . . . . . . . . . 651
57.6.2 Updating Conda . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 651
57.6.3 Creating a Conda Environment . . . . . . . . . . . . . . . . . . . 652
57.6.4 Listing Available Conda Environments . . . . . . . . . . . . 653
57.6.5 Activating a Conda Environment . . . . . . . . . . . . . . . . . . 654
57.6.6 Deactivating a Conda Environment . . . . . . . . . . . . . . . . 655
57.6.7 Listing the Modules Loaded into a Conda
Environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 655
57.6.8 Removing an Anaconda Environment . . . . . . . . . . . . . 656
57.6.9 Installing a Module into a Conda Environment . . . . . . 656
57.7 Anaconda in PyCharm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 658
57.8 Online Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 658
Chapter 1
Introduction

1.1 Introduction

I have heard many people over the years say that Python is an easy language to learn
and that Python is also a simple language.
To some extent both of these statements are true; but only to some extent.
While the core of the Python language is easy to lean and relatively simple (in
part thanks to its consistency), the sheer richness of the language constructs and
flexibility available can be overwhelming. In addition the Python environment, its
eco system, the range of libraries available, the often competing options available,
etc., can make moving to the next-level daunting.
Once you have learned the core elements of the language such as how classes
and inheritance work, how functions work, what are protocols and Abstract Base
Classes, etc. where do you go next?
The aim of this book is to delve into those next steps. The book is organised into
eleven different topics:
1. Advanced Language Features. The first section in the book covers topics
that are often missed out from introductory Python books such as slots, weak
references __init__() versus __new__() and metaclasses.
2. Computer Graphics. The book covers Computer Graphics and Computer
Generated Art in Python as well as graphical user interfaces and graphing/
charting via Matplotlib.
3. Games Programming. This topic is covered using the pygame library.
4. Testing and Mocking. Testing is an important aspect of any software develop-
ment; this book introduces testing in general and the PyTest module in detail.
It also considers mocking within testing including what and when to mock.

© Springer Nature Switzerland AG 2023 1

J. Hunt, Advanced Guide to Python 3 Programming,
Undergraduate Topics in Computer Science,
https://doi.org/10.1007/978-3-031-40336-1_1
2 1 Introduction

5. File Input/Output. The book covers text file reading and writing as well as
reading and writing CSV and Excel files. Although not strictly related to file
input, regulator expressions are included in this section as they can be used to
process textual data held in files.
6. Database Access. The book introduces databases and relational database in
particular. It then presents the Python DB-API database access standard and
one implementation of this standard, the PyMySQL module used to access a
MySQL database.
7. Logging. An often missed topic is that of logging. The book therefore introduces
logging the need for logging, what to log and what not to log as well as the Python
logging module.
8. Concurrency and Parallelism. The book provides extensive coverage of
concurrency topics including threads, processes and inter-thread or process
synchronisation. It also presents futures and AsyncIO.
9. Reactive Programming. This section of the book introduces Reactive Program-
ming using the PyRx Reactive Programming library.
10. Network Programming. The book introduces socket and web service commu-
nications in Python. It looks at both the Flask and the Django web service
libraries.
11. Data Analytics. A very hot topic for any potential Python programmer is data
analytics (and the related use of machine learning). The book concludes by
introducing these topics and there Pandas and scikit-learn (or SK-learn as it is
sometimes known) libraries.
Each section is introduced by a chapter providing the background and key concepts
of that topic. Subsequent chapters then cover various aspects of the topic.
For example, the second topic covered is on Computer Graphics. This section
has an introductory chapter on Computer Graphics in general. It then introduces the
Turtle Graphics Python library which can be used to generate a graphical display.
The following chapter considers the subject of Computer Generated Art and
uses the Turtle Graphics library to illustrate these ideas. Thus several examples
are presented that might be considered art. The chapter concludes by presenting the
well-known Koch Snowflake and the Mandelbrot Fractal set.
This is followed by a chapter presenting the Matplotlib library used for generating
2D and 3D charts and graphs (such as a line chart, bar chart or scatter graph).
The section concludes with a chapter on graphical user interfaces (or GUIs) using
the wxpython library. This chapter explores what we mean by a GUI and some of
the alternatives available in Python for creating a GUI.
Other topics follow a similar pattern.
Each programming or library-oriented chapter also includes numerous sample
programs that can be downloaded from the GitHub repository and executed. These
chapters also include one or more end of chapter exercises (with sample solutions
also in the GitHub repository).
1.2 Useful Python Resources 3

The topics within the book can be read mostly independently of each other. This
allows the reader to dip into subject areas as and when required. For example, the
File Input/Output section and the Database Access section can read independently
of each other (although in this case assessing both technologies may be useful in
selecting an appropriate approach to adopt for the long-term persistent storage of
data in a particular system).
Within each section there are usually dependencies; for example, it is neces-
sary to understand pygame library from the ‘Building Games with pygame’ intro-
ductory chapter, before exploring the worked case study presented by the chapter
on the StarshipMeteors game. Similarly it is necessary to have read the threading
and multiprocessing chapters before reading the inter-thread/process synchronisation
chapter.

1.2 Useful Python Resources

There are a wide range of resources on the web for Python; we will highlight a few
here that you should bookmark. We will not keep referring to these to avoid repetition
but you can refer back to this section whenever you need to:
• https://en.wikipedia.org/wiki/Python_Software_Foundation Python Software
Foundation.
• https://docs.python.org/3/ The main Python 3 documentation site. It contains
tutorials, library references, set up and installation guides as well as Python
how-tos.
• https://docs.python.org/3/library/index.html A list of all the built-in features for
the Python language—this is where you can find online documentation for the
various class and functions that we will be using throughout this book.
• https://pymotw.com/3/ the Python 3 Module of the week site. This site contains
many, many Python modules with short examples and explanations of what the
modules do. A Python module is a library of features that build on and expand
the core Python language. For example, if you are interested in building games
using Python then pygame is a module specifically designed to make this easier.
• https://www.fullstackpython.com/email.html is a monthly newsletter that
focusses on a single Python topic each month, such as a new library or module.
• http://www.pythonweekly.com/ is a free weekly summary of the latest Python
articles, projects, videos and upcoming events.
Each section of the book will provide additional online references relevant to the
topic being discussed.
 Part I
Advanced Language Features
Chapter 2
Python Type Hints

2.1 Introduction

Python is a dynamically typed language right—well yes it is however there is a

feature known as Type Hints that allows typing information to be provided when
functions and methods are defined. These Type Hints are extremely useful and can
help a developer understand what types are expected by a function or a method and
indeed what types are likely to be returned.
Recent versions of Python, including 3.10 and 3.11, have increased the support
for Type Hints so that they are now quite usable. These can be used by analysis tools
and IDEs to help developers create more stable and reliable applications.

2.2 Pythons Type System

Many people consider Python to be an untyped programming language. However,

that is not quite true. The type system in Python can be referred to as representing
a dynamically typed programming language. That is a variable holds a value and
the type of that value is known and understood by the language. At runtime Python
checks that what you are trying to do is valid given the types involved. For example,
you can use the type() function to find out what type of thing a variable holds at
any point in time:
a_variable = 42
print(f'a_variable type = {type(a_variable)} = {a_variable}')
a_variable = 1.345
print(f'a_variable type = {type(a_variable)} = {a_variable}')
a_variable = "Hello"
print(f'a_variable type = {type(a_variable)} = {a_variable}')
a_variable = True
print(f'a_variable type = {type(a_variable)} = {a_variable}')

© Springer Nature Switzerland AG 2023 7

J. Hunt, Advanced Guide to Python 3 Programming,
Undergraduate Topics in Computer Science,
https://doi.org/10.1007/978-3-031-40336-1_2
8 2 Python Type Hints

This produces as output:

a_variable type = <class 'int'> = 42
a_variable type = <class 'float'> = 1.345
a_variable type = <class 'str'> = Hello
a_variable type = <class 'bool'> = True

Of course, as the above shows, a variable in Python can hold different types of
things at different types, hence the term dynamically typed.
As Python knows what types variables hold it can check at runtime that your
programs are valid/correct given the types involved for example in a particular oper-
ation. Thus, it is valid to add two integers together and indeed two strings together
(as this is string concatenation) but attempting to add an integer to a string will result
in a TypeError:
print(1 + 1)
print(1.2 + 3.4)
print("Hello" + "world")
print("Hello" + 1)

This code produced the following output including the TypeError:

2
4.6
Helloworld
Traceback (most recent call last):
File "/Users/jeh/temp/pythonProjects/course/main.py", line 15,
in <module>
print("Hello" + 1)
TypeError: can only concatenate str (not "int") to str

2.3 The Challenge for Python Developers

The challenge for Python developers comes when they need to understand what types
are required by, or work with, some API. As a very simple example, consider the
following function:
def add(x, y):
return x + y

What types can be used with this function?

In essence any type can be used for the parameter x that supports the plus operator
(+) with the type in y. From the above we know that integers and strings can be used,
but we can also use floating point numbers, for example:
print(add(1, 2))
print(add(1.2, 3.4))
2.4 Static Typing 9

print(add(1, 3.4))
print(add(5.5, 1))
print(add("Hi", "There"))

All of the above are valid parameters, and the output produced from the above
code is:
3
4.6
4.4
6.5
HiThere

Even custom types can be used if they implement the special __add__(self,
other) operator method, for example:
class Quantity:
def __init__(self, amount):
self.amount = amount

def __add__(self, other):

return Quantity(self.amount + other.amount)

def __str__(self):
return f"Quantity({self.amount})"

q1 = Quantity(5)
q2 = Quantity(4)
print(add(q1, q2))

The __add__() method allows the custom type (class) being defined to be used
with the add operator (‘+’). Thus this program generates the following output:
Quantity(9)

However, what was the intent of the designer of this add() function? What did
they expect you to add together? The only option in traditional Python code is for
the developer to provide some form of documentation, for example in the form of a
docstring:
def add(x, y):
"""adds two integers together and
returns the resulting integer."""
return x + y

2.4 Static Typing

Languages such as Java, C# and C are statically typed languages. That is when a
variable, object attribute, parameter or return type is defined then the type of that
element is specified statically a compile time.
10 2 Python Type Hints

Thus, an add() method on a Java class Calculator might be written as

follows:
package com.jjh;

public class Calculator {

public int add(int x, int y) {

return x + y;
}

This makes it clear to a Java programmer and to the Java compiler that the add()
method will only handle integers and will return as a result an integer type. Thus,
there is no possibility that a developer might try to add a number to a Boolean value,
etc. In fact the compiler will not even allow it!
The Java Calculator class can be used as shown below, note that this code
will not even compile if the developer tries to add two strings together. In this case
we are adding two integers together, so all is fine:
package com.jjh;

public class App {

public static void main( String[] args ) {
System.out.println( "Starting" );
Calculator calc = new Calculator();
System.out.println(calc.add(4, 5));
System.out.println("Done");
}
}

As the above program uses valid integer types with the add() method, the output
from the compiled and executed program is:
Starting
9
Done

2.5 Python Type Hints

Python’s Type Hints are more like a half-way house between traditional Python’s
lack of typing information at all and the very strict string static typing approach of
languages such as Java.
A Type Hint is additional type information that can be used with a function
definition to indicate what types parameters should be and what type is returned.
This is illustrated below:
def add(x: int, y: int) -> int:
2.6 Type Hint Layout 11

return x + y

In this case it makes it clear that both x and y should be of type int (integer
types) and the returned result will be an int. However, adding Type Hints as shown
above has no effect on the runtime execution of the program; they are only hints and
are not enforced by Python per se. For example, it is still possible to pass a string
into the add() function as far as Python is concerned.
However, static analysis tools (such as MyPy) can be applied to the code to check
for such misuse. Some editors, such as the widely used PyCharm, already have such
tools integrated into their code checking behaviour.
If you want to use a tool such as mypy instead, or in addition to that available in
your IDE, then you can install it using
pip install mypy

Or if you want to use conda/Anaconda by using

conda install mypy

You can now analyse your code by applying MyPy to a Python file, for example:
% mypy main.py
main.py:3: error: Incompatible types in assignment (expression has
type "float", variable has type "int")
main.py:5: error: Incompatible types in assignment (expression has
type "str", variable has type "int")
main.py:24: error: Argument 1 to "add" has incompatible type "str";
expected "int"
main.py:24: error: Argument 2 to "add" has incompatible type "str";
expected "int"
main.py:44: error: Argument 1 to "add" has incompatible type
"Quantity"; expected "int"
main.py:44: error: Argument 2 to "add" has incompatible type
"Quantity"; expected "int"
Found 6 errors in 1 file (checked 1 source file)

2.6 Type Hint Layout

The Python Style Guide defined by Python Enhancement Proposal 8 (PEP 8) provides
some guidance for using Type Hints, for example:
• Use normal rules for colons, that is, no space before and one space after a colon:
text: str.
• Use spaces around the = sign when combining an argument annotation with a
default value: align: bool = True.
• Use spaces around the -> arrow: def headline(…) -> str.
12 2 Python Type Hints

2.7 Type Hints for Multiple Types

Of course our add() function could work with floating point numbers as well as it
works with integers. It would therefore be useful to be able to state this in terms of the
Type Hints. Prior to Python 3.10 this could be done using a Union type, for example
Union[int, float] which while it worked was a little unwieldy. Since Python
3.10 we can use the style syntax bar ‘|’ for example int | float as shown below:
def add(x: int | float, y: int | float) -> int:
return x + y

2.8 The Self Type

Python 3.11 introduced the Self type which is defined in PEP 673. This can be used
to indicate that a method returns a reference to itself, for example:
from typing import Self

class Shape:

def __init__(self):
self.scale = 0.0

def set_scale(self, scale: float) -> Self:

self.scale = scale
return self

2.9 The Benefits of Type Hints

There are a range of benefits to using Type Hints in Python, for example:
• They help catch some errors within programs. Obviously, the biggest benefit
is that Type Hints can help developers catch certain types of problems in their
code (assuming that some form of type checker is used).
• They provide documentation. Type Hints can also act as a level of document
that editors such as IDEs can pick up and display to other developers.
• They can be work with IDEs. They can help with code generation and IDE
auto-complete functionality.
• They can make developers stop and think. They can help ensure that developers
think about their code and what types should be supported.
• They can improve understanding of libraries. Although Type Hints may offer
little advantage in a single use script, or throw away program, they can be of
significant benefit when a library is being created. Such libraries will be used
2.11 Online Resources 13

by a range of different developers, and some may be released into the wild, for
example via PyPI, the Python Package Index. The use of Type Hints can greatly
enhance others understanding of the APIs provided by these libraries.

2.10 Summary

If you are just starting out with Python, or you are writing scripts that will only be used
once, then Type Hints may not be particularly useful. However, if you are creating
libraries or developing larger more complex applications with teams of developers,
then they can be very useful indeed.

2.11 Online Resources

• https://docs.python.org/3.10/ Python 3.10 documentation.

• https://docs.python.org/3.11/ Python 3.11 documentation.
• https://en.wikipedia.org/wiki/Type_system Wikipedia Type System page.
• https://en.wikipedia.org/wiki/Dynamic_programming_language Dynamically
typed languages.
• https://docs.python.org/3/library/exceptions.html Exception handling in Python.
• https://www.pythontutorial.net/python-basics/python-type-hints/ Tutorial on
Python Type Hints.
• https://pypi.org/project/mypy/ MyPy static type hint analysis tool.
• https://mypy.readthedocs.io/en/stable/ MyPy documentation.
• https://www.jetbrains.com/pycharm/ PyCharm IDE tool.
• https://peps.python.org/pep-0008/ Python (PEP 8) Style Guide including guid-
ance of how to layout Type Hints.
• https://peps.python.org/pep-0673/ Information on the Self type.
• https://pypi.org/ The Python package Index PyPi.
Exploring the Variety of Random
Documents with Different Content
close upon a thousand volumes, including rare alchemical and early
chemical works, and of complete sets of some of the most valuable
of the serial publications of the science.
It was in grateful recognition of this liberal and warm-hearted
encouragement of the objects for which the Society was instituted
that his old pupils resolved to commemorate his connection with it
by placing, on the occasion of his eightieth birthday, his bust in the
library which he had done so much to enrich, and at the same time
to offer a replica of Mr. Drury’s admirable work to his family.
Roscoe was one of the original members of the Society of
Chemical Industry, and took a leading part in establishing it on its
present basis as a national institution, with local sections in many of
our principal towns, and branches in certain of our colonies, and in
America. How it originated may be gleaned from the following letter.
There had been a previous attempt to establish a local society with
special reference to the South Lancashire district. But Roscoe, with
others of its projectors, had conceived the idea of placing it on a
wider plane, and the meeting referred to was called to ascertain the
general feeling as to the expediency of the action.

The Owens College, Manchester,

April 13, ’80.
We are going to have a meeting here on Monday
next, at 7 p.m., of chemical manufacturers and others,
to consider the question of the establishment of a
Lancashire Chemical Society, or of a more General
Institute or Society of Chemical Engineers.
If you can come over I should be very glad, as it is
likely to be an important meeting and your opinion
would be of value.
I congratulate you (and Mrs. Thorpe) on your
completion of the paper. We are to decide, or begin to
 decide, about the Chemical Society Medal on Thursday,
and I hope it will be settled to your satisfaction.

At this gathering it was decided to attempt to establish a society

to subserve the general interests of chemical industry and not
merely those of the South Lancashire district, and to call a meeting
in London to discuss the project. He presided at this meeting in the
rooms of the Chemical Society when the Society was formally
inaugurated. He served as its first President, and was the first
chairman of its Manchester section. He opened the proceedings of
its first annual general meeting in 1881 with an account of the
reasons for the formation of such a society; he indicated its
proposed scope, and dwelt upon the many advantages to science
and industry that might be expected to follow its creation. What he
said then is not less pertinent now. Perhaps, indeed, it is even more
so. His words were words of wisdom and of warning. Read in the
fierce light of current events, and of present disabilities, we may well
inquire whether the generation that has passed has been as mindful
as he had hoped it would be of its opportunities. If it had quickened
its energies and marshalled its forces as he encouraged it to do, we
should have been better able to meet the strenuous times that are
now in store for us. For upwards of a century—ever since, indeed,
the renascence of chemical science which originated with Lavoisier—
far-sighted men have been preaching the same story. But to the
great majority in this country it has been as seed fallen by the
wayside. Not so abroad. Thanks mainly to a clearer recognition of
the part that science plays under the changing and progressive
conditions of modern life, other nations have been more heedful: the
seed with them fell upon more receptive soil. The catastrophe which
has overtaken us has brought a rude awakening. We are beginning
to realize the imperfection of a system of national education which
has no adequate relation to present-day necessities.
Roscoe, one is consoled to think, lived to see the evidence of this
quickening. The doctrine which he preached with an insistency and
pertinacity that never flagged, during more than half a century, is
now, under the stress of necessity, coming home to men’s business
and bosoms as it never did before.
One proof of the Society’s recognition of its indebtedness to its
first President may be seen in the award to him of its medal on the
occasion of the Nottingham meeting in 1914.
In 1909 he was Honorary President of the Seventh International
Congress of Applied Chemistry which met in London in that year. It
was his wish that his friend, Dr. Ludwig Mond, as an eminent
industrial chemist, should be appointed to that office, but Mond
declined the position and proposed Roscoe’s name instead at the
preceding meeting in Rome.
To Roscoe, therefore, fell the honour of introducing the Prince and
Princess of Wales—our present King and Queen—to the meeting of
three thousand industrial chemists assembled in the Albert Hall,
when the Prince welcomed the gathering in a felicitous speech. The
foreign delegates were afterwards received in private audience by
King Edward VII, when Roscoe, as Honorary President, had the
honour of presenting them.
Roscoe’s first introduction to the British Association was at the
Glasgow meeting in 1855, when the late Lord Playfair was President
of the Chemical Section, and when he himself acted as the secretary.
At this meeting he read a paper on the results of a joint investigation
with Bunsen on the action of light upon chlorine water, an
examination and extension of Wittwer’s work on the same subject.
This was subsequently published in the Journal of the Chemical
Society and in Liebig’s Annalen. As already stated, he acted as one
of the local secretaries at the Manchester meeting of 1861, when he
presented a report jointly with the late Drs. Schunck and R. Angus
Smith, “On the Condition of Manufacturing Chemistry in the South
Lancashire District” (Brit. Assoc. Rep. 1861, pp. 108-128). At the
Bath meeting in 1864 he gave one of the evening lectures on the
chemical action of light. At the Liverpool meeting of 1870 he
presided over the Chemistry Section.
 Then, as now, France and Germany were at war, and that fact
naturally called for reference. But, Eheu! neither the cosmopolitan
character of Science to which he then alluded, nor upwards of forty
years of that comity among those interested in Science and its
applications which he confidently hoped would “render impossible
the breaking out of disasters so fatal to the progress of Science and
to the welfare of humanity!” as he then witnessed, have served to
avert an even more fearful disaster. The small but living fire which
he contended would in the end surely serve to melt down national
animosities has been now almost wholly extinguished by the
arrogant pride and lust of power which has obsessed a nation
claiming to be the most enlightened in the world.
In 1884 he again served as President of the Chemical Section at
the meeting in Montreal. In 1887 he was President of the Association
at the Manchester meeting in that year—an honour he prized all the
more on account of his long association with that city. The meeting
was notable as being the largest held since the foundation of the
Association, and was specially characterized by the number of
foreign chemists who were present. He continued to take an active
interest in the affairs of the Association up to the time of his death,
and was a constant attendant at the meetings of its Council.
 CHAPTER X
PUBLIC SERVICES—POLITICAL AND
PROFESSIONAL WORK

Roscoe’s position in the educational world, and in scientific circles,

coupled with his well-known business capacity and sound judgment,
frequently led to his being invited to place his knowledge and
experience at the service of the State in connection with Royal
Commissions and Departmental Committees. During his tenure of his
professorship at Manchester he served on two important Royal
Commissions; the first in 1876, when Mr. Cross, then Home
Secretary, nominated him as a member of Lord Aberdare’s
Commission on Noxious Vapours, which led to the amended Alkali
Acts of 1891 and 1892; the second in 1881, when Mr. A. J. Mundella,
then Vice-President of the Education Council, appointed him a
member of Sir Bernhard Samuelson’s Commission on Technical
Instruction—one of the most important Commissions ever issued by
reason of its influence on the industrial history of this country.
Roscoe threw himself heart and soul into its work. The task was
thoroughly congenial to him, for he was profoundly convinced of its
importance. It required long and frequent visits abroad in order to
inquire into the methods of the continental trade-schools and
polytechnics, and to judge by direct observation of their results. The
preparation of the Report was a tedious and complicated business,
but with the help of his colleagues, whom he invited to his holiday-
home in the Lakes, it was gradually, as he says, “licked into shape,”
the last touches to its recommendations being made at the
Chairman’s country house in Devonshire.
During the ten years that followed the publication of the Report,
Roscoe, in common with several of his colleagues, addressed
innumerable public meetings throughout the country in order to
make its lessons as widely known as possible. The work of the
Commission bore fruit in the Technical Instruction Act of 1889, and
still later and to a greater extent in the Education Act of 1902. This
last measure was preceded by the Royal Commission on Secondary
Education, of which Roscoe was a member under the chairmanship
of the present Lord Bryce. In 1896 he introduced a strong and
representative deputation to urge upon the Lord President of the
Council the desirability of taking steps to enforce its
recommendations. It was then intimated that it was the intention of
the Government to introduce legislation dealing with the
organization of our secondary schools—thus foreshadowing the Act
of 1902.
Although more than thirty years have passed since the Report of
the Technical Instruction Commission was issued, it may still be read
with profit. Indeed, the lessons it teaches are singularly applicable to
the present juncture. In spite of what has been accomplished,
Roscoe was far from being satisfied with our national position. In
1906 he wrote:

Much remains for us in England to accomplish in the

organization of our secondary and scientific training, in
which our competitors are before us, and of which the
importance and the effects are well summed up in the
following opinion of an eminent German manufacturer:
“We in Germany do not care whether you in England
are Free-traders or Protectionists, but what we are
afraid of is that some day your people will wake up to
the necessity of having a complete system of technical
and scientific education, and then with your energetic
population, with your insular position, and with your
stores of raw material it will be difficult, or it may be
impossible, for us to compete.”
 In 1884 a knighthood was conferred on him, as stated in Mr.
Gladstone’s letter when intimating the Queen’s pleasure, “in
acknowledgment of his distinguished service on the Technical
Education Commission.”

Roscoe has recorded in his autobiography the circumstances,

altogether unexpected by him, which led to his introduction to active
political life. He was elected for South Manchester—a constituency
largely composed of the upper and middle class—in 1885, the first
Member of Parliament for the division in which the University is
situated and the only Liberal then returned for the city. He held his
seat during two succeeding elections (1886 and 1892), but lost it in
1895, by a narrow majority, to the Marquis of Lorne. Although
frequently solicited to re-enter Parliament he felt, to use his own
phrase, that he “had had enough.”
Roscoe was a strong and consistent Liberal, a member of the
Manchester School of Economists, and a devoted adherent of Mr.
Gladstone, whom he followed in the Home Rule split. During the
greater part of his parliamentary career, that is from 1886 to 1892,
he sat on the Opposition benches, and had therefore comparatively
little opportunity of accomplishing much in the way of legislative
achievement. On questions involving scientific matters he could
always secure the ear of the House, especially when these related to
the comfort and well-being of its members, as when he took in hand
the ventilation, lighting, and drainage of the Palace of Westminster.
In 1888, and again in 1889, he introduced a Technical Education Bill,
but it failed to reach the statute-book. In the latter year, however,
the Government passed the Technical Instruction Act already
referred to; this, although not wholly in accord with the views he
had put forward, he gladly accepted as a satisfactory instalment. His
efforts to pass an amending Bill in the following year met with no
success. In 1891 the National Association for the Promotion of
Technical Education, which was founded as a result of the Report of
the Royal Commission of 1881, entrusted him with a Bill to remove
certain disabilities which had been found to attend the working of
the Act of 1889, and this he succeeded in carrying. It was one of the
few private Bills of the session of 1891 that became law.
He was frequently called upon to serve upon select committees,
and in his last session he was Chairman of the Select Committee on
Weights and Measures which led to Mr. Balfour’s Bill for legalizing the
use of the metrical system in this country.
Roscoe was a Vice-President of the Decimal Association, and lost
no opportunity of advocating the use of a system of weights and
measures which practically every other civilized community has
found it expedient to adopt. The reform of the method of holding
parliamentary elections, continuation schools, opening museums on
Sundays, the housing of the science collections at South Kensington,
grants to University colleges, industrial employment in Ireland,
limitation of moisture in weaving sheds, river pollution—were all
questions upon which he was able to exercise his influence and
knowledge, and most of which he lived to see satisfactorily settled.
But, on the whole, he found little satisfaction in his parliamentary
life. There was much in it that was irksome and distasteful to a man
of his active and independent mind. It was unfortunate for him that
the greater part of his political career should have to be spent in
opposition, thus affording him only limited opportunities of initiating
legislative action. Owing to the political circumstances of the time
many questions with which he was specially qualified to deal never
came up for consideration. Others were only discussed for the
purpose of “marking time,” and he deplored the loss of opportunity
and waste of effort thereby involved. His rejection in 1895,
therefore, occasioned him no very great concern. Whatever feeling
of disappointment he may have felt soon passed away, and he
quickly went back to his old occupations, and to pursuits more
congenial to him than haunting the precincts of the House of
Commons. The following letter from Woodcote, under date July 20,
1895, affords some indication of the way in which he regarded the
loss of his seat.
 Many thanks for your kind note. As you surmise, I
do not feel personally much regret at my own defeat. I
could tell you something of the way the thing was
worked.
Now I feel an “old freeman,” and able to do much
more what I like. But this not always—for I do not see
my way just now to accept your invitation. I have been
worked up with the election, and have to be careful, so
that with this, and with the present uncertainty of
weather, I think I am safer on entire dry land.
We are thankful for rain which loveth the thirsty land
and makes things green again.
Is it true that the “burning bush” manufactures
C₂H₅OH [alcohol]? If so, that is really interesting.[27]
Harden and I have found some most interesting
results as regards the genesis of the atomic theory,
and I am going to work them up.
How about Davy?… The editor asks for more, and I
should be pleased to satisfy his maw by giving him a
lump of Davy. Kindest regards.

Shortly after his removal to London he became interested in the

sewage problem of the Metropolis, and was called upon to advise
the Metropolitan Board of Works with respect to methods for
improving the condition of the river Thames. In connection with this
work he established a laboratory, specially equipped for studying its
problems, in the Earl’s Court Road, not far from his London
residence. During the year 1887 he was engaged, with the
assistance of his former pupil Mr. Harry Baker, in reporting to Lord
Magheramorne, the Chairman of the Metropolitan Board of Works,
on the chemical methods employed for the deodorization of sewage
(a) in the metropolitan sewers, and (b) at the outfalls. Early in 1888
much larger problems were submitted to him: viz. the purification of
the sewage, the disposal of the sludge, and the effect of the
discharge of sewage sludge at sea on the foreshores of the estuary
of the Thames. In connection with these subjects he became
impressed with the importance of obtaining accurate scientific
methods for determining the changes which polluted water
experiences during its natural purification. Some of the results of his
inquiries he published in conjunction with his pupil Mr. Joseph Lunt
in two memoirs, one “On Schützenberger’s Process for the
Estimation of Dissolved Oxygen in Water,” communicated to the
Chemical Society in 1889, and published in the Transactions,[28] and
the other entitled “Contributions to the Chemical Bacteriology of
Sewage,” which appeared in the Philosophical Transactions of the
Royal Society.[29] The former paper contained the results of a
careful investigation of the conditions under which this method is
alone trustworthy, and served to explain the causes of the
discrepancy between the statements of previous observers who had
critically examined it. The latter paper gave the results of a
protracted examination of the chemical and bacteriological
phenomena of crude sewage with the object of ascertaining the
species of organisms present, both pathogenic and saprophytic, and
of determining their chemical characteristics.
These investigations were carried on for more than two years,
concurrently with the technical and outside work required. During
this time purification works had been established at Crossness and
Barking outfalls, a sludge ship had been provided for the disposal of
the sewage sludge at sea, and the effect of the discharge had been
studied in the lower reaches and estuary of the Thames, and a
chemical survey of the condition of the foreshores had been
completed. But the formation of the London County Council, with
Lord Rosebery as the first Chairman, involved new arrangements.
This circumstance, combined with the death of Sir Joseph
Bazalgette, the Chief Engineer, and the opposition of the Labour
Party, resulted in Roscoe resigning his post as Scientific Adviser.
 Mr. Lunt transferred his services to the British Institute of
Preventive Medicine, but Roscoe continued to carry on his laboratory
with the assistance of Mr. Frank Scudder until 1898. During this
period he acted as chemical adviser to sanitary authorities all over
the country on questions of sewage purification and water-supply;
and was frequently consulted by manufacturers on works-processes,
and on legal, patent, and trade-mark cases, and in connection with
parliamentary inquiries, e.g. humidity and purity of air in textile mills,
flashpoint of paraffin oils, etc. He was further concerned in the
promotion of Bills for the creation of rivers boards, as, for example,
those of the Mersey and Irwell and West Riding. He gave
considerable attention to the question of the manufacture and use of
water gas (“carburetted” and “blue” gas), and inspected most of the
water-gas plants then in operation in England and on the Continent,
sending Mr. Scudder to visit and report on the installations in the
principal cities of America. In 1898 they both gave evidence before a
Parliamentary Committee on the question of restricting the amount
of the poisonous carbonic oxide in town gas.
In 1891 Roscoe’s services were retained by the Mersey and Irwell
Joint Committee to report on the influence of the various
manufacturing works in the Mersey and Irwell basins in polluting the
streams, and as to the best means of preventing it. In 1893 the
Committee made the position of Scientific Adviser a permanent
appointment, and established a properly equipped laboratory in
Manchester in connection with its work. Roscoe retained the
appointment until 1905, when the frequent journeys to attend the
meetings of the Joint Committee began to tell upon his health, and
at his suggestion Mr. Scudder was appointed to succeed him. The
London laboratory was given up in 1908.
 CHAPTER XI
UNIVERSITY OF LONDON—ETON COLLEGE—
UNIVERSITY COLLEGE OF DUNDEE—SCOTTISH
UNIVERSITIES COMMISSION—ROYAL
COMMISSION OF THE 1851 EXHIBITION—
CARNEGIE TRUST: SCOTTISH UNIVERSITIES—
SCIENCE AND ART DEPARTMENT: SCIENCE
MUSEUMS—LISTER INSTITUTE OF
PREVENTIVE MEDICINE

Roscoe was long and honourably connected with the University of

London. A graduate in 1853, he acted as examiner in chemistry from
1874 to 1878. It was largely through his action that practical
laboratory work was included in the curriculum in chemistry for
science degrees. This not only greatly enhanced their status, but
reacted beneficially upon the general character of laboratory
instruction throughout the country. On relinquishing parliamentary
work he became a member of the Senate, and took part in the
movement for the reform of the University which led incidentally to
the formation of an association of teachers and others for the
promotion of a so-called Professorial University, of which Huxley was
President. The following letter refers to this circumstance:
 Manchester,
June 26, 1892.
I am delighted to hear that Huxley has joined and is
to be President of the Association. It will give me
pleasure to act as a Vice-President with Jebb.
Things look very well, and our views must greatly
influence the Royal Commission.
I will try to secure names here. My wife sends a list
per parcel post.

He made proposals with the idea of uniting what have come to be

called the Internal and External sides of the University, and in his
evidence before Lord Cowper’s Commission he suggested a
machinery of a less cumbrous and, as he hoped, of a more
satisfactory character than that which became law in 1898. In 1896
he succeeded Sir Julian Goldsmid as Vice-Chancellor. It was during
his term of office that the Act of 1898, which reconstituted the
University as the result of Earl Cowper’s Commission, was passed. As
Vice-Chancellor it became his duty to watch the progress of the
measure, and to use his influence in promoting its passage through
Parliament.
Unfortunately the University was as a house divided against itself.
One section of its Senate, numerically not very strong, was avowedly
hostile to its reconstitution as a teaching body. Some members of
Convocation acted as if their conception of the sole purpose of a
University was the holding of examinations and the giving of
degrees. Their object, apparently, was to strengthen by all possible
means the influence of Convocation; to make it, in fact, the main
controlling power. Accordingly, they used such parliamentary support
as they could command to wreck the Bill, or failing that, so to modify
its provisions as to preserve as far as possible the existing
constitution of the institution, and to perpetuate its restricted
functions. Thanks, however, to the action and alertness of Lord
Bryce, Lord Haldane, Sir W. Priestley, and Sir John Gorst, and the
firmness of the Government, the measure was steered safely
through Parliament and received the Royal Assent.
The statutory commission which followed the University of London
Act of 1898 reported in 1900; its provisions were approved by
Parliament in June of that year, and the new Senate held its first
meeting in the following October. Roscoe took an active share in the
rearrangements consequent on the reconstitution of the University,
and in the changes necessitated by its removal from Burlington
Gardens to the buildings of the Imperial Institute at South
Kensington.
This last step was a somewhat delicate matter. As housed in
Burlington Gardens the University was only moderately well provided
for as regards examination-rooms and administrative offices, but
such laboratories and store-rooms as it possessed were wholly
inadequate for the practical work required in the examinations for
science and medical degrees. The pressure on the limited space
grew more severe each session, and for some time previous to 1898
the necessity of making fresh provision had forced itself upon the
notice of the authorities. The wants of the University in this respect
had been freely ventilated in the course of the discussion on the Bill.
Accordingly, overtures were made to the Senate to take over some
portion of the building of the Imperial Institute as a home for the
reconstituted University. The offer was not received with any great
enthusiasm. The Imperial Institute had not fulfilled the anticipations
of its projectors; its associations, to say the least, were not
altogether academic, and this circumstance naturally created a
prejudice against it. Moreover, the building itself, although grandiose
in design, and possessing an admirable façade, was rather like the
geometrical definition of a line—length without breadth; and when
that portion of it intended to be assigned to the University was
measured up, it was actually not much, if any, larger in superficial
area than was available in Burlington Gardens. There was, however,
more space in the neighbourhood, and a certain amount of
rearrangement and new construction was possible. Moreover, the
authorities of the Science and Art Department were projecting new
laboratories for chemistry and physics, and it was hoped that
facilities might be granted to the University to enable them, under
certain conditions, to use them, or some portion of them, for their
practical examinations in those sciences. But objections were raised
in regard to the geographical position of the building, its distance
from the main-line stations, etc. Its possible association with what
was styled “the South Kensington clique” was another rock of
offence.
There were possible difficulties also with the Council of the
Institute as to the partition of the structure, use of the main
entrance, etc. But all these matters were adjusted eventually by the
skill, tact, and firmness of the Vice-Chancellor, with the concurrence
of the Treasury and of the Office of Works; and the University
entered into the possession of the eastern half of the building.
Not the least of the services which Roscoe rendered to the
University was his action with regard to the selection of the late Sir
Arthur Rücker as its first Principal. It was entirely through his efforts
that the appointment was made. Its success, he says in his “Life and
Experiences,” more than justified those efforts, and he always spoke
of it as the best day’s work he ever did for the University.
Roscoe resigned the Vice-Chancellorship in 1902, when he
presented to the University the handsome mace which now lies on
the table during the meetings of the Senate, and which is used on
ceremonial occasions. It was so employed, draped in crape, at the
memorial service held in Rosslyn Hill Chapel at his death.
He remained a member of the Senate until 1910, when age and
increasing deafness necessitated his retirement. In the resolution of
condolence which the Senate passed at its first meeting after his
death, they recalled with gratitude and admiration the great services
he had rendered to the University during the twenty-one years of his
membership of the Senate, at first as Fellow and later as one of the
representatives of His Majesty in Council; and especially the wisdom,
born of long experience in academic administration, with which he
guided the University during the six years of his Vice-Chancellorship,
which witnessed its reconstitution under the Act of 1898.

Roscoe was a Fellow of Eton College, as a representative of the

Royal Society, from 1889 to 1912, and did what he could during the
twenty-three years he served on the Governing Body to overcome
what he terms “the enormous inertia of this ancient machine.” He
sought to further the teaching of physical science in the School by
himself giving lectures, and through his efforts it is the richer by no
less than one large and one small physical laboratory, a physics
lecture-room, a workshop, and two more chemical laboratories, with
store-rooms, etc. He also reorganized the system of teaching, and
introduced graduated courses, which have resulted in an all-round
improvement. Nevertheless, the results have not been
commensurate with all the hard work and enthusiasm he put into his
efforts. They have been largely discounted by factors over which he
had no control. No one realized this more clearly than himself, and
he felt keenly the disappointment of his hopes, so much so that
more than once he considered the advisability of resigning his
Fellowship as a protest. He earned the gratitude of the science staff
by his uniform kindness and sympathy, and by the readiness with
which he would discuss their difficulties with them and help them
with advice and encouragement.
As the representative of the Royal Society, his chief interests lay
with the teaching of physical science, but they did not rest there. No
Fellow worked harder for the general welfare of the School. In order
to make himself acquainted first-hand with facts connected with the
subjects to be discussed at the meetings of the Provost and Fellows,
he constantly visited Eton. His opinion and advice on all sorts of
questions were sought and respected, and he has left behind him a
record of whole-hearted service to the School that will long be
remembered with appreciation and gratitude.
 Roscoe’s experience as an educationist, and his success in
furthering the development of Owens College, naturally caused him
to be consulted when institutions of a similar type were projected,
and he was occasionally induced to take part in their foundation and
government. Thus he had a large share in the arrangement of the
curriculum of the University College of Dundee in 1881, and he was
afterwards concerned, as a member of the Scottish Universities
Commission, in establishing the connection of that College with the
University of St. Andrews. He was appointed by the Duke of
Devonshire to a governorship of University College, Liverpool, a
position particularly gratifying to him as a member of a distinguished
Liverpool family. He represented the University of London on the
Council of Firth College, Sheffield, which has since risen to the
dignity of a university, and was of service with respect to its science
curriculum.
In 1888 Roscoe acted as a member of an Executive Commission
appointed to carry out the provisions of the Scottish Universities Act.
The Commission succeeded in devising ordinances which in many
respects revolutionized the systems of the Scottish Universities, and
by providing new avenues to degrees are destined, it may be hoped,
to have an important effect upon the character of scientific
education in Scotland.

In 1890 he was appointed a member of a Committee along with

Lord Playfair, Lord Kelvin, Professor Huxley, Mr. Mundella, Sir Norman
Lockyer, and Dr. William Garnett, to advise the Commissioners of the
1851 Exhibition on the question of establishing scholarships to aid
the development of scientific education in the manufacturing districts
of the country.
Roscoe remained a member of that body after it had presented its
Report and had been entrusted with the duty of putting into
operation the scheme which had been devised for the distribution
and regulation of the scholarships. The character of these
scholarships cannot be better described than in his own words:—
 It was decided that these should be of a higher
order than most of those existing; in fact, that their
functions should begin where the ordinary educational
curriculum ends, this system having been adopted with
excellent effect in the French École Pratique des
Hautes Études. In other words, the scholarships were
to be entirely confined to research, and strict
conditions were laid down as to the capability of the
candidate to carry out original investigation. The
scholarships were to be £150 a year in value, tenable
for two years, and to be limited to those branches of
science the knowledge of which is specially important
for our national industries. The Commissioners from
time to time were to select a number of institutions
throughout the Empire in which high scientific
instruction is given. Each university or college was to
have the power of nominating a student to a
scholarship on the condition that he exhibited a
capacity for advancing science or its applications;
these scholarships, when awarded, were to be tenable
in any university, either at home or abroad, or other
approved institutions.…
The fact that a large number of the nominating
institutions are situated in the colonies, and that the
scholars from these institutions come to the mother-
country to carry out their scientific work, must exert an
important influence in strengthening the relations
between different parts of the Empire. The value of
this system is fully appreciated by the authorities at
home and abroad, and many are the instances in
which men of ability, who would otherwise have been
unable to follow a scientific career, or to assist the
progress of our national industries, are now coming to
the front in both respects. It has been well remarked
that if, in the course of a century, even one Faraday
 should have been discovered, the sum spent would
have been amply repaid.

The policy laid down by the original committee has since been
pursued without essential variation, and what was at first regarded
as an educational experiment has proved itself by the test of time an
entirely successful undertaking, and one which has served as a
model for the institution of similar foundations, both in this country
and abroad.
In recognition of the services rendered on this committee, Roscoe
was elected, in 1891, a member of the Royal Commission, and five
years afterwards, in 1896, he became a member of the Board of
Management, and at the same time succeeded Lord Playfair as
Chairman of the Scholarships’ Committee.
As Chairman, the control and direction of the Committee’s work
was very largely in his hands, and the care he devoted to every
detail of the scholarship work undoubtedly contributed to the
successful operation of the scheme.

But there was something more to value (says Mr.

Evelyn Shaw, who contributes the above particulars)
than the part he played in the proceedings of the
Committee. The charm and sympathy of his
personality were felt by so many scholars who had
occasion to consult him upon their work, and who
often afterwards remembered and were grateful for
some kind and helpful advice. He never failed to watch
with interest the careers of past scholars, as he
regarded their record as the most convincing proof of
the value of the Commissioners’ Endowment.

In 1901 he consented, on the invitation of Lord Elgin, the

Chairman, to join the executive committee of the trustees appointed
to carry out the administration of Mr. Andrew Carnegie’s munificent
gift to the Scottish Universities for the benefit of scientific education;
and he assisted in the inauguration of a system of Carnegie
Scholarships and Fellowships for the encouragement of original
investigation, resembling that of the Royal Commissioners of the
1851 Exhibition.

Roscoe acted as chief examiner in chemistry of the Science and

Art Department in succession to the late Sir Edward Frankland, but
resigned the appointment on his election to Parliament. He took a
great interest in the aims of the department, and worked cordially
with its administrative officers, especially in the abolition of the old
system of “payment on results” for the elementary stage of science
subjects, and in remodelling organized science schools.
The importance of properly housing the valuable science
collections at South Kensington was constantly being pressed by him
upon the Government. In 1909 he accompanied a strong deputation,
and presented an influentially signed memorial to the Board of
Education, pleading for larger and better accommodation for the
unique and almost priceless exhibits of historically important objects
which the museum possesses, some of which are absolutely
irreplaceable. He pointed out how valuable such a collection was as
an adjunct to the systematic teaching of science and technology.
Each model, or piece of apparatus, or specimen of historic interest,
was selected to bring into prominence underlying principles, or to
illustrate various stages of industrial progress. In the temporary
buildings in which the collections were placed, there was not only no
room for the necessary expansion, but the objects were so crowded
together that proper arrangement and inspection were impossible.
What was needed was a building adequate to the proper exhibition
of the present collection, and one worthy of British Science. He
pointed out that one consequence of storing the collections in so
haphazard and unsatisfactory a manner was that persons possessing
objects of interest naturally felt indisposed to present them to the
nation, and some of these when offered had to be refused through
want of space. Land sufficient for the purpose was in the hands of
the Government, and the Royal Commissioners for the 1851
Exhibition, so long ago as 1878, offered to contribute £100,000
towards a building for the Science Museum. Roscoe’s arguments
were strongly supported by other members of the deputation, and
Mr. Runciman, who was then at the Board of Education, expressed
himself as convinced by their weight, and as wholly in sympathy with
the object of the speakers. A gratifying result of this action is to be
seen in the new buildings now in course of erection.

Roscoe’s high appreciation of Pasteur’s work as a chemist was, we

may presume, the immediate cause of the great interest with which
he had followed his remarkable discoveries concerning the causes
and cure of chicken cholera, anthrax, and the silkworm disease—an
interest quickened, no doubt, by the fact that he had made the
personal acquaintance of that distinguished man as far back as the
early ’sixties. He had specially informed himself of the working of the
Institut Pasteur in Paris, and of the anti-rabic treatment, and had
borne his share in combating the mischievous prejudices of those in
this country who sought to misrepresent the character and objects
of Pasteur’s work. In 1886 he had used his parliamentary influence
to induce Mr. Chamberlain, who was then President of the Local
Government Board, to appoint a Government Commission, consisting
of the late Lord Lister, Sir James Paget, Professor Ray Lankester, and
himself, with Sir Victor Horsley as secretary, to inquire and report on
the efficacy of Pasteur’s treatment of hydrophobia. The Commission
came to the conclusion, based upon irrefragable proof, that this
treatment had been the means of saving a large number of lives that
otherwise would have been sacrificed to a dreadful and torturing
death. Their report induced Sir James Whitehead, when Lord Mayor
of London, to call a Mansion House meeting for the purpose of
raising a fund partly to defray the cost of sending poor persons, who
may need treatment, to the Pasteur Institute in Paris, and partly to
repay some of our indebtedness to Pasteur and his co-workers for
having treated some two hundred of our countrymen gratuitously.
The Royal Society requested Roscoe, with Sir James Paget and
Professor Lankester, to represent them at the Lord Mayor’s meeting,
and they supported the action by a formal letter from the President.
Roscoe seized the opportunity of having to respond for “Science” at
a Royal Academy banquet to direct further attention to the subject,
and he subsequently spoke in the House of Commons of the great
value of experiments on living animals in opposition to an
amendment designed to impede the working of the Vivisection Acts.
By memorials, popular lectures, and articles in the periodical press,
he kept the subject continually before the public eye. Nor were his
colleagues less active in instructing and forming public opinion. Their
efforts eventually resulted in the establishment of an institute in
London with aims similar to those of that in Paris. Thanks to the
munificent action of Lord Iveagh, it has been housed and equipped
not less worthily than its sister foundation. The London Institute of
Preventive Medicine now bears the honoured name of Lord Lister, its
first President.
Roscoe was its Treasurer from 1891 to 1904 and Chairman from
1904 to 1912, and again from 1914 to the time of his death. The
building has now been completed at a cost of £28,000, entirely paid
out of income, and there has been a gradual and considerable
increase in the scientific staff and in the volume of work done. The
formation of the Medical Research Committee was thought by
Roscoe to affect the interests of the Institute, and he considered
that it might be better to bring about a working arrangement
between the two bodies. He felt that the independent existence of
two such schemes of research might lead to rivalry rather than to
co-operation, and that the superior resources of the Government
Committee might operate to the disadvantage of the Institute. It
was also thought that the addition of the resources of the Institute
to those at the disposal of the Committee together with the union of
the two scientific staffs would prove a great advantage to each and
contribute largely to the success of both. Another more practical
point was that the amalgamation scheme would remove from the
Institute the burdensome necessity of having to earn money by
routine diagnosis work in order to provide a sufficient income to
support the scientific work, and to permit of its increase. This
contemplated action gave rise to a considerable difference of
opinion. As a question of policy it obviously admitted of two sides,
and when the matter came up for decision the preponderating
feeling was to let well alone and to allow the Institute to continue to
develop along independent lines.
 CHAPTER XII
DIGNITIES AND HONOURS—THE DEUTSCHE
REVUE—GERMANY AND ENGLAND—WORLD
SUPREMACY OR WAR

Roscoe’s services to science and to the cause of education were

widely recognized. He was an honorary graduate of many
universities at home and abroad, and an honorary or corresponding
member of many foreign scientific societies. He was a D.C.L. of
Oxford; and LL.D. of Cambridge (1883), Dublin (1878), Glasgow
(1901), and Montreal (1884); and D.Sc. of Aberdeen, Liverpool, and
Victoria. On the occasion of the eighth jubilee of the foundation of
Heidelberg University he was made an honorary M.D.
He served as a member of the jury for chemical products of the
English section of the French Exhibition of 1878, and was made an
officer of the Legion of Honour, and in 1889 a corresponding
member of the French Institute of the Academy of Sciences. He was
an honorary member of the American Philosophical Society of
Philadelphia and of the New York Academy of Sciences; of the
Chemical Society of Berlin; of the Bunsen Gesellschaft, of the Verein
für Naturwissenschaft of Brunswick, and of the Physikalische Verein
of Frankfort; a corresponding member of the Bavarian Academy of
Sciences of Munich, of the Royal Society of Sciences of Göttingen, of
the Royal Accad. Lincei of Rome, and of the Academy of Natural
Science of Catania; a member of the Leop. Carol. Akad. of Halle, and
of the Physiogr. Sällsk of Lund. He was an honorary member of the
Royal Irish Academy, and of the Literary and Philosophical Society of
Manchester. In 1912 the Franklin Institute awarded him the Elliott
Cresson Gold Medal.
 He was sworn of the Privy Council in 1909—an honour which he
accepted not only as a personal distinction but as a recognition of
the claims of Science.
But of all the distinctions and marks of appreciation he received in
the course of his long and busy life, none afforded him a truer or
more heart-felt gratification than the action taken by his old pupils in
celebrating the fiftieth anniversary of the date—March 25, 1854—on
which he took his Heidelberg degree. The warmth and cordiality with
which the idea of commemorating his jubilee was received, not only
by his former students but by every teaching institution, academic
body, and scientific society with which he was or had been
connected, was a striking testimony to the regard and esteem in
which he was universally held. The University of Heidelberg renewed
its diploma of Doctor Philosophiæ Naturalis, and accompanied it by
an address from the Grand Duke of Baden, Rector of the University,
the Pro-Rector, Senior Dean, and the other professors of the
Philosophical Faculty. Addresses were also sent by University College,
London, the Victoria University, and the Universities of London,
Liverpool, Birmingham, Wales, Scotland, Montreal, Melbourne, New
Zealand, and Tokyo; King’s College, London, the Yorkshire College,
and the University Colleges of Sheffield, Newcastle, and Dundee; the
Royal and Chemical Societies, the Society of Chemical Industry,
British Association, the Lister and Pasteur Institutes, and a number
of the academies and scientific societies of Germany, Italy, Holland,
and America. In addition a large number of congratulatory letters
and messages were received from distinguished friends, chemists,
and physicists throughout Europe, America, and the British
Dominions beyond the Seas.
The celebration was held on April 22, 1904, in the beautiful
Whitworth Hall of the Victoria University in the presence of a large
and enthusiastic gathering of former students and colleagues, and of
friends who had journeyed to Manchester to present addresses.
Of all these addresses, the one, he says in his autobiography, that
touched him most nearly was that from his former pupils. It ran as