Python High Performance Programming Boost the performance of your Python programs using advanced techniques 1st Edition Gabriele Lanaro download
Python High Performance Programming Boost the performance of your Python programs using advanced techniques 1st Edition Gabriele Lanaro download
https://ebookname.com/product/python-high-performance-2nd-
edition-gabriele-lanaro/
https://ebookname.com/product/fundamentals-of-python-first-
programs-1st-edition-kenneth-a-lambert/
https://ebookname.com/product/beginning-python-using-
python-2-6-and-python-3-1-wrox-programmer-to-programmer-james-
payne/
https://ebookname.com/product/religion-and-politics-in-the-
middle-east-identity-ideology-institutions-and-attitudes-second-
edition-edition-robert-d-lee/
Gilbert Law Summary on Criminal Procedure 19th ed.
Edition Paul Marcus
https://ebookname.com/product/gilbert-law-summary-on-criminal-
procedure-19th-ed-edition-paul-marcus/
Dream Lucky When FDR was in the White House Count Basie
was on the radio and everyone wore a hat 1st Edition
Roxane Orgill
https://ebookname.com/product/dream-lucky-when-fdr-was-in-the-
white-house-count-basie-was-on-the-radio-and-everyone-wore-a-
hat-1st-edition-roxane-orgill/
https://ebookname.com/product/making-sense-of-the-dollar-
exposing-dangerous-myths-about-trade-and-foreign-exchange-1st-
edition-chandler/
https://ebookname.com/product/constitutional-and-administrative-
law-helen-fenwick/
https://ebookname.com/product/the-syriac-bible-according-to-the-
mosul-edition-yusuf-dawud/
The Bible in the Syriac Tradition Third Edition
Sebastian Brock
https://ebookname.com/product/the-bible-in-the-syriac-tradition-
third-edition-sebastian-brock/
Python High Performance
Programming
Gabriele Lanaro
BIRMINGHAM - MUMBAI
Python High Performance Programming
All rights reserved. No part of this book may be reproduced, stored in a retrieval
system, or transmitted in any form or by any means, without the prior written
permission of the publisher, except in the case of brief quotations embedded in
critical articles or reviews.
Every effort has been made in the preparation of this book to ensure the accuracy
of the information presented. However, the information contained in this book is
sold without warranty, either express or implied. Neither the author, nor Packt
Publishing, and its dealers and distributors will be held liable for any damages
caused or alleged to be caused directly or indirectly by this book.
Packt Publishing has endeavored to provide trademark information about all of the
companies and products mentioned in this book by the appropriate use of capitals.
However, Packt Publishing cannot guarantee the accuracy of this information.
ISBN 978-1-78328-845-8
www.packtpub.com
Reviewers Proofreader
Daniel Arbuckle Linda Morris
Mike Driscoll
Albert Lukaszewski Indexer
Rekha Nair
Acquisition Editors
Owen Roberts Production Coordinators
Pooja Chiplunkar
Harsha Bharwani
Manu Joseph
Commissioning Editor
Shaon Basu Cover Work
Pooja Chiplunkar
Technical Editors
Akashdeep Kundu
Faisal Siddiqui
About the Author
Mike Driscoll has been programming in Python since Spring 2006. He enjoys
writing about Python on his blog at http://www.blog.pythonlibrary.org/.
Mike also occasionally writes for the Python Software Foundation, i-Programmer,
and Developer Zone. He enjoys photography and reading a good book. Mike has
also been a technical reviewer for Python 3 Object Oriented Programming, Python
2.6 Graphics Cookbook, and Tkinter GUI Application Development Hotshot.
Did you know that Packt offers eBook versions of every book published, with PDF
and ePub files available? You can upgrade to the eBook version at www.PacktPub.
com and as a print book customer, you are entitled to a discount on the eBook copy.
Get in touch with us at service@packtpub.com for more details.
At www.PacktPub.com, you can also read a collection of free technical articles, sign
up for a range of free newsletters and receive exclusive discounts and offers on Packt
books and eBooks.
TM
http://PacktLib.PacktPub.com
Why Subscribe?
• Fully searchable across every book published by Packt
• Copy and paste, print and bookmark content
• On demand and accessible via web browser
[ ii ]
Preface
Python is a programming language renowned for its simplicity, elegance, and
the support of an outstanding community. Thanks to the impressive amount
of high-quality third-party libraries, Python is used in many domains.
Python is an easier language to deal with and it can be used to quickly write
complex applications. Thanks to its tight integration with C, Python is able to
avoid the performance drop associated with dynamic languages. You can use
blazing fast C extensions for performance-critical code and retain all the
convenience of Python for the rest of your application.
In this book, you will learn, in a step-by-step method how to find and speedup
the slow parts of your programs using basic and advanced techniques.
The style of the book is practical; every concept is explained and illustrated with
examples. This book also addresses common mistakes and teaches how to avoid
them. The tools used in this book are quite popular and battle-tested; you can be
sure that they will stay relevant and well-supported in the future.
This book starts from the basics and builds on them, therefore, I suggest you
to move through the chapters in order.
Chapter 2, Fast Array Operations with NumPy is a guide to the NumPy package.
NumPy is a framework for array calculations in Python. It comes with a clean
and concise API, and efficient array operations.
The book was written and tested on Ubuntu 13.10. The examples will likely run on
Mac OS X with little or no changes.
[2]
Preface
However, the scope of this book is broad and the concepts can be applied to any
domain. Since the book addresses both basic and advanced topics, it contains
useful information for programmers with different Python proficiency levels.
Conventions
In this book, you will find a number of styles of text that distinguish between
different kinds of information. Here are some examples of these styles, and an
explanation of their meaning.
Code words in text, database table names, folder names, filenames, file extensions,
pathnames, dummy URLs, user input, and Twitter handles are shown as follows:
"The plot function included in matplotlib can display our particles as points
on a Cartesian grid and the FuncAnimation class can animate the evolution of
our particles over time."
def visualize(simulator):
[3]
Preface
When we wish to draw your attention to a particular part of a code block, the
relevant lines or items are set in bold:
In [1]: import purepy
In [2]: %timeit purepy.loop()
100 loops, best of 3: 8.26 ms per loop
In [3]: %timeit purepy.comprehension()
100 loops, best of 3: 5.39 ms per loop
In [4]: %timeit purepy.generator()
100 loops, best of 3: 5.07 ms per loop
New terms and important words are shown in bold. Words that you see on the
screen, in menus or dialog boxes, for example, appear in the text like this: "You
can navigate to the Call Graph or the Caller Map tabs by double-clicking on the
rectangles."
Reader feedback
Feedback from our readers is always welcome. Let us know what you think about
this book—what you liked or may have disliked. Reader feedback is important for
us to develop titles that you really get the most out of.
If there is a topic that you have expertise in and you are interested in either writing
or contributing to a book, see our author guide on www.packtpub.com/authors.
[4]
Preface
Customer support
Now that you are the proud owner of a Packt book, we have a number of things to
help you to get the most from your purchase.
Errata
Although we have taken every care to ensure the accuracy of our content, mistakes
do happen. If you find a mistake in one of our books—maybe a mistake in the text or
the code—we would be grateful if you would report this to us. By doing so, you can
save other readers from frustration and help us improve subsequent versions of this
book. If you find any errata, please report them by visiting http://www.packtpub.
com/submit-errata, selecting your book, clicking on the errata submission form link,
and entering the details of your errata. Once your errata are verified, your submission
will be accepted and the errata will be uploaded on our website, or added to any list of
existing errata, under the Errata section of that title. Any existing errata can be viewed
by selecting your title from http://www.packtpub.com/support.
Piracy
Piracy of copyright material on the Internet is an ongoing problem across all media.
At Packt, we take the protection of our copyright and licenses very seriously. If you
come across any illegal copies of our works, in any form, on the Internet, please
provide us with the location address or website name immediately so that we can
pursue a remedy.
We appreciate your help in protecting our authors, and our ability to bring you
valuable content.
Questions
You can contact us at questions@packtpub.com if you are having a problem with
any aspect of the book, and we will do our best to address it.
[5]
Benchmarking and Profiling
Recognizing the slow parts of your program is the single most important task when it
comes to speeding up your code. In most cases, the bottlenecks account for a very small
fraction of the program. By specifically addressing those critical spots you can focus on
the parts that need improvement without wasting time in micro-optimizations.
You may also want to assess the total execution time of your program and see how
it is affected by your changes. We will learn how to write benchmarks and how to
accurately time your programs.
In the early development stages, the design of the program can change quickly,
requiring you to rewrite and reorganize big chunks of code. By testing different
prototypes without bothering about optimizations, you learn more about your
program, and this will help you make better design decisions.
Benchmarking and Profiling
The mantras that you should remember when optimizing your code, are as follows:
• Make it run: We have to get the software in a working state, and be sure that
it produces the correct results. This phase serves to explore the problem that
we are trying to solve and to spot major design issues in the early stages.
• Make it right: We want to make sure that the design of the program is solid.
Refactoring should be done before attempting any performance optimization.
This really helps separate the application into independent and cohesive
units that are easier to maintain.
• Make it fast: Once our program is working and has a good design we want
to optimize the parts of the program that are not fast enough. We may also
want to optimize memory usage if that constitutes an issue.
In this section we will profile a test application—a particle simulator. The simulator
is a program that takes some particles and evolves them over time according to a
set of laws that we will establish. Those particles can either be abstract entities or
correspond to physical objects. They can be, for example, billiard balls moving on
a table, molecules in gas, stars moving through space, smoke particles, fluids in a
chamber, and so on.
Those simulations are useful in fields such as Physics, Chemistry, and Astronomy,
and the programs used to simulate physical systems are typically performance-
intensive. In order to study realistic systems it's often necessary to simulate the
highest possible number of bodies.
In our first example, we will simulate a system containing particles that constantly
rotate around a central point at various speeds, like the hands of a clock.
The necessary information to run our simulation will be the starting positions of
the particles, the speed, and the rotation direction. From these elements, we have
to calculate the position of the particle in the next instant of time.
[8]
Chapter 1
(vx, vy)
(x, y)
(0, 0)
The basic feature of a circular motion is that the particles always move
perpendicularly to the direction connecting the particle and the center, as shown in
the preceding image. To move the particle we simply change the position by taking a
series of very small steps in the direction of motion, as shown in the following figure:
[9]
Benchmarking and Profiling
Another class, called ParticleSimulator will encapsulate our laws of motion and
will be responsible for changing the positions of the particles over time. The __
init__ method will store a list of Particle instances and the evolve method will
change the particle positions according to our laws.
We want the particles to rotate around the point (x, y), which, here, is equal to (0, 0),
at constant speed. The direction of the particles will always be perpendicular to the
direction from the center (refer to the first figure of this chapter). To find this vector
v=(vx ,vy)
(corresponding to the Python variables v_x and v_y) it is sufficient to use these
formulae:
If we let one of our particles move, after a certain time dt, it will follow a circular path,
reaching another position. To let the particle follow that trajectory we have to divide
the time interval dt into very small time steps where the particle moves tangentially
to the circle. The final result, is just an approximation of a circular motion and, in fact,
it's similar to a polygon. The time steps should be very small, otherwise the particle
trajectory will diverge quickly, as shown in the following figure:
[ 10 ]
Chapter 1
In a more schematic way, to calculate the particle position at time dt we have to carry
out the following steps:
for i in range(nsteps):
for p in self.particles:
p.x += d_x
p.y += d_y
# 3. repeat for all the time steps
[ 11 ]
Benchmarking and Profiling
We can use the matplotlib library to visualize our particles. This library is not
included in the Python standard library. To install it, you can follow the instructions
included in the official documentation at:
http://matplotlib.org/users/installing.html
The plot function included in matplotlib can display our particles as points on
a Cartesian grid and the FuncAnimation class can animate the evolution of our
particles over time.
The visualize function accomplishes this by taking the particle simulator and
displaying the trajectory in an animated plot.
• Setup the axes and display the particles as points using the plot function
• Write an initialization function (init) and an update function
(animate) that changes the x, y coordinates of the data points using the
line.set_data method
• Create a FuncAnimation instance passing the functions and some parameters
• Run the animation with plt.show()
def visualize(simulator):
fig = plt.figure()
ax = plt.subplot(111, aspect='equal')
line, = ax.plot(X, Y, 'ro')
# Axis limits
plt.xlim(-1, 1)
plt.ylim(-1, 1)
[ 12 ]
Chapter 1
def animate(i):
# We let the particle evolve for 0.1 time units
simulator.evolve(0.01)
X = [p.x for p in simulator.particles]
Y = [p.y for p in simulator.particles]
line.set_data(X, Y)
return line,
simulator = ParticleSimulator(particles)
visualize(simulator)
if __name__ == '__main__':
test_visualize()
We need a test that checks whether the results produced by the simulation are
correct or not. In the optimization process we will rewrite the code to try different
solutions; by doing so we may easily introduce bugs. Maintaining a solid test suite
is essential to avoid wasting time on broken code.
[ 13 ]
Benchmarking and Profiling
Our test will take three particle and let the system evolve for 0.1 time units. We
then compare our results, up to a certain precision, with those from a reference
implementation:
def test():
particles = [Particle( 0.3, 0.5, +1),
Particle( 0.0, -0.5, -1),
Particle(-0.1, -0.4, +3)]
simulator = ParticleSimulator(particles)
simulator.evolve(0.1)
if __name__ == '__main__':
test()
We also want to write a benchmark that can measure the performance of our
application. This will provide an indication of how much we have improved
over the previous implementation.
def benchmark():
particles = [Particle(uniform(-1.0, 1.0),
uniform(-1.0, 1.0),
uniform(-1.0, 1.0))
for i in range(1000)]
simulator = ParticleSimulator(particles)
simulator.evolve(0.1)
if __name__ == '__main__':
benchmark()
[ 14 ]
Chapter 1
The time command is not available for Windows, but can be found
in the cygwin shell that you can download from the official website
http://www.cygwin.com/.
• real: The actual time spent in running the process from start to finish, as if it
was measured by a human with a stopwatch
• user: The cumulative time spent by all the CPUs during the computation
• sys: The cumulative time spent by all the CPUs during system-related tasks
such as memory allocation
Notice that sometimes user + sys might be greater than real, as multiple processors
may work in parallel.
time also offers several formatting options; for an overview you can
explore its manual (by using the man time command). If you want a
summary of all the metrics available, you can use the -v option.
The Unix time command is a good way to benchmark your program. To achieve
a more accurate measurement, the benchmark should run long enough (in the
order of seconds) so that the setup and tear-down of the process become small,
compared to the execution time. The user metric is suitable as a monitor for the
CPU performance, as the real metric includes also the time spent in other
processes or waiting for I/O operations.
Another useful program to time Python scripts is the timeit module. This module
runs a snippet of code in a loop for n times and measures the time taken. Then, it
repeats this operation r times (by default the value of r is 3) and takes the best of
those runs. Because of this procedure, timeit is suitable to accurately time small
statements in isolation.
[ 15 ]
Benchmarking and Profiling
The timeit module can be used as a Python module, from the command line, or
from IPython.
IPython is a Python shell designed for interactive usage. It boosts tab completion and
many utilities to time, profile, and debug your code. We will make use of this shell to
try out snippets throughout the book. The IPython shell accepts magic commands—
statements that start with a % symbol—that enhance the shell with special behaviors.
Commands that start with %% are called cell magics, and these commands can be
applied on multi-line snippets (called cells).
In the following code we show how to use timeit from IPython, from the command
line and as a Python module:
# IPython Interface
$ ipython
In [1]: from simul import benchmark
In [2]: %timeit benchmark()
1 loops, best of 3: 782 ms per loop
# Python Interface
# put this function into the simul.py script
import timeit
result = timeit.timeit('benchmark()',
setup='from __main__ import
benchmark', number=10)
# result is the time (in seconds) to run the whole loop
[ 16 ]
Random documents with unrelated
content Scribd suggests to you:
rough stone slab upon which the offenders were seated still remains
in place.
Stroud is a larger and better-appearing town, whose ten thousand
inhabitants depend mainly upon the manufacture of English
broadcloths. The whole section, in fact, was once the center of cloth
manufacture, but the advent of the steam engine and more modern
methods superseded the watermills. All about are half-ruined factory
buildings, some of them once of vast extent, with shattered windows
and sagging roofs. Here and there one has survived in a small way
or has been adapted to some other industry. In the neighborhood
are many country houses, once the residences of wealthy cloth-
makers, but now either deserted or turned into farm tenements.
The country is hilly and wooded, and we had few points of vantage
that afforded views more picturesque and far-reaching than from
some of the upland roads overlooking these Gloucestershire
landscapes. The road sweeps around the hills, rising at times far
above the valleys, affording a panorama of the Avon gleaming
through the dense green foliage that half conceals it. The vale
presents the most charming characteristics of rural England. One
sees the irregular patchwork of the little fields, the great parks with
their sunny meadowlands and groups of ancient trees, the villages
lying in the valleys or clinging to the hillsides, and the gray church
towers that lend a touch of majesty and solemn sentiment to almost
every glimpse of Britain.
We missed the main road from Bath to Wells, wandering through a
maze of unmarked byroads, and were able to proceed only by
frequent inquiry. We did not regain the highway until just entering
the town and had been a comparatively long time in going a short
distance. After a few minutes’ pause to admire the marvelous west
front of the cathedral, with its endless array of crumbling prophets,
saints and kings, weatherworn to a soft-gray blur, we were away on
the highroad leading across the wold to Cheddar, famed for its
stupendous cliffs, its caverns—and its cheese. The caverns and cliffs
are there, but little cheese now comes from Cheddar, even though it
bears the name. As we ascended the exceedingly steep and winding
road we were astonished—overwhelmed. We had not expected to
find natural scenery upon such an amazing scale in the heart of
England—gray pinnacled cliffs rising, almost sheer, five hundred feet
into the sky. Not often may British scenery be styled imposing, but
the towering cliffs of Cheddar surely merit such description. In the
midst of the gorge between the great cliffs there are two prehistoric
caverns extending far into the earth. We entered one of them, now a
mere passageway, now a spacious cavern whose domelike roof
glistens with translucent stalactites. Here we pass a still, mirrorlike
pool, and there a deep fissure from which comes the gurgle of a
subterranean river. Altogether, there is much that is interesting and
impressive. Perhaps it all seems a little gaudy and unnatural because
of the advertising methods and specious claims of the owner and
alleged discoverer, but none the less a visit is worth while. The
museum of relics found in the cavern contains a remarkable
prehistoric skull, with low, thick frontal bone and heavy square jaw,
but its queerest feature is little spurlike projections of the temporal
bone just above the ear. It is estimated by archaeologists that the
possessor of this curious skull had lived at least forty thousand years
ago and mayhap had made his dwelling-place in the Cheddar Caves.
We were assured that an offer from the British Museum of five
thousand pounds for the relic had been refused.
The sun was low when we left Cheddar, and Taunton seemed the
nearest place where we might be sure of good accommodations. We
soon reached Axbridge, a gray little market town, so ancient that a
hunting-lodge built by King John still stands on the market square.
Near Bridgewater, a few miles farther, is the Isle of Athelney—once
an island in a marsh, perhaps—where King Alfred made his last
desperate stand against the Danish invaders, defeating them and
finally expelling them from Britain. Not less in interest, though
perhaps less important in its issues, was the Battle of Sedgemoor,
fought here in 1685, when the Duke of Monmouth was disastrously
defeated by the Royal Army—the last battle worthy of the name to
be waged on English soil.
But we were to learn more of Monmouth at Taunton and to have
again impressed upon us how easy it is in Britain for one to hasten
through places of the deepest historic interest quite unaware of their
tragic story. We had passed through Taunton before, seeing little but
a staid old country town with a church tower of unmatched
gracefulness and dignified proportions; but Taunton’s tragic part in
the parliamentary wars and her fatal connection with “King
Monmouth” never occurred to us, if, indeed, we knew of it at all.
Taunton was strongly for the Parliament, but it was a storm center
and was taken and retaken until the iron hand of Fairfax crushed the
Royalists before its walls. Its record stood against it when the King
“came into his own again.” Its walls were leveled to the ground, its
charter taken away and many of its citizens thrown into prison.
Discontent and hatred of the Stuarts were so rampant that any
movement against their rule was welcomed by the Taunton Whigs,
though it is hard to see any consistency in the unreasoning support
they gave to the Duke of Monmouth—the son of Charles II. and one
of his many mistresses—in his pretensions to the throne occupied by
James II. Monmouth entered Taunton amidst the wildest
acclamations, and it was from the market square of the rebellious
town that he issued his proclamation assuming the title of King. He
was followed by an ill-organized and poorly equipped army of seven
thousand men, who were defeated by four thousand Royal Troops.
Then followed a reign of terror in Taunton. The commander of the
King’s forces hanged, without pretense of trial, many of his
prisoners, using the sign of the old White Hart Inn as a gallows.
Then came the Bloody Assizes, held by Jeffreys of infamous memory,
in the great hall of the castle. After trials no more than travesties of
brutal jests and savage cruelty, more than three hundred
Somersetshire men were sentenced, according to the terrible
customs of the time, to be “hanged, drawn and quartered,” and a
thousand were doomed to transportation. Here the active history of
Taunton may be said to have ended.
But Taunton has little to remind us of these dark and bloody times as
we glide through her fine old streets and draw up in front of the
London Hotel, where the host himself in evening dress welcomes us
at the door. Every attention is given us and The London certainly
deserves its official appointment by the Royal Automobile Club as
well as the double distinction accorded it by the infallible Baedeker.
It is one of the charming old-fashioned inns, such as perhaps
inspired the poet Shenstone with the sentiment expressed in his
well-known quatrain:
Our website is not just a platform for buying books, but a bridge
connecting readers to the timeless values of culture and wisdom. With
an elegant, user-friendly interface and an intelligent search system,
we are committed to providing a quick and convenient shopping
experience. Additionally, our special promotions and home delivery
services ensure that you save time and fully enjoy the joy of reading.
ebookname.com