Parallel Programming for Multicore and Cluster

Systems 3rd Edition Thomas Rauber pdf download

https://ebookmeta.com/product/parallel-programming-for-multicore-
and-cluster-systems-3rd-edition-thomas-rauber/

Download more ebook from https://ebookmeta.com

We believe these products will be a great fit for you. Click
the link to download now, or visit ebookmeta.com
to discover even more!

Data Parallel C++: Programming Accelerated Systems

Using C++ and SYCL 2nd Edition James Reinders

https://ebookmeta.com/product/data-parallel-c-programming-
accelerated-systems-using-c-and-sycl-2nd-edition-james-reinders/

Multicore and GPU programming An Integrated approach

2nd Gerassimos Barlas

https://ebookmeta.com/product/multicore-and-gpu-programming-an-
integrated-approach-2nd-gerassimos-barlas/

Multicore and GPU Programming An Integrated Approach

2nd Edition Gerassimos Barlas

https://ebookmeta.com/product/multicore-and-gpu-programming-an-
integrated-approach-2nd-edition-gerassimos-barlas/

The Micro Economy Today, 16th Edition Bradley Schiller

https://ebookmeta.com/product/the-micro-economy-today-16th-
edition-bradley-schiller/
Indigenous Peoples National Parks and Protected Areas A
New Paradigm Linking Conservation Culture and Rights
1st Edition Stan Stevens

https://ebookmeta.com/product/indigenous-peoples-national-parks-
and-protected-areas-a-new-paradigm-linking-conservation-culture-
and-rights-1st-edition-stan-stevens/

Collected Short Fiction 2022nd Edition Keith Roberts

https://ebookmeta.com/product/collected-short-fiction-2022nd-
edition-keith-roberts/

Exploring a Business Case for High Value Continuing

Professional Development Proceedings of a Workshop 1st
Edition And Medicine Engineering National Academies Of
Sciences
https://ebookmeta.com/product/exploring-a-business-case-for-high-
value-continuing-professional-development-proceedings-of-a-
workshop-1st-edition-and-medicine-engineering-national-academies-
of-sciences/

World Prehistory A Brief Introduction 10th Edition

Brian M. Fagan

https://ebookmeta.com/product/world-prehistory-a-brief-
introduction-10th-edition-brian-m-fagan/

The Tainted Course Sugarbury Falls Mystery 4 Diane

Weiner

https://ebookmeta.com/product/the-tainted-course-sugarbury-falls-
mystery-4-diane-weiner/
Messianism and Sociopolitical Revolution in Medieval
Islam 1st Edition Said Amir Arjomand

https://ebookmeta.com/product/messianism-and-sociopolitical-
revolution-in-medieval-islam-1st-edition-said-amir-arjomand/
Parallel Programming
Thomas Rauber • Gudula Rünger

Parallel Programming
for Multicore and Cluster Systems

Third Edition
Thomas Rauber Gudula Rünger
Lehrstuhl für Angewandte Informatik II Fakultät für Informatik
University of Bayreuth Chemnitz University of Technology
Bayreuth, Bayern, Germany Chemnitz, Sachsen, Germany

Second English Edition was a translation from the 3rd German language edition: Parallele
Programmierung (3. Aufl. 2012) by T. Rauber and G. Rünger, Springer-Verlag Berlin
Heidelberg 2000, 2007, 2012.

ISBN 978-3-031-28923-1 ISBN 978-3-031-28924-8 (eBook)

https://doi.org/10.1007/978-3-031-28924-8

© The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature Switzerland
AG 2010, 2013, 2023
This work is subject to copyright. All rights are solely and exclusively licensed by the Publisher, whether
the whole or part of the material is concerned, specifically the rights of reprinting, reuse of illustrations,
recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission or
information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar
methodology now known or hereafter developed.
The use of general descriptive names, registered names, trademarks, service marks, etc. in this publication
does not imply, even in the absence of a specific statement, that such names are exempt from the relevant
protective laws and regulations and therefore free for general use.
The publisher, the authors, and the editors are safe to assume that the advice and information in this book
are believed to be true and accurate at the date of publication. Neither the publisher nor the authors or
the editors give a warranty, expressed or implied, with respect to the material contained herein or for any
errors or omissions that may have been made. The publisher remains neutral with regard to jurisdictional
claims in published maps and institutional affiliations.

This Springer imprint is published by the registered company Springer Nature Switzerland AG
The registered company address is: Gewerbestrasse 11, 6330 Cham, Switzerland
Preface

Innovations in hardware architecture, such as hyper-threading or multicore proces-

sors, make parallel computing resources available for computer systems in differ-
ent areas, including desktop and laptop computers, mobile devices, and embedded
systems. However, the efficient usage of the parallel computing resources requires
parallel programming techniques. Today, many standard software products are al-
ready based on concepts of parallel programming to use the hardware resources of
multicore processors efficiently. This trend will continue and the need for parallel
programming will extend to all areas of software development. The application area
will be much larger than the area of scientific computing, which used to be the main
area for parallel computing for many years. The expansion of the application area
for parallel computing will lead to an enormous need for software developers with
parallel programming skills. Some chip manufacturers already demand to include
parallel programming as a standard course in computer science curricula. A more
recent trend is the use of Graphics Processing Units (GPUs), which may comprise
several thousands of cores, for the execution of compute-intensive non-graphics ap-
plications.
This book covers the new development in processor architecture and parallel
hardware. Moreover, important parallel programming techniques that are necessary
for developing efficient programs for multicore processors as well as for parallel
cluster systems or supercomputers are provided. Both shared and distributed address
space architectures are covered. The main goal of the book is to present parallel
programming techniques that can be used in many situations for many application
areas and to enable the reader to develop correct and efficient parallel programs.
Many example programs and exercises are provided to support this goal and to show
how the techniques can be applied to further applications. The book can be used as a
textbook for students as well as a reference book for professionals. The material of
the book has been used for courses in parallel programming at different universities
for many years.
This third edition of the English book on parallel programming is an updated
and revised version based on the second edition of this book from 2013. The three
earlier German editions appeared in 2000, 2007, and 2012, respectively. The update

V
VI Preface

of this new English edition includes an extended update of the chapter on com-
puter architecture and performance analysis taking new developments such as the
aspect of energy consumption into consideration. The description of OpenMP has
been extended and now also captures the task concept of OpenMP. The chapter
on message-passing programming has been extended and updated to include new
features of MPI such as extended reduction operations and non-blocking collec-
tive communication operations. The chapter on GPU programming also has been
updated. All other chapters also have been revised carefully.
The content of the book consists of three main parts, covering all areas of parallel
computing: the architecture of parallel systems, parallel programming models and
environments, and the implementation of efficient application algorithms. The em-
phasis lies on parallel programming techniques needed for different architectures.
The first part contains an overview of the architecture of parallel systems, includ-
ing cache and memory organization, interconnection networks, routing and switch-
ing techniques as well as technologies that are relevant for modern and future mul-
ticore processors. Issues of power and energy consumption are also covered.
The second part presents parallel programming models, performance models,
and parallel programming environments for message passing and shared memory
models, including the message passing interface (MPI), Pthreads, Java threads, and
OpenMP. For each of these parallel programming environments, the book intro-
duces basic concepts as well as more advanced programming methods and enables
the reader to write and run semantically correct and computationally efficient par-
allel programs. Parallel design patterns, such as pipelining, client-server, or task
pools are presented for different environments to illustrate parallel programming
techniques and to facilitate the implementation of efficient parallel programs for a
wide variety of application areas. Performance models and techniques for runtime
analysis are described in detail, as they are a prerequisite for achieving efficiency
and high performance. A chapter gives a detailed description of the architecture of
GPUs and also contains an introduction into programming approaches for general
purpose GPUs concentrating on CUDA and OpenCL. Programming examples are
provided to demonstrate the use of the specific programming techniques introduced.
The third part applies the parallel programming techniques from the second part
to representative algorithms from scientific computing. The emphasis lies on basic
methods for solving linear equation systems, which play an important role for many
scientific simulations. The focus of the presentation is the analysis of the algorithmic
structure of the different algorithms, which is the basis for a parallelization, and not
so much on mathematical properties of the solution methods. For each algorithm,
the book discusses different parallelization variants, using different methods and
strategies.
Many colleagues and students have helped to improve the quality of this book.
We would like to thank all of them for their help and constructive criticisms. For
numerous corrections we would like to thank Robert Dietze, Jörg Dümmler, Mar-
vin Ferber, Michael Hofmann, Ralf Hoffmann, Sascha Hunold, Thomas Jakobs,
Oliver Klöckner, Matthias Korch, Ronny Kramer, Raphael Kunis, Jens Lang, Isabel
Mühlmann, John O’Donnell, Andreas Prell, Carsten Scholtes, Michael Schwind,
Preface VII

and Jesper Träff. Many thanks to Thomas Jakobs, Matthias Korch, Carsten Scholtes
and Michael Schwind for their help with the program examples and the exercises.
We thank Monika Glaser and Luise Steinbach for their help and support with the
LATEX typesetting of the book. We also thank all the people who have been involved
in the writing of the three German versions of this book. It has been a pleasure work-
ing with the Springer Verlag in the development of this book. We especially thank
Ralf Gerstner for his continuous support.

Bayreuth and Chemnitz, January 2023 Thomas Rauber

Gudula Rünger
Contents

1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.1 Classical Use of Parallelism . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.2 Parallelism in Today’s Hardware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.3 Basic Concepts of parallel programming . . . . . . . . . . . . . . . . . . . . . . . 4
1.4 Overview of the Book . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

2 Parallel Computer Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

2.1 Processor Architecture and Technology Trends . . . . . . . . . . . . . . . . . . 10
2.2 Power and Energy Consumption of Processors . . . . . . . . . . . . . . . . . . 13
2.3 Memory access times . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
2.3.1 DRAM access times . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
2.3.2 Multithreading for hiding memory access times . . . . . . . . . . . 18
2.3.3 Caches for reducing the average memory access time . . . . . . 18
2.4 Flynn’s Taxonomy of Parallel Architectures . . . . . . . . . . . . . . . . . . . . 19
2.5 Memory Organization of Parallel Computers . . . . . . . . . . . . . . . . . . . . 21
2.5.1 Computers with Distributed Memory Organization . . . . . . . . 22
2.5.2 Computers with Shared Memory Organization . . . . . . . . . . . . 25
2.6 Thread-Level Parallelism . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
2.6.1 Simultaneous Multithreading . . . . . . . . . . . . . . . . . . . . . . . . . . 29
2.6.2 Multicore Processors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
2.6.3 Architecture of Multicore Processors . . . . . . . . . . . . . . . . . . . . 32
2.7 Interconnection Networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
2.7.1 Properties of Interconnection Networks . . . . . . . . . . . . . . . . . . 38
2.7.2 Direct Interconnection Networks . . . . . . . . . . . . . . . . . . . . . . . 41
2.7.3 Embeddings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
2.7.4 Dynamic Interconnection Networks . . . . . . . . . . . . . . . . . . . . . 49
2.8 Routing and Switching . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
2.8.1 Routing Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
2.8.2 Routing in the Omega Network . . . . . . . . . . . . . . . . . . . . . . . . 64
2.8.3 Switching . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66
2.8.4 Flow control mechanisms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74

IX
X Contents

2.9 Caches and Memory Hierarchy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74

2.9.1 Characteristics of Caches . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76
2.9.2 Write Policy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84
2.9.3 Cache coherency . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86
2.9.4 Memory consistency . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94
2.10 Examples for hardware parallelism . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100
2.10.1 Intel Cascade Lake and Ice Lake Architectures . . . . . . . . . . . 101
2.10.2 Top500 list . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105
2.11 Exercises for Chapter 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106

3 Parallel Programming Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109

3.1 Models for parallel systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109
3.2 Parallelization of programs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112
3.3 Levels of parallelism . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114
3.3.1 Parallelism at instruction level . . . . . . . . . . . . . . . . . . . . . . . . . 115
3.3.2 Data parallelism . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116
3.3.3 Loop parallelism . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118
3.3.4 Functional parallelism . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121
3.3.5 Explicit and implicit representation of parallelism . . . . . . . . . 122
3.3.6 Parallel programming patterns . . . . . . . . . . . . . . . . . . . . . . . . . 125
3.4 SIMD Computations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 130
3.4.1 Execution of vector operations . . . . . . . . . . . . . . . . . . . . . . . . . 130
3.4.2 SIMD instructions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132
3.5 Data distributions for arrays . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133
3.5.1 Data distribution for one-dimensional arrays . . . . . . . . . . . . . 134
3.5.2 Data distribution for two-dimensional arrays . . . . . . . . . . . . . 135
3.5.3 Parameterized data distribution . . . . . . . . . . . . . . . . . . . . . . . . . 136
3.6 Information exchange . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 136
3.6.1 Shared variables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 138
3.6.2 Communication operations . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139
3.7 Parallel matrix-vector product . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145
3.7.1 Parallel computation of scalar products . . . . . . . . . . . . . . . . . . 147
3.7.2 Parallel computation of the linear combinations . . . . . . . . . . . 150
3.8 Processes and Threads . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 151
3.8.1 Processes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 151
3.8.2 Threads . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153
3.8.3 Synchronization mechanisms . . . . . . . . . . . . . . . . . . . . . . . . . . 158
3.8.4 Developing efficient and correct thread programs . . . . . . . . . 161
3.9 Further parallel programming approaches . . . . . . . . . . . . . . . . . . . . . . 163
3.10 Exercices for Chapter 3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 166

4 Performance Analysis of Parallel Programs . . . . . . . . . . . . . . . . . . . . . . . 169

4.1 Performance Evaluation of Computer Systems . . . . . . . . . . . . . . . . . . 170
4.1.1 Evaluation of CPU Performance . . . . . . . . . . . . . . . . . . . . . . . . 170
4.1.2 MIPS and MFLOPS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 172
Contents XI

4.1.3 Performance of Processors with a Memory Hierarchy . . . . . . 174

4.1.4 Benchmark Programs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 176
4.2 Performance Metrics for Parallel Programs . . . . . . . . . . . . . . . . . . . . . 181
4.2.1 Parallel Runtime and Cost . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 181
4.2.2 Speedup and Efficiency . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 182
4.2.3 Weak and Strong Scalability . . . . . . . . . . . . . . . . . . . . . . . . . . . 185
4.3 Energy Measurement and Energy Metrics . . . . . . . . . . . . . . . . . . . . . . 191
4.3.1 Performance and Energy Measurement Techniques . . . . . . . . 191
4.3.2 Modeling of Power and Energy Consumption for DVFS . . . . 196
4.3.3 Energy Metrics for Parallel Programs . . . . . . . . . . . . . . . . . . . 197
4.4 Asymptotic Times for Global Communication . . . . . . . . . . . . . . . . . . 200
4.4.1 Implementing Global Communication Operations . . . . . . . . . 201
4.4.2 Communications Operations on a Hypercube . . . . . . . . . . . . . 207
4.4.3 Communication Operations on a Complete Binary Tree . . . . 215
4.5 Analysis of Parallel Execution Times . . . . . . . . . . . . . . . . . . . . . . . . . . 218
4.5.1 Parallel Scalar Product . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 219
4.5.2 Parallel Matrix-vector Product . . . . . . . . . . . . . . . . . . . . . . . . . 221
4.6 Parallel Computational Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 223
4.6.1 PRAM Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 223
4.6.2 BSP Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 226
4.6.3 LogP Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 228
4.7 Loop Scheduling and Loop Tiling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 231
4.7.1 Loop Scheduling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 232
4.7.2 Loop Tiling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 242
4.8 Exercises for Chapter 4 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 244

5 Message-Passing Programming . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 249

5.1 Introduction to MPI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 250
5.1.1 MPI point-to-point communication . . . . . . . . . . . . . . . . . . . . . 252
5.1.2 Deadlocks with Point-to-point Communications . . . . . . . . . . 257
5.1.3 Nonblocking Point-to-Point Operations . . . . . . . . . . . . . . . . . . 260
5.1.4 Communication modes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 262
5.2 Collective Communication Operations . . . . . . . . . . . . . . . . . . . . . . . . . 266
5.2.1 Collective Communication in MPI . . . . . . . . . . . . . . . . . . . . . . 266
5.2.2 Deadlocks with Collective Communication . . . . . . . . . . . . . . 280
5.2.3 Nonblocking Collective Communication Operations . . . . . . . 283
5.3 Process Groups and Communicators . . . . . . . . . . . . . . . . . . . . . . . . . . . 285
5.3.1 Process Groups in MPI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 285
5.3.2 Process Topologies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 290
5.3.3 Timings and aborting processes . . . . . . . . . . . . . . . . . . . . . . . . 295
5.4 Advanced topics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 295
5.4.1 Dynamic Process Generation and Management . . . . . . . . . . . 296
5.4.2 One-sided communication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 298
5.5 Exercises for Chapter 5 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 308
XII Contents

6 Thread Programming . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 313

6.1 Programming with Pthreads . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 314
6.1.1 Creating and Merging Threads . . . . . . . . . . . . . . . . . . . . . . . . . 315
6.1.2 Thread Coordination with Pthreads . . . . . . . . . . . . . . . . . . . . . 318
6.1.3 Condition Variables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 325
6.1.4 Extended Lock Mechanism . . . . . . . . . . . . . . . . . . . . . . . . . . . . 330
6.1.5 One-time initialization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 332
6.2 Parallel programming patterns with Pthreads . . . . . . . . . . . . . . . . . . . . 333
6.2.1 Implementation of a Task Pool . . . . . . . . . . . . . . . . . . . . . . . . . 333
6.2.2 Parallelism by Pipelining . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 339
6.2.3 Implementation of a Client-Server Model . . . . . . . . . . . . . . . . 345
6.3 Advanced Pthread features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 349
6.3.1 Thread Attributes and Cancellation . . . . . . . . . . . . . . . . . . . . . 349
6.3.2 Thread Scheduling with Pthreads . . . . . . . . . . . . . . . . . . . . . . . 358
6.3.3 Priority Inversion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 361
6.3.4 Thread-specific Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 365
6.4 Java Threads . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 366
6.4.1 Thread Generation in Java . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 367
6.4.2 Synchronization of Java Threads . . . . . . . . . . . . . . . . . . . . . . . 370
6.4.3 Wait and Notify . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 379
6.4.4 Extended Synchronization Patterns . . . . . . . . . . . . . . . . . . . . . 385
6.4.5 Thread Scheduling in Java . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 389
6.4.6 Package java.util.concurrent . . . . . . . . . . . . . . . . . . . . . 391
6.5 OpenMP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 398
6.5.1 Compiler directives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 399
6.5.2 Execution environment routines . . . . . . . . . . . . . . . . . . . . . . . . 406
6.5.3 Coordination and synchronization of threads . . . . . . . . . . . . . 407
6.5.4 OpenMP task model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 413
6.6 Exercises for Chapter 6 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 419

7 General Purpose GPU Programming . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 423

7.1 The Architecture of GPUs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 424
7.2 Introduction to CUDA programming . . . . . . . . . . . . . . . . . . . . . . . . . . 429
7.3 Synchronization and Shared Memory . . . . . . . . . . . . . . . . . . . . . . . . . . 435
7.4 CUDA Thread Scheduling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 440
7.5 Efficient Memory Access and Tiling Technique . . . . . . . . . . . . . . . . . 441
7.6 Introduction to OpenCL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 447
7.7 Exercises for Chapter 7 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 449

8 Algorithms for Systems of Linear Equations . . . . . . . . . . . . . . . . . . . . . . 451

8.1 Gaussian Elimination . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 452
8.1.1 Gaussian Elimination and LU Decomposition . . . . . . . . . . . . 452
8.1.2 Parallel Row-Cyclic Implementation . . . . . . . . . . . . . . . . . . . . 456
8.1.3 Parallel Implementation with Checkerboard Distribution . . . 460
8.1.4 Analysis of the Parallel Execution Time . . . . . . . . . . . . . . . . . 464
Contents XIII

8.2 Direct Methods for Linear Systems with Banded Structure . . . . . . . . 470
8.2.1 Discretization of the Poisson Equation . . . . . . . . . . . . . . . . . . 470
8.2.2 Tridiagonal Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 475
8.2.3 Generalization to Banded Matrices . . . . . . . . . . . . . . . . . . . . . 489
8.2.4 Solving the Discretized Poisson Equation . . . . . . . . . . . . . . . . 491
8.3 Iterative Methods for Linear Systems . . . . . . . . . . . . . . . . . . . . . . . . . . 493
8.3.1 Standard Iteration Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . 494
8.3.2 Parallel implementation of the Jacobi Iteration . . . . . . . . . . . . 498
8.3.3 Parallel Implementation of the Gauss-Seidel Iteration . . . . . . 499
8.3.4 Gauss-Seidel Iteration for Sparse Systems . . . . . . . . . . . . . . . 501
8.3.5 Red-black Ordering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 505
8.4 Conjugate Gradient Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 512
8.4.1 Sequential CG method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 512
8.4.2 Parallel CG Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 514
8.5 Cholesky Factorization for Sparse Matrices . . . . . . . . . . . . . . . . . . . . . 518
8.5.1 Sequential Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 518
8.5.2 Storage Scheme for Sparse Matrices . . . . . . . . . . . . . . . . . . . . 525
8.5.3 Implementation for Shared Variables . . . . . . . . . . . . . . . . . . . . 526
8.6 Exercises for Chapter 8 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 532

References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 537

Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 547
Chapter 1
Introduction

About this Chapter

Parallel programming is increasingly important for the software develop-

ment today and in the future. This introduction outlines the more classical
use of parallelism in scientific computing using supercomputers as well as
the parallelism available in today’s hardware, which broadens the use of par-
allelism to a larger class of applications. The basic concepts of parallel pro-
gramming are introduced on less than two pages by informally defining key
definitions, such as task decomposition or potential parallelism, and bring-
ing them into context. The content of this book on parallel programming is
described and suggestions for course structures are given.

1.1 Classical Use of Parallelism

Parallel programming and the design of efficient parallel programs is well-establi-

shed in high performance, scientific computing for many years. The simulation
of scientific problems is an important area in natural and engineering sciences of
growing importance. More precise simulations or the simulation of larger problems
lead to an increasing demand for computing power and memory space. In the last
decades, high performance research also included the development of new parallel
hardware and software technologies, and steady progress in parallel high perfor-
mance computing can be observed. Popular examples are simulations for weather
forecast based on complex mathematical models involving partial differential equa-
tions or crash simulations from car industry based on finite element methods. Other
examples include drug design and computer graphics applications for film and ad-
vertising industry.
Depending on the specific application, computer simulation is the main method
to obtain the desired result or it is used to replace or enhance physical experiments.

© The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 1

T. Rauber, G. Rünger, Parallel Programming, https://doi.org/10.1007/978-3-031-28924-8_1
2 1 Introduction

A typical example for the first application area is weather forecasting where the fu-
ture development in the atmosphere has to be predicted, which can only be obtained
by simulations. In the second application area, computer simulations are used to
obtain results that are more precise than results from practical experiments or that
can be performed at lower cost. An example is the use of simulations to determine
the air resistance of vehicles: Compared to a classical wind tunnel experiment, a
computer simulation can get more precise results because the relative movement of
the vehicle in relation to the ground can be included in the simulation. This is not
possible in the wind tunnel, since the vehicle cannot be moved. Crash tests of ve-
hicles are an obvious example where computer simulations can be performed with
lower cost.
Computer simulations often require a large computational effort. Thus, a low
performance of the computer system used can restrict the simulations and the ac-
curacy of the results obtained significantly. Using a high-performance system al-
lows larger simulations which lead to better results and therefore, parallel comput-
ers have usually been used to perform computer simulations. Today, cluster systems
built up from server nodes are widely available and are now also often used for
parallel simulations. Additionally, multicore processors within the nodes provide
further parallelism, which can be exploited for a fast computation. To use paral-
lel computers or cluster systems, the computations to be performed must be parti-
tioned into several parts which are assigned to the parallel resources for execution.
These computation parts should be independent of each other, and the algorithm
performed must provide enough independent computations to be suitable for a par-
allel execution. This is normally the case for scientific simulation, which often use
one- or multi-dimensional arrays as data structures and organize their computations
in nested loops. To obtain a parallel program for parallel excution, the algorithm
must be formulated in a suitable programming language. Parallel execution is often
controlled by specific runtime libraries or compiler directives which are added to
a standard programming language, such as C, Fortran, or Java. The programming
techniques needed to obtain efficient parallel programs are described in this book.
Popular runtime systems and environments are also presented.

1.2 Parallelism in Today’s Hardware

Parallel programming is an important aspect of high performance scientific com-

puting but it used to be a niche within the entire field of hardware and software
products. However, more recently parallel programming has left this niche and will
become the mainstream of software development techniques due to a radical change
in hardware technology.
Major chip manufacturers have started to produce processors with several power-
efficient computing units on one chip, which have an independent control and can
access the same memory concurrently. Normally, the term core is used for single
computing units and the term multicore is used for the entire processor having sev-
1.2 Parallelism in Today’s Hardware 3

eral cores. Thus, using multicore processors makes each desktop computer a small
parallel system. The technological development toward multicore processors was
forced by physical reasons, since the clock speed of chips with more and more tran-
sistors cannot be increased at the previous rate without overheating.
Multicore architectures in the form of single multicore processors, shared mem-
ory systems of several multicore processors, or clusters of multicore processors with
a hierarchical interconnection network will have a large impact on software develop-
ment. In 2022, quad-core and oct-core processors are standard for normal desktop
computers, and chips with up to 64 cores are already available for a use in high-
end systems. It can be predicted from Moore’s law that the number of cores per
processor chip will double every 18 – 24 months and in several years, a typical
processor chip might consist of dozens up to hundreds of cores where some of the
cores will be dedicated to specific purposes such as network management, encryp-
tion and decryption, or graphics [138]; the majority of the cores will be available
for application programs, providing a huge performance potential. Another trend in
parallel computing is the use of GPUs for compute-intensive applications. GPU ar-
chitectures provide many hundreds of specialized processing cores that can perform
computations in parallel.
The users of a computer system are interested in benefitting from the perfor-
mance increase provided by multicore processors. If this can be achieved, they can
expect their application programs to keep getting faster and keep getting more and
more additional features that could not be integrated in previous versions of the soft-
ware because they needed too much computing power. To ensure this, there should
definitely be support from the operating system, e.g., by using dedicated cores for
their intended purpose or by running multiple user programs in parallel, if they are
available. But when a large number of cores is provided, which will be the case in
the near future, there is also the need to execute a single application program on
multiple cores. The best situation for the software developer would be that there is
an automatic transformer that takes a sequential program as input and generates a
parallel program that runs efficiently on the new architectures. If such a transformer
were available, software development could proceed as before. But unfortunately,
the experience of the research in parallelizing compilers during the last 20 years has
shown that for many sequential programs it is not possible to extract enough paral-
lelism automatically. Therefore, there must be some help from the programmer and
application programs need to be restructured accordingly.
For the software developer, the new hardware development toward multicore ar-
chitectures is a challenge, since existing software must be restructured toward paral-
lel execution to take advantage of the additional computing resources. In particular,
software developers can no longer expect that the increase of computing power can
automatically be used by their software products. Instead, additional effort is re-
quired at the software level to take advantage of the increased computing power. If
a software company is able to transform its software so that it runs efficiently on
novel multicore architectures, it will likely have an advantage over its competitors.
There is much research going on in the area of parallel programming languages
and environments with the goal of facilitating parallel programming by providing
Another Random Scribd Document
with Unrelated Content
 Mūshteri led him to the door of the council chamber, and bade him
go to his home and friends. “Alas!” said he, “I have no home—I have
no friends. I have outraged the confidence of the Shāh, there is no
room for me in his dominions, and even the father who taught me
the lessons of hypocrisy is now ashamed of his son. I have no home
but the desert—no friend but death.”
He went away, but the disappointed malice, and the hopeless
future, had wrought a change in the strong man that he was
powerless to overcome; he returned to his caravan which had been
restored to him by the intercession of Meher and Mūshteri, but in a
few days his lifeless body was found upon the plains, and his
servants claimed that he had died by his own hand.
 CHAPTER XXI.
MEHER AND MŪSHTERI—CONCLUDED.

THE WEDDING—A COUNCIL—ROYAL CAVALCADE—THE MESSENGER—

RECEPTION.

A pavilion was built beneath the palm trees, and the fire-flies lit
their signals afresh in the thickets of foliage, for it was amidst the
shades of the garden that the singers were placed, whose sweetest
notes were to be poured forth at the royal wedding. Within the
palace, the courts were all ablaze with light and loveliness; lamps of
graven silver were swinging from the fretted roof, suspended by long
chains, and fed with the perfumed oils of distant lands. Their soft
light fell on silken hangings and tapestries from Eastern looms, while
crystal vases gleamed here and there, filled with branches of orange
trees or sprays of magnolia blossoms. It was here that Meher
received his royal bride, and when the ceremony was finished, the
notes of music floated in through the casement, and mingled with
the breath of the flowers. Still nearer seemed to come the dream-
like harmonies, as the tones of pipe and lute were mingled with the
voices of the singers and the musical ripple of the fountains.
Then the dancing girls floated into the bright halls, and swayed
gracefully through the soft measures, and all was motion, light and
jewels. Golden chains were woven in their dark hair, and silver
bangles gleamed upon the shapely ankles, where little bells kept
time with gliding feet. Each dancer held a dainty lute of gold and
sandal wood, which answered to the swaying of her arms and the
soft beat of graceful hands. And still the music from without floated
through the lattice and mingled with the harmonies within. But in
this festal scene Love was the honored guest. He came to rule the
court and grove; his were the symphonies that breathed a richer
note than all the garden singers; his were the harmonies that
shaped the loyal lives, and led the happy feet along the aisles of
time.
Bewildered with the beauty and love of his bride, Meher lived for
weeks unheeding the lapse of time, for all the days were crowned
with gold and radiant with the blossoms of love. But there came a
morning when the picture of his grieving mother was forced upon
his heart and mind with all its power, and he remembered that not
alone to his lovely wife belonged his fealty.
They were sitting together beneath the sheltering branches of a
great magnolia tree, whose creamy flowers were bursting from the
green sheath of the bud, and the air was rich with fragrance. On the
green bank beyond them, the peacocks drew their gorgeous trains,
and birds sang in the tall trees in the distance.
The dark eyes of the prince had a look of sadness in them, and
there was a cloud, the first that Nahīd had ever seen upon his
handsome brow; she drew closer within the sheltering arm, and her
soft, dark eyes looked anxiously into his. His own heart read her
pleading question, even before her lips had framed it, and then he
told her of the loving mother who was grieving her life away amidst
the splendors of another court—of the faithful heart that looked
longingly for his return and refused to be comforted, because he
came not.
“But what can we do?” questioned the princess. “Thou canst not
leave the wife to go even to the mother.”
“No,” answered the prince, “but can I not take my bride with me?
Can I not take my peerless pearl to the royal court which is my
rightful inheritance? Can I not bear to her arms the beauteous
daughter that I have given her? Surely my wife should receive my
mother’s blessing! Let me take thee there before the faithful mother-
heart is cold in death.”
“But my father,” faltered Nahīd, “will he consent? Will he allow
thee to bear me away to a strange land to claim the lost
inheritance?”
“The king should remember that not only filial love demands my
return, but I can never make my bride the queen that she should be
—I can never place a royal crown upon her lovely brow unless I
return to the land of my fathers,” answered the prince; and then he
told her, with loving thought, of the land where the palms grew
higher by striving toward the sun, of the marble palaces of Istakhar,
standing beside the river that came down from the heights rippling
with low harmonies, as the waves dashed on the sanded shores; told
her, too, of the mountains beyond the marble city, where the wild
swans came to their nesting places,—white voyagers on the seas of
blue, calling, in soft notes, down the line, while love was leading
them homeward, to the sheltered nooks beside the pools of the
mountain stream.
Long they stayed in loving converse, and when they turned to the
palace court, the prince had won from his bride a promise that she
would see the king, and win, if possible, his consent to the long
bridal trip, that she now looked forward to with pleasure.

ROYAL CAVALCADE.

The king listened patiently to the plea of Nahīd, for though he

knew that the long journey would take her from him, perhaps
forever, he also knew that the throne of Persia might be waiting for
their coming, and at last he consented that the prince should bear
his bride away to wear a crown in the halls of the proud Sassanian
kings.
But she should not go dowerless to the home of her husband.
Keiwan therefore gave orders for the fitting out of a magnificent
cavalcade, comprising a thousand camels of the purest Syrian blood,
a thousand splendid Arabian steeds and a thousand Indian slaves,
besides a military escort composed of the finest troops in the service
of the king.
The morning was radiant with golden sunlight when the splendid
procession left the city of Khārizm; the streets were gorgeous with
flags, and branches of flowering trees stood by every doorway, while
the palace itself was covered with silken banners, lightly draped with
wreaths of flowers. The excited horses, with their golden caparisons,
tossed their heads in the air, and pranced with joy as the strains of
music rang out from the balconies around them, and the camels
gently shook their light-toned bells in every passing breeze.
Hundreds of banners floated above the troops and waved like the
wings of birds in the sunlight; the gleaming swords of the warriors
were pointed up to heaven, and a thousand voices rang with joyous
acclamation. Keiwan and his queen rode in the imperial chariot
immediately behind the camels bearing the luxurious cushions of the
prince and his bride, for they traveled a day’s journey with them
before bidding their children farewell, and then returned sorrowfully
to their lonely palace home.
The gorgeous cavalcade moved slowly onward, over hill and plain,
and through a forest where all the branches laughed with songs of
birds, and trusses of scarlet pomegranate blossoms gleamed here
and there through the rich foliage. When night came down upon the
landscape an encampment was made beside a river, and pavilions of
scarlet and gold were furnished with costly cushions that invited
repose.

THE MESSENGER.

The uneventful days passed slowly by, and still the great
cavalcade was far from its destination, when Meher ordered his
especial attendant to mount one of the swiftest Arabian horses and
carry a letter to his father asking if he wished him to return.
The Persian monarch was sitting in the council hall surrounded by
his counselors, and they were considering an important affair of
state when a messenger was announced. He was ordered into an
adjoining room to wait until King Shapur was ready to receive him,
and here he could look upon the once familiar form of his sovereign.
He was astonished to see how greatly the Shāh had changed with
the passing years; only three times had the seasons made their
cycles, and yet the stalwart form was bent as if with age, the dark
hair was already silvered and the furrows upon the weary brow told
that grief and remorse were leaving their impress upon his once
serene countenance. At last the word was brought that the
messenger could now approach the king, but he replied that his was
a secret mission, he must see his majesty alone, and after a time he
was ushered into the private audience room.
He then told the king that he brought him news from Behrām,
who had obtained a magnificent caravan under the pretext of finding
the prince. The king listened eagerly while the messenger gave a
graphic description of the pursuit and capture of Mūshteri but his
brow darkened with an ominous frown as the recital continued. He
had been the prey of evil advisers who cared only to flatter him for
their own gain, and in the years that had gone he sadly missed the
faithful advice and unfailing loyalty of his old Vizir. He often
reproached himself as the indirect cause of his death, and decreed in
his heart that if the banished son could be found he should be
recompensed, so far as lay in his power, for all sufferings of the past.
When, therefore, he learned of the persistent brutality of Behrām his
anger grew almost uncontrolable. He inquired anxiously for the
prince. “You bring me bad news enough;” he cried, “can you give
me no knowledge of my son?” And he answered: “Oh, king, great
and mighty ruler of the wide realm, I can bring thee news of the
prince, for I have seen him in a foreign court.” “Where didst thou see
him? What is he doing, and why does he not return to the land of
his fathers?” he rapidly questioned. “He has risen, oh, king, to great
eminence at the court of a foreign potentate, and he hath no need
to return to thee, but his heart yearns for his native land; he cares
much to spend his years near to the father whom he still loves, and
he longs to take his beloved mother into his arms again. I have
brought thee a letter from him,” and then he placed the document in
the royal hand. “A letter!” cried the Shāh, “a letter from my son!”
and he ceased to be a king, for now he was only a father, and the
manly tears coursed down his cheeks as he caught the precious
missive and pressed the hand of the messenger.
As soon as he could read the communication from Meher he called
for writing materials, and with his own hand he penned a long and
loving letter to his son, telling him that not only his home but also
the Persian crown awaited his coming, urging him to return and
bring with him the faithful friend who had suffered so much on
account of his loyalty to the prince. Then he hastened the
messenger away, that he might reach Meher at the earliest possible
moment, and he himself went to bear the glad tidings to the
sorrowful queen.
The next day a proclamation was issued that the heir of the
throne was coming to the capital city, and orders were given to the
Grand Vizir, to the chamberlains and other officers of the crown that
suitable preparations be made to welcome the prince and his bride.

THE RECEPTION.

The announcement of his coming was a signal for general

rejoicing; even the children loved the young heir and knew the story
of fraternal affection between him and Mūshteri. The Shāh had been
bitterly blamed in the hearts of his subjects, although such was the
force of Oriental despotism that a man scarcely knew the thought of
his neighbor. Never were the imperial orders more willingly obeyed
than when the Shāh commanded a festal scene to be arranged for
the reception of Meher, and never was the marble city fairer than
when the coming of the royal cavalcade was announced. Silken
banners waved in triumph from every wall and battlement, while
strains of martial music floated through the air, and the streets were
strewn with white lilies and the fragrant roses of Persia. Gilded
barges on the river wore their festal flags, and bore the minstrels
down the stream to the shore, where the voices of singers were
mingled with the notes of lute and psaltery.
Without the city the Persian road of palms was festooned with
arches of roses and strewn with the flowers of the valley, for all the
way was glad with blossoms and vocal with the songs of welcome.
In the early morning a swiftly-mounted courier had been stationed
on an eminence a few miles from the city, where he could see the
approaching cavalcade far down the valley, and when he rode into
the city with the message that the advance guard was already in
sight there were loud acclamations of joy. For hours the finest horses
in the royal stables had stood impatient, with tossing plumes and
gorgeous trappings, waiting for the advance, and now the Shāh,
with his chosen guard, rode out in royal state to meet the coming
prince.
The white Arabian steeds, the costly armor of the troops and the
rich raiment of the Shāh, made a gorgeous picture in the sunlight,
when they swept down through the rose-covered arches and under
the palms. As they rode onward a new strain of music saluted their
ears, and a long line of camels came swinging slowly into view, their
heads tufted with bright tassels, while their light-toned bells were
shaking silvery notes upon the air, and their drivers were singing and
playing on pipes. But lo! the lines were opened for a small troop of
horsemen who galloped towards the Shāh, and Meher, swinging
gracefully down from the saddle, came to his father’s feet.
King Shapur quickly recognized the familiar face, and hastily
dismounting, he caught his son in his arms. The hardy Persian
soldiers turned away from the sacred scene with tears in their eyes,
but after a time Mūshteri came forward, and humbly kneeling at the
monarch’s feet he craved forgiveness. The Shāh laid his hand upon
the head of him who, in his childhood, had seemed almost as near
as his own son, and freely gave the royal pardon; then the lines
were reformed, Meher and Mūshteri riding on either side of the king,
and the horses were turned toward Istakhar.
The sun was sinking behind the western mountains when the
cavalcade approached the gates of the city, and the dark thickets by
the roadside were vocal with the song of the nightingale; but his
voice was soon hushed by the notes of martial music and the
triumphant shouts of welcome that greeted their first appearance to
the people who had been held back by the spears of the soldiery.
Although the distant peaks still wore the crimson crowns of sunset,
the side of the mountain was already dark with the gathering shades
of twilight, and signal fires flashed from the gray depths of the forest
or blazed upon the leafless slopes of granite beyond them. Within
the city all was joyous tumult; but Meher had little heart for the
general rejoicing, and scarcely waiting to be announced he hurried
away to the apartments of his mother. A little later the Princess
Nahīd was ushered into the rooms of the queen, and was folded
closely to the warm, loving heart, so fully prepared to receive her.
Little cared the mother for the wondrous beauty of the princess, but
much she valued the loyal heart which had been given so fully into
the keeping of her son, and from that day forth she was cherished
as a loving daughter in the royal household.
The days flew by on joyous feet, but King Shapur was weary of
the cares of state—weary of a life whose very pleasures were
burdened with responsibility and embittered with the knowledge that
treachery waited only for a favorable opportunity to show her cruel
fangs. He therefore abdicated in favor of his son, and voluntarily
invested Meher with the robes of sovereignty.
All the resources of the kingdom were taxed to provide for the
splendors of the coronation ceremony. Again the royal procession
swept through the streets, and feasts were given where the richest
wines of the East were poured in jeweled cups and the tables were
laden with the choicest viands from many climes. There were
plantains, golden and green, and grapes of gold; there were apples
and pomegranates from Kabūl, apricots from the fairest gardens of
Īrān, and the sunniest fruits in all the lands of the Orient.
Again the dark face of the mountain blazed forth at night with the
signal-lights of victory, the river was covered with barges bearing
illuminations, and the night rivaled the day in the splendor of its
offerings at the feet of the new Shāh, and Mūshteri, his Grand Vizir.
 CHAPTER XXII.
CONCLUSION.

SUMMARY—PRIESTLY RULE—RUSSIAN OPPRESSION.

We have now passed in review the principal features of a great

literature from its early mythology to the time when the rule of
priestcraft, combined with political tyranny, seems to have quenched
the fire of Persian genius.
The empire gathered to herself the culture of ancient Nineveh and
the poetic dreams of Chaldea, but, not content with the heritage
which she received from more ancient kingdoms, she developed,
from resources peculiarly her own, a literature which is rich in all
that pertains to Oriental fancy. Her mythology, like that of other
Āryan races, is traceable to the system of sensual idolatry which
flourished in the valley of the Euphrates; the origin of her myths was
found in the “sacred groves of Baal,” and around the altars of
Ashtaroth.
Merodach and Nebo, Moloch and Chemosh reappeared in later
times in other lands, and under different names, but still with the
same characteristics which they had in the land of their birth.
We have seen the gradual growth of her Zend-Avesta with the
inauguration of her system of worship, and noted the fact there was
originally a close connection between the Veda and the Zend-Avesta,
although the Persian work was of later origin.
Some of the Hindū gods bear the same names in the Avesta that
were applied to them in the Sanskṛit poems, although in the later
books they may appear as evil spirits, and the same god is
sometimes represented as an angel, and again as a fiend.
 Indra, the storm-king of the Veda, was the god of war, for whom
the Ṛishis made and drank the intoxicating Soma, while in the
Vendīdad[288] he is expressly mentioned in the list of evil spirits, and
is second only to Ahriman, the arch-fiend of the Avesta. But another
name for Indra in the Vedic songs is Vṛitrahā, and this name is
recognized as that of the angel Verethragna; hence it follows that
under one name the god is cursed and feared as a fiend, while
under another he is worshipped as an angel.
The name of Deva in the Vedas, and in all Brāhmanical literature,
is applied to divine beings who are still worshipped by the Hindūs,
while in the Avesta, from the earliest to the latest texts, and even in
modern Persian literature, Deva is a name applied to a fiend. The
word Asura, although used in a good sense in the early songs of the
Rig-veda, becomes, in the later portions of that literature, as well as
in the Brāhmaṇas and Purāṇas, a term which is applied only to evil
spirits; they are represented as the constant enemies of the Hindū
gods, always making attacks upon the sacrifices offered by devotees.
In the Avesta, Asura, in the form of Ahura, becomes a component
part of Ahura Mazda, which is the name of God among the Pārsīs,
whose faith is called “the Ahura religion”[289] in order to distinctively
indicate its opposition to the Deva religion.
The Vedic god, Vāyu (the wind), is readily recognized in the spirit
Vāyu in the Avesta, who is supposed to be roaming everywhere.
Another instance of a deity who is scarcely changed in any way is
Mithra, the Sanskṛit form of which is Mitra. In the Ṛig-veda “Mitra
calls men to their work; Mitra is preserving earth and heaven; Mitra
looks upon the nations always without shutting his eyes.” In the
Avesta he is also the lord of the morning, the god of day, and the
object of profound adoration.
These are but a few out of many similarities, and the careful
student of the Veda and the Avesta will also notice the identity of
many terms referring to priestly functions. The very name of “priest”
in the Zend-Avesta is atharva, and it is merely another form of
atharvan, which is the term applied to the priest of fire and Soma, in
the Vedas.[290]
 These and many other similarities do not necessarily prove that
the Zend-Avesta was partially copied from the Veda, but they do
prove that “the Veda and the Zend-Avesta are two rivers flowing
from one fountain head; the stream of the Veda is the fuller and
purer, and has remained truer to its original character; that of the
Zend-Avesta has been in various ways polluted, has altered its
course, and cannot, with certainty, be traced back to its source.”[291]
Nevertheless, their common origin must be assigned to the early
Indo-Īraniān traditions.
Besides the official copy of the sacred books, which was burned by
Alexander with the palace of the Persian king at Persepolis, there
were other copies, or at least portions of them, and these the first
Sassanian kings collected, and compiled from them, as far as
possible, their sacred literature. For more than five centuries after
Alexander, the empire of Persia suffered from foreign despotism and
internal dissensions, but during this long period of political unrest,
the Sassanian kings were able to collect a large proportion of the old
writings, even though the literature which was thus restored
consisted chiefly of fragments; it appears, however, that some
portion of nearly every book was recovered by the zeal of these
monarchs, and therefore the total disappearance of some of them
must be assigned to more recent times.
A still greater disaster awaited the books of the Persians at the
hand of the Moslem invader, when the Arabian horde swept over the
hills and valleys of Persia like a simoon from the desert. Every tree
and flower seemed to feel the withering touch of the barbarian, and
the authority of the Korān was enforced with the logic of the sword.
“Ye know your option, ye Christian dogs; the Korān, tribute, or the
sword,” was the dictum of the conqueror wherever the Moslem flag
was triumphant, and at last the Star and Crescent floated over the
land of “the Lion and the Sun”—her nationality was humiliated and
crushed, while her treasures of literature were again destroyed by a
foreign foe. The kingdom of Persia now entered upon the long night
of Mohammedan rule. Her sacred books were swept from the land,
the Korān became the successor of the Zend-Avesta, and many of
the Pārsīs went into voluntary exile, finding upon the shores of India
that freedom which was denied them upon their native soil. Even the
Persian tongue was placed under a ban, and Arabic became the legal
language of court and council hall.
The Persians were conquered, but not subdued; the national spirit
still lived in their hearts, and in more than one instance the conquest
was repeated—for, in the defence of their nationality and their faith,
they rebelled in different portions of the country and fought
desperately against the hated Arab. They were subjugated at last,
and, to a certain extent, accepted even the religion of the invader,
but the vitality of the Persian character was not destroyed.
After a time, a few of the subordinate rulers, who were natives,
rebelled against the tyranny of the Arabic tongue, and succeeded in
establishing the Persian language to a great extent in its rightful
domain. The national spirit again rallied, Persian poets were
encouraged, traditions of the empire were once more collected, and
the composition of a great national epic became possible. The Shāh
Nāmah, which was written under royal patronage, has lived through
the vicissitudes of more than eight hundred years, and is still the
most popular Persian classic. Other centuries followed, bearing the
names of distinguished poets and scholars, the cities of Bokhāra,
Samarcānd and Bagdad became great literary centers, their colleges
and libraries being celebrated throughout the East.
But again the power of brute force was destined to sweep away
the bulwarks of civilization, and Genghis Khān, the Tartar chief, came
down like a mountain storm upon the fairest provinces of the Orient.
The principal cities were pillaged and burned by the Tartar horde,
colleges were destroyed, and the most valuable books in the libraries
were thrown into the Tigris.
These were times which tried the hearts of men, for more than
two hundred thousand lives were sacrificed to the cruelty of the
invading host. Scholars were driven to various places of refuge, and
the science of letters received an almost fatal blow.
There are, however, a few illustrious names upon the records of
the Persian literati, even after the close of the thirteenth century,
and such was the intellectual vitality of the people that lyric poetry
and rhetoric were well developed during these stormy times in the
political and military world, for the empire had still many men of
culture, and also boasted of one literary king.

PRIESTLY RULE.

Nations, as well as individuals, have their periods of growth,

prosperity and decay. It is seldom that they arise from an age of
great prostration and regain their former strength and brilliancy.
History, however, furnishes bright exceptions to this general rule,
and Persia has repeatedly recovered herself from the ravages of
foreign conquest. Three times her territory has been invaded when
the design of the conqueror has apparently been the extermination
of the science of letters, and three times she has rallied bravely from
the shock and rebuilded her institutions of learning, founding a new
national literature upon the ruins of the old.
Her literature of to-day is profuse in quantity, consisting largely of
the various forms of romance,[292] but the best works of Persian
authors belong to the centuries past. Perhaps she might rally even
the fourth time, and resume her old position in the world of letters,
but she is held in a state of lethargy by the benumbing influence of
a Mohammedan priesthood. Even the Shāh rules only by the
permission of this power, being looked upon as the vicegerent of the
prophet, and the laws of the nation are subject to the dictation of
the priests.
They stand in the way of all progress, as even a railway cannot be
laid without their permission, much less can institutions of learning
be carried on outside of their control. Official corruption, which
seems to threaten the very existence of some of the Eastern nations,
gathers new power from the influence of these Mohammedan
mollāhs, and a large share of the money which is appropriated for
public improvement eventually finds its way into the coffers of the
king’s ministers.
There is little hope of intellectual growth under this baneful
influence. At the beginning of the present century Īrānian poetry
assumed a dramatic form, but, like the Greek drama, and the
“Mysteries” of the Middle ages, it is the offspring of a religious
ceremony, and the great attraction of the Persian stage is a Moslem
passion play,[293] even the drama of the empire being under the
control of her conquerors.

RUSSIAN OPPRESSION.

Not only is the nation firmly held in the chains of priestly rule, but
her political position is far from enviable. Upon her northern border
stands the most unscrupulous power among the nations of the East.
The black eagles of the Czar are ever watching for an opportunity to
invade her dominions, ever looking for some unusual sign of internal
weakness which may throw her completely into their power. Russia
has justly earned a reputation which, for political treachery, is
unequaled among the children of men. She makes treaties and signs
the most solemn pledges of national co-operation, apparently with
the utmost sincerity, and then breaks them, without even a word of
apology, whenever she can gain a point or a province by so doing.
For centuries Russia has coveted Constantinople as the key of the
East. For centuries she has looked with envious eyes upon the
wealth of India, and she has hesitated at no policy which might
advance her interests by extending her boundary line.
Great Britain stands as the strongest bulwark in the world of
nations against the insidious diplomacy of the Muscovite, which
seems to be the enemy of all civilization, and therefore in every
move that is made in the political world of either Europe or Asia,
Russia is ever on the alert to defeat the plans of England, and the
coming conflict in the Old World will doubtless be led by these two
great powers. Intending some day to wrest India from the hand of
Great Britain, she finds Persia standing in the way of her design, and
it must therefore be conquered or absorbed. By the most
unscrupulous methods known to nations, she has already acquired
much of Persian territory, and the process of absorption is renewed
whenever the opportunity offers.
She hesitates at no oppression, and has already ruined Persian
commerce, as far as lay in her power, by permitting the
transportation of goods across her territory, only under restrictions
which are practically prohibitory. Flattering promises are carefully
combined with threats in order to promote her designs, and the
emissaries of Russia are abundant in Persia, and even in Northern
India, where their mission is to educate a public sentiment by
constantly instilling into the minds of the people false ideas of the
magnificence and generosity of Russia. These men are not Russians,
for they would attract attention and arouse public apprehension, but
they are Asiatics, who are kept at work by Russian gold, making
lavish promises of Muscovite benevolence when northern barbarism
shall succeed English civilization.
While engaged in thus duping the Asiatic tribes, she is pushing her
railway as rapidly as possible toward India, and preparing for war on
a greater scale than ever before in her history.
The record of her political policy proves that she will fasten her
iron hand upon the vitals of a nation, and crush out, as far as
possible, every effort toward progress, until the crippled empire falls
into her fatal embrace. Persia has little hope of escaping the Russian
policy of oppression and absorption, unless either English or German
troops are allied with her native forces against the common foe.
There is no longer, therefore, a hope that Persian literature may be
revived, and the intellectual resources of the empire again
developed, unless the civilized nations of Europe come to her rescue.
The yoke of Mohammedan rule must be broken, and the tyranny of
the northern barbarian removed, before the Persian mind and heart
can be stimulated to intellectual and moral activity.
 INDEX.

A.

Abbas, Shah, 335.

Accad, 1, 2, 3, 4, 33, 51.
Accadian, 66.
Accadian Tongue, 6, 31, 55.
Achæmenes, n 46.
Achæmenian, 44, 46, 48.
Achæmenidæ, 50.
Adarbad, 114.
Adonis, 68, 77.
Adonis Phrygian, 42.
Afrasiyab, the Tartar Chief, 244, 245, 247.
Age, Babylonian, 7.
Agathokles, 10.
Agni, 66, 107.
Ahasuerus, 49.
Ahriman, 48, 54, 99, 106, 129.
Ahura Mazda, 86, 94, 99, 108, 128, 161.
Akbar, Shah, 334, 335.
Akhuni, 59.
Alborz, Mount, 91, 98, 103.
Alexander, 13, 14, 20, 21, 52, 112, 113, 406;
book of, 333.
Allah-il-Allah, 23.
Al-Fateh, 333.
Almokaffa, 190.
Alp Arslan, 27.
Amicable Instruction, 189, 203.
Anāhid, 69.
Anat, 56.
Anderson, n 96.
Angra Mainyu, 127, 129.
Annatu, 70.
Antelope and Crow, 189, 205.
Anthon, 76.
Anu, 56, 69, 70.
Anwāri, 284.
Anwār-i-Suhali, 189, 215;
history of, 190;
preface of, 192.
Apaosha, 101, 105.
Aphrodite, 76.
Apis Egyptian, 87.
Arabs, 166, 176, 183, 188.
Arabia, 7, 46, 48, 166.
Arabian Conquest, 1, 22, 25.
Arabian Nights, 28.
Arbela, 40.
Architecture of Persepolis, 18.
Ardvi Sura Anāhita, 93, 94.
Arjasp, 272;
defeated by Isfendiyār, 276.
Armenia, 15, 48, 69.
Arrianus, n 36.
Art, Asiatic, 17.
Art, Greek, 5.
Art, Persian, 1, 16, 17, 18.
Artaxerxes, 13, 30, 44, 49, 50.
Aryan, 23.
Arzang, the Demon Chief, 259.
Asia, 1, 8, 23, 39, 48.
Ashtaroth, 8, n 10, n 54, 67, 68, 77, 108.
Assur, 53, 58, 59.
Assur-banipal, 6, 36, n 37, 55, 69.
Assyria, 6, 15, 33, 34, 35, 39, 46, 48.
Assyriologists, 34.
Astarte, n 10, 68.
Asura, 404.
Asteria, 68, 70.
Atar, 86, 100.
Augustus, 7.
Aulād, 252;
capture of, 257.
Avesta, 23, 54, 104, 107, 114, 115, 116, 404.

B.

Baal, 8, 53, 66, 76, 107, 108.

Baal Moloch, 10.
Babylon, 1, 6, 7, 10, 17;
fall of, 30, 36, 39, 41, 43.
Babylonia, 3, 6, 15, 33, 55, 61;
North, 3, 40, 65.
Babylonians, 9, 41.
Bāhram Gor, 214.
Bagdad, 311, 321, 407.
Balder, 77.
Bardes, 47.
Bāzindah, 195;
misfortunes of, 198;
return of, 201.
Beer, 44
Bees and their Habits, 189, 193.
Behistun Inscriptions, 17, 30, 45, 114.
Behistun Rock, n 20.
Bel, 10, 36, 40, 42, 52.
Bel Merodach, 56, 61, 62.
Bellona, 69.
Belshazzar, 35, 40, 42, 43.
Benfey, 44.
Berosus, 69.
Beth-anath, 56.
Bi-frost, (Rainbow Bridge), 103.
Birds, Mythical, 86, 95.
Bird, My, 321, 329.
Birs, Nimrud, 37.
Blackstone, Sir Wm., 156.
Blind Man and his Whip, 189, 201.
Bokhāra, 309, 323, 407.
Bombay, 161.
Bores, 309, 320.
Boscawen, W. St. Chad, 40, n 43.
Bosphorus, 13.
Bournouf, 31, 44, 116.
Brāhman and Ichneumon, 208.
Brockhaus, 116.
Buddha, 227.
Budge, E.A., n 5, n 37.
Bundehesh, 112.
Bustān, 25, 225, 309;
extracts from, 313.

C.

Calligraphy, 19, 335.

Cambyses, 44, 45, 87.
Canaan, 63.
Canopus, lights of, 190.
Captives, The, 383.
Capua, John of, 190.
Capture, the, 351, 352, 362.
Cappadocia, 48, 69.
Carthage, 48.
Carthaginians, 10.
Caucasus, 7.
Cerberus, 102.
Chaldea, 4, 64, 65, 103.
Chaos, wife of Hea, 59, 60.
Chemosh, 53, 60, 63, 64, 65, 108.
Chinvat Bridge, 86, 103, 141, 153, 157, 158.
Chips from a German Workshop, n 109, n 126.
Christians persecuted, 104, 121.
Christianity, 125, 167.
Citagriva, 204.
Circe, 74.
Commentary, 127, 140.
Comrades, the two, 338, 340.
Conquest, Mohammedan, 165.
Constellations, King of, 67.
Coptic Girl, 174, 184, 185.
Creation, legend of, 54.
Cuneiform inscriptions, 3, 6, 30.
Curtius, n 36, n 113.
Cutha, 54, 61.
Cyprus, 6.
Cyrus, 12, 15, 20, 28, 30, 40, 44, 47, 52, 118;
decree of, 20.

D.

Dagon, 63.
Daniel, n 36, 40, 41.
Damascus, 311.
Darius, 9, 13, 20, 29, 45, 47, 49, 50, 52;
and his Horsekeeper, 309, 316.
Darmesteter, Prof., 20, n 110, n 150, n 156.
Dastur, 125.
Datilla, River of Death, 107.
Dead, Disposition of, 146, 153.
Delos, 68.
Devas, 105.
Demon, White, 248, 252, 257, 260;
slain, 260.
Desert, the, 351, 353.
Diana, 75, 108.
Dihkans, 22.
Diodorus, 5, n 113.
Diocletian, 114.
Domitian, 50.
Dogs, importance of, 147, 152, 153, 156;
of Yama, 153.
Druj, 152.
Dungi, 3.
Duza, 42.
Dynasty, Achæmenian, 25;
Gaznevides, 24, 25;
Sassanian, 20, 21, 22.

E.

Eastwick, Prof., xiii, 191.

Ecbatana, 20, n 43.
Ecclesiastes, book of, 310.
Egebi, 9.
Egypt, 6, 7, 15, 39, 46, 48, 49, 61.
Elam, 7, 11, 69.
Elamites, 3.
Elephant and Jackal, 189, 209.
England, 16, 411.
Epic, Persian, 24, 25, 214.
Ephesus, 76, 108.
Ephesians, 76.
Eridu, 56, 67.
Esar-haddon, 9, n 37.
Euphrates, 3, 5, 16, 35, 37.
Evil Spirits, seven, 56, 57.
Exiles, the, 351.
Exodus, 3.

F.

Features, physical, 1, 15.

Fimbul Eddic, 130.
Fire god, 56, 57, 66, 107, 124.
Fire, sacred, 100, 124, 154, 155.
Fire worshippers, 124.
Firdusi, 22, 25, 26, 118.
Firdusi, life of, 216;
invective of, 214, 219;
death of, 214, 223.
Flattery, wiles of, 206, 209, 210, 212.
Flood, 33.
Formulas, 140.
Fravishas, 140, 160.
France, 16.
Frey, 77.
Freyja, 76.
Fugitives, the, 365.

G.

Ganges, 92, 96, 107.

Garuda, 96.
Garden scene, 365, 377, 381.
Gāthas, 20, 111, 113, 119, 127, 130, 144.
Gātha, Last, 135.