Download the Full Version of ebook for Fast Typing at ebookmass.

com

The Art of Multiprocessor Programming 2nd Edition

Maurice Herlihy

https://ebookmass.com/product/the-art-of-multiprocessor-
programming-2nd-edition-maurice-herlihy/

OR CLICK BUTTON

DOWNLOAD NOW

Download More ebook Instantly Today - Get Yours Now at ebookmass.com

 Recommended digital products (PDF, EPUB, MOBI) that
you can download immediately if you are interested.

The Art of Multiprocessor Programming. Second Edition

Maurice Herlihy

https://ebookmass.com/product/the-art-of-multiprocessor-programming-
second-edition-maurice-herlihy/

ebookmass.com

Oncoplastic Surgery of the Breast 2nd Edition Maurice Y.

Nahabedian

https://ebookmass.com/product/oncoplastic-surgery-of-the-breast-2nd-
edition-maurice-y-nahabedian/

ebookmass.com

The Art of Computer Programming - Combinatorial Algorithms

Donald Knuth

https://ebookmass.com/product/the-art-of-computer-programming-
combinatorial-algorithms-donald-knuth/

ebookmass.com

The AWK Programming Language, 2nd Edition Aho

https://ebookmass.com/product/the-awk-programming-language-2nd-
edition-aho/

ebookmass.com
Colour in Art (World of Art) 2nd Edition John Gage

https://ebookmass.com/product/colour-in-art-world-of-art-2nd-edition-
john-gage/

ebookmass.com

The Art of Assembly Language Programming using PIC

Technology. Core Fundamentals Theresa Schousek

https://ebookmass.com/product/the-art-of-assembly-language-
programming-using-pic-technology-core-fundamentals-theresa-schousek-2/

ebookmass.com

The Art of Assembly Language Programming Using PIC®

Technology : Core Fundamentals. Theresa Schousek

https://ebookmass.com/product/the-art-of-assembly-language-
programming-using-pic-technology-core-fundamentals-theresa-schousek/

ebookmass.com

The Art of Computer Programming, Volume 4B: Combinatorial

Algorithms, Part 2 Donald E. Knuth

https://ebookmass.com/product/the-art-of-computer-programming-
volume-4b-combinatorial-algorithms-part-2-donald-e-knuth/

ebookmass.com

The Design of the UNIX Operating System Maurice J. Bach

https://ebookmass.com/product/the-design-of-the-unix-operating-system-
maurice-j-bach/

ebookmass.com
 The Art of
Multiprocessor
Programming
 The Art of
Multiprocessor
Programming
Second Edition

Maurice Herlihy
Nir Shavit
Victor Luchangco
Michael Spear
Morgan Kaufmann is an imprint of Elsevier
50 Hampshire Street, 5th Floor, Cambridge, MA 02139, United States
Copyright © 2021 Elsevier Inc. All rights reserved.
No part of this publication may be reproduced or transmitted in any form or by any means, electronic
or mechanical, including photocopying, recording, or any information storage and retrieval system,
without permission in writing from the publisher. Details on how to seek permission, further
information about the Publisher’s permissions policies and our arrangements with organizations such
as the Copyright Clearance Center and the Copyright Licensing Agency, can be found at our website:
www.elsevier.com/permissions.
This book and the individual contributions contained in it are protected under copyright by the
Publisher (other than as may be noted herein).
Notices
Knowledge and best practice in this field are constantly changing. As new research and experience
broaden our understanding, changes in research methods, professional practices, or medical treatment
may become necessary.
Practitioners and researchers must always rely on their own experience and knowledge in evaluating
and using any information, methods, compounds, or experiments described herein. In using such
information or methods they should be mindful of their own safety and the safety of others, including
parties for whom they have a professional responsibility.
To the fullest extent of the law, neither the Publisher nor the authors, contributors, or editors, assume
any liability for any injury and/or damage to persons or property as a matter of products liability,
negligence or otherwise, or from any use or operation of any methods, products, instructions, or ideas
contained in the material herein.

Library of Congress Cataloging-in-Publication Data

A catalog record for this book is available from the Library of Congress

British Library Cataloguing-in-Publication Data

A catalogue record for this book is available from the British Library

ISBN: 978-0-12-415950-1

For information on all Morgan Kaufmann publications

visit our website at https://www.elsevier.com/books-and-journals

Publisher: Katey Birtcher

Acquisitions Editor: Stephen R. Merken
Editorial Project Manager: Beth LoGiudice
Production Project Manager: Beula Christopher
Designer: Renee Duenow
Typeset by VTeX
For my parents, David and Patricia Herlihy, and for Liuba, David,
and Anna.
– M.H.

For Noun and Aliza, Shafi, Yonadav, and Lior, and for Luisa.
– N.S.

For my family, especially my parents, Guilly and Maloy Luchangco,

and for God, who makes all things possible.
– V.L.

For Emily, Theodore, Bernadette, Adelaide, Teresa, Veronica, Phoebe,

Leo, and Rosemary.
– M.S.
Contents

Preface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xv
Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xix
Suggested ways to teach the art of multiprocessor programming . . . . . . . . . . . xxi
CHAPTER 1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.1 Shared objects and synchronization . . . . . . . . . . . . . . . . . . . . 3
1.2 A fable . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
1.2.1 Properties of a mutual exclusion protocol . . . . . . . . . . 8
1.2.2 The moral . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
1.3 The producer–consumer problem . . . . . . . . . . . . . . . . . . . . . . 9
1.4 The readers–writers problem . . . . . . . . . . . . . . . . . . . . . . . . . 11
1.5 The harsh realities of parallelization . . . . . . . . . . . . . . . . . . . . 12
1.6 Parallel programming . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
1.7 Chapter notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
1.8 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

PART 1 Principles
CHAPTER 2 Mutual exclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
2.1 Time and events . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
2.2 Critical sections . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
2.3 Two-thread solutions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
2.3.1 The LockOne class . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
2.3.2 The LockTwo class . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
2.3.3 The Peterson lock . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
2.4 Notes on deadlock . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
2.5 The filter lock . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
2.6 Fairness . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
2.7 Lamport’s Bakery algorithm . . . . . . . . . . . . . . . . . . . . . . . . . 34
2.8 Bounded timestamps . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
2.9 Lower bounds on the number of locations . . . . . . . . . . . . . . . 39
2.10 Chapter notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
2.11 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
CHAPTER 3 Concurrent objects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
3.1 Concurrency and correctness . . . . . . . . . . . . . . . . . . . . . . . . . 49
3.2 Sequential objects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
3.3 Sequential consistency . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
3.3.1 Sequential consistency versus real-time order . . . . . . . 55
3.3.2 Sequential consistency is nonblocking . . . . . . . . . . . . . 56

vii
viii Contents

3.3.3 Compositionality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
3.4 Linearizability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58
3.4.1 Linearization points . . . . . . . . . . . . . . . . . . . . . . . . . . 58
3.4.2 Linearizability versus sequential consistency . . . . . . . . 59
3.5 Quiescent consistency . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59
3.5.1 Properties of quiescent consistency . . . . . . . . . . . . . . . 60
3.6 Formal definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
3.6.1 Histories . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
3.6.2 Linearizability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
3.6.3 Linearizability is compositional . . . . . . . . . . . . . . . . . . 63
3.6.4 Linearizability is nonblocking . . . . . . . . . . . . . . . . . . . 63
3.7 Memory consistency models . . . . . . . . . . . . . . . . . . . . . . . . . 64
3.8 Progress conditions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64
3.8.1 Wait-freedom . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65
3.8.2 Lock-freedom . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65
3.8.3 Obstruction-freedom . . . . . . . . . . . . . . . . . . . . . . . . . . 66
3.8.4 Blocking progress conditions . . . . . . . . . . . . . . . . . . . 67
3.8.5 Characterizing progress conditions . . . . . . . . . . . . . . . 67
3.9 Remarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68
3.10 Chapter notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69
3.11 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70
CHAPTER 4 Foundations of shared memory . . . . . . . . . . . . . . . . . . 75
4.1 The space of registers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76
4.2 Register constructions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81
4.2.1 Safe MRSW registers . . . . . . . . . . . . . . . . . . . . . . . . . 82
4.2.2 A regular Boolean MRSW register . . . . . . . . . . . . . . . 83
4.2.3 A regular M-valued MRSW register . . . . . . . . . . . . . . 84
4.2.4 An atomic SRSW register . . . . . . . . . . . . . . . . . . . . . . 85
4.2.5 An atomic MRSW register . . . . . . . . . . . . . . . . . . . . . 87
4.2.6 An atomic MRMW register . . . . . . . . . . . . . . . . . . . . . 90
4.3 Atomic snapshots . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92
4.3.1 An obstruction-free snapshot . . . . . . . . . . . . . . . . . . . . 92
4.3.2 A wait-free snapshot . . . . . . . . . . . . . . . . . . . . . . . . . . 93
4.3.3 Correctness arguments . . . . . . . . . . . . . . . . . . . . . . . . 97
4.4 Chapter notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98
4.5 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99
CHAPTER 5 The relative power of primitive synchronization
operations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103
5.1 Consensus numbers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104
5.1.1 States and valence . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105
5.2 Atomic registers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106
5.3 Consensus protocols . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109
5.4 FIFO queues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110
 Contents ix

5.5 Multiple assignment objects . . . . . . . . . . . . . . . . . . . . . . . . . . 113

5.6 Read–modify–write operations . . . . . . . . . . . . . . . . . . . . . . . 116
5.7 Common2 RMW operations . . . . . . . . . . . . . . . . . . . . . . . . . 117
5.8 The compareAndSet operation . . . . . . . . . . . . . . . . . . . . . . . . . 119
5.9 Chapter notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120
5.10 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121
CHAPTER 6 Universality of consensus . . . . . . . . . . . . . . . . . . . . . . . 129
6.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129
6.2 Universality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 130
6.3 A lock-free universal construction . . . . . . . . . . . . . . . . . . . . . 130
6.4 A wait-free universal construction . . . . . . . . . . . . . . . . . . . . . 134
6.5 Chapter notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140
6.6 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141

PART 2 Practice
CHAPTER 7 Spin locks and contention . . . . . . . . . . . . . . . . . . . . . . . 147
7.1 Welcome to the real world . . . . . . . . . . . . . . . . . . . . . . . . . . . 147
7.2 Volatile fields and atomic objects . . . . . . . . . . . . . . . . . . . . . . 150
7.3 Test-and-set locks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 150
7.4 Exponential back-off . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 154
7.5 Queue locks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 156
7.5.1 Array-based locks . . . . . . . . . . . . . . . . . . . . . . . . . . . . 156
7.5.2 The CLH queue lock . . . . . . . . . . . . . . . . . . . . . . . . . . 159
7.5.3 The MCS queue lock . . . . . . . . . . . . . . . . . . . . . . . . . . 161
7.6 A queue lock with timeouts . . . . . . . . . . . . . . . . . . . . . . . . . . 163
7.7 Hierarchical locks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 166
7.7.1 A hierarchical back-off lock . . . . . . . . . . . . . . . . . . . . 167
7.7.2 Cohort locks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 167
7.7.3 A cohort lock implementation . . . . . . . . . . . . . . . . . . . 170
7.8 A composite lock . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 171
7.9 A fast path for threads running alone . . . . . . . . . . . . . . . . . . . 178
7.10 One lock to rule them all . . . . . . . . . . . . . . . . . . . . . . . . . . . . 180
7.11 Chapter notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 180
7.12 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 181
CHAPTER 8 Monitors and blocking synchronization . . . . . . . . . . . 183
8.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 183
8.2 Monitor locks and conditions . . . . . . . . . . . . . . . . . . . . . . . . . 183
8.2.1 Conditions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 185
8.2.2 The lost-wakeup problem . . . . . . . . . . . . . . . . . . . . . . 187
8.3 Readers–writers locks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 189
8.3.1 Simple readers–writers lock . . . . . . . . . . . . . . . . . . . . 190
8.3.2 Fair readers–writers lock . . . . . . . . . . . . . . . . . . . . . . . 192
x Contents

8.4 Our own reentrant lock . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 194

8.5 Semaphores . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 194
8.6 Chapter notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 196
8.7 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 197
CHAPTER 9 Linked lists: The role of locking . . . . . . . . . . . . . . . . . 201
9.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 201
9.2 List-based sets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 202
9.3 Concurrent reasoning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 204
9.4 Coarse-grained synchronization . . . . . . . . . . . . . . . . . . . . . . . 206
9.5 Fine-grained synchronization . . . . . . . . . . . . . . . . . . . . . . . . . 207
9.6 Optimistic synchronization . . . . . . . . . . . . . . . . . . . . . . . . . . 211
9.7 Lazy synchronization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 215
9.8 Nonblocking synchronization . . . . . . . . . . . . . . . . . . . . . . . . . 220
9.9 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 225
9.10 Chapter notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 226
9.11 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 226
CHAPTER 10 Queues, memory management, and the ABA
problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 229
10.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 229
10.2 Queues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 230
10.3 A bounded partial queue . . . . . . . . . . . . . . . . . . . . . . . . . . . . 230
10.4 An unbounded total queue . . . . . . . . . . . . . . . . . . . . . . . . . . . 235
10.5 A lock-free unbounded queue . . . . . . . . . . . . . . . . . . . . . . . . 236
10.6 Memory reclamation and the ABA problem . . . . . . . . . . . . . . 240
10.6.1 A naïve synchronous queue . . . . . . . . . . . . . . . . . . . . . 244
10.7 Dual data structures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 244
10.8 Chapter notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 248
10.9 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 248
CHAPTER 11 Stacks and elimination . . . . . . . . . . . . . . . . . . . . . . . . . . 251
11.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 251
11.2 An unbounded lock-free stack . . . . . . . . . . . . . . . . . . . . . . . . 251
11.3 Elimination . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 254
11.4 The elimination back-off stack . . . . . . . . . . . . . . . . . . . . . . . . 255
11.4.1 A lock-free exchanger . . . . . . . . . . . . . . . . . . . . . . . . . 255
11.4.2 The elimination array . . . . . . . . . . . . . . . . . . . . . . . . . 257
11.5 Chapter notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 260
11.6 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 261
CHAPTER 12 Counting, sorting, and distributed coordination . . . 265
12.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 265
12.2 Shared counting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 265
12.3 Software combining . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 266
12.3.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 267
 Contents xi

12.3.2 An extended example . . . . . . . . . . . . . . . . . . . . . . . . . 274

12.3.3 Performance and robustness . . . . . . . . . . . . . . . . . . . . 275
12.4 Quiescently consistent pools and counters . . . . . . . . . . . . . . . 276
12.5 Counting networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 276
12.5.1 Networks that count . . . . . . . . . . . . . . . . . . . . . . . . . . 276
12.5.2 The bitonic counting network . . . . . . . . . . . . . . . . . . . 279
12.5.3 Performance and pipelining . . . . . . . . . . . . . . . . . . . . . 287
12.6 Diffracting trees . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 288
12.7 Parallel sorting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 292
12.8 Sorting networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 293
12.8.1 Designing a sorting network . . . . . . . . . . . . . . . . . . . . 294
12.9 Sample sorting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 296
12.10 Distributed coordination . . . . . . . . . . . . . . . . . . . . . . . . . . . . 298
12.11 Chapter notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 299
12.12 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 300
CHAPTER 13 Concurrent hashing and natural parallelism . . . . . . 305
13.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 305
13.2 Closed-address hash sets . . . . . . . . . . . . . . . . . . . . . . . . . . . . 306
13.2.1 A coarse-grained hash set . . . . . . . . . . . . . . . . . . . . . . 308
13.2.2 A striped hash set . . . . . . . . . . . . . . . . . . . . . . . . . . . . 310
13.2.3 A refinable hash set . . . . . . . . . . . . . . . . . . . . . . . . . . . 311
13.3 A lock-free hash set . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 315
13.3.1 Recursive split-ordering . . . . . . . . . . . . . . . . . . . . . . . 315
13.3.2 The BucketList class . . . . . . . . . . . . . . . . . . . . . . . . . . 318
13.3.3 The LockFreeHashSet<T> class . . . . . . . . . . . . . . . . . . . 319
13.4 An open-address hash set . . . . . . . . . . . . . . . . . . . . . . . . . . . . 323
13.4.1 Cuckoo hashing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 323
13.4.2 Concurrent cuckoo hashing . . . . . . . . . . . . . . . . . . . . . 324
13.4.3 Striped concurrent cuckoo hashing . . . . . . . . . . . . . . . 329
13.4.4 A refinable concurrent cuckoo hash set . . . . . . . . . . . . 331
13.5 Chapter notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 332
13.6 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 334
CHAPTER 14 Skiplists and balanced search . . . . . . . . . . . . . . . . . . . 335
14.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 335
14.2 Sequential skiplists . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 335
14.3 A lock-based concurrent skiplist . . . . . . . . . . . . . . . . . . . . . . 337
14.3.1 A bird’s-eye view . . . . . . . . . . . . . . . . . . . . . . . . . . . . 337
14.3.2 The algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 339
14.4 A lock-free concurrent skiplist . . . . . . . . . . . . . . . . . . . . . . . . 345
14.4.1 A bird’s-eye view . . . . . . . . . . . . . . . . . . . . . . . . . . . . 345
14.4.2 The algorithm in detail . . . . . . . . . . . . . . . . . . . . . . . . 348
14.5 Concurrent skiplists . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 355
14.6 Chapter notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 356
xii Contents

14.7 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 356

CHAPTER 15 Priority queues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 359
15.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 359
15.1.1 Concurrent priority queues . . . . . . . . . . . . . . . . . . . . . 359
15.2 An array-based bounded priority queue . . . . . . . . . . . . . . . . . 360
15.3 A tree-based bounded priority queue . . . . . . . . . . . . . . . . . . . 361
15.4 An unbounded heap-based priority queue . . . . . . . . . . . . . . . . 363
15.4.1 A sequential heap . . . . . . . . . . . . . . . . . . . . . . . . . . . . 363
15.4.2 A concurrent heap . . . . . . . . . . . . . . . . . . . . . . . . . . . . 365
15.5 A skiplist-based unbounded priority queue . . . . . . . . . . . . . . . 371
15.6 Chapter notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 374
15.7 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 375
CHAPTER 16 Scheduling and work distribution . . . . . . . . . . . . . . . . 377
16.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 377
16.2 Analyzing parallelism . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 384
16.3 Realistic multiprocessor scheduling . . . . . . . . . . . . . . . . . . . . 387
16.4 Work distribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 389
16.4.1 Work stealing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 389
16.4.2 Yielding and multiprogramming . . . . . . . . . . . . . . . . . 390
16.5 Work-stealing deques . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 390
16.5.1 A bounded work-stealing deque . . . . . . . . . . . . . . . . . 391
16.5.2 An unbounded work-stealing deque . . . . . . . . . . . . . . . 395
16.5.3 Work dealing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 397
16.6 Chapter notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 400
16.7 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 401
CHAPTER 17 Data parallelism . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 405
17.1 MapReduce . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 407
17.1.1 The MapReduce framework . . . . . . . . . . . . . . . . . . . . . . 408
17.1.2 A MapReduce-based WordCount application . . . . . . . . . . 410
17.1.3 A MapReduce-based KMeans application . . . . . . . . . . . . . 411
17.1.4 The MapReduce implementation . . . . . . . . . . . . . . . . . . 411
17.2 Stream computing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 414
17.2.1 A stream-based WordCount application . . . . . . . . . . . . . 416
17.2.2 A stream-based KMeans application . . . . . . . . . . . . . . . 417
17.2.3 Making aggregate operations parallel . . . . . . . . . . . . . 419
17.3 Chapter notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 422
17.4 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 423
CHAPTER 18 Barriers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 431
18.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 431
18.2 Barrier implementations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 432
18.3 Sense reversing barrier . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 433
18.4 Combining tree barrier . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 434
 Contents xiii

18.5 Static tree barrier . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 436

18.6 Termination detection barriers . . . . . . . . . . . . . . . . . . . . . . . . 438
18.7 Chapter notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 442
18.8 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 443
CHAPTER 19 Optimism and manual memory management . . . . . . 451
19.1 Transitioning from Java to C++ . . . . . . . . . . . . . . . . . . . . . . . 451
19.2 Optimism and explicit reclamation . . . . . . . . . . . . . . . . . . . . . 451
19.3 Protecting pending operations . . . . . . . . . . . . . . . . . . . . . . . . 454
19.4 An object for managing memory . . . . . . . . . . . . . . . . . . . . . . 455
19.5 Traversing a list . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 456
19.6 Hazard pointers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 458
19.7 Epoch-based reclamation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 462
19.8 Chapter notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 465
19.9 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 466
CHAPTER 20 Transactional programming . . . . . . . . . . . . . . . . . . . . . 467
20.1 Challenges in concurrent programming . . . . . . . . . . . . . . . . . 467
20.1.1 Problems with locking . . . . . . . . . . . . . . . . . . . . . . . . . 467
20.1.2 Problems with explicit speculation . . . . . . . . . . . . . . . 468
20.1.3 Problems with nonblocking algorithms . . . . . . . . . . . . 470
20.1.4 Problems with compositionality . . . . . . . . . . . . . . . . . 471
20.1.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 472
20.2 Transactional programming . . . . . . . . . . . . . . . . . . . . . . . . . . 472
20.2.1 An example of transactional programming . . . . . . . . . . 473
20.3 Hardware support for transactional programming . . . . . . . . . . 475
20.3.1 Hardware speculation . . . . . . . . . . . . . . . . . . . . . . . . . 475
20.3.2 Basic cache coherence . . . . . . . . . . . . . . . . . . . . . . . . . 475
20.3.3 Transactional cache coherence . . . . . . . . . . . . . . . . . . . 476
20.3.4 Limitations of hardware support . . . . . . . . . . . . . . . . . 477
20.4 Transactional lock elision . . . . . . . . . . . . . . . . . . . . . . . . . . . 478
20.4.1 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 480
20.5 Transactional memory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 481
20.5.1 Run-time scheduling . . . . . . . . . . . . . . . . . . . . . . . . . . 482
20.5.2 Explicit self-abort . . . . . . . . . . . . . . . . . . . . . . . . . . . . 483
20.6 Software transactions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 483
20.6.1 Transactions with ownership records . . . . . . . . . . . . . . 485
20.6.2 Transactions with value-based validation . . . . . . . . . . . 490
20.7 Combining hardware and software transactions . . . . . . . . . . . 492
20.8 Transactional data structure design . . . . . . . . . . . . . . . . . . . . . 493
20.9 Chapter notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 494
20.10 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 494
APPENDIX A Software basics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 497
A.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 497
A.2 Java . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 497
xiv Contents

A.2.1 Threads . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 497

A.2.2 Monitors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 498
A.2.3 Yielding and sleeping . . . . . . . . . . . . . . . . . . . . . . . . . 501
A.2.4 Thread-local objects . . . . . . . . . . . . . . . . . . . . . . . . . . 502
A.2.5 Randomization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 503
A.3 The Java memory model . . . . . . . . . . . . . . . . . . . . . . . . . . . . 504
A.3.1 Locks and synchronized blocks . . . . . . . . . . . . . . . . . . 505
A.3.2 Volatile fields . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 506
A.3.3 Final fields . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 506
A.4 C++ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 508
A.4.1 Threads in C++ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 508
A.4.2 Locks in C++ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 509
A.4.3 Condition variables . . . . . . . . . . . . . . . . . . . . . . . . . . . 510
A.4.4 Atomic variables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 512
A.4.5 Thread-local storage . . . . . . . . . . . . . . . . . . . . . . . . . . 513
A.5 C# . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 514
A.5.1 Threads . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 514
A.5.2 Monitors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 515
A.5.3 Thread-local objects . . . . . . . . . . . . . . . . . . . . . . . . . . 517
A.6 Appendix notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 518
APPENDIX B Hardware basics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 519
B.1 Introduction (and a puzzle) . . . . . . . . . . . . . . . . . . . . . . . . . . 519
B.2 Processors and threads . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 522
B.3 Interconnect . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 522
B.4 Memory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 523
B.5 Caches . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 523
B.5.1 Coherence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 524
B.5.2 Spinning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 526
B.6 Cache-conscious programming, or the puzzle solved . . . . . . . 526
B.7 Multicore and multithreaded architectures . . . . . . . . . . . . . . . 527
B.7.1 Relaxed memory consistency . . . . . . . . . . . . . . . . . . . 528
B.8 Hardware synchronization instructions . . . . . . . . . . . . . . . . . . 529
B.9 Appendix notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 530
B.10 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 531

Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 533
Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 541
Preface

In the decade since the first edition, this book has become a staple of undergraduate
and graduate courses at universities around the world. It has also found a home on
the bookshelves of practitioners at companies large and small. The audience for the
book has, in turn, advanced the state of the art in multiprocessor programming. In
this second edition, we aim to continue this “virtuous cycle” by providing new and
updated content. Our goal is the same as with the first edition: to provide a textbook
for a senior-level undergraduate course and a reference for practitioners.

Organization
The first part of this book covers the principles of concurrent programming, show-
ing how to think as a concurrent programmer, developing fundamental skills such
as understanding when operations “happen,” considering all possible interleavings,
and identifying impediments to progress. Like many skills—driving a car, cooking
a meal, or appreciating caviar—thinking concurrently must be cultivated, and it can
be learned with moderate effort. Readers who want to start programming right away
may skip most of this section but should still read Chapters 2 and 3, which cover the
basic ideas necessary to understand the rest of the book.
We first look at the classic mutual exclusion problem (Chapter 2). This chapter
is essential for understanding why concurrent programming is a challenge. It covers
basic concepts such as fairness and deadlock. We then ask what it means for a con-
current program to be correct (Chapter 3). We consider several alternative conditions
and the circumstances under which one might want to use each one. We examine
the properties of shared memory essential to concurrent computation (Chapter 4),
and we look at the kinds of synchronization primitives needed to implement highly
concurrent data structures (Chapters 5 and 6).
We think it is essential that anyone who wants to become truly skilled in the art
of multiprocessor programming spend time solving the problems presented in the
first part of this book. Although these problems are idealized, they distill the kind
of thinking necessary to write effective multiprocessor programs. Most importantly,
they distill the style of thinking necessary to avoid the common mistakes committed
by nearly all novice programmers when they first encounter concurrency.
The second part of the book describes the practice of concurrent programming.
For most of this part, we give examples in Java to avoid getting mired in low-level
details. However, we have expanded this edition to include discussion of some low-
level issues that are essential to understanding multiprocessor systems and how to
program them effectively. We use examples in C++ to illustrate these issues.

xv
Discovering Diverse Content Through
Random Scribd Documents
authority whatever in his own country; for as serki-n-turáwa he had
to levy the tax of ten mithkáls on every camel-load of merchandise,
and this he is said to have done with some degree of severity. After
a long conversation on the steps of the terrace, we parted, the best
possible friends.
Not so pleasant to me, though not without interest, was the visit
of another great man—Belróji, the támberi or war-chieftain of the
Ighólar Im-esághlar. He was still in his prime, but my Kél-owí (who
were always wrangling like children) got up a desperate fight with
him in my very room, which was soon filled with clouds of dust; and
the young Slimán entered during the row, and joining in it, it
became really frightful. The Kél-owí were just like children; when
they went out they never failed to put on all their finery, which they
threw off as soon as they came within doors, resuming their old dirty
clothes.
It was my custom in the afternoon, when the sun had set behind
the opposite buildings, to walk up and down in front of our house;
and while so doing to-day I had a long conversation with two chiefs
of the Itísan on horseback, who came to see me, and avowed their
sincere friendship and regard. They were fine, tall men, but rather
slim, with a noble expression of countenance and of light colour.
Their dress was simple but handsome, and arranged with great care.
All the Tuarek, from Ghát as far as Háusa, and from Alákkos to
Timbúktu, are passionately fond of the tobes and trousers called
“tailelt” (the Guinea-fowl), or “filfil” (the pepper), on account of their
speckled colour. They are made of silk and cotton interwoven, and
look very neat. The lowest part of the trousers, which forms a
narrow band about two inches broad, closing rather tightly, is
embroidered in different colours. None of the Tuarek of pure blood
would, I think, degrade themselves by wearing on their head the red
cap.
Monday, October 21.—Early in the morning I went with Hámma to
take leave of the Sultan, who had been too busy for some days to
favour me with an audience; and I urged my friend to speak of the
treaty though I was myself fully aware of the great difficulty which
so complicated a paper, written in a form entirely unknown to the
natives, and which must naturally be expected to awaken their
suspicion, would create, and of the great improbability of its being
signed while the Sultan was pressed with a variety of business. On
the way to the fáda we met Áshu, the present serki-n-turáwa, a
large-sized man, clad in an entirely white dress, which may not
improbably be a sign of his authority over the white men (Turáwa).
He is said to be a very wealthy man. He replied to my compliments
with much kindness, entered into conversation with me about the
difference of our country and theirs, and ordered one of his
companions to take me to a small garden which he had planted near
his house in the midst of the town, in order to see what plants we
had in common with them. Of course there was nothing like our
plants; and my cicerone conceived rather a poor idea of our country
when he heard that all the things which they had we had not—
neither senna, nor bamia, nor indigo, nor cotton, nor Guinea-corn,
nor, in short, the most beautiful of all trees of the creation, as he
thought—the talha, or Mimosa ferruginea; and he seemed rather
incredulous when told that we had much finer plants than they.
We then went to the fáda. The Sultan seemed quite ready for
starting. He was sitting in the courtyard of his palace, surrounded by
a multitude of people and camels, while the loud murmuring noise of
a number of schoolboys who were learning the Kurán proceeded
from the opposite corner, and prevented my hearing the
conversation of the people. The crowd and the open locality were, of
course, not very favourable to my last audience, and it was
necessarily a cold one. Supported by Hámma, I informed the Sultan
that I expected still to receive a letter from him to the Government
under whose auspices I was travelling, expressive of the pleasure
and satisfaction he had felt in being honoured with a visit from one
of the mission, and that he would gladly grant protection to any
future traveller who should happen to visit his country. The Sultan
promised that such a letter should be written; however, the result
proved that either he had not quite understood what I meant, or,
what is more probable, that in his precarious situation he felt himself
not justified in writing to a Christian government, especially as he
had received no letter from it.
When I had returned to my quarters, Hámma brought me three
letters, in which ʿAbd el Káder recommended my person and my
luggage to the care of the Governors of Kanó, Kátsena, and Dáura,
and which were written in rather incorrect Arabic, and in nearly the
same terms. They were as follows:—

“In the name of God, etc.

“From the Emír of Ahír, ʿAbd el Káder, son of the Sultan
Mohammed el Bákeri, to the Emír of Dáura, son of the late
Emír of Dáura, Is-hhák. The mercy of God upon the eldest
companions of the Prophet, and His blessing upon the
Khalífa; ‘Amín.’ The most lasting blessing and the highest
wellbeing to you without end. I send this message to you
with regard to a stranger, my guest, of the name of ʿAbd el
Kerím, who came to me, and is going to the Emír el Mumenín
[the Sultan of Sókoto], in order that, when he proceeds to
you, you may protect him and treat him well, so that none of
the freebooters and evil-doers may hurt him or his property,
but that he may reach the Emír el Mumenín. Indeed, we
wrote this on account of the freebooters, in order that you
may protect him against them in the most efficacious manner.
Farewell.”

These letters were all sealed with the seal of the Sultan.
Hámma showed me also another letter which he had received
from the Sultan, and which I think interesting enough to be here
inserted, as it is a faithful image of the turbulent state of the country
at that time, and as it contains the simple expression of the sincere
and just proceedings of the new Sultan. Its purport was as follows,
though the language in which it is written is so incorrect that several
passages admit of different interpretations:—
 “In the name of God, etc.
“From the Commander, the faithful Minister of Justice, the
Sultan ʿAbd el Káder, son of the Sultan Mohammed el Bákeri,
to the chiefs of all the tribe of Eʾ Núr, and Hámed, and Sëis,
and all those among you who have large possessions, perfect
peace to you.
“Your eloquence, compliments, and information are
deserving of praise. We have seen the auxiliaries sent to us
by your tribe, and we have taken energetic measures with
them against the marauders, who obstruct the way of the
caravans of devout people, and the intercourse of those who
travel, as well as those who remain at home. On this account
we desire to receive aid from you against their incursions.
The people of the Kél-fadaye, they are the marauders. We
should not have prohibited their chiefs to exercise rule over
them, except for three things: first, because I am afraid they
will betake themselves from the Aníkel [the community of the
people of Aír] to the Awelímmiden; secondly, in order that
they may not make an alliance with them against us, for they
are all marauders; and thirdly, in order that you may approve
of their paying us the tribute. Come, then, to us quickly. You
know that what the hand holds it holds only with the aid of
the fingers; for without the fingers the hand can seize
nothing.
“We therefore will expect your determination, that is to say
your coming, after the departure of the salt-caravan of the
Itísan, fixed among you for the fifteenth of the month. God!
God is merciful and answereth prayer! Come therefore to us,
and we will tuck up our sleeves, and drive away the
marauders, and fight valiantly against them as God (be He
glorified!) hath commanded.
“Lo, corruption hath multiplied on the face of the earth!
May the Lord not question us on account of the poor and
needy, orphans and widows, according to His word: ‘You are
 all herdsmen, and ye shall all be questioned respecting your
herds, whether ye have indeed taken good care of them or
dried them up.’
“Delay not, therefore, but hasten to our residence, where
we are all assembled; for ‘zeal in the cause of religion is the
duty of all;’ or send thy messenger to us quickly with a
positive answer; send thy messenger as soon as possible.
Farewell!”

The whole population was in alarm, and everybody who was able
to bear arms prepared for the expedition. About sunset the “égehen”
left the town, numbering about four hundred men, partly on camels,
partly on horseback, besides the people on foot. Bóro as well as
Áshu accompanied the Sultan, who this time was himself mounted
on a camel. They went to take their encampment near that of
Astáfidet, in Tagúrast, ʿAbd el Káder pitching a tent of grey colour,
and in size like that of a Turkish aghá, in the midst of the Kél-gerés,
the Kél-ferwán, and the Emgedesíye; while Astáfidet, who had no
tent, was surrounded by the Kél-owí. The Sultan was kind and
attentive enough not to forget me even now; and having heard that
I had not yet departed, Hámma not having finished his business in
the town, he sent me some wheat, a large botta with butter and
vegetables (chiefly melons and cucumbers), and the promise of
another sheep.
In the evening the drummer again went his rounds through the
town, proclaiming the strict order of the Sultan that everybody
should lay in a large supply of provisions. Although the town in
general had become very silent when deserted by so many people,
our house was kept in constant bustle, and in the course of the night
three mehára came from the camp, with people who could get no
supper there, and sought it with us. Bóro sent a messenger to me
early the next morning, urgently begging for a little powder, as the
“Mehárebín” of the Imghád had sent off their camels and other
property, and were determined to resist the army of the Sultan.
However, I could send him but very little. My amusing friend
Mohammed spent the whole day with us, when he went to join the
ghazzia. I afterwards learnt that he obtained four head of cattle as
his share. There must be considerable herds of cattle in the more
favoured valleys of Asben; for the expedition had nothing else to live
upon, as Mohammed afterwards informed me, and slaughtered an
immense quantity of them. Altogether, the expedition was
successful, and the Fádë-ang and many tribes of the Imghád lost
almost all their property. Even the influential Háj Beshír was
punished, on account of his son having taken part in the expedition
against us. I received also the satisfactory information that ʿAbd el
Káder had taken nine camels from the man who retained my méheri;
but I gained nothing thereby, neither my own camel being returned
nor another given me in its stead. The case was the same with all
our things; but nevertheless the proceeding had a good effect,
seeing that people were punished expressly for having robbed
Christians, and thus the principle was established that it was not less
illegal to rob Christians than it was to rob Mohammedans, both
creeds being placed, as far as regards the obligations of peace and
honesty, on equally favourable terms.
Tuesday, October 22.—I spent the whole of Tuesday in my house,
principally in taking down information which I received from the
intelligent Ghadámsi merchant Mohammed, who, having left his
native town from fear of the Turks, had resided six years in Ágades,
and was a well-informed man.
Wednesday, October 23.—My old friend the blacksmith Hámmeda,
and the tall Elíyas, went off this morning with several camels laden
with provisions, while Hámma still stayed behind to finish the
purchases; for on account of the expedition, and the insecure state
of the road to Damerghú, it had been difficult to procure provisions
in sufficient quantity. Our house therefore became almost as silent
and desolate as the rest of the town; but I found a great advantage
in remaining a few days longer, for my chivalrous friend and
protector, who, as long as the Sultan and the great men were
present, had been very reserved and cautious, had now no further
scruple about taking me everywhere, and showing me the town
“within and without.”
We first visited the house of Ídder,
a broker, who lived at a short distance
to the south from our house, and had
also lodged Háj ʿAbdúwa during his
stay here. It was a large, spacious
dwelling, well arranged with a view to
comfort and privacy, according to the
conception and customs of the
inhabitants, while our house (being a
mere temporary residence for Ánnur’s
people occasionally visiting the town)
was a dirty, comfortless abode. We
entered first a vestibule, about twenty-five feet long and nine broad,
having on each side a separate space marked off by that low kind of
balustrade mentioned in my description of the Sultan’s house. This
vestibule or ante-room was followed by a second room of larger size
and irregular arrangement; opposite the entrance it opened into
another apartment, which, with two doors, led into a spacious inner
courtyard, which was very irregularly circumscribed by several rooms
projecting into it, while to the left it was occupied by an enormous
bedstead (1). These bedsteads are a most characteristic article of
furniture in all the dwellings of the Sónghay. In Ágades they are
generally very solidly built of thick boards, and furnished with a
strong canopy resting upon four posts, covered with mats on the top
and on three sides, the remaining side being shut in with boards.
Such a canopied bed looks like a little house by itself. On the wall of
the first chamber, which on the right projected into the courtyard,
several lines of large pots had been arranged, one above the other
(2), forming so many warm nests for a number of turtle-doves which
were playing all about the courtyard; while on the left, in the half-
decayed walls of two other rooms (3), about a dozen goats were
fastened each to a separate pole. The background of the courtyard
contained several rooms, and in front of it a large shade (4) had
been built of mats, forming a rather pleasant and cool resting-place.
Numbers of children were gambolling about, who gave to the whole
a very cheerful appearance. There is something very peculiar in
these houses, which are constructed evidently with a view to
comfort and quiet enjoyment.
We then went to visit a female friend of Hámma, who lived in the
south quarter of the town, in a house which likewise bespoke much
comfort; but here, on account of the number of inmates, the
arrangement was different, the second vestibule being furnished on
each side with a large bedstead instead of mats, though here also
there was in the courtyard an immense bedstead. The courtyard was
comparatively small, and a long corridor on the left of it led to an
inner courtyard or “tsakangída,” which I was not allowed to see. The
mistress of the house was still a very comely person, although she
had borne several children. She had a fine figure, though rather
under the middle size, and a fair complexion. I may here remark that
many of the women of Ágades are not a shade darker than Arab
women in general. She wore a great quantity of silver ornaments,
and was well dressed in a gown of coloured cotton and silk. Hámma
was very intimate with her, and introduced me to her as his friend
and protégé, whom she ought to value as highly as himself. She was
married, but her husband was residing in Kátsena, and she did not
seem to await his return in the Penelopean style. The house had as
many as twenty inmates, there being no less than six children, I
think, under five years of age, and among them a very handsome
little girl, the mother’s favourite; besides, there were six or seven
full-grown slaves. The children were all naked, but wore ornaments
of beads and silver.
After we had taken leave of this Emgedesíye lady, we followed the
street towards the south, where there were some very good houses,
although the quarter in general was in ruins; and here I saw the
very best and most comfortable-looking dwelling in the town. All the
pinnacles were ornamented with ostrich eggs. One will often find in
an eastern town, after the first impression of its desolate appearance
is gone by, many proofs that the period of its utter prostration is not
yet come, but that even in the midst of the ruins there is still a good
deal of ease and comfort. Among the ruins of the southern quarter
are to be seen the pinnacled walls of a building of immense
circumference and considerable elevation; but unfortunately I could
not learn from Hámma for what purpose it had been used; however,
it was certainly a public building, and probably a large khán rather
than the residence of the chief. With its high, towering walls, it still
forms a sort of outwork on the south side of the town, where in
general the wall is entirely destroyed, and the way is everywhere
open. Hámma had a great prejudice against this desolate quarter.
Even the more intelligent Mohammedans are often afraid to enter
former dwelling-places of men, believing them to be haunted by
spirits; but he took me to some inhabited houses, which were all
built on the same principle as that described, but varying greatly in
depth and in the size of the courtyard; the staircases (abi-n-háwa)
leading to the upper story are in the courtyard, and are rather
irregularly built of stones and clay. In some of them young ostriches
were running about. The inhabitants of all the houses seemed to
have the same cheerful disposition, and I was glad to find scarcely a
single instance of misery. I give here the ground-plan of another
house.
The artisans who work in leather (an
occupation left entirely to females) seem to live
in a quarter by themselves, which originally was
quite separated from the rest of the town by a
sort of gate; but I did not make a sufficient
survey of this quarter to mark it distinctly on the
ground-plan of the town. We also visited some
of the mat-makers.
Our maimólo of the other day, who had
discovered that we had slaughtered our sheep,
paid us a visit in the evening, and for a piece of meat entertained
me with a clever performance on his instrument, accompanied with
a song. Hámma spent his evening with our friend the Emgedesíye
lady, and was kind enough to beg me to accompany him. This I
declined, but gave him a small present to take to her.
I had a fair sample of the state of morals in Ágades the following
day, when five or six girls and women came to pay me a visit in our
house, and with much simplicity invited me to make merry with
them, there being now, as they said, no longer reason for reserve,
“as the Sultan was gone.” It was indeed rather amusing to see what
conclusions they drew from the motto “Serki yátafi.” Two of them
were tolerably pretty and well-formed, with fine black hair hanging
down in plaits or tresses, lively eyes, and very fair complexion. Their
dress was decent, and that of one of them even elegant, consisting
of an under-gown reaching from the neck to the ankles, and an
upper one drawn over the head, both of white colour; but their
demeanour was very free, and I too clearly understood the caution
requisite in a European who would pass through these countries
unharmed and respected by the natives, to allow myself to be
tempted by these wantons. It would be better for a traveller in these
regions, both for his own comfort and for the respect felt for him by
the natives, if he could take his wife with him; for these simple
people do not understand how a man can live without a partner. The
Western Tuarek, who in general are very rigorous in their manners,
and quite unlike the Kél-owí, had nothing to object against me
except my being a bachelor. But as it is difficult to find a female
companion for such journeys, and as by marrying a native he would
expose himself to much trouble and inconvenience on the score of
religion, he will do best to maintain the greatest austerity of
manners with regard to the other sex, though he may thereby
expose himself to a good deal of derision from some of the lighter-
hearted natives. The ladies, however, became so troublesome that I
thought it best to remain at home for a few days, and was thus
enabled at the same time to note down the information which I had
been able to pick up. During these occupations I was greatly pleased
with the companionship of a diminutive species of finches which
frequent all the rooms in Ágades, and, as I may add from later
experience, in Timbúktu also; the male, with its red neck, in
particular looks extremely pretty. The poults were just about to
fledge.
Sunday, October 27.—There was one very characteristic building in
the town, which, though a most conspicuous object from the terrace
of our house, I had never yet investigated with sufficient accuracy.
This was the mesállaje, or high tower rising over the roof of the
mosque. The reason why this building in particular (the most famous
and remarkable one in the town) had been hitherto observed by me
only from a distance, and in passing by, must be obvious. Difference
of religious creed repelled me from it; and so long as the town was
full of strangers, some of them very fanatical, it was dangerous for
me to approach it too closely. I had often inquired whether it would
not be possible to ascend the tower without entering the mosque;
but I had always received for answer that the entrance was locked
up. As soon, however, as the Sultan was gone, and when the town
became rather quiet, I urged Hámma to do his best that I might
ascend to the top of this curious building, which I represented to
him as a matter of the utmost importance to me, since it would
enable me not only to control my route by taking a few angles of the
principal elevations round the valley Aúderas, but also to obtain a
distant view over the country towards the west and south, which it
was not my good luck to visit myself. To-day Hámma promised me
that he would try what could be done.
 Having once more visited the lively house of Ídder, we took our
way over the market-places, which were now rather dull. The
vultures looked out with visible greediness and eagerness from the
pinnacles of the ruined walls around for their wonted food—their
share of offal during these days, when so many people were absent,
being of course much reduced, though some of them probably had
followed their fellow-citizens on the expedition. So few people being
in the streets, the town had a more ruined look than ever, and the
large heap of rubbish accumulated on the south side of the butchers’
market seemed to me more disgusting than before. We kept along
the principal street between Dígi and Arrafíya, passing the deep well
Shedwánka on our right, and on the other side a school, which
resounded with the shrill voices of about fifty little boys repeating
with energy and enthusiasm the verses of the Kurán, which their
master had written for them upon their little wooden tablets. Having
reached the open space in front of the mosque, and there being
nobody to disturb me, I could view at my leisure this simple but
curious building, which in the subsequent course of my journey
became still more interesting to me, as I saw plainly that it was built
on exactly the same principle as the tower which rises over the
sepulchre of the famed conqueror Háj Mohammed Áskiá (the
“Ischia” of Leo).
The mesállaje starts up from the platform or terrace formed by the
roof of the mosque, which is extremely low, resting apparently, as
we shall see, in its interior, upon four massive pillars. It is square,
and measures at its base about thirty feet, having a small lean-to, on
its east side, on the terrace of the mosque, where most probably
there was formerly the entrance. From this the tower rises
(decreasing in width, and with a sort of swelling or entasis in the
middle of its elevation, something like the beautiful model adopted
by nature in the deléb palm, and imitated by architects in the
columns of the Ionic and Corinthian orders) to a height of from
ninety to ninety-five feet. It measures at its summit not more than
about eight feet in width. The interior is lighted by seven openings
on each side. Like most of the houses in Ágades, it is built entirely of
clay; and in order to strengthen a building so lofty and of so soft a
material, its four walls are united by thirteen layers of boards of the
dúm-tree, crossing the whole tower in its entire breadth and width,
and coming out on each side from three to four feet, while at the
same time they afford the only means of getting to the top. Its
purpose is to serve as a watch-tower, or at least was so at a former
time, when the town, surrounded by a strong wall and supplied with
water, was well capable of making resistance, if warned in due time
of an approaching danger. But at present it seems rather to be kept
in repair only as a decoration of the town.
The mesállaje in its present state was only six years old at the
time of my visit (in 1850), and perhaps was not even quite finished
in the interior, as I was told that the layers of boards were originally
intended to support a staircase of clay. About fifty paces from the
south-western corner of the mosque, the ruins of an older tower are
seen still rising to a considerable height, though leaning much to one
side, more so than the celebrated Tower of Pisa, and most probably
in a few years it will give way to an attack of storm and rain. This
more ancient tower seems to have stood quite detached from the
mosque.
 Having sufficiently surveyed the exterior of the tower, and made a
sketch of it, I accompanied my impatient companion into the interior
of the mosque, into which he felt no scruple in conducting me. The
lowness of the structure had already surprised me from without; but
I was still more astonished when I entered the interior, and saw that
it consisted of low, narrow naves, divided by pillars of immense
thickness, the reason of which it is not possible at present to
understand, as they have nothing to support but a roof of dúm-tree
boards, mats, and a layer of clay; but I think it scarcely doubtful that
originally these naves were but the vaults or cellars of a grand
superstructure, designed but not executed; and this conjecture
seems to be confirmed by all that at present remains of the mosque.
The gloomy halls were buried in a mournful silence, interrupted only
by the voice of a solitary man, seated on a dirty mat at the western
wall of the tower, and reading diligently the torn leaves of a
manuscript. Seeing that it was the kádhi, we went up to him and
saluted him most respectfully; but it was not in the most cheerful
and amiable way that he received our compliments—mine in
particular—continuing to read, and scarcely raising his eyes from the
sheets before him. Hámma then asked for permission to ascend the
tower, but received a plain and unmistakable refusal, the thing being
impossible, there being no entrance to the tower at present. It was
shut up, he said, on account of the Kél-gerés, who used to ascend
the tower in great numbers. Displeased with his uncourteous
behaviour, and seeing that he was determined not to permit me to
climb the tower, were it ever so feasible, we withdrew and called
upon the imám, who lives in a house attached to these vaults, and
which looked a little neater from having been whitewashed;
however, he had no power to aid us in our purpose, but rather
confirmed the statement of the kádhi. This is the principal mosque
of the town, and seems to have been always so, although there are
said to have been formerly as many as seventy mosques, of which
ten are still in use. They deserve no mention, however, with the
exception of three, the Msíd Míli, Msíd Éheni, and Msíd el Mékki. I
will only add here that the Emgedesíye, so far as their very slender
stock of theological learning and doctrine entitles them to rank with
any sect, are Malekíye, as well as the Kél-owí.
Resigning myself to the disappointment of not being able to
ascend the tower, I persuaded my friend to take a longer walk with
me round the northern quarter of the town. But I forgot to mention
that besides Hámma, I had another companion of a very different
character. This was Zúmmuzuk, a reprobate of the worst description,
and whose features bore distinct impress of the vile and brutal
passions which actuated him; yet being a clever fellow, and (as the
illegitimate son, or “dan néma,” of an Emgédesi woman) fully master
of the peculiar idiom of Ágades, he was tolerated not only by the old
chief Ánnur, who employed him as interpreter, but even by me. How
insolent the knave could be I shall soon have occasion to mention.
With this fellow, therefore, and with Hámma, I continued my walk,
passing the kófa-n-alkáli, and then, from the ruins of the quarter
Ben-Gottára, turning to the north. Here the wall of the town is in a
tolerable state of preservation, but very weak and insufficient,
though it is kept in repair, even to the pinnacles, on account of its
surrounding the palace of the Sultan. Not far from this is an open
space called Azarmádarangh, “the place of execution,” where
occasionally the head of a rebellious chieftain or a murderer is cut
off by the “dóka;” but as far as I could learn, such things happen
very seldom. Even on the north side, two gates are in a tolerable
state of preservation.
Having entered the town from this side, we went to visit the
quarter of the leather-workers, which, as I stated before, seems to
have formed originally a regular ward; all this handicraft, with the
exception of saddle-work, is carried on by women, who work with
great neatness. Very beautiful provision-bags are made here,
although those which I brought back from Timbúktu are much
handsomer. We saw also some fine specimens of mats, woven of a
very soft kind of grass, and dyed of various colours. Unfortunately, I
had but little with me wherewith to buy; and even if I had been able
to make purchases, the destination of our journey being so distant,
there was not much hope of carrying the things safely to Europe.
The blacksmiths’ work of Ágades is also interesting, although showy
and barbarous, and not unlike the work with which the Spaniards
used to adorn their long daggers.
Monday, October 28.—During all this time I prosecuted inquiries
with regard to several subjects connected with the geography and
ethnography of this quarter of the world. I received several visits
from Emgédesi tradesmen, many of whom are established in the
northern provinces of Háusa, chiefly in Kátsena and Tasáwa, where
living is infinitely cheaper than in Ágades. All these I found to be
intelligent men, having been brought up in the centre of intercourse
between a variety of tribes and nations of the most different
organization, and, through the web of routes which join here,
receiving information of distant regions. Several of them had even
made the pilgrimage, and thus come in contact with the relatively
high state of civilization in Egypt and near the coast; and I shall not
easily forget the enlightened view which the mʿallem Háj
Mohammed ʿOmár, who visited me several times, took of Islamism
and Christianity. The last day of my stay in Ágades, he reverted to
the subject of religion, and asked me, in a manner fully expressive of
his astonishment, how it came to pass that the Christians and
Moslemín were so fiercely opposed to one another, although their
creeds, in essential principles, approximated so closely. To this I
replied by saying that I thought the reason was that the great
majority both of Christians and Moslemín paid less regard to the
dogmas of their creeds than to external matters, which have very
little or no reference to religion itself. I also tried to explain to him
that in the time of Mohammed Christianity had entirely lost that
purity which was its original character, and that it had been mixed up
with many idolatrous elements, from which it was not entirely
disengaged till a few centuries ago, while the Mohammedans had
scarcely any acquaintance with Christians except those of the old
sects of the Jacobites and Nestorians. Mutually pleased with our
conversation, we parted from each other with regret.
In the afternoon I was agreeably surprised by the arrival of the
Tinýlkum Ibrahim, for the purpose of supplying his brother’s house
with what was wanted; and being determined to make only one
day’s stay in the town, he had learned with pleasure that we were
about to return by way of Áfasás, the village whither he himself was
going. I myself had cherished this hope, as all the people had
represented that place as one of the largest in the country, and as
pleasantly situated. Hámma had promised to take me this way on
our return to Tin-téllust; but having stayed so much longer in the
town than he had intended, and being afraid of arriving too late for
the salt-caravan of the Kél-owí on their way to Bilma, which he was
to supply with provisions, he changed his plan, and determined to
return by the shortest road. Meanwhile he informed me that the old
chief would certainly not go with us to Zínder till the salt-caravan
had returned from Bilma.
Fortunately, in the course of the 29th a small caravan with corn
arrived from Damerghú, and Hámma completed his purchases. He
had, however, first to settle a disagreeable affair; for our friend
Zúmmuzuk had bought, in Hámma’s name, several things for which
payment was now demanded. Hámma flew into a terrible rage, and
nearly finished the rogue. My Arab and Tawáti friends, who heard
that we were to start the following day, though they were rather
busy buying corn, came to take leave of me, and I was glad to part
from all of them in friendship. But before bidding farewell to this
interesting place, I shall make a few general observations on its
history.
 CHAPTER XVIII.
HISTORY OF ÁGADES

Previously to Mr. Cooley’s perspicuous inquiries into the Negroland

of the Arabs, this place was identified with Aúdaghost, merely on
account of a supposed similarity of name. But Ágades, or rather
Égedesh, is itself a pure Berber word, in no way connected with
Aúdaghost. It is of very frequent occurrence, particularly among the
Awelímmiden, and means “family,” and the name was well chosen
for a town consisting of mixed elements. Moreover, while we find
Aúdaghost in the far west in the twelfth century, we have the
distinct statement of Marmol that Ágades was founded a hundred
and sixty years before the time when he wrote (that is to say, in
1460), the truth of which statement, harmonizing as it does with
Leo’s more general account, that it was a modern town, we have no
reason to doubt. Neither of these authors tells us who built it; but as
we know that the great Sónghay conqueror Háj Mohammed Áskiá,
who conquered the town of Ágades in the year of the Hejra 921, or
1515 of our era, expelled from it the five Berber tribes who,
according to the information collected by me during my stay in
Ágades, and which I shall soon lay before my readers, must have
been long resident in the town, it appears highly probable that these
Berbers were its founders. And if this be assumed, there will be no
difficulty in explaining why the language of the natives of the place
at present is a dialect of the Sónghay language, as it is most
probable that this great and enlightened conqueror, after he had
driven out the old inhabitants, established in this important place a
new colony of his own people. In a similar way we find the Sónghay
nation, which seems not to have originally extended to a great
distance eastward of Gágho or Gógo, now extending into the very
heart of Kébbi, although we shall find other people speaking the
same language in the neighbourhood of Ágades, and perhaps may
be able in the course of our researches to trace some connection
between the Sónghay and ancient Egypt.
It is therefore highly probable that those five Berber tribes formed
the settlement in question as an entrepôt for their commerce with
Negroland, though the foundation of such a grand settlement on the
border of the desert presumes that they had at that time a
preponderating influence in all these regions; and the whole affair is
so peculiar that its history could not fail to gratify curiosity if more
could be known of it. From Bello’s account, it would appear that
they, or at least one of these tribes (the Aújila), conquered the
whole of Aïr.
It is certainly remarkable to see people from five places, separated
from each other by immense tracts, and united only by the bond of
commerce and interest, founding a large colony far away from their
homes and on the very border of the desert. For, according to all
that I could learn by the most sedulous inquiries in Ágades, those
tribes belonged to the Gurára of Tawat, to the Tafimáta, to the Beni
Wazít and the Tésko of Ghadámes, to the once powerful and
numerous tribe of the Masráta, and finally to the Aújila; and as the
names of almost all these different tribes, and of their divisions, are
still attached to localities of the town, we can scarcely doubt the
correctness of this information, and must suppose that Sultan Bello
was mistaken in referring the five tribes (settled in Ágades) to Aújila
alone.
Though nothing is related about the manner in which Háj
Mohammed Áskiá took possession of the town, except that it is
stated distinctly that he drove out the five tribes, it seems, from the
traditions current in Ágades, that a considerable number of the
Berbers, with five hundred “jákhfa” (cages mounted on camels, such
as only wealthy people can afford to keep for carrying their wives),
left the town, but were all massacred. But no one who regards with
the least attention the character of the present population of the
town can doubt for a moment that a considerable number of the
Berber population remained behind, and in course of time mixed
with the Sónghay colonists; for, even if we set aside the
consideration of the language (which is greatly intermixed with
Berber words), there is evidently much Berber blood in the
population even at the present day, a fact which is more evident in
the females than in the males.
It is a pity that Leo says nothing about the language spoken in
Ágades; for he lived just at the very period during which the town,
from a Berber settlement, became a Negro town. His expression
certainly implies that he regarded it as a Negro town. But, while
well-informed in general respecting the great conquests of
Mohammed Áskiá (or, as he calls him, Ischia, whom he erroneously
styles King of Timbúktu), he does not once mention his expedition
against Ágades, of which he might have heard as easily as of those
against Kátsena and Kanó, which preceded the former only by two
years. From his account it would seem that the town was then in a
very flourishing state, full of foreign merchants and slaves, and that
the king, though he paid a tribute of one hundred and fifty thousand
ducats to the King of Timbúktu (Gágho), enjoyed a great degree of
independence, at least from that quarter, and had even a military
force of his own. Besides, it is stated expressly that he belonged to
the Berber race. But it would almost seem as if Leo, in this passage,
represented the state of things as it was when he visited the town,
before Áskiá’s time, and not at the date when he wrote, though the
circumstance of the tribute payable to that king may have been
learnt from later information. In general, the great defect in Leo’s
description is that the reader has no exact dates to which to refer
the several statements, and that he cannot be sure how far the
author speaks as an eye-witness, and how far from information.
Of course it is possible that the Berbers found a Sónghay
population, if not in the place itself, which most probably did not
exist before the time of their arrival, yet in the district around it; and
it would seem that there existed in ancient times, in the celebrated
Valley of Ír-n-allem, a small town of which some vestiges are said to
remain at the present day, as well as two or three date-trees, the
solitary remains of a large plantation. From this town, tradition says,
the present inhabitants of Ágades were transplanted. But be this as
it may, it is certain that the same dialect of the Sónghay language
which is spoken in Ágades is also still spoken in a few places in the
neighbourhood, by the tribe of the Íghdalén, or Ighedálen, whose
whole appearance, especially their long hair, shows them to be a
mixed race of Sónghay and Berbers, and there is some reason to
suppose that they belonged originally to the Zenága or Senhája.
These people live in and around Íngal, a small town four days’
journey from Ágades, on the road to Sókoto, and in and around
Tegídda, a place three days’ journey from Íngal, and about five from
Ágades west-south-west. This latter place is of considerable interest,
being evidently identical with the town of the same name mentioned
by Ebn Khaldún and by Ebn Batúta as a wealthy place, lying
eastward from Gógo, on the road to Egypt, and in intimate
connection and friendly intercourse with the Mzáb and Wárgela. It
was governed by a Berber chief, with the title of Sultan. This place,
too, was for some time subject to Gógo, or rather to the empire of
Méle or Málli, which then comprised Sónghay, in the latter part of
the fourteenth century; and the circumstance that here too the
Sónghay language is still spoken may be best explained by referring
it to colonization, since it is evident that Áskiá, when he took
possession of Ágades, must have occupied Tegídda also, which lay
on the road from Gógo to that place. However, I will not indulge in
conjectures, and will merely enter into historical questions so far as
they contribute to furnish a vivid and coherent picture of the tribes
and countries with which my journey brought me into contact. I will
therefore only add that this place, Tegídda or Tekádda, was famous,
in the time of Ebn Batúta, for its copper mines, the ores of which
were exported as far as Bórnu and Góber, while at present nothing is
known of the existence of copper hereabouts; but a very good
species of salt of red colour (já-n-gísherí), which is far superior to
that of Bilma, is obtained here, as well as in Íngal. But I recommend
this point to the inquiry of future travellers. I have mentioned above
the presence of loadstone on the border of Aír.
 Having thus attempted to elucidate and illustrate the remarkable
fact that the language of Ágades is derived from and akin to the
Sónghay—a fact which of course appeared to me more surprising
before I discovered, in the course of 1853, that this language
extends eastward far beyond the so-called Niger—I return once
more to the settlement of the Berbers in Ágades. It is evident that
this settlement, if it was of the nature described above, was made
for the purpose of serving as a great commercial entrepôt for the
commerce with another country; and if we duly consider the
statements made by el Bekri, Ebn Batúta, Leo, Ca da Mosto, and by
the author of the “History of Sónghay,” with regard to the
importance of the market of Gógo, and if we pay due attention to
that circuitous route which led from Gógo by way of Tegídda, not
only to Egypt, but even to Tawát, there cannot be the least doubt
that Ágades was founded by those Berber tribes with the distinct
purpose that it might serve them as a secure abode and fortified
magazine in their commercial intercourse with that splendid capital
of the Sónghay empire, the principal article of which was gold, which
formed also the chief article in the former commerce of Ágades. For
Ágades had its own standard weight of this precious metal, the
mithkál, which even at the present day regulates the circulating
medium. And this mithkál of Ágades is totally different from the
standard of the same name which is in use in Timbúktu, the latter
being, in regard to the value of the Spanish dollar, as 1⅓ to 1, and
the former only as ⅖ to 1. But for wholesale business a greater
weight was in use, called “kárruwe,” the smaller kárruwe containing
thirty-three mithákel, or mithkáls, and a third, equal to two rottls and
a sixth, while the larger kárruwe contained a hundred mithkáls, and
was equal to six rottls and a half.
The importance of the trade of Ágades, and the wealth of the
place in general, appear very clearly from the large tribute, of a
hundred and fifty thousand ducats, which the King of Ágades was
able to pay to that of Sónghay, especially if we bear in mind that
Leo, in order to give an idea of the great expense which this same
King of Sónghay had incurred on his pilgrimage to Mekka, states in
another passage that having spent all he took with him, he
contracted a debt amounting to that very sum. As for the King of
Ágades, his situation was at that time just what it is now; and we
cannot better describe his precarious position, entirely dependent on
the caprice and intrigues of the influential chiefs of the Tuarek, than
by using the very words of Leo, “Alle volte scacciano il re e pongono
qualche suo parente in luogo di lui, nè usano ammazzar alcuno; e
quel che più contenta gli abitatori del diserto è fatto re in Agadez.”
Unfortunately, we are not able to fix a date for that very peculiar
covenant between the different tribes with regard to the installation
of the Sultan of Ágades, and the establishing of the principle that he
must belong to a certain family, which is regarded as of sheríf
nobility, and lives not in Ágades, nor even in the country of Aír, but
in a town of Góber. I was once inclined to think that this was an
arrangement made in consequence of the power and influence
which the Emír of Sókoto had arrogated to himself; but I have now
reason to doubt this, for even the grandfather of ʿAbd el Káder was
Sultan. Certainly even now, when the power of the Fulfúlde or
Féllani empire is fast crumbling to pieces, the Emír of Sókoto has a
certain influence upon the choice of the Sultan of Ágades. Of this
fact I myself became witness during my stay in Sókoto in April,
1853, when Hámed eʾ Rufäy was once more sent out to succeed
ʿAbd el Káder. Indeed, Ittegáma, ʿAbd el Káder’s brother, who
thought that I enjoyed the favour and confidence of the Emír, called
upon me (as I shall relate in due time) expressly to entreat me most
urgently to exert my influence in order to restore my former host to
his authority.
I have described already in what way the union of the tribes of the
Itísan, the Kél-gerés, and the Kél-owí is expressed in installing the
Sultan; but though without the presence and assent of the former
the new prince could never arrive at his place of residence, the final
decision seems to rest with the chief Ánnur, the inhabitants of the
town having no voice in the matter. The Sultan is rather a chief of
the Tuarek tribes residing in Ágades than the ruler of Ágades. How
difficult and precarious his position must be may be easily conceived
if it be considered that these tribes are generally at war with one
another; the father of Hámed eʾ Rufäy was even killed by the Kél-
gerés. Nevertheless, if he be an intelligent and energetic man, his
influence in the midst of this wild conflict and struggle of clashing
interests and inclinations must be very beneficial.
What the revenue of the Sultan may at present amount to it is
difficult to say. His means and income consist chiefly in the presents
which he receives on his accession to authority, in a contribution of
one bullock’s hide or kulábu (being about the value of half a Spanish
dollar) from each family, in a more considerable but rather uncertain
tribute levied upon the Imghád, in the tax of ten mithkáls or four
Spanish dollars which he levies on each camel-load of foreign
merchandise which enters the town of Ágades (articles of food being
exempt from charge), in a small tribute derived from the salt
brought from Bilma, and in the fines levied on lawless people and
marauders, and often on whole tribes. Thus it is very probable that
the expedition which ʿAbd el Káder undertook immediately after his
accession, against the tribes who had plundered us, enriched him
considerably. As for the inhabitants of Ágades themselves, I was
assured that they do not pay him any tribute at all, but are only
obliged to accompany him on his expeditions. Of course in earlier
times, when the commerce of the town was far greater than at
present, and when the Imghád (who had to provide him with cattle,
corn, fruit, and vegetables) were strictly obedient, his income far
exceeded that of the present day. When taken altogether it is
certainly considerably under twenty thousand dollars. His title is
Amanókal, or Amanókal Imakóren, in Temáshight, Kókoy bére in the
Emgédesi, and Babá-n-Serkí in the Háusa language.
The person second in authority in the town, and in certain
respects the Vizier, is now, and apparently was also in ancient times,
the “kókoy gerégeré” (i.e. master of the courtyard or the interior of
the palace). This is his real indigenous character, while the
foreigners, who regarded him only in his relation to themselves,
called him Sheikh el ʿArab, or, in the Háusa language, Serkí-n-
turáwa (the Chief of the Whites), and this is the title by which he is
generally known. For it was he who had to levy the tax on the
merchandise imported into the town, an office which in former
times, when a considerable trade was carried on, was of great
importance. But the chief duty of the “serkí-n-turáwa,” at the present
time, is to accompany annually the salt-caravan of the Kél-gerés,
which supplies the western part of Middle Sudán with the salt of
Bilma, from Ágades to Sókoto, and to protect it on the road as well
as to secure it against exorbitant exactions on the part of the Fúlbe
of Sókoto. For this trouble he receives one “kántu,” that is to say the
eighth part (eight kántu weighing three Turkish kantars or quintals)
of a middle-sized camel-load, a contribution which forms a
considerable income in this country, probably of from eight to ten
thousand Spanish dollars, the caravan consisting generally of some
thousand camels, not all equally laden, and the kántu of salt fetching
in Sudán from five thousand to seven and eight thousand kurdí or
shells, which are worth from two to three dollars. Under such
circumstances those officers, who at the same time trade on their
own account, cannot but amass considerable wealth. Mohammed
Bóro as well as Áshu are very rich, considering the circumstances of
the country.
After having escorted the salt-caravan to Sókoto, and settled the
business with the Emír of this place, the serkí-n-turáwa in former
times had to go to Kanó, where he received a small portion of the
six hundred kurdí, the duty levied on each slave brought to the
slave-market, after which he returned to Ágades with the Kél-gerés
that had frequented the market of Kanó. I had full opportunity, in
the further course of my journey, to convince myself that such is not
now the case; but I cannot say what is the reason of this custom
having been discontinued, though it may be the dangerous state of
the road between Sókoto and Kanó. Mohammed Bóro, the former
serkí-n-turáwa, has still residences as well in Kanó and Zínder as in
Sókoto and Ágades. From what I have said it is clear that at present
the serkí-n-turáwa has much more to do with the Tuarek and Fúlbe
than with the Arabs, and at the same time is a sort of mediator
between Agades and Sókoto. Of the other persons in connection
with the Sultan, the “kókoy kaina” or “bába-n-serkí” (the chief
eunuch), at present Ámagay, the fádawa-n-serkí (the aides-de-camp
of the Sultan), as well as the kádhi or alkáli, and the war-chief Sídi
Ghalli, I have spoken in the diary of my residence in the place.
I have already stated above that the southern part of the town,
which at present is almost entirely deserted, formed the oldest
quarter, while katánga, or “báki-n-bírni,” seems to have been its
northern limit. Within these limits the town was about two miles in
circuit, and when thickly peopled may have contained about thirty
thousand inhabitants; but after the northern quarter was added the
whole town had a circuit of about three miles and a half, and may
easily have mustered as many as fifty thousand inhabitants, or even
more. The highest degree of power seems to have been attained
before the conquest of the town by Mohammed Áskiá in the year
1515, though it is said to have been a considerable and wealthy
place till about sixty years ago (reckoned from 1850), when the
greatest part of the inhabitants emigrated to the neighbouring towns
of Háusa, chiefly Kátsena, Tasáwa, Marádi, and Kanó. The exact
circumstances which brought about this deplorable desertion and
desolation of the place I was not able to learn; and the date of the
event cannot be made to coincide with the period of the great
revolution effected in Middle Sudán by the rising of the Jihádi, “the
Reformer,” ʿOthmán da-n-Fódiye, which it preceded by more than
fifteen years; but it coincides with or closely follows upon an event
which I shall have to dwell upon in the further course of my
proceedings. This is the conquest of Gáo, or Gógo (the former
capital of the Sónghay empire, and which since 1591 had become a
province of the empire of Morocco), by the Tuarek. As we have seen
above that Ágades had evidently been founded as an entrepôt for
the great trade with this most flourishing commercial place on the
Ísa, or Niger, at that time the centre of the gold trade, of course the
ransacking and wholesale destruction of this town could not but
affect in the most serious manner the wellbeing of Agades, cutting
away the very roots through which it received life.
 1, House where I lodged; 2, Great Mosque, or Mesállaje; 3, Palace, or Fáda; 4,
Káswa-n-delélti, or Tama-n-lókoy; 5, Káswa-n-rákoma; 6, Katánga; 7, Erárar-n-
zákan; 8, Mohammed Bóro’s house; 9, House of the Kádhi; 10, Well Shedwánka;
11, Pools of Stagnant Water; 12, Kófa-n-Alkáli; 13, Masráta Hogúme; 14, Suburb
of Ben Gottára.

At present I still think that I was not far wrong in estimating the
number of the inhabited houses at from six hundred to seven
hundred, and the population at about seven thousand, though it
must be borne in mind that, as the inhabitants have still preserved
their trading character, a great many of the male inhabitants are
always absent from home, a circumstance which reduces the armed
force of the place to about six hundred. A numerical element,
capable of controlling the estimated amount of the population, is
offered by the number of from two hundred and fifty to three
hundred well-bred boys, who at the time of my visit were learning a
little reading and writing, in five or six schools scattered over the
town; for it is not every boy who is sent to school, but only those
belonging to families in easy circumstances, and they are all about
the same age, from eight to ten years old.
 With regard to the names of the quarters of the town, which are
interesting from an historical point of view, I was not able to learn
exactly the application of each of the names; and I am sure very few
even of the inhabitants themselves can now tell the limits of the
quarters, on account of the desolate state of many of them. The
principal names which can be laid down with certainty in the plan
are Masráta, Gobetáren, Gáwa-Ngírsu, Dígi or Dégi, Katánga,
Terjemán, and Arrafía, which comprise the south-western quarter of
the town. The names of the other quarters, which I attempted to lay
down on the plan sent to Government together with my report, I
now deem it prudent to withdraw, as I afterwards found that there
was some uncertainty about them. I therefore collect here, for the
information of future travellers, the names of the other quarters of
the place, besides those mentioned above and marked in the plan—
Lárelóg, Churúd, Hásena, Amaréwuël, Imurdán (which name, I was
assured afterwards, has nothing in common with the name of the
tribe of the Imghád), Tafimáta (the quarter where the tribe of the
same name lived), Yobímme (“yobu-mé” meaning the mouth of the
market), Dégi-n-béne, or the Upper Dégi, and Bosenrára. Kachíyu
(not Kachín) seems to have been originally the name of a pool, as I
was assured that, besides the three ponds still visible, there were
formerly seven others, namely Kudúru, Kachíyu, Chikinéwan,
Lángusúgázará, Kurungúsu, and Rabafáda, this latter in the square
of the palace.
The whole ground upon which the town is built (being the edge of
a tableland which coincides with the transition from granite to
sandstone) seems to be greatly impregnated with salt at a certain
depth, of which not only the ponds, but even the wells bear
evidence, two of the three wells still in use having saltish water, and
only that of Shedwánka being, as to taste, free from salt, though it
is still regarded as unwholesome, and all the water used for drinking
is brought from the wells outside the walls. Formerly, it is said, there
were nine wells inside the town.
From what I have said above, it may be concluded that the
commerce of Ágades is now inconsiderable. Its characteristic feature
is that no kind of money whatever is current in the market—neither
gold, nor silver, nor kurdí, nor shells; while strips of cotton, or
gábagá (the Kanúri, and not the Háusa term being employed in this
case, because the small quantity of this stuff which is current is
imported from the north-western province of Bórnu), are very rare,
and indeed form almost as merely nominal a standard as the
mithkál. Nevertheless the value of the mithkál is divided into ten
rijáls, or érjel, which measure means eight drʿa, or cubits, of
gábagá. The real standard of the market, I must repeat, is millet or
dukhn (“géro” in Háusa, “éneli” in Temáshight, Pennisetum
typhoïdeum), durra, or Holcus sorghum, being scarcely ever brought
to market. And it is very remarkable, that with this article a man may
buy everything at a much cheaper rate than with merchandise,
which in general fetches a low price in the place; at least it did so
during my stay, when the market had been well stocked with
everything in demand, by the people who had come along with us.
English calico of very good quality was sold by me at 20 per cent.
less than it had been bought for at Múrzuk. Senna in former times
formed an article of export of some importance; but the price which
it fetches on the coast has so decreased that it scarcely pays the
carriage, the distance from the coast being so very great; and it
scarcely formed at all an article in request here, nor did we meet on
our whole journey a single camel laden with it, though it grows in
considerable quantities in the valleys hereabouts.
Ágades is in no respect a place of resort for wealthy merchants,
not even Arabs, while with regard to Europe its importance at
present consists in its lying on the most direct road to Sókoto and
that part of Sudán. In my opinion it would form for a European
agent a very good and comparatively healthy place from which to
open relations with Central Africa. The native merchants seem only
to visit the markets of Kátsena, Tasáwa, Marádi, Kanó, and Sókoto,
and, as far as I was able to learn, never go to the northern markets
of Ghát or Múrzuk, unless on a journey to Mekka, which several of
them have made. Neither does there seem to exist any intercourse
at present with Gágho, or Gógo, or with Timbúktu; but the Arabs of
Azawád and those parts, when undertaking a pilgrimage, generally
go by way of Ágades.
I must here add, that I did not observe that the people of Ágades
use manna in their food, nor that it is collected in the neighbourhood
of the town; but I did not inquire about it on the spot, not having
taken notice of the passage of Leo relating to it.
My stay in Ágades was too short to justify my entering into detail
about the private life of the people, but all that I saw convinced me
that, although open to most serious censure on the part of the
moralist, it presented many striking features of cheerfulness and
happiness, and nothing like the misery which is often met with in
towns which have declined from their former glory. It still contains
many active germs of national life, which are most gratifying to the
philosophic traveller. The situation, on an elevated plateau, cannot
but be healthy, as the few waterpools, of small dimensions, are
incapable of infecting the air. The disease which I have mentioned in
my diary as prevalent at the time of my sojourn was epidemic.
Besides, it must be borne in mind that the end of the rainy season
everywhere in the tropical regions is the most unhealthy period of
the year.
Welcome to our website – the ideal destination for book lovers and
knowledge seekers. With a mission to inspire endlessly, we offer a
vast collection of books, ranging from classic literary works to
specialized publications, self-development books, and children's
literature. Each book is a new journey of discovery, expanding
knowledge and enriching the soul of the reade

Our website is not just a platform for buying books, but a bridge
connecting readers to the timeless values of culture and wisdom. With
an elegant, user-friendly interface and an intelligent search system,
we are committed to providing a quick and convenient shopping
experience. Additionally, our special promotions and home delivery
services ensure that you save time and fully enjoy the joy of reading.

Let us accompany you on the journey of exploring knowledge and

personal growth!

ebookmass.com