Visit https://ebookfinal.

com to download the full version and

explore more ebooks

Linux Programming by Example The Fundamentals

1st Edition Robbins

_____ Click the link below to download _____

https://ebookfinal.com/download/linux-programming-by-
example-the-fundamentals-1st-edition-robbins/

Explore and download more ebooks at ebookfinal.com

Here are some suggested products you might be interested in.
Click the link to download

Access 2002 Programming by Example 1st Edition Bob

Villareal

https://ebookfinal.com/download/access-2002-programming-by-
example-1st-edition-bob-villareal/

Your Wish is My Command Programming by Example 1st Edition

Henry Lieberman

https://ebookfinal.com/download/your-wish-is-my-command-programming-
by-example-1st-edition-henry-lieberman/

Professional Linux Programming 1st Edition Jon Masters

https://ebookfinal.com/download/professional-linux-programming-1st-
edition-jon-masters/

Linux System Programming 1st Edition Robert Love

https://ebookfinal.com/download/linux-system-programming-1st-edition-
robert-love/
Django By Example 1st Edition Antonio Melé

https://ebookfinal.com/download/django-by-example-1st-edition-antonio-
mele/

Version Control by Example 1st Edition Eric Sink

https://ebookfinal.com/download/version-control-by-example-1st-
edition-eric-sink/

Beginning Linux programming 4th Edition Neil Matthew

https://ebookfinal.com/download/beginning-linux-programming-4th-
edition-neil-matthew/

Beginning Linux programming 3rd Edition Neil Matthew

https://ebookfinal.com/download/beginning-linux-programming-3rd-
edition-neil-matthew/

SAS Functions by Example Second Edition Ron Cody

https://ebookfinal.com/download/sas-functions-by-example-second-
edition-ron-cody/
Linux Programming by Example The Fundamentals 1st
Edition Robbins Digital Instant Download
Author(s): Robbins, Arnold
ISBN(s): 9780131429642, 0131429647
Edition: 1
File Details: PDF, 3.31 MB
Year: 2004
Language: english
Linux Programming
by Example

Arnold Robbins

PRENTICE HALL
Professional Technical Reference
Upper Saddle River, NJ 07458
www.phptr.com
 Prentice Hall
Open Source Software Development Series
Arnold Robbins, Series Editor

“Real world code from real world applications”

Open Source technology has revolutionized the computing world. Many large-scale projects are
in production use worldwide, such as Apache, MySQL, and Postgres, with programmers writing
applications in a variety of languages including Perl, Python, and PHP. These technologies are in
use on many different systems, ranging from proprietary systems, to Linux systems, to traditional
UNIX systems, to mainframes.
The Prentice Hall Open Source Software Development Series is designed to bring you the
best of these Open Source technologies. Not only will you learn how to use them for your
projects, but you will learn from them. By seeing real code from real applications, you will learn
the best practices of Open Source developers the world over.

Titles currently in the series include:

Linux® Debugging and Performance Tuning: Tips and Techniques
Steve Best
0131492470, Paper, 10/14/2005
The book is not only a high-level strategy guide but also a book that combines strategy with
hands-on debugging sessions and performance tuning tools and techniques.
Linux Programming by Example: The Fundamentals
Arnold Robbins
0131429647, Paper, 4/12/2004
Gradually, one step at a time, Robbins teaches both high level principles and “under the hood”
techniques. This book will help the reader master the fundamentals needed to build serious
Linux software.
The Linux® Kernel Primer: A Top-Down Approach for x86 and PowerPC Architectures
Claudia Salzberg, Gordon Fischer, Steven Smolski
0131181637, Paper, 9/21/2005
A comprehensive view of the Linux Kernel is presented in a top down approach—the big picture
first with a clear view of all components, how they interrelate, and where the hardware/software
separation exists. The coverage of both the x86 and the PowerPC is unique to this book.
 © 2004 Pearson Education, Inc.
Publishing as Prentice Hall Professional Technical Reference
Upper Saddle River, New Jersey 07458

Prentice Hall PTR offers discounts on this book when ordered in quantity for bulk purchases or special sales. For
more information, please contact: U.S. Corporate and Government Sales, 1-800-382-3419,
corpsales@pearsontechgroup.com. For sales outside of the United States, please contact: International Sales,
1-317-581-3793, international@pearsontechgroup.com.

Portions of Chapter 1, Copyright © 1994 Arnold David Robbins, first appeared in an article in Issue 16 of Linux
Journal, reprinted by permission.
Portions of the documentation for Valgrind, Copyright © 2003 Julian Seward, reprinted by permission.
Portions of the documentation for the DBUG library, by Fred N. Fish, reprinted by permission.
The GNU programs in this book are Copyright © 1985-2003, Free Software Foundation, Inc.. The full list of files
and copyright dates is provided in the Preface. Each program is “free software; you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by the Free Software Foundation; either version
2 of the License, or (at your option) any later version.” Appendix C of this book provides the text of the GNU
General Public License.
All V7 Unix code and documentation are Copyright © Caldera International Inc. 2001-2002. All rights reserved.
They are reprinted here under the terms of the Caldera Ancient UNIX License, which is reproduced in full in
Appendix B.
Cover image courtesy of Parks Sabers, Inc. The Arc-Wave(tm) saber is manufactured by Parks Sabers, Inc., Copyright
© 2001, www.parksabers.com. Parks Sabers is not associated with any Lucasfilm Ltd. property, film, or franchise.
The programs and applications presented in this book have been included for their instructional value. They have
been tested with care but are not guaranteed for any particular purpose. The publisher does not offer any warranties
or representations, nor does it accept any liabilities with respect to the programs or applications. UNIX is a registered
trademark of The Open Group in the United States and other countries.
Microsoft, MS, and MS-DOS are registered trademarks, and Windows is a trademark of Microsoft Corporation in
the United States and other countries. Linux is a registered trademark of Linux Torvalds.
All company and product names mentioned herein are the trademarks or registered trademarks of their respective
owners.
This material may be distributed only subject to the terms and conditions set forth in the Open Publication License,
v1.0 or later (the latest version is presently available at http://www.opencontent.org/openpub/), with License
Option B.
Printed in the United States of America
ISBN 0-13-142964-7
Text printed on recycled paper
First printing
Pearson Education LTD.
Pearson Education Australia PTY, Limited
Pearson Education Singapore, Pte. Ltd.
Pearson Education North Asia Ltd.
Pearson Education Canada, Ltd.
Pearson Educación de Mexico, S.A. de C.V.
Pearson Education—Japan
Pearson Education Malaysia, Ptd. Ltd.
 To my wife Miriam,
and my children,
Chana, Rivka, Nachum, and Malka.
 Contents

Preface ........................................................................................................... xvii

PART I Files and Users ............................................ 1

Chapter 1 Introduction .............................................................................. 3
1.1 The Linux/Unix File Model ............................................................................. 4
1.1.1 Files and Permissions ................................................................................ 4
1.1.2 Directories and Filenames ........................................................................ 6
1.1.3 Executable Files ........................................................................................ 7
1.1.4 Devices ..................................................................................................... 9
1.2 The Linux/Unix Process Model ........................................................................ 10
1.2.1 Pipes: Hooking Processes Together .......................................................... 12
1.3 Standard C vs. Original C ................................................................................ 12
1.4 Why GNU Programs Are Better ....................................................................... 14
1.4.1 Program Design ....................................................................................... 15
1.4.2 Program Behavior ..................................................................................... 16
1.4.3 C Code Programming .............................................................................. 16
1.4.4 Things That Make a GNU Program Better .............................................. 17
1.4.5 Parting Thoughts about the “GNU Coding Standards” ........................... 19
1.5 Portability Revisited ......................................................................................... 19
1.6 Suggested Reading ............................................................................................ 20
1.7 Summary .......................................................................................................... 21
Exercises ...................................................................................................................... 22
Chapter 2 Arguments, Options, and the Environment ................................ 23
2.1 Option and Argument Conventions ................................................................. 24
2.1.1 POSIX Conventions ................................................................................. 25
2.1.2 GNU Long Options ................................................................................. 27

v
vi Contents

2.2 Basic Command-Line Processing ...................................................................... 28

2.2.1 The V7 echo Program ............................................................................ 29
2.3 Option Parsing: getopt() and getopt_long() ....................................... 30
2.3.1 Single-Letter Options ............................................................................... 30
2.3.2 GNU getopt() and Option Ordering .................................................. 33
2.3.3 Long Options ........................................................................................... 34
2.3.3.1 Long Options Table ........................................................................... 34
2.3.3.2 Long Options, POSIX Style ............................................................... 37
2.3.3.3 getopt_long() Return Value Summary ........................................ 37
2.3.3.4 GNU getopt() or getopt_long() in User Programs ................ 39
2.4 The Environment ............................................................................................. 40
2.4.1 Environment Management Functions ...................................................... 41
2.4.2 The Entire Environment: environ ........................................................ 43
2.4.3 GNU env ................................................................................................ 44
2.5 Summary .......................................................................................................... 49
Exercises ...................................................................................................................... 50
Chapter 3 User-Level Memory Management ............................................... 51
3.1 Linux/Unix Address Space ................................................................................ 52
3.2 Memory Allocation .......................................................................................... 56
3.2.1 Library Calls: malloc(), calloc(), realloc(), free() .............. 56
3.2.1.1 Examining C Language Details ........................................................... 57
3.2.1.2 Initially Allocating Memory: malloc() ........................................... 58
3.2.1.3 Releasing Memory: free() .............................................................. 60
3.2.1.4 Changing Size: realloc() .............................................................. 62
3.2.1.5 Allocating and Zero-filling: calloc() .............................................. 65
3.2.1.6 Summarizing from the GNU Coding Standards .................................. 66
3.2.1.7 Using Private Allocators ...................................................................... 67
3.2.1.8 Example: Reading Arbitrarily Long Lines ........................................... 67
3.2.1.9 GLIBC Only: Reading Entire Lines: getline() and getdelim() . 73
3.2.2 String Copying: strdup() ..................................................................... 74
3.2.3 System Calls: brk() and sbrk() .......................................................... 75
3.2.4 Lazy Programmer Calls: alloca() ........................................................ 76
3.2.5 Address Space Examination ...................................................................... 78
3.3 Summary .......................................................................................................... 80
Exercises ...................................................................................................................... 81
Contents vii

Chapter 4 Files and File I/O ....................................................................... 83

4.1 Introducing the Linux/Unix I/O Model ........................................................... 84
4.2 Presenting a Basic Program Structure ............................................................... 84
4.3 Determining What Went Wrong ..................................................................... 86
4.3.1 Values for errno ..................................................................................... 87
4.3.2 Error Message Style .................................................................................. 90
4.4 Doing Input and Output ................................................................................. 91
4.4.1 Understanding File Descriptors ................................................................ 92
4.4.2 Opening and Closing Files ....................................................................... 93
4.4.2.1 Mapping FILE * Variables to File Descriptors ................................. 95
4.4.2.2 Closing All Open Files ........................................................................ 96
4.4.3 Reading and Writing ................................................................................ 96
4.4.4 Example: Unix cat .................................................................................. 99
4.5 Random Access: Moving Around within a File ................................................. 102
4.6 Creating Files ................................................................................................... 106
4.6.1 Specifying Initial File Permissions ............................................................ 106
4.6.2 Creating Files with creat() .................................................................. 109
4.6.3 Revisiting open() ................................................................................... 110
4.7 Forcing Data to Disk ........................................................................................ 113
4.8 Setting File Length ........................................................................................... 114
4.9 Summary .......................................................................................................... 115
Exercises ...................................................................................................................... 115
Chapter 5 Directories and File Metadata .................................................... 117
5.1 Considering Directory Contents ....................................................................... 118
5.1.1 Definitions ............................................................................................... 118
5.1.2 Directory Contents ................................................................................... 120
5.1.3 Hard Links ............................................................................................... 122
5.1.3.1 The GNU link Program .................................................................. 123
5.1.3.2 Dot and Dot-Dot ............................................................................... 125
5.1.4 File Renaming .......................................................................................... 125
5.1.5 File Removal ............................................................................................ 126
5.1.5.1 Removing Open Files ......................................................................... 127
5.1.5.2 Using ISO C: remove() .................................................................. 127
5.1.6 Symbolic Links ......................................................................................... 128
viii Contents

5.2 Creating and Removing Directories .................................................................. 130

5.3 Reading Directories .......................................................................................... 132
5.3.1 Basic Directory Reading ........................................................................... 133
5.3.1.1 Portability Considerations .................................................................. 136
5.3.1.2 Linux and BSD Directory Entries ....................................................... 137
5.3.2 BSD Directory Positioning Functions ...................................................... 138
5.4 Obtaining Information about Files ................................................................... 139
5.4.1 Linux File Types ....................................................................................... 139
5.4.2 Retrieving File Information ...................................................................... 141
5.4.3 Linux Only: Specifying Higher-Precision File Times ................................ 143
5.4.4 Determining File Type ............................................................................. 144
5.4.4.1 Device Information ............................................................................ 147
5.4.4.2 The V7 cat Revisited ........................................................................ 150
5.4.5 Working with Symbolic Links .................................................................. 151
5.5 Changing Ownership, Permission, and Modification Times ............................. 155
5.5.1 Changing File Ownership: chown(), fchown(), and lchown() ....... 155
5.5.2 Changing Permissions: chmod() and fchmod() .................................. 156
5.5.3 Changing Timestamps: utime() ............................................................ 157
5.5.3.1 Faking utime(file, NULL) ......................................................... 159
5.5.4 Using fchown() and fchmod() for Security ....................................... 161
5.6 Summary .......................................................................................................... 162
Exercises ...................................................................................................................... 163
Chapter 6 General Library Interfaces — Part 1 ............................................ 165
6.1 Times and Dates .............................................................................................. 166
6.1.1 Retrieving the Current Time: time() and difftime() ...................... 167
6.1.2 Breaking Down Times: gmtime() and localtime() ......................... 168
6.1.3 Formatting Dates and Times .................................................................... 170
6.1.3.1 Simple Time Formatting: asctime() and ctime() ...................... 170
6.1.3.2 Complex Time Formatting: strftime() ........................................ 171
6.1.4 Converting a Broken-Down Time to a time_t ...................................... 176
6.1.5 Getting Time-Zone Information .............................................................. 178
6.1.5.1 BSD Systems Gotcha: timezone(), Not timezone ...................... 179
6.2 Sorting and Searching Functions ...................................................................... 181
6.2.1 Sorting: qsort() ................................................................................... 181
Contents ix

6.2.1.1 Example: Sorting Employees .............................................................. 183

6.2.1.2 Example: Sorting Directory Contents ................................................. 188
6.2.2 Binary Searching: bsearch() ................................................................ 191
6.3 User and Group Names .................................................................................... 195
6.3.1 User Database .......................................................................................... 196
6.3.2 Group Database ....................................................................................... 199
6.4 Terminals: isatty() ..................................................................................... 202
6.5 Suggested Reading ............................................................................................ 203
6.6 Summary .......................................................................................................... 203
Exercises ...................................................................................................................... 205
Chapter 7 Putting It All Together: ls ......................................................... 207
7.1 V7 ls Options ................................................................................................. 208
7.2 V7 ls Code ..................................................................................................... 209
7.3 Summary .......................................................................................................... 225
Exercises ...................................................................................................................... 226
Chapter 8 Filesystems and Directory Walks ................................................ 227
8.1 Mounting and Unmounting Filesystems .......................................................... 228
8.1.1 Reviewing the Background ....................................................................... 228
8.1.2 Looking at Different Filesystem Types ..................................................... 232
8.1.3 Mounting Filesystems: mount ................................................................. 236
8.1.4 Unmounting Filesystems: umount .......................................................... 237
8.2 Files for Filesystem Administration ................................................................... 238
8.2.1 Using Mount Options .............................................................................. 239
8.2.2 Working with Mounted Filesystems: getmntent() .............................. 241
8.3 Retrieving Per-Filesystem Information ............................................................. 244
8.3.1 POSIX Style: statvfs() and fstatvfs() ........................................ 244
8.3.2 Linux Style: statfs() and fstatfs() ............................................... 252
8.4 Moving Around in the File Hierarchy .............................................................. 256
8.4.1 Changing Directory: chdir() and fchdir() ...................................... 256
8.4.2 Getting the Current Directory: getcwd() ............................................. 258
8.4.3 Walking a Hierarchy: nftw() ................................................................. 260
8.4.3.1 The nftw() Interface ....................................................................... 261
8.4.3.2 The nftw() Callback Function ........................................................ 263
x Contents

8.5 Walking a File Tree: GNU du ......................................................................... 269

8.6 Changing the Root Directory: chroot() ....................................................... 276
8.7 Summary .......................................................................................................... 277
Exercises ...................................................................................................................... 278

PART II Processes, IPC, and Internationalization ....... 281

Chapter 9 Process Management and Pipes ................................................. 283
9.1 Process Creation and Management ................................................................... 284
9.1.1 Creating a Process: fork() ..................................................................... 284
9.1.1.1 After the fork(): Shared and Distinct Attributes ............................. 285
9.1.1.2 File Descriptor Sharing ....................................................................... 286
9.1.1.3 File Descriptor Sharing and close() ............................................... 288
9.1.2 Identifying a Process: getpid() and getppid() ................................ 289
9.1.3 Setting Process Priority: nice() ............................................................. 291
9.1.3.1 POSIX vs. Reality ............................................................................... 293
9.1.4 Starting New Programs: The exec() Family .......................................... 293
9.1.4.1 The execve() System Call .............................................................. 294
9.1.4.2 Wrapper Functions: execl() et al. .................................................. 295
9.1.4.3 Program Names and argv[0] .......................................................... 297
9.1.4.4 Attributes Inherited across exec() ................................................... 298
9.1.5 Terminating a Process .............................................................................. 300
9.1.5.1 Defining Process Exit Status ............................................................... 300
9.1.5.2 Returning from main() .................................................................... 301
9.1.5.3 Exiting Functions ............................................................................... 302
9.1.6 Recovering a Child’s Exit Status ............................................................... 305
9.1.6.1 Using POSIX Functions: wait() and waitpid() ......................... 306
9.1.6.2 Using BSD Functions: wait3() and wait4() ............................... 310
9.2 Process Groups ................................................................................................. 312
9.2.1 Job Control Overview .............................................................................. 312
9.2.2 Process Group Identification: getpgrp() and getpgid() .................. 314
9.2.3 Process Group Setting: setpgid() and setpgrp() ............................ 314
9.3 Basic Interprocess Communication: Pipes and FIFOs ...................................... 315
9.3.1 Pipes ......................................................................................................... 315
9.3.1.1 Creating Pipes .................................................................................... 316
9.3.1.2 Pipe Buffering .................................................................................... 318
Contents xi

9.3.2 FIFOs ....................................................................................................... 319

9.4 File Descriptor Management ............................................................................ 320
9.4.1 Duplicating Open Files: dup() and dup2() ......................................... 321
9.4.2 Creating Nonlinear Pipelines: /dev/fd/XX ........................................... 326
9.4.3 Managing File Attributes: fcntl() ........................................................ 328
9.4.3.1 The Close-on-exec Flag ...................................................................... 329
9.4.3.2 File Descriptor Duplication ................................................................ 331
9.4.3.3 Manipulation of File Status Flags and Access Modes .......................... 332
9.4.3.4 Nonblocking I/O for Pipes and FIFOs ............................................... 333
9.4.3.5 fcntl() Summary ........................................................................... 336
9.5 Example: Two-Way Pipes in gawk .................................................................. 337
9.6 Suggested Reading ............................................................................................ 341
9.7 Summary .......................................................................................................... 342
Exercises ...................................................................................................................... 344
Chapter 10 Signals ....................................................................................... 347
10.1 Introduction ..................................................................................................... 348
10.2 Signal Actions ................................................................................................... 348
10.3 Standard C Signals: signal() and raise() ............................................... 349
10.3.1 The signal() Function ........................................................................ 349
10.3.2 Sending Signals Programmatically: raise() .......................................... 353
10.4 Signal Handlers in Action ................................................................................. 353
10.4.1 Traditional Systems .................................................................................. 353
10.4.2 BSD and GNU/Linux .............................................................................. 356
10.4.3 Ignoring Signals ....................................................................................... 356
10.4.4 Restartable System Calls ........................................................................... 357
10.4.4.1 Example: GNU Coreutils safe_read() and safe_write() ...... 359
10.4.4.2 GLIBC Only: TEMP_FAILURE_RETRY() ....................................... 360
10.4.5 Race Conditions and sig_atomic_t (ISO C) ...................................... 361
10.4.6 Additional Caveats ................................................................................... 363
10.4.7 Our Story So Far, Episode I ..................................................................... 363
10.5 The System V Release 3 Signal APIs: sigset() et al. .................................... 365
10.6 POSIX Signals .................................................................................................. 367
10.6.1 Uncovering the Problem .......................................................................... 367
10.6.2 Signal Sets: sigset_t and Related Functions ........................................ 368
xii Contents

10.6.3 Managing the Signal Mask: sigprocmask() et al. ............................... 369

10.6.4 Catching Signals: sigaction() ............................................................ 370
10.6.5 Retrieving Pending Signals: sigpending() .......................................... 375
10.6.6 Making Functions Interruptible: siginterrupt() ............................. 376
10.6.7 Sending Signals: kill() and killpg() ............................................... 376
10.6.8 Our Story So Far, Episode II .................................................................... 378
10.7 Signals for Interprocess Communication .......................................................... 379
10.8 Important Special-Purpose Signals ................................................................... 382
10.8.1 Alarm Clocks: sleep(), alarm(), and SIGALRM ............................... 382
10.8.1.1 Harder but with More Control: alarm() and SIGALRM ................. 382
10.8.1.2 Simple and Easy: sleep() ................................................................ 383
10.8.2 Job Control Signals .................................................................................. 383
10.8.3 Parental Supervision: Three Different Strategies ....................................... 385
10.8.3.1 Poor Parenting: Ignoring Children Completely .................................. 385
10.8.3.2 Permissive Parenting: Supervising Minimally .................................... 386
10.8.3.3 Strict Parental Control ........................................................................ 393
10.9 Signals Across fork() and exec() ............................................................... 398
10.10 Summary .......................................................................................................... 399
Exercises ...................................................................................................................... 401
Chapter 11 Permissions and User and Group ID Numbers ........................... 403
11.1 Checking Permissions ....................................................................................... 404
11.1.1 Real and Effective IDs .............................................................................. 405
11.1.2 Setuid and Setgid Bits .............................................................................. 406
11.2 Retrieving User and Group IDs ........................................................................ 407
11.3 Checking As the Real User: access() ........................................................... 410
11.4 Checking as the Effective User: euidaccess() (GLIBC) ............................. 412
11.5 Setting Extra Permission Bits for Directories .................................................... 412
11.5.1 Default Group for New Files and Directories ........................................... 412
11.5.2 Directories and the Sticky Bit ................................................................... 414
11.6 Setting Real and Effective IDs .......................................................................... 415
11.6.1 Changing the Group Set .......................................................................... 416
11.6.2 Changing the Real and Effective IDs ........................................................ 416
11.6.3 Using the Setuid and Setgid Bits .............................................................. 419
11.7 Working with All Three IDs: getresuid() and setresuid() (Linux) .... 421
Contents xiii

11.8 Crossing a Security Minefield: Setuid root ..................................................... 422

11.9 Suggested Reading ............................................................................................ 423
11.10 Summary .......................................................................................................... 424
Exercises ...................................................................................................................... 426
Chapter 12 General Library Interfaces — Part 2 ............................................ 427
12.1 Assertion Statements: assert() .................................................................... 428
12.2 Low-Level Memory: The memXXX() Functions .............................................. 432
12.2.1 Setting Memory: memset() ................................................................... 432
12.2.2 Copying Memory: memcpy(), memmove(), and memccpy() ............. 433
12.2.3 Comparing Memory Blocks: memcmp() ................................................. 434
12.2.4 Searching for a Byte Value: memchr() .................................................... 435
12.3 Temporary Files ............................................................................................... 436
12.3.1 Generating Temporary Filenames (Bad) ................................................... 437
12.3.2 Creating and Opening Temporary Files (Good) ....................................... 441
12.3.3 Using the TMPDIR Environment Variable ................................................ 443
12.4 Committing Suicide: abort() ....................................................................... 445
12.5 Nonlocal Gotos ................................................................................................ 446
12.5.1 Using Standard Functions: setjmp() and longjmp() ........................ 447
12.5.2 Handling Signal Masks: sigsetjmp() and siglongjmp() ............... 449
12.5.3 Observing Important Caveats ................................................................... 450
12.6 Pseudorandom Numbers .................................................................................. 454
12.6.1 Standard C: rand() and srand() ........................................................ 455
12.6.2 POSIX Functions: random() and srandom() ..................................... 457
12.6.3 The /dev/random and /dev/urandom Special Files .......................... 460
12.7 Metacharacter Expansions ................................................................................ 461
12.7.1 Simple Pattern Matching: fnmatch() ................................................... 462
12.7.2 Filename Expansion: glob() and globfree() ................................... 464
12.7.3 Shell Word Expansion: wordexp() and wordfree() ......................... 470
12.8 Regular Expressions .......................................................................................... 471
12.9 Suggested Reading ............................................................................................ 480
12.10 Summary .......................................................................................................... 481
Exercises ...................................................................................................................... 482
xiv Contents

Chapter 13 Internationalization and Localization .......................................... 485

13.1 Introduction ..................................................................................................... 486
13.2 Locales and the C Library ................................................................................. 487
13.2.1 Locale Categories and Environment Variables .......................................... 487
13.2.2 Setting the Locale: setlocale() .......................................................... 489
13.2.3 String Collation: strcoll() and strxfrm() ..................................... 490
13.2.4 Low-Level Numeric and Monetary Formatting: localeconv() ........... 494
13.2.5 High-Level Numeric and Monetary Formatting: strfmon()
and printf() ........................................................................................ 498
13.2.6 Example: Formatting Numeric Values in gawk ....................................... 501
13.2.7 Formatting Date and Time Values: ctime() and strftime() ........... 503
13.2.8 Other Locale Information: nl_langinfo() ......................................... 504
13.3 Dynamic Translation of Program Messages ...................................................... 507
13.3.1 Setting the Text Domain: textdomain() ............................................. 507
13.3.2 Translating Messages: gettext() ......................................................... 508
13.3.3 Working with Plurals: ngettext() ....................................................... 509
13.3.4 Making gettext() Easy to Use ............................................................ 510
13.3.4.1 Portable Programs: "gettext.h" ................................................... 511
13.3.4.2 GLIBC Only: <libintl.h> ........................................................... 513
13.3.5 Rearranging Word Order with printf() .............................................. 514
13.3.6 Testing Translations in a Private Directory .............................................. 515
13.3.7 Preparing Internationalized Programs ....................................................... 516
13.3.8 Creating Translations ............................................................................... 517
13.4 Can You Spell That for Me, Please? .................................................................. 521
13.4.1 Wide Characters ....................................................................................... 523
13.4.2 Multibyte Character Encodings ................................................................ 523
13.4.3 Languages ................................................................................................. 524
13.4.4 Conclusion ............................................................................................... 525
13.5 Suggested Reading ............................................................................................ 526
13.6 Summary .......................................................................................................... 526
Exercises ...................................................................................................................... 527
Chapter 14 Extended Interfaces ................................................................... 529
14.1 Allocating Aligned Memory: posix_memalign() and memalign() ......... 530
14.2 Locking Files .................................................................................................... 531
Contents xv

14.2.1 File Locking Concepts .............................................................................. 531

14.2.2 POSIX Locking: fcntl() and lockf() .............................................. 533
14.2.2.1 Describing a Lock ............................................................................... 533
14.2.2.2 Obtaining and Releasing Locks ........................................................... 536
14.2.2.3 Observing Locking Caveats ................................................................ 538
14.2.3 BSD Locking: flock() .......................................................................... 539
14.2.4 Mandatory Locking .................................................................................. 540
14.3 More Precise Times .......................................................................................... 543
14.3.1 Microsecond Times: gettimeofday() ................................................ 544
14.3.2 Microsecond File Times: utimes() ....................................................... 545
14.3.3 Interval Timers: setitimer() and getitimer() ............................. 546
14.3.4 More Exact Pauses: nanosleep() ......................................................... 550
14.4 Advanced Searching with Binary Trees ............................................................. 551
14.4.1 Introduction to Binary Trees .................................................................... 551
14.4.2 Tree Management Functions .................................................................... 554
14.4.3 Tree Insertion: tsearch() .................................................................... 554
14.4.4 Tree Lookup and Use of A Returned Pointer: tfind() and
tsearch() ............................................................................................ 555
14.4.5 Tree Traversal: twalk() ......................................................................... 557
14.4.6 Tree Node Removal and Tree Deletion: tdelete() and tdestroy() . 561
14.5 Summary .......................................................................................................... 562
Exercises ...................................................................................................................... 563

PART III Debugging and Final Project ........................ 565

Chapter 15 Debugging ................................................................................. 567
15.1 First Things First .............................................................................................. 568
15.2 Compilation for Debugging ............................................................................. 569
15.3 GDB Basics ...................................................................................................... 570
15.3.1 Running GDB ......................................................................................... 571
15.3.2 Setting Breakpoints, Single-Stepping, and Setting Watchpoints ............... 574
15.4 Programming for Debugging ............................................................................ 577
15.4.1 Compile-Time Debugging Code .............................................................. 577
15.4.1.1 Use Debugging Macros ...................................................................... 577
15.4.1.2 Avoid Expression Macros If Possible ................................................... 580
15.4.1.3 Reorder Code If Necessary ................................................................. 582
xvi Contents

15.4.1.4 Use Debugging Helper Functions ....................................................... 584

15.4.1.5 Avoid Unions When Possible ............................................................. 591
15.4.2 Runtime Debugging Code ....................................................................... 595
15.4.2.1 Add Debugging Options and Variables .............................................. 595
15.4.2.2 Use Special Environment Variables .................................................... 597
15.4.2.3 Add Logging Code ............................................................................. 601
15.4.2.4 Runtime Debugging Files ................................................................... 602
15.4.2.5 Add Special Hooks for Breakpoints .................................................... 603
15.5 Debugging Tools .............................................................................................. 605
15.5.1 The dbug Library — A Sophisticated printf() ................................... 606
15.5.2 Memory Allocation Debuggers ................................................................. 612
15.5.2.1 GNU/Linux mtrace ......................................................................... 613
15.5.2.2 Electric Fence ..................................................................................... 614
15.5.2.3 Debugging Malloc: dmalloc ............................................................ 619
15.5.2.4 Valgrind: A Versatile Tool .................................................................. 623
15.5.2.5 Other Malloc Debuggers .................................................................... 629
15.5.3 A Modern lint ...................................................................................... 631
15.6 Software Testing ............................................................................................... 632
15.7 Debugging Rules .............................................................................................. 633
15.8 Suggested Reading ............................................................................................ 637
15.9 Summary .......................................................................................................... 638
Exercises ...................................................................................................................... 639
Chapter 16 A Project That Ties Everything Together .................................... 641
16.1 Project Description ........................................................................................... 642
16.2 Suggested Reading ............................................................................................ 644

PART IV Appendixes ................................................. 647

Appendix A Teach Yourself Programming in Ten Years ................................. 649
Appendix B Caldera Ancient UNIX License .................................................... 655
Appendix C GNU General Public License ....................................................... 657
Index .............................................................................................................. 667
 Preface

ne of the best ways to learn about programming is to read well-written pro-

O grams. This book teaches the fundamental Linux system call APIs—those
that form the core of any significant program—by presenting code from production
programs that you use every day.
By looking at concrete programs, you can not only see how to use the Linux APIs,
but you also can examine the real-world issues (performance, portability, robustness)
that arise in writing software.
While the book’s title is Linux Programming by Example, everything we cover, unless
otherwise noted, applies to modern Unix systems as well. In general we use “Linux”
to mean the Linux kernel, and “GNU/Linux” to mean the total system (kernel, li-
braries, tools). Also, we often say “Linux” when we mean all of Linux, GNU/Linux
and Unix; if something is specific to one system or the other, we mention it explicitly.

Audience
This book is intended for the person who understands programming and is familiar
with the basics of C, at least on the level of The C Programming Language by Kernighan
and Ritchie. (Java programmers wishing to read this book should understand C pointers,
since C code makes heavy use of them.) The examples use both the 1990 version of
Standard C and Original C.
In particular, you should be familiar with all C operators, control-flow structures,
variable and pointer declarations and use, the string management functions, the use of
exit(), and the <stdio.h> suite of functions for file input/output.
You should understand the basic concepts of standard input, standard output, and
standard error and the fact that all C programs receive an array of character strings
representing invocation options and arguments. You should also be familiar with the
fundamental command-line tools, such as cd, cp, date, ln, ls, man (and info if you

xvii
xviii Preface

have it), rmdir, and rm, the use of long and short command-line options, environment
variables, and I/O redirection, including pipes.
We assume that you want to write programs that work not just under GNU/Linux
but across the range of Unix systems. To that end, we mark each interface as to its
availability (GLIBC systems only, or defined by POSIX, and so on), and portability
advice is included as an integral part of the text.
The programming taught here may be at a lower level than you’re used to; that’s
OK. The system calls are the fundamental building blocks for higher operations and
are thus low-level by nature. This in turn dictates our use of C: The APIs were designed
for use from C, and code that interfaces them to higher-level languages, such as C++
and Java, will necessarily be lower level in nature, and most likely, written in C. It may
help to remember that “low level” doesn’t mean “bad,” it just means “more challenging.”

What You Will Learn

This book focuses on the basic APIs that form the core of Linux programming:
• Memory management
• File input/output
• File metadata
• Processes and signals
• Users and groups
• Programming support (sorting, argument parsing, and so on)
• Internationalization
• Debugging

We have purposely kept the list of topics short. We believe that it is intimidating to
try to learn “all there is to know” from a single book. Most readers prefer smaller, more
focused books, and the best Unix books are all written that way.
So, instead of a single giant tome, we plan several volumes: one on Interprocess
Communication (IPC) and networking, and another on software development and
code portability. We also have an eye toward possible additional volumes in a Linux
Preface xix

Programming by Example series that will cover topics such as thread programming and
GUI programming.
The APIs we cover include both system calls and library functions. Indeed, at the C
level, both appear as simple function calls. A system call is a direct request for system
services, such as reading or writing a file or creating a process. A library function, on the
other hand, runs at the user level, possibly never requesting any services from the oper-
ating system. System calls are documented in section 2 of the reference manual (viewable
online with the man command), and library functions are documented in section 3.
Our goal is to teach you the use of the Linux APIs by example: in particular, through
the use, wherever possible, of both original Unix source code and the GNU utilities.
Unfortunately, there aren’t as many self-contained examples as we thought there’d be.
Thus, we have written numerous small demonstration programs as well. We stress
programming principles: especially those aspects of GNU programming, such as “no
arbitrary limits,” that make the GNU utilities into exceptional programs.
The choice of everyday programs to study is deliberate. If you’ve been using
GNU/Linux for any length of time, you already understand what programs such as ls
and cp do; it then becomes easy to dive straight into how the programs work, without
having to spend a lot of time learning what they do.
Occasionally, we present both higher-level and lower-level ways of doing things.
Usually the higher-level standard interface is implemented in terms of the lower-level
interface or construct. We hope that such views of what’s “under the hood” will help
you understand how things work; for all the code you write, you should always use the
higher-level, standard interface.
Similarly, we sometimes introduce functions that provide certain functionality and
then recommend (with a provided reason) that these functions be avoided! The primary
reason for this approach is so that you’ll be able to recognize these functions when you
see them and thus understand the code using them. A well-rounded knowledge of a
topic requires understanding not just what you can do, but what you should and should
not do.
Finally, each chapter concludes with exercises. Some involve modifying or writing
code. Others are more in the category of “thought experiments” or “why do you
think …” We recommend that you do all of them—they will help cement your under-
standing of the material.
xx Preface

Small Is Beautiful: Unix Programs

Hoare’s law:
“Inside every large program is a small program
struggling to get out.”
—C.A.R. Hoare—
Initially, we planned to teach the Linux API by using the code from the GNU utilities.
However, the modern versions of even simple command-line programs (like mv and
cp) are large and many-featured. This is particularly true of the GNU variants of the
standard utilities, which allow long and short options, do everything required by POSIX,
and often have additional, seemingly unrelated options as well (like output highlighting).
It then becomes reasonable to ask, “Given such a large and confusing forest, how
can we focus on the one or two important trees?” In other words, if we present the
current full-featured program, will it be possible to see the underlying core operation
of the program?
That is when Hoare’s law1 inspired us to look to the original Unix programs for ex-
ample code. The original V7 Unix utilities are small and straightforward, making it
easy to see what’s going on and to understand how the system calls are used. (V7 was
released around 1979; it is the common ancestor of all modern Unix systems, including
GNU/Linux and the BSD systems.)
For many years, Unix source code was protected by copyrights and trade secret license
agreements, making it difficult to use for study and impossible to publish. This is still
true of all commercial Unix source code. However, in 2002, Caldera (currently operating
as SCO) made the original Unix code (through V7 and 32V Unix) available under an
Open Source style license (see Appendix B, “Caldera Ancient UNIX License,” page 655).
This makes it possible for us to include the code from the early Unix system in this book.

Standards
Throughout the book we refer to several different formal standards. A standard is a
document describing how something works. Formal standards exist for many things,
for example, the shape, placement, and meaning of the holes in the electrical outlet in

1 This famous statement was made at The International Workshop on Efficient Production of Large Programs in
Jablonna, Poland, August 10–14, 1970.
Preface xxi

your wall are defined by a formal standard so that all the power cords in your country
work in all the outlets.
So, too, formal standards for computing systems define how they are supposed to
work; this enables developers and users to know what to expect from their software and
enables them to complain to their vendor when software doesn’t work.
Of interest to us here are:
1. ISO/IEC International Standard 9899: Programming Languages — C, 1990.
The first formal standard for the C programming language.
2. ISO/IEC International Standard 9899: Programming Languages — C, Second
edition, 1999. The second (and current) formal standard for the C programming
language.
3. ISO/IEC International Standard 14882: Programming Languages — C++, 1998.
The first formal standard for the C++ programming language.
4. ISO/IEC International Standard 14882: Programming Languages — C++, 2003.
The second (and current) formal standard for the C++ programming language.
5. IEEE Standard 1003.1–2001: Standard for Information Technology — Portable
Operating System Interface (POSIX®). The current version of the POSIX stan-
dard; describes the behavior expected of Unix and Unix-like systems. This
edition covers both the system call and library interface, as seen by the C/C++
programmer, and the shell and utilities interface, seen by the user. It consists
of several volumes:
• Base Definitions. The definitions of terms, facilities, and header files.
• Base Definitions — Rationale. Explanations and rationale for the choice of
facilities that both are and are not included in the standard.
• System Interfaces. The system calls and library functions. POSIX terms them
all “functions.”
• Shell and Utilities. The shell language and utilities available for use with shell
programs and interactively.

Although language standards aren’t exciting reading, you may wish to consider pur-
chasing a copy of the C standard: It provides the final definition of the language. Copies
xxii Preface

can be purchased from ANSI2 and from ISO.3 (The PDF version of the C standard is
quite affordable.)
The POSIX standard can be ordered from The Open Group.4 By working through
their publications catalog to the items listed under “CAE Specifications,” you can find
individual pages for each part of the standard (named “C031” through “C034”). Each
one’s page provides free access to the online HTML version of the particular volume.
The POSIX standard is intended for implementation on both Unix and Unix-like
systems, as well as non-Unix systems. Thus, the base functionality it provides is a subset
of what Unix systems have. However, the POSIX standard also defines optional exten-
sions—additional functionality, for example, for threads or real-time support. Of most
importance to us is the X/Open System Interface (XSI) extension, which describes facilities
from historical Unix systems.
Throughout the book, we mark each API as to its availability: ISO C, POSIX, XSI,
GLIBC only, or nonstandard but commonly available.

Features and Power: GNU Programs

Restricting ourselves to just the original Unix code would have made an interesting
history book, but it would not have been very useful in the 21st century. Modern pro-
grams do not have the same constraints (memory, CPU power, disk space, and speed)
that the early Unix systems did. Furthermore, they need to operate in a multilingual
world—ASCII and American English aren’t enough.
More importantly, one of the primary freedoms expressly promoted by the Free
Software Foundation and the GNU Project5 is the “freedom to study.” GNU programs
are intended to provide a large corpus of well-written programs that journeyman pro-
grammers can use as a source from which to learn.

2 http://www.ansi.org
3 http://www.iso.ch
4 http://www.opengroup.org
5 http://www.gnu.org
Preface xxiii

By using GNU programs, we want to meet both goals: show you well-written,
modern code from which you will learn how to write good code and how to use the
APIs well.
We believe that GNU software is better because it is free (in the sense of “freedom,”
not “free beer”). But it’s also recognized that GNU software is often technically better
than the corresponding Unix counterparts, and we devote space in Section 1.4, “Why
GNU Programs Are Better,” page 14, to explaining why.
A number of the GNU code examples come from gawk (GNU awk). The main
reason is that it’s a program with which we’re very familiar, and therefore it was easy
to pick examples from it. We don’t otherwise make any special claims about it.

Summary of Chapters
Driving a car is a holistic process that involves multiple simultaneous tasks. In many
ways, Linux programming is similar, requiring understanding of multiple aspects
of the API, such as file I/O, file metadata, directories, storage of time information,
and so on.
The first part of the book looks at enough of these individual items to enable studying
the first significant program, the V7 ls. Then we complete the discussion of files and
users by looking at file hierarchies and the way filesystems work and are used.
Chapter 1, “Introduction,” page 3,
describes the Unix and Linux file and process models, looks at the differences be-
tween Original C and 1990 Standard C, and provides an overview of the principles
that make GNU programs generally better than standard Unix programs.
Chapter 2, “Arguments, Options, and the Environment,” page 23,
describes how a C program accesses and processes command-line arguments and
options and explains how to work with the environment.
Chapter 3, “User-Level Memory Management,” page 51,
provides an overview of the different kinds of memory in use and available in a
running process. User-level memory management is central to every nontrivial
application, so it’s important to understand it early on.
xxiv Preface

Chapter 4, “Files and File I/O,” page 83,

discusses basic file I/O, showing how to create and use files. This understanding
is important for everything else that follows.
Chapter 5, “Directories and File Metadata,” page 117,
describes how directories, hard links, and symbolic links work. It then describes
file metadata, such as owners, permissions, and so on, as well as covering how to
work with directories.
Chapter 6, “General Library Interfaces — Part 1,” page 165,
looks at the first set of general programming interfaces that we need so that we
can make effective use of a file’s metadata.
Chapter 7, “Putting It All Together: ls,” page 207,
ties together everything seen so far by looking at the V7 ls program.
Chapter 8, “Filesystems and Directory Walks,” page 227,
describes how filesystems are mounted and unmounted and how a program
can tell what is mounted on the system. It also describes how a program can
easily “walk” an entire file hierarchy, taking appropriate action for each object
it encounters.

The second part of the book deals with process creation and management, interprocess
communication with pipes and signals, user and group IDs, and additional general
programming interfaces. Next, the book first describes internationalization with GNU
gettext and then several advanced APIs.

Chapter 9, “Process Management and Pipes,” page 283,

looks at process creation, program execution, IPC with pipes, and file descriptor
management, including nonblocking I/O.
Chapter 10, “Signals,” page 347,
discusses signals, a simplistic form of interprocess communication. Signals also
play an important role in a parent process’s management of its children.
Chapter 11, “Permissions and User and Group ID Numbers,” page 403,
looks at how processes and files are identified, how permission checking works,
and how the setuid and setgid mechanisms work.
Preface xxv

Chapter 12, “General Library Interfaces — Part 2,” page 427,

looks at the rest of the general APIs; many of these are more specialized than the
first general set of APIs.
Chapter 13, “Internationalization and Localization,” page 485,
explains how to enable your programs to work in multiple languages, with almost
no pain.
Chapter 14, “Extended Interfaces,” page 529,
describes several extended versions of interfaces covered in previous chapters, as
well as covering file locking in full detail.

We round the book off with a chapter on debugging, since (almost) no one gets
things right the first time, and we suggest a final project to cement your knowledge of
the APIs covered in this book.
Chapter 15, “Debugging,” page 567,
describes the basics of the GDB debugger, transmits as much of our programming
experience in this area as possible, and looks at several useful tools for doing dif-
ferent kinds of debugging.
Chapter 16, “A Project That Ties Everything Together,” page 641,
presents a significant programming project that makes use of just about everything
covered in the book.

Several appendices cover topics of interest, including the licenses for the source code
used in this book.
Appendix A, “Teach Yourself Programming in Ten Years,” page 649,
invokes the famous saying, “Rome wasn’t built in a day.” So too, Linux/Unix ex-
pertise and understanding only come with time and practice. To that end, we
have included this essay by Peter Norvig which we highly recommend.
Appendix B, “Caldera Ancient UNIX License,” page 655,
covers the Unix source code used in this book.
Appendix C, “GNU General Public License,” page 657,
covers the GNU source code used in this book.
xxvi Preface

Typographical Conventions
Like all books on computer-related topics, we use certain typographical conventions
to convey information. Definitions or first uses of terms appear in italics, like the word
“Definitions” at the beginning of this sentence. Italics are also used for emphasis, for
citations of other works, and for commentary in examples. Variable items such as argu-
ments or filenames, appear like this. Occasionally, we use a bold font when a point
needs to be made strongly.
Things that exist on a computer are in a constant-width font, such as filenames
(foo.c) and command names (ls, grep). Short snippets that you type are additionally
enclosed in single quotes: ‘ls -l *.c’.
$ and > are the Bourne shell primary and secondary prompts and are used to display
interactive examples. User input appears in a different font from regular computer
output in examples. Examples look like this:
$ ls -1 Look at files. Option is digit 1, not letter l
foo
bar
baz

We prefer the Bourne shell and its variants (ksh93, Bash) over the C shell; thus, all
our examples show only the Bourne shell. Be aware that quoting and line-continuation
rules are different in the C shell; if you use it, you’re on your own!6
When referring to functions in programs, we append an empty pair of parentheses
to the function’s name: printf(), strcpy(). When referring to a manual page (acces-
sible with the man command), we follow the standard Unix convention of writing the
command or function name in italics and the section in parentheses after it, in regular
type: awk(1), printf (3).

Where to Get Unix and GNU Source Code

You may wish to have copies of the programs we use in this book for your own ex-
perimentation and review. All the source code is available over the Internet, and your
GNU/Linux distribution contains the source code for the GNU utilities.

6 See the csh(1) and tcsh(1) manpages and the book Using csh & tcsh, by Paul DuBois, O’Reilly & Associates, Se-
bastopol, CA, USA, 1995. ISBN: 1-56592-132-1.
Preface xxvii

Unix Code
Archives of various “ancient” versions of Unix are maintained by The UNIX Heritage
Society (TUHS), http://www.tuhs.org.
Of most interest is that it is possible to browse the archive of old Unix source code
on the Web. Start with http://minnie.tuhs.org/UnixTree/. All the example code
in this book is from the Seventh Edition Research UNIX System, also known as “V7.”
The TUHS site is physically located in Australia, although there are mirrors of the
archive around the world—see http://www.tuhs.org/archive_sites.html.
This page also indicates that the archive is available for mirroring with rsync.
(See http://rsync.samba.org/ if you don’t have rsync: It’s standard on
GNU/Linux systems.)
You will need about 2–3 gigabytes of disk to copy the entire archive. To copy the
archive, create an empty directory, and in it, run the following commands:
mkdir Applications 4BSD PDP-11 PDP-11/Trees VAX Other

rsync -avz minnie.tuhs.org::UA_Root .

rsync -avz minnie.tuhs.org::UA_Applications Applications
rsync -avz minnie.tuhs.org::UA_4BSD 4BSD
rsync -avz minnie.tuhs.org::UA_PDP11 PDP-11
rsync -avz minnie.tuhs.org::UA_PDP11_Trees PDP-11/Trees
rsync -avz minnie.tuhs.org::UA_VAX VAX
rsync -avz minnie.tuhs.org::UA_Other Other

You may wish to omit copying the Trees directory, which contains extractions of
several versions of Unix, and occupies around 700 megabytes of disk.
You may also wish to consult the TUHS mailing list to see if anyone near you can
provide copies of the archive on CD-ROM, to avoid transferring so much data over
the Internet.
The folks at Southern Storm Software, Pty. Ltd., in Australia, have “modernized” a
portion of the V7 user-level code so that it can be compiled and run on current systems,
most notably GNU/Linux. This code can be downloaded from their web site.7
It’s interesting to note that V7 code does not contain any copyright or permission
notices in it. The authors wrote the code primarily for themselves and their research,
leaving the permission issues to AT&T’s corporate licensing department.

7 http://www.southern-storm.com.au/v7upgrade.html
xxviii Preface

GNU Code
If you’re using GNU/Linux, then your distribution will have come with source code,
presumably in whatever packaging format it uses (Red Hat RPM files, Debian DEB
files, Slackware .tar.gz files, etc.). Many of the examples in the book are from the
GNU Coreutils, version 5.0. Find the appropriate CD-ROM for your GNU/Linux
distribution, and use the appropriate tool to extract the code. Or follow the instructions
in the next few paragraphs to retrieve the code.
If you prefer to retrieve the files yourself from the GNU ftp site, you will find them
at ftp://ftp.gnu.org/gnu/coreutils/coreutils-5.0.tar.gz.
You can use the wget utility to retrieve the file:
$ wget ftp://ftp.gnu.org/gnu/coreutils/coreutils-5.0.tar.gz Retrieve the distribution
… lots of output here as file is retrieved …
Alternatively, you can use good old-fashioned ftp to retrieve the file:
$ ftp ftp.gnu.org Connect to GNU ftp site
Connected to ftp.gnu.org (199.232.41.7).
220 GNU FTP server ready.
Name (ftp.gnu.org:arnold): anonymous Use anonymous ftp
331 Please specify the password.
Password: Password does not echo on screen
230-If you have any problems with the GNU software or its downloading,
230-please refer your questions to <gnu@gnu.org>.
... Lots of verbiage deleted
230 Login successful. Have fun.
Remote system type is UNIX.
Using binary mode to transfer files.
ftp> cd /gnu/coreutils Change to Coreutils directory
250 Directory successfully changed.
ftp> bin
200 Switching to Binary mode.
ftp> hash Print # signs as progress indicators
Hash mark printing on (1024 bytes/hash mark).
ftp> get coreutils-5.0.tar.gz Retrieve file
local: coreutils-5.0.tar.gz remote: coreutils-5.0.tar.gz
227 Entering Passive Mode (199,232,41,7,86,107)
150 Opening BINARY mode data connection for coreutils-5.0.tar.gz (6020616 bytes)
#################################################################################
#################################################################################
...
226 File send OK.
6020616 bytes received in 2.03e+03 secs (2.9 Kbytes/sec)
ftp> quit Log off
221 Goodbye.
Preface xxix

Once you have the file, extract it as follows:

$ gzip -dc < coreutils-5.0.tar.gz | tar -xvpf - Extract files
. . . lots of output here as files are extracted . . .
Systems using GNU tar may use this incantation:
$ tar -xvpzf coreutils-5.0.tar.gz Extract files
. . . lots of output here as files are extracted . . .
In compliance with the GNU General Public License, here is the Copyright infor-
mation for all GNU programs quoted in this book. All the programs are “free software;
you can redistribute it and/or modify it under the terms of the GNU General Public
License as published by the Free Software Foundation; either version 2 of the License,
or (at your option) any later version.” See Appendix C, “GNU General Public License,”
page 657, for the text of the GNU General Public License.
Coreutils 5.0 File Copyright dates
lib/safe-read.c Copyright © 1993–1994, 1998, 2002
lib/safe-write.c Copyright © 2002
lib/utime.c Copyright © 1998, 2001–2002
lib/xreadlink.c Copyright © 2001
src/du.c Copyright © 1988–1991, 1995–2003
src/env.c Copyright © 1986, 1991–2003
src/install.c Copyright © 1989–1991, 1995–2002
src/link.c Copyright © 2001–2002
src/ls.c Copyright © 1985, 1988, 1990, 1991, 1995–2003
src/pathchk.c Copyright © 1991–2003
src/sort.c Copyright © 1988, 1991–2002
src/sys2.h Copyright © 1997–2003
src/wc.c Copyright © 1985, 1991, 1995–2002
Gawk 3.0.6 File Copyright dates
eval.c Copyright © 1986, 1988, 1989, 1991–2000
xxx Preface

Gawk 3.1.3 File Copyright dates

awk.h Copyright © 1986, 1988, 1989, 1991–2003
builtin.c Copyright © 1986, 1988, 1989, 1991–2003
eval.c Copyright © 1986, 1988, 1989, 1991–2003
io.c Copyright © 1986, 1988, 1989, 1991–2003
main.c Copyright © 1986, 1988, 1989, 1991–2003
posix/gawkmisc.c Copyright © 1986, 1988, 1989, 1991–1998, 2001–2003
Gawk 3.1.4 File Copyright dates
builtin.c Copyright © 1986, 1988, 1989, 1991–2004
GLIBC 2.3.2 File Copyright dates
locale/locale.h Copyright © 1991, 1992, 1995–2002
posix/unistd.h Copyright © 1991–2003
time/sys/time.h Copyright © 1991–1994, 1996–2003
Make 3.80 File Copyright dates
read.c Copyright © 1988–1997, 2002

Where to Get the Example Programs Used in This Book

The example programs used in this book can be found at http://authors.
phptr.com/robbins.

About the Cover

“This is the weapon of a Jedi Knight …, an elegant weapon for
a more civilized age. For over a thousand generations the Jedi
Knights were the guardians of peace and justice in the Old
Republic. Before the dark times, before the Empire.”
—Obi-Wan Kenobi—
You may be wondering why we chose to put a light saber on the cover and to use it
throughout the book’s interior. What does it represent, and how does it relate to Linux
programming?
Preface xxxi

In the hands of a Jedi Knight, a light saber is both a powerful weapon and a thing
of beauty. Its use demonstrates the power, knowledge, control of the Force, and arduous
training of the Jedi who wields it.
The elegance of the light saber mirrors the elegance of the original Unix API design.
There, too, the studied, precise use of the APIs and the Software Tools and GNU design
principles lead to today’s powerful, flexible, capable GNU/Linux system. This system
demonstrates the knowledge and understanding of the programmers who wrote all its
components.
And, of course, light sabers are just way cool!

Acknowledgments
Writing a book is lots of work, and doing it well requires help from many people.
Dr. Brian W. Kernighan, Dr. Doug McIlroy, Peter Memishian, and Peter van der
Linden reviewed the initial book proposal. David J. Agans, Fred Fish, Don Marti, Jim
Meyering, Peter Norvig, and Julian Seward provided reprint permission for various
items quoted throughout the book. Thanks to Geoff Collyer, Ulrich Drepper, Yosef
Gold, Dr. C.A.R. (Tony) Hoare, Dr. Manny Lehman, Jim Meyering, Dr. Dennis M.
Ritchie, Julian Seward, Henry Spencer, and Dr. Wladyslaw M. Turski, who provided
much useful general information. Thanks also to the other members of the GNITS
gang: Karl Berry, Akim DeMaille, Ulrich Drepper, Greg McGary, Jim Meyering,
François Pinard, and Tom Tromey, who all provided helpful feedback about good
programming practice. Karl Berry, Alper Ersoy, and Dr. Nelson H.F. Beebe provided
valuable technical help with the Texinfo and DocBook/XML toolchains.
Good technical reviewers not only make sure that an author gets his facts right, they
also ensure that he thinks carefully about his presentation. Dr. Nelson H.F. Beebe,
Geoff Collyer, Russ Cox, Ulrich Drepper, Randy Lechlitner, Dr. Brian W. Kernighan,
Peter Memishian, Jim Meyering, Chet Ramey, and Louis Taber acted as technical re-
viewers for the entire book. Dr. Michael Brennan provided helpful comments on
Chapter 15. Both the prose and many of the example programs benefited from their
reviews. I hereby thank all of them. As most authors usually say here, “Any remaining
errors are mine.”
I would especially like to thank Mark Taub of Pearson Education for initiating this
project, for his enthusiasm for the series, and for his help and advice as the book moved
xxxii Preface

through its various stages. Anthony Gemmellaro did a phenomenal job of realizing my
concept for the cover, and Gail Cocker’s interior design is beautiful. Faye Gemmellaro
made the production process enjoyable, instead of a chore. Dmitry Kirsanov and
Alina Kirsanova did the figures, page layout, and indexing; they were a pleasure to
work with.
Finally, my deepest gratitude and love to my wife, Miriam, for her support and en-
couragement during the book’s writing.

Arnold Robbins
Nof Ayalon
ISRAEL
 I

Files and Users

Chapter 1 Introduction page 3

Chapter 2 Arguments, Options, and the Environment page 23
Chapter 3 User-Level Memory Management page 51
Chapter 4 Files and File I/O page 83
Chapter 5 Directories and File Metadata page 117
Chapter 6 General Library Interfaces — Part 1 page 165
Chapter 7 Putting It All Together: ls page 207
Chapter 8 Filesystems and Directory Walks page 227

1
This page intentionally left blank
 1

Introduction

In this chapter

1.1 The Linux/Unix File Model page 4

1.2 The Linux/Unix Process Model page 10
1.3 Standard C vs. Original C page 12
1.4 Why GNU Programs Are Better page 14
1.5 Portability Revisited page 19
1.6 Suggested Reading page 20
1.7 Summary page 21
Exercises page 22

3
 f there is one phrase that summarizes the primary GNU/Linux (and therefore
I Unix) concepts, it’s “files and processes.” In this chapter we review the Linux
file and process models. These are important to understand because the system calls
are almost all concerned with modifying some attribute or part of the state of a file
or a process.
Next, because we’ll be examining code in both styles, we briefly review the major
difference between 1990 Standard C and Original C. Finally, we discuss at some
length what makes GNU programs “better,” programming principles that we’ll see
in use in the code.
This chapter contains a number of intentional simplifications. The full details are
covered as we progress through the book. If you’re already a Linux wizard, please
forgive us.

1.1 The Linux/Unix File Model

One of the driving goals in the original Unix design was simplicity. Simple concepts
are easy to learn and use. When the concepts are translated into simple APIs, simple
programs are then easy to design, write, and get correct. In addition, simple code is
often smaller and more efficient than more complicated designs.
The quest for simplicity was driven by two factors. From a technical point of view,
the original PDP-11 minicomputers on which Unix was developed had a small address
space: 64 Kilobytes total on the smaller systems, 64K code and 64K of data on the large
ones. These restrictions applied not just to regular programs (so-called user level code),
but to the operating system itself (kernel level code). Thus, not only “Small Is Beautiful”
aesthetically, but “Small Is Beautiful” because there was no other choice!
The second factor was a negative reaction to contemporary commercial operating
systems, which were needlessly complicated, with obtuse command languages, multiple
kinds of file I/O, and little generality or symmetry. (Steve Johnson once remarked that
“Using TSO is like trying to kick a dead whale down a beach.” TSO is one of the obtuse
mainframe time-sharing systems just described.)

1.1.1 Files and Permissions

The Unix file model is as simple as it gets: A file is a linear stream of bytes. Period.
The operating system imposes no preordained structure on files: no fixed or varying

4
1.1 The Linux/Unix File Model 5

record sizes, no indexed files, nothing. The interpretation of file contents is entirely up
to the application. (This isn’t quite true, as we’ll see shortly, but it’s close enough for
a start.)
Once you have a file, you can do three things with the file’s data: read them, write
them, or execute them.
Unix was designed for time-sharing minicomputers; this implies a multiuser environ-
ment from the get-go. Once there are multiple users, it must be possible to specify a
file’s permissions: Perhaps user jane is user fred’s boss, and jane doesn’t want fred
to read the latest performance evaluations.
For file permission purposes, users are classified into three distinct categories: user:
the owner of a file; group: the group of users associated with this file (discussed shortly);
and other: anybody else. For each of these categories, every file has separate read, write,
and execute permission bits associated with it, yielding a total of nine permission bits.
This shows up in the first field of the output of ‘ls -l’:
$ ls -l progex.texi
-rw-r--r-- 1 arnold devel 5614 Feb 24 18:02 progex.texi

Here, arnold and devel are the owner and group of progex.texi, and -rw-r--r--
are the file type and permissions. The first character is a dash for regular file, a d for
directories, or one of a small set of other characters for other kinds of files that aren’t
important at the moment. Each subsequent group of three characters represents read,
write, and execute permission for the owner, group, and “other,” respectively.
In this example, progex.texi is readable and writable by the owner, and readable
by the group and other. The dashes indicate absent permissions, thus the file is not ex-
ecutable by anyone, nor is it writable by the group or other.
The owner and group of a file are stored as numeric values known as the user ID
(UID) and group ID (GID); standard library functions that we present later in the book
make it possible to print the values as human-readable names.
A file’s owner can change the permission by using the chmod (change mode)
command. (As such, file permissions are sometimes referred to as the “file mode.”)
A file’s group can be changed with the chgrp (change group) and chown (change
owner) commands.1

1 Some systems allow regular users to change the ownership on their files to someone else, thus “giving them away.”
The details are standardized by POSIX but are a bit messy. Typical GNU/Linux configurations do not allow it.
6 Chapter 1 • Introduction

Group permissions were intended to support cooperative work: Although one person
in a group or department may own a particular file, perhaps everyone in that group
needs to be able to modify it. (Consider a collaborative marketing paper or data from
a survey.)
When the system goes to check a file access (usually upon opening a file), if the UID
of the process matches that of the file, the owner permissions apply. If those permissions
deny the operation (say, a write to a file with -r--rw-rw- permissions), the operation
fails; Unix and Linux do not proceed to test the group and other permissions.2 The
same is true if the UID is different but the GID matches; if the group permissions deny
the operation, it fails.
Unix and Linux support the notion of a superuser: a user with special privileges. This
user is known as root and has the UID of 0. root is allowed to do anything; all bets
are off, all doors are open, all drawers unlocked.3 (This can have significant security
implications, which we touch on throughout the book but do not cover exhaustively.)
Thus, even if a file is mode ----------, root can still read and write the file. (One
exception is that the file can’t be executed. But as root can add execute permission,
the restriction doesn’t prevent anything.)
The user/group/other, read/write/execute permissions model is simple, yet flexible
enough to cover most situations. Other, more powerful but more complicated, models
exist and are implemented on different systems, but none of them are well enough
standardized and broadly enough implemented to be worth discussing in a general-
purpose text like this one.

1.1.2 Directories and Filenames

Once you have a file, you need someplace to keep it. This is the purpose of the direc-
tory (known as a “folder” on Windows and Apple Macintosh systems). A directory is
a special kind of file, which associates filenames with particular collections of file
metadata, known as inodes. Directories are special because they can only be updated by
the operating system, by the system calls described in Chapter 4, “Files and File I/O,”

2 The owner can always change the permission, of course. Most users don’t disable write permission for themselves.
3 There are some rare exceptions to this rule, all of which are beyond the scope of this book.
1.1 The Linux/Unix File Model 7

page 83. They are also special in that the operating system dictates the format of direc-
tory entries.
Filenames may contain any valid 8-bit byte except the / (forward slash) character
and ASCII NUL, the character whose bits are all zero. Early Unix systems limited file-
names to 14 bytes; modern systems allow individual filenames to be up to 255 bytes.
The inode contains all the information about a file except its name: the type, owner,
group, permissions, size, modification and access times. It also stores the locations on
disk of the blocks containing the file’s data. All of these are data about the file, not the
file’s data itself, thus the term metadata.
Directory permissions have a slightly different meaning from those for file permissions.
Read permission means the ability to search the directory; that is, to look through it to
see what files it contains. Write permission is the ability to create and remove files in
the directory. Execute permission is the ability to go through a directory when opening
or otherwise accessing a contained file or subdirectory.

NOTE If you have write permission on a directory, you can remove files in that
directory, even if they don’t belong to you! When used interactively, the rm
command notices this, and asks you for confirmation in such a case.
The /tmp directory has write permission for everyone, but your files in /tmp
are quite safe because /tmp usually has the so-called sticky bit set on it:
$ ls -ld /tmp
drwxrwxrwt 11 root root 4096 May 15 17:11 /tmp
Note the t is the last position of the first field. On most directories this position
has an x in it. With the sticky bit set, only you, as the file’s owner, or root may
remove your files. (We discuss this in more detail in Section 11.5.2, “Directories
and the Sticky Bit,” page 414.)

1.1.3 Executable Files

Remember we said that the operating system doesn’t impose a structure on files?
Well, we’ve already seen that that was a white lie when it comes to directories. It’s also
the case for binary executable files. To run a program, the kernel has to know what part
of a file represents instructions (code) and what part represents data. This leads to the
notion of an object file format, which is the definition for how these things are laid out
within a file on disk.
Exploring the Variety of Random
Documents with Different Content
 60
The movement for a more effective union was partly the work of
far-sighted leaders who could look beyond state boundaries to the
larger interests of the country as a whole ... another group were
beginning to see that the weakness of Congress might have
something to do with troubles nearer home.
Thus the causes of the Federal union were the need of external
defense and the need of reconciling the many diverse groups in the
population of the new nation. These groups are summarized by Carl
R. Fish:
61
There were thirteen distinct and separate state governments,
and Vermont had its own local authority which defied the rest....
Differences in the original stock, emphasized by different physical
conditions and by the isolated life of the colonial period, had created
several great sections or divisions in the country, which had sufficient
similarity within themselves and sufficient unlikeness to each other, to
make them permanent entities, and to cause sectionalism to be a
permanent factor in American history.
He then goes on to enumerate the groups: New England, with its
Puritan English ancestry and its agriculture and fishing; the South,
with its English Episcopalians, its aristocratic ideals, its plantation
life; the piedmont country, with its independent small farmers and
self-governing townships; and in between the commercial valleys of
the Hudson, the Delaware and the Susquehanna. These again had
their local differences of character:
62
The Hudson had been settled by the Dutch, although many
English, New Englanders, Germans and others had mixed with them.
The Delaware region was largely occupied by English and German
Quakers. The Susquehanna Valley contained a large proportion of
Germans, still using their native tongue, and also many Scotch, Irish
and English. There existed well defined interests, the mercantile, the
agricultural; the German, the Dutch and the Quakers; the city, the
country.

Beyond all these was the frontier, with its own natural conditions,
type of inhabitants, and economic problems, a democratic
community separate from all the rest.
 The Constitutional convention of 1787 in Philadelphia was typical
of this situation. Every problem found divergent interests and
63
opinions; every solution was effected by compromise. There was
state jealousy of all central authority; the opposition between large
and small states; that between the industries of the north and the
agriculture of the south; the slave trade, with its complex of moral
and economic problems; the sectionalism of the settled east and the
frontier west; the protection of the property-holding class and the
satisfaction of the radicals with their demand for liberty and equality.
Every one of these conflicts had to be settled if possible, or at least
(as with slavery) brought to a temporary status to avoid sharp
struggle. There were, of course, certain unifying factors. The
majority of the settlers were English, and most of these Protestants.
The non-English speaking elements were very largely of Teutonic
blood and Protestant religion also. There was a common political
experience, and a democratic urge typical of the frontier. Most
important of all, there was an eight-year war fought together against
a common enemy and under the same Commander-in-Chief. The
American government, then, with its new Constitution, was not a
simple unity from the outset. It was rather a highly complex unity,
containing within itself many minor groups, many different
viewpoints, and many integrations of the sub-group for the benefit
of the nation as a whole.

2.
An interesting illustration of this, and for our purpose a crucial
one, is in the religious life of the thirteen original states. Before the
Revolution, the states might be divided into four groups as regards
their religious organization: there were congregational
establishments in Massachusetts and Plymouth, New Haven,
Connecticut, and New Hampshire; Church of England establishments
in Virginia and the two Carolinas; four states formerly under various
regimes had had the Church of England forced on them—Maryland,
at first under Catholic rule, but with freedom of residence for all
Christians; New York and New Jersey, which had been dominated by
the Dutch Reformed Church; and Georgia, founded with almost
complete religious liberty. Only three states had no established
church—Rhode Island, Pennsylvania and its offshoot, Delaware. Of
these last, Rhode Island was founded by Roger Williams in 1636,
under the radical, not to say revolutionary principle of complete
separation of church and state, with right of residence and
citizenship for all persons, even including Jews and atheists.
Pennsylvania, chartered in 1681, was founded by William Penn, the
Quaker, with liberty of residence for all “believers in Almighty God”;
but the English government insisted on the condition that all voters
and office holders “shall be such as profess faith in Jesus Christ” and
the Protestant religion. What the new nation had, then, was not
religious liberty, but rather a clash of many different points of view.
64
Massachusetts set up its theocratic state with its chief interest in
the Church; Virginia established its civil state, with the church as a
subject member; while Rhode Island boldly denied the purposes and
premises of both, placing an impassable gulf between the State and
the Church and relegating to the individual conscience and to
voluntary association all concern and action touching the Church and
religious matters.
What, then, should be the upshot of this confusion of religious
groups, with their ancient hatreds and prejudices, ingrown with
history and overlaid with former strife and martyrdom? It was
obviously impossible to make the United States Calvinist or
Episcopal; it was necessary to have some sacrifice of each for the
good of all. But it might have been possible to make the nation
Protestant Christian, as was actually the case with the state of New
Hampshire until 1877. Various minor causes here entered in. Warfare
with England meant some opposition, at least, to the Church of
England. The distance from the actual seat of old-world struggles,
the character of the colonists and their longing for every type of
freedom, helped much. The new theories of the French
Encyclopedists, as adopted by Jefferson, certainly had great
influence. But most important of all was the existence of the many
minor sects, with the few important ones, of which all longed to rule
but none wished to be dominated by any other.
The upshot was religious freedom, the separation of church and
state, according to Article VI, Section 3, of the Federal Constitution:
“No religious test shall ever be required as a qualification to any
office or public trust under the United States.” This clause was
opposed on both sides—by Massachusetts as being too liberal, by
Virginia and Rhode Island as not liberal enough. Virginia had two
years before this overthrown her state church and given complete
freedom of conscience—not toleration—to all her people. The
opposition even to toleration was becoming crystallized in the words
of Thomas Paine: “Toleration is not the opposite of intolerance, but
the counterfeit of it. Both are despotisms: the one assumes to itself
the right of withholding liberty of conscience, the other of granting
it.” So the first amendment to the Constitution, adopted immediately
afterward by motion of the first Congress, and by the required two-
thirds of the states, was: “Congress shall make no law respecting an
establishment of religion, or prohibiting the free exercise thereof.”
This was tremendously significant of the growing and newly
conscious group mind of the United States of America. It was equally
important for the future of the nation and its unity in days to come.
Religious liberty was not a matter of doctrine in its inception; it was
the product of the birth and development of the group mind of the
nation. It meant the relinquishment of the racial habits, of the state
laws, of the old urge to persecute (common to almost every group,
even those who were themselves refugees from persecution), and
the adoption of a national standard to which every state, every
church and every sect should bring its sacrifice. And if this sacrifice
was not of their own right to live, but only of the right to make
others miserable, it was nevertheless the sacrifice of something so
important that the demand had convulsed France, Germany and
England not many years before. Religious liberty, indeed, however
firmly based on law and political ideals, never became the habit of
thought and action which intolerance had been. A recurrent
phenomenon of American life has been the breaking up into
religious, racial and sectional groups, with a further synthesis of
Americanization, through some common interest to unite them. The
conflict among the many groups prior to the adoption of the
Constitution, and its solution in that document with its Bill of Rights,
has been paralleled at least four times from that period to the
present day.

3.
These four reactions against the immigrant correspond with the
four peaks of the curve of immigration into the United States, with
two great alterations in the process, corresponding to the Civil War
and the World War.
Up till 1830, immigration into the United States was small in
amount and fairly regular. The first wave stretched from 1831 to
1861, reaching its peak in 1855; its total amounted to four millions
of foreigners, of whom the largest group and the first to come were
the Irish, the second in number and date of arrival the Germans.
The growth of intolerance against these newcomers was shown in
the movement known as Know-Nothingism or the American Party.
The second wave, of similar nationalities, was from 1862 to 1877;
the reply to this appeared in the anti-alien planks in the political
platforms of 1876. The third wave, from 1878 to 1897, was larger
than these earlier ones; it included many Scandinavians and, after
1882, growing numbers of Italians, Russians, and Austro-
Hungarians, the two last being composed in part of persecuted
Jews, in part of impoverished peasants. The Nativist reaction against
this immigrant trend appears rather in the form of religious
opposition, for the American Protective Association of those days
was predominantly anti-Catholic. The fourth wave began in 1898 and
extended until 1914, when the outbreak of the World War in Europe
caused a sudden drop to almost nothing; in its highest years, 1907
and 1913, more than 1,200,000 entered our ports annually; and the
greatest number of these new arrivals came from Italy, Austria-
Hungary and Russia. The reaction against these new immigrants was
under way, but the war interrupted its progress, and the Ku Klux
Klan arrived at its full power only after the war, when new conditions
swayed the group mind of America.
In each of these cases, the height of the movement against the
immigrant came just after the peak of the wave of immigration, at
the time when it had had time to impress itself on the native-born.
The philosophies of these four movements varied according to the
nationality of the immigrants against whom the natives were
protesting, and according to the general philosophy of life in vogue
at the time. The first such movement, the Know-Nothing or
American Party, originated in New York State in 1852 “as a secret
65
organization with passwords, oath, grip and ritual.” Its creed was
summed up in two words: Americanism and Protestantism. Its
special target was the two million Irish who had come into the
country; they were poor laborers, with a low standard of living,
ignorant, hereditary enemies of England, and Catholics into the
bargain. No wonder there were anti-Irish riots in New York,
Philadelphia and Boston; that it was rumored the Pope would soon
be dictator of America; or that the secret anti-alien society was
begun. But the course of the movement was spectacular and brief. It
entered national politics, thus both making bitter enemies for itself
and taking off the secrecy which was its chief source of power. Then
came the abolitionist movement, and the American party was split
into northern and southern branches. Most important of all, the peak
of immigration was passed, the Irish adopted the American standard
of living, became a part of communal life, without any danger of
Catholic overthrow of our cherished institutions—Othello’s
occupation was gone, and the Know-Nothing party disappeared.
The next wave of immigration and the next reaction against it
were minor ones. The immigrants met groups of their own origin
already absorbed into the common life of America, and fitted in with
little difficulty. The attempt in 1876 to prevent the use of public
funds for sectarian schools was itself comparatively slight.
 But in between came the tremendous crisis of the Civil War. Here
the opposition was not between native and immigrant, but between
north and south, an industrial society of free laborers against an
agricultural society of castes,—planters, poor whites, and negro
slaves. I shall not go further into this conflict, because it is too
familiar and has comparatively little to do with the particular
application of my viewpoint. But, from our point of view, it is
important to see the place of the first Ku Klux Klan of 1865–71. This
was again a secret organization, adding the feature of disguise, for
the terrifying effect on the negroes whom it was the object of the
Klan to overawe. The Klan was a partisan and sectional organization,
of Southern white men of Confederate sympathies, to maintain their
group supremacy over the newly freed negroes and the “carpet
baggers” from the North. The victors had, as usual, indulged in
oppression over the losers, and the grievance was a very real one.
The Klan was partially successful in its object, but at once fell into
numerous abuses, was used by partisans to vent personal grudges,
fell into the hands of a lawless element, and was formally disbanded
in 1871 by General Nathan B. Forrest, its national commander or
Grand Wizard. Its slogan of “white supremacy” shows its animus
against the negroes and the North, not against the alien. Some of its
partisans claim that the Klan did not disband when it was formally
ordered so to do, but persisted in its underground activity until as
66
late as 1877. However that may be, its character and purpose are
very clear; it was sectional, timely, and for the one aim of white
supremacy. It appealed to its members and frightened its enemies
by its methods of disguise and secrecy, no less than by the beatings,
burnings and other outrages which were carried on either under its
auspices or by the false use of its insignia and methods. Its defiance
of the law imposed by force, and its use of force in reply, are the
vestiges of war psychology; It was the legitimate, if unlovely,
offspring of the Reconstruction. It had no function left when the
white South regained control of the states, but its memory still
lingers as part of the idealization of the “lost cause” of the
Confederacy.
 The third reaction against immigration was primarily anti-Catholic
in trend. This was the A. P. A., or American Protective Association,
another secret society, organized in 1887, which reached its greatest
popularity in 1894 and 1895. At this same period there were several
other societies with the same purpose, notably the National League
for the Protection of American Institutions, which had a number of
extremely prominent men among its members. At this time the so-
called “new immigration” was growing strong, with its large numbers
of Italian and Austrian Catholics, added to those of German and Irish
origin already on the ground. The old fear of political domination by
the Papacy, expressed at the time of the adoption of the first
amendment to the Constitution, and then refuted, was again
revived. There was an orgy of purported “confessions” of nuns and
priests; there was circulated a forged oath of the Knights of
Columbus, in which the members agreed to place the papal
authority above their national allegiance; and a false encyclical of
Pope Leo XIII. Thousands of patriotic Americans believed all this
obvious nonsense, stirred up by the fear of a dominant Church; the
A. P. A. had as many as two million members and threatened to
drive out of public life the twelve million Catholics then in the
country, without regard to their race, nation, service to America, or
the number of generations they had lived in the United States. The
mob spirit, once aroused, crystalized in the breaking of the Northern
group mind of Civil War days into various sub-groups, Catholic, anti-
Catholic and indifferent. But the financial panic of the 1890’s resulted
in a sudden drop in immigration; the older settlers learned English
and were absorbed into the American cultural group; the A. P. A. had
no reason for existence, and again substantial unity was achieved by
the mind of the American people.
In this connection we must give a passing glance to what is still
our single greatest problem of groups, the existence of a ten per
cent. negro minority in the United States. These people were
brought here by force as slaves; as a subject class they were refused
education, though at the same time their own language, religion and
customs were thrown into disrepute and have been largely
forgotten. Though freed from economic slavery, they are still
politically a subject class in our southern states, while in northern
and border states they are gaining a political balance of power.
Finally, they rest everywhere under social disabilities, from the “Jim
Crow” cars of the South to the subtler distaste and ostracisms of the
North. The result is that they are forming complete, self-contained
Negro communities within the larger cities of the North and South
alike; that they are growing increasingly self-conscious as a group;
and that the large number of mulattoes, who in the British and
French West Indies would rank as a third group, between the racial
divisions, are here forced to make common cause with negroes. The
negroes are thus a self-conscious group, though their culture is
imitative. The grouping of the negroes apart is easy, on the whole,
because of the gross external signs, such as skin color and texture
of hair, so that the mass of the whites of the United States regard
them definitely as a different and a lower race. That anthropologists
are not so certain of all this makes little difference, because the
group mind is based rather on old habits of thought than on the
understanding of new and difficult facts. Here seems a problem of a
different order, then, than the racial and religious groupings of the
sub-varieties of the white race, which are constantly being overcome
and regrouped in a larger union of social life. In this study it will be
impossible to do more than point out the existence of this distinct
problem, with its similar mental background to the rest but its
immeasurably more terrible implications.
The fourth wave of immigration was by far the greatest in number
of newcomers, and by far the most variegated in racial and national
composition. It brought a million a year or more for six years during
this period. And its members had 75% of persons from southern and
eastern Europe, while the immigration prior to 1890 had included
only 20% of these races, and had been chiefly the English, Irish,
Germans and Scandinavians. It is no wonder that the race theory
began to be popular in America, under the spectacular leadership of
Lothrop Stoddard and Grant Madison, and that many began to
agitate for a greater or less limitation of the flood of immigration.
Even so sober a student of society as Professor Edward A. Ross of
Wisconsin held that it was wise to assimilate people of different
group mind more slowly than we were doing at the time. He said:
67
There have come among us in the last half century more than
twenty million European immigrants with all manner of mental
background, many of them having tradition which will no more blend
with American traditions than oil will blend with water.
And he proceeded to point out their inexperience with democratic
institutions, their lack of respect for law and for women, their
disbelief in progress. In addition, we need only note that many of
these people were Catholics and Jews; the total number of the
former in the United States in 1923 being estimated at 18,000,000
and of the latter at 3,600,000. And the Jews were far more
conspicuous than their numbers, on account of their massing in the
great cities and their concentration in certain lines of industry. Thus
the ground was fully prepared for a new anti-alien movement,
expressing itself this time in the form of efforts to restrict
immigration. This movement was under way in 1914, and would
probably have followed in the course of its precursors. But world-
shaking events ensued which altered the course of groups in
America as well. The outbreak of the World War in 1914, the
entrance of the United States into the war in 1917, altered all
groups, profoundly affected the American group mind, and made the
relation between the sub-groups and the mind of America very
different from what it had been. The results of this process are still
evident, and it is among them that we can look for anti-Semitism,
together with many other types of intolerance and group opposition.
 CHAPTER V.
THE WORLD WAR AND ITS AFTERMATH

1.
With the outbreak of the World War in August 1914, the mind of
America suddenly became strikingly distinct from that of Europe.
They were fighting; we were watching. President Wilson appealed to
the United States to be “neutral in fact as in name ... impartial in
thought as well as in action.” The older American stock sympathized,
on the whole, with England, except for the Irish and Germans; the
newer immigrants had different racial and national affinities and
memories, some holding allegiance to their former governments,
some, like the oppressed Russian Jews, being especially bitter
against their former rulers. In this situation, American neutrality was
the result, not of indifference, but of lack of understanding on the
part of many groups in our population and of a stalemate between
the rest.
One definite result certainly was that all these diverse groups of
new and old immigrants began to feel themselves a unity, an
American people. They felt their distinction from the warring nations
overseas, their own interest, their own reaction to the complex
problems at issue. Meanwhile, however, both parties were trying
every means to bring the United States into the war on their own
sides. Germany tried to bring about an embargo on munitions sold
to the Allies and in default of that, to obstruct their shipment by
both peaceful and warlike means. Great Britain, more especially,
tried to influence American public opinion in favor of the Allies and
against Germany. Within, there were pacifists and advocates of
preparedness, both trying to mold opinion. This formation of an
American mind, and the difficulty of determining its future direction,
came to a head in the election of 1916, when the German-Americans
opposed President Wilson, and when Hughes was supported by
Roosevelt, the arch-interventionist. During this period we
experienced the first development of what we have since grown to
know intimately as “propaganda,” a deliberate, elaborate technique
for influencing the mind of the group.
The declaration of war by the United States in April, 1917, unified
the American mind in a manner and to a degree that were almost
inconceivable. Every immigrant group began to pass resolutions
favoring the government; the foreign language newspapers
commenced an intensive propaganda for the prosecution of the war.
Volunteers came from every section of the country and every type of
origin, as many from the children of Germans as from any other
group. The draft law was passed with apparent general approval;
and its enforcement met with surprisingly little difficulty. Huge loans
were made to the Allied governments. Tremendous bond issues were
raised by the American government, with general approval and the
coercion of any minority objectors. The National Council of Defense,
founded in August, 1916, was able in many cases to overcome the
dominant profit-motive of our society in gaining self-sacrificing
patriotism of manufacturers and merchants.
Along with this voluntary and spontaneous unification of the
group mind, came repression and coercion directed to forcing into
agreement any unabsorbed minority groups. The Committee on
Public Information was founded in September, 1917, to exert
propaganda through the sources of public information, to send out
favorable news and opinion, and even through censorship to
suppress material considered dangerous to the general cause. The
censorship exercised by the military forces on war bulletins, war
correspondents and the personal letters of soldiers, was applied less
strictly to the general population. The secret service, greatly
expanded to cope with German spies, began hunting out strikers,
radicals or any others who—in the minds of the detectives or of any
other government officials—might possibly obstruct the war efforts.
Emergency acts gave the President unusual power in these and
other directions.
This use of force was characteristic, not only of the government,
but of local groups as well. In one place a German sympathizer (real
or supposed) might be made to kiss the flag; in another a strike
leader might be lynched. In Milwaukee, where public opinion was
sensitive on account of the large number of German-Americans, a
quota of Liberty Bonds was assigned arbitrarily to every person, and
he was practically forced to purchase them, irrespective of his ability
to do so, by threats of ostracism, by influence of his creditors, by
every sort of social pressure,—in order that Milwaukee might rank as
a real American community and go “over the top” in every “drive.”
The military language applied to these campaigns was matched by a
growing technique of organization. Professional propagandists
perfected a method of meetings, songs, card-catalogs, and quotas,
by which any cause might be assured of huge sums of money. The
greater propaganda of our government and foreign governments
was matched by the little propaganda of every subgroup, as long as
this was not in conflict with the general purpose.
A striking illustration of this is in the successful drives of the
various war-work agencies, the Red Cross, American Library
Association, Young Men’s Christian Association, Jewish Welfare
Board, Knights of Columbus, and the rest; and especially in their
enormous joint campaign just after the signing of the armistice.
Every American felt that this joint campaign, first, would help the
soldiers and the common cause; and second, indicated by its
inclusiveness the complete unification of America. Along with this
general unification came the similar process in many of the
68
immigrant groups themselves. Professor Miller tells how this was
reflected in the Czecho-Slovak group in America, so that bitter
atheists united with Catholic priests on joint committees for national
freedom in their old home in Europe.

2.
 This internal unification was accomplished by a high emotional
tension, a national and personal uncertainty, and a common hate.
The prejudice against the various immigrant groups, arising as a
result of the great wave of immigration, was abated for the moment;
all the little prejudices were summed up in one great hatred of the
common enemy, Germany. This was reflected in avoidance of
everything German in this country as well; German instruction was
withdrawn from many high schools, German music from the opera
houses, German fried potatoes from the restaurants. The term,
“German-American,” formerly in good repute, now became a
byword, and with it every form of “hyphen.” The demand now was
for “hundred per cent.” Americanism.
In the prevailing ignorance of foreign languages and peoples, or
even if this ignorance had not existed in its full measure, the hatred
against the Germans was transferred in part to other groups as well,
even those with most reason to be anti-German or anti-Austrian.
Foreign language newspapers fell under popular suspicion and
official censorship much heavier than that of the English language
periodicals. Some states passed laws, later declared unconstitutional,
forbidding teaching, preaching or public meetings in languages other
than English. Foreign sounding names attracted suspicion, and were
changed in large numbers. Altogether, America begun to repeat the
oppression of subject groups which had caused permanent
resentment and sown the seeds of rebellion in almost every land in
Europe, to create her own Ireland, Alsace-Lorraine or Poland.
Americanization became a synonym for compulsory adoption of
American standards and group habits.
Americanization had had a long, if somewhat unsatisfactory, trial
before the war. It was the attempt, at that time, to bring American
culture to the supposedly uncultured immigrant through settlements,
night schools, and other cultural agencies. The attempt was
satisfactory in a comparatively small proportion of the total
immigrant population; and the earnest workers blamed this fact on
the poorness of their textbooks, the unsuitability of their buildings,
or the weariness of the people after a day of arduous labor. Now, all
of these were undoubtedly true, but a more fundamental cause of
the weakness of Americanization methods lay in the fact that they
were all one-sided; they consisted in attempts to change the
immigrant into an American, rather than attempts to join many
groups together into a composite unity. Even the conference on
Americanization called by the Secretary of the Interior in 1918
passed friendly and practical resolutions, but still one-sided and
consequently superficial.
The few individuals who persisted in their individuality, who
refused to be absorbed in the group purpose, formed no clearly
marked group of themselves. They were the “conscientious
objectors,” who refused any type of activity that might help the
military machine; the “slackers,” who evaded the draft for selfish
reasons; various religious groups, such as the Quakers; a few
economic dissenters, such as the Industrial Workers of the World.
They received, as they must have expected, the violent disapproval
of the group, expressed in terms of mob attack, legal imprisonment,
or at least, extreme social disapproval. They were the unassimilated
residuum of personality in the general unification of the American
group under the pressure of an external foe.

3.
Then came the armistice in November, 1918. As Dr. Drachsler
remarks:
69
The war lasted long enough to make America painfully conscious
of her peculiar problem of nationalism, but was not of long enough
duration to fuse the divergent ethnic elements permanently.
The artificial unity of war-time had no longer a purpose, and began
instantly to dissolve into its component elements. But the high
emotional tone of the war-time remained. Men still hated violently,
but they could no longer release this hatred in battle or in sending
others to battle. The repressive agencies remained in existence and
in excellent running order; groups had learned how to use
propaganda as an instrument; the habit of group pressure on
subgroups and on different and opposing groups had been
strengthened. Most of all, great masses of Americans had a new
group consciousness of America as a group, with the uniformity of
habit, opinion and conduct characteristic of their own subgroup
taken as normal for the whole.
The first result, then, was that the original subgroups fell apart
and that their opposition was stronger and more open than before
the war. This was due certainly to the heightened emotional tone,
not only of the American mind, but every group mind the world over.
During the war men and nations lived habitually under conditions of
excitement, uncertainty and tension. After the war the same
emotional tone remained to color whatever group ideas might
become associated with its action. So the whites who had drafted
negroes to fight for them resented these same negroes coming
home with the new pride of soldiers, remembering new equality of
treatment they had received from the French. The daughters of the
rich no longer danced with the poor, ignorant farm boys as they had
in every cantonment. Prejudice against the uniform returned, and
girls of certain classes would no longer care to be seen with soldiers
or sailors; as they had when those men were expressing the group
purpose by their very garments. And the hatred of the various
immigrant groups for each other—the hatred of the older American
groups against the immigrant, the Catholic and the Jew, returned
with redoubled force. As the present writer found occasion to note
directly after the close of the war:
70
During the war we felt that prejudice between men of different
groups and different faiths was lessening day by day, that our
common enthusiasm in our common cause had brought Catholics,
Protestants and Jews nearer together on the basis of their ardent
Americanism. Especially we who were at the front felt this in the first
flush of our co-operation, our mutual interest and our mutual
helpfulness.
This disappointment was common to many of us who had allowed
our hopes to run beyond our knowledge.
 Another cause of this unusual strength of group hatreds was the
very repression of the war period. Individuals and sub-groups had
sacrificed their prejudices for the common purpose, but they had
done so without pleasure and as a sacrifice. Now they resumed their
group intolerance with redoubled zest due to long repression,
whether that had been voluntary or forced. The “white, gentile,
Protestant American” may have resented fighting on an equality with
the negro, or under the orders of a foreigner—now that resentment
had its vent. Never has group feeling run higher in America than in
this reaction from the sudden, violent and partially artificial unity
during our participation in the World War.
One notable result of this sudden relaxation of unity, this sudden
predominance of the subgroups, appeared in the phenomena of
displacement. Displacement is a common matter among paranoiacs,
where one object is substituted for another with the same meaning
and the same feeling-tone of resentment or of pleasure. It is also a
common characteristic of mobs, which may be called for this and
other reasons, a sort of social paranoiacs; the lynching mob will turn
from its intended victim to hang instead a public official or a
71
bystander who objects even mildly to its program. In this way the
hatreds of war-time were displaced. The hatred for the German was
displaced to the alien as a whole. The hatred and suspicion of
Russia, aroused when that nation drew out of the war, and
intensified when it adopted the radical economic program of the
Bolsheviki and the novel political rule of the Soviets, was displaced
and applied to all economic radicals, whether Russian or American.
Finally, the Jew was identified as a foreigner (even though he might
be American-born and a veteran of the war); and as a radical (even
though he might be an ultra-conservative capitalist). The ancient,
lingering anti-Judaism and anti-Semitism of ages past appeared
again; the Jew was not only a Christ-killer or a boor or a Semite,—
for no accusation was ever entirely dropped—he was also an alien
and a radical, an international banker and an enemy of gentile
civilization.
 CHAPTER VI.
THE KU KLUX KLAN AND OTHER GROUP
REACTIONS
The outstanding phenomenon of the post-war period was the Ku
Klux Klan. Other events which accompanied it were the new laws for
the limitation of immigration and the general suppression of civil
liberties of many kinds. The Klan had something to do with both of
these as cause and as effect. Moreover, all three—Klan, anti-alien
movement, anti-radical movement—were largely anti-Semitic in
sentiment; in addition to which there was a separate movement of
anti-Semitism based on the imported anti-Semitism from Europe.
Therefore in any study of anti-Semitism as a group reaction we must
also study these three group reactions of the post-war period, all of
them partially anti-Semitic, and all of them associated with the same
group-ideas and the same group-will as anti-Semitism itself.

1.
The Ku Klux Klan of the present is not the one of the
Reconstruction period in any sense. It has taken over the name, the
garb and much of the high-sounding ritual. But it has a new motive
and a new psychology. The old Klan was sectional; the new is
national. The old was anti-Northern and anti-negro; the new is anti-
alien, anti-negro, anti-Catholic and anti-Jewish. The old met a certain
emergency and was then disbanded by compulsion of the Federal
government and the action of its own leaders; the new has
expanded from the character of a fraternal society to that of a
nation-wide propaganda movement, has entered politics, and
become one of the leading political issues of the campaign of 1924.
In other words, its real ancestors are: not the Ku Klux Klan of the
south in 1866–71, but the Know-Nothing Party of the 50’s and the A.
P. A. of the 90’s.
The Ku Klux Klan was organized in 1915 in Atlanta, Ga., by
William J. Simmons, a former Protestant minister of strong
convictions, intense if narrow intellect, and great interest in the
organization and spreading of fraternal orders. For five years it grew
slowly and inconspicuously, during the period of the war and for two
years thereafter; in June 1920 it had about five thousand members
and was in financial straits. At this juncture it was taken up by Mr.
Edward Young Clarke and Mrs. Elizabeth Tyler, who had had
experience in the new technique of propaganda. Under their skilled
hands the Klan at once grew with astounding rapidity; paid
organizers entered state after state, organized “Klaverns,” and
reaped great profits for themselves and for the heads of the
organization. But the commercial motive, while probably strong in a
few persons, was in no sense important in the actual membership of
the Klan and their acts. “Its official documents indicate that the Klan
originally was a purely fraternal and patriotic organization, one of
72
the hundreds of similar secret societies throughout the country.”
The New York World investigated the Klan in 1921, and a
Congressional investigation followed in October of that year, but both
served rather to advertise than to harm the organization. It spread
rapidly throughout the Union, claiming at one time as many as four
million members, elected senators and governors in a few instances,
and in several became the outstanding issue of state elections,
sponsored or was accused of innumerable acts of mob violence,
ranging from warnings to certain persons to discontinue their
bootlegging or immorality, up to beatings, tar-and-feather parties,
and the notorious Mer Rouge murders of 1922 in Louisiana.
We have already discussed the expansion of propaganda, so that
its enormous utilization by the Klan is quite comprehensible. But
even the constant reiteration of laudable motives and grandiloquent
phrases about Americanism cannot account for this sudden rise to
power; two other elements must be included—group prejudice and
secrecy. The Klan capitalized every prejudice of its group, which was
predominantly a small-town one, of American birth, Protestant
religion, and Anglo-Saxon either in race or in their opinion of their
race. And the Klan met in utter secrecy, did not divulge the names of
its members, paraded the streets in the disguise of robes and masks,
and carried out its deeds of violence in the same awe-inspiring
anonymity.
Clearly, the Klan is typical of the tendencies we have found in the
American mind after the war. It represents a subgroup revolting
against its voluntary sacrifices for the nation during the war. It
represents the anti-alien, anti-Catholic and now also anti-Jewish
sentiment, the reaction against the enormous wave of immigration
just at an end. It includes also the fear and hatred of the negro,
strongest in the old South but spreading to the North with the
northern migration of many negroes during and after the war. On
the Pacific coast the fear of the Japanese immigration enters into the
complex of hatreds. In other words, the Klan is the third wave of
Nativism. It is the great reaction of the subgroup to the intense
sacrifice for the nation during the war.

2.
Various other motives are implicated in this general complex. The
South furnished the original soil of the Klan; its second center was
the middle west, the old home of the A. P. A. It was weakest on the
Atlantic and Pacific coasts (except Oregon) where the various
immigrant groups actually live. It was weak in the heterogeneous
masses of the cities with their aliens, Catholics and Jews; strongest
in the small town, where men may talk of the Papal menace without
actually knowing many Catholics, of the Elders of Zion without
seeing personally more than one or two Jews a year. The attitude of
Nativism, the reaction to the immigration of huge masses of
foreigners, is still strongest where these foreigners themselves are
not in evidence.
 This suggests that other motives must enter in, that something
else in the small-town American must have made the Klan congenial.
That something else is monotony, standardization (the “Main Street”
attitude), and the appeal of the Klan to these people lay largely in its
glamor of mystery, secrecy and hidden power. The rise of fraternal
orders is one of the note-worthy movements in American life; there
are now over six hundred of these societies in the United States, of
which four hundred ninety were organized between 1880 and 1895.
Over seven per cent. of our population is affiliated with these orders,
and their greatest strength is precisely in the small town, where they
are a bright spot in the dull social life, and give a factitious
importance to their “nobles” and “exalted rulers,” as well as to the
many who are permitted to enter into their secrets and to parade in
73
their regalia. Professor Mecklin classifies secret societies in three
groups: the beneficial societies, with whom secrecy is merely
protective; the social organizations, devised to give “variety and
interest to our poverty-stricken American life”; and finally, militant
societies with a general program which affects the entire nation, like
the old Ku Klux Klan, the Mafia, and the Fenians. He concludes that
the present Klan, while undoubtedly furnishing for many of its
members the release from monotony, the sense of power, the revolt
against repression, that is characteristic of the second class of
organizations, has also the characteristics of the third type and is
therefore a public problem. As he points out elsewhere in his book,
the disguise of the mask is a further danger, as it may be adopted by
members to persecute non-members in nameless ways, and even
presents an opportunity for non-Klansmen to indulge in violence
practically without fear of detection.
Professor Mecklin’s analysis of Klan psychology in Chapter IV of
his book presents several suggestive points. He says:
74
The strength of the Klan lies in that large, well-meaning, but
more or less ignorant and unthinking middle class, whose inflexible
loyalty has preserved with uncritical fidelity the traditions of the
original American stock.
 75
Membership in a vast mysterious Empire means a sort of mystic
glorification of his petty self.
The Klan insists on like-mindedness, in the sense of adopting the
Anglo-Saxon ideals as the norm for America. Finally,
76
The Klan has literally battened upon the irrational fear
psychology that followed on the heels of the war.
Father John A. Ryan contributes an additional motive,
77
There is a particular manifestation of public opinion which
deserves emphasis as a cause of the recent intolerance. This is the
conviction which seized large and numerous groups of individuals that
they were justified in becoming extra legal agents for law
enforcement.... Either the spirit or the letter of the law is violated in
the name of the law itself.

Frank Tannenbaum covers similar points in the first chapter of

“Darker Phases of the South,” where he deals with the Klan. He
holds, first, that
78
The Klan is an attempt to maintain static what has become
79
dynamic. The war left a common mood upon the world ... the hate
is generated as a means of justifying the thrill to be derived from
abusing the people hated. The Klan is a reaction to boredom; it is a
means of fulfilling the millennial hopes frustrated by the outcome of
80
the war; it gives vent to a type of war hysteria. The idealization of
the white women in the South is partly the unconscious self-protection
on the part of the white men from their own bad habits, notions,
beliefs, attitudes and practises, a matter of over-compensation.
To his keen psychoanalytic study I must add a few words from an
81
article by Frank Bohn in the American Journal of Sociology. Mr.
Bohn points out that the Klan, once organized, had to find
something to do, that its violence was a natural outcome of disguise,
organization and aimlessness. He attributes its origin chiefly to the
disillusionment of the American people over the break-down of their
simple, democratic ideals when applied to a huge nation of complex
population; and to the changing character of the racial and social
composition of the people, with the revolt of the older stocks. He
concludes:
The civilization of the United States is suffering rapid changes, not
only as regards its basic institutions, but also in the nature and quality
of its human composition. The hooded figures of the Ku Klux Klan are
an expression of pain, of sorrow and of solemn warning. Its methods
arise from anger and fear, not from knowledge and forethought.

3.
A word may be needed especially on our narrower topic, the
relation of the Ku Klux Klan to the Jew. Its preliminary questions to
the candidate for “naturalization” include two that exclude the
Catholic, two the Jew, one the alien and one the negro. The most
inclusive is number 2: “Are you a native born, white, Gentile
American citizen?” Number 4 is: “Do you believe in the tenets of the
Christian religion?” Imperial Wizard H. W. Evans gave out an
interview in Indianapolis early in 1924 when he made the following
statement, repeated several time later in other connections:
By deliberate election he (the Jew) is unassimilable. He rejects
intermarriage. His religious and social rites and customs are inflexibly
segregative. Law-abiding, healthy, moral, mentally alert, energetic,
loyal and reverent in his home life, the Jew is yet by primal instinct a
Jew, indelibly marked by persecution, with no deep national
attachment, a stranger to the emotion of patriotism as the Anglo-
Saxon feels it. Klansmen have no quarrel with him, no hatred of him,
no thought of persecuting him. As Protestants are unavailable for
membership in all-Jewish societies, so Jews are unavailable for
membership in an all-Protestant society like the Klan. Moreover, their
jealously guarded separatism unfits them for co-operation in a
movement dedicated to the thorough unification of the dominant
strains in American life.
Here are the same themes of racial superiority, like-mindedness of
America, identification of Americanism and Protestantism. But
elsewhere we meet with direct attacks on the Jew, as on the
Catholic, negro and foreigner—not merely the assertion of their
inferiority. Speaking at Dallas, Texas, December 7, 1922, Mr. Evans
said:
The Jew produces nothing anywhere on the face of the earth. He
does not till the soil. He does not create or manufacture anything for
common use. He adds nothing to the sum of human welfare.
Everywhere he stands between the producer and the consumer and
sweats the toil of the one and the necessity of the other for his gains.
This sounds like an economic motive, but it may be merely repetition
of stock charges of traditional anti-Semitism. Mr. Bohn hints at such
an economic purpose when he remarks:
One factor has been the recent invasion of the smaller western and
southern towns by Jewish retail merchants. These are disliked and
opposed by their native American competitors for purely commercial
reasons.
These facts seem to me erroneous; there have always been Jewish
merchants and peddlers throughout the country, and they have
always had Christian competitors; probably they have merely been a
point of vantage for the aroused prejudices of the group. Dr. Mecklin
says:
82
The Klan insists, in the published statements of its ideals, upon
complete religious toleration while in actual practise it encourages
83
boycotts of Catholic and Jew in business and social relations. The
eternal quarrel of the Klan with the Jew and the Negro is that mental
and physical differences seem to have conspired to place them in
groups entirely to themselves.... The Negro is granted a place in
American society only upon his willingness to accept a subordinate
position. The Jew is tolerated largely because native Americanism
cannot help itself. The Jew is disliked because of the amazing tenacity
with which he resists absolute Americanization, a dislike that is not
unmingled with fear; the Negro is disliked, because he is considered
essentially an alien and unassimilable element in society.

4.
The Klan has now passed the zenith of its aggressiveness and its
influence. The campaign of exposure, while it made thousands of
members, also made thousands of enemies and robbed the Klan of
the secrecy which was so essential an element of its strength. Many
of its members lost interest, others were positively estranged by
certain methods and ideals of the organization. The trials for murder
at Mer Rouge, La., brought the Klan into bad odor generally. Most
important of all, the Klan went into politics, and in this followed
exactly the cycle of the Know-Nothings and A. P. A.’s—secrecy,
growth, propaganda, politics, enemies, decline. In 1924 the Klan
was an element in the national conventions of the two major parties.
The Republicans considered planks opposing and favoring the
organization and finally took no action. The Democrats had to take
up the issue because of the movement to nominate as their
presidential candidate Alfred E. Smith, governor of New York, and a
professing Catholic. While Mr. Smith had political supporters in his
own state of every religious denomination, still the entire strength of
the Klan was thrown against him. At the same time, the many Irish
Catholics belonging to the Democratic party resented the attempt of
the Klan to dictate the nomination and introduced a resolution
attacking the Klan by name. The conflict of that convention is now
historic, and resulted in thoroughly disorganizing the Democratic
party for the ensuing campaign.
Finally, the passage of the immigration bills of 1921 and 1924
robbed the Klan of its chief reason for existence, its most potent
argument. Immigration was abruptly cut down. Not only that, but its
national origin was totally altered so as to favor the peoples of
northern and western Europe, and to keep out the Italian Catholics
and Russian Jews. It is no longer possible to stimulate fear or hatred
on such a large scale again, now that immigration is no longer a
large factor in American life, and the group integration is once more
proceeding at its accustomed rate.

5.
The anti-immigration movement must not be regarded as a result
of the Klan but as a parallel phenomenon, with the same motives
and philosophy. The original political theory and economic situation,
by which all immigrants were welcomed into the United States to
help build up the country and to become full Americans has been
slowly altering. The first law of limitation, passed in 1882, and
followed up by later amendments, merely excluded convicts, persons
affected with contagious diseases, persons likely to become public
charges, and similar individuals for individual reasons. Other
legislation of economic trend excluded Chinese and later Japanese
laborers, and contract labor. In 1917 the demand to limit the
numbers of common labor, voiced by the American Federation of
Labor, met the desire to limit numbers and to select racial groups,
and the literacy test was embodied in the law, excluding all who
could not read or write in any language. But this was satisfactory to
neither the friends nor foes of immigration; it was merely a
temporary device.
In May 1921 a temporary law was passed limiting the number of
each nation to enter the United States annually to 3% of natives of
that nation residing here in 1910. This limited the total immigration
at once from the 1,285,349 of 1907, the peak year, to a total of
357,803. This total is in addition to immigrants from Canada, Mexico,
Newfoundland, Cuba and Central and South America; it does not
deduct the emigrants who often amount to as many or more than
those entering the country. It is simply a means of cutting down
numbers and altering proportions. It is directly a result of Klan
preachments, of Nordic theories, of the reaction of the native,
gentile, Protestant American to the growing complexity and
heterogeneity of the nation, and to the need of revising his mental
stereotypes of the United States. He must grow to think of his nation
as a nation of many elements, many beliefs, many backgrounds,
most of them different from his own—to him America is a Protestant
country, a white man’s country, a gentile country, and he intends
that it shall remain so.
Therefore the permanent immigration bill enacted in May, 1924,
changed the percentage from three to two, and the date on which
the quota is to be estimated from 1910 to 1890. The result of this
double change is to alter radically the racial and national composition
of the immigration stream and hence the total character of the
United States. As Chairman Albert Johnson of the House Committee
on Immigration, after whom the bill was named, phrased its double
purpose:
84
The committee took a very important step in recommending a
permanent percentage law and thus recognizing the principle that the
United States should never keep its doors wide open. Second, the
percentage is based on the census of 1890 instead of the census of
1910, as in the present law. The new measure thus aims to change
the character of our future immigration by cutting down the number
of aliens who can come from southern and eastern Europe. In other
words, it is recognized that, on the whole, northern and western
Europe furnish the best material for citizenship.
The total immigration, therefore, was reduced from 357,000 to
164,667 and the emigrants have to be deducted from this to
ascertain the actual annual increase. The Italian quota was reduced
from 42,000 to 3,845; the Russian from 24,000 to 2,200; the Polish
from 30,000 to 6,000. On the other hand, the German quota was
reduced only from 67,000 to 51,000; the Norwegian from 12,000 to
6,400; the British and Irish from 77,000 to 62,500. The bill carried
out radically the intentions of its sponsors, to cut down the flood of
immigration and to discriminate against the racial and religious
groups which they consider inferior because they appear externally
to be different. It is a group reaction of the same order and
motivation as the Ku Klux Klan.

6.
A concurrent phenomenon, arising from the same group mind but
essentially different in manifestation, is the suppression of civil
liberties which began during the war and continued afterward, an
expression of the same impulse toward compulsory like-mindedness,
but taking its criterion from the economic rather than the cultural,
religious or racial aspects of the differing groups. As Father Ryan put
it:
 85
These deplorable phenomena are three-fourths due to war
legislation and surviving war hysteria and one-fourth due to industrial
factors.... By means of clever, unscrupulous and wholesale
propaganda, nine-tenths of the American people were led to believe
that the steel strike of 1919 was revolutionary, bolshevistic, and aimed
immediately at the overthrow of the government. As a matter of fact,
there was no more bolshevism in that contest than in any one of a
dozen important disputes that have occurred in the last ten years.
Attorney General Palmer asserted that there was an organized
attempt to overthrow the government of the United States sufficiently
widespread to merit the attention of Congress. As a matter of fact,
there was no such danger.
86
Dr. Harry F. Ward of Union Theological Seminary, in the same
Proceedings of the American Sociological Society, has a fine
summary of the “Repression of Civil Liberties in the United States
(1918–23).” He enumerates the new Supreme Court interpretation of
the free-speech clause of the first amendment to the Constitution,
by which a “clear and present danger” justifies its violation; the state
laws on syndicalism or sedition or anarchy; the attacks on the right
of labor to strike; the use of the Department of Justice of the United
States to repress radical economic movements; the mob violence
increasingly widespread and regular; and the national organizations
engaged in repression, such as the National Civic Federation, the
National Security League, and the Better American Federation.
The material is too wide in range and too full of important
instances to be even cursorily examined here. The trend, however,
was definitely a part of the post-war attitude of the American mind,
the breaking up into violently opposing groups, each claiming to
assert the true American spirit. The same attitude of repression
appears in the churches in the form of heresy trials and an
aggressive Fundamentalism. It appears in the form of legislative acts
to prohibit the teaching of evolution in the state universities of
several Southern states (most of which failed of passage). Dr. Ward
feels that the
Mob attacks, lynchings and prosecutions involving the use of free
speech reached their peak at the end of 1922, declining rapidly in.
 1923. Interference with meetings by public authorities and private
groups reached a peak at the end of 1921, fell sharply in 1922, and
87
then went up again to a midway point in 1923.... We have a
manifest abatement of post-war repression, but that experience has
left us a heritage of repressive laws and ordinances and a technique
of administrative illegality all ready to be used on due occasion. It has
also strengthened our lynching habit of mind, with its determination
to enforce its type of goodness, and our traditional demand for
conformity already overstimulated by the increasing standardization of
life. The occasions for the use of those qualities and instruments of
repression are increasing rather than diminishing.
Attempts were made during the height of the anti-Russian and
anti-radical movement to connect Jews with Bolshevism in Russia
and with radicalism in the United States, so that this movement also
has its anti-Semitic phase. Thus anti-Semitism is bound up with the
Ku Klux Klan, with the immigration bills, with the economic
repression,—it is an integral part of the group reaction from national
unity, and appears in every phase of the post-war group reactions.
 CHAPTER VII.
ANTI-SEMITISM
In “Loyalties” by John Galsworthy, there occur two statements of
anti-Semitism so powerful and so keen that they may serve as a key
to the whole situation. The young Jew has accused a Christian
aristocrat of stealing his purse. The gentile girl, naturally a liberal,
has to choose her loyalty. She says: “Oh! I know lots of splendid
Jews, and I rather like little Ferdie; but when it comes to the point—
they all stick together; why shouldn’t we! It’s in the blood....
Prejudices—or are they loyalties—I don’t know—criss-cross—we all
cut each other’s throats from the best of motives.” And later on an
English grocer of the lower middle class confesses: “To tell you the
truth, I don’t like—well, not to put too fine a point on it—‘ebrews.
They work harder; they’re more sober; they’re honest; and they’re
everywhere. I’ve nothing against them, but the fact is—they get on
so.”

1.
Anti-Semitism is, then, a typical because a violent group attitude.
In America in its newest manifestation it is a part of the complex of
group revolts after the World War; it is intimately associated with the
Ku Klux Klan, anti-immigration movement, and repression generally,
at the same time that it has distinctive phases of its own. As Lewis
S. Gannett wrote:
88
Because anti-Semitism is world-wide it is easy to assume that
it has the same causes everywhere; but conditions in America are
very different from conditions in countries where religion is taught in
the schools, where the Jews are virtually all middlemen, where the
Welcome to our website – the ideal destination for book lovers and
knowledge seekers. With a mission to inspire endlessly, we offer a
vast collection of books, ranging from classic literary works to
specialized publications, self-development books, and children's
literature. Each book is a new journey of discovery, expanding
knowledge and enriching the soul of the reade

Our website is not just a platform for buying books, but a bridge
connecting readers to the timeless values of culture and wisdom. With
an elegant, user-friendly interface and an intelligent search system,
we are committed to providing a quick and convenient shopping
experience. Additionally, our special promotions and home delivery
services ensure that you save time and fully enjoy the joy of reading.

Let us accompany you on the journey of exploring knowledge and

personal growth!

ebookfinal.com