Visit https://ebookgate.

com to download the full version and

explore more ebooks

Linear programming with MATLAB 1st Edition Michael

C. Ferris

_____ Click the link below to download _____

https://ebookgate.com/product/linear-programming-with-
matlab-1st-edition-michael-c-ferris/

Explore and download more ebooks at ebookgate.com

Here are some recommended products that might interest you.
You can download now and explore!

Linear Control System Analysis and Design with MATLAB

Sixth Edition Constantine H. Houpis

https://ebookgate.com/product/linear-control-system-analysis-and-
design-with-matlab-sixth-edition-constantine-h-houpis/

ebookgate.com

Linear Control System Analysis and Design with MATLAB 6th

Edition Constantine H. Houpis

https://ebookgate.com/product/linear-control-system-analysis-and-
design-with-matlab-6th-edition-constantine-h-houpis/

ebookgate.com

Programming embedded systems with C and GNU development

tools 2nd Edition Michael Barr

https://ebookgate.com/product/programming-embedded-systems-with-c-and-
gnu-development-tools-2nd-edition-michael-barr/

ebookgate.com

Programming With C 3rd Edition Bl Juneja

https://ebookgate.com/product/programming-with-c-3rd-edition-bl-
juneja/

ebookgate.com
CryENGINE Game Programming with C C and Lua 1st Edition
Lundgren

https://ebookgate.com/product/cryengine-game-programming-with-c-c-and-
lua-1st-edition-lundgren/

ebookgate.com

MATLAB Graphical Programming Practical hands on MATLAB

solutions 1st Edition Cesar Perez Lopez

https://ebookgate.com/product/matlab-graphical-programming-practical-
hands-on-matlab-solutions-1st-edition-cesar-perez-lopez/

ebookgate.com

Programming with ANSI C 2nd ed Edition Trivedi

https://ebookgate.com/product/programming-with-ansi-c-2nd-ed-edition-
trivedi/

ebookgate.com

Fundamentals of Computer Programming with C The Bulgarian

C Book 1st Edition Svetlin Nakov

https://ebookgate.com/product/fundamentals-of-computer-programming-
with-c-the-bulgarian-c-book-1st-edition-svetlin-nakov/

ebookgate.com

Programming Windows Store Apps with C 1st Edition Matthew

Baxter-Reynolds

https://ebookgate.com/product/programming-windows-store-apps-
with-c-1st-edition-matthew-baxter-reynolds/

ebookgate.com
MP07_Ferris_FMA.qxp 10/4/2007 2:22 PM Page 1

LINEAR PROGRAMMING
WITH MATLAB
MP07_Ferris_FMA.qxp 10/4/2007 2:22 PM Page 2

MPS-SIAM Series on Optimization

This series is published jointly by the Mathematical Programming Society and the Society
for Industrial and Applied Mathematics. It includes research monographs, books on
applications, textbooks at all levels, and tutorials. Besides being of high scientific quality,
books in the series must advance the understanding and practice of optimization. They
must also be written clearly and at an appropriate level.

Editor-in-Chief
Philippe Toint, University of Namur (FUNDP)

Editorial Board
Oktay Gunluk, IBM T.J. Watson Research Center
Matthias Heinkenschloss, Rice University
C.T. Kelley, North Carolina State University
Adrian S. Lewis, Cornell University
Pablo Parrilo, Massachusetts Institute of Technology
Daniel Ralph, University of Cambridge
Mike Todd, Cornell University
Laurence Wolsey, Université Catholique de Louvain
Yinyu Ye, Stanford University

Series Volumes
Ferris, Michael C., Mangasarian, Olvi L., and Wright, Stephen J., Linear Programming with
MATLAB
Attouch, Hedy, Buttazzo, Giuseppe, and Michaille, Gérard, Variational Analysis in Sobolev
and BV Spaces: Applications to PDEs and Optimization
Wallace, Stein W. and Ziemba, William T., editors, Applications of Stochastic Programming
Grötschel, Martin, editor, The Sharpest Cut: The Impact of Manfred Padberg and His Work
Renegar, James, A Mathematical View of Interior-Point Methods in Convex Optimization
Ben-Tal, Aharon and Nemirovski, Arkadi, Lectures on Modern Convex Optimization:
Analysis, Algorithms, and Engineering Applications
Conn, Andrew R., Gould, Nicholas I. M., and Toint, Phillippe L., Trust-Region Methods
MP07_Ferris_FMA.qxp 10/4/2007 2:22 PM Page 3

LINEAR PROGRAMMING
WITH MATLAB

Michael C. Ferris
Olvi L. Mangasarian
Stephen J. Wright
University of Wisconsin–Madison
Madison, Wisconsin

Society for Industrial and Applied Mathematics Mathematical Programming Society

Philadelphia Philadelphia
MP07_Ferris_FMA.qxp 10/4/2007 2:22 PM Page 4

Copyright © 2007 by the Society for Industrial and Applied Mathematics and the
Mathematical Programming Society

10 9 8 7 6 5 4 3 2 1

All rights reserved. Printed in the United States of America. No part of this book may be
reproduced, stored, or transmitted in any manner without the written permission of the
publisher. For information, write to the Society for Industrial and Applied Mathematics,
3600 Market Street, 6th floor, Philadelphia, PA 19104-2688 USA.

Trademarked names may be used in this book without the inclusion of a trademark symbol.
These names are used in an editorial context only; no infringement of trademark is intended.

Maple is a registered trademark of Waterloo Maple, Inc.

MATLAB is a registered trademark of The MathWorks, Inc. For MATLAB product information,
please contact The MathWorks, Inc., 3 Apple Hill Drive, Natick, MA 01760-2098 USA,
508-647-7000, Fax: 508-647-7101, info@mathworks.com, www.mathworks.com.

Library of Congress Cataloging-in-Publication Data

Ferris, Michael C.
Linear programming with MATLAB / Michael C. Ferris, Olvi L. Mangasarian, Stephen J. Wright.
p. cm. — (MPS-SIAM series on optimization ; 7)
Includes bibliographical references and index.
ISBN 978-0-898716-43-6 (alk. paper)
1. Linear programming—Data processing. 2. MATLAB. 3. Mathematical optimization. 4. Algebras,
Linear. I. Mangasarian, Olvi L., 1934- II. Wright, Stephen J., 1960- III. Title.

QA402.5.F425 2007
519.7’2—dc22
2007061748

is a registered trademark. is a registered trademark.

MP07_Ferris_FMA.qxp 10/4/2007 2:22 PM Page 5

To Jane, Claire, and Jill


MP07_Ferris_FMA.qxp 10/4/2007 2:22 PM Page 6
 Contents

Preface xi

1 Introduction 1
1.1 An Example: The Professor’s Dairy . . . . . . . . . . . . . . . . . . . 2
1.1.1 The Setup . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.1.2 Formulating the Problem and a Graphical Solution . . . . 2
1.1.3 Changing the Problem . . . . . . . . . . . . . . . . . . . 4
1.1.4 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . 6
1.2 Formulations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
1.3 Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
1.3.1 The Diet Problem . . . . . . . . . . . . . . . . . . . . . 8
1.3.2 Linear Surface Fitting . . . . . . . . . . . . . . . . . . . 9
1.3.3 Load Balancing Problem . . . . . . . . . . . . . . . . . . 10
1.3.4 Resource Allocation . . . . . . . . . . . . . . . . . . . . 10
1.3.5 Classification . . . . . . . . . . . . . . . . . . . . . . . . 11
1.3.6 Minimum-Cost Network Flow . . . . . . . . . . . . . . . 12
1.4 Algorithms and Complexity . . . . . . . . . . . . . . . . . . . . . . . 14
1.4.1 The Simplex Method . . . . . . . . . . . . . . . . . . . . 14
1.4.2 Interior-Point Methods . . . . . . . . . . . . . . . . . . . 15

2 Linear Algebra: A Constructive Approach 17

2.1 Jordan Exchange . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
2.2 Linear Independence . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
2.3 Matrix Inversion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
2.4 Exact Solution of m Equations in n Unknowns . . . . . . . . . . . . . 32
2.5 Solving Linear Equations Efficiently . . . . . . . . . . . . . . . . . . 39
2.6 LU Decomposition . . . . . . . . . . . . . . . . . . . . . . . . . . . 41

3 The Simplex Method 45

3.1 A Simple Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
3.2 Vertices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
3.3 The Phase II Procedure . . . . . . . . . . . . . . . . . . . . . . . . . 53
3.4 The Phase I Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . 60
3.5 Finite Termination . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65

vii
viii Contents

3.5.1 The Nondegenerate Case . . . . . . . . . . . . . . . . . 65

3.5.2 Cycling . . . . . . . . . . . . . . . . . . . . . . . . . . . 66
3.5.3 The General Case . . . . . . . . . . . . . . . . . . . . . 67
3.6 Linear Programs in Nonstandard Form . . . . . . . . . . . . . . . . . 72
3.6.1 Transforming Constraints and Variables . . . . . . . . . . 72
3.6.2 Scheme I . . . . . . . . . . . . . . . . . . . . . . . . . . 76
3.6.3 Scheme II . . . . . . . . . . . . . . . . . . . . . . . . . 80
3.6.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . 86

4 Duality 89
4.1 Duality and Rank in Linear Systems . . . . . . . . . . . . . . . . . . 89
4.2 Duality in Linear Programming . . . . . . . . . . . . . . . . . . . . . 94
4.3 Interpretation of Linear Programming Duality . . . . . . . . . . . . . 96
4.4 Duality Theory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97
4.5 KKT Optimality Conditions . . . . . . . . . . . . . . . . . . . . . . . 100
4.6 Dual Simplex Method . . . . . . . . . . . . . . . . . . . . . . . . . . 102
4.7 General Linear Programs . . . . . . . . . . . . . . . . . . . . . . . . 107
4.8 Big M Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110
4.9 Applications of Duality . . . . . . . . . . . . . . . . . . . . . . . . . 112

5 Solving Large Linear Programs 117

5.1 Foundations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118
5.1.1 Basic Feasible Solutions and Basis Matrices . . . . . . . 118
5.1.2 Geometric Viewpoint . . . . . . . . . . . . . . . . . . . 121
5.2 The Revised Simplex Method . . . . . . . . . . . . . . . . . . . . . . 123
5.2.1 Upper and Lower Bounds . . . . . . . . . . . . . . . . . 129
5.2.2 Generating Basic Feasible Solutions . . . . . . . . . . . 134
5.2.3 Basis Updates . . . . . . . . . . . . . . . . . . . . . . . 139
5.2.4 Advanced Pivot Selection Mechanisms . . . . . . . . . . 142
5.3 Network Flow Problems . . . . . . . . . . . . . . . . . . . . . . . . . 143
5.3.1 Minimum-Cost Network Flow . . . . . . . . . . . . . . . 144
5.3.2 Shortest-Path Problem . . . . . . . . . . . . . . . . . . . 145
5.3.3 Max-Flow Problem . . . . . . . . . . . . . . . . . . . . 146
5.3.4 Transportation Problem . . . . . . . . . . . . . . . . . . 147
5.3.5 Assignment Problem . . . . . . . . . . . . . . . . . . . . 149
5.3.6 Network Simplex Method . . . . . . . . . . . . . . . . . 149

6 Sensitivity and Parametric Linear Programming 151

6.1 Sensitivity Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . 151
6.2 Adding New Variables or Constraints . . . . . . . . . . . . . . . . . . 155
6.3 Parametric Optimization of the Objective Function . . . . . . . . . . . 158
6.4 Parametric Optimization of the Right-Hand Side . . . . . . . . . . . . 164

7 Quadratic Programming and Complementarity Problems 169

7.1 Nonlinear Programs: Optimality Conditions . . . . . . . . . . . . . . 169
7.2 Quadratic Programming . . . . . . . . . . . . . . . . . . . . . . . . . 172
Contents ix

7.2.1 Basic Existence Result . . . . . . . . . . . . . . . . . . . 172

7.2.2 KKT Conditions . . . . . . . . . . . . . . . . . . . . . . 173
7.2.3 Duality . . . . . . . . . . . . . . . . . . . . . . . . . . . 176
7.3 Linear Complementarity Problems . . . . . . . . . . . . . . . . . . . 177
7.4 Lemke’s Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 178
7.5 Bimatrix Games . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 185
7.5.1 Computing Nash Equilibria . . . . . . . . . . . . . . . . 186
7.5.2 Zero-Sum Games As Dual Linear Programs . . . . . . . 192

8 Interior-Point Methods 195

8.1 Motivation and Outline . . . . . . . . . . . . . . . . . . . . . . . . . 195
8.2 Newton’s Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . 197
8.3 Primal-Dual Methods . . . . . . . . . . . . . . . . . . . . . . . . . . 201
8.3.1 An Affine-Scaling Approach . . . . . . . . . . . . . . . . 202
8.3.2 Path-Following Methods . . . . . . . . . . . . . . . . . . 204
8.3.3 Solution of the Linear System at Each Interior-Point
Iteration . . . . . . . . . . . . . . . . . . . . . . . . . . 208
8.3.4 Practical Primal-Dual Methods . . . . . . . . . . . . . . 209
8.4 Interior-Point vs. Simplex . . . . . . . . . . . . . . . . . . . . . . . . 212
8.5 Extension to Quadratic Programming . . . . . . . . . . . . . . . . . . 212

9 Approximation and Classification 217

9.1 Minimax Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . 217
9.2 Approximation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 218
9.2.1 Chebyshev Approximation . . . . . . . . . . . . . . . . . 219
9.2.2 L1 Approximation . . . . . . . . . . . . . . . . . . . . . 221
9.2.3 Approximate Solutions to Systems with Inequality
Constraints . . . . . . . . . . . . . . . . . . . . . . . . . 223
9.2.4 Least-Squares Problems . . . . . . . . . . . . . . . . . . 224
9.3 Huber Estimation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 227
9.4 Classification Problems . . . . . . . . . . . . . . . . . . . . . . . . . 230

A Linear Algebra, Convexity, and Nonlinear Functions 237

A.1 Linear Algebra . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 237
A.2 Convex Sets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 239
A.3 Smooth Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 242
A.4 Convex Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 242
A.5 Quadratic Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . 244
A.6 Norms and Order Notation . . . . . . . . . . . . . . . . . . . . . . . 247
A.7 Taylor’s Theorem . . . . . . . . . . . . . . . . . . . . . . . . . . . . 249

B Summary of Available MATLAB Commands 251

B.1 Basic MATLAB Operations . . . . . . . . . . . . . . . . . . . . . . . 251
B.2 MATLAB Functions Defined in This Book . . . . . . . . . . . . . . . 252
x Contents

Bibliography 257

Index 261
 Preface

This text has grown over many years from a set of class notes for an undergraduate linear
programming course offered at the University of Wisconsin-Madison. Though targeted to
Computer Science undergraduates, the course has attracted both undergraduates and be-
ginning graduate students from many other departments, including Industrial and Systems
Engineering, Statistics, and Mathematics. The course aims to provide a one-semester ele-
mentary introduction to linear programming formulations, algorithms, computations, and
applications. Only basic knowledge of linear algebra and calculus is required.
One feature of our approach is the use of MATLAB codes to demonstrate the com-
putational aspects of the course, from the elementary manipulations that form the building
blocks of algorithms to full implementations of revised simplex and interior-point methods.
(The latter are clearly not robust or efficient enough to solve larger practical problems,
but they do illustrate the basic principles of the computational methods in question.) The
MATLAB codes (and associated mex files) are distributed on the web site associated with
the book: www.siam.org/books/mp07.
We have included a chapter on quadratic programs and complementarity problems,
which are topics whose importance in a number of application areas appears to be growing
by the day. The final chapter deals with approximation and classification problems, which
are of interest to statisticians and others, showing how these problems can be formulated and
solved as linear or quadratic programs. An earlier chapter deals with the topic of duality,
which is of interest not only because of the insight it provides into the beautiful theory
underlying linear programming but also because of its usefulness in formulating practical
problems. (The dual of a problem may be easier to solve than the primal, or it might provide
a bound on the optimal solution value or other useful information.)
A one-semester undergraduate class should include most of the chapters in the text.
If time is pressing, some of the later chapters could be omitted in part or in their entirety.
However, we believe that all topics covered are interesting and relevant to the intended
student audience, and so we hope that most teachers can find a way to incorporate them into
their curriculum.
We thank the students and colleagues who have given us feedback on the manuscript
during its development, particularly our colleague Bob Meyer. We are also grateful to the
referees of the manuscript who read it thoroughly and provided valuable suggestions for
improvement, most of which we adopted. Finally, we thank our wives and families for their
love and support over many years.
Madison, Wisconsin, USA
Spring 2007
xi
Chapter 1

Introduction

Nothing happens in the universe that does not have a sense of either certain maximum or
minimum. L. Euler, Swiss Mathematician and Physicist, 1707–1783

Optimization is a fundamental tool for understanding nature, science, engineering,

economics, and mathematics. Physical and chemical systems tend to a state that minimizes
some measure of their energy. People try to operate man-made systems (for example, a
chemical plant, a cancer treatment device, an investment portfolio, or a nation’s economy)
to optimize their performance in some sense. Consider the following examples:

1. Given a range of foods to choose from, what is the diet of lowest cost that meets an
individual’s nutritional requirements?

2. What is the most profitable schedule an airline can devise given a particular fleet of
planes, a certain level of staffing, and expected demands on the various routes?

3. Where should a company locate its factories and warehouses so that the costs of
transporting raw materials and finished products are minimized?

4. How should the equipment in an oil refinery be operated, so as to maximize rate of

production while meeting given standards of quality?

5. What is the best treatment plan for a cancer patient, given the characteristics of the
tumor and its proximity to vital organs?

Simple problems of this type can sometimes be solved by common sense, or by using
tools from calculus. Others can be formulated as optimization problems, in which the goal
is to select values that maximize or minimize a given objective function, subject to certain
constraints. In the next section, we show how a practical problem can be formulated as a
particular type of optimization problem known as a linear program.

1
2 Chapter 1. Introduction

1.1 An Example: The Professor’s Dairy

1.1.1 The Setup
University professors sometimes engage in businesses to make a little extra cash. Professor
Snape and his family run a business that produces and sells dairy products from the milk
of the family cows, Daisy, Ermentrude, and Florence. Together, the three cows produce 22
gallons of milk each week, and Snape and his family turn the milk into ice cream and butter
that they then sell at the Farmer’s Market each Saturday morning.
The butter-making process requires 2 gallons of milk to produce one kilogram of
butter, and 3 gallons of milk is required to make one gallon of ice cream. Professor Snape
owns a huge refrigerator that can store practically unlimited amounts of butter, but his freezer
can hold at most 6 gallons of ice cream.
Snape’s family has at most 6 hours per week in total to spend on manufacturing their
delicious products. One hour of work is needed to produce either 4 gallons of ice cream or
one kilogram of butter. Any fraction of one hour is needed to produce the corresponding
fraction of product.
Professor Snape’s products have a great reputation, and he always sells everything
he produces. He sets the prices to ensure a profit of $5 per gallon of ice cream and $4 per
kilogram of butter. He would like to figure out how much ice cream and butter he should
produce to maximize his profit.

1.1.2 Formulating the Problem and a Graphical Solution

The first step in formulating this problem is to identify the two variables, which are the
quantities that we are able to vary. These are the number of gallons of ice cream, which we
denote by x, and the number of kilograms of butter, which we denote by y. Next, we figure
out how the objective function depends on these variables. We denote the objective (which
in this case is the profit) by z, and note that it is simply z = 5x + 4y dollars in this example.
Since we aim to maximize the production, it is generally in our interest to choose x
and y as large as possible. However, the constraints on production mentioned above prevent
us from making these variables too large. We now formulate the various constraints in the
description above algebraically.

• The 6-gallon constraint on freezer capacity causes us to impose the constraint x ≤ 6.

• The total amount of labor required to produce x gallons of ice cream and y kilograms
of butter is .25x + y. Since the family can labor for a total of at most 6 hours during
the week, we have the constraint .25x + y ≤ 6.

• We look at the amount of milk needed by the production process. The total number
of gallons of milk used is 3x + 2y, and since there are 22 gallons of milk available,
we have the constraint 3x + 2y ≤ 22.

• Finally, the problem must include the simple constraints x ≥ 0, y ≥ 0, because it

does not make sense to produce negative amounts of ice cream or butter.
1.1. An Example: The Professor’s Dairy 3

labor constraint (.25 x + y <= 6) freezer constraint (x <= 6)

y (butter) solution (4,5)

milk constraint (3x+2y <= 22)

z=
40
x (ice cream)
z=

z=
z=
10

30
20

Figure 1.1. The Professor’s dairy: Constraints and objective.

Summarizing, we can express the linear program mathematically as follows:

max z = 5x + 4y
x,y
subject to x ≤ 6,
.25x + y ≤ 6, (1.1)
3x + 2y ≤ 22,
x, y ≥ 0.
Figure 1.1 illustrates this problem graphically, plotting the variable x along the hori-
zontal axis and y along the vertical axis. Each constraint is represented by a line, the shaded
side of the line representing the region of the (x, y) plane that fails to satisfy the constraint.
For example, the constraint 3x + 2y ≤ 22 is represented by the line 3x + 2y = 22 (obtained
by replacing the inequality by an equality), with the “upper” side of the line shaded. In
general, we can determine which side of the line satisfies the constraint and which does not
by picking a point that does not lie on the line and determining whether or not the constraint
is satisfied at this point. If so, then all points on this side of the line are feasible; if not, then
all points on this side of the line are infeasible.
The set of points satisfying all five of the constraints is known as the feasible region.
In this problem the feasible region is the five-sided polygonal region in the middle of the
figure.
The linear programming problem is to the find a point in this feasible region that
maximizes the objective z = 5x + 4y. As a step towards this goal, we plot in Figure 1.1
a dotted line representing the set of points at which z = 20. This line indicates feasible
points such as (x, y) = (0, 5) and (x, y) = (2, 2.5) that yield a profit of $20. Similarly,
we plot the line z = 5x + 4y = 30—the set of points that achieves a profit of $30. Note
4 Chapter 1. Introduction

that this line (and all other lines of constant z) is parallel to the line z = 20. In fact, we can
maximize profit over the feasible region by moving this line as far as possible to the right
while keeping some overlap with the feasible region and keeping it parallel to the z = 20
line. It is not difficult to see that this process will lead us to a profit of z = 40 and that
this line intersects the feasible region at the single point (x, y) = (4, 5). Note that this
point is a “corner point” of the feasible region, corresponding to the point at which two
of the constraints—the limit of milk supply and the limit on labor supply—are satisfied as
equalities.

1.1.3 Changing the Problem

The graphical representation of Figure 1.1 can be used to see how the solution changes when
the data is changed in certain ways. An investigation of this type is known as sensitivity
analysis and will be discussed in Chapter 6. We discuss two possible changes to the example
problem here. A first time reader may skip this section without loss of continuity since it is
meant primarily as an intuitive graphical introduction to duality and sensitivity.
First, we look at what happens if Professor Snape decides to increase the price of
ice cream, while leaving the price of butter (and all the other problem data) the same. We
ask the question, How much can we increase the price of ice cream without changing the
solution (4, 5)? It is intuitively clear that if the profit on ice cream is much greater than on
butter, it would make sense to make as much ice cream as possible subject to meeting the
constraints, that is, 6 gallons. Hence, if the price of ice cream increases by more than a
certain amount, the solution will move away from the point (4, 5).
Suppose for instance that we increase the profit on ice cream to $5.50, so that the
objective function becomes z = 5.5x + 4y. If we plot the contours of this new objective
(see Figure 1.2), we find that they are rotated slightly in the clockwise direction from the
contours in Figure 1.1. It is clear that for a $42 profit, (4, 5) is still the optimum. However,
if the profit on ice cream is increased further, the contours will eventually have exactly the
same slope as the “milk” constraint, at which point every point on the line joining (4, 5) to
(6, 2) will be a solution. What ice cream profit p will make the contours of the objective
z = px + 4y parallel to the line 3x + 2y = 22? By matching the slopes of these two lines,
we find that the operative value is p = 6. If the price of ice cream is slightly higher than 6,
the point (6, 2) will be the unique optimum.
Exercise 1-1-1. Plot a figure like Figures 1.1 and 1.2 for the case in which the objective is
z = 8x + 4y, while the constraints remain the same. Verify from your figure that (6, 2) is
the optimum.
Returning to the original problem, we could ask a slightly different question. Suppose
that Professor Snape’s neighbor, Professor Crouch, has some excess milk and is offering to
sell it to Snape for $1 per gallon. Given that Snape still wants to maximize his profits, and
given that his other constraints are still in place (labor and freezer capacity), should he buy
any milk from Crouch and, if so, how much?
To answer this question, we note first that if Snape purchases c gallons, the milk
constraint becomes 3x + 2y ≤ 22 + c. Graphically, the boundary of this constraint shifts
upward and to the right, as we see in Figure 1.3. Provided c is not too large, the contours of
the objective will not be greatly affected by this change to the problem, and so the solution
1.1. An Example: The Professor’s Dairy 5

labor constraint (.25 x + y <= 6) freezer constraint (x <= 6)

y (butter) solution (4,5)

milk constraint (3x+2y <= 22)

z=
40
x (ice cream)
z=

z=
20

30
Figure 1.2. The Professor’s dairy: After increasing the profit on ice cream to
$5.50, the objective contours rotate slightly clockwise, but the optimum is still (4, 5).

labor constraint (.25 x + y <= 6) freezer constraint (x <= 6)

solution (4+.4c,5−.1c)
y (butter)

milk constraint (3x+2y <= 22+c)

x (ice cream)

Figure 1.3. The Professor’s dairy: If the professor purchases c gallons from his
neighbor, the milk constraint shifts upward and to the right.
6 Chapter 1. Introduction

will still occur at the intersection of the labor constraint with the milk constraint, that is, at
the point (x, y) that satisfies the following two equalities:

.25x + y = 6,
3x + 2y = 22 + c.

The solution is
(x, y) = (4 + .4c, 5 − .1c),
and the objective function value at this point (allowing for the $1 per gallon purchase price
of milk from Crouch) is

z = 5x + 4y − c = 5(4 + .4c) + 4(5 − .1c) − c = 40 + .6c.

It follows that it is definitely to Snape’s advantage to buy some milk from Crouch, as he
earns an extra 60 cents in profit for each gallon purchased.
However, if c is too large, the solution will no longer be at the intersection of the
labor and milk constraints, and there is no further advantage to be gained. This happens
when the milk constraint is shifted so far that it intersects with both the labor limit and the
freezer limit at the point (6, 4.5), which is true when c = 5. As c increases above this value,
the solution stays at (6, 4.5) while the profit actually starts to decline, as Snape is buying
surplus milk unnecessarily without producing any more of either butter or ice cream.
Analysis of this type will be discussed further when we cover the subject of duality
in Chapter 4.
The graphical analysis used in this section is sufficient for understanding problems
with two variables. However, when extra variables are added (for example, if the professor
decides to make cottage cheese and gourmet yogurt as well), it is hard to solve or analyze the
problem using graphical techniques alone. This book describes computational techniques,
motivated by the graphical analysis above, that can be used to solve problems with many
variables and constraints. Solution of this problem using an algebraic approach, namely the
simplex method, is given in Section 3.

1.1.4 Discussion
The example of this section has three important properties.

• Its variables (the amounts of ice cream and butter to produce) are continuous variables.
They can take on any real value, subject to satisfying the bounds and constraints.

• All constraints and bounds involve linear functions of the variables. That is, each
term of the sum is either a constant or else a constant multiple of one of the variables.

• The objective function—profit, in this case—is also a linear function of the variables.

Problems with these three essential properties are known as linear programming prob-
lems or linear programs. Most of our book is devoted to algorithms for solving this class
of problems. Linear programming can be extended in various ways to give broader classes
of optimization problems. For instance, if we allow the objective function to be a quadratic
1.2. Formulations 7

function of the variables (but still require the constraint to be linear and the variables to
be continuous), we obtain quadratic programming problems, which we study in Chapter 7.
If we allow both constraints and objective to be nonlinear functions (but still require con-
tinuous variables), the problem becomes a nonlinear program. If we restrict some of the
variables to take on integer values, the problem becomes an integer program. We give
several references for nonlinear and integer programming in the Notes and References at
the end of this chapter.
Since 1947, when George B. Dantzig proposed his now classic simplex method for
solving linear programs, the utilization of linear programming as a tool for modeling and
computation has grown tremendously. Besides becoming a powerful tool in the area for
which it was originally designed (economic planning), it has found a myriad of applications
in such diverse areas as numerical analysis, approximation theory, pattern recognition, and
machine learning. It has become a key tool in the important disciplines of operations
research and management science.

1.2 Formulations
Throughout this book, we will refer to the following form of the linear program as the
standard form:
min z = p1 x1 + · · · + pn xn
x1 ,x2 ,...,xn
subject to A11 x1 + ··· + A1n xn ≥ b1 ,
.. .. .. .. (1.2)
. . . .
Am1 x1 + ··· + Amn xn ≥ bm ,
x1 , x2 , . . . , xn ≥ 0.
By grouping the variables x1 , x2 , . . . , xn into a vector x and constructing the following
matrix and vectors from the problem data,
     
A11 · · · A1n b1 p1
 .. .   .   
A= . . .. ..  , b =  ..  , p =  ...  ,
Am1 · · · Amn bm pn
we can restate the standard form compactly as follows:
min z = p x
x
subject to Ax ≥ b, x ≥ 0,
where p denotes the transpose of the column vector p, which is known as the cost vector.
Every linear program can be put into this standard form. We show in Chapter 3
how problems with equality constraints, free variables, and so on can be reformulated as
standard-form problems. Problem (1.1) of the previous section can be expressed in standard
form by setting x to be the vector made up of the two scalars x and y, while
   
1 0 6  
    5
A = − .25 1 , b=− 6 , p=− .
4
3 2 22
8 Chapter 1. Introduction

To perform this conversion, we changed “≤” inequality constraints into “≥” inequalities by
simply multiplying both sides by −1. We also noted that maximization of a function (which
we do in (1.1)) is equivalent to minimization of the negation of this function, which is why
we have negative entries in p above.
In Chapter 5 we introduce another formulation in which all the general constraints are
assumed to be equality constraints. This is known as the canonical form and is written as
follows:
min z = p x
subject to Ax = b, x ≥ 0.
As with the standard form, any linear program can be put into this form by appropriate
transformations of the constraints and variables. We could express our example (1.1) in
canonical form by first replacing (x, y) by (x1 , x2 ) in (1.1) and then introducing three slack
variables x3 , x4 , and x5 to represent the amount by which the right-hand sides exceed the
left-hand sides of the three constraints. We then obtain the following formulation:

min z = −5x1 − 4x2

x
subject to x1 + x3 = 6,
.25x1 + x2 + x4 = 6,
3x1 + 2x2 + x5 = 22,
x1 , x2 , x3 , x4 , x5 ≥ 0.

We can verify that the problem is in canonical form by setting

   
    5 x1
1 0 1 0 0 6 4  x2 
   
A = .25 1 0 1 0 , b =  6 , p = − 
0 , x= 
 x3  .
3 2 0 0 1 22 0  x4 
0 x5

1.3 Applications
In this section, we discuss several other practical problems that can be formulated as linear
programs.

1.3.1 The Diet Problem

In an early application, linear programming was used to determine the daily diet for a person.
From among a large number of possible foods, a diet was determined that achieved all the
nutritional requirements of the individual while minimizing total cost.
To formulate as a linear program, we suppose that the n possible foods are indexed by
j = 1, 2, . . . , n and that the m nutritional categories are indexed by i = 1, 2, . . . , m. We
let xj be the amount of food j to be included in the diet (measured in number of servings),
and denote by pj the cost of one serving of food j . We let bi denote the minimum daily
requirement of nutrient i and Aij be the amount of nutrient i contained in one serving of
food j . By assembling this data into matrices and vectors in the usual way, we find that the
1.3. Applications 9

linear program to determine the optimal diet can be formulated as follows:

min z = p x
x
subject to Ax ≥ b, x ≥ 0.

The bounds x ≥ 0 indicate that only nonnegative amounts of each food will be considered,
while the “≥” inequality constraints require the diet to meet or exceed the nutritional re-
quirements in each category i = 1, 2, . . . , m. If we wish to place an upper limit of dj on
the number of servings of food j to be included in the diet (to ensure that the diet does
not become too heavy on any one particular food), we could add the constraints xj ≤ dj ,
j = 1, 2, . . . , n, to the model.

1.3.2 Linear Surface Fitting

Suppose that we have a set of observations (Ai· , bi ), i = 1, 2, . . . , m, where each Ai· is a
(row) vector with n real elements, and each bi is a single real number. We would like to find
a vector x ∈ Rn and a constant γ such that

Ai· x + γ ≈ bi for each i = 1, 2, . . . , m.

The elements of the vector x can be thought of as “weights” that are applied to the components
of Ai· to yield a prediction of each scalar bi . For example, m could be the number of people
in a population under study, and the components of each Ai· could represent the income
of person i, the number of years they completed in school, the value of their house, their
number of dependent children, and so on. Each bi could represent the amount of federal
income tax they pay.
To find the “best” pair (x, γ ), we need to measure the misfit between Ai· x + γ and bi
over all the i. One possible technique is to sum the absolute values of all the mismatches,
that is,
m
|Ai· x + γ − bi |.
i=1

We can formulate a linear program to find the (x, γ ) that minimizes this measure. First,
define the matrix A and the vector b by
   
A1· b1
 A2·   b2 
   
A =  . , b =  . .
 ..   .. 
Am· bm

Next, write the linear program as follows:

min z = e y
x,γ ,y
subject to −y ≤ Ax + γ e − b ≤ y.

In this formulation, e = (1, 1, . . . , 1) ∈ Rm , so that the objective is the sum of the elements
of y. The constraints ensure that each yi is no smaller than the absolute value |Ai· x +γ −bi |,
10 Chapter 1. Introduction

while the fact that we are minimizing the sum of yi ’s ensures that each yi is chosen no larger
than it really needs to be. Hence, the minimization process chooses each yi to be equal to
|Ai· x + γ − bi |.
When n = 1 (that is, each Ai· has just a single element), this problem has a simple
geometric interpretation. Plotting Ai· on the horizontal axis and bi on the vertical axis,
this formulation finds the line in the two-dimensional (Ai· , bi ) space such that the sum of
vertical distances from the line to the data points bi is minimized.

1.3.3 Load Balancing Problem

Consider the task of balancing computational work among n processors, some of which
may already be loaded with other work. We wish to distribute the new work in such a way
that the lightest-loaded processor has as heavy a load as possible. We define the data for the
problem as follows:

pi = current load of processor i = 1, 2, . . . , n (nonnegative),

L= additional total load to be distributed,
xi = fraction of additional load L distributed to processor i, with xi ≥ 0 and
n
i=1 xi = 1,
γ = minimum of final loads after distribution of workload L.

Assuming that the new work can be distributed among multiple processors without incurring
any overhead, we can formulate the problem as follows:

max γ
x,γ
subject to γ e ≤ p + xL, e x = 1, x ≥ 0,

where e = (1, 1, . . . , 1) is the vector of 1’s with n components.

Interestingly, this is one of the few linear programs that can be solved in closed form.
When pi ≤ L/n for all i = 1, 2, . . . , n, the optimal γ is (e p + L)/n, and all processors
have the same workload γ . Otherwise, the processors that had the heaviest loads to begin
with do not receive any new work; the solution is slightly more complicated in this case
but can be determined by sorting the pi ’s. Similar solutions are obtained for the continuous
knapsack problem that we mention later.

1.3.4 Resource Allocation

Consider a company that needs to decide how to allocate its resources (for example, raw
materials, labor, or time on rented equipment) in a certain period to produce a variety of
finished products. Suppose the company is able to to produce m types of finished products
(indexed i = 1, 2, . . . , m) and that it uses n resources (indexed by j = 1, 2, . . . , n). Each
unit of finished product i yields ci dollars in revenue, whereas each unit of resource j
costs dj dollars. Suppose too that one unit of product i requires Aij units of resource j
to manufacture and that a maximum of bj units of resource j are available in this period.
The manufacturer aims to maximize their profit (defined as total revenue minus total cost)
subject to using no more resources than are available.
1.3. Applications 11

The variables in this problem are yi , i = 1, 2, . . . , m, which is the number of units of

product i, and xj , j = 1, 2, . . . , n, the number of units of resource j consumed. The linear
programming formulation is as follows:

max z = c y − d  x
x,y
subject to x = A y, x ≤ b, x, y ≥ 0.

To further explain the constraint x = A y better, we consider the j th equation of this system,
which is

xj = A1j y1 + A2j y2 + · · · + Amj ym .

Each term Aij yi indicates the amount of resource j used to manufacture the desired amount
of product i, and so the summation represents the total amount of resource j required to
make the specified amounts of the products. The bound x ≤ b ensures that we do not exceed
the available resources, and the nonnegativity constraint y ≥ 0 constrains us to produce a
nonnegative amount of each product. (The constraint x ≥ 0 is actually redundant and can
be omitted from the formulation; since all the elements of y and A are nonnegative, all
elements of x = A y must also be nonnegative.)

1.3.5 Classification
In classification problems, we are given two sets of points in the space of n dimensions Rn .
Our aim is to find a hyperplane in the space Rn that separates these two sets as accurately
as possible. We use this hyperplane to classify any new points that arise; if the new point
lies on one side of the hyperplane, we classify it as an element of the first set, while if it lies
on the other side, we place it in the second set.
Linear programming can be used to find the separating hyperplane, which is defined
by a vector w ∈ Rn and a scalar γ . Ideally, we would like each point t in the first set to
satisfy w t ≥ γ , while each point t in the second set satisfies w  t ≤ γ . To guard against
a trivial answer (note that the conditions just specified are trivially satisfied by w = 0 and
γ = 0!), we seek to enforce the stronger conditions w  t ≥ γ + 1 for points in the first
set and w t ≤ γ − 1 for points in the second set. Moreover, because the two sets may
be intermingled, it may not be able to enforce a clean separation. We define the objective
function in the linear program to be the sum of the average violations of the classification
conditions over each set.
We set up the linear program by constructing an m × n matrix M whose ith row
contains the n components of the ith points in the first set. Similarly, we construct a k × n
matrix B from the points in the second set. The violations of the condition w  t ≥ γ + 1
for points in the first set are measured by a vector y, which is defined by the inequalities
y ≥ −(Mw − γ e) + e, y ≥ 0, where e = (1, 1, . . . , 1) ∈ Rm . Similarly, violations of the
condition w t ≤ γ − 1 for points in the second set are measured by the vector z defined
by z ≥ (Bw − γ e) + e, z ≥ 0, where e ∈ Rk . In general, e will be a vector of ones of
appropriate dimension. The average violation on the first set is e y/m and on the second
12 Chapter 1. Introduction

O
O O O O
O O
x O O O O O
O O
x x x
x x O O
O
O
O x O
O
O
O M
x x O O O
x x x O x O
x x x O O
x O x x O O O
x x x
x O O O
x x x x O
x x O O O
x x x x
x x O O O
x
x x x
x O
x x x x w’x=γ+1
x x
x x
x x
B x
x
x x
x

x x
x
x
w’x=γ−1
w’x=γ

Figure 1.4. Classification using the plane w  x = γ .

set is e z/k, and so we can write the linear program as follows:
1 
min m
ey + k1 e z
w,γ ,y,z
subject to y ≥ −(Mw − γ e) + e,
z ≥ (Bw − γ e) + e,
(y, z) ≥ 0.
Figure 1.4 shows the separation in a particular example arising in breast cancer diag-
nosis (Mangasarian, Street & Wolberg (1995)). The first set M (indicated by circles in the
diagram) consists of fine needle aspirates (samples) taken from malignant tumors. Their
location in the two-dimensional space is defined by the measures of two properties of each
tumor, for example, the average cell size and the average deviation from “roundness” of the
cells in the sample. The second set B (indicated by crosses) consists of fine needle aspirates
taken from benign tumors. Note that the hyperplane w  x = γ (which in two dimensions is
simply a line) separates most of the benign points from most of the malignant points.
Another interesting application of the linear programming approach to classification
is described by Bosch & Smith (1998), who use a separating plane in three dimensions that
count the frequencies of certain words to determine that 12 disputed Federalist Papers were
probably authored by James Madison rather than Alexander Hamilton.

1.3.6 Minimum-Cost Network Flow

Network problems, which involve the optimization of a flow pattern in a network of nodes
and arcs, are important because of their applicability to many diverse practical problems.
1.3. Applications 13

4 7
1 9

10

Figure 1.5. Nodes and arcs in a network.

We consider here a particular kind of network problem known as minimum-cost network

flow, where the “flow” consists of the movement of a certain commodity along the arcs of
a network, from the nodes at which the commodity is produced to the nodes where it is
consumed. If the cost of transporting the commodity along an arc is a fixed multiple of the
amount of commodity, then the problem of minimizing the total cost can be formulated as
a linear program.
Networks, such as that depicted in Figure 1.5, consist of nodes N and arcs A, where
the arc (i, j ) connects an origin node i to a destination node j . Associated with each node i
is a divergence bi , which represents the amount of product produced or consumed at node i.
When bi > 0, node i is a supply node, while if bi < 0, it is a demand node. Associated with
each arc (i, j ) are a lower bound lij and an upper bound uij of the amount of the commodity
that can be moved along that arc. Each variable xij in the problem represents the amount of
commodity moved along the arc (i, j ). The cost of moving one unit of flow along arc (i, j )
is cij . We aim to minimize the total cost of moving the commodity from the supply nodes
to the demand nodes.
Using this notation, we can formulate the minimum-cost network flow problem as
follows:
min z= cij xij
x
(i,j )∈A
subject to xij − xj i = bi for all nodes i ∈ N ,
j :(i,j )∈A j :(j,i)∈A
lij ≤ xij ≤ uij for all arcs (i, j ) ∈ A.
The first constraint states that the net flow through each node should match its divergence.
The first summation represents the total flow out of node i, summed over all the arcs that have
node i as their origin. The second summation represents total flow into node i, summed over
14 Chapter 1. Introduction

all the arcs having node i as their destination. The difference between inflow and outflow
is constrained to be the divergence bi .
By relabeling the flow variables as x1 , x2 , . . . , xn , where n is the total number of
arcs, we can put the problem into a more general programming form. However, the special
notation used above reveals the structure of this application, which can be used in designing
especially efficient versions of the simplex method. Note, in particular, that the coefficient
matrix arising from the flow constraints contains only the numbers 0, 1, and −1. If all
the problem data is integral, it can be shown that the solution x also contains only integer
components.

1.4 Algorithms and Complexity

Though easy to state, linear programs can be quite challenging to solve computationally.
The essential difficulty lies in determining which of the inequality constraints and bounds are
active (that is, satisfied as equalities) at the solution and which are satisfied but inactive. (For
example, the constraint 2x1 + x2 ≤ 8 is active at the point (x1 , x2 ) = (1, 6); it is satisfied but
inactive at the point (2, 2); it is violated at the point (4, 1).) To determine which constraints
are active at the solution would seem to be a combinatorial problem: If there are l inequality
constraints and bounds, and each of them can be either active or inactive, we may have a
total of 2l active/inactive combinations. The situation hardly improves if we make use of
the fact that a solution occurs at one of the vertices of the feasible region, defined as a point
at which at least n of the constraints are active. A problem in Rn with a total of l inequality
constraints and bounds (and no equality constraints) may have as many as
l l!
=
n (l − n)!n!
vertices. Even for a small problem with n = 10 variables and l = 20 inequality constraints
and bounds, there may be 184, 756 vertices, and possibly 1, 048, 576 active/inactive com-
binations. A “brute force” algorithm that examines all these possibilities will be much too
slow for practical purposes.

1.4.1 The Simplex Method

From a geometrical point of view, the simplex method is easy to understand. It starts
by determining whether the feasible region is empty. If so, it declares the problem to be
infeasible and terminates. Otherwise, it finds a vertex of the feasible region to use as a
starting point. It then moves from this vertex to an adjacent vertex for which the value of
the objective z is lower—in effect, sliding along an edge of the feasible region until it can
proceed no further without violating one of the constraints. This process is repeated; the
algorithm moves from vertex to (adjacent) vertex, decreasing z each time. The algorithm
can terminate in one of two ways. First, it may encounter a vertex whose value of z is less
than or equal to all adjacent vertices. In this case, it declares this vertex to be a solution of
the linear program. Second, it may detect that the problem is unbounded. That is, it may
find a direction leading away from the current vertex that remains feasible (no matter how
long a step is taken along it) such that the objective z decreases to −∞ along this direction.
In this case, it declares the problem to be unbounded.
1.4. Algorithms and Complexity 15

Suppose in our two-variable example of Figure 1.1 that the simplex algorithm starts at
the origin (0, 0). It could find the optimum (4, 5) by moving along one of two paths where,
due to conversion to a minimization problem, z is the negative of that depicted in Figure 1.1.

Path 1 Path 2
(0,0) z=0
(0,0) z=0
(6,0) z = −30
(0,6) z = −24
(6,2) z = −38
(4,5) z = −40
(4,5) z = −40

Note that both adjacent vertices of the initial point (0, 0) have lower objective values,
and hence each one is a valid choice for the next iterate. The simplex method uses a pivot
selection rule to select from among these possibilities; different variants of the simplex
method use different pivot rules, as we see in Chapters 3 and 5.

1.4.2 Interior-Point Methods

Although the simplex method performs well on most practical problems, there are patho-
logical examples (Klee & Minty (1972)) in which the number of iterations required is
exponential in the number of variables. On such examples, linear programming seems to
reveal a combinatorial nature. A surprising development occurred in 1979, when a (theoret-
ically) more efficient method was discovered by Khachiyan (1979). For problems in which
the data A, b, c were integer or rational numbers, Khachiyan’s ellipsoid method can solve the
problem in a time that is bounded by a polynomial function of the number of bits L needed
to store the data and the number of variables n. However, the ellipsoid method proved to
be difficult to implement and disappointingly slow in practice. Karmarkar (1984) proposed
a new algorithm with a similar polynomial bound. He made the additional claim that a
computational implementation of his algorithm solved large problems faster than existing
simplex codes. Though this claim was never fully borne out, Karmarkar’s announcement
started a surge of new research into interior-point methods, so named because their iterates
move through the interior of the feasible region toward a solution, rather than traveling
from vertex to vertex around the boundary. Software based on interior-point methods is
often significantly faster than simplex codes on large practical problems. We discuss these
methods further in Chapter 8.

Notes and References

The use of the word “programming” in connection with linear programming is some-
what anachronistic. It refers to the step-by-step mathematical procedure used to solve this
optimization problem, not specifically to its implementation in a computer program. The
term “linear programming” was coined in the 1940s, well before the word “programming”
became strongly associated with computers.
The definition of the term standard form is itself not “standard”; other authors use
a definition different from the one we provide in (1.2). The term canonical form is not
widely used and is also not standard terminology, but we use it here as a convenient way to
distinguish between the two formulations, both of which appear throughout the book.
16 Chapter 1. Introduction

The classic text on the simplex method is by the inventor of this method, George
B. Dantzig (1963). In 1939, the Russian Nobel Laureate Leonid V. Kantorovich had also
proposed a method for solving linear programs; see Kantorovich (1960).
More advanced treatments of linear programming than ours include the books of
Chvátal (1983) and Vanderbei (1997). Wright (1997) focuses on interior-point methods.
Several advanced chapters on linear programming (both simplex and interior-point) also
appear in the text of Nocedal & Wright (2006). The latter text also contains material on
more general optimization problems, especially nonlinear optimization problems with and
without constraints. The text of Wolsey (1998) provides an excellent introduction to integer
programming.
Chapter 2

Linear Algebra:
A Constructive Approach

In Section 1.4 we sketched a geometric interpretation of the simplex method. In this chapter,
we describe the basis of an algebraic interpretation that allows it to be implemented on a
computer. The fundamental building block for the simplex method from linear algebra is the
Jordan exchange. In this chapter, we describe the Jordan exchange and its implementation
in MATLAB. We use it in a constructive derivation of several key results in linear algebra
concerning linear independence and the solution of linear systems of equations. In the
latter part of the chapter, we discuss the LU factorization, another linear algebra tool that
is important in implementations of the simplex method.
In this chapter and the rest of the book, we assume basic familiarity with MATLAB.
There are many books and web sites that will get you started in MATLAB; we recommend
the MATLAB Primer by Sigmon & Davis (2004).
We first describe the Jordan exchange, a fundamental building block of linear algebra
and the simplex algorithm for linear programming.

2.1 Jordan Exchange

Consider the following simple linear equation in the one-dimensional variables x and y:

y = ax.

The form of the equation indicates that x is the independent variable and y is the dependent
variable: Given a value of x, the equation tells us how to determine the corresponding value
of y. Thus we can think of the dependent variable as a function of the independent variable;
that is, y(x) := ax. If we assume that a  = 0, we can reverse the roles of y and x as follows:

1
x = ãy, where ã = .
a
Note that now we have a function x(y) in which x is determined as a function of y. This
exchange in roles between dependent and independent variables gives a very simple proce-
dure for solving either the equation ax − b = 0 or the inequality ax − b ≥ 0, using the

17
18 Chapter 2. Linear Algebra: A Constructive Approach

following simple equivalences:

ax − b = 0 ⇐⇒ ax = y, y=b ⇐⇒ x = ãy, y = b,
(2.1)
ax − b ≥ 0 ⇐⇒ ax = y, y≥b ⇐⇒ x = ãy, y ≥ b.

In particular, the second formula gives an explicit characterization of the values of x that
satisfy the inequality ax − b ≥ 0, in terms of an independent variable y for which y ≥ b.
The Jordan exchange is a generalization of the process above. It deals with the
case in which x ∈ Rn is a vector of independent variables and y ∈ Rm is a vector of
dependent variables, and we wish to exchange one of the independent variables with one
of the dependent variables. The Jordan exchange plays a crucial role in solving systems
of equations, linear inequalities, and linear programs. In addition, it can be used to derive
fundamental results of linear algebra and linear programming.
We now demonstrate a Jordan exchange on the system y = Ax by exchanging the roles
of a component yr of y and a component xs of x. First, we write this system equation-wise as

yi = Ai1 x1 + Ai2 x2 + · · · + Ain xn , i = 1, 2, . . . , m, (2.2)

where the independent variables are x1 , x2 , . . . , xn and the dependent variables are
y1 , y2 , . . . , ym , and the Aij ’s are the coefficients. We can think of the yi ’s as (linear)
functions of xj ’s, that is,

yi (x) := Ai1 x1 + Ai2 x2 + · · · + Ain xn , i = 1, 2, . . . , m, (2.3)

or, more succinctly, y(x) := Ax. This system can also be represented in the following
tableau form:
x1 · · · xs · · · xn
y1 = A11 · · · A1s · · · A1n
.. .. .. ..
. . . .
yr = Ar1 · · · Ars · · · Arn
.. .. .. ..
. . . .
ym = Am1 ··· Ams ··· Amn
Note that the tableau is nothing more than a compact representation of the system of equa-
tions (2.2) or the functions determining the dependent variables from the independent vari-
ables (2.3). All the operations that we perform on the tableau are just simple algebraic
operations on the system of equations, rewritten to conform with the tableau representation.
We now describe the Jordan exchange or pivot operation with regard to the tableau
representation. The dependent variable yr will become independent, while xs changes from
being independent to being dependent. The process is carried out by the following three
steps.
(a) Solve the rth equation

yr = Ar1 x1 + · · · + Ars xs + · · · + Arn xn

for xs in terms of x1 , x2 , . . . , xs−1 , yr , xs+1 , . . . , xn . Note that this is possible if and

only if Ars  = 0. (Ars is known as the pivot element, or simply the pivot.)
2.1. Jordan Exchange 19

(b) Substitute for xs in all the remaining equations.

(c) Write the new system in a new tableau form as follows:

x1 ··· xs−1 yr xs+1 ··· xn

y1 = B11 ··· B1s ··· B1n
.. .. .. ..
. . . .
yr−1 =
xs = Br1 ··· Brs ··· Brn
yr+1 =
.. .. .. ..
. . . .
ym = Bm1 ··· Bms ··· Bmn

To determine the elements Bij in terms of the elements Aij , let us carry out the algebra
specified by the Jordan exchange. As will be our custom in this book, we will describe
and produce corresponding MATLAB m-files for the important algebraic operations that
we perform. Solution of the rth equation

n
yr = Arj xj + Ars xs
j =1
j =s

for xs gives
n n
1 −Arj
xs = yr + xj = Brs yr + Brj xj , (2.4)
Ars j =1
Ars j =1
j =s j =s

where
1 −Arj
Brs = , Brj = ∀j  = s. (2.5)
Ars Ars

These formulae define the rth row Br· of the transformed tableau. We can express them
in MATLAB by first defining J to represent the columns of the tableau excluding the sth
column, that is,

J = [1:s-1,s+1:n];

and then writing

B(r,s) = 1.0/A(r,s); B(r,J) = -A(r,J)/A(r,s);

This “vector index” facility is an important feature of MATLAB which enables terse coding
of expressions such as those given in (2.5).
20 Chapter 2. Linear Algebra: A Constructive Approach

We can now proceed with part (b) of the Jordan exchange. Substituting of the expres-
sion for xs from (2.4) in the ith equation of the tableau (i  = r), we have
 
n
 1 n
−Arj 
yi = Aij xj + Ais 
 Ars y r + xj 

j =1 j =1
A rs
j  =s j =s
n
= Bij xj + Bis yr ,
j =1
j  =s

where
Ais Ais  
Bis = , Bij = Aij − Arj = Aij − Bis Arj ∀i  = r, j  = s. (2.6)
Ars Ars

These formulae define rows Bi· , i = 1, 2, . . . , m, i  = r, of the transformed tableau. We

can also write these equations succinctly in MATLAB notation by defining J as above and
defining I to represent all the rows of the tableau except the rth row, that is,

I = [1:r-1,r+1:m];

and writing

B(I,s) = A(I,s)/A(r,s);
B(I,J) = A(I,J) - B(I,s)*A(r,J);

The complete description of one step of the Jordan exchange with pivot Ars is coded
in jx.m—the function jx in MATLAB.
Note that we have introduced the “function” facility of MATLAB. Any function can be
defined in a file of the same name as the function, but with suffix .m, just as the function jx
is stored in jx.m. It can then be invoked from within MATLAB—either from the command
window or from within other functions—by simply typing the function name together with
its arguments. The following example shows a call to the function jx.

Example 2-1-1. Solve the following system of equations for x1 , x2 in terms of y1 , y2 :

y1 = 2x1 + x2 ,
y2 = 3x1 + x2 .

Working from the MATLAB command window, one first loads the data file containing
the matrix A of Example 2-1-1 and then invokes jx twice to perform the two required
Jordan exchanges:

load ex2-1-1
B = jx(A,1,1)
B = jx(B,2,2)
2.1. Jordan Exchange 21

MATLAB file jx.m: Jordan exchange

function B = jx(A,r,s)
% syntax: B = jx(A,r,s)
% input: matrix A, integers r,s
% perform a Jordan exchange with pivot A(r,s)

[m,n] = size(A); B = zeros(m,n);

I = [1:r-1,r+1:m]; J = [1:s-1,s+1:n];

% update pivot row

B(r,s) = 1.0/A(r,s); B(r,J) = -A(r,J)/A(r,s);

% update pivot column

B(I,s) = A(I,s)/A(r,s);

% update remainder of tableau

B(I,J) = A(I,J)-B(I,s)*A(r,J);
return;

Note that we overwrite B at each step to hold the cumulative effects of the sequence of
exchanges.
We now introduce a MATLAB structure that stores a complete tableau—the row and
column labels corresponding to the dependent and independent variables, along with the
matrix of coefficients that defines the relationship between these quantities. The totbl
command can be used to construct a tableau as follows:

load ex2-1-1 x1 x2
T = totbl(A); y1 = 2 1
y2 = 3 1

The row labels (dependent variables) are assigned the default values y1 and y2 and the
column labels (independent variables) the default values x1 and x2 ; other forms of the
totbl command, discussed later, will allow the user to define their own labels. The
command tbl can be used to print out the tableau along with its associated labels.
To perform Jordan exchanges on the tableau representation (rather than on just the
matrix), we use the labeled Jordan exchange function ljx in place of jx as follows:

T = ljx(T,1,1); y1 x2
x1 = 0.5 −0.5
y2 = 1.5 −0.5
22 Chapter 2. Linear Algebra: A Constructive Approach

T = ljx(T,2,2); y 1 y2
x1 = −1 1
x2 = 3 −2

In addition to making the algebraic changes to the matrix, ljx swaps the row and column
labels as required by the exchange and prints the modified tableau using tbl. The trailing
semicolon should not be omitted after the call to ljx since it results in the printing of
additional unnecessary information about the structure T.

The following simple theorem provides a formal justification of the Jordan exchange
formulae (2.5) and (2.6) as well as their extension to multiple pivots in succession. The
result will enable us to use the Jordan exchange to give some constructive derivations of key
results in linear algebra and linear programming.

Theorem 2.1.1. Consider the linear function y defined by y(x) := Ax, where A ∈ Rm×n .
After k pivots (with appropriate reordering of rows and columns) denote the initial and kth
tableaus as follows:
xJ1 xJ2 yI1 xJ2
yI1 = AI1 J1 AI1 J2 xJ1 = BI1 J1 BI1 J2
yI2 = AI2 J1 AI2 J2 yI2 = BI2 J1 BI2 J2

Here I1 , I2 is a partition of {1, 2, . . . , m} and J1 , J2 is a partition of {1, 2, . . . , n}, with I1

and J1 containing the same number of elements. Then for all values of x ∈ Rn

xJ1 = BI1 J1 yI1 (x) + BI1 J2 xJ2 ,

yI2 (x) = BI2 J1 yI1 (x) + BI2 J2 xJ2 .

That is, the original linear functions y satisfy the new linear relationships given by the kth
tableau.

Proof. We show the result for one pivot. The result for k pivots follows by induction.
For a pivot on the (r, s) element, we have I1 = {r}, I2 = {1, . . . , r − 1, r + 1, . . . , m},
J1 = {s}, and J2 = {1, . . . , s − 1, s + 1, . . . , n}. Then for all x, we have

n
xJ1 − BI1 J1 yI1 (x) − BI1 J2 xJ2 = xs − Brs yr (x) − Brj xj
j =1
j =s
 
n n
1  −Arj
= xs − Arj xj  − xj
Ars j =1 j =1
Ars
j =s
=0
2.2. Linear Independence 23

and

yI2 (x) − BI2 J1 yI1 (x) − BI2 J2 xJ2

 
 n
 i = 1, 2, . . . , m,
=
yi (x) − Bis yr (x) − Bij xj 
, i = r
j =1
j  =s

 
 n
Ais
n n
Ais Arj  i = 1, 2, . . . , m,
=
 Aij xj − Arj xj − Aij − xj 
,
Ars Ars i = r
j =1 j =1 j =1
j =s
= 0,

verifying the claims.

The following result shows that if two tableaus have identical dependent variables for
all possible values of the independent variables, then the tableau entries are also identical.

Proposition 2.1.2. If the linear function y is defined by y(x) = Ax and also by y(x) = Bx,
then A = B.

Proof. Since (A − B)x = 0 for all x ∈ Rn , we can choose x = I·i , where I·i is the ith
column of the identity matrix. We deduce that the ith columns of A and B are identical.
Since this fact is true for all i = 1, 2, . . . , n, we conclude that A = B.

2.2 Linear Independence

A simple geometric way to solve a system of two equations in two unknowns is to plot the
corresponding lines and determine the point where they intersect. Of course, this technique
fails when the lines are parallel to one another. A key idea in linear algebra is that of
linear dependence, which is a generalization of the idea of parallel lines. Given a matrix
A ∈ Rm×n , we may ask if any of its rows are redundant. In other words, is there a row Ak·
that can be expressed as a linear combination of the other rows? That is,
m
Ak· = λi Ai· . (2.7)
i=1
i =k

If so, then the rows of A are said to be linearly dependent.

As a concrete illustration, consider the matrix
 
1 2 4
A = 3 4 8  .
5 6 12
24 Chapter 2. Linear Algebra: A Constructive Approach

The third row of this matrix is redundant, because it can be expressed as a linear combination
of the first two rows as follows:

A3· = 2A2· − A1· .

If we rearrange this equation, we see that

 
−1 2 −1 A = 0,
 
that is, for z = −1 2 −1 , z A = 0 with z  = 0.
We define linear dependence of the rows of a matrix A formally as follows:

z A = 0 for some nonzero z ∈ Rm .

(We see that (2.7) can be expressed in this form by taking zi = λi , i  = k, zk = −1.) The
negation of linear dependence is linear independence of the rows of A, which is defined by
the implication
z A = 0 ⇒ z = 0.
The idea of linear independence extends also to functions, including the linear func-
tions y defined by y(x) := Ax that we have been considering above. The functions yi (x),
i = 1, 2, . . . , m, defined by y(x) := Ax are said to be linearly dependent if

z y(x) = 0 ∀x ∈ Rn for some nonzero z ∈ Rm

and linearly independent if

z y(x) = 0 ∀x ∈ Rn ⇒ z = 0. (2.8)

The equivalence of the linear independence definitions for matrices and functions is clear
when we note that

z Ax = 0 ∀x ∈ Rn for some nonzero z ∈ Rm

⇐⇒ z A = 0 for some nonzero z ∈ Rm .

Thus the functions y(x) are linearly independent if and only if the rows of the matrix A are
linearly independent.
Proposition 2.2.1. If the m linear functions yi are linearly independent, then any p of them
are also linearly independent, where p ≤ m.

Proof. The proof is obvious from contrapositive statement: yi , i = 1, 2, . . . , p, are linearly

dependent implies yi , i = 1, 2, . . . , m, are linearly dependent.

Proposition 2.2.2. If the linear functions y defined by y(x) = Ax, A ∈ Rm×n , are linearly
independent, then m ≤ n. Furthermore, in the tableau representation, all m dependent yi ’s
can be made independent; that is, they can be exchanged with m independent xj ’s.
2.2. Linear Independence 25

Proof. Suppose that the linear functions y(x) = Ax are linearly independent. Exchange
y’s and x’s in the tableau until no further pivots are possible, at which point we are blocked
by a tableau of the following form (after a possible rearrangement of rows and columns):

yI1 xJ2
xJ1 = BI1 J1 BI1 J2
yI2 = BI2 J1 0

If I2  = ∅, we have by Theorem 2.1.1 that

yI2 (x) = BI2 J1 yI1 (x) ∀x ∈ Rn ,

which we can rewrite as follows:

 
  yI1 (x)
−BI2 J1 I = 0.
yI2 (x)
 
By taking z to be any row of the matrix −BI2 J1 I , note that z is nonzero and that
z y(x) = 0. According to definition (2.8), the existence of z implies that the functions
y(x) = Ax are linearly dependent. Hence, we must have I2 = ∅, and therefore m ≤ n and
all the yi ’s have been pivoted to the top of the tableau, as required.

Example 2-2-1. For illustration, consider the matrix defined earlier:

 
1 2 4
A = 3 4 8  .
5 6 12

Then

load ex2-2-1 x1 x2 x3
T = totbl(A); y1 = 1 2 4
y2 = 3 4 8
y3 = 5 6 12

T = ljx(T,1,1); y1 x2 x3
x1 = 1 −2 −4
y2 = 3 −2 −4
y3 = 5 −4 −8

T = ljx(T,2,2); y1 y2 x3
x1 = −2 1 0
x2 = 1.5 −0.5 −2
y3 = −1 2 0
26 Chapter 2. Linear Algebra: A Constructive Approach

Note that we cannot pivot any more y’s to the top and that y3 = −y1 + 2y2 . This final
relationship indicates the linear dependence relationship amongst the rows of A, namely
that A3· = −A1· + 2A2· .
The above result can be strengthened to the following fundamental theorem, which
can be taken as an alternative and constructive definition of linear independence.
Theorem 2.2.3 (Steinitz). For a given matrix A ∈ Rm×n , the linear functions y, defined by
y(x) := Ax, are linearly independent if and only if for the corresponding tableau all the
yi ’s can be exchanged with some m independent xj ’s.

Proof. The “only if” part follows from Proposition 2.2.2, and so we need to prove just
the “if” part. If all the yi ’s can be exchanged to the top of the tableau, then we have (by
rearranging rows and columns if necessary) that
xJ1 xJ2 y xJ2
y = A·J1 A·J2 −→ xJ1 = B·J1 B·J2

Suppose now that there is some z such that z A = 0. We therefore have that z Ax = 0 for
all x ∈ Rn . In the right-hand tableau above, we may set the independent variables y = z,
xJ2 = 0, whereupon xJ1 = B·J1 z. For this particular choice of x and y, we have y = Ax
from Theorem 2.1.1, and so it follows that

0 = z Ax = z y = z z,

implying that z = 0. We have shown that the only z for which z A = 0 is the zero vector
z = 0, verifying that the rows of A and hence the functions y are linearly independent.

A consequence of this result is that given a matrix A, the number of linearly indepen-
dent rows in A is the maximum number of components of y that can be exchanged to the
top of the tableau for the functions y(x) := Ax.
When not all the rows of A are linearly independent, we reach a tableau in which one
or more of the yi ’s are expressed in terms of other components of y. These relationships
show the linear dependencies between the functions y(x) and, therefore, between the rows
of the matrix A.
Example 2-2-2. Let the matrix A be defined by
 
−1 0 3
A=  2 −2 4 .
0 −2 10
By using ljx.m, find out how many linearly independent rows A has. If there are linear
dependencies, write them out explicitly.
After entering the matrix A into MATLAB, we construct a tableau and perform two
Jordan exchanges to make y1 and y2 independent variables:
T=totbl(A); x 1 x2 x3
y1 = −1 0 3
y2 = 2 −2 4
y3 = 0 −2 10
2.3. Matrix Inversion 27

T=ljx(T,2,1); y2 y1 x3
T=ljx(T,1,2); x2 = −0.5 −1 5
x1 = 0 −1 3
y3 = 1 2 0

We cannot exchange y3 with x3 because there is a zero in the pivot position. We

conclude that this matrix has two linearly independent rows. By reading across the final
row of the tableau, we see that the components of y are related as follows:

y3 = y2 + 2y1 .

In this example, we could have done the Jordan exchanges in some other way; for
example, T=ljx(T,1,1) followed by T=ljx(T,3,3). However, the relationship that
we derive between the components of y will be equivalent.
Exercise 2-2-3. Let  
1 2 3 4
A= 3 1 3 0 .
1 3 −3 −8
Using ljx.m, find out how many linearly independent rows A has. By working with A ,
find out how many linearly independent columns it has. If there are linear dependencies,
write them out explicitly.

2.3 Matrix Inversion

An n × n matrix is nonsingular if the rows of A are linearly independent; otherwise the
matrix is singular.
Theorem 2.3.1. The system y = Ax with A ∈ Rn×n can be inverted to x = By if and only
if A is nonsingular. In this case, the matrix B is unique and is called the inverse of A and
is denoted by A−1 . It satisfies AA−1 = A−1 A = I .

Proof. Apply Steinitz’s theorem (Theorem 2.2.3) to A to get B such that x = By. B is
unique by Proposition 2.1.2. Finally, y = Ax = ABy for all y shows that I − AB = 0,
and x = By = BAx for all x shows that I − BA = 0.

Example 2-3-1. Invert the matrix

 
2 −1 −1
A = 0 2 −1
0 −1 1
using MATLAB and the ljx function.
load ex2-3-1 x1 x2 x3
T = totbl(A); y1 = 2 −1 −1
y2 = 0 2 −1
y3 = 0 −1 1
28 Chapter 2. Linear Algebra: A Constructive Approach

T = ljx(T,1,1); y1 x2 x3
x1 = 0.5 0.5 0.5
y2 = 0 2 −1
y3 = 0 −1 1

T = ljx(T,2,2); y1 y2 x3
x1 = 0.5 0.25 0.75
x2 = 0 0.5 0.5
y3 = 0 −0.5 0.5

T = ljx(T,3,3); y1 y2 y3
x1 = 0.5 1 1.5
x2 = 0 1 1
x3 = 0 1 2

Note that the inverse of A is now found in this tableau, that is,
 
0.5 1 1.5
invA = T.val A−1 =  0 1 1
0 1 2

The command T.val extracts the matrix from the tableau T (that is, it strips off the labels).
We can check that the computed matrix is indeed A−1 by evaluating AA−1 − I using
the following code:

A*invA-eye(3)

(where eye(3) is MATLAB notation for a 3 × 3 identity matrix). This code should result
in a 3 × 3 matrix whose elements are zero or very small numbers (which may not be exactly
zero because of numerical roundoff error).
As an alternative method for solving this problem, we show that it is possible not to
pivot along the diagonal. For the matrix B of Theorem 2.3.1 to be A−1 , the subscript indices
of x and y must be both arranged in ascending order. If this is not the case, reordering of
rows and/or columns is necessary.

load ex2-3-1 x1 x2 y1
T = totbl(A); x3 = 2 −1 −1
y2 = −2 3 1
T = ljx(T,1,3); y3 = 2 −2 −1
2.3. Matrix Inversion 29

T = ljx(T,3,1); y3 x2 y1
x3 = 1 1 0
y2 = −1 1 0
x1 = 0.5 1 0.5

T = ljx(T,2,2); y 3 y2 y1
x3 = 2 1 0
x2 = 1 1 0
x1 = 1.5 1 0.5

Notice that the numbers in this final tableau are identical to those obtained after the previous
sequence of pivots but that the rows and columns have been reordered according to the
labels. To restore the correct ordering of the rows and columns and recover the inverse A−1 ,
we use the command

Atemp = T.val;

to extract the matrix from the tableau and then use standard MATLAB commands to reorder
the rows and columns of the 3 × 3 matrix Atemp to obtain A−1 . In this case, we note from
the row labels that rows 1, 2, and 3 of T must appear as rows 3, 2, and 1 of A−1 , respectively;
while from the column labels we see that columns 1, 2, and 3 of T must appear as rows
3, 2, and 1 of A−1 , respectively. We can define permutation vectors I and J to effect the
reordering and then define A−1 as follows:

I=[3 2 1]; J=[3 2 1];

invA(I,J)=Atemp;

Alternatively, we can avoid generating Atemp and instead use

invA(I,J)=T.val;

Note that the permutations correspond to the row and column label orderings after the final
Jordan exchange. That is, the vector I shows the final ordering of the row labels (x3 , x2 ,
x1 ), and the vector J shows the final ordering of the column labels (y3 , y2 , y1 ). This scheme
for choosing the reordering vectors will work in general, provided we put the reordering on
the left-hand side of the assignment of T.val to invA, as above.

Of course, for the example above, one can avoid the final reordering step by pivoting
down the diagonal in order. However, there may be problems for which such a pivot sequence
is not possible. A simple example is given by the matrix
 
0 1
A= ,
1 1

for which the (1, 1) element cannot be used as the first pivot. Another example follows.
30 Chapter 2. Linear Algebra: A Constructive Approach

Example 2-3-2. Calculate A−1 using ljx, where

 
0 1 3
A = 4 3 2  .
1 6 6

load ex2-3-2 x1 x2 x3
T = totbl(A); y1 = 0 1 3
y2 = 4 3 2
y3 = 1 6 6

T = ljx(T,3,1); y3 x2 x3
y1 = 0 1 3
y2 = 4 −21 −22
x1 = 1 −6 −6

T = ljx(T,1,2); y3 y1 x3
x2 = 0 1 −3
y2 = 4 −21 41
x1 = 1 −6 12

T = ljx(T,2,3); y3 y1 y2
x2 = 0.2927 −0.5366 −0.0732
x3 = −0.0976 0.5122 0.0244
x1 = −0.1707 0.1463 0.2927

We now extract the numerical values from the tableau, define the permutation vectors, and
perform the reordering as follows:
 
0.1463 0.2927 −0.1707
I=[2 3 1]; J=[3 1 2]; A−1 = −0.5366 −0.0732 0.2927 
invA(I,J) = T.val; 0.5122 0.0244 −0.0976

Exercise 2-3-3. Calculate A−1 using ljx.m, where

 
1 2 3
A = 2 4 2 .
1 1 1
2.3. Matrix Inversion 31

Exercise 2-3-4. Use ljx.m to find the inverses of the following matrices in MATLAB:
     
1 0 1 1 0 1 2 0 1
A = 2 1 1 , B = 2 1 1 , C = 1 2 0.5 .
3 0 0 1 1 0 0 1 1
If a matrix is singular, show the linear dependence between the rows of the matrix. (Use
T.val and perform any reordering needed on the resulting MATLAB matrix to obtain the
final result.)
At this point, we note that the pivot rules described in Section 2.1 in terms of matrix
elements have simple matrix block analogues. That is, instead of pivoting on Ars , we could
instead pivot on the submatrix ARS , where R ⊆ {1, 2, . . . , m} and S ⊆ {1, 2, . . . , n} are two
index sets with the same number of elements. The only real difference is that the inverse
of ARS is used in place of 1/Ars . MATLAB code for the block Jordan exchange is given
in bjx.m. The code incorporates changes to the labels, and so it is an extension of ljx
rather than of jx. In addition, the code updates the matrix A instead of creating the new
matrix B, and so it needs to perform the operations in a slightly different order from the
analogous operations in ljx.
Two successive Jordan exchanges can be effected as one block pivot of order two.
Thus by induction, any sequence of Jordan exchanges can be effected by a single block
pivot. What are the algebraic consequences of this observation? Consider a system that
leads to the following tableau:
(2.9)
A B
C D
where A is square and invertible. Applying a block pivot to the matrix A, the transformed
tableau is
(2.10)
A−1 −A−1 B
CA−1 D − CA−1 B
 B
The matrix D − CA−1 B is called the Schur complement of A in CA D . The algebraic
formula for the block pivot operation will prove to be very useful in what follows. For
example, if A−1 exists, then the original matrix (2.9) is invertible if and only if the Schur
complement D − CA−1 B is invertible.
Exercise 2-3-5. The following tableau expresses how the dependent variables yI1 and yI2
can be expressed in terms of the independent variables xJ1 and xJ2 :
xJ1 xJ2
yI1 = A B
yI2 = C D
By performing some simple manipulations, derive the following formulae:
xJ1 = A−1 yI1 − A−1 BxJ2 and yI2 = CA−1 yI1 + (D − CA−1 B)xJ2 ,
thus justifying the block Jordan exchange formula (2.10).
32 Chapter 2. Linear Algebra: A Constructive Approach

MATLAB file bjx.m: Labeled block Jordan exchange

function A = bjx(A,R,S)
% syntax: B = bjx(A,R,S)
% input: tableau A, integer vectors R,S
% perform a block Jordan exchange with pivot A(R,S)

R = R(:); S = S(:);
[m,n] = size(A.val);

% setdiff(1:m,R) := {1,...,m}\R
I = setdiff(1:m,R); J = setdiff(1:n,S);

% note that values are updated in place

% update pivot column
A.val(R,S) = inv(A.val(R,S));
A.val(I,S) = A.val(I,S)*A.val(R,S);

% update remainder of tableau

A.val(I,J) = A.val(I,J)-A.val(I,S)*A.val(R,J);

% update pivot row

A.val(R,J) = -A.val(R,S)*A.val(R,J);

% now update the labels

swap = A.bas(R);
A.bas(R) = A.nonbas(S);
A.nonbas(S) = swap;

if isfield(A,’dualbas’)
swap = A.dualbas(S);
A.dualbas(S) = A.dualnonbas(R);
A.dualnonbas(S) = swap;
end

tbl(A);

return;

2.4 Exact Solution of m Equations in n Unknowns

At this stage, we have considered only square systems where the number of variables is
the same as the number of equations. In optimization applications, it is more likely that
the systems under consideration have different numbers of variables than equations. Such
Random documents with unrelated
content Scribd suggests to you:
Still for a moment he wavered in doubt.
“And if I comply, sir, with your request,” he said slowly, “what
guarantee have I that this lady shall suffer no harm?”
“My word of honour, sir,” I answered quietly.
“Honour?” my lady cried quickly. “If you look for aught of honour
from this gentleman, I warn you that you will look in vain!”
“I thank you for your good opinion of me, madam,” I answered
calmly, “which, coming from your ladyship, was such as I should
have expected. And I can the more readily overlook your words,
since I know how inopportune must be my presence. But I think that
for the present it is for me to command and for you to obey.”
“I waited a moment, but she did not speak. She only looked at me.
Yet that look hurt me more than words. Despite my assumed air of
carelessness, the scorn and loathing in her eyes made me wince.
“And I will give your ladyship a word of future advice,” I continued
with a sneer. “Open windows consort ill with conspiracy!”
“I trust, sir, that you do not include me in that category,” the colonel
broke in suddenly, “for I think that you have no evidence to support
such an assertion.”
“No evidence at all,” the stout man cried, recovering somewhat from
his first shock of surprise. Though if ever conscious guilt looked out
of a man’s eyes, it was in his, and his twitching lip belied his air of
assurance. “Not a shadow of it!” he continued, growing bolder at the
sound of his own voice. “And if you were listening, captain, you
heard what I was saying. As a magistrate of this country it is my
duty—hum!—strictly in the interests of government to acquaint
myself with the opinions of the most influential families. Had you
delayed your appearance a moment longer, you would have heard
the measures I was about to take to—hum!—counteract the scheme
of this misguided lady!”
“As to that,” I replied—and I could afford to forgive him his
cowardice—for the man was playing into my hands, “I am willing to
take your word for it, Mr.——?”
“Wetherell,” he answered—“Nicholas Wetherell, at your service. I am
well known, sir! I believe that I can honestly say,” he added
pompously, “that I am a man of some weight in the city of Exeter.”
I looked him steadily in the face.
“I do not doubt it,” I answered quietly.
The young baronet gave a short laugh and the colonel’s mouth
writhed itself into a grim smile. Nay, for a moment even I thought
that my lady’s face relaxed somewhat; but, meeting my eyes, she
stiffened again into her old mask of scorn.
“Yet I do not know, Mr. Wetherell,” I continued dryly, “that the
government would entirely approve of your method of obtaining
information.”
“There is no more loyal subject, sir, in England than I!” he cried
hastily.
“Indeed,” I answered, “I think that your loyalty, sir, is as strong as
your courage!”
“And I trust that the latter does not need to be questioned, captain,”
he blustered.
“It does not!” I replied, with I fear some of the contempt I felt for
the man appearing in my voice. “But I will put your loyalty to the
test. You have wine there,” I continued, stepping to the table, the
whilst they watched me in silence. “As loyal subjects, therefore, you
will not refuse to join me in drinking a toast.”
I slowly filled four glasses, three of which I placed in the centre of
the table.
“Gentlemen,” I cried clearly, raising the fourth, “to the health of his
Majesty, King William!”
For it may be ten seconds they sat silent. The room was very still. I
gazed full at my lady where she stood upon the hearth. She was
leaning slightly forward, her hands pressed against her bosom,
watching the scene before her with fascinated eyes. It was a battle
of our wills and I had won!
Suddenly the stout man stretched out a shaking hand.
“To the health of King William, sir!” he cried quickly, all the more
anxious to make up for his first slight hesitation. “I desire no better
toast.” And he drained his glass.
“Nor I also,” said the colonel more slowly. “To King William, sir! I
wish him success in his campaign!”
But I noticed that he avoided meeting my lady’s eyes.
I turned to the young baronet. And here I expected trouble.
“Sir Rupert Courtenay,” I said quietly, “it is your turn!”
For a moment he hesitated, and I saw a quick glance pass between
him and my lady. Then, to my surprise, he gave a light laugh and
raised his glass.
“I drink to the king!” he cried clearly, with a defiant look at me.
I noted mentally his reservation, and I knew whom he meant. But it
was not my cue to quarrel with him. I was more than content with
my victory.
“Good!” I said aloud, following his example. “That is sufficient, sir!”
and I replaced my glass upon the table. “I will not detain you longer,
gentlemen,” I added. “I have already said that for reasons of my
own I am willing to overlook your presence here. Whether I take
further steps in the matter will depend entirely upon your future
conduct. But for the present, gentlemen, there is the door. I should
recommend you to take your departure with all speed. The night air
is somewhat chilly!”
And they needed no further urging. Without a word the stout man
replaced his wig upon his head and, catching up his hat, started for
the door. Perhaps he feared that I might yet repent my clemency.
And the colonel was not slow to follow his example. True, he had the
decency first to turn to my lady, as if to make his adieux; but one
glance at her rigid figure and burning eyes was sufficient for him.
“This comes of having a woman in it,” he said shortly, turning on his
heel and leaving the room.
I glanced at my lady. Her head was bent. I could not see her face.
Suddenly, with a quick, defiant movement she upraised it, tossing
the hair from her forehead.
“Cousin Rupert,” she cried proudly, “your arm!”
The young baronet hastened to do her bidding, and she moved by
me, drawing her scented skirts aside, lest they should touch me in
passing. At the door, still with her hand upon his arm, she turned.
“You have beaten me twice, sir,” she said, in a voice trembling with
passion. “For your own sake beware of the third time!” And she
swept out.
 CHAPTER VI
OF HOW I PLAYED KNIGHT-ERRANT, AND OF
MY LADY’S GRATITUDE

When my lady had left the room and the swish of her silken skirts
had ceased I hastened to take my departure, for I did not doubt that
if it was discovered I had come alone and that the troopers of whose
presence I had talked so glibly existed only in my own imagination,
her ladyship would take active measures for the recovery of the
paper in my possession.
I stepped through the window, therefore, and sought the spot where
I had left my horse. I took the bridle in my hand, and, avoiding the
avenue leading to the gates, led him through the tangled
undergrowth until we came out, some hundred yards below, upon
the road.
Here I halted and looked back. There was no sound to break the
stillness save the soft sighing of the night wind in the branches
overhead. At my feet the white road stretched away into the night,
silent and lonely.
I lost no time, therefore, in mounting, and a few minutes later,
without molestation, reached the village.
As I rode up the main street I saw that the troopers had quartered
themselves upon the inhabitants; for a dozen or more were lounging
in the open doorways upon either side of the road, who upon
catching sight of my face, stiffened their backs and saluted me as I
passed. Before the inn itself a noisier group were gathered, whose
laughter and jests died away for a moment as I dismounted, to
again break out with renewed clamour when I had crossed the
threshold.
Calling for a light, I made my way to the little chamber that I had
previously occupied. Carefully fastening the door behind me and
satisfying myself that I was indeed alone, I set the candle upon the
little table and drew the precious paper from my breast—the paper
big with the fate of my lady’s freedom! Perhaps in the temper of the
times her life! For I well knew that in the nation’s present state of
panic the government would have little consideration and respect for
either rank or sex. And ’twas well known that the Earl of Ingram was
one of James’s most trusted councillors.
The paper itself was a small sheet of coarse white parchment and
was folded into three.
As I stood turning it over in my hands, on a sudden my lady’s face
seemed to confront me.
Again I went over the scene so lately enacted, again I heard her
clear voice raised in pleading, and moved by some impulse of I know
not what, I laid the paper unopened upon the table, and with the
candle in my hand I went down on my knees upon the dusty floor.
Presently in the corner of the room farthest from the window I found
that which I sought—a loosened board.
With the aid of my sword I raised one end of this, and in the cavity
beneath I slipped the fateful parchment, replacing afterwards the
board and removing all traces of my handiwork. It was not until this
was accomplished that I breathed more freely, and I sought my
couch with a distinctly lighter heart.
It was late next morning when I awoke, and the bright June
sunshine was streaming into the room through the little latticed
window. I dressed hastily and descended to the inn parlour, where I
found the two cornets already at the table. During the meal I
gathered from Cornet Graham some information respecting Cleeve.
The village itself and the surrounding farms belonging to the Ingram
family contained some eight hundred inhabitants; but of this
number, fully two hundred—being the young and able-bodied men—
were serving under the earl in Ireland. Of the remaining number
some eighty only were males, and these the greybeards or youths
too young to endure the arduous duties of the Irish campaign.
When I had made myself acquainted with such gossip as he had at
his command, I gave the cornets their orders for the day, and
returning to my room, arrayed myself in full uniform. Also whilst in
London I had taken the opportunity of visiting a tailor’s shop in the
Haymarket and purchasing to myself sundry little fripperies of attire,
such as a lace cravat and fashionable peruke, the which I smiled at
myself for donning. Nevertheless, I was minded to look my best
when I again presented myself at Cleeve.
My vanity satisfied, I made my way to the stables, and mounting my
horse, set out for the manor.
Evidently my arrival had been expected, and my lady had given her
orders as to my future treatment; for as I reined in my horse and
dismounted before the door, of a sudden it opened, and the steward
came to the head of the steps. Behind him I caught a glimpse of a
little group of servants, most of whom were women, who regarded
me with feminine curiosity.
“Is it your pleasure to enter, sir?” said the steward respectfully,
though the sullen hate in his eyes belied his deferential manner.
“And my horse?” I said curtly, ascending the steps.
“Shall be looked to, never fear,” he replied; then, raising his voice:
“Martin!” he cried; and at his call the youth who had accompanied
him upon the ride to Exeter came slowly forward from the group
behind. “Take this gentleman’s horse to the stables.”
“And see that he lacks for nothing,” I said grimly, “or you will regret
it, my friend!”
For I had not forgotten this same youth’s insolence to me upon the
road, and I was determined to show these people that for the
present, at least, I was their master.
When he had moved away, muttering and cursing to himself, I
followed the steward to the dining hall, which I have previously
described. Everything in it was as I had seen it upon my first arrival,
only the room was empty.
“My lady is out?” I said carelessly.
“No,” the steward answered slowly; “she is not out, but——”
“She is not desirous of seeing me, you would say?” I answered
quickly. “Be frank, man! I understand. And for the present it is a
matter of indifference to me, as I intend to make myself acquainted
with the gardens and the cliffs adjoining. But you will prepare a
room for me,” I continued, watching him out of the corner of my
eye; “and also inform your mistress that I dine at noon, and must
then request their ladyships’ presence.”
The start he gave did not escape me.
“My ladies will dine with you?” he stammered.
“Certainly,” I replied harshly. “Is there any reason why they should
not? Or does it require another necklet to teach you that I am a man
to be obeyed?” And with this parting shot I left him. At the threshold
of the room I turned. He still stood in the centre of the floor,
apparently bereft of speech.
“The hour is noon,” I said quietly, “and I beg that they will not keep
me waiting.”
Arrived in the open air, for the first time I remembered that I had
not as yet seen the gigantic figure of Sampson Dare; and
encountering the youth Martin, I asked what had become of him.
He looked up for a moment from his task.
“He is not here,” he said, scowling.
“I asked you where he was,” I answered dryly.
“In London, then, if it will please you,” he retorted sullenly.
“Ah!” was all I said in reply. And turning on my heel, I left him. But I
understood perfectly upon what errand the man had gone, and I
saw by this, even if I had not received ample proof already, that my
lady was not a woman to let the grass grow under her feet.
Doubtless she had powerful friends in London whom she trusted to
interest government on her behalf. If so, I thought, time alone will
show whether her confidence was misplaced.
For the present, at least, I held the winning cards.
From the stables I took the first broad path, and finally came to the
cliffs, where I stumbled upon the little path which we had ascended
fourteen days previously in the darkness and storm.
And the longer I looked at it, the more I wondered that our attempt
on such a night had not ended in disaster. Even by daylight it was no
pleasant task to tread the narrow ledge, as I discovered for myself
when I essayed to make the descent. No pleasant knowledge, that a
single slip or a momentary giddiness would precipitate me into the
sapphire sea, slumbering peacefully a hundred feet below.
Nevertheless, I persevered in the attempt, and in due time reached
the beach.
On the flat slab of rock at the bottom of the steps I paused. The tide
was fast coming in, and the first waves were gently rocking the boat,
that still lay where we had before seen it, in the little bay beneath
the cliff.
The sun beat fiercely down upon my head and sparkled upon the
crests of the waves, until the whole surface of the water resembled
a sea of burnished silver. Presently my eyes lighted upon something
that roused in me a momentary curiosity. The beach, as I have
previously said, was composed of small, grey pebbles; but about ten
feet from the foot of the cliff lay a patch of hard sand, some twelve
feet square, its smooth, yellow surface showing plainer by reason of
the slate-coloured shingle surrounding it. How it had come in such a
place was beyond my imagination to fathom.
For some minutes I stood gazing over the sea; then, turning, I
retraced my steps, and passing idly through the sunlit gardens,
reached the house.
It wanted a few minutes to the hour when I entered the dining hall,
and I saw that my instructions had been obeyed.
In the centre of the room, a table, laden with massive silver plate,
had been set out for three. I laid aside my hat and sword and waited
for my guests’ appearance with a heart that beat more rapidly than
was its wont.
Nevertheless, it was five minutes past the hour of noon when the
steward appeared in the doorway.
“How is this?” I said curtly. “You carried my message?”
“My ladies would know if you command their presence?” he
answered slowly.
“Aye,” I cried impatiently. “Command them, then, in the devil’s
name!”
“I will not fail to give them your orders,” he retorted quickly, and was
gone ere I could make reply.
And presently they came.
Long ere they entered the room I heard the soft rustling of their
gowns and the click of their heels upon the polished oaken floor. The
younger sister’s face was pale, with a certain shrinking timidity in the
covert glance she stole at me. My lady’s face was stone, and her
eyes were hard and glittering. I bowed low to them on their
entering, a salute which the younger woman returned with a slight
acknowledgment, and which my lady totally ignored.
When they had seated themselves opposite me, and the steward
had placed the covers upon the table, I lost some of my original
nervousness in fulfilling the civilities of the table. And I strove to
appear entirely at my ease, praising the beauty of the gardens and
the cooking of the dishes before us. But to all my conversation—nay,
even to direct questions, the younger woman replied only in
monosyllables—my lady not at all.
And they made no pretence of eating what I pressed upon them.
Stately and still, they sat and looked at me. And gradually I froze!
They could not have hit upon a better device whereby to unnerve
me. Minute by minute, try as I would, my assumed air of assurance
left me. Beneath the gaze of my lady’s scornful eyes, of her rigid
figure and curled lip—that all proclaimed my presence as pollution—I
felt the hot blood mantling to my brow, and the words died
stammering from my tongue.
I strove to gather courage from the wine before me, and my hand
was shaking.
Presently I could bear it no longer, and I pushed my plate from me
and rose to my feet.
“If you would prefer for the future to dine alone, madam,” I said
sullenly, “you have my permission to do so.”
“I thank you for the concession, sir,” my lady said coldly; “which,
being prisoners, we have no right to expect. Also as our jailer,” she
continued with bitter emphasis, “is it permitted us to take our daily
ride alone?”
“Certainly,” I answered; though the scathing contempt in her voice
and eyes made me wince. “I see no reason against your doing so!
And believe me, madam,” I continued earnestly, “excepting in so far
as my duty here compels me, I have no desire to interfere with your
actions or restrain your freedom.” Yet I might have spared myself
the confession, for without a word of thanks from either of them
they moved away. “And one word more, madam,” I continued, as
they reached the doorway; “the paper——?” and I hesitated.
My lady turned quickly at the words.
“The paper which you stole, sir?” she said coldly.
“Which I—— You use a harsh phrase, madam,” I answered
irrelevantly, advancing a few paces nearer to her. On a sudden I was
minded to prove to her that she was unjust to me—to shame her for
her words. “I had no wish,” I continued in a low voice, “to use it to
your injury, my lady, and so——”
“You can spare me further, sir!” she interrupted icily. “Your insolence
I can bear for the present, but not your hypocrisy. That you are
present here to-day is insult enough to my womanhood; for the rest,
I pray you keep as far from me as the limits of this house permit.”
For a moment I stood before her tongue-tied and trembling. Then:
“Very well, madam,” I said harshly. “As you please! I was about to
say that this paper is no longer in my possession, but in the hands of
those who will make ruthless use of it should any accident befall me!
I am aware that the cliffs of Cleeve are dangerous,” I added
meaningly; and with a bow I went back to my seat.
But alone with my thoughts, I cursed the foolishness that had led
me to demand their presence, and I laid the humiliation I felt at my
lady’s door. Bah! ’twas a pitiful little triumph, and one worthy of her!
As I recalled the look that had shone in her eyes I bit my lip
savagely and strove with what appetite I could rally to complete my
meal. But the taste had gone from the dishes, the wine was sour.
Presently, in sheer ill humour with things in general, and with my
lady in particular, I rose from the table, and making my way to the
stables, mounted my horse and set out for the village.
Arrived there, I assembled the astonished troopers, and there for an
hour I drilled them in the little square before the inn. And well for
the knaves that they gave me no offence upon which to vent my
rage, for it would have gone hard with the offender, whomsoever he
might have been.
It was whilst engaged in this duty that I became aware of the clatter
of a horse’s hoofs approaching rapidly down the village street, and
looked up to meet my lady’s scornful eyes, a smile of disdain upon
her lips.
’Twas but a moment’s glimpse I gained of her, the next and she had
passed; but for me the pleasure had gone from my task and again
my thoughts were embittered.
Accordingly I dismissed the men somewhat curtly, and turning my
horse’s head, rode past the church in the direction which my lady
had taken. I went at a foot pace, however, for I had no desire to
overtake the woman in front; at the same time, I was minded to ride
as far as Cleevesborough, the tall hill rising to the south of the
village. To overtake her? My faith, no! Henceforth I would avoid her
and so follow her advice. At the foot of the hill I fastened my horse
to a straggling thorn bush and ascended the rest of the way on foot.
From the summit I glanced below me.
Some two miles distant my lady showed as a rapidly moving speck
upon the ribbon of white road that wound down the valley. A mile or
so beyond her I could distinguish the upper portion of a large black-
timbered mansion rising above the surrounding trees. Long after she
had disappeared beneath the latter, I still remained upon the summit
of the hill, admiring the beauty of the scene. At length I descended
to where I had left my horse, and mounting, rode slowly down to
the level of the road. Arrived there, however, I determined that I
would not return by the way that I had come, but would strike
inland and make my way in a half circle back to Cleeve.
With this intention I crossed the road and entered the trees upon
the opposite side. I found that these did not continue for more than
a few hundred yards, giving place to a succession of little grass-
grown hollows, covered with sparse, brown bracken, and here and
there a scattered boulder breaking through the coarse turf.
In most of these depressions were pools of stagnant water and
patches of black bogland that required some caution in avoiding, so
that it was not without satisfaction that about a mile farther I came
upon a path. It was a mere bridle track running down the centre of
the hollow, and was little used, as I saw at once by the marks of the
hoof prints upon its surface, which were fully a week old. I set my
face in the direction of Cleeve, and the sorrel broke into a trot. The
track led from hollow to hollow, some broad, some narrow, and for
the most part with marshy, reed-grown ground on either side.
I rode with slackened rein, my hat pressed low upon my brow, and I
gave myself up to my thoughts—thoughts in which my lady strangely
mingled.
The sun beat down with almost tropical heat upon me; and what
little wind there had been on the uplands above was here shut out
by the slopes upon either side.
How far I had proceeded thus I do not know, for lulled by the
rhythmic stroke of my horse’s hoofs, with half-closed eyes I took no
account of time.
On a sudden the sorrel stopped with a jerk that all but unseated me.
Then he commenced to back, and I felt his flanks quiver. In an
instant I was alert, and I searched the pathway in front with my
eyes, thinking perhaps some viper or creature of the bog had
startled him. But I could see nothing.
Slipping from the saddle, therefore, I soothed the frightened animal
as best I could, and glanced keenly around me. I was in a deeper
depression than any I had as yet traversed—a circular, cup-shaped
hollow, its sides sloping some quarter of a mile to the ridge above.
In front of me stretched the path, to right and left of it patches of
bogland, its black surface covered with slime and green water-weed.
Stay, was it the path? I bent and gazed fixedly at the track before
me, then, unbuckling my sword, I slowly advanced, probing the
ground in front of me. At the sixth step my sword encountered no
resistance. I drew back with a shudder, and, despite the heat of the
day, a cold chill ran down my back. Again I tested the spot with my
sword. The green mass quivered at my touch, but there was no solid
ground, and in a moment the truth flashed upon me.
The bog upon the right, which stretched some distance up the slope,
had slipped, undermining the path, which at a casual glance still
retained its ordinary appearance by reason of the green entangled
weed floating upon its stagnant surface.
Never had Nature cunningly concealed a man-trap more
treacherous. But for my horse’s sagacity I should have been—pah!
the very thought of sinking slowly beneath the noisome ooze made
me shudder. I turned away, and taking the bridle rein in my hand, I
skirted round the smaller patch of bogland on the left and regained
the track beyond.
Arrived there, however, I did not at once ride forward. On the
contrary, I retraced my steps to the brink of the bog, for I was
minded to see how far the pathway was undermined. To the spot
where I had stood before upon the opposite side was, as near as I
could judge, some fifteen feet. I cast about for some means of
warning travellers of their danger; but there was nothing but the
bare hillside around me, and with a shrug I turned away. After all,
what concern was it of mine? Resolving, however, for the future to
take to the ridges and to avoid the low-lying ground, I remounted
my horse and headed straight for the top of the hollow, some
quarter of a mile distant. I had all but reached the summit of the
slope when a dull sound struck upon my ear—the regular thud, thud
of a horse’s hoofs and of a horse ridden at speed.
I drew rein in idle curiosity as to who the rider might be in such an
unfrequented spot. Nearer and nearer came the sound behind me,
and a minute or so later the horse and its rider flashed into sight.
It was my lady herself!
My lady mounted on the chestnut mare that I had so admired. And
in a moment I understood. The green track that had nearly proved
fatal to me led to the mansion I had seen from the summit of
Cleevesborough, the smooth turf forming a more pleasant bridle
path to the village than the dusty road.
Along this pathway my lady was sweeping at full gallop, was
sweeping to the death that lurked below! And I hated her!
But even as the thought came to me I gathered up the reins, drove
in my spurs, and in a moment I was thundering down the slope.
Even now, though years have lapsed, how the memory of that ride
comes back to me!
Less than a quarter of a mile distant, at right angles to me, and
somewhat nearer than I to the green pitfall below, came the
chestnut at full gallop, spurning the ground from beneath her flying
hoofs. And with a certain shrinking of the heart I could in nowise
account for, moment by moment I realised that the sorrel I bestrode
was no match for the more thoroughbred and lighter weight of the
animal before me. And seeing this, I rose in my stirrups and
shouted. It may have been that my words were drowned in the
thunder of our galloping hoofs. It may be, seeing that it was I—nay,
more likely, I thought bitterly, because it was I—my lady took no
heed. And in my heart I cursed the wilfulness of this woman who
would take naught from me, not even words of warning. Yet this
very wilfulness made me but the more resolved to save her—to save
her at all hazards! I settled myself, therefore, more firmly in the
saddle, and the race continued. And now less than two hundred
yards divided us—two hundred yards and the morass between.
Also, now that I was nearer, I realised that for once I had done my
lady an injustice, and that the animal beneath her was far beyond
her control. So clear, indeed, was the air, so brilliant the glaring light,
that I could mark the chestnut’s straining eyes and the white foam
flying from her lips—aye, and I could clearly distinguish the outline
of my lady’s graceful figure as she rose and fell to the mare’s stride
—could see the dainty head and glowing cheek, the proud, oval
face, and the stray tresses of golden hair escaping from beneath her
grey-velvet riding hat. And now she was but forty yards distant from
the brink, and spur as I would, the sorrel might do no more.
Again I rose in my stirrups.
“For God’s sake, beware of the bog, madam!” I shouted, pointing to
the ground between us. “Turn to the left! To the left!” For I saw that
her only chance lay in skirting round the smaller patch as I had
previously done. And now, indeed, despite her contempt for me,
something in my voice must have attracted my lady’s attention, and,
accustomed as she was to the nature of the surrounding country,
her quick eyes discerned her danger; for I saw her throw her whole
weight upon the left rein.
But it was too late, nothing could stop the chestnut now, and seeing
this, my lady abandoned the attempt, and arriving at the brink, she
lifted the mare with her hand and essayed the leap instead.
For a moment, as she rose into the air, I closed my eyes, and but
opened them upon hearing the dull thud with which the mare
alighted in safety upon the near side.
Not quite, for her hind feet striking upon the very edge of the path,
the soft, spongy turf gave way beneath the shock, and she rolled
back into the bog.
Yet my lady was safe. Even in mid air she had freed her foot from
the stirrup, and as the mare struck the ground she sprang clear—to
fall, indeed, upon her hands and knees on the soft turf a dozen
paces distant.
A moment later I reached the spot and flung myself from the saddle.
My lady had already regained her feet.
“You are not hurt, madam?” I cried anxiously, forgetful for the
moment of the hatred between us.
She did not answer me. No doubt the fall had dazed her. Instead,
she turned in the direction of her horse and took two steps forward.
But I was too quick for her.
“Have a care, madam!” I cried, barring her further progress. “This is
no woman’s task, and the ground is dangerous. Trust me,” I
continued earnestly, “to do all that is possible to save your
favourite.” I turned from her as I spoke and made my way to the
edge of the bog.
With eyes dilated with terror and blood-red nostrils distended, the
mare still struggled to regain her footing. At no little risk to myself of
being drawn into the bog, I succeeded at length in laying hold of the
rein, and I drove my heels into the turf and exerted all my strength
—aye, till the muscles of my back and arms cracked beneath the
strain—in a vain endeavour to assist her efforts. But though her
forefeet, indeed, rested upon the more solid ground, her struggles
were growing fainter and she was sinking rapidly. I saw that it was a
question of moments only, and there was but one expedient.
Loosing hold of the rein with my right hand, I drew my sword and
thrust her lightly in the breast.
At the touch of the steel she gave a snort of mingled pain and terror
and rose from her knees. Her hoofs caught, slipped upon the soft,
wet turf, caught again as I threw my weight into the scale, and the
next moment I was on my back upon the grass, and the hollow spun
round me in a golden mist. ’Twas but a second or two I lay there,
however, for the mare’s head had caught me fairly in the chest and
the breath was gone from my body. Then I slowly rose to my feet
and turned to look behind me. Twenty paces away my lady stood
soothing the frightened animal, that now stood quivering with terror
from head to foot.
I approached her slowly, with a feeling of exultation in my breast.
For had I not proved my words to her and succeeded? Had I not by
service rendered placed her in my debt? Surely I had earned this
woman’s gratitude, and I would take it as my just reward. She did
not look at me as I approached. Instead, she drew a snow-white
kerchief from the bosom of her gown and with ostentatious care
began to staunch the blood that welled from the wound I had
inflicted upon the mare’s breast. One might almost have supposed
that she thought more of this slight wound than if the animal itself
had been engulfed. Three paces from her was a large flat boulder,
one of many that lay scattered upon the turf. By the side of this I
halted. Still she did not turn her head.
Her hat had fallen back, revealing the tresses of golden hair straying
in wild disorder upon her neck. I had leisure to observe more closely
the exquisite symmetry of her figure, displayed as it was to its best
advantage by the tight-fitting riding coat she wore.
Feeling, I suppose, my eyes upon her, she deliberately turned her
back on me and continued her task as before. I waited two—three—
minutes, still she did not speak.
“Am I to have no thanks, madam?” I said at length in a low voice.
“It was a praiseworthy action,” she answered icily; “and as such
doubtless carries its own reward.”
On a sudden my exultation vanished at her words. It was borne in
on me that she would rather have been beholden to the meanest
beggar upon the road than to me. Yet I would not be discouraged so
easily. Again I broke the silence:
“I do not think that the animal is much hurt, madam,” I said humbly.
“’Tis but a flesh wound at most. Nevertheless, in case of further
mishap, may I be permitted to return with you?”
And then indeed she looked at me.
“The road is public property, I believe, sir!” she replied in the same
icy tone. “And I cannot prevent you, if you force your presence on
me. But if you were anything but what you are—if you laid any
pretence to being a gentleman, you would spare me the loathing of
your company!”
I fell back then, indeed, as if she had struck me, and without a word
in reply I returned slowly to the sorrel’s side and made pretence of
tightening the girths with fingers that were trembling.
This was her gratitude! This my reward! Yet I consoled myself with
the thought that even yet she would be obliged to seek my
assistance in remounting, and I determined that I would not be the
one to again make advances. But I did not know my lady, nor had I
taken into consideration the fact of the boulders scattered plentifully
around.
As with my back to her I fumbled at my saddle, I heard the mare’s
footsteps receding; and, turning sharply, was in time to see her
ladyship move slowly away.
Erect in the saddle, with never a backward glance, she urged the
mare into a canter, breasted the green slope, disappeared, and left
me there in the sunlit hollow—alone!
 CHAPTER VII
OF CERTAIN PASSAGES IN THE ROSE GARDEN

Neither that day nor the following one did I again see my lady. And if
I yet retained some lingering hope of her relenting in her old attitude
towards me, I was doomed to be disappointed.
Nevertheless, in pursuance of my duties many a visit I paid to
solitary farms and to the houses of the gentry scattered along the
coast; at some to meet with the respect which my uniform and the
nature of my errand warranted; at others—and these for the most
part belonging to Catholic gentlemen—to encounter an ill-concealed
hostility that sufficiently testified with whom their sympathies rested.
And everywhere I found the same brooding spirit of alarm and
discontent. The whole country was on the tiptoe of expectation.
Rumours of William’s defeat and death were freely circulated by
James’s infatuated followers. Scarcely a fisherman upon the coast
but who woke fully expecting to see a French army drawn up upon
the shore. Indeed, throughout all the western counties the Jacobites
were secretly arming, awaiting but the success of the Stuart cause in
Ireland to kindle anew the flames of civil war.
Such was the condition of England in this, the month of June, 1690.
’Twas some ten days after the incident in the hollow that I again had
speech of my lady. Intelligence had reached me of a rumoured
landing of arms in the neighbourhood of Teignmouth. Leaving De
Brito, therefore, and twenty men to guard the village, with the
remaining two-score troopers at my heels, I set out northwards.
Whether, for once, rumour had lied, or whether the Jacobites had
got wind of our presence I do not know; but certain it is that though
we lay all that day and the next concealed in a beech wood near to
the town nothing occurred, nor did we see any sign of a vessel off
the coast.
On the third day, therefore, empty handed, we returned to Cleeve. It
was evening, when, dismissing the troopers at the entrance gates, I
rode up to the manor.
Seldom, I think, in all my wanderings have I witnessed a more lovely
night. Behind the torrs, in a golden glory the sun was sinking to its
rest, gilding the foliage of the oaks with a dozen varying shades of
orange, pink and purple, and in the light of which the house before
me stood refulgent, as if ’twas bathed in lambent flame.
I rode slowly to the stables, and having seen my horse safely stalled,
I passed by way of the terrace to the house.
The doors and windows stood wide open, for ’twas a warm June
night and the smoke curled lazily from the tall chimneys into the still
evening air; but there was no other sign of life about it, and I
entered and made my way to the dining hall without encountering
any one.
Here indeed, though I met with the same solitude, I found a cold
collation upon the oaken table, to the which I readily applied myself,
wondering the while at the silence of the house and half wishing—
such is man’s inconsistency—for their ladyships’ presence. Once, my
eyes travelling through the open window, rested upon the figure of a
woman passing swiftly down one of the terrace walks. But the
distance and the gathering dusk left me uncertain as to whether it
was my lady or no.
Presently I rose and sauntered slowly through the gardens to the
cliffs; and here, upon the highest point, I flung myself upon the
grass and gazed in genuine admiration upon the scene.
Long I lay there watching the lights spring up, one by one, in the
village below me, until the crimson glow faded from the fleecy
clouds above; then at last I rose and slowly retraced my steps. As I
passed through the misty, scented gardens, idly culling the roses
that lined the pathway on either side, the bats were fluttering
around me on their silent wings, and faintly in the deepening dusk
came the hoot of wandering owls. Somewhere in the trees around
the house a nightingale poured forth its flood of song, as slowly
upon the quiet landscape fell the peaceful stillness of the summer
night.
Presently I saw upon my right a green arch of yew, and passing
beneath this, I came upon a spot the like of which I have never seen
to equal. Surely, I told myself, this is my lady’s garden, and one well
worthy to match with her in point of loveliness. For it was a veritable
bower of roses—a smooth stretch of green lawn, interspersed with
beds of flowers of every conceivable shade of colour. The thick yew
hedge enclosing it was cut in the stiff and formal manner of the
Dutch, a fashion brought with William from The Hague. In the centre
stood a white marble fountain, the jet from which fell with a
pleasant plash into the wide basin beneath. One side of this
enclosure was fenced by the low stone wall that ran above the moat,
and facing me, another leafy arch gave entrance to the terrace walk
beyond. Yet it was not admiration for the scene before me that
brought me to a sudden halt and caused my heart to quicken its
pulsations; for upon the broad steps at the fountain foot a woman
was seated with a canvas in her hand, a brush and palette at her
side. At the sound of my footsteps she turned her head, and I saw
that it was my lady’s sister, Mistress Grace.
“Captain Cassilis,” she said with a winning smile, “saw you ever a
more lovely night? Alas! I fear that my poor efforts fall far short of
the reality. But you shall judge, sir, of their merits for yourself.” And
she held the canvas out to me.
For a moment I stared at her in sheer astonishment. Then with a
beating heart I took the canvas in my hand; for although she had
not shown the same hostility towards me that my lady had done, yet
never before had she addressed me of her own free will.
“You will let me see it?” I said humbly.
“You have been a great traveller, I believe, sir,” she answered; “and
therefore your opinion should carry weight.
“But you do not speak!” I held the painting from me, the better to
observe it; though had it been the finest masterpiece that Rome or
Florence could produce, I doubt if in that moment I had marked its
beauty; for my head was in a whirl. Was the barrier between myself
and these women to be broken down at last?
“Madam,” I replied hastily, “to say that it is beautiful, is only——”
“To flatter me!” she interrupted quickly, as I hesitated, racking my
brains for a phrase she might not deem extravagant. “And,” she
continued lightly, “is it your custom, sir, to judge of a picture’s
excellence upside down?”
“Madam,” I stammered in utter confusion, “I crave your pardon——”
“Fie, sir!” she continued, smiling. “Do you not know that to wound a
woman’s vanity is to make an enemy for life? And I had inferred,”
she continued in the same tone of raillery, glancing at the roses in
my hand, “that you were a lover of Nature yourself.”
“If, madam,” I said hesitatingly, “you would permit me in some slight
measure to retrieve my unfortunate error, and would so far honour
me as to accept this humble tribute of my regret——”
“I love flowers,” she said simply, taking my proffered gift and raising
it to her face, perhaps to hide her heightened colour.
“And I also,” I replied, looking down at her slight, girlish figure, as
with deft fingers she rearranged the roses in her lap. “For to one
fresh from the reeking kennels of London, where pestilence stalks
hand in hand with crime, this old world garden where you live comes
as a glimpse of Eden.”
“Yet I have read, sir,” she said impulsively, “that every Eden has its
——”
She broke off abruptly without finishing the sentence; and even in
the dusk I saw the warm blood mantle to her brow.
“Its serpent, you would say,” I said quietly, reading her thoughts;
“aye, madam?”
For a moment or two there was silence—a silence broken only by
the soft sound of the falling water and the voices of the night. I
glanced at the woman before me and my heart sank. What a gulf
there was between her life and mine!
Presently she spoke.
“It was a thoughtless speech,” she said in a low voice. “I pray you
forget my hasty words.”
Again there was silence between us. But the memory of my errand
in this place, of my lady’s open scorn, and of the haunting feeling of
unrest that I had previously felt recurred to me again with double
force.
“You, at least, do not hate me, madam,” I said bitterly, leaning upon
the marble basin and gazing into the water below.
“It may be that I have not my sister’s pride,” she answered slowly,
“or it may be that my nature is not formed for hatred. And then—”
she continued, bending lower over the flowers, so that I could not
see her face.
“Yes, madam?” I said inquiringly, as she paused.
“Forgive me if I am wrong,” she replied; “but I do not think that you
are happy here.”
“Happy?” I cried, startled out of my self-control. “God knows that I
am not! Do you imagine that I have no feeling? That it is pleasant
for me to be shunned as if I were a leper—unfit for human ken? But
for your kindly speech of me to-night, since first I came to Cleeve I
have encountered naught save contumely and cruel words. Yet I
would remind you, madam, that another in my place might not have
dealt with you so leniently.”
In the silence that followed on my words—a silence in which the
woman before me rose to her feet, and, laying the flowers that I had
given her upon the fountain rim, stooped to collect her scattered
colours—a quick step sounded on the terrace walk, and my lady’s
figure appeared beneath the arch of yew.
“Why, Gracie?” she cried gaily. The fountain was between us—she
had not discerned my presence. “I have been seeking everywhere
for you! And what is this?” she continued, catching sight of the
flowers, and raising them to inhale their fragrant odour. “Roses? Ah,
now I understand! Cousin Rupert has been here, and the painting, I
fear me, was but a pretext!”
“Indeed but you are wrong!” Mistress Grace replied with pretty
confusion. “It was this gentleman who gave them to me!”
At her words I stepped aside, and my lady and I were face to face.
It sent a swift pang to my heart to see the sudden change upon her
face. A moment before she had been gay and smiling, but now, at
sight of me the smile was frozen on her lips, and the hand in which
she held the flowers fell to her side again. For it may be twenty
seconds thus we stood, her eyes hardening with the pride I knew so
well. Then she spoke. “It was kind indeed of Captain Cassilis to give
us of our own,” she said coldly. “But in the house yonder there are of
flowers enough and yet to spare. And for your compassion, sir,
toward our fallen state, it may follow—these!”
She had been standing near to the wall that ran above the moat,
and now, suiting the action to her words, she tossed the roses
contemptuously into the black water below.
“Come, Gracie,” she added, turning on her heel.
In a few strides I had barred her further progress.
“One moment, madam,” I said hoarsely; “I desire a word with you.”
“The desire is not reciprocal, sir,” she answered icily. “Permit me to
pass.”
“Not until you have heard me, madam,” I cried desperately. “Even
the greatest of criminals can claim so much right.”
“In that case,” she replied with bitter irony, “your claim is
indisputable. Say on, sir. We are but two unarmed women here.”
For a moment, speechless, I stared at her, with the hot blood
flushing to my face. How this woman hated me!
“Well, sir,” she cried impatiently, “have you nothing to say? No
further insults for your prisoners?”
“Madam, madam!” I burst out passionately, “what have I done to
you that you should hate me so?”
“Hate you?” she answered slowly, gazing at me with hard, cruel
eyes. “I think that you mistake me, sir. You are too mean, too base a
thing to hate. I loathe you!”
And as with bent head, to hide the pain her words caused me, I
stood aside, without further notice, save, indeed, one pitying glance
from Mistress Grace, they passed me by, and I heard their footsteps
die away into the night.
Long I remained where they had left me, my brain a chaos, a tumult
in my breast. The song of the nightingale still quivered on the
peaceful air, and the moon rose high in the heavens, silvering the
tops of the surrounding oaks and flinging the shadows of their
twisted boles upon the grass. Yet still I lingered by the fountain, in
nowise conscious of the flight of time, whilst the very leaves,
whispering to the passing breeze, seemed but to mock me with the
echo of my lady’s words. Presently my brain grew clearer. What was
this woman to me that I should imagine that her words could wound
me? Or what concern of mine the opinion that she held of me?
’Twas but a week or two at most, and Cleeve, its fortunes, and its
mistress would but linger in my memory—a vanished dream. Or, at
the most, the vision of my lady would shine athwart the pathway of
my chequered life, like as a radiant star above my head shot
suddenly across the lighted heavens and vanished in illimitable
space.
With a grim smile and a firm step I made my way to the house.
Arrived in the hall, however, a surprise awaited me. Within the main
entrance, seeming to fill the doorway, and conversing with my lady
in low tones, stood the formidable figure of Sampson Dare.
The actual words I could not catch, but of the failure of his mission I
could readily guess, both by his dejected face and also by the fact
that at the sound of my footstep my lady turned swiftly away, and
with bent head and averted face passed up the oaken stair. When
the last rustle of her skirts had died away I slowly crossed the hall.
“Well, my friend,” I said, addressing the giant before me, not wisely,
I admit, “what news of London?”
He scowled at me for a moment without speaking, then bending
suddenly, he thrust his face within a foot of mine, and I felt his hot
breath on my cheek.
“Hark you!” he growled hoarsely, opening and shutting his great
hands, “but for my orders I would twist your neck and think no more
of it than if I killed a rat!”
“Tush, man!” I made answer, meeting his gaze firmly, though I
confess I was considerably startled at his words, for well I knew that
once within his iron grip, the man had strength to carry out his
threat, “and bring the troopers down upon the house? You should
best know in that case what consideration your mistress would be
like to meet with at their hands. No, no,” I continued coolly, twisting
my moustaches, “I give you credit for more sense than that, unless,
indeed, your body’s growth has dwarfed your brain.”
His face had fallen at my words. I suppose he saw their wisdom, but
he still regarded me with a look of vindictive hatred.
“So you shelter yourself behind the women, do you?” he said at
length. “Very well, Master Chicken-heart! Only, should it happen that
they cannot shield you——”
“In that event, my friend,” I answered, lightly tapping the butt of the
pistol in my sash, “the bigger the bulk, the easier the mark. You
understand?” And without further words I left him. But that night on
retiring to my chamber, for the first time I took the precaution of
sounding all the walls and flooring with my sword, and having
assured myself as to the non-existence of a secret entrance, I placed
my sword and pistols within easy reach of my hand; for with so
resolute a man now at her bidding, I was by no means certain that
my lady might not attempt some desperate scheme against me. Nay,
as I stood at the open window gazing across the woods to where
the moonlight fell upon the old church tower and the slumbering
hamlet below, I was minded for the moment to transfer my quarters
to the village inn. But shame at such a course kept me to my post,
and I flung myself at last upon my couch, conscious that the day
had brought to me another formidable antagonist, a relentless foe.
Under the circumstances, therefore, it is not to be wondered at that
I slept but ill, or that it was with a distinct feeling of relief I awoke
from a troubled slumber to hear the distant clock upon the church
strike five.
I sprang from my couch and set the window wide. By my faith! ’twas
a morning to put heart in any man. The fresh sea breeze stole softly
through the casement, fragrant with the scent of opening flowers.
Overhead a few fleecy clouds drifted idly beneath a dome of deepest
blue, whilst in the gardens below me, flooded by the summer
sunshine, gorgeous butterflies on painted wings flitted above the
dew-bespangled grass.
As, lost in thought, I leaned upon the sill, the fancy for a swim in the
distant, murmuring sea commended itself so strongly to my mind
that, hastily dressing, I took my sheathed sword beneath my arm
and descended the stairs. No one was as yet abroad, and all was
silent in the house when I quietly unbarred the outer door and
stepped upon the terrace. And now that I was in the open air, my
courage, that had somewhat waned the night before, returned to
me. I laughed at the fears that then had power to shake me, and I
told myself with a thrill of pardonable pride that over all this fair
domain and that which it contained I was the master, and would
prove my power. But on arrival at the cliffs my exultation suffered a
sudden check, for I had scarcely set foot upon the narrow path
when my eyes, travelling to the beach below, fell upon that which
caused me to fling myself down upon the grass and to peer
cautiously over the edge of the cliff; for in the little bay beneath, a
boat, urged by a single pair of oars, was rapidly approaching to the
land. Presently it grounded on the shingle, and in the solitary
occupant who sprang ashore I recognised the figure of Sampson
Dare. He beached the boat high and dry above the receding tide,
and, stooping, took from thence a lantern and a cloak. This done, he
straightened his back, and, unconscious of the watcher overhead,
shading his eyes, stood looking out to sea. What in the name of
mystery was the man doing here—here at this early hour, with a
lantern in his hand? Clearly, the very fact of this proved that he had
been out all night. Yet for what was he watching now? Or what did
the man expect?
And suddenly the answer came, for the thick haze that had hitherto
hung upon the surface of the water was rapidly dissipating beneath
the sun’s increasing power, and as with its disappearance the
prospect widened, away out at sea, some two miles distant, I caught
sight of the dark hull of a small vessel with a cloud of white canvas
above, looking for all the world like some great seabird riding on the
wave.
Short time had I to note her appearance, however, for even as I
gazed the helm swung round, and heading for the open sea, she
vanished in the golden mist beyond.
Not till then did the figure of the man below me turn and with a
quick step ascend the beach. And seeing this, I quickly withdrew,
concealing myself among the shrubs, where I could command both a
view of the path by which he must come and also of the house itself,
and setting myself to wait for what should follow.
And presently my patience was rewarded, for on arriving within sight
of the house, at the beginning of the sloping lawns, he came to a
sudden halt, and after closely scanning its windows, fell to pacing up
and down the grass. Nor had he long to wait before the door by
which I had left the house suddenly opened and my lady herself
stepped forth into the light.
Welcome to our website – the ideal destination for book lovers and
knowledge seekers. With a mission to inspire endlessly, we offer a
vast collection of books, ranging from classic literary works to
specialized publications, self-development books, and children's
literature. Each book is a new journey of discovery, expanding
knowledge and enriching the soul of the reade

Our website is not just a platform for buying books, but a bridge
connecting readers to the timeless values of culture and wisdom. With
an elegant, user-friendly interface and an intelligent search system,
we are committed to providing a quick and convenient shopping
experience. Additionally, our special promotions and home delivery
services ensure that you save time and fully enjoy the joy of reading.

Let us accompany you on the journey of exploring knowledge and

personal growth!

ebookgate.com