From Statistical Physics to Data-Driven

Modelling: with Applications to Quantitative

Biology Simona Cocco pdf download

https://ebookmass.com/product/from-statistical-physics-to-data-
driven-modelling-with-applications-to-quantitative-biology-
simona-cocco/

Explore and download more ebooks at ebookmass.com

We have selected some products that you may be interested in
Click the link to download now or visit ebookmass.com
for more options!.

Introduction to Quantitative Ecology: Mathematical and

Statistical Modelling for Beginners Essington

https://ebookmass.com/product/introduction-to-quantitative-ecology-
mathematical-and-statistical-modelling-for-beginners-essington/

Introduction to Quantitative Ecology: Mathematical and

Statistical Modelling for Beginners Timothy E. Essington

https://ebookmass.com/product/introduction-to-quantitative-ecology-
mathematical-and-statistical-modelling-for-beginners-timothy-e-
essington/

An Introduction to Statistical Learning with Applications

in R eBook

https://ebookmass.com/product/an-introduction-to-statistical-learning-
with-applications-in-r-ebook/

Data-driven Solutions to Transportation Problems Yinhai

Wang

https://ebookmass.com/product/data-driven-solutions-to-transportation-
problems-yinhai-wang/
Statistical Topics and Stochastic Models for Dependent
Data With Applications Vlad Stefan Barbu

https://ebookmass.com/product/statistical-topics-and-stochastic-
models-for-dependent-data-with-applications-vlad-stefan-barbu/

Statistical Process Monitoring Using Advanced Data-Driven

and Deep Learning Approaches: Theory and Practical
Applications 1st Edition Fouzi Harrou
https://ebookmass.com/product/statistical-process-monitoring-using-
advanced-data-driven-and-deep-learning-approaches-theory-and-
practical-applications-1st-edition-fouzi-harrou/

An Introduction to Data-Driven Control Systems Ali Khaki-

Sedigh

https://ebookmass.com/product/an-introduction-to-data-driven-control-
systems-ali-khaki-sedigh/

Renewable-Energy-Driven Future: Technologies, Modelling,

Applications, Sustainability and Policies Jingzheng Ren
(Eds)
https://ebookmass.com/product/renewable-energy-driven-future-
technologies-modelling-applications-sustainability-and-policies-
jingzheng-ren-eds/

Data Driven: Harnessing Data and AI to Reinvent Customer

Engagement 1st Edition Tom Chavez

https://ebookmass.com/product/data-driven-harnessing-data-and-ai-to-
reinvent-customer-engagement-1st-edition-tom-chavez/
FROM STATISTICAL PHYSICS TO DATA-DRIVEN
MODELLING
From Statistical Physics to Data-Driven
Modelling
with Applications to Quantitative Biology

Simona Cocco
Rémi Monasson
Francesco Zamponi
 Great Clarendon Street, Oxford, OX2 6DP,
United Kingdom
Oxford University Press is a department of the University of Oxford.
It furthers the University’s objective of excellence in research, scholarship,
and education by publishing worldwide. Oxford is a registered trade mark of
Oxford University Press in the UK and in certain other countries
© Simona Cocco, Rémi Monasson, and Francesco Zamponi 2022
The moral rights of the authors have been asserted
Impression: 1
All rights reserved. No part of this publication may be reproduced, stored in
a retrieval system, or transmitted, in any form or by any means, without the
prior permission in writing of Oxford University Press, or as expressly permitted
by law, by licence or under terms agreed with the appropriate reprographics
rights organization. Enquiries concerning reproduction outside the scope of the
above should be sent to the Rights Department, Oxford University Press, at the
address above
You must not circulate this work in any other form
and you must impose this same condition on any acquirer
Published in the United States of America by Oxford University Press
198 Madison Avenue, New York, NY 10016, United States of America
British Library Cataloguing in Publication Data
Data available
Library of Congress Control Number: 2022937922
ISBN 978–0–19–886474–5
DOI: 10.1093/oso/9780198864745.001.0001
Printed and bound by
CPI Group (UK) Ltd, Croydon, CR0 4YY
Links to third party websites are provided by Oxford in good faith and
for information only. Oxford disclaims any responsibility for the materials
contained in any third party website referenced in this work.
Contents

1 Introduction to Bayesian inference 1

1.1 Why Bayesian inference? 1
1.2 Notations and definitions 2
1.3 The German tank problem 4
1.4 Laplace’s birth rate problem 7
1.5 Tutorial 1: diffusion coefficient from single-particle tracking 11
2 Asymptotic inference and information 17
2.1 Asymptotic inference 17
2.2 Notions of information 23
2.3 Inference and information: the maximum entropy principle 29
2.4 Tutorial 2: entropy and information in neural spike trains 32
3 High-dimensional inference: searching for principal compo-
nents 39
3.1 Dimensional reduction and principal component analysis 39
3.2 The retarded learning phase transition 43
3.3 Tutorial 3: replay of neural activity during sleep following task
learning 52
4 Priors, regularisation, sparsity 59
4.1 Lp -norm based priors 59
4.2 Conjugate priors 64
4.3 Invariant priors 67
4.4 Tutorial 4: sparse estimation techniques for RNA alternative splic-
ing 71
5 Graphical models: from network reconstruction to Boltzmann
machines 81
5.1 Network reconstruction for multivariate Gaussian variables 81
5.2 Boltzmann machines 86
5.3 Pseudo-likelihood methods 92
5.4 Tutorial 5: inference of protein structure from sequence data 97
6 Unsupervised learning: from representations to generative
models 107
6.1 Autoencoders 107
6.2 Restricted Boltzmann machines and representations 112
6.3 Generative models 120
6.4 Learning from streaming data: principal component analysis re-
visited 125
vi Contents

6.5 Tutorial 6: online sparse principal component analysis of neural

assemblies 132
7 Supervised learning: classification with neural networks 137
7.1 The perceptron, a linear classifier 137
7.2 Case of few data: overfitting 143
7.3 Case of many data: generalisation 146
7.4 A glimpse at multi-layered networks 152
7.5 Tutorial 7: prediction of binding between PDZ proteins and pep-
tides 156
8 Time series: from Markov models to hidden Markov models 161
8.1 Markov processes and inference 161
8.2 Hidden Markov models 164
8.3 Tutorial 8: CG content variations in viral genomes 171
References 175
Index 181
Preface

Today’s science is characterised by an ever-increasing amount of data, due to instru-

mental and experimental progress in monitoring and manipulating complex systems
made of many microscopic constituents. While this tendency is true in all fields of
science, it is perhaps best illustrated in biology. The activity of neural populations,
composed of hundreds to thousands of neurons, can now be recorded in real time
and specifically perturbed, offering a unique access to the underlying circuitry and
its relationship with functional behaviour and properties. Massive sequencing has per-
mitted us to build databases of coding DNA or protein sequences from a huge variety
of organisms, and exploiting these data to extract information about the structure,
function, and evolutionary history of proteins is a major challenge. Other examples
abound in immunology, ecology, development, etc.
How can we make sense of such data, and use them to enhance our understanding
of biological, physical, chemical, and other systems? Mathematicians, statisticians,
theoretical physicists, computer scientists, computational biologists, and others have
developed sophisticated approaches over recent decades to address this question. The
primary objective of this textbook is to introduce these ideas at the crossroad between
probability theory, statistics, optimisation, statistical physics, inference, and machine
learning. The mathematical details necessary to deeply understand the methods, as
well as their conceptual implications, are provided. The second objective of this book is
to provide practical applications for these methods, which will allow students to really
assimilate the underlying ideas and techniques. The principle is that students are given
a data set, asked to write their own code based on the material seen during the theory
lectures, and analyse the data. This should correspond to a two- to three-hour tutorial.
Most of the applications we propose here are related to biology, as they were part of
a course to Master of Science students specialising in biophysics at the Ecole Normale
Supérieure. The book’s companion website1 contains all the data sets necessary for
the tutorials presented in the book. It should be clear to the reader that the tutorials
proposed here are arbitrary and merely reflect the research interests of the authors.
Many more illustrations are possible! Indeed, our website presents further applications
to “pure” physical problems, e.g. coming from atomic physics or cosmology, based on
the same theoretical methods.
Little prerequisite in statistical inference is needed to benefit from this book. We
expect the material presented here to be accessible to MSc students not only in physics,
but also in applied maths and computational biology. Readers will need basic knowl-
edge in programming (Python or some equivalent language) for the applications, and
in mathematics (functional and linear analysis, algebra, probability). One of our major
goals is that students will be able to understand the mathematics behind the meth-

1 https://github.com/StatPhys2DataDrivenModel/DDM_Book_Tutorials
viii Preface

ods, and not act as mere consumers of statistical packages. We pursue this objective
without emphasis on mathematical rigour, but with a constant effort to develop in-
tuition and show the deep connections with standard statistical physics. While the
content of the book can be thought of as a minimal background for scientists in the
contemporary data era, it is by no means exhaustive. Our objective will be truly ac-
complished if readers then actively seek to deepen their experience and knowledge by
reading advanced machine learning or statistical inference textbooks.
As mentioned above, a large part of what follows is based on the course we gave
at ENS from 2017 to 2021. We are grateful to A. Di Gioacchino, F. Aguirre-Lopez,
and all the course students for carefully reading the manuscript and signalling us the
typos or errors. We are also deeply indebted to Jean-François Allemand and Maxime
Dahan, who first thought that such a course, covering subjects not always part of the
standard curriculum in physics, would be useful, and who strongly supported us. We
dedicate the present book to the memory of Maxime, who tragically disappeared four
years ago.

Paris, January 2022.

Simona Cocco1 , Rémi Monasson1,2 and Francesco Zamponi1

1
Ecole Normale Supérieure, Université PSL & CNRS
2
Department of Physics, Ecole Polytechnique
1
Introduction to Bayesian inference

This first chapter presents basic notions of Bayesian inference, starting with the def-
initions of elementary objects in probability, and Bayes’ rule. We then discuss two
historically motivated examples of Bayesian inference, in which a single parameter has
to be inferred from data.

1.1 Why Bayesian inference?

Most systems in nature are made of small components, interacting in a complex way.
Think of sand grains in a dune, of molecules in a chemical reactor, or of neurons in a
brain area. Techniques to observe and characterise quantitatively these systems, or at
least part of them, are routinely developed by scientists and engineers, and allow one
to ask fundamental questions, see figure 1.1:
• What can we say about the future evolution of these systems? About how they will
respond to some perturbation, e.g. to a change in the environmental conditions?
Or about the behaviour of the subparts not accessible to measurements?
• What are the underlying mechanisms explaining the collective properties of these
systems? How do the small components interact together? What is the role played
by stochasticity in the observed behaviours?
The goal of Bayesian inference is to answer those questions based on observations,
which we will refer to as data in the following. In the Bayesian framework, both the

Fig. 1.1 A. A large complex system includes many components (black dots) that interact
together (arrows). B. An observer generally has access to a limited part of the system and
can measure the behaviour of the components therein, e.g. their characteristic activities over
time.
Visit https://ebookmass.com today to explore
a vast collection of ebooks across various
genres, available in popular formats like
PDF, EPUB, and MOBI, fully compatible with
all devices. Enjoy a seamless reading
experience and effortlessly download high-
quality materials in just a few simple steps.
Plus, don’t miss out on exciting offers that
let you access a wealth of knowledge at the
best prices!
2 Introduction to Bayesian inference

Fig. 1.2 Probabilistic description of models and data in the Bayesian framework. Each point
in the space of model parameters (triangle in the left panel) defines a distribution of possible
observations over the data space (shown by the ellipses). In turn, a specific data set (diamond
in right panel) corresponding to experimental measurements is compatible with a portion of
the model space: it defines a distribution over the models.

data and the system under investigation are considered to be stochastic. To be more
precise, we assume that there is a joint distribution of the data configurations and
of the defining parameters of the system (such as the sets of microscopic interactions
between the components or of external actions due to the environment), see figure 1.2.
The observations collected through an experiment can be thought of as a realisation
of the distribution of the configurations conditioned by the (unknown) parameters of
the system. The latter can thus be inferred through the study of the probability of the
parameters conditioned by the data. Bayesian inference offers a framework connecting
these two probability distributions, and allowing us in fine to characterise the system
from the data. We now introduce the definitions and notations needed to properly set
this framework.

1.2 Notations and definitions

1.2.1 Probabilities
We will denote by y a random variable taking values in a finite set of q elements:

y ∈ {a1 , a2 , · · · , aq } = A , (1.1)

and by
pi = Prob(y = ai ) = p(y = ai ) = p(y) (1.2)
the probability that y takes a given value y = ai , which will be equivalently written
in one of the above forms depending on the context.
For a two dimensional variable

y = (y1 , y2 ) ∈ A × B (1.3)

the corresponding probability will be denoted by

 Notations and definitions 3

pij = Prob[y = (ai , bj )] = p(y1 = ai , y2 = bj ) = p(y1 , y2 ) = p(y) . (1.4)

In this case, we can also define the “marginal probabilities”

X X
p(y1 ) = p(y1 , y2 ) , p(y2 ) = p(y1 , y2 ) . (1.5)
y2 ∈B y1 ∈A

By definition, y1 and y2 are independent variables if their joint probability factorises

into the product of their marginals, then

p(y1 , y2 ) = p(y1 ) × p(y2 ) . (1.6)

1.2.2 Conditional probabilities and Bayes’ rule

The “conditional probability” of y1 = ai , conditioned to y2 = bj , is

p(y1 , y2 )
p(y1 |y2 ) = . (1.7)
p(y2 )

Note that this definition makes sense only if p(y2 ) > 0, but of course if p(y2 ) = 0, then
also p(y1 , y2 ) = 0. The event y2 never happens, so the conditional probability does not
make sense. Furthermore, according to Eq. (1.7), p(y1 |y2 ) is correctly normalised:
P
y1 ∈A p(y1 , y2 )
X
p(y1 |y2 ) = =1. (1.8)
p(y2 )
y1 ∈A

Eq. (1.7) allows us to derive Bayes’ rule:

p(y1 , y2 ) p(y2 |y1 ) p(y1 )

p(y1 |y2 ) = = . (1.9)
p(y2 ) p(y2 )

This simple identity has a deep meaning for inference, which we will now discuss.
Suppose that we have an ensemble of M “data points” yi ∈ RL , which we denote
by Y = {yi }i=1,··· ,M , generated from a model with D (unknown to the observer)
“parameters”, which we denote by θ ∈ RD . We can rewrite Bayes’ rule as

p(Y |θ)p(θ)
p(θ|Y ) = . (1.10)
p(Y )

The objective of Bayesian inference is to obtain information on the parameters θ.

We will call p(θ|Y ) the “posterior distribution”, which is the object we are interested
in: it is the probability of θ conditioned to the data Y we observe. Bayes’ rule expresses
the posterior in terms of a “prior distribution” p(θ), which represents our information
on θ prior to any measurement, and of the “likelihood” p(Y |θ) of the model, i.e. the
probability that the data Y are generated by the model having defining parameters
θ. It is important to stress that the likelihood expresses, in practice, a “model of the
experiment”; such a model can be known exactly in some cases, but in most situations
it is unknown and has to be guessed, as part of the inference procedure. Last of all, p(Y )
4 Introduction to Bayesian inference

is called the “evidence”, and is expressed in terms of the prior and of the likelihood
through Z
p(Y ) = dθ p(Y |θ)p(θ) . (1.11)

Its primary role is to guarantee the normalisation of the posterior p(θ|Y ).

We will now illustrate how Bayesian inference can be applied in practice with two
examples.

1.3 The German tank problem

The German tank problem was a very practical issue that arose during World War II.
The Allies wanted to estimate the numbers of tanks available to the Germans, in order
to estimate the number of units needed for the Normandy landings. Some information
was available to the Allies. In fact, during previous battles, some German tanks were
destroyed or captured and the Allies then knew their serial numbers, i.e. a progressive
number assigned to the tanks as they were produced in the factories. The problem can
be formalised as follows [1].

1.3.1 Bayes’ rule

The available data are a sequence of integer numbers

1 ≤ y1 < y2 < y3 < · · · < yM ≤ N , (1.12)

where N is the (unknown) total number of tanks available to the Germans, and
Y = {y1 , · · · , yM } ∈ NM are the (known) factory numbers of the M destroyed tanks.
Note that, obviously, M ≤ yM ≤ N . Our goal is to infer N given Y .
In order to use Bayes’ rule, we first have to make an assumption about the prior
knowledge p(N ). For simplicity, we will assume p(N ) to be uniform in the interval
[1, Nmax ], p(N ) = 1/Nmax . A value of Nmax is easily estimated in practical applications,
but for convenience we can also take the limit Nmax → ∞ assuming p(N ) to be
a constant for all N . Note that in this limit p(N ) is not normalisable (it is called
“an improper prior”), but as we will see this may not be a serious problem: if M is
sufficiently large, the normalisation of the posterior probability is guaranteed by the
likelihood, and the limit Nmax → ∞ is well defined.
The second step consists in proposing a model of the observations and in computing
the associated likelihood. We will make the simplest assumption that the destroyed
tanks are randomly and uniformly sampled from the total number of available tanks.
Note that this assumption could be incorrect in practical applications, as for example
the Germans could have decided to send to the front the oldest tanks first, which
would bias the yi towards smaller values. The choice of the likelihood expresses our
modelling of the data generation process. If the true model is unknown, we have to
make a guess, which
has an impact on our inference result.
N
There are M ways to choose M ordered numbers y1 < y2 < · · · < yM in [1, N ].
Assuming that all these choices have equal probability, the likelihood of a set Y given
N is
 The German tank problem 5

1
p(Y |N ) = N

1(1 ≤ y1 < y2 < · · · < yM ≤ N ) , (1.13)
M

because all the ordered M -uples are equally probable. Here, 1(c) denotes the “indicator
function” of a condition c, which is one if c is satisfied, and zero otherwise.
Given the prior and the likelihood, using Bayes’ rule, we have

p(Y |N )p(N )
p(N |Y ) = P 0 0
, (1.14)
N 0 p(Y |N )p(N )

which for an improper uniform prior reduces to

N −1


p(Y |N ) M 1(N ≥ yM )
p(N |Y ) = P = = p(N |yM ; M ) . (1.15)
p(Y |N 0) N 0 −1
P

N 0
0
N ≥yM M

Note that, because Y is now given and N is the variable, the condition N ≥ yM >
yM −1 > · · · > y1 ≥ 1 is equivalent to N ≥ yM , and as a result the posterior probability
of N depends on the number of data, M (here considered as a fixed constant), and on
the largest observed value, yM , only. It can be shown [1] that the denominator of the
posterior is
X N 0 −1 yM −1 yM
= , (1.16)
0
M M M −1
N ≥yM

leading to the final expression for the posterior,

N −1


M 1(N ≥ yM )
p(N |yM ; M ) = . (1.17)
yM −1 yM


M M −1

Note that for large N , we have p(Y |N ) ∝ N −M . Therefore, if M = 1, the posterior in

Eq. (1.15) is not normalisable if the prior is uniform, and correspondingly Eq. (1.17) is
not well defined, see the M −1 term in the denominator. Hence, if only one observation
is available, some proper prior distribution p(N ) is needed for the Bayesian posterior
to make sense. Conversely, if M ≥ 2, the posterior is normalisable thanks to the fast-
decaying likelihood, and Eq. (1.17) is well defined, so the use of an improper prior is
acceptable when at least two observations are available.

1.3.2 Analysis of the posterior

Having computed the posterior probability, we can deduce information on N . First of
all, we note that p(N |M, yM ) obviously vanishes for N < yM . For large N , we have
p(N |M, yM ) ∝ N −M , and recall that we need M ≥ 2 for the posterior to be well
defined. We can then compute:
1. The typical value of N , i.e. the value N ∗ that maximises the posterior. As the
posterior distribution is a monotonically decreasing function of N ≥ yM we have
N ∗ = yM .
6 Introduction to Bayesian inference

0.10

Posterior distribution P(NjM; yM )

0.08

­ ®
0.06 N
yM

0.04

0.02

» N ¡M
0.00
75 100 125 150 175 200
N

Fig. 1.3 Illustration of the posterior probability for the German tank problem, for M = 10
and yM = 100. The dashed vertical lines locate the typical (= yM ) and average values of N .

2. The average value of N ,

X M −1
hN i = N p(N |yM ; M ) = (yM − 1) . (1.18)
M −2
N

Note that, for large M , we get

yM − M
hN i ∼ yM + + ··· , (1.19)
M

i.e. hN i is only slightly larger than yM . This formula has a simple interpretation.
For large M , we can assume that the yi are equally spaced in the interval [1, yM ].
The average spacing is then ∆y = (yM − M )/M . This formula tells us that
hN i = yM + ∆y is predicted to be equal to the next observation.
3. The variance of N ,

2 2 M −1
σN = N 2 − hN i = (yM − 1)(yM − M + 1) . (1.20)
(M − 2)2 (M − 3)

The variance gives us an estimate of the quality of the prediction for N . Note that
for large M , assuming yM ∝ M , we get σN /hN i ∝ 1/M , hence the relative stan-
dard deviation decreases with the number of observations. Notice that Eq. (1.18)
for the mean and Eq. (1.20) for the variance make sense only if, respectively,
M > 2 and M > 3. Again, we see a consequence of the use of an improper prior:
moments of order up to K are defined only if the number M of observations is
larger than K + 2.
For example, one can take M = 10 observations, and y10 = 100. A plot of the posterior
is given in figure 1.3. Then, hN i = 111.4, and the standard deviation is σN = 13.5.
One can also ask what the probability is that N is larger than some threshold, which
 Laplace’s birth rate problem 7

is arguably the most important piece of information if you are planning the Normandy
landings. With the above choices, one finds
p(N > 150|y10 = 100) = 2.2 · 10−2 ,
p(N > 200|y10 = 100) = 1.5 · 10−3 , (1.21)
−4
p(N > 250|y10 = 100) = 2 · 10 .
More generally,
 −(M −1)
Nlower bound
p(N > Nlower bound |yM ) ∝ . (1.22)
hN i

1.4 Laplace’s birth rate problem

We now describe a much older problem, which was studied by Laplace at the end of the
eighteenth century. It is today well known that the number of boys and girls is slightly
different at birth, although the biological origin of this fact is not fully understood. At
his time, Laplace had access to the historical record of the number of boys and girls
born in Paris from 1745 to 1770: out of a total number M = 493472 of observations,
the number of girls was y = 241945, while the number of boys was M − y = 251527.
This observation suggests a slightly lower probability of giving birth to a girl, but could
this be explained by random fluctuations due to the limited number of observations?
1.4.1 Posterior distribution for the birth rate
Laplace wanted to determine the probability of a newborn baby to be a girl, which is
a single parameter, θ ∈ [0, 1]. To do so, we first need to introduce a prior probability1
over θ; let us assume p(θ) = 1 to be constant in the interval θ ∈ [0, 1] (and zero
otherwise), i.e. no prior information. To obtain the likelihood, we can assume that
each birth is an independent event, in which case the distribution of y conditioned to
θ is a binomial,  
M y
p(y|θ; M ) = θ (1 − θ)M −y , (1.23)
y
where M is a fixed, known integer. Note that this is a very simple model, because in
principle there could be correlations between births (e.g. for brothers, twins, etc.). We
will discuss later how to assess the quality of a given model. For now we use Bayes’
rule to obtain the posterior probability density over the birth rate,
p(y|θ; M )p(θ) θy (1 − θ)M −y
p(θ|y; M ) = = R1 . (1.24)
p(y; M ) dθ 0 (θ 0 )y (1 − θ 0 )M −y
0

Eq. (1.24) is a particular case of the “beta distribution”,

θα−1 (1 − θ)β−1
Beta(θ; α, β) = ,
B(α, β)
Z 1
(1.25)
Γ(α)Γ(β)
B(α, β) = dθ θα−1 (1 − θ)β−1 = ,
0 Γ(α + β)
1 Note that this is now a probability density, because θ is a continuous variable.
8 Introduction to Bayesian inference

R∞
with α = y + 1 and β = M − y + 1, and Γ(x) = 0 dθ θx−1 e−θ stands for the Gamma
function. The following properties of the beta distribution are known:
• The typical value of θ, i.e. which maximizes Beta, is θ∗ = α−1
α+β−2 .
α
• The average value is hθi = α+β .
αβ
• The variance is σθ2 = (α+β)2 (α+β+1) .
A first simple question is: what would be the distribution of θ if there were no girls
observed? In that case, we would have y = 0 and

p(θ|0; M ) = (M + 1)(1 − θ)M . (1.26)

The most likely value would then be θ∗ = 0, but the average value would be hθi = 1/M .
This result is interesting: observing no girls among M births should not be necessarily
interpreted that their birth rate θ is really equal to zero, but rather that θ is likely
to be smaller than 1/M , as the expected number of events to be observed to see one
girl is 1/θ. From this point of view, it is more reasonable to estimate that θ ∼ 1/M ,
than θ = 0.
In the case of Laplace’s data, from the observed numbers, we obtain

θ∗ = 0.490291 , hθi = 0.490291 , σθ = 0.000712 . (1.27)

The possibility that θ = 0.5 seems then excluded, because θ∗ ∼ hθi differs from 0.5 by
much more than the standard deviation,

|θ∗ − 0.5|  σθ , | hθi − 0.5|  σθ , (1.28)

but we would like to quantify more precisely the probability that, yet, the “true” value
of θ is equal to, or larger than 0.5.

1.4.2 Extreme events and Laplace’s method

The analysis above suggests that the birth rate of girls is smaller than 0.5. To be
more quantitative let us estimate the tiny probability that the observations (number
of births) are yet compatible with θ > 0.5. The posterior probability that θ > 0.5 is
given by Z 1
p(θ > 0.5|y; M ) = dθ p(θ|y; M ) . (1.29)
0.5
Unfortunately this integral cannot be computed analytically, but this is precisely why
Laplace invented his method for the asymptotic estimation of integrals! Expressing
the posterior in terms of θ∗ = y/M instead of y, he observed that
∗ ∗
p(θ|θ∗ ; M ) ∝ θM θ (1 − θ)M (1−θ )
= e−M fθ∗ (θ) , (1.30)

where
fθ∗ (θ) = −θ∗ log θ − (1 − θ∗ ) log(1 − θ) . (1.31)
∗
A plot of fθ∗ (θ) for the value of θ that corresponds to Laplace’s observations is given
in figure 1.4. fθ∗ (θ) has a minimum when its argument θ reaches the typical value θ∗ .
 Laplace’s birth rate problem 9

2.2

2.0

1.8

1.6
µ¤

fµ ¤ (µ)
1.4

1.2

1.0

0.8

0.0 0.2 0.4 0.6 0.8 1.0

µ

Fig. 1.4 Illustration of the posterior minus log-probability fθ∗ (θ) = − log p(θ|θ∗ , M )/M for
Laplace’s birth rate problem.

Because of the factor M in the exponent, for large M , a minimum of fθ∗ (θ) induces a
very sharp maximum in p(θ|θ∗ ; M ).
We can use this property to compute the normalisation factor at the denominator
in Eq. (1.24). Expanding fθ∗ (θ) in the vicinity of θ = θ∗ , we obtain

1
fθ∗ (θ) = fθ∗ (θ∗ ) + (θ − θ∗ )2 fθ00∗ (θ∗ ) + O((θ − θ∗ )3 ) , (1.32)
2
with
1
fθ00∗ (θ∗ ) = . (1.33)
θ∗ (1 − θ∗ )
In other words, next to its peak value the posterior distribution is roughly Gaussian,
and this statement is true for all θ away from θ∗ by deviations of the order of M −1/2 .
This property helps us compute the normalisation integral,
Z 1 Z 1 ∗ ∗ 2
1
dθ e−M fθ∗ (θ) ∼ dθ e−M [fθ∗ (θ )+ 2θ∗ (1−θ∗ ) (θ−θ ) ]
0 0 (1.34)
∗ p
' e−M fθ∗ (θ ) × 2πθ∗ (1 − θ∗ )/M ,

because we can extend the Gaussian integration interval to the whole real line without
affecting the dominant order in M . We deduce the expression for the normalised
posterior density of birth rates,
∗
e−M [fθ∗ (θ)−fθ∗ (θ )]
p(θ|θ∗ ; M ) ∼ p . (1.35)
2πθ∗ (1 − θ∗ )/M

In order to compute the integral in Eq. (1.29), we need to study the regime where
θ − θ∗ is of order 1, i.e. a “large deviation” of θ. For this, we use Eq. (1.35) and expand
it around θ = 0.5, i.e.
10 Introduction to Bayesian inference

1 ∗
e−M [fθ∗ (θ)−fθ∗ (θ )]
Z
p(θ > 0.5|θ∗ ; M ) = dθ p
0.5 2πθ∗ (1 − θ∗ )/M
∗ Z 1
eM fθ∗ (θ ) 0
=p dθe−M [fθ∗ (0.5)+fθ∗ (0.5)(θ−0.5)+··· ] (1.36)
∗ ∗
2πθ (1 − θ )/M 0.5
∗
e−M [fθ∗ (0.5)−fθ∗ (θ )]
∼ p .
fθ0 ∗ (0.5) 2πθ∗ (1 − θ∗ )M

With Laplace’s data, this expression for the posterior probability that θ ≥ 0.5 could
be evaluated to give
p(θ > 0.5|θ∗ ; M ) ∼ 1.15 · 10−42 , (1.37)
which provides a convincing statistical proof that, indeed, θ < 0.5.
We conclude this discussion with three remarks:
1. The above calculation shows that the posterior probability that θ deviates from its
typical value θ∗ decays exponentially with the number of available observations,
∗
+a)−fθ∗ (θ ∗ )]
p(θ − θ∗ > a|θ∗ ; M ) ∼ e−M [fθ∗ (θ . (1.38)

A more general discussion of the large M limit will be given in Chapter 2.

2. The maximum of the function f , fθ∗ (θ∗ ), is equal to the entropy of a binary
variable y = 0, 1 with corresponding probabilities 1 − θ∗ , θ∗ . We will introduce
the notion of entropy and discuss the general connection with inference in the
so-called asymptotic regime (namely, for extremely large M ) in the next chapter.
3. Note that Eq. (1.36) is strongly reminiscent of a Boltzmann distribution. Assume
that a thermal particle moves on a one-dimensional line and feels a potential
U (x) depending on its position x. Let us call x∗ the absolute minimum of U . At
low temperature T , the density of probability that the particle is in x reads (the
Boltzmann constant kB is set to unity here):
∗
e−(U (x)−U (x ))/T
ρ(x; T ) ∼ p , (1.39)
2π U 00 (x∗ )/T

where the denominator comes from the integration over the harmonic fluctuations
of the particle around the bottom of the potential. We see that the above expres-
sion is identical to Eq. (1.36) upon the substitutions θ∗ → x∗ , 0.5 → x, fθ∗ →
U, M → 1/T . Not surprisingly, having more observations reduces the uncertainty
about θ and thus the effective temperature.
 Tutorial 1: diffusion coefficient from single-particle tracking 11

1.5 Tutorial 1: diffusion coefficient from single-particle tracking

Characterising the motion of biomolecules (proteins, RNA, etc.) or complexes (vesi-
cles, etc.) inside the cell is fundamental to the understanding of many biological pro-
cesses [2]. Optical imaging techniques now allow for the tracking of single particles in
real time [3]. The goal of this tutorial is to understand how the diffusion coefficient can
be reconstructed from the recordings of the trajectory of a particle, and how the ac-
curacy of the inference is affected by the number of data points (recording length) [4].
The diffusion coefficient depends on the diffusive properties of the environment and
on the size of the object. Supposing that the data are obtained in water, from the
diffusion coefficient reconstructed from the data the characteristic size of the diffusing
object will then be extracted, and a connection with characteristic biological sizes will
be made.

1.5.1 Problem
We consider a particle undergoing diffusive motion in the plane, with position r(t) =
(x(t), y(t)) at time t. The diffusion coefficient (supposed to be isotropic) is denoted by
D, and we assume that the average velocity vanishes. Measurements give access to the
positions (xi , yi ) of the particles at times ti , where i is a positive integer running from
1 to M .

Data:
Several trajectories of the particle can be downloaded from the book webpage2 , see
tutorial 1 repository. Each file contains a three-column array (ti , xi , yi ), where ti is the
time, xi and yi are the measured coordinates of the particle, and i is the measurement
index, running from 1 to M . The unit of time is seconds and displacements are in µm.

Questions:

1. Write a script to read the data. Start by the file dataN1000d2.5.dat, and plot the
trajectories in the (x, y) plane. What are
p their characteristics? How do they fill
the space? Plot the displacement ri = x2i + yi2 as a function of time. Write the
random-walk relation between displacement and time in two dimensions, defining
the diffusion coefficient D. Give a rough estimate of the diffusion coefficient from
the data.
2. Write down the probability density p({xi , yi }|D; {ti }) of the time series {xi , yi }i=1,...,M
given D, and considering the measurement times as known, fixed parameters. De-
duce, using Bayes’ rule, the posterior probability density for the diffusion coeffi-
cient, p(D|{xi , yi }; {ti }).
3. Calculate analytically the most likely value of the diffusion coefficient, its average
value, and its variance, assuming a uniform prior on D.
4. Plot the posterior distribution of D obtained from the data. Compute, for the
given datasets, the values of the mean and of the variance of D, and its most
likely value. Compare the results obtained with different number M of measures.

2 https://github.com/StatPhys2DataDrivenModel/DDM_Book_Tutorials
Visit https://ebookmass.com today to explore
a vast collection of ebooks across various
genres, available in popular formats like
PDF, EPUB, and MOBI, fully compatible with
all devices. Enjoy a seamless reading
experience and effortlessly download high-
quality materials in just a few simple steps.
Plus, don’t miss out on exciting offers that
let you access a wealth of knowledge at the
best prices!
Exploring the Variety of Random
Documents with Different Content
 national, 38, 39, 41, 100;
possessed by U. S. and by States, 47;
State distinguished from federal, 51;
as to commerce, 63;
of Congress over outlying possessions, 161–163
State (in the Union), quasi-sovereign, 2;
Legislature, 2;
sovereignty of, 6–9, 55, 100;
supremacy of, 34;
powers of Legislatures derivative, 35, 36
State, powers of legislature extinguished by Congress, 36;
implied powers of, 36;
power to punish counterfeiting, 41, 42;
police power of, 43, 44, 45, 79, 81, 91, 92, 98, 121, 205, 206,
207, 208;
meaning of “State” in the Constitution, 47, 48, 141;
and U. S. possess sovereignty, 47;
subdivisions of, 48;
system of State government distinguished from federal, 51;
power of, over commerce, 64, 67;
limitation of jurisdiction of, 82;
may be petitioned, not suable, 114, 115;
what constitutes a republican form of, 128;
suability, 140;
the word “States” in the Constitution, 141;
jurisdiction of, determined, 147;
principle of relation of State to State, 147, 148;
law of in federal courts, 148;
the States mutually foreign to one another, 148;
rights of citizens of, 149;
admission of a, 156, 157;
the States indestructible, 158;
new States, 173;
limitation of power of, 176–181;
the States as limitations on the U. S., 179, 180;
 appoints presidential electors, 179;
subdivision of, 180;
guaranteed a republican form of government, 180;
States and amendment of the Constitution, 180, 181;
citizenship, 213
Suffrage, 223
Supreme Court of the U. S., Marshall’s decisions, 28;
principle of interpretation, 31, 34, 39, 186;
on boundary between the federal and State systems, 51;
part of the judicial department, 56;
powers not delegated, 59;
has not defined power over commerce, 63;
nature of power of U. S. over commerce, 65, 76, 77;
decisions on Anti-Trust Act, 83, 84;
on obligation of contracts, 93;
adequacy of its authority, 106;
on executive and ministerial powers, 112;
judicial power of, 113, 114 et seq.;
jurisdiction, original and appellate, 119–136;
on the war power, 127;
nature of jurisdiction, 129–142;
determines constitutional law, 133–135;
jurisdiction under the Judiciary Act, 141;
relation to State tribunals, 144, 145;
as to republican form of government, 155, 156;
decision of as to power of Congress over Territories and
possessions, 160–163, 183, 184, 201;
on delegated powers, 175;
jurisdiction when a State is a party, 178;
function of the Judiciary, 185;
power of U. S. to acquire territory, 193;
power vested in, 194;
on the nature of American institutions, 210, 211;
on citizens’ rights, 214;
on “due process of law,” 220;
 on the Fifteenth Amendment, 222, 223
“Sweeping Clause,” 26

Tax, export, 22, 52;

essentials of a good, 52, 60, 61;
income, 58 (and see under Amendment);
exemptions, 60;
direct, indirect, 61
Taxation, law of, 51–62;
by a State, 52–56;
national, 54;
power of U. S. over, 65, 66
Tender, legal, 38, 39
Territory, when sovereign, 47;
power of Congress over, 48, 141, 183;
the law of, and of territories, 146–163;
becoming a State, 157, 158;
governed by Congress, 159, 160, 162, 163
Texas, law of, regulating commerce, 80
Tickets, lottery, 43, 83, 85
Treason, 172
Treaties, 104;
in the Senate, 180

United States, supreme law of, 2 et seq.;

laws of, by whom made, 2;
guarantees republican form of government, 6;
sovereignty of, 8–13;
organization of, reflects popular will, 18;
 governmental functions of, 19, 20;
powers of, 20 et seq.;
admiralty, jurisdiction of, 45, 46;
war power of, 46;
and States possess sovereignty, 47;
civil system of, distinguished from State, 51;
powers of, as to States, 54–57;
power over commerce, 63–88;
the peace of, 85;
police power of, 95;
citizenship, 214, 216, 217

Veto, 2
Vice-President, 108

Waite, Chief Justice, quoted, 92

Waters, navigable, 46
Webster, Daniel, his definition of law, 205
Weights, 36, 37
Wilson, James, on the Constitution a Bill of Rights, 187, 188
 FOOTNOTES
1
Art. vi., 2, 3, and Preamble.
2
Art. i., 7: 2.
3
The Supreme Court of Mississippi in Sproule v.
Fredericks, 69 Miss. 898 (1892), decided that the
Constitutional Convention of that State (1890) “wielded
the powers of sovereignty specially delegated to it, for
the purpose and the occasion, by the whole electoral
body, for the good of the whole Commonwealth.” The
Supreme Court of Pennsylvania in Wells v. Bain, 75 Pa.
St. 39 (1874), decided that the Convention of 1872 was
“not a co-ordinate branch of the government,” and
possessed only “delegated powers.” The Supreme Court
of the United States, through Marshall, C. J., decided in
McCulloch v. Maryland, 4 Wheaton, 316 (1819), that the
Constitution which came from the hands of the Federal
Convention of 1787 “was a mere proposal, without
obligation, or pretensions to it. By the Convention, by
Congress, and by the State Legislatures, the instrument
was submitted to the people. They acted upon it, in the
only manner in which they can act safely, effectively,
and wisely, on such a subject, by assembling in
convention. It is true they assembled in their several
States; and where else should they have assembled? No
political dreamer was ever wild enough to think of
 breaking down the lines which separate the States, and
of compounding the American people into one common
mass. Of consequence when they act, they act in their
States. But the measures they adopt do not, on that
account, cease to be the measures of the people
themselves, or become the measures of the State
governments. From these conventions the Constitution
derives its whole authority. The government proceeds
directly from the people; is “ordained and established”
in the name of the people; and is declared to be
ordained, “in order to@ form a more perfect union,
establish justice, insure domestic tranquillity, and secure
the blessings of liberty to themselves and to their
posterity.” The assent of the States in their sovereign
capacity is implied in calling a convention, and thus
submitting that instrument to the people. But the
people were at perfect liberty to accept or reject it; and
their act was final. It required not the affirmance, and
could not be negatived by the State governments. The
Constitution when thus adopted was of complete
obligation, and bound the State sovereignties.” The
character of the Constitution, its purport and principles,
is examined in Martin v. Hunter’s Lessee, 1 Wheaton,
304 (1816). Decision by Story, J.
4
No. xxxix.
5
Art. v.
6
Constitution (1780 to date) Pt. I. Art. iv. The words
“substitutes and agents” may be considered equivalent
to the modern words “administrative officers.”
7
Art. iv., 4.
8
McCulloch v. Maryland, note, supra.
9
McCulloch v. Maryland, note, supra.
10
Idem. (The language of the Court slightly paraphrased.)
11
Art. v.
12
The Federalist, No. lxii.
13
McCulloch v. Maryland.
14
Art. i., 8: 1; but see Amendment XVI.
15
McCulloch v. Maryland.
16
Articles i., 8: 5; ii., 2: 2; i., 10: 3; i., 8: 2.
17
The License Cases, 5 Howard, 504 (1846); Kimmish v.
Ball, 129 U. S., 217 (1889); Cook v. Marshall Company,
196 U. S., 261.
18
Discussed at length in the chapters on State Comity,
and Commerce.
19
Art. iv. (and preceding note).
20
See also Chapters XII and XIII.
21
Barron v. Baltimore, 7 Peters, 243 (1833).
22
No. xxxix.
23
Cohens v. Virginia, 6 Wheaton, 382 (1821). Madison’s
thought is incorporated into Weston et al. v. the City of
Charleston, 2 Peters, 466 (1829.)
24
The Federalist, No. lxxxii.
25
Gallatin’s Writings, i., 11.
26
Sustained by Downes v. Bidwell, 182 U. S., 244 (1901).
27
The American Insurance Company v. Canter, 1 Peters,
511 (1828).
28
Compare the Preamble. The entire discussion in The
Federalist is of the conformity of the Constitution to a
republican government and of the necessity of
governmental powers adequate to governmental
purposes.
29
Art. x.
30
Marbury v. Madison, 1 Cranch, 176 (1803).
31
Every question in constitutional law, in the United
States, sooner or later leads back to a question of
sovereignty. What that sovereignty is can be known
only by its operation,—that is, by political experience.
What powers are delegated by the Constitution is the
question answered (at least in part) by courts of law
and legislatures, by publicists and by the actual
administration of government. Widely divergent
interpretations of that sovereignty and that law have
been held throughout our history as a nation. These
divergent opinions are recorded in the Debates during
the formation and ratification of the Constitution; in the
discussions incident to the Kentucky and Virginia
Resolutions of 1798; in the discussions relating to
Nullification, in 1833; again in 1860 and immediately
prior; and in various decisions of the Supreme Court of
the United States. Chief Justice Marshall’s decisions
(some thirty-six in number), the opinion of that Court in
his time, remain the classic interpretation of national
sovereignty. The Federalist remains the classic
contemporaneous interpretation of the Constitution.
The issue involved is, fundamentally, one of
functions, and is viewed at different times with different
understandings. As a practical question, it is one of
jurisdiction as legally understood, but as a question of
service as politically understood. Here enter many and
 diverse factors as morals, industry, communal interest,
public safety, social needs, and the like. Questions
growing out of these are not and cannot be decided
finally by any generation. Each generation interprets
these factors. Thus constitutional interpretation
becomes, not a fixed quantity, but an adjustment to
reason and necessity. Prudence dictates that
interpretation be conservative. The constitutional and
political history of America must be read along with its
constitutional law. In addition to cases already cited in
this chapter, the following may advantageously be read,
though each contains matter of special application to
other aspects of the subject: Ex parte Siebold, 100
U. S., 371 (1879); The Civil Rights Cases, 109 U. S., 3
(1883); Rogers v. Alabama, 192 U. S., 226 (1904).
32
Marbury v. Madison, 1 Cranch, 177.
33
Art. i., 1.
34
Art. vi., 2.
35
A typical formulation in Massachusetts, (1780) Pt. I.,
xxx. Discussed in Taylor v. Place, 4 R. I., 324 (1856.)
36
McCulloch v. Maryland, 4 Wheaton, 316.
37
Art. i., v.; Amendment XII.
38
“The Sources and Authorship of the Constitution,” in the
author’s Constitutional History of the United States, iii.,
464–515.
39
Art. i., 8: 1.
40
Art. i., 9: 2.
41
Id., 3.
42
Id., 5.
43
Id., 6.
44
Id., 7.
45
See the Chapters on The Law of Limitations, and The
Law of Fundamental Rights.
46
Pennsylvania, 1873, Art. i., 26.
47
Thus annulling Art. i., 2: 3.
48
It will be profitable to compare this amendment with
the doctrine laid down in Marbury v. Madison, 1 Cranch,
137. See also The Reconciliation of Government and
Liberty, J. W. Burgess (1915).
49
The fundamental principle of judicial interpretation is
laid down in Marbury v. Madison; the principle is
examined in the Chapter on The Law of Judicial Power.
50
Art. iii.
51
This point is elaborated and examined by the Supreme
Court in the decision declaring the Civil Rights Bill of
April 9, 1866, unconstitutional. Civil Rights Cases, 109
U. S., 3 (1883). The doctrine annunciated is that
Congress has no power to legislate generally upon
subjects, power over which is reserved to the States by
the Tenth Amendment.
52
See authorities at close of preceding Chapter; also
Chapter XI.
53
In this connection as to the President see Field v. Clark,
143 U. S., 649 (1892).
54
Art. i., 8: 18.
55
Preamble. As to “necessary and proper,” see United
States v. Fisher, 2 Cranch, 396; McCulloch v. Maryland,
4 Wheaton, 421.
56
U. S. v. Fisher, supra.
57
The great opinions interpretative of the Constitution
have each their historical setting. Illustration of this is
given in the annotated editions of Marshall’s decisions,
e. g., J. P. Cotton’s edition, 2 vols. 1905.
58
For a detailed history of the first fifteen amendments
see the author’s Constitutional History of the United
States; the social and political history from 1789 to
1870 are related, respectively, by John Bach McMaster
in his History of the People of the United States, and by
James Schouler in his History of the United States. J. F.
Rhodes in his History of the United States from the
Compromise of 1850, 7 vols. (1850–1877), gives the
history of congressional legislation and of judicial
interpretation during the period. Much of the history
relevant to the great decisions of the Court is given in
the decisions.
59
McCulloch v. Maryland, 4 Wheaton, 316 (1819). Many
later decisions apply this principle.
For an examination of the character and scope of the
Legislative Department, see
Taylor v. Place, 4 R. I., 324 (1856); Dalby v. Wolf, 14
Iowa, 228 (1862); Stone v. City of Charleston, 114
Mass., 214 (1873); Barrno v. Baltimore, 7 Peters, 243
(1833); Calder v. Bull, 3 Dallas, 386 (1798).
The powers of Congress over taxation, commerce,
the currency, war, territories, outlying possessions, etc.,
 are particularly examined under appropriate headings in
later chapters.
In addition to cases cited in the present Chapter, and
to the above, and relating to the powers of Congress,
see Gibbons v. Ogden, 9 Wheaton, 1 (1824); The
Mayor, etc., of the City of New York v. Miln, 11 Peters,
102 (1837); The License Cases, 5 Howard, 504 (1847);
Sinnot v. Davenport, 22 Howard, 227 (1859); Gilman v.
Philadelphia, 3 Wallace, 713 (1865); Henderson et al.
Mayor of the City of New York, et al. Commissioners of
Immigration v. North German Lloyd, 92 U. S., 259
(1875); Hull v. De Cuir, 95 U. S., 485 (1877); Pensacola
Telegraph Co. v. Western Union Telegraph Co., 96 U. S.,
1 (1877); County of Mobile v. Kimball, 102 U. S., 691
(1880); Williamette Iron Bridge Co. v. Hatch, 125 U. S.,
1 (1888).
The best brief treatise on the legislative in America is
American Legislatures and Legislative Methods, by Paul
S. Reinsch, 1907; the most exhaustive and authoritative
treatise is Constitutional Limitations, by Thomas M.
Cooley. The general powers of Congress are discussed
by Justice Story in his Commentaries on the
Constitution, and by Chancellor Kent in his
Commentaries on American Law.
See also the authorities cited in the present work on
The Law of the Judicial Power.
60
No. xvi.
61
No. xxxi.
62
McCulloch v. Maryland, 4 Wheaton, 316 (1819).
63
The Federalist, xxxii.
64
Idem. and Weaver v. Fegely, 29 Pennsylvania State, 27
(1857).
65
Moore v. Houston, 3 S. and R. (Pa.), 179, and the cases
cited in Weaver v. Fegely.
66
See cases as under preceding note.
67
Baldwin v. Hale, 1 Wallace, 223 (1863).
68
Baldwin v. Hale, supra.
69
Juilliard v. Greenman, 110 U. S., 421 (1884), citing and
quoting McCulloch v. Maryland.
70
Art. i., 8: 1, 2, 5.
71
Distinctions as to United States notes, coin, currency,
legal tender, etc., are brought out in Juilliard v.
Greenman, supra; Hepburn v. Griswold, 8 Wallace, 603
(1869); Parker v. Davis, 12 Wallace, 79 (1871);
Trebilcock v. Wilson, 12 Wallace, 687 (1871).
72
Knox v. Lee, Parker v. Davis, 12 Wallace, 554 (1871).
73
An account of the struggles of political parties, and of
the successive decisions of the Supreme Court as to
Legal Tender Acts belongs to the history of the law
rather than to a statement of the essentials of present
constitutional law. Accounts of this struggle, available in
histories of the United States, may be compared with
Justice Stephen J. Field’s account in J. Norton
Pomeroy’s Some Account of the Work of Stephen J.
Field as a Legislator, State Judge, and Justice of the
Supreme Court of the United States (1881), (Edition by
George C. Gorham, 1895) pp. 65–86. Mr. Justice Field’s
dissenting opinions from the decisions of the Supreme
Court which sustain the constitutionality of the Acts are
 based largely on his conception of the principle of the
obligation of a contract as contained in the Constitution
respecting “gold and silver coin.” For the history of the
Acts, the decision of the Court invalidating them (1869);
the increase of the membership of the Court (1870);
the reversal of the earlier decisions (1871), and the final
decision in Juilliard v. Greenman (1883), consult
Rhodes, vi., 268, 270–273, and Note.
74
Art. i., 10: 1.
75
Briscoe v. Bank of Kentucky, 11 Peters, 257 (1837).
76
Darrington v. The Bank of Alabama, 13; Howard, 12
Briscoe v. Bank of Kentucky, supra.
77
Art. i., 8: 6.
78
Id. 5, 10: 1.
79
United States v. Marigold, 9 Howard, 560 (1849); Fox v.
Ohio, 5 Howard, 410.
80
In re Rapier, 143 U. S., 110 (1892); Battle v. U. S., 209
U. S., 36.
81
Wheaton v. Peters, 8 Peters, 591 (1834).
82
Vanini et al. v. Paine et al. 1 Harr. (Del.) 65, quoted in
Patterson v. Kentucky, 97 U.S., 501 (1878).
83
Id. See also Herdic v. Roessler, 109 New York, 127
(1888); Hill and Co. Lmtd. v. Hoover, 220 U.S., 329.
“Where a suit is brought on a contract of which a patent
is the subject matter, either to enforce such contract, or
to annul it, the case arises on the contract and not
under the patent laws.” Hartell v. Tilghman, 99 U.S.,
558. See also Dale Tile Mfg. Co. v. Hyatt, 125 U.S., 46
(1888).
84
Rex v. Dawson, 5 State Trials.
85
U. S. v. Smith, 5 Wheaton, 153 (1820).
86
Art. iii., 2: 1.
87
U. S. v. Rodgers, 150 U. S., 249 (1893).
88
Art. i., 8: 11; The Prize Cases, 2 Black, 635 (1862).
89
Brown v. U. S., 8 Cranch, 110; American Insurance Co.
v. Canter, 1 Peters, 511; Lamar ex. v. Browne et al., 92
U. S., 187; Mormon Church v. U. S., 136 U. S., 1.
90
Hepburn v. Ellzey, 2 Cranch, 445 (1804).
91
Barron v. Baltimore, 7 Peters, 243 (1833).
92
Metropolitan R. R. Co. v. District of Columbia, 132 U. S.,
1 (1889).
93
Ft. Leavenworth R. R. Co. v. Lowe, 114 U. S., 525
(1885).
94
Art. iv., 3.
95
Dorr v. U. S., 195 U. S., 138 (1904); Hawaii v. Mankichi,
190 U. S., 197 (1903); Dooley v. U. S., 183 U. S., 151
(1901); Downes v. Bidwell, 182 U. S. (1901);
Rasmussen v. U. S., 197 U. S., Weems v. U. S., 217
U. S., 349. (But see dissenting opinions in above cases.)
96
Downes v. Bidwell, supra, and cases and laws therein
cited and quoted.
97
Idem.
98
There are powerful dissenting opinions in the various
Insular Cases. The chief objection to the unlimited
control of insular territory by Congress is that Congress
 itself, by the Constitution, possesses only limited
powers. How can a limited Congress exercise unlimited
powers?
99
Downes v. Bidwell, supra. (The Court cites, in
confirmation, the history of Congress and of the British
Parliament.)
100
Bank of Commerce v. New York City, 2 Black, 620
(1862) quoting from McCulloch v. Maryland, 4 Wheaton,
431 (1819). The principle is laid down in the decision
that “the sovereignty of a State extends to everything
which exists by its own authority, or is introduced by its
permission; but it does not extend to these means
which are employed by Congress to carry into execution
powers conferred on that body by the people of the
United States.” Id. 429.
101
Bank of Commerce v. New York City, supra.
102
Loan Association v. Topeka, 20 Wallace, 655 (1874),
quoting Cooley on Constitutional Limitations, 479.
103
P. R. Co. v. Pennsylvania, 15 Wallace, 300 (1872). The
constitutional use of the taxing power by the United
States and by the several States is examined by
Hamilton in The Federalist, No. xxxii.,—the classic
contemporaneous exposition of the taxing clauses of
the Constitution. For a judicial examination of these
clauses see Transportation Company v. Wheeling, 99
U. S., 273 (1878). The idea held both by Hamilton and
by the Court is that taxation is the exercise of sovereign
power; that “all subjects over which the sovereign
power of a State extends are objects of taxation,” but
that “objects over which it does not extend, as for
example, the means and instruments of the general
government, are exempt from taxation.” (The quotation
 in Transportation Co. v. Wheeling, from McCulloch v.
Maryland is not verbally accurate.)
104
The phrase (Federalist, No. lxii.) may be Hamilton’s or
Madison’s.
105
P. R. Co., v. Pennsylvania, 15 Wallace, 300.
106
This principle applies also in international law.
107
The principle is established in McCulloch v. Maryland.
108
Kirtland v. Hotchkiss, 100 U. S., 491 (1879).
109
Kirtland v. Hotchkiss, supra. Thus, “If the law treats the
mortgagee’s interest in the land as real estate for his
protection, it is not easy to see why the law should
forbid it to be treated as real estate for the purpose of
taxation.” Savings and Loan Society v. Multnomah
County, 169 U. S., 421 (1898).
110
McCulloch v. Maryland, supra, quoted in The Collector v.
Day, 11 Wallace, 113 (1870).
111
The Collector v. Day, supra. (The Court quotes the
Tenth Amendment, in this connection, as the basis of its
decision.)
112
Idem.
113
The Collector v. Day, supra.
114
Id.
115
The Collector v. Day, supra.
116
Id.
117
Amendment XVI.
118
Compare the effect of the Thirteenth Amendment, the
Fourteenth and Fifteenth Amendments on the decision
of the Supreme Court in Scott v. Sandford, 19 Howard,
393 (1857).
119
To what extent a salaried official of a State is exempt
from inclusion of his salary as income taxable under the
Sixteenth Amendment is as yet not determined by
judicial decision. “The corporate franchises, the
property, the business, the income of corporations
created by a State may undoubtedly be taxed by the
State; but in imposing such taxes care should be taken
not to interfere with or hamper, directly or by
indirection, interstate or foreign commerce, or any other
matter exclusively within the jurisdiction of the Federal
government. This is a principle so often announced by
the courts, and especially by this court (the Supreme
Court of the United States) that it may be received as
an axiom of our constitutional jurisprudence.”
Philadelphia and Southern Steamship Company v.
Pennsylvania, 122 U. S., 326 (1887).
120
United States v. R. R. Co., 17 Wallace, 322 (1873).
121
See the Chapter on The Law of Fundamental Rights,
post.
122
Wisconsin Central R. R. Co. v. Price County, 133 U. S.,
496 (1890). As to exemptions, the decisions are
conflicting. Not infrequently notices may be seen of
exemption of manufacturing plants, or other industrials,
from taxation, if they locate within a community.
Mississippi in its constitution of 1890 made such
exemptions by special ordinance. Such exemption has
been held valid in Franklin Needle Co. v. Franklin, 65
N. H., 177; Florida Central Railway Co. v. Reynolds, 183
 U. S., 476; Per contra, Brewer Brick Co. v. Brewer, 62
Maine, 62.
123
Loan Association v. Topeka, 20 Wallace, 655 (1874);
Kingman v. City of Brockton, 153 Mass., 255 (1891); an
admirable note citing decisions as to a good tax may be
found in L. B. Evans, Leading Cases on American
Constitutional Law (Ed. 1916), p. 211.
124
Art. i., 2: 3; 8: 1.
125
Art. i., 8: 1. Kentucky Railroad Tax Cases, 115 U. S., 321
(1885); Kelly v. Pittsburgh, 104 U. S., 78 (1881); French
v. Barber Asphalt Paving Co., 181 U. S., 324 (1901);
Veazie Bank v. Fenne, 8 Wallace, 533 (1869);
Corporation Tax Cases, 220 U. S., 611 (1911).
126
South Carolina v. United States, 199 U. S., 437 (1905).
The State conducted dispensatories and derived profit
from them. It was held liable for internal revenue. The
exercise by the State, as a dispenser, was held not to
exempt it from the operation of the law.
127
Art. i., viii., 3.
128
Brig Wilson v. U. S., 1 Brockenbrough, 437 (1820).
129
See decision of the Supreme Court sustaining the
“Webb-Kenyon” Law decommercializing (interstate)
intoxicating liquors, Clark Distilling Company v. W. Md.
R. R. Co.; Id. v. Am. Ex. Co. and State of W. Va.
(January 8, 1917).
The power of Congress to deal with the hours of work
and wages of employees engaged in interstate
commerce is examined in Wilson v. New and Ferris,
Receivers, Mo. Ok., & G. Railway Co., March 19, 1917.
(Constitutionality of the “Adamson” law.)
130
Art. vi., 2.
131
Marbury v. Madison, 1 Cranch, 177 (1803).
132
McCulloch v. Maryland, 4 Wheaton, 430 (1819).
133
Brown v. Maryland, 12 Wheaton, 419 (1827).
134
Gibbons v. Ogden, 9 Wheaton, 1 (1824).
135
Pensacola Telegraph Co. v. Western Union Telegraph
Co., 96 U. S., 1 (1877).
136
So in Gibbon v. Ogden, supra.
137
Henderson v. Mayor of New York, 92 U. S., 259 (1875);
L. S. & M. S. Railway Co. v. Ohio, 173 U. S. (1899);
Railroad Co. v. Husen, 95 U. S., 465 (1877); Brimmer v.
Rebman, 138 U. S., 78 (1891); Morgan’s S. S. Co. v.
Louisiana Board of Health, 118 U. S., 455 (1886); Leisy
v. Hardin, 135 U. S., 100 (1890); Schellenberger v.
Pennsylvania, 171 U. S., 1 (1898).
138
The trend of these respective lines is disclosed by the
decisions in the cases cited in this Chapter.
139
Pensacola Telegraph Co. v. Western Union Telegraph
Co., 96 U. S., 1 (1877). The important word here is
“jurisdiction.” “To bring the transportation within the
control of the State, as part of its domestic commerce,
the subject transported must be within the entire
voyage under the exclusive jurisdiction of the State.”
Hanley v. Kansas City Southern Railroad Co., 187 U. S.,
617 (1903). The Immigration Law (February 20, 1897,
amended March 26, 1910), contains the protective
features the State would demand through exercise of its
police power. So too the Federal Meat Inspection Act
(March 4, 1907).
140
The Daniel Ball, 10 Wallace, 557 (1870).
141
Act of Congress, March 2, 1893.
142
“The insurance business does not constitute interstate
commerce.” Paul v. Virginia, 8 Wallace, 168 (1868). But
the power to regulate commerce doubtless includes
legislation placing common carriers engaged in
interstate commerce under such federal control as to
constitute federal ownership of railroads, telegraph and
telephone lines, steamships, sailing vessels, etc., etc.
Such ownership is illustrated in France, Germany, Italy,
Russia, and in other countries.
143
The Sherman Anti-Trust Law of July 2, 1890, and
decisions of the Supreme Court concerning it, are
illustrations.
144
See the Hours of Service Act (March 4, 1907); the
Adamson Act (1916), and other acts indicative of the
trend in the congressional exercise of the power.
145
United States v. E. C. Knight Co., 156 U. S., 1 (1895).
146
Art. i., 8: 3.
147
Coe v. Errol, 116 U. S., 525.
148
Kidd v. Pearson, 128 U. S., 1.
149
Cooley v. Board of Wardens of the Port of Philadelphia,
12 Howard, 299 (1851).
150
Escanaba Company v. Chicago, 107 U. S., 678 (1882).
151
Harman v. Chicago, 147 U. S., 396 (1893).
152
Sands v. Manistee River Improvement Company, 123
U. S., 238.
153
Brown v. Maryland, 12 Wheaton, 419 (1827).
154
Brown v. Maryland, 12 Wheaton, 419 (1827).
155
Walton v. Missouri, 91 U. S., 275 (1875).
156
The evil effect of discriminating State legislation, and
the like, during the Articles of Confederation, are dwelt
on by the Court in Walton v. Missouri, supra.
157
Walton v. Missouri, supra.
158
Robbins v. Shelby County Taxing District, 120 U. S., 489
(1887).
159
Idem.
160
Crutcher v. Kentucky, 141 U. S., 47 (1891).
161
As by the act forbidding the transportation of lottery
tickets through the mails.
162
Crutcher v. Kentucky, 141 U. S., 47 (1891).
163
Idem. Cases decisive of the police powers of a State are
numerous. The principle involved may be deduced from
Railroad Company v. Huson, 95 U. S., 465 (1877);
Brimmer v. Rebman, 138 U. S., 78 (1891); Morgan’s
S. S. Company v. Louisiana Board of Health, 118 U. S.,
455 (1886); Leisy v. Hardin, 135 U. S., 100 (1890); L. S.
and M. S. R. R. v. Ohio, 173 U. S., 285 (1899).
164
Crutcher v. Kentucky, supra.
165
Brown v. Houston, 114 U. S., 622 (1885), in which the
cases are cited.
166
Telegraph Company v. Texas, 105 U. S., 460 (1881).
167
Leisy v. Hardin, 135 U. S., 100 (1890). An act of the
Legislature, or a constitutional provision prohibiting the
manufacture or sale of intoxicating liquors within a
State, is an example of exercise of the police power by
a State. See also Rhodes v. Iowa, 170 U. S., 412
(1898). Schellenberger v. Pennsylvania, 171 U. S., 1
(1898); and cases cited supra touching State police
power.
168
The Passenger Cases, 7 Howard, 283.
169
R. R. Co. v. Huson, 95 U. S., 465 (1877).
170
Turner v. Maryland, 107 U. S., 38 (1882).
171
Inman S. S. Co. v. Tinker, 94 U. S., 238 (1876).
172
Packet Co. v. Keokuk, 95 U. S., 80 (1877).
173
Transportation Co. v. Wheeling, 99 U. S., 273 (1878).
174
Lottery Cases, 188 U. S., 321 (1903).
175
Id.
176
26 Statutes at Large, 209.
177
Northern Securities Company v. United States, 193
U. S., 197 (1904).
178
Beef-Trust case, Swift and Co. v. U. S., 196 U. S., 375.
179
Danbury Hatters’ Case, Loewe v. Lawler, 208 U. S., 274;
see also Pullman Car Company, 64 Fed. Reporter, 724.
180
In re Neagle, 135 U. S., 1 (1889).
181
In re Debs, 158 U. S., 564 (1895).
182
The Addystone Pipe & Steel Company v. United States,
175 U. S., 211 (1899).
183
The Shreveport Case, (Houston, East and West Texas
Railway Co. v. United States; Texas and Pacific Railway
Co. v. United States) 234 U. S., 342 (1914).
Note.—Cases further illustrating prohibition of a
business or activity by operation of laws passed under
the commerce clause: United States v. Holliday, 3
Wallace, 407 (1866); Buttfield v. Stranahan, 192 U. S.,
470 (1904); U. S. v. Del. & Hudson Ry., 213 U. S., 366
(1909); Hope v. U. S., 227 U. S., 308 (1913).
Cases illustrating exercise of the power over
commerce given by the clause and exercising
jurisdiction over commerce claimed to be intrastate but
forming as it were a link in the chain of interstate
commerce: Lord v. S. S. Co., 102 U. S., 541 (1880);
Wilmington Transportation Co. v. California Railroad
Commission, 236 U. S., 151 (1915); Hanley v. Kansas
City Southern Ry., 187 U. S., 617 (1903).
It will be well to read the dissenting opinions in any
of these cases as these usually emphasize the power of
the State over commerce.
184
Art. i., 10: 1.
185
McCrackin v. Hayward, 2 Howard, 608 (1844).
186
Woodruff v. Trapnall, 10 Howard, 190 (1850).
187
Woodruff v. Trapnall, 10 Howard, 190 (1850).
188
Murray v. Charleston, 96 U. S., 432 (1877).
189
Idem.
190
Salt Company v. East Saginaw, 13 Wallace, 373 (1871).
191
Fisk v. Jefferson Police Jury, 116, U. S., 131 (1885).
192
Trustees of Dartmouth College v. Woodward, 4
Wheaton, 518 (1819).
193
Case of the conjunction of Washington and Jefferson
Colleges, Pennsylvania College Cases, 13 Wallace, 190
(1871).
194
Boyd v. Alabama, 94 U. S., 645.
195
Beer Company v. Massachusetts, 97 U. S., 25 (1877).
196
Douglas v. Kentucky, 168 U. S., 488 (1897).
197
Douglas v. Kentucky, supra; New Orleans Gas Co. v.
Louisiana Light Co., 115 U. S., 650 (1885).
198
See the cases cited in New Orleans Gas Co. v.
Louisiana, supra.
199
Georgia R. R. and Banking Co. v. Smith, 128 U. S., 174
(1888); East Hartford v. Hartford Bridge Co., 10
Howard, 511 (1850). But a judgment (judicial decision)
is not a contract in the meaning of the Constitution.
Morley v. L. S. & M. S. R. R., 146 U. S., 162 (1892).
200
McCrackin v. Hayward, 2 Howard, 608 (1844). All legal
remedies for the enforcement of a contract belonging to
it at the time and place when and where it is made are
a part of its obligation. Any provision of a State law or
constitution impairing such remedies are void. Gunn v.
Barry, 15 Wallace, 610 (1872); Mitchell v. Clark, 110
U. S. (1884). But the prohibition, in the Constitution, of
any State to make any law impairing the obligation of
contracts “did not give to Congress power to provide
laws for the general enforcement of contracts; nor
power to invest the courts of the United States with
 jurisdiction over contracts, so as to enable parties to
sue upon them in those courts. It did, however, give the
power to provide remedies by which the impairment of
contracts by State legislation might be counteracted and
corrected: and this power was exercised.” Civil Rights
Cases, 109 U. S., 3 (1883).
201
Juilliard v. Greenman, 110 U. S., 421 (1884), and see
note supra, p. 92.
202
Consult Mitchell v. Clark, 110 U. S., 633 (1884) from
which the quotation is taken.
203
This raises the whole question of national sovereignty.
204
Amendment V.; XIV.
205
Missouri Pacific Ry. v. Nebraska, 164 U. S., 403 (1896).
206
Pennoyer v. Neff, 95 U. S., 714 (1877); Arndt v. Griggs,
134 U. S., 316 (1890).
207
Cunnius v. Reading School District, 198 U. S., 458
(1905), sustaining a Pennsylvania statute that provided
for administration upon estates of persons presumed to
be dead by reason of long absence from the State.
Mattingly v. District of Columbia, 97 U. S., 687 (1878);
that which a State Legislature may have dispensed with
by a prior statute it may dispense with by a subsequent
one; an irregularity or defect which might be made
immaterial by prior law, the Legislature has power to
make immaterial by a subsequent law. Cooley,
Constitutional Limitations, 371.
208
License Cases, 5 Howard, 588.
209
Bartemeyer v. Iowa, 18 Wallace, 129.
210
Foster v. Kansas, 112 U. S., 201.
211
Mugler v. Kansas, 123 U. S., 623 (1887).
212
Idem.
213
Amendment V.
214
Pumpelly v. Green Bay Co., 13 Wallace, 166 (1871).
215
Idem.
216
Preceding case and Central Bridge Corporation v. City of
Lowell, Gray (Mass.), 474 (1855).
217
Pierce v. Drew, 136 Mass., 75 (1883). The case grew
out of plaintiff’s claim for damages because the town
had granted a telegraph company the right to erect its
poles, wires, etc., along the highway abutting plaintiff’s
land. The highway being land in public use, plaintiff
claimed indirect or consequential damages because of
the erection of the poles, wires, etc., of the duly
franchised telegraph company. Plaintiff’s complaint was
(inter alia) that said poles, wires, etc., disfigured and
depreciated his property. See also Bedford v. U. S., 192
U. S., 217 (1904); the principle therein further
examined.
218
Kohl v. United States, 91 U. S., 367 (1875).
219
Kohl v. United States, 91 U. S., 367 (1875).
220
Art. i., 2: 5; 3: 6.
221
Compare Art. i., 6: 1, 2; 9: 8; Art. ii., 1: 1; Art., 5, 8;
“officer” in Art. ii., 2: 1,2; Art. ii., 4: 1; “offices” in Art.
iii., 1: 1; vi., 3. There is every reason that the framers of
the Constitution used words with profound discernment
and discriminating care.
222
Art. ii., 2: 1.
223
Id. i., 8: 11.
224
Id. i., 7: 2.
225
Constitution of Pennsylvania, 1873, iv., 16.
226
Art. ii., 2: 2.
227
Id. The Federalist, No. lxxv.
228
Art. ii., 2: 2.
229
Id., 2: 1.
230
July (14?), 1864. Lincoln’s Works (Century Ed.) i., 548.
231
In re Neagle, 135 U. S., 1 (1889).
232
Spaulding v. Vilas, 161 U. S., 483; U. S. v. Windom, 137
U. S., 636; U. S. v. Blaine, 139 U. S., 306. Marbury v.
Madison, 1 Cranch, 137; Kendall v. U. S., 12 Peters,
524; U. S. v. Black, 128 U. S., 40; Mississippi v.
Johnson, 4 Wallace, 475; Georgia v. Stanton, 6 Wallace,
57.
233
Ex parte Garland, 4 Wallace, 333 (1886).
234
Art. i., 3: 6.
235
Id., 3: 4, 5.
236
Notes of conversation, etc., C. E., Stevens, Sources of
the Constitution of the United States, 169.
237
Id., 168.
238
Mississippi v. Johnson, 4 Wallace, 475 (1866).
239
Idem.
240
Mississippi v. Johnson, 4 Wallace, 475 (1866).
Welcome to our website – the perfect destination for book lovers and
knowledge seekers. We believe that every book holds a new world,
offering opportunities for learning, discovery, and personal growth.
That’s why we are dedicated to bringing you a diverse collection of
books, ranging from classic literature and specialized publications to
self-development guides and children's books.

More than just a book-buying platform, we strive to be a bridge

connecting you with timeless cultural and intellectual values. With an
elegant, user-friendly interface and a smart search system, you can
quickly find the books that best suit your interests. Additionally,
our special promotions and home delivery services help you save time
and fully enjoy the joy of reading.

Join us on a journey of knowledge exploration, passion nurturing, and

personal growth every day!

ebookmasss.com