Probability and Statistics Cookbook. pdf
Probability and Statistics
Cookbook
c Matthias Vallentin, 2011
Copyright
vallentin@icir.org
12th December, 2011
12 Parametric Inference
12.1 Method of Moments . . . . . . . . .
12.2 Maximum Likelihood . . . . . . . . .
versity of California in Berkeley but also influenced by other
12.2.1 Delta Method . . . . . . . . .
sources [4, 5]. If you find errors or have suggestions for further
12.3 Multiparameter Models . . . . . . .
topics, I would appreciate if you send me an email. The most re12.3.1 Multiparameter delta method
cent version of this document is available at http://matthias.
12.4 Parametric Bootstrap . . . . . . . .
This cookbook integrates a variety of topics in probability theory and statistics. It is based on literature [1, 6, 3] and in-class
material from courses of the statistics department at the Uni-
vallentin.net/probability-and-statistics-cookbook/. To
reproduce, please contact me.
2 Probability Theory
3 Random Variables
3.1 Transformations . . . . . . . . . . . . .
6
7
1 Distribution Overview
1.1 Discrete Distributions . . . . . . . . . .
1.2 Continuous Distributions . . . . . . . .
4 Expectation
5 Variance
6 Inequalities
7 Distribution Relationships
8 Probability
Functions
and
Moment
Generating
9 Multivariate Distributions
9.1 Standard Bivariate Normal . . . . . . .
9.2 Bivariate Normal . . . . . . . . . . . . .
9.3 Multivariate Normal . . . . . . . . . . .
10 Convergence
10.1 Law of Large Numbers (LLN) . . . . . .
10.2 Central Limit Theorem (CLT) . . . . .
11 Statistical Inference
11.1 Point Estimation . . . . . . . . . .
11.2 Normal-Based Confidence Interval
11.3 Empirical distribution . . . . . . .
11.4 Statistical Functionals . . . . . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
13 Hypothesis Testing
14 Bayesian Inference
14.1 Credible Intervals . . . .
14.2 Function of parameters .
3
3
14.3 Priors . . . . . . . . . .
4
14.3.1 Conjugate Priors
14.4 Bayesian Testing . . . .
6
Contents
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
15 Exponential Family
16 Sampling Methods
16.1 The Bootstrap . . . . . . . . . . . . . .
7
16.1.1 Bootstrap Confidence Intervals .
7
16.2 Rejection Sampling . . . . . . . . . . . .
16.3 Importance Sampling . . . . . . . . . . .
8
17 Decision Theory
8
17.1 Risk . . . . . . . . . . . . . . . . . . . .
17.2 Admissibility . . . . . . . . . . . . . . .
17.3 Bayes Rule . . . . . . . . . . . . . . . .
9
17.4 Minimax Rules . . . . . . . . . . . . . .
9
9 18 Linear Regression
9
18.1 Simple Linear Regression . . . . . . . .
9
18.2 Prediction . . . . . . . . . . . . . . . . .
18.3 Multiple Regression . . . . . . . . . . .
9
18.4 Model Selection . . . . . . . . . . . . . .
10
10
19 Non-parametric Function Estimation
19.1 Density Estimation . . . . . . . . . . . .
10
19.1.1 Histograms . . . . . . . . . . . .
10
19.1.2 Kernel Density Estimator (KDE)
11
19.2 Non-parametric Regression . . . . . . .
11
11
19.3 Smoothing Using Orthogonal Functions
11 20 Stochastic Processes
20.1 Markov Chains . . . . . . . . . .
11
20.2 Poisson Processes . . . . . . . . .
12
12
21 Time Series
12
21.1 Stationary Time Series . . . . . .
13
21.2 Estimation of Correlation . . . .
13
21.3 Non-Stationary Time Series . . .
21.3.1 Detrending . . . . . . . .
13
21.4 ARIMA models . . . . . . . . . .
21.4.1 Causality and Invertibility
14
21.5 Spectral Analysis . . . . . . . . .
14
14 22 Math
22.1 Gamma Function . . . . . . . . .
15
22.2 Beta Function . . . . . . . . . . .
15
22.3 Series . . . . . . . . . . . . . . .
15
22.4 Combinatorics . . . . . . . . . .
16
16
16
16
17
17
17
17
17
18
18
18
18
19
19
19
20
20
20
21
21
21
22
. . . . 22
. . . . 22
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
23
23
24
24
24
24
25
25
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
26
26
26
27
27
Distribution Overview
1.1
Discrete Distributions
Notation1
FX (x)
Unif {a, . . . , b}
Uniform
Bernoulli
Bern (p)
Binomial
0
1
Hypergeometric
≈Φ
Hyp (N, m, n)
Negative Binomial
NBin (r, p)
Geometric
Geo (p)
Poisson
Po (λ)
!
e−λ
MX (s)
I(a < x < b)
b−a+1
a+b
2
(b − a + 1)2 − 1
12
eas − e−(b+1)s
s(b − a)
px (1 − p)1−x
p
p(1 − p)
1 − p + pes
np
np(1 − p)
(1 − p + pes )n
x
x ∈ N+
p(1 − p)x−1
x
X
λi
i!
i=0
Uniform (discrete)
r
1−p
p
Binomial
1−p
p2
1
p
1−p
p2
λ
λ
x ∈ N+
λx e−λ
x!
r
r
p
1 − (1 − p)es
eλ(e
s
−1)
Poisson
●
p = 0.2
p = 0.5
p = 0.8
●
●
●
λ=1
λ=4
λ = 10
●
●
●
●
●
0.2
0.6
●
PMF
●
0.4
●
PMF
0.15
●
n
PMF
1
●
0.10
●
●
●
0.05
x
1 We
●
●
●
0
●
●
●
10
● ●
● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●
20
x
30
40
●
●
●
●
●
0.0
●
●
0.0
● ● ● ●
●
●
●
b
●
●
●
a
0.1
0.2
●
●
0.00
PMF
0.20
0.3
0.25
n = 40, p = 0.3
n = 30, p = 0.6
n = 25, p = 0.9
!n
p
1 − (1 − p)es
Geometric
●
pi e
si
i=0
nm(N − n)(N − m)
N 2 (N − 1)
nm
N
n−x
N
x
k
X
npi (1 − pi )
npi
!
x+r−1 r
p (1 − p)x
r−1
Ip (r, x + 1)
1 − (1 − p)x
V [X]
k
X
n!
x
px1 1 · · · pkk
xi = n
x1 ! . . . xk !
i=1
m m−x
Mult (n, p)
x − np
p
np(1 − p)
E [X]
!
n x
p (1 − p)n−x
x
I1−p (n − x, x + 1)
Bin (n, p)
Multinomial
xb
(1 − p)1−x
⌊x⌋−a+1
b−a
fX (x)
0.8
1
0
2
4
6
8
10
●
0
x
5
●
●
●
●
●
10
●
●
●
●
●
15
●
●
●
●
●
20
x
use the notation γ(s, x) and Γ(x) to refer to the Gamma functions (see §22.1), and use B(x, y) and Ix to refer to the Beta functions (see §22.2).
3
1.2
Continuous Distributions
Notation
FX (x)
Uniform
Unif (a, b)
Normal
N µ, σ 2
0
x2
(α − 1)2 (α − 2)2
1
exp µT s + sT Σs
2
(1 − 2s)−k/2 s < 1/2
d
F
F(d1 , d2 )
Exponential
Exp (β)
Gamma
Inverse Gamma
Dirichlet
Beta
Weibull
Pareto
I
d1 x
d1 x+d2
Gamma (α, β)
InvGamma (α, β)
d1 d1
,
2 2
d1
, 2
2
1 −x/β
e
β
γ(α, x/β)
Γ(α)
Γ α, βx
Γ (α)
1
xα−1 e−x/β
Γ (α) β α
β α −α−1 −β/x
x
e
Γ (α)
P
k
k
Γ
i=1 αi Y α −1
xi i
Qk
i=1 Γ (αi ) i=1
Γ (α + β) α−1
x
(1 − x)β−1
Γ (α) Γ (β)
Ix (α, β)
Weibull(λ, k)
1 − e−(x/λ)
Pareto(xm , α)
x α
1−
xB
d1
1 − e−x/β
Dir (α)
Beta (α, β)
(d1 x)d1 d2 2
(d1 x+d2 )d1 +d2
m
x
k
x ≥ xm
k x k−1 −(x/λ)k
e
λ λ
xα
m
α α+1
x
x ≥ xm
αi
Pk
i=1 αi
α
α+β
1
λΓ 1 +
k
αxm
α>1
α−1
E [Xi ] (1 − E [Xi ])
Pk
i=1 αi + 1
αβ
(α + β)2 (α + β + 1)
2
λ2 Γ 1 +
− µ2
k
xα
m
α>2
(α − 1)2 (α − 2)
1
(s < 1/β)
1 − βs
α
1
(s < 1/β)
1 − βs
p
2(−βs)α/2
−4βs
Kα
Γ(α)
1+
∞
X
k=1
∞
X
n=0
k−1
Y
r=0
α+r
α+β+r
!
sk
k!
s n λn
n
Γ 1+
n!
k
α(−xm s)α Γ(−α, −xm s) s < 0
4
Normal
Log−normal
Student's t
1.0
µ = 0, σ2 = 3
µ = 2, σ2 = 2
µ = 0, σ2 = 1
µ = 0.5, σ2 = 1
µ = 0.25, σ2 = 1
µ = 0.125, σ2 = 1
ν=1
ν=2
ν=5
ν=∞
0.2
PDF
PDF
0.4
φ(x)
PDF
0.6
0.6
0.3
0.8
0.8
µ = 0, σ2 = 0.2
µ = 0, σ2 = 1
µ = 0, σ2 = 5
µ = −2, σ2 = 0.5
0.4
Uniform (continuous)
●
0.1
−4
−2
0
x
2
4
0.0
0.0
0.2
0.2
●
b
0.0
●
a
0.0
0.5
1.0
1.5
x
χ
2
F
2.5
3.0
−4
−2
0
Exponential
4
Gamma
2.0
β=2
β=1
β = 0.4
α = 1, β = 2
α = 2, β = 2
α = 3, β = 2
α = 5, β = 1
α = 9, β = 0.5
0.4
3.0
d1 = 1, d2 = 1
d1 = 2, d2 = 1
d1 = 5, d2 = 2
d1 = 100, d2 = 1
d1 = 100, d2 = 100
2
x
0.3
PDF
0.2
1.0
PDF
PDF
1.5
4
6
8
0.1
0
1
2
3
4
5
0.0
0
1
2
x
x
Inverse Gamma
Beta
5
0
5
10
15
20
x
Pareto
xm = 1, α = 1
xm = 1, α = 2
xm = 1, α = 4
λ = 1, k = 0.5
λ = 1, k = 1
λ = 1, k = 1.5
λ = 1, k = 5
2.0
2.5
4
4
Weibull
α = 0.5, β = 0.5
α = 5, β = 1
α = 1, β = 3
α = 2, β = 2
α = 2, β = 5
3.0
α = 1, β = 1
α = 2, β = 1
α = 3, β = 1
α = 3, β = 0.5
3
x
0
1
2
3
x
4
5
0.0
0.2
0.4
0.6
x
0.8
1.0
2
0
0.0
0
0.0
0.5
0.5
1
1
1.0
1.0
PDF
PDF
1.5
PDF
2
PDF
1.5
3
2.0
3
2
2.5
0
0.0
0.0
0.0
0.5
0.1
0.5
1.0
0.2
PDF
0.3
2.0
1.5
2.5
0.4
0.5
k=1
k=2
k=3
k=4
k=5
2.0
x
0.5
●
0.4
1
b−a
0.0
0.5
1.0
1.5
x
2.0
2.5
0
1
2
3
4
5
x
5
2
Probability Theory
Law of Total Probability
Definitions
•
•
•
•
P [B] =
Sample space Ω
Outcome (point or element) ω ∈ Ω
Event A ⊆ Ω
σ-algebra A
Inclusion-Exclusion Principle
[
n
X
n
Ai =
(−1)r−1
i=1
3
Ai
r=1
X
Ω=
n
G
Ai
i=1
r
\
Aij
i≤i1
Cookbook
c Matthias Vallentin, 2011
Copyright
vallentin@icir.org
12th December, 2011
12 Parametric Inference
12.1 Method of Moments . . . . . . . . .
12.2 Maximum Likelihood . . . . . . . . .
versity of California in Berkeley but also influenced by other
12.2.1 Delta Method . . . . . . . . .
sources [4, 5]. If you find errors or have suggestions for further
12.3 Multiparameter Models . . . . . . .
topics, I would appreciate if you send me an email. The most re12.3.1 Multiparameter delta method
cent version of this document is available at http://matthias.
12.4 Parametric Bootstrap . . . . . . . .
This cookbook integrates a variety of topics in probability theory and statistics. It is based on literature [1, 6, 3] and in-class
material from courses of the statistics department at the Uni-
vallentin.net/probability-and-statistics-cookbook/. To
reproduce, please contact me.
2 Probability Theory
3 Random Variables
3.1 Transformations . . . . . . . . . . . . .
6
7
1 Distribution Overview
1.1 Discrete Distributions . . . . . . . . . .
1.2 Continuous Distributions . . . . . . . .
4 Expectation
5 Variance
6 Inequalities
7 Distribution Relationships
8 Probability
Functions
and
Moment
Generating
9 Multivariate Distributions
9.1 Standard Bivariate Normal . . . . . . .
9.2 Bivariate Normal . . . . . . . . . . . . .
9.3 Multivariate Normal . . . . . . . . . . .
10 Convergence
10.1 Law of Large Numbers (LLN) . . . . . .
10.2 Central Limit Theorem (CLT) . . . . .
11 Statistical Inference
11.1 Point Estimation . . . . . . . . . .
11.2 Normal-Based Confidence Interval
11.3 Empirical distribution . . . . . . .
11.4 Statistical Functionals . . . . . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
13 Hypothesis Testing
14 Bayesian Inference
14.1 Credible Intervals . . . .
14.2 Function of parameters .
3
3
14.3 Priors . . . . . . . . . .
4
14.3.1 Conjugate Priors
14.4 Bayesian Testing . . . .
6
Contents
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
15 Exponential Family
16 Sampling Methods
16.1 The Bootstrap . . . . . . . . . . . . . .
7
16.1.1 Bootstrap Confidence Intervals .
7
16.2 Rejection Sampling . . . . . . . . . . . .
16.3 Importance Sampling . . . . . . . . . . .
8
17 Decision Theory
8
17.1 Risk . . . . . . . . . . . . . . . . . . . .
17.2 Admissibility . . . . . . . . . . . . . . .
17.3 Bayes Rule . . . . . . . . . . . . . . . .
9
17.4 Minimax Rules . . . . . . . . . . . . . .
9
9 18 Linear Regression
9
18.1 Simple Linear Regression . . . . . . . .
9
18.2 Prediction . . . . . . . . . . . . . . . . .
18.3 Multiple Regression . . . . . . . . . . .
9
18.4 Model Selection . . . . . . . . . . . . . .
10
10
19 Non-parametric Function Estimation
19.1 Density Estimation . . . . . . . . . . . .
10
19.1.1 Histograms . . . . . . . . . . . .
10
19.1.2 Kernel Density Estimator (KDE)
11
19.2 Non-parametric Regression . . . . . . .
11
11
19.3 Smoothing Using Orthogonal Functions
11 20 Stochastic Processes
20.1 Markov Chains . . . . . . . . . .
11
20.2 Poisson Processes . . . . . . . . .
12
12
21 Time Series
12
21.1 Stationary Time Series . . . . . .
13
21.2 Estimation of Correlation . . . .
13
21.3 Non-Stationary Time Series . . .
21.3.1 Detrending . . . . . . . .
13
21.4 ARIMA models . . . . . . . . . .
21.4.1 Causality and Invertibility
14
21.5 Spectral Analysis . . . . . . . . .
14
14 22 Math
22.1 Gamma Function . . . . . . . . .
15
22.2 Beta Function . . . . . . . . . . .
15
22.3 Series . . . . . . . . . . . . . . .
15
22.4 Combinatorics . . . . . . . . . .
16
16
16
16
17
17
17
17
17
18
18
18
18
19
19
19
20
20
20
21
21
21
22
. . . . 22
. . . . 22
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
23
23
24
24
24
24
25
25
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
26
26
26
27
27
Distribution Overview
1.1
Discrete Distributions
Notation1
FX (x)
Unif {a, . . . , b}
Uniform
Bernoulli
Bern (p)
Binomial
0
1
Hypergeometric
≈Φ
Hyp (N, m, n)
Negative Binomial
NBin (r, p)
Geometric
Geo (p)
Poisson
Po (λ)
!
e−λ
MX (s)
I(a < x < b)
b−a+1
a+b
2
(b − a + 1)2 − 1
12
eas − e−(b+1)s
s(b − a)
px (1 − p)1−x
p
p(1 − p)
1 − p + pes
np
np(1 − p)
(1 − p + pes )n
x
x ∈ N+
p(1 − p)x−1
x
X
λi
i!
i=0
Uniform (discrete)
r
1−p
p
Binomial
1−p
p2
1
p
1−p
p2
λ
λ
x ∈ N+
λx e−λ
x!
r
r
p
1 − (1 − p)es
eλ(e
s
−1)
Poisson
●
p = 0.2
p = 0.5
p = 0.8
●
●
●
λ=1
λ=4
λ = 10
●
●
●
●
●
0.2
0.6
●
PMF
●
0.4
●
PMF
0.15
●
n
PMF
1
●
0.10
●
●
●
0.05
x
1 We
●
●
●
0
●
●
●
10
● ●
● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●
20
x
30
40
●
●
●
●
●
0.0
●
●
0.0
● ● ● ●
●
●
●
b
●
●
●
a
0.1
0.2
●
●
0.00
PMF
0.20
0.3
0.25
n = 40, p = 0.3
n = 30, p = 0.6
n = 25, p = 0.9
!n
p
1 − (1 − p)es
Geometric
●
pi e
si
i=0
nm(N − n)(N − m)
N 2 (N − 1)
nm
N
n−x
N
x
k
X
npi (1 − pi )
npi
!
x+r−1 r
p (1 − p)x
r−1
Ip (r, x + 1)
1 − (1 − p)x
V [X]
k
X
n!
x
px1 1 · · · pkk
xi = n
x1 ! . . . xk !
i=1
m m−x
Mult (n, p)
x − np
p
np(1 − p)
E [X]
!
n x
p (1 − p)n−x
x
I1−p (n − x, x + 1)
Bin (n, p)
Multinomial
xb
(1 − p)1−x
⌊x⌋−a+1
b−a
fX (x)
0.8
1
0
2
4
6
8
10
●
0
x
5
●
●
●
●
●
10
●
●
●
●
●
15
●
●
●
●
●
20
x
use the notation γ(s, x) and Γ(x) to refer to the Gamma functions (see §22.1), and use B(x, y) and Ix to refer to the Beta functions (see §22.2).
3
1.2
Continuous Distributions
Notation
FX (x)
Uniform
Unif (a, b)
Normal
N µ, σ 2
0
x2
(α − 1)2 (α − 2)2
1
exp µT s + sT Σs
2
(1 − 2s)−k/2 s < 1/2
d
F
F(d1 , d2 )
Exponential
Exp (β)
Gamma
Inverse Gamma
Dirichlet
Beta
Weibull
Pareto
I
d1 x
d1 x+d2
Gamma (α, β)
InvGamma (α, β)
d1 d1
,
2 2
d1
, 2
2
1 −x/β
e
β
γ(α, x/β)
Γ(α)
Γ α, βx
Γ (α)
1
xα−1 e−x/β
Γ (α) β α
β α −α−1 −β/x
x
e
Γ (α)
P
k
k
Γ
i=1 αi Y α −1
xi i
Qk
i=1 Γ (αi ) i=1
Γ (α + β) α−1
x
(1 − x)β−1
Γ (α) Γ (β)
Ix (α, β)
Weibull(λ, k)
1 − e−(x/λ)
Pareto(xm , α)
x α
1−
xB
d1
1 − e−x/β
Dir (α)
Beta (α, β)
(d1 x)d1 d2 2
(d1 x+d2 )d1 +d2
m
x
k
x ≥ xm
k x k−1 −(x/λ)k
e
λ λ
xα
m
α α+1
x
x ≥ xm
αi
Pk
i=1 αi
α
α+β
1
λΓ 1 +
k
αxm
α>1
α−1
E [Xi ] (1 − E [Xi ])
Pk
i=1 αi + 1
αβ
(α + β)2 (α + β + 1)
2
λ2 Γ 1 +
− µ2
k
xα
m
α>2
(α − 1)2 (α − 2)
1
(s < 1/β)
1 − βs
α
1
(s < 1/β)
1 − βs
p
2(−βs)α/2
−4βs
Kα
Γ(α)
1+
∞
X
k=1
∞
X
n=0
k−1
Y
r=0
α+r
α+β+r
!
sk
k!
s n λn
n
Γ 1+
n!
k
α(−xm s)α Γ(−α, −xm s) s < 0
4
Normal
Log−normal
Student's t
1.0
µ = 0, σ2 = 3
µ = 2, σ2 = 2
µ = 0, σ2 = 1
µ = 0.5, σ2 = 1
µ = 0.25, σ2 = 1
µ = 0.125, σ2 = 1
ν=1
ν=2
ν=5
ν=∞
0.2
0.4
φ(x)
0.6
0.6
0.3
0.8
0.8
µ = 0, σ2 = 0.2
µ = 0, σ2 = 1
µ = 0, σ2 = 5
µ = −2, σ2 = 0.5
0.4
Uniform (continuous)
●
0.1
−4
−2
0
x
2
4
0.0
0.0
0.2
0.2
●
b
0.0
●
a
0.0
0.5
1.0
1.5
x
χ
2
F
2.5
3.0
−4
−2
0
Exponential
4
Gamma
2.0
β=2
β=1
β = 0.4
α = 1, β = 2
α = 2, β = 2
α = 3, β = 2
α = 5, β = 1
α = 9, β = 0.5
0.4
3.0
d1 = 1, d2 = 1
d1 = 2, d2 = 1
d1 = 5, d2 = 2
d1 = 100, d2 = 1
d1 = 100, d2 = 100
2
x
0.3
0.2
1.0
1.5
4
6
8
0.1
0
1
2
3
4
5
0.0
0
1
2
x
x
Inverse Gamma
Beta
5
0
5
10
15
20
x
Pareto
xm = 1, α = 1
xm = 1, α = 2
xm = 1, α = 4
λ = 1, k = 0.5
λ = 1, k = 1
λ = 1, k = 1.5
λ = 1, k = 5
2.0
2.5
4
4
Weibull
α = 0.5, β = 0.5
α = 5, β = 1
α = 1, β = 3
α = 2, β = 2
α = 2, β = 5
3.0
α = 1, β = 1
α = 2, β = 1
α = 3, β = 1
α = 3, β = 0.5
3
x
0
1
2
3
x
4
5
0.0
0.2
0.4
0.6
x
0.8
1.0
2
0
0.0
0
0.0
0.5
0.5
1
1
1.0
1.0
1.5
2
1.5
3
2.0
3
2
2.5
0
0.0
0.0
0.0
0.5
0.1
0.5
1.0
0.2
0.3
2.0
1.5
2.5
0.4
0.5
k=1
k=2
k=3
k=4
k=5
2.0
x
0.5
●
0.4
1
b−a
0.0
0.5
1.0
1.5
x
2.0
2.5
0
1
2
3
4
5
x
5
2
Probability Theory
Law of Total Probability
Definitions
•
•
•
•
P [B] =
Sample space Ω
Outcome (point or element) ω ∈ Ω
Event A ⊆ Ω
σ-algebra A
Inclusion-Exclusion Principle
[
n
X
n
Ai =
(−1)r−1
i=1
3
Ai
r=1
X
Ω=
n
G
Ai
i=1
r
\
Aij
i≤i1