Probability - Basis

1. Basic Probability

Types of uncertainties

• Inherent variability or randomness
• Statistical uncertainty
• Measurement error
• Model error
• Human error

Statistical uncertatinty could be reduced through accumulation of data.

1.1 Basic Definitions

Trials (experiments or observations):

• probability is concerned with the outcome of trials
• trail refers to an event whose outcome is unknown
• e.g. flip 2 coins, roll a die

Sample Space $S$:

• the collection of all possible outcomes of a trial
• e.g. for flip 2 coins $S={HH,HT,TH,TT}$

Sample Point $x$: each indivisual outcome

Event $E$: any collection of sample points (subset of $S$)

General relation: $x\in S$, $E\subseteq S$, $S=\bar{E}\cup E$ Special case: $E=S$: certain event; $E=\empty$: null event

$P(E)$: likelihood of occurrence of event $E$ in its sample space $S$

Axioms: (1) $P(E)\geq 0$ (2) $P(S)=1$ (3) $P(\bar{E})= 1-P(E)$

1.2 Permutation, Combination

Permutation: 从 n 个人中挑选 k 个排成一列，有多少种挑选及排列方式

公式只用于计算，其实是不好理解的。正确的理解方式为，第 1 个人有 n 种选择，第 2 个人有 n-1 种选择…第 k 个人有 n-k+1 种选择

Combination: 从 n 个人中挑选 k 个，有多少种挑选方式

在 permutation 的基础上，去掉那些因为位置不同导致的重复

1.3 Relationship between Events

Mutually exclusive: $P(E,F)=0$ (不能同时发生)

• $P(E\cup F)=P(E) + P(F)$ if mutually exclusive
• $P(E\cup F)=P(E) + P(F) - P(E,F)$ if not mutually exclusive

Independent: $P(E,F)=P(E)P(F)$ (S.I. statistica)

• $P(E,F)=P(E)P(F\vert E)$ if dependent
• $P(E\vert F)=P(E)$ in other words

Two mutually exlusive events are dependent, 因为如果一个发生我们就知道另一个不会发生 ($P(E,F)=0$)

Conditional Independent: \(p(x,y\vert z)=p(x\vert z)p(y\vert z)\)

Complementary: $E\bar{F}=\emptyset$

Collective exhaustive events: $\bigcup_{i=1}^M E_i=S$

1.4 Random Varaibles

A ramdom variable is a mapping from sample space $S$ to a real number space $R$

e.g. Tossing tow coins, $S={HH,HT,TH,TT}$

denote $X$ as number of H, then $S_X={0,1,2}$

2. Multiple Events

For multiple events: $E_1,E_2,…,E_M$

2.1 Associate Properties

(1) $(E_1\cup E_2)\cup E_3 = E_1\cup(E_2 \cup E_3) = E_1\cup E_2\cup E_3$

(2) $(E_1E_2)E_3 = E_1(E_2E_3) = E_1E_2E_3$

2.2 De Morgan’s Rules

\(\tag{2.1}\overline{\bigcup_{i=1}^M E_i}=\bigcap_{i=1}^M \overline{E_i}\)

That is $\overline{E_i\cup E_2\cup…\cup E_M}=\overline{E_1}…\overline{E_M}$

That is $\overline{E_1…E_M}=\overline{E_1}\cup\overline{E_1}\cup\overline{E_M}$

Representation through circuits

Let $E_i$ indicates the failure event of $i$-th component, and $E_{sys}$ indicates the whole system’s failure. The parallel and series circuits are shown below:

Series connections: $E_{sys} = E_1\cup E_2\cup E_3$,     $\overline{E_{sys}}=\bar{E_1}\bar{E_2}\bar{E_3}$

Parallel connections: $E_{sys} = E_1E_2E_3$,     $\overline{E_{sys}}=\bar{E_1}\cup\bar{E_2}\cup\bar{E_3}$

3. Rules of Probability Theory

3.1 Inclusion-Exclusion Rule

For simplest case: $P(E_1\cup E_2)=P(E_1) + P(E_2) - P(E_1E_2)$

For general cases: \(P(\bigcup_{i=1}^nE_i)=\sum_{i=1}^nP(E_i) - \sum_{i=1}^{n-1}\sum_{j=i+1}^nP(E_iE_j)+...+(-1)^{n-1}P(E_1E_2...E_n)\)

Statistical Independence

Two events are statistically independant (S.I.) if the occurrence of one event doesn’t change the probability of the other, that is, $P(E_1\vert E_2)=P(E_1)$ and $P(E_2\vert E_1)=P(E_2)$.

Only when $E_1,E_2$ are S.I., the formulation $P(E_1E_2)=P(E_1)\cdot P(E_2)$ works

3.2 Total Probability Rule

Let $E_1,E_2,…,E_M$ satisfy $\begin{cases}E_iE_j=\empty \text{ for any } i\neq j \ \bigcup_{i=1}^ME_i=S\end{cases}$

then $P(A)=\sum_{i=1}^MP(A\vert E_i)P(E_i)$

That’s because $=\sum_{i=1}^MP(A, E_i)=P(AE_1\cup…\cup AE_M)=P(A(E_1\cup…\cup E_M))=P(AS)=P(A)$

3.3 Bayes’s Rule

where

• $P(A\vert B)$ posterior probability
• $P(A)$ prior probability
• $P(B\vert A)$ likelihood
• $P(B)=P(B\vert A)P(A) + P(B\vert\sim A)P(\sim A)$

3.4 Other Calculations

$P(A,B)=P(A\vert B) P(B)$

$P(A,B\vert C)=P(A\vert B,C)P(B\vert C)$

$ $