Probability and Probabilistic Reasoning for Electrical Engineering

Chapter 1: Background, Events, and Data
Brief Historical Background to Probability. Important Examples of Random Phenomena. Modeling Random Phenomena I: Random Experiment and Sample Space. Modeling Random Phenomena II: Events and Combinations of Events. Modeling Random Phenomena III: Event Collections and Algebras. Dealing with Chance Data–Statistics. Appendix 1: Sets. Appendix 2: Functions. Appendix 3: Pseudorandom Numbers Generated by Matlab.



Chapter 2: Classical Probability
Choosing at Random. Enumeration of Ordered Sequences. Enumeration of Sets: Binomial Coefficients. Application to Entropy and Data Compression. Application to Graphs. Application to Statistical Mechanics. Multinomial Counting. Conditional Classical Probability. Independence in Classical Probability.



Chapter 3: Probability Foundations

Hierarchy of Probability Structural Concepts. Interpretations of Probability. Long-run Time Averages and Relative Frequencies. Modeling Random Phenomena IV: Kolmogorov Axioms. Elementary Consequences of the Axioms K0--K4. Boole's Inequality. Inclusion-Exclusion Principle. Convex Combinations of Probability Measures. Application to Target Detection. Appendix: Proof of Equivalence Theorem.





Chapter 4: Describing Probability I: Countable
Probability Mass Functions. Application to Modeling English Text Letter Frequencies. Commonly Encountered PMFs. Poisson as a Rare Events Limit of the Binomial.



Chapter 5: Describing Probability II: Uncountable and Distributions
Cumulative Distribution Functions. Properties of Univariate CDFs. General Representation and Decomposition of Univariate CDFs. The Empirical CDF.



Chapter 6: Describing Probability III: Uncountable and Densities
Probability Density Functions. Representing PMFs by PDFs. Closure Under Convex Combinations. PDFs over Finite Intervals. PDFs over Semi-infinite Intervals. PDFs over all of R. Appendix: Beta Density Unit Normalization.



Chapter 7: Multivariate Distribution Functions and Densities
Multivariate or Joint CDFs. Multivariate PDF Representation of Multivariate CDF. Examples of Multivariate PDFs. Application to Reliability.



Chapter 8: Functions of Random Variables
Random Variables. Single Input--Single Output (SISO) Functions. Simulation and the Probability Integral Transformation. Multiple Input-Multiple Output (MIMO): n = m. MIMO Applications. MIMO Transformation: m < n.



Chapter 9: Expectation and Moments

The Concept of Expectation. Interpretation 1: Mean. Elementary Properties of Expectation. Interpretation 2: Definite Integral. Interpretation 3: Long-run Average Outcome. Interpretation 4: Fair Price. Expectation of a Function of a Random Variable. Convex Functions and Jensen's Inequality. Moments, Especially Variance. Correlation, Covariance, and the Schwarz Inequality. Linear Dependence and Least Mean Square Estimation. Extensions of Expectation to Vector, Matrix, and Complex-Valued Variables. Correlation and Covariance for Real Random Vectors. Linear Transformation and Synthesis of (Gaussian) Random Vectors. Appendix: Summary of Basic Properties of Expectation. Appendix: Correlation for Complex Random Vectors.



Chapter 10: Linear Systems, Signals, and Filtering
Background to Linear Systems. Random Signals and Autocorrelation Functions. Moments of Outputs of Linear Systems. Application to Wiener Filtering. Application to Kalman Filtering.



Chapter 11: Discrete Conditional Probability
Conditional Probability. Properties of Discrete Conditional Probability. Multiplication/Sequence Theorem. Total Probability Theorem and Its Applications. Inverting Cause and Effect: Bayes' Theorem and Its Applications. Three Problems that Challenge Our Intuition. Appendix: Discrete Conditional Probability Summary.



Chapter 12: Mixed Conditional Probability
Conditional Probability: P(A|B) for P(B) = 0. Bivariate Conditional Density. Multivariate Conditional Densities. Applications of Conditional Densities. Useful Families of Conditional Densities. Extensions of the Total Probability Theorem. Applications of the Extended Total Probability Theorem. Extensions of the Bayes' Theorem. Applications of the Extended Bayes' Theorem. Appendix: Conditional Probability Summary I. Appendix: Conditional Probability Summary II.



Chapter 13: MAP, MLE, and Neyman-Pearson Rules
Binary Decision-Making Setup. Minimum Error Probability Design: MAP Rule. Hypothesis Testing. Additive Measurement Noise.



Chapter 14: Independence

Independent Pairs of Events. Mutually Independent Events. Independent Random Variables and Product Factorization. Derivations of Two Probability Models. Example of Maxima and Minima of Random Variables. Expectation and Independence. Application to Estimation of the Mean and Variance.



Chapter 15: Characteristic Functions
Characteristic Function Definition and Examples. Properties of the Characteristic Function. Relationship Between Characteristic Functions and Moments. Extension to Random Vectors. Linear Transformations and the Multivariate Normal Revisited. Characteristic Functions and Independence. Characteristic Functions of Sums. Central Limit Theorem. Generating Function.



Chapter 16: Probability Bounds and Sums
Generalized Chebychev Bounds. Chernoff Bounds. Hoeffding Bounds.



Chapter 17: Conditional Expectation
Conditional Expectation Basics. Example of the Multivariate Normal. MMSE Estimation. Two Applications of MMSE Estimation. Application to Kalman Filtering Revisited. Random Sums. Appendix: Conditioning on Event Algebras.



Chapter 18: Bayesian Inference
Setup and Notation for Bayesian Decision-Making. Bayes Approach to Decision-Making. Calculating Bayes Rules. Particular Bayes Solutions. Appendix: An Informal Primer on Loss Functions.



Chapter 19: Limits and Laws of Large Numbers

Jacob Bernoulli on the Laws of Large Numbers. LLN for Mean Square Convergence. Convergence of Sequences of Functions. Convergence of Sequences of Random Variables. Mutual Convergence. Laws of Large Numbers. An Example of Divergence of Time Averages. Appendix: Proof of Theorem 19.5.1 C1, C2.



Chapter 20: Random Processes
Definition of a Random Process. Borel-Cantelli Lemma. Independent Random Variables. Stationary and Ergodic Processes. Markov Chains in Discrete Time. Independent Increment Random Processes. Martingales. Calculus. Wide Sense Stationary Processes and Power Spectral Density. Gaussian Random Process.



Bibliography. Index.