Markov Chains: from Theory to Implementation and Experimentation

This book takes a new approach to Markov chains, based on four convergent lines that include mathematics, implementation, simulation and experimentation. The author introduces readers to the art of stochastic modeling, shows how to design computer implementations, and provides extensive worked out examples and case studies. This book contains nine chapters, with Chapter 1 providing a general introduction of history of probability theory. The introduction to discrete-time is made using quantifiable examples, showing how the field of probability theory arrived in recent times at the notion of dependent variables (Markov model) from experiments related to independent variables (Bernoulli model). Next, Chapter 2 and 3 provide an introduction to simple stochastic matrices and transition probabilities. This is followed by a simulation of a two-state Markov Chain. Chapter 4 examines the notion of steady state and discusses it in connection with the long-run distribution behavior of the Markov chain. Predictions based on Markov chains with more than two states are covered in Chapter 5. In Chapter 6, the notion of absorbing Markov Chains it is approached by example, while Chapter 7 covers a topic linked to the average time spent in a state. Next, Chapter 8 discusses different configurations of chains. Finally, Chapter 9 covers the simulation of an n-state Markov chain used for verifying experiments of various diagram configurations.