This book presents recent research in decision making under uncertainty, in particular reinforcement learning and learning with expert advice. The core elements of decision theory, Markov decision processes and reinforcement learning have not been previously collected in a concise volume. Our aim with this book was to provide a solid theoretical foundation with elementary proofs of the most important theorems in the field, all collected in one place, and not typically found in introductory textbooks. This book is addressed to graduate students that are interested in statistical decision making under uncertainty and the foundations of reinforcement learning.