COMPGI16 - Approximate Inference and Learning in Probabilistic Models

This database contains 2016-17 versions of the syllabuses. For current versions please see here.

Year MSc
Prerequisites COMPGI18 Probabilstic and Unsupervised Learning
Term 1
Taught By Maneesh Sahani (Gatsby Computational Neuroscience Unit) (100%)
Aims The module will present the foundations of approximate inference and learning in probabilistic graphical models (e.g. Bayesian networks and Markov networks), with particular focus on models composed from conditional exponential family distributions. Both stochastic (Monte Carlo) methods and deterministic approximations will be covered. The methods will be discussed in relation to practical problems in real-world inference in Machine Learning, including problems in tracking and learning.
Learning Outcomes Students will be able to understand how to derive and implement state-of-the-art approximate inference techniques and be able to make contributions to research in this area.


Nonlinear, hierarchical (deep), and distributed models.

Independent component analysis, Boltzmann machines, Dirichlet topic models, manifold discovery.

Mean-field methods, variational approximations and variational Bayes.

Expectation propagation

Loopy belief propagation, the Bethe free energy and extensions.

Convex methods and convexified bounds.

Monte-Carlo methods: including rejection and importance sampling, Gibbs, Metropolis-Hastings, anealed importance sampling, Hamiltonian Monte-Carlo, slice sampling, sequential Monte-Carlo (particle filtering)

Other topics as time permits.

Method of Instruction:

Lecture presentations with associated class problems.


The course has the following assessment components:

  • Written Examination (2.5 hours, 50%)
  • Coursework Section (3 pieces, 50%)

 To pass this course, students must:

  • Obtain an overall pass mark of 50% for all sections combined.

The examination rubric is:

Answer all questions



There is no required textbook. However, the following in an excellent sources for many of the topics covered here.

David J.C. MacKay (2003) Information Theory, Inference, and Learning Algorithms, Cambridge University Press. (also available online)

Web resources for Gatsby Unit courses