Class Times: Tuesday, 10:00-13:00 Location: Pearson Bg., Room 305 (10:00--12:00), Room 229 (12:00--13:00) Instructors: Massimiliano Pontil Email Contact : gi12@cs.ucl.ac.uk Course description
The course introduces the elements of information theory and illustrate their relevance in AI, especially machine learning and pattern recognition. We will also review the mathematical concepts which will be used in the subsequent part of the course. This includes in particular the elements of probability theory, linear algebra, and optimization.Prerequisites
A good background in university-level mathematics (calculus, basic probability, linear algebra).Grading
The course has the following assessment components: 1) Written Examination (2.5 hours, 80%) , 2) Coursework Section (4 pieces, 20%). To pass this course, students must obtain at least 40% on the coursework component and an average of at least 50% when the coursework and exam components of a course are weighted together.
Problem sets
Problem set #1: PDF (Due: Noon, October 14)
Problem set #2: PDF (Due: Noon, October 21)
Problem set #3: PDF (Due: Noon, November 4)
Problem set #4: PDF (Due: Noon, November 18)
Syllabus
The tentative schedule of the course is listed below. Follow the link for each class to find a detailed description, suggested readings, and lecture slides.
Date Title Tuesday, October 5 Elements of Probability Theory Tuesday, October 12 Entropy and Mutual Information Tuesday, October 19 Coding and Data Compression Friday, October 22 (Room 229) Stochastic Processes, Exercises Tuesday, October 26 Channel Capacity Tuesday, November 2 Channel Capacity (cont.), Exercises Tuesday, November 9 No lectures (reading week) Tuesday, November 16 Continuous Channels Tuesday, November 23 Universal Coding Friday, November 26 (Room 203) Rate Distortion Theory Tuesday, November 30 Eigenvalue Methods for Data Compression Reading List
- T.M. Cover and J.A. Thomas. Elements of Information Theory, Wiley, 1991.
- Hamming, R. W. Coding and Information Theory. Prentice-Hall, 2nd edition, 1986.
- A.I. Khinchin. Mathematical foundation of information theory. Dover Pub. Inc., 1957.
- D.J.C. MacKay. Information Theory, Pattern Recognition and Neural Networks, Cambridge Press, 2003.
- McEliece, R. J. The Theory of Information and Coding: A Mathematical Framework for Communication. Addison-Wesley, 1977.