Class Times: Tuesday, 14:00-17:00, Friday 10:00-13:00 Location: Computer Science Dept, Room 104 Instructors: Massimiliano Pontil Email Contact : gi13@cs.ucl.ac.uk Course description
The course presents the elements of kernel-based methods from a machine learning perspective. It introduces the theoretical basis for studying these methods (theory of positive definite kernels, associated reproducing kernel Hilbert spaces and techniques to construct kernel functions) and present selected topics in this area. This includes learning algorithms such as regularization networks, support vector machines, kernel principal component analysis, kernel canonical correlation analysis, anomaly detection, etc., as well as a discussion of the value of these algorithms for applications. The material is primarily based on a recent book and on research publications.Students should gain an in-depth familiarity with kernel methods and be able to individually read and discuss in class research papers in the field.
Prerequisites
A good background in university-level mathematics (calculus, elements of linear algebra and optimization), Supervised Learning (4C55/GI01).Grading
The course has the following assessment components: 1) Written Examination (2.5 hours, 50%); 2) Orally assessed coursework (24 hours, 25%) Reports (24 hours, 25%).To pass this course, students must obtain at least 40% on the coursework component and an average of at least 50% when the coursework and exam components of a course are weighted together.
In-class presentations
Each student needs to choose a topic, either based on a chapter/section from the course book or a research paper (see list below). S/he will read and then prepare a presentation. The presentation's aim is to describe and explain the topic to the class (allowing for questions). The following are the requirements:1. Topics are allocated to students on a first-come first-get basis. To be allocated, a student needs to send a request to the lecturer stating the name of the chosen topic. The list will be updated online to indicate which topics are still available. Once a topic has been allocated, there will be no changes.
2. All requests for topic allocation must be sent to the lecturer no later than 12:00am, Monday Jan 24, 2005. The first group of presentations will be on Feb 4. The second group on Feb 11.
3. The presentation needs to be done using power-point on a laptop (will be provided) or on a transparency-projector (preferably using Latex but handwriting is o.k. provided it is clear). The amount of material per slide should be equivalent to approximately 2/4 minutes of presentation time per slide. The time allocated for each presentation is 30-50 minutes.
4. The presentation material needs to be prepared in electronic format having 4 slides per page (preferably in .pdf or .ps but PowerPoint is also possible) and sent to the lecturer at least 48 hours prior to the presentation date. The dates/times of each presentation will be announced in class.
Title Who When Singular value decomposition, Ch. 6.1 and proof of Prop. 6.12 Hassan Feb 4 Direction of max covariance and generalized eigenvalue problem, Ch. 6.3 6.4, pp 155-164 Feng Yuan Feb 4 Canonical correlation analysis, Ch. 6.5, pp 164-175 Xi Chen Feb 11 Fisher discriminant analysis, Ch. 5.4, pp 132-137 Christopher Feb 11 Kernel perceptron algorithm, Ch. 7.4, pp 241-248 Phil Feb 4 Novelty detection, Ch 7.1.1, 7.1.3 Oliver Feb 4 nu-support vector machine, Ch 7.2 Stefania Feb 11 Support vector machine regression, Ch 7.3.3 Jia Zhou Feb 4 Regularized multi-task learning Sylvain Feb 11 Probabilistic outputs for SVMs Cheng Yuan Feb 11 SMO Method to train SVMs Available Learning multiple parameters with SVMs Available Reduced sets methods Available Kernel PCA paper Tao Feb 4 Geometry of SVM classifiers Available Syllabus
The tentative schedule of the course is listed below. Follow the link for each class to find a detailed description, suggested readings, and lecture slides.
Date Title Tuesday, January 11 Kernels in Machine Learning -- Ridge Regression
-- Feature maps
-- Positive definite kernels
-- kernel construction
-- kernel on Euclidean spaces
Friday, January 12 Kernel-Based Learning Algorithms -- Simple operation in feature space
-- Kernel PCA
-- Novelty detection
-- Optimal separating hyperplane
-- Soft margin separation
-- Support vector machines
-- Extensions
Friday, January 21 Reproducing Kernel Hilbert Spaces -- Hilbert spaces
-- Reproducing kernel Hilbert space
-- Mercer theorem
-- Regularization, representer theorem
Friday, February 4 Individual Presentations I -- TBA
Friday, February 11 Individual Presentations II -- TBA
Reading List
Main Book:
- J. Shawe-Taylor and N. Cristianini. Kernel Methods for Pattern Analysis. Cambridge University Press, 2004.
Other suggested references:
- N. Cristianini and J. Shawe-Taylor.An Introduction to Support Vector Machines. Cambridge University Press, 2000.
- T. Hastie, R. Tibshirani, and J. Friedman. The Elements of Statistical Learning: Data Mining, Inference, and Prediction. Springer Series in Statistics, 2002.
- B.Sch\"olkopf and A.J. Smola.Learning with Kernels. MIT Press, 2002.
- V.N. Vapnik.Statistical Learning Theory. Wiley, New York, 1998.
- G. Wahba. Splines Models for Observational Data. Series in Applied Mathematics, Vol. 59, SIAM, Philadelphia, 1990.