# Machine Learning

## Announcements:

- Re-Exam results
- Exam results
**There will be no lectures on the 28th and 30th of January. Furthermore no tutorials will take place in the same week.****The registration for the exercise groups has closed. Find your choice in this list: Slot assignments****The exercise groups have started on Wednesday, the 24th of October****Please subscribe to our machine learning google group, which will be used as a means to discuss the exercise content and we are likely going to post course updates there: Google group.**

## Overview

In a broader perspective machine learning tries to automatize the process of empirical sciences – namely extracting knowledge about natural phenomena from measured data with the goal to either understand better the underlying processes or to make good predictions. Machine learning methods are therefore widely used in different fields: bioinformatics, computer vision, information retrieval, computer linguistics, robotics,…

The lecture gives a broad introduction into machine learning methods. After the lecture the students should be able to solve and analyze learning problems. The lecture is based on the machine learning lecture of Matthias Hein.

## Course Information

- Semester: winter term
- Year: 2018/2019
- Type: Core lecture (Stammvorlesung), 9 credit points
- Time and Location:
- Lectures
- Monday 2 pm - 4 pm, lecture hall E 1.3 HS002
- Wednesday 10 am - 12 am, lecture hall E 1.3 HS002

- Exercise groups
- Wednesday 8 am - 10 am, E 1.4 room 024
- Thursday 8 am - 10 am, E 1.4 room 024
- Friday 4 pm - 6 pm, E 1.4 room 024

- Exams
- 26.02.19 10:00 - 13:00, Günter Hotz Hörsaal E 2.2 0.01 and E 2.5 HS I
- 28.03.19 10:00 - 13:00, Günter Hotz Hörsaal E 2.2 0.01

- Lectures

- Exercises
- handed in in groups of 3
- 50% of the points (up to that point) are needed to take part in the exams (end-term / re-exam).

- Lecturers:
- Tutors:
- Apratim Bhattacharyya
- Max Losch
- Maksym Andriushchenko

## List of topics (tentative)

- Reminder of probability theory
- Maximum Likelihood/Maximum A Posteriori Estimators
- Bayesian decision theory
- Linear classification and regression
- Kernel methods
- Model selection and evaluation of learning methods
- Feature selection
- Nonparametric methods
- Boosting, Decision trees
- Neural networks
- Structured Output
- Semi-supervised learning
- Unsupervised learning (Clustering, Independent Component Analysis)
- Dimensionality Reduction and Manifold Learning
- Statistical learning theory

Previous knowledge of machine learning is not required. The participants should be familiar with linear algebra, analysis and probability theory on the level of the local `Mathematics for Computer Scienticists I-III’ lectures. In particular, attendees should be familiar with

- Discrete and continuous probability theory (marginals, conditional probability, random variables, expectation etc.)

The first three chapters of: L. Wasserman: All of Statistics, Springer, (2004) provide the necessary background - Linear algebra (rank, linear systems, eigenvalues, eigenvectors (in particular for symmetric matrices), singular values, determinant)

A quick reminder of the basic ideas of linear algebra can be found in the tutorial of Mark Schmidt (I did not check it for correctness!). Apart from the LU factorization this summarizes all what is used in the lecture in a non-formal way. You might also find the following sheets on matrix identities and gaussian identities useful from Sam Roweis useful. - Multivariate analysis (integrals, gradient, Hessian, extrema of multivariate functions)

## Lecture

### Notes

- Machine Learning Notes

Acknowledgement: We are grateful to Matthias Hein for providing the Machine Learning lecture notes

### Slides

- Introduction-17-10-18
- Probaility-Recap-22-10-18
- Bayes-Decision-24-10-18
- Empirical-Risk-Minimization-29-10-18
- Linear-Regression-31-10-18
- Smooth-Optimization-05-11-18
- Linear-Classification-12-11-18
- Linear-Classification-SVM-14-11-18
- SVMKernel-19-11-18
- Kernel2-21-11-18
- PerfMeaStatTest-26-11-18
- CompClassModels-28-11-18
- Feature Selection-03-12-18
- Boosting-10-12-18
- DecisionTrees-12-12-18
- Semisupervised-learning-17-12-18
- Spectral-clustering-19-12-18
- Hierarchical-clustering-7-1-19
- Dimensionality-Reduction-09-01-19
- additional slides on Laplacian Eigenmaps

- Backpropagation-and-NeuralNetworks-14-01-19
- Convolutional Neural Networks
- Stochastic Gradient Descent and Training Neural Networks
- Statistical Learning Theory

## Exercises

- Exercise sheet 0 -- (Solution)
- Exercise sheet 1 (due on 29th October) -- (Solution)
- Exercise sheet 2 (due on 5th November) -- (Solution)
- Exercise sheet 3 (due on 12th November)(Data)(cvx package) -- (Solution)
- Exercise sheet 4 (due on 19th November)(Data) -- (Solution)
- Exercise sheet 5 (due on 26th November)(Data) -- (Solution)
- Exercise sheet 6 (due on 3rd December)(Data) -- (Solution)
- Exercise sheet 7 (due on 10th December)(Data) -- (Solution)
- Exercise sheet 8 (due on 17th December)(Data) -- (Solution)
- Exercise sheet 9 (due on 7th January)(Data) -- (Solution)
- Exercise sheet 10 (due on 14th January)(Data) -- (Solution)
- Exercise sheet 11 (due on 21th January)(Data) -- (Solution)
- Exercise sheet 12 (due on 4th February) -- (Solution)

## Literature

The lecture will be partially based on the following books and partially on recent research papers:

- R.O. Duda, P.E. Hart, and D.G.Stork: Pattern Classification, Wiley, (2000).
- B. Schoelkopf and A. J. Smola: Learning with Kernels, MIT Press, (2002).
- J. Shawe-Taylor and N. Christianini: Kernel Methods for Pattern Analysis, Cambridge University Press, (2004).
- C. M. Bishop: Pattern recognition and Machine Learning, Springer, (2006).
- T. Hastie, R. Tibshirani, J. Friedman: The Elements of Statistical Learning, Springer, second edition, (2008).
- L. Devroye, L. Gyoerfi, G. Lugosi: A Probabilistic Theory of Pattern Recognition, Springer, (1996).
- L. Wasserman: All of Statistics, Springer, (2004).
- S. Boyd and L. Vandenberghe: Convex Optimization, Cambridge University Press, (2004).

## Other resources

- Matlab is accessible via our campus license. Details how to use it can be found here

Access from outside should be possible via ssh: ssh -X username@computername.studcs.uni-sb.de - Material for Matlab:
- Slides (Theory part ) and (Practical part ) from an (outdated) Matlab tutorial.
- Matlab tutorial by David F. Griffiths