Meeting time & location: Tuesday and Thursday 8:20–9:55 am at WH–G02.
Office hours: Thursday 10–11:30 am
This course is a 4-credit course, which means that in addition to the scheduled lectures/discussions,
students are expected to do at least 9.5 hours of course-related work each week during the
semester. This includes things like: completing assigned readings, participating in lab sessions,
studying for tests and examinations, preparing written assignments, completing internship or
clinical placement requirements, and other tasks that must be completed to earn credit in the
course.
Prerequisite
Scientific programming in a language such as R, Matlab, or Python.
Linear regression and its inference
Matrix algebra, preferrably including orthogonality, eigenvalues and eigenvectors, and singular value decomposition.
Description
This course is a survey of statistical learning and data mining methods. It will cover major statistical learning methods and concepts for both supervised and unsupervised learning. Topics covered include regression methods with sparsity or other regularizations, model selection, graphical models, statistical learning pipeline and best practice, introduction to classification, including discriminant analysis, logistic regression, support vector machines, and kernel methods, nonlinear methods, dimension reduction, including matrix factorization-based approaches - principal component analysis and non-negative matrix factorization-, multidimensional scaling, and independent component analysis, clustering, decision trees, random forest, boosting and ensemble learning.
Learning Outcomes
Students will learn how and when to apply statistical learning techniques, their comparative strengths and weaknesses, and how to critically evaluate the performance of learning algorithms. Students completing this course should be able to
process and visualize different data types,
apply basic statistical learning methods to build predictive models or perform exploratory analysis,
have basic understanding of the underlying mechanism of predictive models and evaluate and interpret such models,
properly tune, select and validate statistical learning models,
use analytical tools and software widely used in practice,
work both independently and in a team to solve problems, and
learn to present and communicate the findings effectively.
Recommended Texts
James, Witten, Hastie and Tibshirani, 2014. An Introduction to Statistical Learning with Applications in R. Book Home Page. The PDF file of the book can be downloaded for free. There is also a R library for this book.
Hastie, Trevor, Tibshirani, Robert, and Friedman, J. H. 2009. The elements of statistical learning: Data mining, inference, and prediction. New York, NY: Springer New York.
Hastie, Trevor, Tibshirani, Robert, and Wainwright, Martin. Statistical Learning with Sparsity: The Lasso and Generalizations. Chapman and Hall/CRC; 1 edition (May 7, 2015). The PDF file of the book can be downloaded for free.
Bühlmann, Peter, and van de Geer, Sara. Statistics for High-Dimensional Data. Springer-Verlag Berlin Heidelberg.
Boyd, Stephen, and Vandenberghe, Lieven. Convex Optimization. Cambridge University Press. The PDF file of the book can be downloaded for free.
Piazza
Please use Piazza (www.piazza.com) for all communications with me rather than email. Piazza is a question-and-answer platform. It supports LaTeX, code formatting, embedding of images, and attaching of files. You are encouraged to ask questions when you have difficulty understanding a concept or working around a piece of code – you can even ask questions anonymously. Moreover, you can also answer questions from your classmates. I constantly monitor the answers and endorse those good answers.
Announcement will be sent to the class using Piazza.
Gradescope
We will use Gradescope to submit and grade homework. This will allow the instructor to efficient grade all the work and give feedback in a timely manner.
Mycourses(Blackboard)
Mycourses(Blackboard) will only be used for recording grades on assignments and exams and for distributing solutions. The code and lecture notes can also be found on blackboard.
Grading
Homework (30%): homework is assigned biweekly.
Midterm exam (30%): a midterm exam focusing on the theoretical part of the course will be administered.
Contest & Reports (35%): a group project will be assigned to each student. Successful completion of the project includes an initial report, a presentation and a final report.
Lecture attendance and participation (5%): attendance will be taken. Active participation on asking meaning questions and answering others’ questions will also count.
Homework Policy
There will be a deduction of 25% of the grade for homeworks that are not typeset using LaTeX. There will be a deduction of 15% of the grade for each day homeworks are late (the final grade for a late homework that is N days late will be 0.85^N times the real grade). Homeworks may be discussed with classmates but must be written and submitted individually.
Data Analysis Contest
Students will compete against each other in a Data Analysis Contest. The competition will begin on Tuesday, Feburary 20 and can be completed in teams of 2 – 4 members. Grades will be based upon a progress report and a final report (one per team) as well as the contest results. Further details about the contest along with specific grading criteria will be given in a separate document and discussed in class.
Software
Jupyter notebook will be used to illustrate both R and Python codes in class. I recommend installing Jupyter notebook via Anaconda.
Students are expected to write homework in LaTex. For users with no experience with LaTex, I suggest using the cloud LaTex editing service at overleaf.com.
Tentative schedule
Lecture
Week
Date
Module
Tentative Topic
Assigned
Due
1
1
Jan-21, Tuesday
I. Regression with Sparsity
Introduction
HW0
2
Jan-23, Thursday
MSE & Least Square
3
2
Jan-28, Tuesday
Ridge Regression
HW1
HW0
4
Jan-30, Thursday
Sparse Regression I
5
3
Feb-4, Tuesday
Sparse Regression II
6
Feb-6, Thursday
Graphical Models & Compressed Sensing
7
4
Feb-11, Tuesday
II. Pipeline for Statistical Learning
Model Selection and Assessment
HW 2
HW 1
8
Feb-13, Thursday
Model Validation
9
5
Feb-18, Tuesday
Case Studies & Logistic Regression
10
Feb-20, Thursday
III. Classification Methods
LR Computing & Other GLMs
11
6
Feb-25, Tuesday
Sparse GLM & Bayes Classifier
HW 3
HW2
12
Feb-27, Thursday
LDA
13
7
Mar-3, Tuesday
SVM I: linear SVM
NA
Mar-5, Thursday
NO CLASS
14
8
Mar-10, Tuesday
SVM II: dual solution, kernel SVM
15
Mar-12, Thursday
IV. Nonlinear Methods
Nonlinear I: RKHS, KRR, Kernel PCA
HW 4
HW3
16
9
Mar-17, Tuesday
Nonlinear II: Polynomial reg., smoothing, GAM, etc.