Math 535 Statistical Learning and Data Mining. Spring 2023.

• Instructor: Xingye Qiao

• Office: WH-134

• Meeting time & location: Tuesday and Thursday 8:10–9:55 am at WH–100E.

• Office hours: the 1/2-hours before and after the class meetings, or by appoitment.

Prerequisite

1. Scientific programming in a language such as R, Matlab, or Python. R and Python are strongly recommended.

2. Linear regression and its inference

3. Matrix algebra, preferrably including orthogonality, eigenvalues and eigenvectors, and singular value decomposition.

Description

This course is a survey of statistical learning and data mining methods. It will cover major statistical learning methods and concepts for both supervised and unsupervised learning. Topics covered include regression methods with sparsity or other regularizations, model selection, graphical models, statistical learning pipeline and best practice, introduction to classification, including discriminant analysis, logistic regression, support vector machines, and kernel methods, nonlinear methods, dimension reduction, including matrix factorization-based approaches - principal component analysis and non-negative matrix factorization-, multidimensional scaling, and independent component analysis, clustering, decision trees, random forest, boosting and ensemble learning.

List of Topics

• ordinary least square regression;

• penalized regression: ridge regression; lasso; other sparse regression methods;

• (optional) Gaussian graphical models;

• model selection and assessment; model validation; cross-validation;

• (optional) generalized linear models;

• classification: Bayes classifier, linear/quadratic discriminant analysis, logistic regression, support vector machines

• nonlinear methods: kernel methods, splines, smoothing, kernel ridge regression, GAM

• (optional) deep neural networks;

• dimension reduction: principal component analysis, nonnegative matrix factorizations, independent component analysis, multidimensional scaling

• clustering: K-means, hierarchical clustering, Gaussian mixtures

• Ensemble methods: classification and regression trees, bagging, random forest, boosting, Gradient Boosting

• (optional) compressed sensing, etc.

• (optional) Text as data.

Learning Outcomes

Students will learn how and when to apply statistical learning techniques, their comparative strengths and weaknesses, and how to critically evaluate the performance of learning algorithms. Students completing this course should be able to

• process and visualize different data types,

• apply basic statistical learning methods to build predictive models or perform exploratory analysis,

• have basic understanding of the underlying mechanism of predictive models and evaluate and interpret such models,

• properly tune, select and validate statistical learning models,

• use analytical tools and software widely used in practice,

• work both independently and in a team to solve problems, and

• learn to present and communicate the findings effectively.

Recommended Texts

• James, Witten, Hastie and Tibshirani, 2014. An Introduction to Statistical Learning with Applications in R. Book Home Page. The PDF file of the book can be downloaded for free. There is also a R library for this book.

• Hastie, Trevor, Tibshirani, Robert, and Friedman, J. H. 2009. The elements of statistical learning: Data mining, inference, and prediction. New York, NY: Springer New York.

• Hastie, Trevor, Tibshirani, Robert, and Wainwright, Martin. Statistical Learning with Sparsity: The Lasso and Generalizations. Chapman and Hall/CRC; 1 edition (May 7, 2015). The PDF file of the book can be downloaded for free.

• Bühlmann, Peter, and van de Geer, Sara. Statistics for High-Dimensional Data. Springer-Verlag Berlin Heidelberg.

• Boyd, Stephen, and Vandenberghe, Lieven. Convex Optimization. Cambridge University Press. The PDF file of the book can be downloaded for free.

• D'Ignazio, C. and Klein, L.F., (2020). Data Feminism. MIT Press. Free copy online: https:data-feminism.mitpress.mit.edu/ Chapter 1 is required reading.

Piazza

Please use Piazza (www.piazza.com) for all communications with me rather than email. Piazza is a question-and-answer platform. It supports LaTeX, code formatting, embedding of images, and attaching of files. You are encouraged to ask questions when you have difficulty understanding a concept or working around a piece of code – you can even ask questions anonymously. Moreover, you can also answer questions from your classmates. I constantly monitor the answers and endorse those good answers.

Announcement will be sent to the class using Piazza.

Gradescope

We will use Gradescope to submit and grade homework. This will allow the instructor to efficient grade all the work and give feedback in a timely manner.

Mycourses(Blackboard)

Mycourses(Blackboard) will only be used for recording grades on assignments and exams and for distributing solutions. The code and lecture notes can also be found on blackboard.

Grading

• Homework (40%): homework is assigned biweekly.

• Midterm exam (30%): a midterm exam focusing on the theoretical part of the course will be administered.

• Contest & Reports (30%): a group project will be assigned to each student. Successful completion of the project includes an initial report, a presentation and a final report.

Homework Policy

There will be a deduction of 15% of the grade for each day homeworks are late (the final grade for a late homework that is N days late will be 0.85^N times the real grade). Homeworks may be discussed with classmates but must be written and submitted individually.

Final Project

Students will compete against each other in a the Final Project. It can be completed in teams of 2 – 4 members. Grades will be based upon a progress report and a final report (one per team) as well as the contest results. Further details about the Final Project along with specific grading criteria will be given in a separate document and discussed in class.

Software

RStudio and Google Colab will be used for completion of the homework assignments.