MAT 6932/3F93 depx/16134 Special Topics in Applied Mathematics (Spring 2021)

Introduction to Deep Learning and Its Mathematical Foundation (II):
“Deep Learning Methods, Theories and Applications”

Instructor:

Name: Yunmei Chen

E-mail: yun@ufl.edu

Objective and Description of the Course:

To better understand the theoretical foundation of the DNNs inspired by variational methods, optimization algorithms and optimal control techniques this, this course will introduce some necessary knowledge on convex and nonconvex analysis and several popular optimization algorithms for solving certain classes of convex/nonconvex and smooth/nonsmooth optimization problems and optimal control problems arising from deep learning approaches.

Course Material: The topic of this course is one of the rapid developing fields. There is no textbook available. References (recent papers) related to the topics of the course are provided below.

References:

LeCun, Y., Bengio, Y. & Hinton, G. Deep learning. Nature 521, 436–444 (2015).

https://sites.google.com/site/deeplearningsummerschool/

http://www.cs.toronto.edu/~hinton/absps/NatureDeepReview.pdf

M. Abadi, P. Barham, J. Chen, Z. Chen, A. Davis, J. Dean, M. Devin, S. Ghemawat, G. Irving, M. Isard, et al. Tensorﬂow: A system for large-scale machine learning. In OSDI, volume 16, pages 265–283, 2016.

J. Adler and O. Oktem, “Learned primal-dual reconstruction,” ¨ IEEE Transactions on Medical Imaging, vol. 37, no. 6, pp. 1322–1332, 2018.

M. Borgerding, P. Schniter, and S. Rangan, “ AMP-inspired deep networks for sparse linear inverse problems,” IEEE Transactions on Signal Processing, vol. 65, no. 16, pp. 4293–4308, 2017.

J. R. Chang, C.-L. Li, B. Poczos, and B. V. Kumar. One network to solve them all: solving linear inverse problems using deep projection models. In 2017 ICCV, pages 5889–5898. IEEE, 2017.

X. Chen, J. Liu, Z. Wang, and W. Yin. Theoretical linear convergence of unfolded ista and its practical weights and thresholds. In NIPS, pages 9061–9071, 2018.

D. Gilton, G. Ongie, and R. Willett, “Neumann networks for linear inverse problems in imaging,” IEEE Transactions on Computational Imaging, 2019.

Gao Huang, Zhuang Liu, Laurens van der Maaten, and Kilian Q Weinberger. Densely Connected Convolutional Networks. CVPR, pages 2261–2269, 2017.

Jie Hu, Li Shen, and Gang Sun. Squeeze-and-Excitation Networks. CVPR, 2018.

K. Hammernik, T. Klatzer, E. Kobler, M. P. Recht, D. K. Sodickson, T. Pock, and F. Knoll. Learning a variational network for reconstruction of accelerated mri data. Magnetic resonance in medicine, 79(6):3055–3071, 2018.

K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in The IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 770–778.

K. He, X. Zhang, S. Ren, and J. Sun. Identity mappings in deep residual networks. In European conference on computer vision, pages 630–645. Springer, 2016.

Y. Li, M. Tofighi, J. Geng, V. Monga, and Y. C Eldar. Efficient and Interpretable Deep Blind Image Deblurring via Algorithm Unrolling. IEEE Trans. Comput. Imaging, vol. 6, pages 666–681, 2020.

T. Meinhardt, M. Moller, C. Hazirbas, and D. Cremers. Learning proximal operators: Using denoising networks for regularizing inverse imaging problems. In ICCV, pages 1781–1790, 2017.

Saining Xie, Ross B Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated Residual Transformations for Deep Neural Networks. CVPR, pages 5987–5995, 2017.

Y. Yang, J. Sun, H. Li, and Z. Xu. Deep admm-net for compressive sensing mri. In D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett, editors, NIPS 29, pages 10–18. Curran Associates, Inc., 2016.

J. Zhang and B. Ghanem. Ista-net: Interpretable optimization-inspired deep network for image compressive sensing. In CVPR, 2018.

J. Zhang, C. Zhao, and W. Gao. Optimization-inspired compact deep compressive sensing. IEEE Journal of Selected Topics in Signal Processing, 2020.

K. Zhang, W. Zuo, S. Gu, and L. Zhang. Learning deep cnn denoiser prior for image restoration. In CVPR, pages 3929–3938, 2017.

Z. Zhang, Y. Liu, J. Liu, F. Wen, and C. Zhu. Amp-net: Denoising based deep unfolding for compressive image sensing. arXiv preprint arXiv:2004.10078, 2020.

S. Zhou, Y. He, Y. Liu, and C. Li. Multi-channel deep networks for block-based image compressive sensing. arXiv preprint arXiv:1908.11221, 2019.

D. Mumford and J. Shah, “Optimal approximations by piecewise smooth functions and associated variational problems,” Communi- cations on Pure and Applied Mathematics, vol. 42, no. 5, pp. 577–685, 1989.

T. F. Chan and L. A. Vese, “Active contours without edges,” IEEE Transactions on Image Processing, vol. 10, no. 2, pp. 266–277, 2001.

S. Esedoglu and F. Otto, “Threshold dynamics for networks with arbitrary surface tensions,” Communications on Pure and Applied Mathematics, vol. 68, no. 5, pp. 808–864, 2015.

O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in International Conference on Medical Image Computing and Computer-Assisted In- tervention. Springer, 2015, pp. 234–241.

F. Jia, J. Liu, and X.-C. Tai, “A regularized convolutional neu- ral network for semantic image segmentation,” arXiv preprint arXiv:1907.05287, 2019.

J.Liu, X.Wang and X.-C. Tai, “Deep Convolutional Neural Networks with Spatial Regularization, Volume and Star-shape Priori for Image Segmentation” arXiv:2002.03989v1, 2020.

Lawrence C. Evans, An Introduction to Mathematical
Optimal Control Theory, https://math.berkeley.edu/~evans/control.course.pdf

T. Q. Chen, Y. Rubanova, J. Bettencourt and D. K. Duvenaud, Neural ordinary differential
equations, in Advances in Neural Information Processing Systems, 2018, 6572–6583.

Q. Li, L. Chen, C. Tai and E. Weinan, Maximum principle based algorithms for deep learning,
The Journal of Machine Learning Research, 18 (2017), 5998–6026.

M. Benning et al., Deep learning as optimal control problems: models and numerical methods, arXiv:1904.05657v3.

Stephen Boyd, Lieven Vandenberghe, Convex Optimization, Cambridge University Press, ISBN-13: 978-0521833783.

Amir Beck, First-Order Methods in Optimization, MOS-SIAM Series on Optimization, 2017.

Leonid I. Rudin, Stanley Osher, and Emad Fatemi. Nonlinear total variation based noise removal algorithms. Physica D, 60(1):259-268, 1992.

T. Goldstein and S. Osher, “The split Bregman method for l1 regularized problems,” SIAM Journal on Imaging Sciences, vol. 2, pp. 323–343, 2009.

A. Beck and M. Teboulle. A fast iterative shrinkage thresholding algorithm for linear inverse problems. SIAM Journal on Imaging Sciences, 2(1):183–202, 2009.

M. R. Hestenes, Multiplier and gradient methods, J. Optim. Theory Appl., 4 (1969), 303–320

A. Chambolle and T. Pock, A ﬁrst-order primal-dual algorithm for convex problems with applications to imaging, J. Math. Imaging Vision, 40 (2011), 120–145.

Meeting Time and Rooms:

MWF 4 online (Synchronous)
Office Hours: MWF 5 or by appointment

Arrangement of the Course::

Unit 1: Basic concepts on convex and nonconvex analysis: (Tentatively week 1-4)

2.1. Convex functions: Definition and basic properties of convex functions, convex conjugate function, properties and calculation of conjugate function;(weeks 1-1.5)

2.2. Functions with Lipschitz continuous gradient: Definition and properties; (weeks 1.5-2)

2.3. Nonsmooth convex and non-convex functions: Sub-gradient and sub-differential for nonsmooth convex function – definition and basic properties, sub-gradient calculus, duality and optimality conditions; Generalized directional derivatives and subgradients of nonconvex nonsmooth functions; (week 3-4)

Unit 2: Variational models and first order optimization algorithms inspired deep neural networks and their applications: (Tentatively week 5-9)

2.1. Gradient decent method, proximal gradient method, accelerated gradient decent methods for unconstrained convex and set constrained convex and nonconvex optimizations: Optimality conditions, local minimum and global minimum, schemes, shrinkage operator and convergence analysis; (weeks 5-6)

2.2. Proximal gradient and gradient decent algorithms inspired convolutional networks for solving inverse problems: Architecture and training of the CNNs for solving inverse problems, case study: Prox-net, ISTA-net, OPINE-net, Neumann-net and Variational-net; (weeks 6-7)

2.3. Alternating direction method of multipliers (ADMM) for equality constrained convex optimization and ADMM inspired CNNs: Architecture and training of the CNNs for solving inverse problems, case study: ADMM-net; (week 8)

2.4. Primal-dual algorithm and Primal-dual algorithm inspired CNNs: Architecture and training of the CNNs for solving inverse problems, case study: PD-nets; (week 9)

Unit 3: Variational methods and variational CNN for image segmentation: (Tentatively week 10-11)

3.1. Mumford-Shah model, Chan-Vese model and Potts model for image segmentation/partition; (week 10)

3.2 Variational CNN for image segmentation and case study: Total variation and soft threshold dynamics regularized variational networks for image segmentation, case study; (week 11)

Unit 4: Deep learning from optimal control perspective: (Tentatively week 12-16)

4.1. Brief introduction mathematical control theory: Controlled dynamics, Calculus of variation, Hamiltonian dynamics, Pontrygin Maximum Principle (PMP), free time fixed endpoint problem, applications; (weeks 12-13)

4.2.Deep neural network and numerical ODE: Connection of discrete dynamical systems with DNN, Neural ODEs for supervised learning and case study: Neural Ordinary Differential Equations; (week 14)

4.3. Deep Learning as optimal control problems: Representability and controllability Problem, Maximum Principle based models and numerical methods, applications for image classification, restoration and reconstruction; (week 15-16)

Additional Information:

Grading:
Students will be required to a presentation related to the course content or problems of particular interest to the individual student. Grades will be assigned on the basis of the presentation. More information on UF grading policy may be found at:
UF Graduate Catalog
Grades and Grading Policies

Honor Code: “UF students are bound by The Honor Pledge which states, “We, the members of the University of Florida community, pledge to hold ourselves and our peers to the highest standards of honor and integrity by abiding by the Honor Code. On all work submitted for credit by students at the University of Florida, the following pledge is either required or implied: “On my honor, I have neither given nor received unauthorized aid in doing this assignment.” The Honor Code specifies a number of behaviors that are in violation of this code and the possible sanctions. Furthermore, you are obligated to report any condition that facilitates academic misconduct to appropriate personnel. If you have any questions or concerns, please consult with the instructor or TAs in this class.”

Class Attendance: “Requirements for class attendance and make-up exams, assignments, and other work in this course are consistent with university policies that can be found at: https://catalog.ufl.edu/ugrad/current/regulations/info/attendance.aspx.”

Accommodations for Students with Disabilities: “Students with disabilities requesting accommodations should first register with the Disability Resource Center (352-392-8565, https://www.dso.ufl.edu/drc/) by providing appropriate documentation. Once registered, students will receive an accommodation le

Online Evaluations: “Students are expected to provide feedback on the quality of instruction in this course by completing online evaluations at https://evaluations.ufl.edu. Evaluations are typically open during the last two or three weeks

Contact information for the Counseling and Wellness Center: https://counseling.ufl.edu/, 392-1575; and the University Police Department: 392-1111 or 9-1-1 for emergencies.

Diversity:
I and the department of Mathematics are committed to diversity and inclusion of all students in this course. I acknowledge, respect, and value the diverse nature, background and perspective of students and believe that it furthers academic achievements. It is our intent to present materials and activities that are respectful of diversity: race, color, creed, gender, gender identity, sexual orientation, age, religious status, national origin, ethnicity, disability, socioeconomic status, and any other distinguishing qualities.

Class Recording:
Our class sessions may be audio-visually recorded for students in the class to refer back and for enrolled students who are unable to attend live. Students who participate with their camera engaged or utilize a profile image are agreeing to have their video or image recorded. If you are unwilling to consent to have your profile or video image recorded, be sure to keep your camera off and do not use a profile image. Likewise, students who un-mute during class and participate orally are agreeing to have their voice recorded. If you are not willing to consent to have your voice recorded during class, you will need to keep your mute button activated and communicate exclusively using the “chat” feature, which allows students to type questions and comments live. The chat will not be recorded or shared. As in all courses, unauthorized recording and unauthorized sharing of recorded materials by students or any other party is prohibited.

Software Use
All faculty, staff, and students of the University are required and expected to obey the laws and legal agreements governing software use. Failure to do so can lead to monetary damages and/or criminal penalties for the individual violator. Because such violations are also against University policies and rules, disciplinary action will be taken as appropriate. We, the members of the University of Florida community, pledge to uphold ourselves and our peers to the highest standards of honesty and integrity.

Student Privacy
There are federal laws protecting your privacy with regards to grades earned in courses and on individual assignments. For more information, please see the Notification to Students of FERPA Rights.

Campus Resources:
Health and Wellness

U Matter, We Care:
If you or a friend is in distress, please contact umatter@ufl.edu or 352 392-1575 so that a team member can reach out to the student.

Counseling and Wellness Center: counseling.ufl.edu/cwc, and 392-1575; and the University Police Department: 392-1111 or 9-1-1 for emergencies.

Sexual Assault Recovery Services (SARS)
Student Health Care Center, 392-1161.

University Police Department at 392-1111 (or 9-1-1 for emergencies), or police.ufl.edu.

Academic Resources
E-learning technical support, 352-392-4357 (select option 2) or e-mail to Learning-support@ufl.edu.

Career Resource Center, Reitz Union, 392-1601. Career assistance and counseling.

Library Support, Various ways to receive assistance with respect to using the libraries or finding resources.

Teaching Center, Broward Hall, 392-2010 or 392-6420. General study skills and tutoring.

Writing Studio, 302 Tigert Hall, 846-1138. Help brainstorming, formatting, and writing papers.

Student Complaints Campus

On-Line Students Complaints

Introduction to Deep Learning and Its Mathematical Foundation (II): “Deep Learning Methods, Theories and Applications”

Introduction to Deep Learning and Its Mathematical Foundation (II):
“Deep Learning Methods, Theories and Applications”