Integration of Variational Methods and Deep Learning Approaches for Image Processing (Part II)

Objective and Description of the Course:

The variational methods have been important tools for solving image analysis problems. They enjoy the advantage of strong theoretical convergence results. However, the quality of the solutions highly depends on the sophisticated models for priors, which however is sometimes absent in many realistic applications. The learning based methods using deep architecture dramatically reduce time complexity, while achieving impressive results. However, training a deep neuron network (DNN) requires large number of labeled training data to estimate millions or billions of parameters. Moreover, DNNs usually are sensitive to the specific problem, in contrast, a typical variational model can solve a class of problems.To bridge the gap between variational methods and learning based methods and take advantage of both, very recently, research on the integration of these two types of methods has attract great attentions.

This course aims to study the current development in this direction. Continuing the study in the previous course (MAP7436, Fall 2018), We will learn the design of variational DNNs that integrates optimization algorithms into a DNN structures to make the DNNs more interpretable and generalizable for image restoration, reconstruction, segmentation or classification. To do this, it requires some basic knowledge on both of variational methods and deep learning approaches in computer vision. Therefore, the course mainly consists of the following three components:
(1). Popular variational models and effective optimization algorithms for image restoration, reconstruction and segmentation;
(2). Learning based algorithms and architectures of DNNs, and their applications to image restoration, reconstruction and segmentation;
(3). Integration of (variational) model based and (deep) learning based methods. Design and training interpretable deep learning networks inspired by variational models and optimization algorithms for image restoration, reconstruction and segmentation.

The topic of this course is one of the rapid developing fields. There is no textbook available. I will provide some references (the papers from Nips, CVPR and ICCV 2016-2018). Students presentations, discussions and projects are required.

References:

Nikos Paragios, Yunmei Chen, and Olivier Faugeras, The Handbook of Mathematical Models in Computer Vision. Springer 2006.

Otmar Scherzer: Handbook of Mathematical Methods in Imaging. Springer 2015.

A. Beck and M. Teboulle. A fast iterative shrinkage thresholding algorithm for linear inverse problems. SIAM Journal on Imaging Sciences, 2(1):183–202, 2009.

J. Eckstein, Augmented Lagrangian and alternating direction methods for convex optimization: a tutorial and some illustrative computational results, 2012.

A. Chambolle and T. Pock, A first-order primal-dual algorithm for convex problems with applications to imaging, J. Math. Imaging Vision, 40 (2011), 120–145.

Diederik P Kingma and Jimmy Ba, Adam: A Method for Stochastic Optimization, ICLR 2015, arXiv:1412.6980v9 [cs.LG] 30 Jan 2017.

 

Generative Adversarial Networks, http://papers.nips.cc/paper/5423-generative-adversarial-nets.pdf, NIPS 2016

Generative Adversarial Networks (GANs), Ian Goodfellow, NIPS 2016 tutorial

J. H. R. Chang, C.-L. Li, B. P´oczos, B. V. K. V. Kumar, and A. C. Sankaranarayanan. One network to solve them all — solving linear inverse problems using deep projection models. ICCV, 2017.

 

S.Wang, S.Fidler and R.Urtasun. Proximal deep structured models. In NIPS, pages 865–873, 2016.

Non-Convex Rank/Sparsity Regularization and Local Minima, ICCV, 2017.

K. Zhang,W. Zuo, S. Gu, and L. Zhang. Learning deep CNN denoiser prior for image restoration. CVPR, 2017.

Y. Yang, J. Sun, H. Li, and Z. Xu. Deep ADMM-Net for compressive sensing MRI. In NIPS, pages 10–18, 2016.

T. Meinhardt, M. Möller, C. Hazirbas, D. Cremers, Learning Proximal Operators: Using Denoising Networks for Regularizing Inverse Imaging Problems, ICCV2017.

Kronecker-Decomposable Component Analysis for Low-Rank Modeling, ICCV2017.

F. Wang, H. Huang, J. Liu, Variational based Mixed Noise Removal with CNN Deep Learning Regularization, arXiv.org > cs > arXiv:1805.08094 (2018).

Kerstin Hammernik et. al., Learning a Variational Network for Reconstruction
of Accelerated MRI Data, Magnetic Resonance in Medicine 79:3055–3071, 2018.

Jian Zhang, Bernard Ghanem, ISTA-Net: Interpretable Optimization-Inspired Deep Network for Image Compressive Sensing, CVPR 2018.

Universal Denoising Networks : A Novel CNN Architecture for Image Denoising, CVPR 2018.

 

Learning deep structured active contours end-to-end, 2018CVPR

Context Encoding for Semantic Segmentation, 2018CVPR

Weakly Supervised Instance Segmentation using Class Peak Response, 2018CVPR

Context Contrasted Feature and Gated Multi-scale Aggregation for Scene Segmentation, 2018CVPR

Learning to Adapt Structured Output Space for Semantic Segmentation, 2018CVPR

Non-local Neural Networks, 2018CVPR

 

 Meeting Time and Rooms:

•   MWF 5 at LIT 223
• Office Hours: MWF 4 or by appointment

 

  Arrangement of the course::

• Unit 1: Variational methods for image restoration and reconstruction (continuation from last semester MAP7438 2018 Fall) (Tentatively week 1-4)

1.1. Convex and non-convex sparsity inducing regularization: L1 norm, nuclear norm, largest eigenvalue, and folded concave penalties in original domain and transformed domains;

1.2. Optimization algorithms and related theories for model solutions;

1.2.1. Proximal gradient method, Shrinkage operator, iterative soft-thresholding algorithm and iterative hard-thresholding algorithm;

1.2.2. Alternating minimization algorithm(AMA), Alternating direction method of multipliers (ADMM) for equality constrained convex optimization;

1.2.3. First-order primal-dual algorithms, Primal-dual hybrid gradient (PDHG) algorithm and Chambolle’s method for solving dual problem

1.2.4. Applications to regularized linear inversion problems in image restoration/reconstruction, MRI and CT image reconstruction with undersampled data.

• Unit 2: Variational methods for image segmentation (Tentatively week 5-8)

2.1. Edge based models: Geodesic Active Contour model, variational level set method.

2.2. Region based models: Mumford-Shah (MS) model for simultaneous smoothing and segmentation, CV-model, region competition model.

2.3. Region based active contour with parametric density estimatior, region based active contour with non-parametric density estimatior, information theoretical approach, Shannon’s entropy and mutual information.

2.4 Soft segmentation methods.

• Unit 3: Learning based methods – deep neuron network approaches for image segmentation (Tentatively week 9-11)

3.1. Context encoding for semantic segmentation

3.2. Context contrasted feature and gated multi-scale aggregation for scene segmentation

3.3. Learning to adapt structured output space for semantic segmentation

3.4. Nonlocal network for segmentation and objective detection

• Unit 4: Integration of variational based and learning based methods – for image segmentation, restoration and reconstruction (Tentatively week 12-16)

4.1. Learning proximal operator using CNN denoiser

4.2. Revisit and beyond ISTA-net, ADMM-net, and variational-net for image reconstruction, and discussion on possible applications to MRI and CT segmentation

4.3. Learning deep structured active contours for segmentation

 
Additional Information:
 
Grading:
Students will be required to present one to two papers or projects related to the course content. The projects may be related to problems of particular interest to the individual student. Grades will be assigned on the basis of the presentations or projects. Current UF grading policies can be found from the following link https://catalog.ufl.edu/ugrad/current/regulations/info/grades.aspx.
 
Honor Code: “UF students are bound by The Honor Pledge which states, “We, the members of the University of Florida community, pledge to hold ourselves and our peers to the highest standards of honor and integrity by abiding by the Honor Code. On all work submitted for credit by students at the University of Florida, the following pledge is either required or implied: “On my honor, I have neither given nor received unauthorized aid in doing this assignment.” The Honor Code specifies a number of behaviors that are in violation of this code and the possible sanctions. Furthermore, you are obligated to report any condition that facilitates academic misconduct to appropriate personnel. If you have any questions or concerns, please consult with the instructor or TAs in this class.”
 
Class Attendance: “Requirements for class attendance and make-up exams, assignments, and other work in this course are consistent with university policies that can be found at: https://catalog.ufl.edu/ugrad/current/regulations/info/attendance.aspx.”
 
Accommodations for Students with Disabilities: “Students with disabilities requesting accommodations should first register with the Disability Resource Center (352-392-8565, https://www.dso.ufl.edu/drc/) by providing appropriate documentation. Once registered, students will receive an accommodation letter which must be presented to the instructor when requesting accommodation. Students with disabilities should follow this procedure as early as possible in the semester.”
 
Online Evaluations: “Students are expected to provide feedback on the quality of instruction in this course by completing online evaluations at https://evaluations.ufl.edu. Evaluations are typically open during the last two or three weeks of the semester, but students will be given specific times when they are open. Summary results of these assessments are available to students at https://evaluations.ufl.edu/results/.”
 
Contact information for the Counseling and Wellness Center: https://counseling.ufl.edu/, 392-1575; and the University Police Department: 392-1111 or 9-1-1 for emergencies.