Integration of Variational Method and Deep Learning Approach for Image Processing

Objective and Description of the Course:

This course aims at studying the integration of variational (energy minimization) method and deep learning approach for image processing, especially medical image processing. We will discuss how to design deep variational network that combines the mathematical structure of variational models and algorithms with deep learning approach to make the learning more efficient. This requires some basic knowledge on both of variational methods for image processing and deep learning in computer vision. Therefore, the course mainly consists of the following four components:
(1). Popular variational models for image restoration, reconstruction and segmentation/partition;
(2). Effective optimization techniques for model solutions (such as ADMM, primal dual, iterative soft thresholding algorithms);
(3). Learning algorithms and architectures of commonly used deep learning networks, such as the convolutional neural network (CNN) and applications to image processing;
(4). Integration of variational and deep learning methods. Design interpretable deep learning networks inspired by variational models and optimization algorithms for image analysis.

The topic of this course is one of the rapid developing fields. There is no textbook available. I will provide some references (recent papers). Students presentations, discussions and projects are expected.

References:

Nikos Paragios, Yunmei Chen, and Olivier Faugeras, The Handbook of Mathematical Models in Computer Vision. Springer 2006.

Otmar Scherzer: Handbook of Mathematical Methods in Imaging. Springer 2015.

Leonid I. Rudin, Stanley Osher, and Emad Fatemi. Nonlinear total variation based noise removal algorithms. Physica D, 60(1):259-268, 1992.

M. Lustig, D. Donoho, and J. M. Pauly. Sparse mri: The application of compressed sensing for rapid mr imaging. Magnetic Resonance in Medicine, 58(6):1182–1195, 2007.

A. Beck and M. Teboulle. A fast iterative shrinkagethresholding algorithm for linear inverse problems. SIAM Journal on Imaging Sciences, 2(1):183–202, 2009.

M. R. Hestenes, Multiplier and gradient methods, J. Optim. Theory Appl., 4 (1969), 303–320

A. Chambolle and T. Pock, A first-order primal-dual algorithm for convex problems with applications to imaging, J. Math. Imaging Vision, 40 (2011), 120–145.

 

https://sites.google.com/site/deeplearningsummerschool/

http://www.cs.toronto.edu/~hinton/absps/NatureDeepReview.pdf

M. Abadi, P. Barham, J. Chen, Z. Chen, A. Davis, J. Dean, M. Devin, S. Ghemawat, G. Irving, M. Isard, et al. Tensorflow: A system for large-scale machine learning. In OSDI, volume 16, pages 265–283, 2016.

J. H. R. Chang, C.-L. Li, B. P´oczos, B. V. K. V. Kumar, and A. C. Sankaranarayanan. One network to solve them all — solving linear inverse problems using deep projection models. ICCV, 2017.

K. Zhang,W. Zuo, S. Gu, and L. Zhang. Learning deep CNN denoiser prior for image restoration. CVPR, 2017.

Y. Yang, J. Sun, H. Li, and Z. Xu. Deep ADMM-Net for compressive sensing MRI. In NIPS, pages 10–18, 2016.

S.Wang, S.Fidler and R.Urtasun. Proximal deep structured models. In NIPS, pages 865–873, 2016.

Kerstin Hammernik et. al., Learning a Variational Network for Reconstruction
of Accelerated MRI Data, Magnetic Resonance in Medicine 79:3055–3071, 2018.

Jian Zhang, Bernard Ghanem, ISTA-Net: Interpretable Optimization-Inspired Deep Network for Image Compressive Sensing, CVPR 2018.

 

 Meeting Time and Rooms:

•   MWF 5 at MAT 114
• Office Hours: MWF 4 or by appointment

 

  Arrangement of the course::

• Unit 1: Image restoration and reconstruction (Tentatively week 1-10)

1. Variational models for image restoration and compressive sensing (CS) reconstruction: (Tentatively week 1-2)

1.1. Data fidelity: Additive and multiplicative noise; Maximum likelihood estimator (MLE) based data fitting; Bayes rule and maximum a priori (MAP) estimation;

1.2. Regularization: Sparsity in transformed domains, such as total variation and/or wavelet regularizers, sparse representations over dictionaries;

2. Optimization algorithms and related theories for model solutions: (Tentatively week 3-5)

2.1. Sub-differential, Shrinkage operator;

2.2. Proximal gradient method and iterative shrinkage-thresholding algorithm (ISTA);

2.3. Alternating minimization algorithm(AMA), Alternating direction method of multipliers (ADMM) for equality constrained convex optimization;

2.4. First-order primal-dual algorithms, Primal-dual hybrid gradient (PDHG) algorithm and Chambolle’s method for solving dual problem

2.5. Applications to regularized linear inversion problems in image reconstruction, MRI and CT image reconstruction with undersampled data.

3. Deep network based image restoration and CS reconstruction: (Tentatively week 6-7)

3.1. Architecture and learning algorithm of convolutional neuron network (CNN).

3.2. Application of CNN in image denoising and reconstruction.

4. Interpretable deep neuron networks inspired by variational models and algorithms (Tentatively week 8-10)

4.1. ISTA-net for solving L2-L1 structured problems and applications to image reconstruction.

4.2. ADMM-net for image reconstruction.

4.3. variational-net for reconstruction of images from partial parallel MR imaging.

• Unit 2: Image segmentation/Partition (Tentatively week 11-16)

1. Variational models for image segmentation: (Tentatively week 11-12)

1.1. Edge based models: Geodesic Active Contour model, parametric and level set formulation.

1.2. Region based models: Mumford-Shah (MS) model for simultaneous smoothing and segmentation, CV-model, region competition model.

2. Algorithms and related theories for model solutions: (Tentatively week 13)

3. Deep network based image segmentation/partition: (Tentatively week 14)

3.1. Application of deep networks such as CNN to image segmentation/partition.

4. Learnable deep neuron networks inspired by variational models and algorithms for image segmentation (Tentatively week 15-16)

4.1. Design novel neuron network inspired by the edge based or region based variational models and ADMM or primal dual algorithm for image segmentation/partition. (Tentatively week 15-16)

 
Additional Information:
 
Grading:
Students will be required to present one to two papers or projects related to the course content. The projects may be related to problems of particular interest to the individual student. Grades will be assigned on the basis of the presentations or projects. Current UF grading policies can be found from the following link https://catalog.ufl.edu/ugrad/current/regulations/info/grades.aspx.
 
Honor Code: “UF students are bound by The Honor Pledge which states, “We, the members of the University of Florida community, pledge to hold ourselves and our peers to the highest standards of honor and integrity by abiding by the Honor Code. On all work submitted for credit by students at the University of Florida, the following pledge is either required or implied: “On my honor, I have neither given nor received unauthorized aid in doing this assignment.” The Honor Code specifies a number of behaviors that are in violation of this code and the possible sanctions. Furthermore, you are obligated to report any condition that facilitates academic misconduct to appropriate personnel. If you have any questions or concerns, please consult with the instructor or TAs in this class.”
 
Class Attendance: “Requirements for class attendance and make-up exams, assignments, and other work in this course are consistent with university policies that can be found at: https://catalog.ufl.edu/ugrad/current/regulations/info/attendance.aspx.”
 
Accommodations for Students with Disabilities: “Students with disabilities requesting accommodations should first register with the Disability Resource Center (352-392-8565, https://www.dso.ufl.edu/drc/) by providing appropriate documentation. Once registered, students will receive an accommodation letter which must be presented to the instructor when requesting accommodation. Students with disabilities should follow this procedure as early as possible in the semester.”
 
Online Evaluations: “Students are expected to provide feedback on the quality of instruction in this course by completing online evaluations at https://evaluations.ufl.edu. Evaluations are typically open during the last two or three weeks of the semester, but students will be given specific times when they are open. Summary results of these assessments are available to students at https://evaluations.ufl.edu/results/.”
 
Contact information for the Counseling and Wellness Center: https://counseling.ufl.edu/, 392-1575; and the University Police Department: 392-1111 or 9-1-1 for emergencies.