Title: Controllability and Uncertainty in Generative Models

Date: Friday, November 3, 2023

Time: 9-10am ET

Location: Coda C0915 Atlantic & Zoom

Cusuh Ham

Machine Learning PhD Student

School of Interactive Computing
Georgia Institute of Technology

Committee

Dr. James Hays (Advisor) - School of Interactive Computing, Georgia Tech

Dr. Judy Hoffman - School of Interactive Computing, Georgia Tech

Dr. Zsolt Kira - School of Interactive Computing, Georgia Tech

Dr. Humphrey Shi - School of Interactive Computing, Georgia Tech

Dr. Jun-Yan Zhu - School of Computer Science, Carnegie Mellon

Abstract

This dissertation describes methods for enhancing generative models with either added controllability or expressiveness of uncertainty, demonstrating how a strong prior enables both features. One general approach is to introduce new architectures or training objectives. However, current trends towards massive upscaling of model size, training data, and computational resources can make retraining or fine-tuning difficult and expensive. Thus, another approach is to build upon existing pre-trained models. We consider both types of approaches with an emphasis on the latter. We first tackle the tasks of controllable image synthesis and uncertainty estimation through training-based methods and then switch focus towards computationally-efficient methods that do not require direct updates to the base model's parameters. We conclude with an overview of our latest work on personalization of text-to-image diffusion models, which efficiently recontextualizes a target concept into new settings and configurations.