Sergey Tulyakov Seminar at the Department of Engineering Science, University of Oxford

“What I cannot create, I do not understand” said the famous writing on Dr. Feynman’s blackboard. The ability to create or to change objects requires us to understand their structure and factors of variation. For example, to draw a face an artist is required to know its composition and have a good command of drawing skills (the latter is particularly challenging for the presenter). Animation additionally requires the knowledge of rigid and non-rigid motion patterns of the object. This talk shows that generation, manipulation and animation skills of deep generative models substantially benefit from such understanding. Moreover we see, the better the models can explain the data they see during training, the higher quality content they are able to generate. Understanding and generation form a loop in which improved understanding improves generation, improving understanding even more. To show this, I detail our works in three areas: video synthesis and prediction, image animation by motion retargeting, and 3D object manipulation and reconstruction. In each of these works, the internal representation was designed to facilitate better understanding of the task, resulting in improved generation abilities. Without a single labeled example, our models are able to understand factors of variation, object parts, their 3D shapes, their motion patterns and perform creative manipulations previously only available to trained professionals equipped with specialized software and hardware.