MemFace: Bringing Talking Faces to Life
Machines that create lifelike video portraits of a person from audio are becoming increasingly common. These talking face generation techniques show up in virtual avatars, online conferencing and animated film.
The one-to-many problem
The current two-stage framework predicts an intermediate representation from the input audio, then uses a renderer to synthesise the video portrait. That approach has made significant gains in realism and lip-sync quality, but it struggles with the one-to-many mapping problem: the same audio can correspond to many valid outputs depending on phoneme context, emotion and lighting.
How MemFace addresses it
MemFace uses two kinds of memory, implicit and explicit, to bridge the gap between audio and visual representation. By supplementing the missing data with memory, the one-to-many mapping becomes tractable.
Implicit memory captures higher-level semantic information by learning habitual patterns and attitudes specific to the target individual.
Explicit memory stores pixel-level information such as wrinkles and shadow detail. It synthesises visual appearance from mouth shapes derived from expression estimates. The memory is composed of 3D face models and accompanying image patches, where the model vertices act as keys and the pixel-level information as values. Querying with the vertices lets the neural rendering model retrieve the pixel detail for each input phrase, which is then used to render the desired appearance.
Where it goes next
Research continues on optimising MemFace and improving realism, lip-sync quality, emotional expression and head motion. Better talking face generation opens the way to more convincing virtual avatars, animation and online services.
MemFace is a useful step against a genuinely hard problem in this space. It also, like every advance in this area, moves synthetic video portraits closer to being indistinguishable from real ones, which is worth tracking as a provenance and authentication question rather than only a graphics one.