NVIDIA Omniverse Audio2Face:Instantly create expressive facial animation from just an audio source using generative AI

8,453 0 80

Table of Contents

About Omniverse Audio2Face

Omniverse Audio2Face by NVIDIA uses generative AI to create expressive facial animation from an audio source or file. It simplifies the animation of a 3D character to match any voice-over track, making it easy to animate characters for games, films, or real-time digital assistants. You can record a voice audio track or generate facial animations live using a microphone, and it can process any language easily. Audio2Face also lets you retarget to any 3D human or human-esque face with just a few clicks.

Instantly create expressive facial animation from just an audio source using generative AI.

Audio-to-Animation Made Easy With Generative AI

NVIDIA Omniverse^™ Audio2Face beta is a foundation application for animating 3D characters facial characteristics to match any voice-over track, whether for a game, film, real-time digital assistant, or just for fun. You can use the Universal Scene Description (OpenUSD)-based app for interactive real-time applications or as a traditional facial animation authoring tool. Run the results live or bake them out, it’s up to you.

How It Works

Audio2Face is preloaded with “Digital Mark”— a 3D character model that can be animated with your audio track, so getting started is simple—just select your audio and upload. The audio input is then fed into a pre-trained Deep Neural Network and the output drives the 3D vertices of your character mesh to create the facial animation in real-time. You also have the option to edit various post-processing parameters to edit the performance of your character. The results you see on this page are mostly raw outputs from Audio2Face with little to no post-processing parameters edited.

Audio Input

Use a Recording, or Animate Live

Simply record a voice audio track, input into the app, and see your 3D face come alive. You can even generate facial animations live using a microphone.

Audio2Face will be able to process any language easily. And we’re continually updating with more and more languages.

Character Transfer

Face-Swap in an Instant

Audio2Face lets you retarget to any 3D human or human-esque face, whether realistic or stylized. This makes swapping characters on the fly—whether human or animal—take just a few clicks.

Scale Output

Express Yourself—or Everyone at Once

It’s easy to run multiple instances of Audio2Face with as many characters in a scene as you like – all animated from the same, or different audio tracks. Breathe life and sound into dialogue between a duo, a sing-off between a trio, an in-sync quartet, and beyond. Plus, you can dial up or down the level of facial expression on each face and batch output multiple animation files from multiple audio sources.

Emotion Control

Bring the Drama

Audio2Face gives you the ability to choose and animate your character’s emotions in the wink of an eye. The AI network automatically manipulates the face, eyes, mouth, tongue, and head motion to match your selected emotional range and customized level of intensity, or, automatically infers emotion directly from the audio clip.

Data Conversion

Connect and Convert

The latest update to Omniverse Audio2Face now enables blendshape conversion and also blendweight export options. Plus, the app now supports export-import with Blendshapes for Blender and Epic Games Unreal Engine to generate motion for characters using their respective Omniverse Connectors.

Discover More Omniverse Apps

Explore the entire Omniverse ecosystem of connections and 3rd party tools, or try out other foundation applications below.

Code

An integrated development environment for developers and power users to easily build Omniverse extensions, apps, or microservices.

Machinima

Remix, recreate, and redefine animated video game storytelling with an AI-powered toolkit for creators.

USD Composer

Accelerate advanced world-building with Pixar USD and interactively assemble, simulate, and render scenes in real time.

USD Presenter

Collaboratively review design projects with this powerful, physically accurate, and photorealistic visualization tool.

Frequently Asked Questions

What is Audio2Face in omniverse?

NVIDIA Omniverse™ Audio2Face is a combination of artificial intelligence (AI)-based technologies that derive facial motion and lip sync animations from an audio source. With Audio2Face, you can: Analyze an audio sample and automatically animate emotions in a character’s performance.

Is omniverse Audio2Face free?

All of the Omniverse tools are free to individual artists.

What languages are supported by Audio2Face?

Audio2Face supports multiple Latin based languages, such as Russian, French, Spanish, and Italian. Non-Latin based languages, such as Asian languages, have more variation which the team is currently working on solving.

Is Omniverse an AI?

Nvidia said the Omniverse platform will leverage the OpenUSD framework and generative AI (AI) to accelerate the creation of virtual worlds and advanced workflows for industrial digitalization.

Is Omniverse a metaverse?

Metaverse is equivalent to the universe but in terms of digital reality. Omniverse, in contrast, is the superset or collection of all the existing universes in terms of digital existence and technology. Sum up of virtual existence of digitalized individuality or community.

What is an Omniverse foundation application?

Omniverse foundation applications are best practice example implementations and configurations of Omniverse extensions. They are provided as a generic template on which developers and customers can customize, extend, and personalize according to their workflow.

Foundation applications can be used out-of-the-box, but, to maximize the true value of the Omniverse platform, customization and extension is highly encouraged. Every developer, customer, user will have their own interpretation of Omniverse foundation applications.

How do I install Omniverse Audio2Face?

To install Omniverse Audio2Face, follow the steps below: