Motionshop

What is Motionshop AI

We present a framework to replace the characters in video with 3D avatars.

Our framework consists of two parts:

1) a video processing pipeline to extract the inpainted background video sequence,

2) a pose estimation and rendering pipeline to generate the avatar video sequence. By parallelizing the two pipelines and using a high performance ray-tracing renderer TIDE, the whole process can be completed in minutes.

Motionshop is a website for AI character animation. It can automatically detect characters in uploaded videos and replace them with 3D cartoon character models, creating interesting AI videos. The product offers a user-friendly interface and powerful AI algorithms, allowing users to easily transform their video content into vivid and entertaining animated works.

Motionshop AI Target Audience:

“Users can use Motionshop to transform their video content into entertaining animated works, suitable for personal entertainment, creativity, and social sharing.”

Use Case Examples:

• Transforming home travel footage into cartoon animations to add fun.
• Converting videos from friend gatherings into cartoon characters to share with friends.
• Transforming dance performance videos into cartoon characters to showcase to fans.

Motionshop AI Features

• Automatically detect characters in videos.
• Replace them with 3D cartoon character models.
• Generate interesting AI videos.

Technical Principles of Motionshop:

1. Video Processing and Background Extraction: Utilizing video processing technology to separate characters from the original video, leaving out the background without people. This step involves complex image analysis and processing to ensure the integrity and coherence of the background.

Character Detection: Motionshop uses a transformer-based framework that tightly integrates textual information with existing closed-set detectors to achieve zero-shot object detection.

Segmentation and Tracking: After successfully detecting the targets, pixel-level target regions are tracked using video object segmentation and tracking methods.

Patchwork: The remaining image regions in the video are completed using video patchwork techniques, including recursive flow completion, dual-domain propagation in the image and feature domains, etc.

2. Pose Estimation: This step utilizes pose estimation technology to analyze the movements of characters in the video. It involves capturing and analyzing human movements, estimating stable human poses using the CVFFS method, and representing 3D human bodies using the SMPL human body model.

3. 3D Cartoon Character Animation Generation: Based on the estimated poses and movements, corresponding 3D animated characters are generated. This process ensures that the movements of the 3D model match those of the characters in the original video.

4. Lighting Estimation: To seamlessly integrate the 3D model into the original video background, the lighting conditions in the video need to be estimated, and appropriate lighting adjustments are made to the 3D model.

5. High-Performance Rendering: Motionshop uses high-performance ray-tracing renderers like TIDE to render the 3D cartoon character models, ensuring they have a high level of realism and visual effects.

6. Video Composition: Finally, Motionshop composites the rendered 3D cartoon characters back into the background-free video, creating the final video output.

How to Use Motionshop AI:

Upload a video clip, where AI automatically identifies the main moving characters in the video and replaces them with interesting 3D character models with just one click, generating unique AI videos.

Step 1: Upload a video containing characters (ensure the characters are complete, and the video duration does not exceed 15 seconds).

Step 2: Select the character to be replaced in the video (auto selection).

Step 3: Choose the virtual character model to be replaced and click on generate, wait for 10 minutes for the results to be generated.

Motionshop is essentially a replacement for Wonder Studio. Wonder Studio can automatically transform characters in real-life scenes into CG characters, and it can also handle animation, lighting, and composition.