Towards High-fidelity Head Blending with Chroma Keying for Industrial Applications

Hah Min Lew1* Sahng-Min Yoo2* Hyunwoo Kang3* Gyeong-Moon Park4
1Klleon AI Research, 2Samsung Research, 3Hyperconnect, 4Kyung Hee University
WACV 2025
(*equal contribution, work done at Klleon AI Research)
Paper

We propose CHANGER, a novel pipeline for Consistent Head blending with predictive AtteNtion Guided foreground Estimation under chroma key setting for industRial applications. CHANGER generates a high-fidelity head blended result.

Head blending results from two source images to three target videos.

High-Fidelity Industrial Head Blending Pipeline

Illustration of our CHANGER pipeline: After acquiring the actor's frames (source), we can seamlessly insert acting scenes into the desired scenes with our CHANGER. Chroma keying ensures high-fidelity backgrounds. Here, both of the source and the target actors are virtual humans.

Various Industrial Application Examples

By leveraging chroma key technique with our proposed CHANGER pipeline, we can obtain various high-fidelity head blended videos in the wild environments. The red boxes represent the source images.

Effects of Our Contributions

Our novel H2 augmentation and FPAT with chroma keying successfully achieves high-fidelity head blending performance. Point and drag your mouse to see the results!

Abstract

We introduce industrial Head Blending pipeline for the task of seamlessly integrating an actor’s head onto a target body in digital content creation, which is essential for applications such as visual effects (VFX), digital human creation, and virtual avatars. The key challenge stems from discrepancies in head shape and hair structure, which lead to unnatural boundaries and blending artifacts. Existing methods treat foreground and background as a single task, resulting in suboptimal blending quality. To address this problem, we propose CHANGER, a novel pipeline that decouples background integration from foreground blending. By utilizing chroma keying for artifact-free background generation and introducing Head shape and long Hair augmentation (H2 augmentation) to simulate a wide range of head shapes and hair styles, CHANGER improves generalization on innumerable various real-world cases. Furthermore, our Foreground Predictive Attention Transformer (FPAT) module enhances foreground blending by predicting and focusing on key head and body regions. Quantitative and qualitative evaluations on benchmark datasets demonstrate that our CHANGER outperforms state-of-the-art methods, delivering high-fidelity, industrial-grade results.

Motivations

We propose CHANGER to consider the real-world application. As shown in (a), the existing work [2] (H2SB) shows severe artifacts on inpainting regions. To inpaint the background flawlessly, we propose to introduce chroma keying in the head blending framework. However, it still shows low-fidelity results to inpaint the neck and the body, which is hidden due to the head shape and hair difference described in a red box of (b). CHANGER generates the high-fidelity foreground with H2 augmentation and Foreground Predictive Attention Transformer (FPAT). CHANGER removes artifacts as shown in the blue boxes of (b) and (c), and easily changes various high-fidelity real-world backgrounds.

Methods

CHANGER consists of H2 augmentation, an encoder (E), a head colorizer, a body blender including Foreground Predictive Attention Transformer (FPAT) modules, and a decoder (D). We visualize our proposed input (X) manipulation method with overall training (blue) and inference (red) schemes. The head colorizer colorizes the gray head of X, and the body blender inpaints the hidden body and the neck with a foreground mask-aware attention mechanism. The Foreground-Prediction module predicts the foreground mask (M) of the body and the neck region, and the attention is reweighted according to M.

Comparisons

We compared qualitative results between CHANGER with the state-of-the-art head blending and face swapping models: H2SB [2], SimSwap [3], RAFSwap [4], BlendFace [5], and E4S [6]. "CK" represents inference on the chromakey configuration.

References

We utilized the pre-trained face reenactment network to generate our head blending results.

[1] Latent Image Animator: Learning to Animate Images via Latent Space Navigation.

[2] Few-Shot Head Swapping in the Wild.

[3] SimSwap: An Efficient Framework For High Fidelity Face Swapping.

[4] Region-Aware Face Swapping.

[5] BlendFace: Re-designing Identity Encoders for Face-Swapping.

[6] E4S: Fine-grained Face Swapping via Regional GAN Inversion.

BibTeX

@article{lew2024towards,
      title={Towards High-fidelity Head Blending with Chroma Keying for Industrial Applications},
      author={Lew, Hah Min and Yoo, Sahng-Min and Kang, Hyunwoo and Park, Gyeong-Moon},
      journal={arXiv preprint arXiv:2411.00652},
      year={2024}
    }