FacePlex: Full-Duplex Joint Speech-Facial Motion Generation for Conversational Avatars

Habin Lim^1*, Jae-Ho Lee^1*, Hah Min Lew^1*, Ji-Su Kang², Gyeong-Moon Park¹

* Equal contribution

arXiv Code (TBD)

Full-Duplex Streaming Joint Speech & Motion Rolling Flow Matching

FacePlex jointly generates speech audio and facial motion token-by-token in a single streaming pass, enabling face-to-face conversational avatars that listen, think, and speak simultaneously.

Abstract

Natural face-to-face conversation requires real-time speech generation together with synchronized facial motion. Existing systems only partially address this problem: speech-only full-duplex models can generate speech in real time but do not produce facial motion, while audio-driven facial motion models animate a face from already available audio rather than jointly generating speech and motion online.

To bridge this gap, we first formalize full-duplex joint speech-facial motion generation, where speech tokens and facial motion tokens are produced together every step. Building on this formulation, we propose FacePlex, a unified streaming framework with two key components. First, Rolling Flow Matching adapts flow matching to online motion generation by committing new motion frames at each streaming step. Second, Rolling Cross-Attention couples the streaming audio queue with the motion queue, allowing speech and facial motion to condition each other as generation progresses.

Through extensive experiments, ablation studies, and a user study, we show that FacePlex enables full-duplex joint speech-facial motion generation under online streaming constraints, while achieving stronger lip-sync quality and motion fidelity than audio-driven facial motion baselines.

Method Overview

FacePlex is a single streaming model that produces speech tokens and facial motion tokens jointly at every step. Two design choices make this possible:

Overview of FacePlex. At each training step, the LLM backbone, audio branch, and motion branch are jointly trained, taking the user audio and previous model streams as input to produce a hidden state and the next audio chunk. The audio chunk is temporarily enqueued so the motion branch can update a rolling motion queue with a short predicted-speech look-ahead. With L = 4 queue slots, the front audio chunk and its aligned facial-motion pair are emitted together, producing synchronized full-duplex avatar output.

Rolling Flow Matching (RFM)

Standard flow matching denoises an entire fixed-length sequence in one shot, which is incompatible with online streaming. Rolling Flow Matching recasts the process as a sliding window: at every step a new motion frame is committed at the head of the queue while the tail is still being refined. This yields a motion stream that is emitted token-by-token alongside the speech stream, with the same low-latency cadence as the audio backbone.

Rolling Cross-Attention (RCA)

Speech and facial motion must condition each other — lips depend on phonemes, and head/jaw motion shapes spectral content. Rolling Cross-Attention couples the streaming audio queue with the motion queue so that, at each step, the two modalities exchange information over a bounded local window. The model therefore stays causal and online while generating tightly synchronized speech and motion.

Quantitative Comparisons

Main comparison against representative full-duplex speech models (Moshi, Freeze-Omni, PersonaPlex) and audio-driven facial motion models (ARTalk, DualTalk, UniLS). FacePlex is the only method that produces full-duplex speech and facial motion jointly within a single streaming forward pass.

Method	Capability		Full-Duplex Speech				Facial Motion
Method	FDS	M	Pause TOR ↓	BC Freq. ↑	Turn Lat. ↓	Intr. Lat. ↓	PLRS ↑	S-LVE ↓	S-MHD ↓	S-FDD ↓	L-FDD ↓	L-PFID ↓
Moshi	✓	✗	0.990	0.001	0.283	0.258	N/A
Freeze-Omni	✓	✗	0.700	0.002	0.955	1.364
PersonaPlex	✓	✗	0.587	0.025	0.078	0.427
ARTalk	✗	✓	N/A				0.186	10.318	2.325	27.254	27.368	0.109
DualTalk	✗	✓					0.208	15.364	3.406	27.272	28.481	0.110
UniLS	✗	✓					0.166	16.138	3.761	34.432	36.250	0.044
FacePlex Ours	✓	✓	0.584	0.028	0.078	0.399	0.239	7.896	1.784	24.629	24.567	0.031

Main comparison. Bold marks the best result, underline the second-best. FDS: full-duplex speech support; M: facial motion generation. Speech-side metrics cover pause handling (Pause TOR), backchanneling (BC Freq., where BC = backchannel), turn-taking latency (Turn Lat.), and interruption latency (Intr. Lat.). PLRS measures audio–motion synchronization; S/L denote speaking and listening intervals. LVE, MHD, FDD, and P-FID are Lip Vertex Error, Mean Head Distance, Upper-Face Dynamics Deviation, and Pose Fréchet Inception Distance, respectively. N/A indicates an unsupported capability or inapplicable metric.

User Study

We further conducted a human evaluation with 25 participants. Each participant watched 5 conversation sessions; for every session, four videos generated from the same input by ARTalk, DualTalk, UniLS, and FacePlex were shown with model identities hidden and order counterbalanced. Participants rated each video on a 1–5 MOS scale across four axes: Lip Synchronization (Lip Sync), Facial Expression & Speech Natural Coherence (Natural & Coherence), Conversational Interaction (Interaction), and Overall Quality (MOS). FacePlex receives the highest mean rating on every axis.

Method	Lip Sync ↑	Natural & Coherence ↑	Interaction ↑	MOS ↑
ARTalk	2.080	1.864	2.832	2.240
DualTalk	3.000	2.888	3.360	3.080
UniLS	2.776	2.816	3.344	2.992
FacePlex Ours	3.592	3.624	3.920	3.736

User study results. Mean rater score on a 1–5 scale (higher is better) across four perceptual axes. Bold = best; FacePlex wins every axis, with the largest absolute gain on Naturalness & Coherence and Lip Sync.

Word-Level Lip-Sync Comparisons

For each keyword we render the same moment from all five methods side-by-side. FacePlex tracks viseme transitions tightly without lagging or smearing. (Click any video to enable audio.)

GT — Ground Truth Audio-Driven Baselines FacePlex (Ours)

GT ARTalk DualTalk UniLS FacePlex (Ours)

Keyword: always

GT ARTalk DualTalk UniLS FacePlex (Ours)

Keyword: because

GT ARTalk DualTalk UniLS FacePlex (Ours)

Keyword: believe

GT ARTalk DualTalk UniLS FacePlex (Ours)

Keyword: better

GT ARTalk DualTalk UniLS FacePlex (Ours)

Keyword: minutes

GT ARTalk DualTalk UniLS FacePlex (Ours)

Keyword: question

GT ARTalk DualTalk UniLS FacePlex (Ours)

Keyword: understand

Pairwise Comparisons vs. Baselines

Ten utterances per baseline, rendered as side-by-side videos (baseline on the left, FacePlex on the right). Audio is identical across both panels — only the generated facial motion differs.

vs. ARTalk

ARTalkFacePlex (Ours)

vs. DualTalk

DualTalkFacePlex (Ours)

vs. UniLS

UniLSFacePlex (Ours)

Full-Duplex Conversational Session

End-to-end conversational sessions in which two avatars communicate. Both speakers' speech and facial motion are generated jointly and online — no audio is provided to the model in advance while others provided pre-given full-utterance audios. Each strip below shows all four methods rendered for the same speaker turn so reviewers can compare lip-sync, prosody-driven expression, and listener behavior on identical content.

GT ARTalk · DualTalk · UniLS FacePlex (Ours)

A Speaker A

GT ARTalk DualTalk UniLS FacePlex (Ours)

B Speaker B

GT ARTalk DualTalk UniLS FacePlex (Ours)

Full-session videos play the entire turn for each speaker (~10 seconds). Click a video to unmute and verify joint speech-motion synchrony.

Gaussian Avatar Integration

Because FacePlex emits speech and facial motion online, its outputs can naturally drive any downstream avatar renderer. Here we plug FacePlex into a conventional, off-the-shelf Gaussian Splatting head avatar — the renderer is used exactly as released by its authors, with no fine-tuning, no architectural change, and no FacePlex-specific adaptation. All clips below are FacePlex outputs rendered with that pretrained GS avatar, demonstrating that our streaming pipeline plugs directly into existing photorealistic head avatars.

FacePlex motion + audio → off-the-shelf GS head avatar photorealistic output

Note The Gaussian Splatting head avatar is a pretrained, third-party renderer used as-is. Any visual artifacts (texture flicker, view-dependent ghosting, hair or boundary glitches, etc.) observed in the clips below are inherited from that conventional GS avatar model — not from FacePlex. FacePlex contributes only the streaming speech and facial-motion signal that drives the renderer; the rendering quality reflects the off-the-shelf avatar, not our method.

Showcase Reels

Each reel stacks five distinct conversational clips side-by-side, all rendered with the Gaussian head avatar from FacePlex's streaming output (twenty examples in total).

Examples 1 – 5

Examples 6 – 10

Examples 11 – 15

Examples 16 – 20

Paired Avatar Conversation

A complete two-speaker dialogue rendered with the Gaussian head avatar. Both speakers' speech and facial motion are jointly generated by FacePlex in a single streaming forward pass.

Speaker A Speaker B

Reminder: rendering artifacts above belong to the off-the-shelf GS avatar renderer, not to FacePlex. FacePlex itself is unchanged across all results.

BibTeX

@article{lim2026faceplex,
  title   = {FacePlex: Full-Duplex Joint Speech-Facial Motion Generation for Conversational Avatars},
  author  = {Lim, Habin and Lee, Jae-Ho and Lew, Hah Min and Kang, Ji-Su and Park, Gyeong-Moon},
  journal = {arXiv e-prints},
  pages   = {arXiv--2606},
  year    = {2026}
}