PAPER_TITLE

FIRST_AUTHOR_LAST, FIRST_AUTHOR_FIRST; SECOND_AUTHOR_LAST, SECOND_AUTHOR_FIRST

DyadTalk: A Unified Pure Audio-Driven Framework for Dyadic 3D Talking Head Generation

Junwu Liu^1,2, Yun Cao^1,2,*, Kangwei Liu^1,2, Bowen Yang^1,2, and Xiaowei Yi^1,2
¹ Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China
² School of Cyber Security, University of Chinese Academy of Sciences, Beijing, China
^* Corresponding author

Comparison between DyadTalk and prior speaker-only or asymmetric interaction models. Existing methods are typically limited to single-speaker monologue settings or asymmetric generation paradigms, where listener motion is conditioned on the speaker’s visual input. In contrast, DyadTalk enables audio-only, bidirectional, and role-aware motion generation for dyadic conversations within a unified framework, providing a practical formulation for modeling coupled conversational behavior from speech alone.

Abstract

Dyadic face-to-face interaction is a fundamental form of human communication, where facial motions are continuously shaped by mutual engagement between two interlocutors rather than being produced independently. In many real-world applications, however, direct visual access to the conversation partner is often restricted or unavailable due to strict privacy requirements, making privacy-preserving audio-only generation especially desirable. Existing methods confront three main limitations: (1) single-speaker 3D talking head approaches generate isolated facial animations, failing to model mutual interactive dependencies; (2) current dyadic interaction frameworks predominantly rely on the partner's visual input as conditioning, rendering them difficult to apply in privacy-sensitive scenarios; and (3) asymmetric speaker-listener paradigms fall short of handling the dynamic turn-taking behavior in genuine bidirectional conversations. To tackle these challenges, we formally define the pure audio-driven dyadic 3D talking head generation task and propose DyadTalk, a unified framework requiring no visual cues during inference. The core of DyadTalk is the Hybrid Interaction Module (HIM) featuring a "Mix-then-Decouple" design. It effectively mixes dual speech streams to capture shared conversational dynamics while decoupling the representations to preserve speaker-specific motion patterns. Extensive experiments validate that DyadTalk consistently outperforms the audio-adapted baseline, DualTalk*. Specifically, it achieves a 43% reduction in expression Fréchet Distance (FD-EXP) and a 35% decrease in Mean Squared Error (MSE-EXP) on the in-domain test split. Furthermore, DyadTalk demonstrates strong out-of-domain robustness, reducing generalization errors by over 23% across key perceptual metrics. By successfully recovering realistic and coherent bidirectional motions solely from synchronized audio, our work provides a viable solution for privacy-friendly conversational avatar generation.

Overview of the proposed DyadTalk framework. Given two input audio streams, the model first extracts speaker-specific acoustic representations and enhances their temporal consistency via the Temporal Enhancer. The core Hybrid Interaction Module then models dyadic interaction through a Mix-then-Decouple strategy, consisting of symmetric implicit interaction fusion and a shared explicit interaction decoder. Finally, two speaker-specific prediction heads map the refined interaction-aware features to blendshape coefficients for generating 3D facial motions of both interlocutors.

Qualitative comparison of pure audio-driven dyadic facial motion synthesis. Compared with DualTalk*, our method generates more expressive and interaction-aware facial motions for both interlocutors, with facial behaviors that more closely resemble the ground truth. The highlighted regions illustrate improved mouth articulation, expression realism, and cross-speaker responsiveness in multi-turn conversations.

Additional qualitative comparison among ground truth, our audio-only model, DualTalk*, and the original vision-dependent DualTalk. The selected examples show that our method produces more natural conversational facial motions than the direct audio-only baseline and narrows the quality gap to the vision-dependent setting.

Qualitative comparison of representative ablation failures. From left to right, the three groups illustrate Role Confusion, Detail Degradation, and Interaction Deficiency, corresponding to w/o explicit decoder, w/o implicit encoder, and w/o partner audio, respectively. The full model remains closer to the ground truth across all cases.

DyadTalk Results

Method Comparison

This section presents video comparisons between our method and existing approaches. As illustrated, DualTalk* which also operates under audio-only conditions exhibits notable limitations including identity confusion and temporal inconsistency in head movements. In contrast, our proposed framework achieves generation quality on par with or surpassing that of vision-conditioned methods, while maintaining strict adherence to privacy-preserving audio-only constraints.

Ablation Analysis

This section presents video visualizations of our ablation studies. As illustrated, replacing or removing individual components leads to varying degrees of degradation in generation quality, thereby validating the contribution of each module to the overall temporal coherence and fidelity.