Sample Page Title

Tencent AI Lab has launched Covo-Audio, a 7B-parameter end-to-end Giant Audio Language Mannequin (LALM). The mannequin is designed to unify speech processing and language intelligence by straight processing steady audio inputs and producing audio outputs inside a single structure.

System Structure

The Covo-Audio framework consists of 4 major parts designed for seamless cross-modal interplay:

• Audio Encoder: The mannequin makes use of Whisper-large-v3 as its major encoder as a result of its robustness in opposition to background noise and different accents. This part operates at a body fee of 50 Hz.
• Audio Adapter: To bridge the encoder and the LLM, a specialised adapter employs three downsampling modules, integrating linear and convolution layers to scale back the body fee from 50 Hz to six.25 Hz.
• LLM Spine: The system is constructed upon Qwen2.5-7B-Base, which has been tailored to course of interleaved sequences of steady acoustic options and textual tokens.
• Speech Tokenizer and Decoder: The tokenizer, primarily based on WavLM-large, makes use of a codebook dimension of 16,384 to provide discrete audio tokens at 25 Hz. The decoder employs a Circulation-Matching (FM) primarily based framework and a BigVGAN vocoder to reconstruct high-fidelity 24K waveforms.

Hierarchical Tri-modal Interleaving

A core contribution of this work is the Hierarchical Tri-modal Speech-Textual content Interleaving technique. Not like conventional strategies that function solely on the phrase or character degree, this framework aligns steady acoustic options $(a_c)$, discrete speech tokens $(a_d)$, and pure language textual content $(t)$.

The mannequin makes use of two major patterns:

1. Sequential Interleaving $(a_c rightarrow t rightarrow a_d)$: Steady options, textual content, and discrete tokens are organized in a progressive chain.
2. Parallel Integration $(a_c rightarrow t | a_d)$: Steady options are aligned with a coupled text-discrete unit.

The hierarchical facet ensures structural coherence through the use of phrase-level interleaving for fine-grained alignment and sentence-level interleaving to protect world semantic integrity in long-form utterances^{. The coaching course of concerned a two-stage pre-training pipeline processing a complete of 2T tokens.}

Intelligence-Speaker Decoupling

To mitigate the excessive price of setting up large-scale dialogue information for particular audio system, the analysis crew proposed an Intelligence Speaker Decoupling technique. This method separates dialogue intelligence from voice rendering, permitting for versatile voice customization utilizing minimal text-to-speech (TTS) information.

The tactic reformats high-quality TTS recordings into pseudo-conversations with masked textual content loss^{. By excluding the textual content response portion from the loss calculation, the mannequin preserves its reasoning skills whereas inheriting the naturalness of the TTS speaker. This allows customized interplay with out the necessity for in depth, speaker-specific dialogue datasets.}

Full-Duplex Voice Interplay

Covo-Audio developed into Covo-Audio-Chat-FD, a variant able to simultaneous dual-stream communication^{. The audio encoder is reformatted right into a chunk-streaming method, and the consumer and mannequin streams are chunk-interleaved in a 1:4 ratio. Every chunk represents 0.16s of audio.}

The system manages conversational states by particular architectural tokens:

• THINK Token: Signifies a listening-only state whereas the mannequin waits to reply.
• SHIFT Token: Signifies the transition to the mannequin’s talking flip.
• BREAK Token: Detects interruption alerts (barge-ins), triggering the mannequin to terminate talking instantly and swap again to listening.

For multi-turn eventualities, the mannequin implements a recursive context-filling technique, the place steady audio options from consumer enter and generated tokens from earlier turns are prefixed as historic context^.

Audio Reasoning and Reinforcement Studying

To boost complicated reasoning, the mannequin incorporates Chain-of-Thought (CoT) reasoning and Group Relative Coverage Optimization (GRPO). The mannequin is optimized utilizing a verifiable composite reward perform:

$$R_{complete} = R_{accuracy} + R_{format} + R_{consistency} + R_{considering}$$

This construction permits the mannequin to optimize for correctness $(R_{accuracy})$, structured output adherence $(R_{format})$, logical coherence $(R_{consistency})$, and reasoning depth $(R_{considering})$.

Analysis and Efficiency

Covo-Audio (7B) exhibits aggressive or superior outcomes on a number of evaluated benchmarks, with strongest claims made for fashions of comparable scale and chosen speech/audio duties. ^{On the MMAU benchmark, it achieved a median rating of 75.30%, the best amongst evaluated 7B-scale fashions. It notably excelled in music understanding with a rating of 76.05%. On the MMSU benchmark, Covo-Audio achieved a number one 66.64% common accuracy.}

Relating to its conversational variants, Covo-Audio-Chat demonstrated robust efficiency on URO-Bench, significantly in speech reasoning and spoken dialogue duties, outperforming fashions like Qwen3-Omni on the Chinese language observe^{. For empathetic interplay on the VStyle benchmark, it achieved state-of-the-art ends in Mandarin for anger (4.89), disappointment (4.93), and nervousness (5.00).}

The analysis crew notes an ‘early-response’ challenge on the GaokaoEval full-duplex setting, the place unusually lengthy silent pauses between vocal fragments could cause untimely responses. This ‘early-response’ conduct correlates with the mannequin’s pause-handling success metric and is recognized as a important path for future optimization.

Key Takeaways

• Unified Finish-to-Finish Structure: Covo-Audio is a 7B-parameter mannequin that natively processes steady audio inputs and generates high-fidelity audio outputs inside a single, unified structure. It eliminates the necessity for cascaded ASR-LLM-TTS pipelines, lowering error propagation and data loss.
• Hierarchical Tri-modal Interleaving: The mannequin employs a specialised technique to align steady acoustic options, discrete speech tokens, and pure language textual content. By interleaving these modalities at each phrase and sentence ranges, it preserves world semantic integrity whereas capturing fine-grained prosodic nuances.
• Intelligence-Speaker Decoupling: Tencent analysis crew introduces a way to decouple dialogue intelligence from particular voice rendering. This permits for versatile voice customization utilizing light-weight Textual content-to-Speech (TTS) information, considerably reducing the price of growing customized conversational brokers.
• Native Full-Duplex Interplay: The Covo-Audio-Chat-FD variant helps simultaneous listening and talking. It makes use of particular architectural tokens—THINK, SHIFT, and BREAK—to handle complicated real-time dynamics resembling easy turn-taking, backchanneling, and consumer barge-ins.
• Superior Parameter Effectivity: Regardless of its compact 7B scale, Covo-Audio achieves state-of-the-art or extremely aggressive efficiency throughout core benchmarks, together with MMAU, MMSU, and URO-Bench. It continuously matches or exceeds the efficiency of a lot bigger methods, resembling 32B-parameter fashions, in audio and speech understanding duties.

Try the Paper, Mannequin on HF and Repo. Additionally, be happy to observe us on Twitter and don’t overlook to affix our 120k+ ML SubReddit and Subscribe to our Publication. Wait! are you on telegram? now you’ll be able to be part of us on telegram as nicely.

Michal Sutter is a knowledge science skilled with a Grasp of Science in Knowledge Science from the College of Padova. With a strong basis in statistical evaluation, machine studying, and information engineering, Michal excels at reworking complicated datasets into actionable insights.