Making Machines Sound Sarcastic: LLM-Enhanced and Retrieval-Guided Sarcastic Speech Synthesis
Zhu Li, Yuqing Zhang, Xiyuan Gao, Shekhar Nayak, Matt Coler
University of Groningen, the Netherlands
Abstract
Sarcasm is a subtle form of non-literal language that poses significant challenges for speech synthesis due to its reliance on nuanced semantic, contextual, and prosodic cues. While existing speech synthesis research has focused primarily on broad emotional categories, sarcasm remains largely unexplored. In this paper, we propose a LLM-enhanced Retrieval-Augmented framework for sarcasm-aware speech synthesis. Our approach combines (1) semantic embeddings from a LoRA-fine-tuned LLaMA 3, which capture pragmatic incongruity and discourse-level cues of sarcasm, and (2) prosodic exemplars retrieved via a RAG module, which provide expressive reference patterns of sarcastic delivery. Integrated within a VITS backbone, this dual conditioning enables more natural and contextually appropriate sarcastic speech. Experiments demonstrate that our method outperforms baselines in both objective measures and subjective evaluations, yielding improvements in speech naturalness, sarcastic expressivity, and downstream sarcasm detection.
Model Architecture
Figure 1:Overview of the sarcasm-aware speech synthesis framework
VITS (Baseline): A standard variational text-to-speech model serving as the baseline, without explicit semantic or stylistic guidance.
VITS + BERT (w/ BERT): Extends VITS with semantic embeddings from a pretrained BERT model, providing additional semantic features during synthesis.
VITS + LLaMA 3 (w/ LLaMA 3): Uses textual embeddings generated by LLaMA 3, enabling richer contextual and pragmatic guidance.
VITS + LLaMA 3-LoRA (w/ LoRA): Incorporates a LoRA-fine-tuned version of LLaMA 3 trained on sarcasm-labeled data, enhancing the ability to capture sarcasm-relevant semantics.
VITS + RAG (w/ RAG): Employs RAG to retrieve sarcastic reference speech samples that are semantically similar to the input text. These references are used to provide prosodic guidance during synthesis.
VITS + LLaMA 3-LoRA + RAG (Proposed): Our proposed unified framework, combining sarcasm-aware semantic embeddings from fine-tuned LLaMA 3 with prosodic exemplars retrieved through RAG. This model is designed to maximize both semantic relevance and expressive alignment with sarcastic intent.
Speaker: CHANDLER
Text
GT
VITS
w/ BERT
w/ LLaMA 3
w/ LoRA
w/ RAG
Proposed
That's a good one too Pheebs, now all that I have to do is find a plane load of people whose resolution is to plummet to their deaths
I'm sorry man there is never a good time to have to .. Stop catching on fire.
I thought if I littered, that crying Indian might come by and save us.
Spanish midgets wrestling Julie! Okay yes I see how they got there.
These people find out that a spelling game is going over there, there will be nobody left at our party.
Really? Struck down in the prime of life.
You said you spent all that money because those boots go with skirts, dresses and pants. What do you plan on wearing? A Chicken suit?
Speaker: DOROTHY
Text
GT
VITS
w/ BERT
w/ LLaMA 3
w/ LoRA
w/ RAG
Proposed
Show them your slides of Hawaii
No I am upset because Joe Piscapo didn't get the lead role in rain man.
What are these? Claws?
No Rose, I am developing pictures from the magellan space probe.
Speaker: HOWARD
Text
GT
VITS
w/ BERT
w/ LLaMA 3
w/ LoRA
w/ RAG
Proposed
They gave him an assistant? If I want a new pen, I have to go to the bank with wire cutters.
She was just so sad all the time. Yeah, I was the only person who could cheer her up. Well, me and Ben and Jerry.
No, she's right. As long as I've known you, you've always been self-conscious.
What about people distracted by irritating passengers?
Great. I want answers and he'd better give them to us.
Apparently, a guy who wants to have his feelings hurt.
