Experiment results for sarcastic TTS
An overview of sarcastic speech synthesis
Update by Feb 23
- Denoised effect
- voicefixer
- Davinci Resolve
- Adobe
- Fine-tuned FastSpeech 2 on MUStARD(MUStARD++)
- w/o denoise
- w/ voicefixer
- w/ Davinci Resolve
- w/ Adobe
- Pre-trained FS2 on Friends
- w/o denoise
- w/ voicefixer
Part 1. Denoised effect
Original Samples
Denoise by voicefixer
Denoise by Davinci Resolve
Denoised by Adobe(from Xiyuan)
Part 2. Fined-tuned FS2 on MUStARD(MUSTARD++)
Finetuned FS2(MUStARD, w/o denoise)
Finetuned FS2(MUStARD, w voicefixer)
Finetuned F2(MUStARD, w DavinciResolve)
Finetuned FS2(MUSTARD++, w Adobe)
Finetuned FS2(MUSTARD++(sarcastic), w Adobe)
Finetuned FS2(MUSTARD++(non-sarcastic), w Adobe)
Pre-trained FS2(LibriTTS)
Part 3. Friends without sarcasm labels
| Spkear | Amount |
|---|---|
| Chandler | 2566 |
| Joey | 3018 |
| Monica | 2598 |
| Phoebe | 2642 |
| Rachel | 2870 |
| Ross | 2917 |
| Total | 16611 |
Pre-trained on Friends TV(w/o denoise)
On test set (from back-translated text)
On validation set(512)
Pre-trained on Friends TV(denoised by voicerfixer)
On test set (from back-translated text, spk: Chandler)
On test set (from back-translated text, spk: Rachel)
On validation set(512)
Friends with sarcasm labels(update on May 10)
| Example2(sarcastic) | ||||
|---|---|---|---|---|
| Fine-tuned FS2 | ||||
| FS2 + Sarcasm Label | ||||
| FS2 + ReferenceEncoder |
MUSTARD_Plus_Plus in different sarcastic & emotional types(update on May 24)
| Fine-tuned FS2(PRO) | ||
| Fine-tuned FS2(ILL) | ||
| Fine-tuned FS2(Neutral) | ||
| Fine-tuned FS2(Happy) |
MUSTARD_Plus_Plus with reference speech(update on June 07)
| Fine-tuned FS2(w/o feedback) | ||||||||
| Fine-tuned FS2(w/o feedback) |
Evaluate on unseen data, from tweet sarcastic texts(update on June-14)
- text1: size on the the toulouse team that pack be monstrous i cant see a welsh region ever win this money talk a they say,0
- text2: villainous pro tip change the device name on her bluetooth device so she doesnt forget u,0
- text3: if there one fictional place i could go christmas at hogwarts would be that place,0
- text4: i mean it a great cause theyre do it for but can we not have new xmas music thats well good,0
- text5: where do they get 1 22xg from without the pen,0
- text6: so the scottish government want people to get their booster shot so badly that the website doesnt even work,1
- text7: sometimes i lay in bed and think about how today will be the day i make my life well exercise drinking water eat healthy then i wake up,1
- text8: max verstappen be such a clean driver he never make dirty move when race,1
- text9: boris look so statesman like in that public address,1
- text10: wow the prime minister be so good at the tell the truth yet again he just cant stop tell the truth about brexit,1
| text1 | |||||||
| text2 | |||||||
| text3 | |||||||
| text4 | |||||||
| text5 | |||||||
| text6 | |||||||
| text7 | |||||||
| text8 | |||||||
| text9 | |||||||
| text10 |
StyleSpeech on MUSTARD Plus Plus (update on June-28)
| GroundTruth | |
| Pre-trained FS2 | |
| Fine-tuned FS2 | |
| Pre-trained StyleSpeech | |
| Fine-tuned StyleSpeech |