The Effects of Input Type and Pronunciation Dictionary Usage in Transfer Learning for Low-Resource Text-to-Speech

Phat Do; Matt Coler; Jelske Dijkstra; Esther Klabbers

doi:10.21437/interspeech.2023-2148

The Effects of Input Type and Pronunciation Dictionary Usage in Transfer Learning for Low-Resource Text-to-Speech

Phat Do, Matt Coler, Jelske Dijkstra, Esther Klabbers

Fryske Akademy (FA)

Research output: Contribution to conference › Paper › Scientific › peer-review

Abstract

We compare phone labels and articulatory features as input for cross-lingual transfer learning in text-to-speech (TTS) for low-resource languages (LRLs). Experiments with FastSpeech 2 and the LRL West Frisian show that using articulatory features outperformed using phone labels in both intelligibility and naturalness. For LRLs without pronunciation dictionaries, we propose two novel approaches: a) using a massively multilingual model to convert grapheme-to-phone (G2P) in both training and synthesizing, and b) using a universal phone recognizer to create a makeshift dictionary. Results show that the G2P approach performs largely on par with using a ground-truth dictionary and the phone recognition approach, while performing generally worse, remains a viable option for LRLs less suitable for the G2P approach. Within each approach, using articulatory features as input outperforms using phone labels.

Original language	English
Pages	5461-5465
DOIs	https://doi.org/10.21437/interspeech.2023-2148
Publication status	Published - 20 Aug 2023
Event	Interspeech 2023 - Convention Centre, Dublin, Ireland Duration: 20 Aug 2023 → 24 Aug 2023 https://interspeech2023.org

Conference

Conference	Interspeech 2023
Country/Territory	Ireland
City	Dublin
Period	20/08/2023 → 24/08/2023
Internet address	https://interspeech2023.org

Access to Document

10.21437/interspeech.2023-2148Licence: CC BY-NC-SA

Cite this

@conference{eadbfc01ea094a798d2654866ede6be5,

title = "The Effects of Input Type and Pronunciation Dictionary Usage in Transfer Learning for Low-Resource Text-to-Speech",

abstract = "We compare phone labels and articulatory features as input for cross-lingual transfer learning in text-to-speech (TTS) for low-resource languages (LRLs). Experiments with FastSpeech 2 and the LRL West Frisian show that using articulatory features outperformed using phone labels in both intelligibility and naturalness. For LRLs without pronunciation dictionaries, we propose two novel approaches: a) using a massively multilingual model to convert grapheme-to-phone (G2P) in both training and synthesizing, and b) using a universal phone recognizer to create a makeshift dictionary. Results show that the G2P approach performs largely on par with using a ground-truth dictionary and the phone recognition approach, while performing generally worse, remains a viable option for LRLs less suitable for the G2P approach. Within each approach, using articulatory features as input outperforms using phone labels.",

author = "Phat Do and Matt Coler and Jelske Dijkstra and Esther Klabbers",

year = "2023",

month = aug,

day = "20",

doi = "10.21437/interspeech.2023-2148",

language = "English",

pages = "5461--5465",

note = "Interspeech 2023 ; Conference date: 20-08-2023 Through 24-08-2023",

url = "https://interspeech2023.org",

}

TY - CONF

T1 - The Effects of Input Type and Pronunciation Dictionary Usage in Transfer Learning for Low-Resource Text-to-Speech

AU - Do, Phat

AU - Coler, Matt

AU - Dijkstra, Jelske

AU - Klabbers, Esther

PY - 2023/8/20

Y1 - 2023/8/20

N2 - We compare phone labels and articulatory features as input for cross-lingual transfer learning in text-to-speech (TTS) for low-resource languages (LRLs). Experiments with FastSpeech 2 and the LRL West Frisian show that using articulatory features outperformed using phone labels in both intelligibility and naturalness. For LRLs without pronunciation dictionaries, we propose two novel approaches: a) using a massively multilingual model to convert grapheme-to-phone (G2P) in both training and synthesizing, and b) using a universal phone recognizer to create a makeshift dictionary. Results show that the G2P approach performs largely on par with using a ground-truth dictionary and the phone recognition approach, while performing generally worse, remains a viable option for LRLs less suitable for the G2P approach. Within each approach, using articulatory features as input outperforms using phone labels.

AB - We compare phone labels and articulatory features as input for cross-lingual transfer learning in text-to-speech (TTS) for low-resource languages (LRLs). Experiments with FastSpeech 2 and the LRL West Frisian show that using articulatory features outperformed using phone labels in both intelligibility and naturalness. For LRLs without pronunciation dictionaries, we propose two novel approaches: a) using a massively multilingual model to convert grapheme-to-phone (G2P) in both training and synthesizing, and b) using a universal phone recognizer to create a makeshift dictionary. Results show that the G2P approach performs largely on par with using a ground-truth dictionary and the phone recognition approach, while performing generally worse, remains a viable option for LRLs less suitable for the G2P approach. Within each approach, using articulatory features as input outperforms using phone labels.

U2 - 10.21437/interspeech.2023-2148

DO - 10.21437/interspeech.2023-2148

M3 - Paper

SP - 5461

EP - 5465

T2 - Interspeech 2023

Y2 - 20 August 2023 through 24 August 2023

ER -

The Effects of Input Type and Pronunciation Dictionary Usage in Transfer Learning for Low-Resource Text-to-Speech

Abstract

Conference

Access to Document

Fingerprint

Cite this