BOOTSTRAPPING NON-PARALLEL VOICE CONVERSION FROM SPEAKER-ADAPTIVE

TEXT-TO-SPEECH

Hieu-Thi Luong

1,2

, Junichi Yamagishi

1,2,3

1

SOKENDAI (The Graduate University for Advanced Studies), Kanagawa, Japan

2

as the interpretations of these two tasks. Moreover by offload-

ing the heavy data demand to the training stage of the TTS

model, our VC system can be built using a small amount of

target speaker speech data. It also opens up the possibility of

using speech in a foreign unseen language to build the system.

Our subjective evaluations show that the proposed framework

is able to not only achieve competitive performance in the

standard intra-language scenario but also adapt and convert

using speech utterances in an unseen language.

Index Terms— voice conversion, cross-lingual, speaker

scenarios, but they share a common goal: generating speech

with a target voice. If we define the TTS system as a syn-

thesis system generating speech using linguistic instructions

(in an abstract sense) obtained from written text [2], then the

VC system can be defined as a synthesis system generating

speech using linguistic instructions extracted from a reference

utterance. This similarity in objective but difference in oper-

ation context makes VC and TTS complement each other and

have their unique role in a spoken dialogue system. More

specifically, as text input is easy to create and modify, TTS is

expected written form (e.g. when text is written in foreign

languages). On the other hand, speech used as a reference

input for VC is more time-consuming and expensive to pro-

duce, but the system can be straightforwardly extended to an

unseen language.

A VC system is usually classified as parallel or non-

parallel depending on the nature of the data available for

development. The parallel VC system is developed using

a parallel corpus containing pairs of utterance spoken by a

deterministic source and target speakers. The parallel speech

utterances of source and target are aligned using dynamic

of its advantages. Acquisition of parallel speech data re-

quires a lot more planning and preparation than acquisition

of non-parallel speech, which in turn makes the former more

expensive to develop, especially when we want a VC system

with the voice of a particular speaker. Therefore, many ap-

proaches have been proposed to develop VC systems using

a non-parallel corpus [6, 7]. For the conventional Gaussian

mixture model (GMM) approach, non-parallel VC can be

adapted from a pretrained parallel VC in the model space

using the maximum a posterior (MAP) method [7, 8] or as

using methods like variational autoencoder (VAE) [12], gen-

erative adversarial networks (GAN) [13, 14] or some other

techniques [15]. For both parallel and non-parallel VC, the

systems usually change the voice but are unable to change

the duration of the utterance. Many recent researches fo-

cused on converting speaking rate along with voices by using

sequence-to-sequence models [11, 16, 17, 18], as speaking

rate is also a speaker characteristic.

arXiv:1909.06532v1 [eess.AS] 14 Sep 2019

Fig. 1. Training the initial speaker-adaptive TTS model.

model that could perform speaker adaptation with untran-

al. [19]. Furthermore, we investigate the performance of our

framework when it adapts and convert using speech utter-

ances of an unseen language. The rest of paper is organized

as follows. Section 2 describes our framework, Section 3

introduces different scenarios in which a VC can operate in

an unseen language and their practical applications, Section 4

describes the experiment setup, Section 5 presents subjective

results, and Section 6 concludes our findings.

2. BOOTSTRAPPING VOICE CONVERSION FROM

SPEAKER-ADAPTIVE TTS

uses neural networks as the approximation function for the x

to y transformation. Neural networks are trained with back-

propagation algorithm and can also be adapted to a new un-

seen domain with backpropagation when the adaptation data

is labeled. In the context of TTS, an acoustic model can

easily be adapted to voices of unseen speakers if the adap-

tation data is transcribed speech but it is not as straightfor-

Fig. 2. Transferring to VC and adapting to the target speaker.

have been proposed to perform adaptation with untranscribed

latent variable z, namely latent linguistic embedding (LLE),

which presumably contains no information about the speak-

ers. The conventional acoustic model then splits into a

speaker-dependent (SD) acoustic decoder Dec and a speaker-

independent (SI) linguistic encoder LEnc. The acoustic

decoder is defined by its SI parameter θ

core

and SD parameter

spk,(k)

, one for each k-th speaker in the training set while the

linguistic encoder is defined by its SI parameter φ

. The TTS

The acoustic encoder is only used to perform the unsuper-

vised speaker adaptation in [19]; however the network cre-

ated by stacking the acoustic encoder and acoustic decoder is

essentially a speech-to-speech stack that potentially could be

used as a VC system. This is the motivation for us to develop

While the original VC is narrowly defined as a function that

directly maps acoustic units of a source to a target speaker

tion learning and disentanglement ability of neural networks.

UNSEEN LANGUAGE

The proposed three-step framework have an interesting char-

acteristic which is the asymmetric nature in terms of data re-

quirements for each step. While the TTS training step re-

quires a transcribed multi-speaker corpus, the adaptation step

only requires a small amount of untranscribed speech from

on data of source speakers and hypothetically can convert ut-

terances of arbitrary speakers out of the box. We could take

advantage of these characteristics to build VC systems for

abundant language as language with transcribed speech cor-

pus and used to train the initial TTS model (seen); in our case

fine multiple scenarios in which our framework can interact

with an unseen language. These scenarios are dictated by the

language scenario of a voice conversion system.

an English speaker but use it to convert Japanese utterances.

Generally speaking, the VC system is used to generate speech

of linguistic instructions that have not been seen. The ability

to generate speech for content that is difficult to represent in

the expected written form (a foreign language in this case) is

one advantage of VC over TTS. This scenario is referred as

cross-language voice conversion [26, 27].

distinguish it from EE-J. Unlike other scenarios involving the

unseen language, cross-language speaker adaptation is rele-

vant for both VC [28] and TTS [30, 25]. Even though it is

not evaluated in this paper, this scenario is also the unsuper-

vised cross-language speaker adaptation scenario of the TTS

system proposed in [19].

Japanese speaker and use it to convert Japanese utterances.

in the training, the adaptation or the conversion stages. This

which is the main topic of the Zero Resource Speech Chal-

lenge 2019 [31]. Even though our scenario has the same

objective as the challenge, the approach is a little different.

The participant of the challenge are encouraged to develop

are more difficult [32, 33, 34]. Our framework is bootstrapped

from an abundant language, which is a more practical ap-

version system with unseen language. In this paper, English

plays the role of the abundant language (seen) while Japanese

plays the role of the low-resource language (unseen).

4. EXPERIMENT

4.1. Dataset

We used 25.6 hours of transcribed speech from 72 English

speakers to train the initial speaker-adaptive TTS model and

the initial SI WaveNet vocoder. The data is a subset of the

VCTK speech corpus [37]. To validate the proposed method,

we reenact the SPOKE task of the Voice Conversion Chal-

the training data in our experiments as our model can convert

speech of an arbitrary source speaker.

in-house bilingual speakers (1 male and 1 female) as the new

targets speakers. These two target speakers can speaker En-

glish and Japanese at almost native level. 400 utterances per

speaker per language are used as training data; 10 more ut-

terances are used for validation. For evaluation, we reuse

2 source speakers (1 male and 1 female) from VCC2018 as

the English source speakers and add 2 native Japanese speak-

Fig. 5. Subjective results for the SPOKE task of the Voice

Conversion Challenge 2018. The lines indicate 95% confi-

dence interval with all results are statistically significant.

Table 1. Detailed subjective results for SPOKE task.

(a) Quality

While Japanese plays the role of the low-resource in our ex-

periments, to provide a good reference, we train an additional

system in which Japanese is the abundant language and then

use it as the upper bound. This intra-lingual VC system for

Japanese, namely JJ-J, is pretrained on 44.9 hours of tran-

scribed speech of 235 native speakers and then adapted to the

Japanese training speech of the two bilingual target.

4.2. Model configuration

Our configuration imitates the setup described in [19] as

closely as possible. The linguistic feature is a 367-dimensional

trained on the 22050 Hz waveform that has been quantized

using 10-bit u-law.

tion task. All results are statistically significant.

the LLE set to 64. Speaker bias is placed at layers B5, B6,

B7, and B8 (Fig. 1 in [19]) in the form of a one-hot-vector

to represent speakers in the training set. The initial multi-

modal architecture is trained using the loss function defined

in Equation 5 with β set to 0.25. The SI WaveNet vocoder

contained 40 dilated causal layers and it is conditioned on a

natural mel-spectrogram. The WaveNet vocoder is fine-tuned

for each target speaker using their respective available train-

ing speech data.

to contain generated utterances of every target speaker from

every system. In summary, each system is judged 1088 times

opinion score. In the similarity question, the participant is

by one VC system and a natural utterance spoken by the

target speaker in a 4-point scale. The setup is the same as in

VCC2018 [38].

The general results can be seen in Fig. 5. While it is not as

the baseline B01 in terms of quality and significantly better in

terms of speaker similarity. One should note that B01 is a

standard Japanese system JJ-J achieved a relatively better

score than the English counterpart EE-E, although technically

they should not be compared directly. The cross-language

converting scenarios, EJ-J and EE-J, are a lot worse than the

score than EE-J. The result for the similarity test is illustrated

in Fig. 8. For the four systems that convert Japanese speech,

robustness of the VC system.

[1] Tomoki Toda, Alan W Black, and Keiichi Tokuda,

[2] Paul Taylor, Text-to-speech synthesis, Cambridge uni-

[3] Masanobu Abe, Satoshi Nakamura, Kiyohiro Shikano,

[5] Ling-Hui Chen, Zhen-Hua Ling, Li-Juan Liu, and Li-

[6] Daniel Erro, Asunci

´

on Moreno, and Antonio Bonafonte,

[8] Chung-Han Lee and Chung-Hsien Wu, “Map-based

[9] Tomoki Toda, Yamato Ohtani, and Kiyohiro Shikano,

[10] Lifa Sun, Kun Li, Hao Wang, Shiyin Kang, and He-

[11] Hiroyuki Miyoshi, Yuki Saito, Shinnosuke Takamichi,

non-parallel corpora using variational auto-encoder,” in

versarial networks,” arXiv preprint arXiv:1711.11293,

2017.

[14] Fuming Fang, Junichi Yamagishi, Isao Echizen, and

Jaime Lorenzo-Trueba, “High-quality nonparallel voice

conversion based on cycle-consistent adversarial net-

work,” in 2018 IEEE International Conference on

Acoustics, Speech and Signal Processing (ICASSP).

IEEE, 2018, pp. 5279–5283.

[15] Andros Tjandra, Berrak Sisman, Mingyang Zhang,

Sakriani Sakti, Haizhou Li, and Satoshi Nakamura,

phoneme posteriorgram sequences,” in Proc. SLT, 2018,

pp. 274–281.

[17] Kou Tanaka, Hirokazu Kameoka, Takuhiro Kaneko, and

Nobukatsu Hojo, “Atts2s-vc: Sequence-to-sequence

voice conversion with attention and context preservation

mechanisms,” in Proc. ICASSP, 2019, pp. 6805–6809.

[18] Jing-Xuan Zhang, Zhen-Hua Ling, and Li-Rong Dai,

“Non-parallel sequence-to-sequence voice conversion

with disentangled linguistic and speaker representa-

tions,” arXiv preprint arXiv:1906.10508, 2019.

ing Pang, Ignacio Lopez Moreno, and Yonghui Wu,

“Transfer learning from speaker verification to mul-

tispeaker text-to-speech synthesis,” arXiv preprint

arXiv:1806.04558, 2018.

Lior Wolf, “Fitting new speakers based on a short un-

transcribed sample,” arXiv preprint arXiv:1802.06984,

´

[23] Hirokazu Kameoka, Takuhiro Kaneko, Kou Tanaka, and

[26] Masanobu Abe, Kiyohiro Shikano, and Hisao

[27] Mikiko Mashimo, Tomoki Toda, Kiyohiro Shikano, and

[30] Yi-Jian Wu, Yoshihiko Nankaku, and Keiichi Tokuda,

[31] Ewan Dunbar, Robin Algayres, Julien Karadayi, Math-

for voice conversion,” arXiv preprint arXiv:1905.11563,

Elan Van Biljon, Ewald van der Westhuizen, Lisa

van Staden, and Herman Kamper, “Unsupervised

acoustic unit discovery for speech synthesis using dis-

crete latent-variable neural networks,” arXiv preprint

arXiv:1904.07556, 2019.

[35] Sunayana Sitaram, Sukhada Palkar, Yun-Nung Chen,

Alok Parlikar, and Alan W Black, “Bootstrapping text-

to-speech for speech processing in languages without an

orthography,” in Proc. ICASSP, 2013, pp. 7992–7996.

ICASSP, 2014, pp. 2594–2598.

nunen, and Zhenhua Ling, “The voice conversion chal-

lenge 2018: Promoting development of parallel and

nonparallel methods,” in Proc. Odyssey, 2018, pp. 195–

202.

[39] Kazuhiro Kobayashi and Tomoki Toda, “sprocket:

Open-source voice conversion software,” in Proc.

Odyssey, 2018, pp. 203–210.

[40] Li-Juan Liu, Zhen-Hua Ling, Yuan Jiang, Ming Zhou,

and Li-Rong Dai, “Wavenet vocoder with limited train-

ing data for voice conversion,” in Proc. INTERSPEECH,