Support German language

[tolnai]

We'd love to support other languages for wake word generation besides English, and some of us would be happy to contribute to working with German. Could you let us know, in what way can we help, what is that is needed to support a new language properly? (E.g. which parts of the code, or what resources to be added, etc.)
Thanks!

[dscripka]

Extending to new languages would require finding language-appropriate versions of the following features:

1. A multi-speaker TTS model that can produced realistic and varied speech. Piper is the default for openWakeWord, mostly because has a model trained on several thousand speakers from the LibriTTS dataset. Checking the Piper voice samples here it looks like there is a German model trained on the MLS dataset with 236 speakers. This may be enough, but it's possible that performance will be lower compared to languages with a much higher number of samples. This is a good place to start, though!

2. A pronunciation dictionary with the ability to search for words that match arbitrary phoneme sequences. This is critical, as it is needed to find words that sound similar to the target wake word to act as adversarial training examples for the model. For English, the pronouncing is used as it has utilities that make this task very easy. But for other languages, alternative options will be needed. It seems like this project may be a good start, but I'm not sure.

So, for adding support for German, what would be of great help is:

• Testing the German MLS Piper model on a random selection of wakewords, and manually reviewing the samples to ensure that it pronounces them correctly, and with enough variety to serve as good training data. This is where assistance from native speakers is critical, as I'm sure there are nuances in pronunciation and tone that I will not be able to identify.

• Developing an alternative approach to find adversarial negative examples based on pronunciation, ideally by producing CMUdict files compatible with the pronouncing library.

[tolnai]

@dscripka thanks for your reply.

1. The mls model there seems to be ok-ish. I checked several speakers from the list and all sounded German. :) the pronunciation is not perfect, but close enough in my view. How should I/we generate wakewords with this model the easiest way, to test them with different speakers as well?
2. Ok, there is quite some thing going on in generate_adversarial_texts... :) The project you linked contains pronunciation of many German words. So technically, we could search for our wakeword there, find similar words based on the pronunciation, and return those words as input for negative TTS generation.

Obviously this limits the choice of wakewords. Ideally, we should have a similar package, which is able to do the word->pronunciation translation. How about using deuPD for example?

Another interesting thing I found is pocketsphinx where someone claims they could use it with these files.

This one also has a dictionary inside, I'm not sure if it is of any other use.

(Some of these are really old.)

But I'm wondering, is it worth to do this much? Can't we simply produce similar words alphabetically, and generate those as negative examples? Would that just make the training less efficient, or would it not work at all? If the problem is that English written form and pronunciation can be fairly different, with some other languages like German (or Hungarian, my native language) basically we-pronounce-what-we-read, so there are less exceptions, and if 2 words are close alphabetically, they would be close in pronunciation as well.

[dscripka]

Thanks for the links @tolnai there are some interesting options here.

As I was thinking about this more in the last few weeks (apologies for the delay), I think you may be right and it is not necessary to fully reproduce the pronouncing library functionality, when really all that is needed is finding words that have enough phoneme overlap to sound similar. To that end, I put together a Google Colab notebook that tests the idea here: https://colab.research.google.com/drive/1WX9jpsYlAbuICmP4j3SjjB3IG22LK6wi?usp=sharing

The notebook installs piper-tts as well as piper-phonemize so that you can test the models (for some reason the MLS German model isn't working, so I'm using a single speaker one for now). Then I use a word list of 50,000 German words, phonemize them, and look for simple phoneme overlap.

Where I could use assistance:

1. Does this simple phoneme overlap approach select that words that would sound similar when spoken? Can you think of any problems that might arise with using this approach for German?
2. How important is stress in German? For English, stress variation is provided by the pronouncing library and is used in the code to find all variations of stress in words when checking for phonetic overlap. The simpler approach likely won't accomplish this as effectively, but not all languages rely on stress in the same way. It might be worth a small decrease in performance if the approach can be kept simple.

[tolnai]

Now it's my turn to have the usual apology, but as it happens, next to work and kids it's difficult to find the time. :)

1. As I see, you were looking for similar words to "lichter". The found words are indeed similar to lichter. The problem that I see after playing with find_similar_sounding_word is that for quite some words there are only a couple similar ones found. A "backtracking" stuff could work, which tries to find results with a higher threshold, but if there is not enough, then running again with a lower threshold, and again. I had to go as low as 0.55 to get results, for example for "Matteo", but then of course the words found at 0.55 are not that close matches. Could we somehow also try to use word combinations? But before spending too much time with this one, it would be worth to just generate some models and start trying them out.
2. Typically it's the first syllable that's stressed, so it's not varying as much as in English. Because of that, it's also easier to detect start of the word, so having similar phonemes in the middle of another word can be differentiated.

[tolnai]

There is a recent comment on the piper-sample-generator, I'm not sure if it helps anything, but perhaps useful for some:
rhasspy/piper-sample-generator#4 (comment)

[tolnai]

@dscripka Any thoughts on how to continue? :)

[Flenser42]

I'm very knew to all of this, but how about using something like Mozilla's Common Voice to train wake words. You have many different speakers, that speak whole sentences.

[dscripka]

Common Voice is a great dataset, and could be a nice source of training data for Piper models in other languages. One open question is the quality of the audio, which is often low, but I agree it is worth exploring.

[dekiesel]

Hi,

Very much interested in this. I am a native german speaker. If there is anything I can help with let me know. I can find my way around python, notebooks and linux, but I am not sure where to start.