Replies: 88 comments 824 replies
-
|
hello @SWivid do you think is possible fine tuning pretrained model on new language planing to add another language italian + english (to avoid catastrophic forgetting) |
Beta Was this translation helpful? Give feedback.
-
|
Yeah, the model should only see hiragana, whether that be at inference or training just because you'd probably need a LOT of data to generalize all those tokens in kanji (other readings etc, also make it suboptimal IMO). You might find this useful if you format data in the same way the GUI does to convert train text files |
Beta Was this translation helpful? Give feedback.
-
def is_japanese(c):
return (
"\u3040" <= c <= "\u309f" # Hiragana
or "\u30a0" <= c <= "\u30ff" # Katakana
or "\uff66" <= c <= "\uff9f" # Half-width Katakana
)
else: # if mixed chinese characters, alphabets and symbols
for c in seg:
if ord(c) < 256:
char_list.extend(c)
elif is_japanese(c):
char_list.append(c)
else:
if c not in "。，、；：？！《》【】—…":
if not char_list or not is_japanese(char_list[-1]):
char_list.append(" ")
char_list.extend(lazy_pinyin(c, style=Style.TONE3, tone_sandhi=True))
else: # if is zh punc
char_list.append(c)Thanks for your repo, i see..., first i need to use pykakasi to preprocessed my dataset，and add the convert code into, then i can use the original vocab to finetuning, is it corrected? if i missing something, tell me thanks! |
Beta Was this translation helpful? Give feedback.
-
Thanks, this help me a lot! |
Beta Was this translation helpful? Give feedback.
-
|
About right, however, a small clarification is that you do not need that conversion code for training, you only need it for inference. The steps would be:
Then after training, you can implement the is_japanese function when you use convert to pinyin in inference |
Beta Was this translation helpful? Give feedback.
-
|
Yeah! got it |
Beta Was this translation helpful? Give feedback.
-
audio files of maybe 3 - 12s duration, i'm not sure what's good, and their transcripts /your_dataset
|-- metadata.csv
|-- wavs/
| |-- audio_0001.wav
| |-- audio_0002.wav
| `-- ...metadata.csv contents:
python scripts/prepare_csv_wavs.py <path_to_your_dataset> <F5-TTS_repo_data_path>/<dataset_name>_pinyinexample: python scripts/prepare_csv_wavs.py /my_pc/your_dataset /my_pc/F5-TTS/data/your_dataset_pinyin
set dataset name to name of your dataset in f5-tts data folder dataset_name = "your_dataset"play around with these parameters and see what give the best results: set max samples to 2, or whatever you seem fit max_samples = 2also play around with learning rate, don't know which one is best learning_rate = 5e-06change epochs and warmup to whatver you seem fit for your dataset epochs = 10 # use linear decay, thus epochs control the slope
num_warmup_updates = 20 # warmup stepsadjust this to your dataset size, eg for 100 audio files and 2 max samples, maybe 500 last_per_steps = 500 # save last checkpoint per steps
hopefully we find good hyperparams for good finetuning results could put it doesn't handle other tokenizers, always assumes english dataset and import sys, os
sys.path.append(os.getcwd())
from pathlib import Path
import json
import shutil
import argparse
from tqdm import tqdm
from datasets.arrow_writer import ArrowWriter
from model.utils import (
convert_char_to_pinyin,
)
PRETRAINED_VOCAB_PATH = Path(__file__).parent.parent / "data/Emilia_ZH_EN_pinyin/vocab.txt"
def is_csv_wavs_format(input_dataset_dir):
fpath = Path(input_dataset_dir)
metadata = fpath / "metadata.csv"
wavs = fpath / 'wavs'
return metadata.exists() and metadata.is_file() and wavs.exists() and wavs.is_dir()
def prepare_csv_wavs_dir(input_dir):
assert is_csv_wavs_format(input_dir), f"not csv_wavs format: {input_dir}"
input_dir = Path(input_dir)
metadata_path = input_dir / "metadata.csv"
audio_path_text_pairs = read_audio_text_pairs(metadata_path.as_posix())
sub_result, durations = [], []
vocab_set = set()
polyphone = True
for audio_path, text in audio_path_text_pairs:
if not Path(audio_path).exists():
print(f"audio {audio_path} not found, skipping")
continue
audio_duration = get_audio_duration(audio_path)
# assume tokenizer = "pinyin" ("pinyin" | "char")
text = convert_char_to_pinyin([text], polyphone=polyphone)[0]
sub_result.append({"audio_path": audio_path, "text": text, "duration": audio_duration})
durations.append(audio_duration)
vocab_set.update(list(text))
return sub_result, durations, vocab_set
def get_audio_duration(audio_path):
import torchaudio
audio, sample_rate = torchaudio.load(audio_path)
num_channels = audio.shape[0]
return audio.shape[1] / (sample_rate * num_channels)
def read_audio_text_pairs(csv_file_path):
import csv
audio_text_pairs = []
parent = Path(csv_file_path).parent
with open(csv_file_path, mode='r', newline='', encoding='utf-8') as csvfile:
reader = csv.reader(csvfile, delimiter='|')
next(reader) # Skip the header row
for row in reader:
if len(row) >= 2:
audio_file = row[0].strip() # First column: audio file path
text = row[1].strip() # Second column: text
audio_file_path = parent / audio_file
audio_text_pairs.append((audio_file_path.as_posix(), text))
return audio_text_pairs
def save_prepped_dataset(out_dir, result, duration_list, text_vocab_set, is_finetune):
out_dir = Path(out_dir)
# save preprocessed dataset to disk
out_dir.mkdir(exist_ok=True, parents=True)
print(f"\nSaving to {out_dir} ...")
# dataset = Dataset.from_dict({"audio_path": audio_path_list, "text": text_list, "duration": duration_list}) # oom
# dataset.save_to_disk(f"data/{dataset_name}/raw", max_shard_size="2GB")
raw_arrow_path = out_dir / "raw.arrow"
with ArrowWriter(path=raw_arrow_path.as_posix(), writer_batch_size=1) as writer:
for line in tqdm(result, desc=f"Writing to raw.arrow ..."):
writer.write(line)
# dup a json separately saving duration in case for DynamicBatchSampler ease
dur_json_path = out_dir / "duration.json"
with open(dur_json_path.as_posix(), 'w', encoding='utf-8') as f:
json.dump({"duration": duration_list}, f, ensure_ascii=False)
# vocab map, i.e. tokenizer
# add alphabets and symbols (optional, if plan to ft on de/fr etc.)
# if tokenizer == "pinyin":
# text_vocab_set.update([chr(i) for i in range(32, 127)] + [chr(i) for i in range(192, 256)])
voca_out_path = out_dir / "vocab.txt"
with open(voca_out_path.as_posix(), "w") as f:
for vocab in sorted(text_vocab_set):
f.write(vocab + "\n")
if is_finetune:
file_vocab_finetune = PRETRAINED_VOCAB_PATH.as_posix()
shutil.copy2(file_vocab_finetune, voca_out_path)
else:
with open(voca_out_path, "w") as f:
for vocab in sorted(text_vocab_set):
f.write(vocab + "\n")
dataset_name = out_dir.stem
print(f"\nFor {dataset_name}, sample count: {len(result)}")
print(f"For {dataset_name}, vocab size is: {len(text_vocab_set)}")
print(f"For {dataset_name}, total {sum(duration_list)/3600:.2f} hours")
def prepare_and_save_set(inp_dir, out_dir, is_finetune: bool = True):
if is_finetune:
assert PRETRAINED_VOCAB_PATH.exists(), f"pretrained vocab.txt not found: {PRETRAINED_VOCAB_PATH}"
sub_result, durations, vocab_set = prepare_csv_wavs_dir(inp_dir)
save_prepped_dataset(out_dir, sub_result, durations, vocab_set, is_finetune)
def cli():
# finetune: python script.py /path/to/input_dir /path/to/output_dir
# pretrain: python script.py /path/to/input_dir /path/to/output_dir --pretrain
parser = argparse.ArgumentParser(description="Prepare and save dataset.")
parser.add_argument('inp_dir', type=str, help="Input directory containing the data.")
parser.add_argument('out_dir', type=str, help="Output directory to save the prepared data.")
parser.add_argument('--pretrain', action='store_true', help="Enable for new pretrain, otherwise is a fine-tune")
args = parser.parse_args()
prepare_and_save_set(args.inp_dir, args.out_dir, is_finetune=not args.pretrain)
if __name__ == "__main__":
cli() |
Beta Was this translation helpful? Give feedback.
-
thanks a lot @kunibald413 |
Beta Was this translation helpful? Give feedback.
-
While using the |
Beta Was this translation helpful? Give feedback.
-
|
the csv_wavs script is mostly copy paste from prepare_emilia.py this is how they deal with different tokenizers: i assume at least this line would need to change, it always assumes pinyin tokenizer: # assume tokenizer = "pinyin" ("pinyin" | "char")
text = convert_char_to_pinyin([text], polyphone=polyphone)[0]i'm not sure about the implications and maybe there's more that needs to change. if you know what's going on there feel free to adjust accordingly and make a merge reuquest @kostum123 |
Beta Was this translation helpful? Give feedback.
-
|
Hi @kunibald413, I'm currently finetuning F5-TTS with gradio interface, it's really amazing. But while I'm trying to finetune it from the previous checkpoint, I'm facing the below issue:
Can you please help me with this. thank you. |
Beta Was this translation helpful? Give feedback.
-
Try changing --file_checkpoint_train into --pretrain in finetune_cli.py (Line 454). There is confusion about the argument between finetune_cli.py and finetune_gradio.py |
Beta Was this translation helpful? Give feedback.
-
|
@kunibald413 Thank you for the script. I've already created something similar here. #62 (comment) can you update this part i think be nice to have like this before after |
Beta Was this translation helpful? Give feedback.
-
|
it's merged into main repo, feel free to adjust to your liking |
Beta Was this translation helpful? Give feedback.
-
|
The code @kunibald413 has provided works. However, when training it seems to initialize from a model with random weights. Can we initialize from the trained model weights instead? |
Beta Was this translation helpful? Give feedback.
-
|
By default, the training script will look for an existing model in If you place a model at that path, it will be loaded in as the base model |
Beta Was this translation helpful? Give feedback.
-
|
Just started my spanish finetune from the facebook libraspeech dataset. Single 4090 so it will take a while. |
Beta Was this translation helpful? Give feedback.
-
|
can you report back the time it takes? |
Beta Was this translation helpful? Give feedback.
-
|
Unfortunately I did it wrong and have to start over. But I ran it over 1 Epoch overnight, which took |
Beta Was this translation helpful? Give feedback.
-
|
Small update on the spanish finetune. original.wav (in training data): finetuned.wav (same input voice): Still much to go |
Beta Was this translation helpful? Give feedback.
-
|
Hello @jpgallegoar , thank you for the model in Spanish. I tried cloning a voice with a Cuban accent, and it’s not bad at all, even though it’s an accent you didn’t use during fine-tuning. I wonder if it’s possible to do fine-tuning starting from the model you trained... I only have 12 hours of Spanish with a Cuban accent. |
Beta Was this translation helpful? Give feedback.
-
|
I am using a Chinese dataset (about 33hr) to fine-tune my model. The loss is continuously decreasing, and the generated voice tone is getting closer to the target. However, as the training steps increase, the pronunciation of words is becoming increasingly unclear.
parm
|
Beta Was this translation helpful? Give feedback.
-
hi can you share the dataset? |
Beta Was this translation helpful? Give feedback.
-
|
@jpgallegoar There are different accents in Chinese, and fine-tuning can also make the model's voice closer to the dataset. |
Beta Was this translation helpful? Give feedback.
-
|
I'm sorry, but my training dataset is private and cannot be provided. |
Beta Was this translation helpful? Give feedback.
-
Ah yes, thank you for the explanation |
Beta Was this translation helpful? Give feedback.
-
Hello friend, I also want to finetune a chinese model, because I think the voice generated by the current model is not very similar to the target voice sometimes. I want to train a generic model that with 5-10s reference voice by any person, the model can mimic that person's voice. May I ask if your finetuned model get better result in mimicing the reference voice? And how many person's voices did you use and how long for each person? Thanks a lot ! |
Beta Was this translation helpful? Give feedback.
-
|
hi i just create gradio interface for easy user-friendly and accessible for beginners you can see here Features |
Beta Was this translation helpful? Give feedback.
-
|
can confirm also training work i training 3 language indonesia-italian-english eng: https://vocaroo.com/1mGEFlRNgouY italian: https://voca.ro/1l6SYplhnSxz (Quattro imperdibili appuntamenti con l’Orchestra da Camera di Caserta e solisti internazionali.) indonesia: it even can do code switching (eng-indonesia): using same config as train 
|
Beta Was this translation helpful? Give feedback.
-
|
Just to confirm: you have used a dataset composed of english, Italian and Indonesian audios do fine-tune this model, right? Can you share how many hours of each you used for each language, and how long are the clips in seconds? |
Beta Was this translation helpful? Give feedback.
-
|
Very good , when was integrated italian language in main f5-tts? |
Beta Was this translation helpful? Give feedback.
-
@acul3 |
Beta Was this translation helpful? Give feedback.
-
|
@acul3 can you tell me how many epochs you used, especially for indo voice? |
Beta Was this translation helpful? Give feedback.
-
|
Boleh minta model fine-tunenya gan? @acul3 |
Beta Was this translation helpful? Give feedback.
-
|
Hi, I was just wondering why you dont try to train on small data first instead of starting with a large dataset. For me, I trained for only 40 hours greek and with 20 hours (LibriTTS-R) focused on English, and it’s working fine speak very well. in half a day about with the 4090, and after about 100k to 150k steps, the model can speak greek and english in same time, very well and have great zero shot , try see if ths working for you i hope this help |
Beta Was this translation helpful? Give feedback.
-
|
Can you please elaborate? Why did you train in English again? |
Beta Was this translation helpful? Give feedback.
-
|
because i want speak both english and greek for example i give english and speak greek , and like i see this working gine with this method , if i train only greek not speak well english |
Beta Was this translation helpful? Give feedback.
-
|
Can you share the greek model? |
Beta Was this translation helpful? Give feedback.
-
|
here the setting i use |
Beta Was this translation helpful? Give feedback.
-
|
here sample greek and english you see my method working great ;) BTW: i working in finetune and plan to suport easy finetune for other language i have some ideas to add and tips , you can see here #143 already merge in main repo |
Beta Was this translation helpful? Give feedback.
-
|
@lpscr how did you load the model for the inference since you have changed the language from chinese to greek the vocab.txt must have chanced and so with the default config it would error out, right? |
Beta Was this translation helpful? Give feedback.
-
|
i have create my api script class to work easy load model and more stuff but if you want now something simple just change in interface-clu.py ckpt_path with the file .pt your model you have finetune other take preetrain ! |
Beta Was this translation helpful? Give feedback.
-
|
Hi all, this is very important and might be confusing for some. You need to copy the original model If you start training without copying this model, it will train from scratch! I’ve created a script called finetune-cli.py that can automate this process. However, before running the script, you need to update all the settings accordingly. Please make sure to do this before you start. or you can run simple run simple change only the dataname my_speak in 3090 with about 60-80 hours dataset working well for 4090 like say @JarodMica working very well and also with very big dataset about the vocab i dont replace anything because suport all symbols in language i train make sure if suport all symbols in your language you want to train if miss symbols not working correct or another idea it's in case miss symbols , you can simple covert all symbols in english language , here how check the vocab in finetune_gradio.py make sure in data/project_name/ you have inside metadata.csv for all text
thats why i make gradio_finetune.py to dont confuse for begin users i hope this help |
Beta Was this translation helpful? Give feedback.
-
|
@jpgallegoar Ah! My bad, yes, I'm pretty much topped out on RAM if you had 32GB, but chrome takes up 2 gb 💀 and vscode another 2 gb in this screenshot However, I just stopped training just to check and I'm idling at 20 GB used... Idk what's taking it up exactly but possible some data didn't cleared. Anywho, you can try to reduce the num_workers that are being used, the default it 16. If I looked at Ipscr's code, it's not setting it here so you can pass it, This should help to lower RAM usage I think as it should reduce how much data is being prepared before hand. One more note, you don't want too little workers or else you're GPU won't saturate and it'll train a little slower, so it's a balancing game |
Beta Was this translation helpful? Give feedback.
-
|
In my case, I encountered a problem with missing symbols. As you suggested, maybe changing out unused symbols could help, so does this mean I should add these missing symbols to the unused symbols (and what are they in the vocab.txt file)? I have my own tokenizer for my language, and if I use it, does that mean I need to train from scratch rather than fine-tune? |
Beta Was this translation helpful? Give feedback.
-
youtube tutorial |
Beta Was this translation helpful? Give feedback.
-
|
@jonnytracker check the gradio finetune i have also video tutorial #143 |
Beta Was this translation helpful? Give feedback.
-
|
@JarodMica I have the same issue of out of RAM memory. I have found the cause? and any solutions for fixing it? My case is on supercomputer which has 500GB ram; it still run out of memory every time. |
Beta Was this translation helpful? Give feedback.
-
|
@jpgallegoar I’m trying to train in Spanish as an experiment , let see this take some hours. I just hope the dataset I’m using is okay since I don’t speak Spanish. I’ll let you know soon.
|
Beta Was this translation helpful? Give feedback.
-
|
i just leave to 60k i dont want burn my gpu more... i need make other test also , i try train spanish only for test stuff , like i see you say now working also for you that's great , |
Beta Was this translation helpful? Give feedback.
-
|
Can you send me your model? Perhaps I can keep training it |
Beta Was this translation helpful? Give feedback.
-
|
lol you have 1k sound and i have only 20 hours . thre is not point to compare or train ... |
Beta Was this translation helpful? Give feedback.
-
|
yeah but end result is what matters no? |
Beta Was this translation helpful? Give feedback.
-
|
Hello, I am a native Spanish speaker, I would be honored to help you if you need it. |
Beta Was this translation helpful? Give feedback.
-
|
Given a large dataset, how important is it that the transcription is 1-1 with the source audio? The reason I ask, most of my datasets are built using a Whisper model, and they often do some text compression and correct misspoken words or stutter. Is this TTS-architecture forgiving for those kinds of variations or inconsistencies in transcription, or should I consider using a more verbose Whisper model for creating this dataset? |
Beta Was this translation helpful? Give feedback.
-
|
83C 388W |
Beta Was this translation helpful? Give feedback.
-
|
for me same 83C here @jpgallegoar you have 4090 ? BTW: |
Beta Was this translation helpful? Give feedback.
-
|
I read up to 90 is safe, so 83 should be fine for a few days. I will not do this forever but keep in mind some people mine crypto 24/7 for YEARS and it doesn't break. Same thing |
Beta Was this translation helpful? Give feedback.
-
|
maybe i put in gradio like this you can see your memory and temp in your gpu right now gpu sleep thats why you see 46C |
Beta Was this translation helpful? Give feedback.
-
oh yes I didnt answer I have 4090 @lpscr That looks cool, PR! |
Beta Was this translation helpful? Give feedback.
-
|
Has anyone here tried finetuning the base model on a single speaker dataset? I tried finetuning with a 6 hr English dataset, but I don’t hear any difference after the training. |
Beta Was this translation helpful? Give feedback.
-
|
Change the line I told you to, you're not using the correct model yet
|
Beta Was this translation helpful? Give feedback.
-
Actually it did use my finetuned model and yeah...I don't hear much difference: Here's the audio generated using the base F5-TTS model: https://voca.ro/1hfgaoFTKcIi Here's the audio generated using my finetuned model: https://voca.ro/1gT2MaNzZXW0 The reference audio: https://voca.ro/15snOJ8WmHF5 |
Beta Was this translation helpful? Give feedback.
-
|
The audio should be closer to 15 seconds, at least 10-12. The first mhm part does not get transcribed so for the model, it's a long silence. You're giving a 6 second audio which is 20% silence that's why it's so unnatural. You should use longer audio and splice together the sentences so there's not much silences in between. Either way, the finetuned audio does sound close to the original voice to me, even if the silences didn't dissapear (that can be fixed with a better input audio) |
Beta Was this translation helpful? Give feedback.
-
|
@jpgallegoar The prompt is: Output files are from the finetune model Old reference input audio(6 Seconds) Multiple Reference Input Audio Files Merged (13 Seconds) |
Beta Was this translation helpful? Give feedback.
-
|
I'm facing the same issue at the moment. Finetuning makes no discernable difference. Did you eventually find a way to make it work? I mean, the quality is fine as is, but I was hoping finetuning would make it even better. |
Beta Was this translation helpful? Give feedback.
-
|
After much testing, I'm gonna have to give up on the spanish finetune for now. Anyway, if anyone wants it, here is the model: Link |
Beta Was this translation helpful? Give feedback.
-
|
gracias por compartir, ahi vere como funciona, justo miraba si seria posible hacer un finetuning, en español. estaba pensando si serviria para ello, utilizar audios generados por bark por ejemplo. que uno puede "Crearlos", de cualquier manera es como que no entiendo aun el proceso de finetuning, estaria dando palos de ciego y nose si mi rtx2060 de 12gb sirve para ese proceso. si logras hacer un avance nuevo , estaria feliz de probarlo, y si hay algo que se pueda hacer para colaborar para hacer uno, tambien. Saludos. |
Beta Was this translation helpful? Give feedback.
-
|
jpgallegoar, pregunta desde la absoluta ignorancia, el archivo model_last.pt como lo utilizas? porque veo que en pinokio los modelos los envia al cache del hub y como nombre ilegible, |
Beta Was this translation helpful? Give feedback.
-
hay que hardcodearlo directamente en load_model() de utils_infer.py por ahora. Si en el futuro tenemos varios finetunes decentes, podemos integrarlos en la aplicacion. |
Beta Was this translation helpful? Give feedback.
-
Por ahora creo que el mayor problema son los datos. Si sería posible entrenar el modelo con audios generados por Bark, pero ten en cuenta que la variabilidad de esos datos no sería muy alta, por lo que el finetuning no sería flexible. Se necesita más VRAM para hacer un finetuning, pero puedes contribuir recopilando datasets de alta calidad, por ejemplo. |
Beta Was this translation helpful? Give feedback.
-
|
i ran 50k steps on a 10 hours of professional Arabic audio recordeing segmented in 2 to 17 seconds chunks, with very good transcribe. but the results are no good. it feels like the chinese is over powering it. any suggestions? can it be the vocab.txt file? |
Beta Was this translation helpful? Give feedback.
-
So, try training with very high epoch count and turning off 8-bit Adam optimizer. |
Beta Was this translation helpful? Give feedback.
-
|
ok. I'll try it and comeback with feedback |
Beta Was this translation helpful? Give feedback.
-
|
after 200k steps the results are no good, it feels like the 90k steps version was kind of better, still no good, ay ideas? |
Beta Was this translation helpful? Give feedback.
-
|
Hi, sorry for late reply. Maybe you can look at this one Issue 464. This thread might be helpful. |
Beta Was this translation helpful? Give feedback.
-
|
Hello, has anyone experimented with creating a finetune on another language while adding english audios in the dataset to prevent catastrophic forgetting?
Thanks! |
Beta Was this translation helpful? Give feedback.
-
|
Hi, have you also checked the ability of the model to generate the speech in a language different from the source? For instance, your source is german, while the target text you generate is english. |
Beta Was this translation helpful? Give feedback.
-
|
It just reads the text in my language (spanish), as it would be read if it were spanish text |
Beta Was this translation helpful? Give feedback.
-
|
Hello everyone. I am trying to finetune a model that was previously already finetuned on German. The model which I want to finetune sounds good, but as soon as I try to finetune it on a small single speaker dataset (around 10 minutes) the output is basically just noise. It almost sounds like it is training from scratch and not finetuning the model. Has anybody have similar issues or am I doing something wrong? Or is a dataset of 10 minutes too small? With others this was plenty enough. |
Beta Was this translation helpful? Give feedback.
-
|
Hello, this happened to me before too. First of all make extra sure you're finetuning the model you want, you can change the code to enforce this. Second of all, try reducing the learning rate and testing, and don't train for too long. To be honest I haven't been successful when trying 10m of audio but around 1 hour was better. |
Beta Was this translation helpful? Give feedback.
-
|
So you think it's because of the length of only 10 minutes? I just thought it needs very little since I stay in the same language. I'll try to use a bigger dataset and try to make sure it's fine-tuning the right model. I only tried it with gradio where I ticket the box with fine-tuning although I don't know exactly what it did in code. Probably trying that tomorrow or the day after. |
Beta Was this translation helpful? Give feedback.
-
10min seems like too little data |
Beta Was this translation helpful? Give feedback.
-
|
Hello. I am trying to do single speaker finetuning, English. It training for only 100 epochs too little? Thanks |
Beta Was this translation helpful? Give feedback.
-
|
ok thank you, will try |
Beta Was this translation helpful? Give feedback.
-
would it be better to train for fewer epochs on a bigger dataset? what size of dataset and number of epochs do you suggest? |
Beta Was this translation helpful? Give feedback.
-
|
Beta Was this translation helpful? Give feedback.
-
|
Btw, when I said " You can easily use that 6.5 hrs single speaker english dataset, and make the model speak with that speaker's voice by training for just 20 epochs. |
Beta Was this translation helpful? Give feedback.
-
understood, will try, thank you for the clarification |
Beta Was this translation helpful? Give feedback.
-
|
alternatively if overtraining is a big problem, modify the learning rate to be slightly less so the model doesn't converge as quickly.
Get Outlook for Android<https://aka.ms/AAb9ysg>
…________________________________
From: Alykasym Begov ***@***.***>
Sent: Saturday, January 4, 2025 1:43:12 PM
To: SWivid/F5-TTS ***@***.***>
Cc: Robert Agee ***@***.***>; Mention ***@***.***>
Subject: Re: [SWivid/F5-TTS] Finetune practice (Discussion #57)
100 epochs seems too much because:
1. The base model already supports English language, so it doesn't need to learn the language but the voice, which doesn't need much training.
2. 6.5 hrs is a small dataset and training it for more than 20-30 epochs will easily overfit the model.
For starters, maybe you can just use Auto Settings from the WebUI, only changing the batch size to match your VRAM, and set the epoch count to something like 1000000, but manually stop around 15-20 epochs during training.
—
Reply to this email directly, view it on GitHub<#57 (reply in thread)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AMWGMPIEKLQRKTUA36G74CL2JATUBAVCNFSM6AAAAABP4ADUBSVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTCNZTGQ4TIMQ>.
You are receiving this because you were mentioned.Message ID: ***@***.***>
|
Beta Was this translation helpful? Give feedback.
-
|
ok thanks |
Beta Was this translation helpful? Give feedback.
-
|
anyone has anyl luck training Arabic? any ideas? i have more audio for the same speaker. more than 100 hours. if that would help |
Beta Was this translation helpful? Give feedback.
-
|
Maybe the issue with the vocab? Did you check vocab whether it contains all the characters? Also can you share your config parameters? Like batch size, learning rate, warmup updates and etc... |
Beta Was this translation helpful? Give feedback.
-
|
How to train the model? Do you have a video tutorial for this? |
Beta Was this translation helpful? Give feedback.
-
|
Have you guys found a good solution for splitting long audio files into shorter ones? |
Beta Was this translation helpful? Give feedback.
-
|
@isolveit-aps The one you shared is much better and works extremely fast. Thank you very much |
Beta Was this translation helpful? Give feedback.
-
Beta Was this translation helpful? Give feedback.
-
|
I wrote a couple of scripts, the basis of which is whisperx, of course together with an aligning (wav2vec) model. So far, this solution has given me the most accurate results, albeit in Hungarian. After cutting, I scan the pieces again with whisperx, I no longer align them here. I compare the second reading with the pieces based on their text, and pronounce the pieces recognized as different languages. I just wrote the cutting script, you can change the distribution curve, the target length and the standard deviation. I'll make a small github repo if you're interested, because my previous database creation repo is out of date. |
Beta Was this translation helpful? Give feedback.
-
|
@sarpba Thank you, that sounds interesting for everyone here! |
Beta Was this translation helpful? Give feedback.
-
|
@ukemamaster Have you been able to make timething actually run? I've now tried it with various versions of python and dependencies, but I am unable to get it running. But the way it's described, it should do exactly what I'm after, so I would love to get it working. |
Beta Was this translation helpful? Give feedback.
-
|
Hi @jpgallegoar, Thanks for your Spanish model, it works great. Could you please share your experience?
Thanks |
Beta Was this translation helpful? Give feedback.
-
|
Hello, I think you should use more speakers if you want a good generalized model, 3 speakers will probably overfit it if trained with 370 hours. I would start from my model and use less hours, but if you manage to get more speakers (>100h of >50 distinct speakers) you can start from scratch. If you decide to start from my model, I would use around 20h of those speakers if you want it to maintain some generalization, and lower the learning rate to 5e-6 or 7.5e-6. I am also in the process of training a peninsular spanish model. best of luck and keep me updated :D |
Beta Was this translation helpful? Give feedback.
-
|
@jpgallegoar Yeah sure, will keep you updating. |
Beta Was this translation helpful? Give feedback.
-
|
@jpgallegoar And what do you mean by "generalization" here? You mean generalization in accent? To have both accents? Or generalization in cloning abilities in zero shot cloning? |
Beta Was this translation helpful? Give feedback.
-
I mean voice cloning generalization. If your dataset only has 1 person, you can only clone that person. If it has more people, the model starts to learn how to create every voice. |
Beta Was this translation helpful? Give feedback.
-
|
I've found F5 for Turkish https://huggingface.co/marduk-ra/F5-TTS-Turkish but it has some turkish character problem like ı,ü,ö,ç Unfortunately, I only have 6800xt. What do you suggest? What could be the problem? I'm considering hiring GPU and train in the cloud. |
Beta Was this translation helpful? Give feedback.
-
|
You can train on Runpod, for example. |
Beta Was this translation helpful? Give feedback.
-
|
@jpgallegoar For sake of using the output for commercial purpose, i cannot train based on the previous training which based on https://huggingface.co/datasets/amphion/Emilia-Dataset How much time needed for this for example? Also i do not have any experience in this field except i'm fullstact web dev and know linux. There is data, there is script so i just need to run (this is my dream, but facts probably differ ^^) |
Beta Was this translation helpful? Give feedback.
-
|
Around 100-150 hours for new language should be good enough, with many different speakers (try max 2h per speaker) so the model learns how to voice clone any voice |
Beta Was this translation helpful? Give feedback.
-
|
Has anyone tested training on fp32 vs fp16 vs bf16? Is there a noticeable quality dropoff? Which is the best? |
Beta Was this translation helpful? Give feedback.
-
The truth is 2. and 3. depends only on the quality of your transcriptions. If your transcriptions are 100% and your reference audio, text and gen text is in the domain of your dataset (not training on normal voice and then using cartoon voice with fast speech or things like that), the generated audio will be perfect. On 100h, 1.2 milliok steps, it's really really good. And number 1 depends on the variety of speech patterns of your dataset. If you have only 5 speakers, it won't be able to clone the voices very well, but if you have 100+ speakers with different voices or something like that, it will learn to generalize and clone any voice. (Again, if you train only on normal voices, don't expect the model to clone a very high pitched cartoon voice, for example) |
Beta Was this translation helpful? Give feedback.
-
|
Hi! Just wanted to share some findings from my recent experiments. I tested the impact of different batch sizes on the quality of speech synthesis. The dataset I used is a 35-hour, high-quality, multi-speaker dataset with accurate annotations. I fine-tuned two models with identical configurations, only changing the batch size: one with a batch size of 3000 and the other with 500. Both were trained for 50 epochs using a learning rate of 1e-5. Surprisingly, the model trained with a batch size of 500 produced more accurate speech compared to the one with a batch size of 3000. The 500 batch size run left more than half of my VRAM unused during fine-tuning and took slightly longer, but the improved results made it worth the trade-off. I’m not entirely sure why this happened, as everything else about the models was identical. I plan to dig into the code and experiment further when I have more time, but for now, I thought I’d share these results in case anyone finds them useful. |
Beta Was this translation helpful? Give feedback.
-
|
I just think the higher batch size one needs longer to learn. I am absolutely certain the ceiling of higher batch size is higher than lower batch size (spent hundreds renting H100 / H200 and locally 4090). If you give it enough time, it will improve. For reference. my 50h dataset was trained on 1300 epochs on 12000 batch size. I'm pretty sure it's 99-100% accurate on a new language. |
Beta Was this translation helpful? Give feedback.
-
|
@Alykasym smaller batch size provides more optimization opportunities because the number of steps increases. This is partly why you may see better results compared to a larger batch size. |
Beta Was this translation helpful? Give feedback.
-
When increasing batch size, you should typically increase the learning rate. A common rule is linear scaling: multiply the learning rate by the ratio. it's compensation for reduced gradient noise in larger batches. If you don't adjust it, training might slow down or become unstable. Can be that's why... |
Beta Was this translation helpful? Give feedback.
-
|
I'm finetuning models with F5-TTS via Pinokio but i'm struggling to identify how to use the models i've trained Would a kind person possibly update the Gradio UI for Pinokio and add an ability to automatically find and be able to select any of the finetuned models that have been trained / created to make it easy please? 😊 |
Beta Was this translation helpful? Give feedback.
-
|
@holycowdude The easyest way use ComfyUI with this cudtom node https://github.com/niknah/ComfyUI-F5-TTS |
Beta Was this translation helpful? Give feedback.
-
|
I'm training 200hrs for pt-br reaching 1M steps, using google colab, half with A100 and half with T4, but it still not perfect, it is actually doing a little inference, but have some misspellings, and for numbers, just does not work. Is it possible to finetune it with a new dataset with only numbers and misspells? will it destroy the previous trainings? |
Beta Was this translation helpful? Give feedback.
-
|
No need to regenerate the data. You could just concatenate your audios if spoken by the same speaker. It doesn't matter, if the audio contains 3 spoken sentences or 1 spoken long sentence. Just make the silence padding consistent between sentences when concatenating. You could also aim for a normal distribution of samples between 3-30 secs. |
Beta Was this translation helpful? Give feedback.
-
|
Yes but if he has the long audios, and the transcriptions were for the long audios, it's better to resplit them to avoid unnatural timings and artifacts when rejoining them. |
Beta Was this translation helpful? Give feedback.
-
|
The single splits should be "logical units", i.e. indepent utterances that make sense standalone. But it's okay to have 3 such utterances concatenated together, like:
|
Beta Was this translation helpful? Give feedback.
-
|
Another question, if I want to make it pt-br + en, can I use the same dataset, with both languages? will it be able to speak both? or for multi language is different? pt-br I used the same vocab.txt from original, I want to add english to it as many words in portuguese are english words, specially on tech industry. |
Beta Was this translation helpful? Give feedback.
-
|
Yes, add good amount, though (e.g. LJSpeech). It will have an portugese accent, though. |
Beta Was this translation helpful? Give feedback.
-
|
Hello, I would like to re-finetune the hungarian model again from the original. I collect about 2600 hours of ultra clear audio from about 50 speakers. Unfortunately, the average audio length is around 5 seconds for me as well. Is there an ideal curve for the distribution of sounds? The amount of data is abundant, I can select a data set corresponding to an ideal distribution curve. edit: I think i need to reroll my dataset, it's worse than I thought. |
Beta Was this translation helpful? Give feedback.
-
That's probably true. The model generalizes better at larger batch values. The question is whether the higher bach size achieved with gradient accumulation is equivalent to the large bach_size achieved in 1 step with higher vram. |
Beta Was this translation helpful? Give feedback.
-
Unfortunately I don't think anyone has made that test yet. |
Beta Was this translation helpful? Give feedback.
-
|
@jpgallegoar @sch0ngut here is the scripts https://github.com/sarpba/ADCS I no have more time now. It's working, but not too nice (gpt translated) and the last scripts is missing. (trash_dropout, numbers_drouout, csv_maker) I'll continue friday night. under windows need to use WSL. I ran a quick test. I processed 20 hours of audio in 35 minutes. it turned out to be about 15 hour usable data. (2x3090) |
Beta Was this translation helpful? Give feedback.
-
|
Awesome, thank you! |
Beta Was this translation helpful? Give feedback.
-
|
@sarpba What software/library did you use to create that kind of graph for distribution of audio durations? |
Beta Was this translation helpful? Give feedback.
-
|
Has anyone able to train this on Multiple-4090 GPU's setup (2 or more)?? I am getting this - #728 (comment) |
Beta Was this translation helpful? Give feedback.
-
|
I was able to do it, but it was from Replicate |
Beta Was this translation helpful? Give feedback.
-
|
Has anyone tried parallelized training with multi GPUs? I mean getting parallel performance, not only more VRAM. Is it even possible? |
Beta Was this translation helpful? Give feedback.
-
|
I'm sorry, I'm not sure. |
Beta Was this translation helpful? Give feedback.
-
|
Since NVIDIA's consumer GPUs, such as the 4090 and 3090, do not support P2P communication, they may not provide significant speedup in multi-GPU training. Instead, you could try training with NVIDIA's data center GPUs. Rather than using 4x 4090, you might consider renting a single H100 or A100. @jpgallegoar |
Beta Was this translation helpful? Give feedback.
-
|
@hcsolakoglu Thank you for your answer, that is in fact what I ended up doing. I even rented H200 because I calculated it was the most efficient. I want to spend the same amount of money (or a little bit more) and train in less time via parallelization. Do you think it would work with these server GPUs? |
Beta Was this translation helpful? Give feedback.
-
|
@hcsolakoglu @jpgallegoar theoretically: So there is an increase in speed, just in a different way. My train with 3200 batch size / 2x3090 GPU (NVlik connected) 6400 owerall batch size -> I have 7-8 update/s (underpowered to 280watt) |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for the answer, the speedup would be nice for commercial purposes. I did 9600 batch size and the results were very good indeed. Perhaps I am confused and it was speeding up, I will have to test again. |
Beta Was this translation helpful? Give feedback.
and others
-
Full finetune is currently supported, lora or adapter not yet.
checkpoint_pathto pretrained model dir intest_train.py,model/trainer.pywill load from there to resume. Reuse thevocab.txtunderdata /Emilia_ZH_EN_pinyin(Emilia_ZH_EN_pinyin <- tokenizer = "pinyin"; dataset_name = "Emilia_ZH_EN" intest_train.pysetting)model/dataset.py. Just need e.g. the audio path, text (tokenized, leverageconvert_char_to_pinyinfunc inmodel/utils.pysee script/prepare_xxxx.py), duration of audio in seconds.grad_accumulation_stepscould be used to simulate a large batchsize. Also other settings, e.g. few warmup steps, 1e-4 lr, etc.We didn't specifically experiment with finetuning, so if you get positive results, welcome to share :)
Some helpful issues, #16 #27
Welcome to share your successful results with finetuning, maybe also start a new tutorial doc helping others to get start with it.
Many Thanks !
Beta Was this translation helpful? Give feedback.
All reactions