Best practices for training new models

[dscripka]

Training new models is an important but complex feature of openWakeWord, and over time I'd like to accumulate a set of best-practices and helpful examples/scripts that can make this process easier and more repeatable.

Right now, these approaches have proven useful as I continue to train and test new models. As confidence in the effectiveness of these increase over time and usage, they will be added to updated training code and example notebooks:

1. Use a development/validation set to track important metrics like Recall and false-positive rate per hour. I've noticed that these metrics can still change substantially even after the loss appears to have reached a steady state, or even begins to increase

2. Train for a relatively large number of epochs (>50), even with large datasets

3. Slowly decay the learning rate over time, and/or use cyclical learning rates to encourage exploration during the initial stages of training and stability at the end of training.

4. If using class weights during training, slowly increase the weight on the negative class as training progresses to penalize false-positive more strongly over time

Feel free to ask questions and give your own suggestions for best-practices when training new models!