Building A New Activity

Simple Activity

As Mema3 is partly based on repurposed Rhasspy which is a smart speaker and home automation platform, it is very open to new activities. This is analogous to adding a new skill to a home automation server or smart speaker. Normally some Python3 will be required for this.

The steps are:

1. Talk to users and then define the voice commands that start the activity. There can be several, see the current file.

2. Put these into sentences.ini via http://<ip_address>:12101/ see left hand menu, save and retrain (usually less than a minute)

3. Add the new intent keyword into the dispatch table in intent_server.py # intents are the options see: https://stackoverflow.com/questions/17881409/whats-an-alternative-to-if-elif-statements-in-python {

    `'TakePhoto'  : run_photo_command,`
    `'TellStory'  : print("tell story found"),   `
    `'Associate'  : print("associate photo and story found"),`
    `'RecordStory': run_record_command,`
    `'RecordVideo': run_video_command,`
    `'KillVideo'  : run_kill_video_command,  `
    `'LabelVideo' : run_label_video_command,       `
    `'SlicePie'   : run_pie,`
    `'DuckDuck'   : run_search_command  `
   

and generally: '<new_intent>' : <run_new_command> in intent_server.py

4. Make a function in the intent server for the new command.

5. If necessary write a script that is called by the function take_picture.py is an example, take a look. record_story.py is more complex because it swaps out rhasspy in order to use the microphone.

6. Normally the scripts return <something>|media_path which is then used for the sqllite3 database

7. Add any necessary entries in etc/mema.ini to be used by the script. Note that there are two versions of mema.ini, one for laptop and one for Pi in the etc folder.

8. Test the script, you have written in 5. by itself on the command line before integration

Also, if there is a web element, templates are in ~/mema/templates/en (to preserve multilingual aspects later on) and static pages are in ~/mema/static (some subdirectories for the media files here too).

This, of course, is not the only way to evolve the project, for example there are cron scripts in mema/etc/cron/ for making a wordcloud and doing background voice to text transcription. None of these are tested or switched on at time of writing as they are probably not crucial to initial demonstrations.

MQTT and Specialised Server Based Activity

Since Rhasspy is publishing and listening to events on an external (to the Docker image) message broker, it is possible to additions directly from MQTT message subscription and run them as a systemd (for example) service, since they'll always need to listen or publish. face_unlock.py and unlock_server.service are examples of this. There are also possible modifications needed in the node-red flow to make sure that anything coded like this doesn't go into the main intent_server.py. Best way to start an activity like this is to examine the face unlock code and configuration.