Setup real dashcam logging

[adeebshihadeh]

This project is to maximize the usable dashcam storage space on a comma 3X. Currently, it's tuned for maximum training/debuggability. We should be able to tradeoff a bit of that for a great dashcam experience.

.hevc and rlog are needed for training and debugging, but they also take up most of the device's storage. They are overkill for dashcam purposes, but the current qcamera isn't quite sufficient either.

Some concrete TODOs

• deleter: setup two classes of files "dashcam" and "training"
• deleter: prioritize "dashcam" segments
• logger: bump up qcamera size to be great dashcam footage
• logger: setup qcamera streams for the wide and driver cams

---

Here's the current size breakdown:

2.2M    qcamera.ts
2.2M    qlog
56M     rlog
72M     dcamera.hevc
72M     fcamera.hevc
73M     ecamera.hevc

[incognitojam]

On the comma 3X I just took out of my car:

• /dev/sda12 /data partition is 88GB (92% used, 7.5G available)
  • 69G /data/media
  • 4.5G /data/safe_staging
  • 3.1G /data/openpilot
  • 1.9G /data/scons_cache
• There are 274 route segment folders in /data/media/0/realdata
  • 1.3G boot folder (925 files) and 2.4G crash folder (486 files)

I wonder if these folders are the same size on other devices?

Assuming all 274 route segments are full minutes that would be 4.56 hours, but if we take the expected file sizes for a 1 minute segment as 277.4MB (qcam+qlog+rlog+dcam...) then 69GB/277.4MB gives 254.7 minutes or 4.25 hours.

[incognitojam]

qcam + qlog (4.4M) is about 1.5% the size of the rest of the logs (275M). If we 4x the size of the qcam and had recordings for all three cameras it would be 28.6M (10.4%). On my 3X that would leave space for 238 minutes (just under 4 hours) without changing the priorities. Will have to see how good a 9MB qcam can be...

[adeebshihadeh]

It's roughly the same on all devices.

• /data is the same size on all 3X
• /data/safe_staging and /data/openpilot will get much smaller with the new non-git updater
• release has no scons cache, so you can ignore that one
• we can tune the 7.5G available in deleter

On my 3X that would leave space for 238 minutes (just under 4 hours) without changing the priorities. Will have to see how good a 9MB qcam can be...

I'm thinking we only keep a tiny percent of training to start (5%?).

[incognitojam]

Is 20 minutes of training data enough? That's about 5.3GB we could keep or 6.6%

[adeebshihadeh]

The actual number we set it to isn't super relevant. We just need to setup the slider, then research can set it to whatever they want or ship something to mine for interesting stuff on-device.