TinyML book Ch8 - strategies for debugging models

drdunc · December 15, 2020, 9:40am

Hi all, I am teaching “Built Environment” students how to use TinyML so that we can instrument our built and natural environment. Working through the TinyML book the examples are all great. However, when I train my own models the results tend to be grim. For example, Chapter 7 micro_speech wake words “yes, no” = success. Chapter 8 micro_speech any other combinations of words (e.g. “on, off”) don’t seem to work for me.

QUESTION: does anyone have any guides for strategies around how to investigate how a model is working or what levers to tweak - for example, where do people start in trying to debug why the model is not recognising “on”. I think this is a really interesting challenge for students since we often give the them examples that work for teaching but then when they try and do their own project they start to struggle.

Thanks!
Duncan

petewarden · December 16, 2020, 12:41am

That’s a great question! I originally started with “on”/“off” but found they were hard for the model to distinguish, so I switched to “yes”/“no” instead.

One thing I like to try is running an evaluation on the Python side, and listening to particular samples that passed and failed. The students might find the “Testing the model accuracy” section in this Colab notebook a good starting point:

github.com

tensorflow/tensorflow/blob/master/tensorflow/lite/micro/examples/micro_speech/train/train_micro_speech_model.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "train_micro_speech_model.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {

This file has been truncated. show original

It loads a selection of WAVs, and runs them through the model. If you add some logging you could figure out which files pass and which fail, and then listen to them to see if there are any clues about why the model is struggling.

drdunc · December 16, 2020, 11:07pm

Cheers - good suggestion - will take a look at logging some of the samples.

machinethink.net · January 6, 2021, 2:01pm

I just recently started to work with the example from ch.8. It worked a lot better after I fixed a few bugs in the audio capture code. This is on the Arduino Nano Sense.

The fixes are in this pull request, in case you’re interested in trying them out (and if they’re not yet merged into the repo by the time you read this): https://github.com/tensorflow/tensorflow/pull/45878

Using a model trained with on/off worked (almost) as well as the yes/no one with these fixes.

I went through the whole code and verified that the spectrograms (aka features) generated on the Arduino are the exact same spectrograms as generated by the TF training code on the same input data. (Using TF 2.1 and CUDA 10.1.)