Gesture Recognition

Learning Goals 5 min

Yesterday you trained the model in Python. Today you embed it into an Arduino sketch and run real-time inference on the Nano 33 BLE Sense. Your "wave" / "punch" / "circle" gestures become detected events. By the end of this lesson you will:

1. Include the TFLite Micro library + your gesture_model.h in an Arduino sketch.
2. Capture a window of accelerometer samples, feed them to the model, get back a class prediction.
3. Trigger different actions per gesture (LED colour, buzzer tone, etc.).

Warm-Up 10 min

Hardware: Nano 33 BLE Sense + the LED + 220 Ω + buzzer for output.

Install the TFLite library

Library Manager → search "Arduino_TensorFlowLite" → install. Note: the official Arduino-maintained one is the most reliable.

Drop in the model

Put gesture_model.h (from yesterday) next to your .ino file in the sketch folder.

New Concept · TFLM inference in 30 lines 25 min

Boilerplate

#include <TensorFlowLite.h>
#include <Arduino_LSM9DS1.h>
#include "gesture_model.h"

#include "tensorflow/lite/micro/all_ops_resolver.h"
#include "tensorflow/lite/micro/micro_error_reporter.h"
#include "tensorflow/lite/micro/micro_interpreter.h"
#include "tensorflow/lite/schema/schema_generated.h"

// Tensor arena — scratch RAM for the interpreter
constexpr int kTensorArenaSize = 32 * 1024;
uint8_t tensor_arena[kTensorArenaSize];

const tflite::Model* model = nullptr;
tflite::MicroInterpreter* interpreter = nullptr;
TfLiteTensor* input = nullptr;
TfLiteTensor* output = nullptr;

const int WINDOW = 150;   // 1.5 s at 100 Hz, matching training
const char* LABELS[] = {"wave", "punch", "circle"};
const int NUM_CLASSES = 3;

setup()

void setup() {
  Serial.begin(9600);
  while (!Serial);

  if (!IMU.begin()) { Serial.println("# IMU init failed"); while(1); }

  static tflite::MicroErrorReporter micro_error_reporter;
  model = tflite::GetModel(gesture_model);
  if (model->version() != TFLITE_SCHEMA_VERSION) {
    Serial.println("# model schema mismatch");
    while (1);
  }

  static tflite::AllOpsResolver resolver;
  static tflite::MicroInterpreter static_interpreter(
      model, resolver, tensor_arena, kTensorArenaSize, &micro_error_reporter);
  interpreter = &static_interpreter;

  TfLiteStatus allocate_status = interpreter->AllocateTensors();
  if (allocate_status != kTfLiteOk) { Serial.println("# alloc fail"); while(1); }

  input  = interpreter->input(0);
  output = interpreter->output(0);

  Serial.println("# ready");
}

The inference loop

void loop() {
  // Wait for motion to begin (acceleration spike above threshold)
  float ax, ay, az;
  if (!IMU.accelerationAvailable()) return;
  IMU.readAcceleration(ax, ay, az);

  float mag = sqrt(ax*ax + ay*ay + az*az);
  if (mag < 1.5) return;     // standing still (just gravity ~1g)

  // Capture WINDOW samples
  int samplesRead = 0;
  while (samplesRead < WINDOW) {
    if (IMU.accelerationAvailable()) {
      IMU.readAcceleration(ax, ay, az);
      input->data.f[samplesRead*3 + 0] = ax;
      input->data.f[samplesRead*3 + 1] = ay;
      input->data.f[samplesRead*3 + 2] = az;
      samplesRead++;
    }
  }

  // Run inference
  if (interpreter->Invoke() != kTfLiteOk) { Serial.println("# invoke failed"); return; }

  // Find the highest-scoring class
  int best = 0;
  float bestScore = output->data.f[0];
  for (int i = 1; i < NUM_CLASSES; i++) {
    if (output->data.f[i] > bestScore) {
      bestScore = output->data.f[i];
      best = i;
    }
  }

  if (bestScore > 0.8) {
    Serial.print("Gesture: ");
    Serial.print(LABELS[best]);
    Serial.print(" ("); Serial.print(bestScore, 2); Serial.println(")");
  } else {
    Serial.println("# unclear");
  }

  delay(500);    // debounce: rest before next gesture
}

That's the whole inference: motion trigger → window capture → invoke model → highest-scoring class. About 100 ms of compute on the Nano 33 BLE.

Worked Example · Trigger actions per gesture 25 min

Wire output devices

RGB LED + 220 Ω each, buzzer on D10.

Action handlers

void onWave() {
  setColor(0, 0, 255);   // blue
  tone(BUZZER, 880, 200);
}

void onPunch() {
  setColor(255, 0, 0);   // red
  tone(BUZZER, 440, 100);
}

void onCircle() {
  setColor(0, 255, 0);   // green
  tone(BUZZER, 660, 300);
}

In the loop, after determining the best class:

if (bestScore > 0.8) {
  if (best == 0) onWave();
  else if (best == 1) onPunch();
  else if (best == 2) onCircle();
}

Test

1. Hold the Nano. Wave it. LED blue + chime.
2. Punch motion. LED red + low beep.
3. Circle. LED green + high note.

Tuning the confidence threshold

If you get false positives (LED lighting on non-gesture motion), raise the threshold from 0.8 to 0.9. If you get false negatives (legit gestures ignored), lower to 0.7. Trade-off.

Recover from drift

After ~30 minutes the gestures may seem less accurate. Re-capture data and re-train if so. Hand position drift, battery effects, etc. accumulate.

Basic 5 min

Challenge 1 5 min

Challenge 2 5 min

Challenge 3 · Fix the gesture picker 10 min

After Invoke(), the sketch must pick the class with the highest score. This test sketch fakes three model outputs, so it runs on any board. The model was trained with its classes in the order wave, punch, circle. There are four mistakes.

const int NUM_CLASSES = 3;
const char* LABELS[] = {"punch", "wave", "circle"};

float testScores[3][NUM_CLASSES] = {
  {0.91, 0.06, 0.03},
  {0.10, 0.85, 0.05},
  {0.30, 0.35, 0.35}
};

void pickGesture(float scores[]) {
  int best = 0;
  float bestScore = scores[0];
  for (int i = 1; i <= NUM_CLASSES; i++) {
    if (scores[i] > bestScore) {
      bestScore = scores[i];
    }
  }
  if (bestScore > 80) {
    Serial.print("Gesture: ");
    Serial.println(LABELS[best]);
  } else {
    Serial.println("# unclear");
  }
}

void setup() {
  Serial.begin(9600);
  for (int t = 0; t < 3; t++) {
    pickGesture(testScores[t]);
  }
}

void loop() {
}

Find each mistake and fix it. Then predict what the three tests should print before you upload.

It works if the Serial Monitor prints Gesture: wave, Gesture: punch and # unclear, in that order.

Recap 5 min

Gesture recognition = capture sensor window → feed into TFLite Micro model → take argmax → trigger action. Confidence threshold tunes false-positive vs false-negative. The Nano 33 BLE Sense runs this inference in ~100 ms. Tomorrow: build a full wand product with multiple gestures + LED feedback + battery.

Tensor arena

Pre-allocated RAM the TFLM runtime uses for intermediate tensors during inference.

Invoke / inference

Running the model on input data to get output. interpreter->Invoke().

argmax

Finding the index of the largest value in an output vector. The predicted class.

Confidence threshold

Minimum predicted probability before accepting the classification. Trades false-positive vs false-negative.

Motion trigger

Logic that captures a window only when motion exceeds a threshold. Avoids running inference on idle data.

Debounce

Delay between accepted gestures so one gesture doesn't register twice.

Extra Mission 5 min

Part 1 — Design a gesture gadget for someone else

A model trained only on your hand may miss other people's gestures. On paper, design a gesture-controlled gadget for someone else to use. It might be a lamp you switch with a flick when your hands are full, or a page-turner for a musician.

Your design must include:

• Who uses it and the three actions it needs.
• Which gesture triggers each action, and why it is easy for that person.
• How many people you need training data from, and why.
• What the gadget does when it is unsure.

Part 2 — Make it

Map your three trained gestures to that gadget's actions using an LED, a buzzer or a servo. Add a counter that prints how many gestures were recognised. Ask two other people to try each gesture ten times, and record the hits.

Bring back next class: your uploaded sketch, a short video of all three actions firing, and a Serial Monitor screenshot with each tester's hit count.