Gesture Recognition

Learning Goals 5 min

Yesterday you trained the model in Python. Today you embed it into an Arduino sketch and run real-time inference on the Nano 33 BLE Sense. Your "wave" / "punch" / "circle" gestures become detected events. By the end of this lesson you will:

1. Include the TFLite Micro library + your gesture_model.h in an Arduino sketch.
2. Capture a window of accelerometer samples, feed them to the model, get back a class prediction.
3. Trigger different actions per gesture (LED colour, buzzer tone, etc.).

Warm-Up 10 min

Hardware: Nano 33 BLE Sense + the LED + 220 Ω + buzzer for output.

Install the TFLite library

Library Manager → search "Arduino_TensorFlowLite" → install. Note: the official Arduino-maintained one is the most reliable.

Drop in the model

Put gesture_model.h (from yesterday) next to your .ino file in the sketch folder.

New Concept · TFLM inference in 30 lines 25 min

Boilerplate

#include <TensorFlowLite.h>
#include <Arduino_LSM9DS1.h>
#include "gesture_model.h"

#include "tensorflow/lite/micro/all_ops_resolver.h"
#include "tensorflow/lite/micro/micro_error_reporter.h"
#include "tensorflow/lite/micro/micro_interpreter.h"
#include "tensorflow/lite/schema/schema_generated.h"

// Tensor arena — scratch RAM for the interpreter
constexpr int kTensorArenaSize = 32 * 1024;
uint8_t tensor_arena[kTensorArenaSize];

const tflite::Model* model = nullptr;
tflite::MicroInterpreter* interpreter = nullptr;
TfLiteTensor* input = nullptr;
TfLiteTensor* output = nullptr;

const int WINDOW = 150;   // 1.5 s at 100 Hz, matching training
const char* LABELS[] = {"wave", "punch", "circle"};
const int NUM_CLASSES = 3;

setup()

void setup() {
  Serial.begin(9600);
  while (!Serial);

  if (!IMU.begin()) { Serial.println("# IMU init failed"); while(1); }

  static tflite::MicroErrorReporter micro_error_reporter;
  model = tflite::GetModel(gesture_model);
  if (model->version() != TFLITE_SCHEMA_VERSION) {
    Serial.println("# model schema mismatch");
    while (1);
  }

  static tflite::AllOpsResolver resolver;
  static tflite::MicroInterpreter static_interpreter(
      model, resolver, tensor_arena, kTensorArenaSize, &micro_error_reporter);
  interpreter = &static_interpreter;

  TfLiteStatus allocate_status = interpreter->AllocateTensors();
  if (allocate_status != kTfLiteOk) { Serial.println("# alloc fail"); while(1); }

  input  = interpreter->input(0);
  output = interpreter->output(0);

  Serial.println("# ready");
}

The inference loop

void loop() {
  // Wait for motion to begin (acceleration spike above threshold)
  float ax, ay, az;
  if (!IMU.accelerationAvailable()) return;
  IMU.readAcceleration(ax, ay, az);

  float mag = sqrt(ax*ax + ay*ay + az*az);
  if (mag < 1.5) return;     // standing still (just gravity ~1g)

  // Capture WINDOW samples
  int samplesRead = 0;
  while (samplesRead < WINDOW) {
    if (IMU.accelerationAvailable()) {
      IMU.readAcceleration(ax, ay, az);
      input->data.f[samplesRead*3 + 0] = ax;
      input->data.f[samplesRead*3 + 1] = ay;
      input->data.f[samplesRead*3 + 2] = az;
      samplesRead++;
    }
  }

  // Run inference
  if (interpreter->Invoke() != kTfLiteOk) { Serial.println("# invoke failed"); return; }

  // Find the highest-scoring class
  int best = 0;
  float bestScore = output->data.f[0];
  for (int i = 1; i < NUM_CLASSES; i++) {
    if (output->data.f[i] > bestScore) {
      bestScore = output->data.f[i];
      best = i;
    }
  }

  if (bestScore > 0.8) {
    Serial.print("Gesture: ");
    Serial.print(LABELS[best]);
    Serial.print(" ("); Serial.print(bestScore, 2); Serial.println(")");
  } else {
    Serial.println("# unclear");
  }

  delay(500);    // debounce: rest before next gesture
}

That's the whole inference: motion trigger → window capture → invoke model → highest-scoring class. About 100 ms of compute on the Nano 33 BLE.

Worked Example · Trigger actions per gesture 25 min

Wire output devices

RGB LED + 220 Ω each, buzzer on D10.

Action handlers

void onWave() {
  setColor(0, 0, 255);   // blue
  tone(BUZZER, 880, 200);
}

void onPunch() {
  setColor(255, 0, 0);   // red
  tone(BUZZER, 440, 100);
}

void onCircle() {
  setColor(0, 255, 0);   // green
  tone(BUZZER, 660, 300);
}

In the loop, after determining the best class:

if (bestScore > 0.8) {
  if (best == 0) onWave();
  else if (best == 1) onPunch();
  else if (best == 2) onCircle();
}

Test

1. Hold the Nano. Wave it. LED blue + chime.
2. Punch motion. LED red + low beep.
3. Circle. LED green + high note.

Tuning the confidence threshold

If you get false positives (LED lighting on non-gesture motion), raise the threshold from 0.8 to 0.9. If you get false negatives (legit gestures ignored), lower to 0.7. Trade-off.

Recover from drift

After ~30 minutes the gestures may seem less accurate. Re-capture data and re-train if so. Hand position drift, battery effects, etc. accumulate.

Try It Yourself 15 min

Mini-Challenge · Demo to a friend 10 min

1. Hand the Nano to a classmate (battery + power bank or USB).
2. Show them the three gestures.
3. Have them try. Does it recognise their gestures?
4. Discuss: did training on only your data cause misses for them? (Almost certainly yes for some gestures.)

Recap 5 min

Gesture recognition = capture sensor window → feed into TFLite Micro model → take argmax → trigger action. Confidence threshold tunes false-positive vs false-negative. The Nano 33 BLE Sense runs this inference in ~100 ms. Tomorrow: build a full wand product with multiple gestures + LED feedback + battery.

Tensor arena

Pre-allocated RAM the TFLM runtime uses for intermediate tensors during inference.

Invoke / inference

Running the model on input data to get output. interpreter->Invoke().

argmax

Finding the index of the largest value in an output vector. The predicted class.

Confidence threshold

Minimum predicted probability before accepting the classification. Trades false-positive vs false-negative.

Motion trigger

Logic that captures a window only when motion exceeds a threshold. Avoids running inference on idle data.

Debounce

Delay between accepted gestures so one gesture doesn't register twice.

Homework 5 min

1. Get 3 gestures working with action triggers (LED + buzzer).
2. Record a video of all 3 firing reliably.
3. Read ahead to ARD-L04-36 (Magic-Wand Gesture Trigger).