Lip-Sync With Costumes — Scratch L4

Where we are now: the puppets are built (Arc 1). Today they get a voice — mouths sync to recorded audio so the short feels alive, not like a silent slideshow.

Learning Goals 3 min

By the end of this lesson you will be able to:

Explain why a random mouth-flap (from L04-25) looks fake the moment a real voice plays — and what "lip-sync" actually means: the mouth opens only while a sound is making noise.
Use a controller script with play sound (line1 v) until done running in parallel with a forever loop that broadcast (mouth-open v)s and broadcast (mouth-closed v)s on a 0.15-second rhythm.
Wire up the mouth sprite with two when I receive hats — one per broadcast — each calling switch costume to (open v) or switch costume to (closed v). Stop the flap when the voice line ends.

Warm-Up — why random flapping looks wrong 7 min

In L04-25 you built Auntie Siti with a mouth that flapped every 0.2 seconds, forever. That works for a silent puppet — the eye is happy as long as something is moving. The moment you add a voice, the eye stops being polite. If the mouth flaps while there's silence, or stays closed while a voice is speaking, the illusion shatters. You've watched a badly-dubbed movie — same feeling.

Predict puzzle. You record a 4-second voice clip of "Selamat datang ke pasar malam!" and import it into the auntie's head sprite as a sound called line1. You add this stack to the head sprite, on top of the existing L04-25 mouth-flap loop:


when flag clicked
forever
  play sound (line1 v) until done
  wait (2) seconds
end

A new parallel script that plays the voice line, waits 2 seconds, repeats.

Click the flag. The voice plays, then silence, then voice again. Meanwhile the mouth from L04-25 keeps flapping at 0.2s the whole time. Is the puppet lip-synced?

Reveal the answer

No. The mouth flaps the same during the voice and during the 2 seconds of silence — completely independent. The mouth is "animated" but not "synced". Worse: even during the voice line, the flap rhythm has no relation to the actual speech. Real makcik speech has fast bits, slow bits, pauses for breath. A constant 0.2s flap matches none of it.

Today we won't try to match every syllable — that's a job for a dedicated tool. We'll do something much simpler that still reads as lip-sync: the mouth flaps only while the sound is playing, and is closed during silence. That alone is 90% of believable lip-sync, and it's a 5-block change.

New Concept — controller + receiver 15 min

To make the mouth obey the sound, we split the puppet's head into two responsibilities and connect them with broadcasts — the messaging system you first met in L04-20.

The two roles

The controller — knows when the voice is playing, and tells the mouth when to open and close.
The receiver — the mouth sprite itself, which just listens for "open" and "closed" messages and switches costume in response.

Why split them? Because the mouth shouldn't have to know about sounds. It just knows two things: "I got the open message → switch to open" and "I got the closed message → switch to closed". Tomorrow you can swap the controller for a totally different rhythm and the mouth doesn't change at all. That separation is the same idea you used with broadcast (load-level v) in cluster D: small, focused sprites listening for small, focused messages.

The controller's job

The controller runs two scripts in parallel on the same sprite (we'll put it on the body sprite to keep the head clean):


when flag clicked
play sound (line1 v) until done
broadcast (mouth-closed v)

Controller script A — plays the voice once, then sends one final "close your mouth" message when it ends.


when flag clicked
forever
  broadcast (mouth-open v)
  wait (0.15) seconds
  broadcast (mouth-closed v)
  wait (0.15) seconds
end

Controller script B — runs at the same time, blasting open/closed messages every 0.15 seconds.

Both scripts start at the flag click and run together. Script A plays the sound (which blocks that script for 4 seconds) and then sends a final mouth-closed message. Script B sends rapid open/closed broadcasts the whole time.

The receiver's job

The mouth sprite has just two short scripts — one per message:


when I receive (mouth-open v)
switch costume to (open v)

"Heard the open message → switch to the open costume."


when I receive (mouth-closed v)
switch costume to (closed v)

"Heard the closed message → switch to the closed costume."

Notice what the mouth doesn't do: it doesn't know about sounds, it doesn't count timers, it doesn't loop. It just reacts to messages. That's why this pattern scales — you can add a second character with their own controller and own messages without touching the mouth code.

The synchronisation moment

Here's what makes it look like lip-sync: script B keeps broadcasting forever, but the rapid open/closed only matters when script A is also running its sound. When script A finishes the play sound (line1 v) until done, it sends a final mouth-closed broadcast that the mouth happens to receive after script B's most recent message — and we add one more trick to stop script B from re-opening it.

Common mistakes

Using start sound instead of play sound (...) until done. The first kicks off the sound and immediately moves on; the second waits for the sound to finish. For lip-sync timing, you almost always want "until done".
Putting the broadcast loop on the mouth sprite. Then the mouth is both the controller and the receiver, and the broadcast becomes pointless — you might as well just switch costume directly. The whole reason to use broadcasts is the separation.
Forgetting the final broadcast (mouth-closed v) after the sound ends. If the sound finishes during an "open" moment of the loop, the mouth gets stuck open. One closing broadcast at the end is your safety net.
0.15 seconds is a guideline, not gospel. Some voices flap better at 0.10 (rapid speech); some at 0.20 (slow storytelling). Tune it by ear.

Worked Example — Auntie Siti welcomes you to the pasar malam 15 min

Start from your L04-25 cutout auntie. We're going to delete her random mouth-flap and replace it with sound-driven lip-sync.

Stage plan. The body sprite becomes the controller — it plays the sound and broadcasts mouth-open / mouth-closed. The head sprite is the receiver — it just switches costume on each message. Two sprites, four scripts, one believable talking puppet.

Step 1 — Record or import the voice line

On the body sprite, click the Sounds tab, then the microphone icon to record. Say: "Selamat datang ke pasar malam!" Keep it short (3–4 seconds). Trim silence at the start and end. Rename the sound to line1.

Step 2 — Remove the random flap from the head sprite

Open the head sprite's scripts. Delete the entire forever mouth-flap loop from L04-25 — but keep the snap-into-place when ⚑ clicked + go to x: (0) y: (60). We'll let broadcasts drive the costume now.

Step 3 — Add receiver scripts to the head sprite

Still on the head sprite, add two new scripts:


when I receive (mouth-open v)
switch costume to (open v)


when I receive (mouth-closed v)
switch costume to (closed v)

You'll need to create the two broadcast names in the dropdown — pick "New message" and type each name. Test: with nothing else built, the mouth stays on whatever costume it had. Receivers without a sender do nothing.

Step 4 — Build the sound controller on the body sprite

Click the body sprite (the one with the recorded sound). Add this stack:


when flag clicked
wait (1) seconds
play sound (line1 v) until done
broadcast (mouth-closed v)
stop [other scripts in sprite v]

Wait one second so the audience has time to look at the auntie, then play, then close the mouth, then halt the flap-loop next to it.

Step 5 — Build the flap controller on the body sprite

Add a second script on the same body sprite:


when flag clicked
wait (1) seconds
forever
  broadcast (mouth-open v)
  wait (0.15) seconds
  broadcast (mouth-closed v)
  wait (0.15) seconds
end

The same 1-second wait so this flap loop starts at the same moment the voice does.

Step 6 — How the two scripts cooperate

At the flag click, both scripts wait 1 second. Then script B starts flapping (open/closed broadcasts every 0.15s), and at the same instant script A starts playing the voice line. The mouth flaps during the voice. When the voice finishes (4 seconds later), script A sends one final mouth-closed and then stop [other scripts in sprite v] kills the forever flap loop. Mouth ends closed. Silence.

Step 7 — Click the flag and watch

Auntie waits a beat, then her mouth flaps for the duration of "Selamat datang ke pasar malam!", then she stops. The flap rhythm isn't perfectly matched to syllables, but the eye reads it as talking. Try it without the stop [other scripts in sprite v] — the mouth keeps flapping after the voice. Add it back — the mouth stops cleanly.

The full assembled stack — body sprite + head sprite


// body sprite — controller A
when flag clicked
wait (1) seconds
play sound (line1 v) until done
broadcast (mouth-closed v)
stop [other scripts in sprite v]

// body sprite — controller B
when flag clicked
wait (1) seconds
forever
  broadcast (mouth-open v)
  wait (0.15) seconds
  broadcast (mouth-closed v)
  wait (0.15) seconds
end

// head sprite — receivers
when I receive (mouth-open v)
switch costume to (open v)

when I receive (mouth-closed v)
switch costume to (closed v)

Two scripts on the body, two scripts on the head. Together they make a talking auntie.

What you just built: the same fake-lip-sync technique used in cheap web cartoons, YouTube shorts, indie visual novels. The mouth doesn't really know what the voice is saying. It just knows when the voice is on, and that's enough.

Try It Yourself — three sync drills 15 min

🟢 Easy

Goal: Tune the flap rhythm. Right now it's 0.15 seconds open + 0.15 seconds closed. Try 0.10/0.10 (faster) and 0.25/0.25 (slower). Which one reads as "talking" best for your recorded voice?


when flag clicked
wait (1) seconds
forever
  broadcast (mouth-open v)
  wait (0.1) seconds
  broadcast (mouth-closed v)
  wait (0.1) seconds
end

Think: Higher voices and excited speech usually flap faster. Lower voices and storytelling flap slower. There's no universal "right" number.

🟡 Medium

Goal: Record a second voice line called line2 (e.g. "Mari beli pisang goreng!"). After line1 finishes, wait 0.5 seconds, then play line2 — and make the mouth flap during both lines.


when flag clicked
wait (1) seconds
play sound (line1 v) until done
broadcast (mouth-closed v)
wait (0.5) seconds
play sound (line2 v) until done
broadcast (mouth-closed v)
stop [other scripts in sprite v]

The flap loop in script B can stay exactly the same — it doesn't care how many lines there are. Only the controller changes.

Think: The mouth still flaps during the 0.5-second gap because script B doesn't know there's a gap. For a more polished result, you'd need to broadcast a "stop flapping" message during the silence — that's tomorrow's territory.

🔴 Stretch

Goal: Give Auntie Siti three mouth costumes — closed, open-small, open-wide — and randomise which "open" she shows each time. The mouth becomes more lifelike because real mouths don't open the same amount on every syllable.


when I receive (mouth-open v)
switch costume to (pick random (2) to (3))

Costumes are numbered: closed = 1, open-small = 2, open-wide = 3. Picking between 2 and 3 gives a random "open" each broadcast.

Think: Even tiny randomness fights the "robot" feel. Three costumes feel many times more alive than two.

Mini-Challenge — the auntie who never stops talking 5 min

"Aisha's frozen-open auntie"

Aisha built lip-sync exactly like the example, but at the end of the voice line, her auntie's mouth gets stuck wide open for one frame, then closes. It's a tiny glitch but very noticeable. Here's her controller A:


when flag clicked
wait (1) seconds
play sound (line1 v) until done
stop [other scripts in sprite v]

Aisha's controller A. The sound plays, then she halts the flap loop.

What did Aisha forget, and why does it cause a frozen-open mouth?

Reveal one valid solution

Aisha is missing the explicit broadcast (mouth-closed v) before the stop. When the sound finishes, the flap loop in controller B was probably mid-cycle — and there's no guarantee which of "open" or "closed" was the most recent broadcast it sent. If the last broadcast happened to be mouth-open, the mouth sprite is showing the open costume the instant Aisha calls stop [other scripts in sprite v]. The flap loop dies, no more broadcasts are sent, and the mouth is frozen wide open.

The fix is one block — send a final mouth-closed broadcast before halting the loop, so the mouth's last instruction is always "close":


when flag clicked
wait (1) seconds
play sound (line1 v) until done
broadcast (mouth-closed v)
stop [other scripts in sprite v]

The order matters: broadcast first, then stop. If you stop first, the receivers don't get the message in time. This kind of "always end in a known state" thinking is the single most useful debugging habit in event-driven Scratch.

Recap 3 min

You learned the controller-and-receiver pattern for lip-sync. Two parallel scripts on a controller sprite — one playing the sound with play sound (line1 v) until done, one broadcasting mouth-open and mouth-closed every 0.15 seconds — drive a mouth sprite that simply switch costume to (open v)s and switch costume to (closed v)s on receive. The mouth doesn't know about sound. The controller doesn't know about costumes. A final closing broadcast plus stop [other scripts in sprite v] guarantees the mouth ends in a tidy closed state — no frozen-open glitches.

Lip-sync: Matching a character's mouth animation to spoken audio. Studio-grade lip-sync matches every syllable; cheap-and-cheerful lip-sync (today's technique) just flaps the mouth while audio is playing.
Controller sprite: A sprite whose job is to drive other sprites via broadcasts. The controller knows about timing and sound; the receivers know about visuals.
Parallel scripts: Two or more scripts that start from the same hat (e.g. two when ⚑ clicked blocks) and run at the same time. Scratch runs them concurrently, swapping between them many times per second.
Broadcast / Receive: Scratch's messaging system. broadcast (name v) sends a named message instantly to every sprite; when I receive (name v) hats fire in response. Used here to decouple the mouth from the sound logic.
play sound until done: The blocking version of the sound block. The script pauses on that block until the audio finishes — letting the script know exactly when to send the "close mouth" cleanup broadcast.

Homework 2 min

The Greeting Card Project. Build a short animated greeting using your L04-25 puppet (or a new one) — for Hari Raya, Deepavali, Chinese New Year, a birthday, or anything you like.

Record two voice lines on your controller sprite: a greeting ("Selamat Hari Raya!") and a follow-up sentence.
Set up the two controller scripts on the controller sprite — one for the sound chain, one for the flap loop. Both wait 1 second before starting.
Set up when I receive (mouth-open v) and when I receive (mouth-closed v) hats on your mouth sprite (head sprite from L04-25).
Make sure the controller ends with a final broadcast (mouth-closed v) + stop [other scripts in sprite v] so the mouth closes cleanly when the audio ends.
Save as HW-L4-26-Greeting-Card.sb3.

Bring back next class:

The .sb3 file.
Your answer to: "What flap interval (0.10s, 0.15s, 0.20s, 0.25s) worked best for your recorded voice, and why?"

Heads up for next class: SCR-L04-27 teaches camera pans — making the world appear to scroll past a stationary character by sliding a wide backdrop-sprite across the Stage. Your auntie will get a moving pasar malam behind her.