I'm working on a project in which LED lights synchronise with audio events. I'm not referring to reactive LED projects that read the ADC from audio amplitude or spectrum, but about having some additional digital data playing along with the audio to indicate several triggers of events.

Initially, the approach I'm taking is that of utilising a stereo file and having a channel for audio and another channel for data. The file is accessed by an SD card module and processed by a codec or mp3 player module. The channel with data has pulses of different sine wave frequencies to indicate different events and trigger specific LED lights.

I managed to do this using a spectrum analyser chip, but it's not the ideal as by using the ADC there are fluctuations, and changes in the volume control affect the signal. I'm hoping to find a digital signal solution, as when the SD card is read and the audio processed, the data from one channel of the file (wav or mp3) must be read real-time in the MCU.

Ideally the MCU would be an ESP32. I experimented with some libraries that can read wav files from an SD card (such as esp32_I2S_player) but I couldn't manage to receive the data for creating the necessary 'if conditions'. Another try was to see if I could do this with a VS1053 codec, but libraries for this chip also don't provide a function to read data off the SD card of a specific channel, and I couldn't find much about this in the datasheet.

So my questions are:

  1. Is there an efficient way for synchronising audio events real-time with LEDs using an MCU? I know how to do this through general Operating System sofware, but now the application requires a bare machine.
  2. If my approach (stereo file = data + audio) makes sense, how can I implement this? Is there a codec or MCU that facilitates this approach? Is there any code example that can help me?

I'm feeling stagnated in this issue so I really appreciate your answers. Thanks.

  • 2
    \$\begingroup\$ I find encoding that kind of thing as audio in a digital file kind of - wrong. MP3 files can contain lyrics tags with time stamps. Encode your led commands as text in lyric tags. \$\endgroup\$ – JRE Apr 23 '19 at 16:31
  • \$\begingroup\$ @JRE: That's actually a good answer, and you should write it up as such. \$\endgroup\$ – Dave Tweed Apr 23 '19 at 16:47
  • \$\begingroup\$ I was checking to see if it is even possible with typical ESP32 or Arduino libraries. It looks like it is. I'll write it up and add some details in a little bit. \$\endgroup\$ – JRE Apr 23 '19 at 16:50
  • \$\begingroup\$ Depends on specs. Placing a low baud rate UART bit stream on one audio channel is pretty simple. \$\endgroup\$ – sstobbe Apr 23 '19 at 18:00
  • \$\begingroup\$ One option would be simultaneous playback of a MIDI stream along with your audio stream where the lighting control data is encoded in the MIDI track. This is actually pretty common with DJ/nightclub lighting control and you can find lots of reference info once you know what you're looking for \$\endgroup\$ – nvuono Apr 23 '19 at 18:51

Using an auxiliary DSP for decompression may make this difficult unless you limit yourself to constant bitrate files - if you need high time accuracy of the light events you may need to account for the processing delay from when you put the data in until the sound comes out, which would be different at different bitrates.

Possibly you could work around a distinct decoder by doing independent timing, starting an MCU time counter at the start of the audio output, and triggering light events at appropriate timestamps. In that case you may want to encode your light data in its own file linked by a naming pattern, or embedded it in interwoven data that shows up a bit ahead of compressed audio it corresponds to and gets held in an MCU buffer until the indicated timestamp.

One potential drastic simplification is to store linear PCM .wav files instead of compressed MP3. Given that an audio CD is only about 3/4 of a gigabyte, even a cheap SD card potentially holds a few hours of uncompressed audio. If you have no compression it's pretty simple for your MCU to just clock the data out a DAC, though preferably use a hardware timer driven DAC (and potentially DMA) or at least an interrupt, not a software delay loop.

  • 1
    \$\begingroup\$ I marked this as the best solution as it worked perfectly for me. I encoded the timestamps in a TXT file and added to an SD card along with the audio (WAV) file. This TXT file was decoded by the MCU, which then initialised a timer and read the WAV file from the SD card. Finally, the code compared the timestamps with the time counter for triggering what was needed. \$\endgroup\$ – Chu Apr 27 '19 at 10:18

I am not fond of encoding command data as an analog signal in a digital file.

I think I would try something like encoding the lighting commands as text blocks in the lyrics block of the ID3 information inside the mp3 file.

The lyrics block is before the sound data, so you should be able to decode it quickly before you start playback.

Typical libraries for mp3 playback don't seem to read all ID3 tags. Some libraries read some tags. You could either extract the lyrics tags in your own read function before playback, or extend one of the existing libraries.

Inside the lyrics tags, you have timestamps and text.

You can encode the timestamps as the ID3 standards define them, or encode your own more precise timestamps in your own format (the ID3 timestamps are only given in seconds.)

The text content is the interesting part. Define your own text encoding for your lights. Say "DMX1:FFFF00000000" for full bright red light on address 1 (that's just encoding DMX data as hexadecimal, with the address included in the header.)

Or something simpler if you just need to turn on and off a handful of LEDs connected directly to the microcontroller.

Or implement your format such that you can use it for DMX, but have an interpreter in your controller for local LEDs.

The content is really up to you. In any case, it is much more flexible than embedding sounds in one channel of your audio file.

There are lyrics editors that you can use to put your commands into the mp3 files. Just type commands in your private format in as lyrics.

  • \$\begingroup\$ This approach seems great, but I can't find any information about codecs (in ICs, not software) or libraries that can decode ID3 lyrics (LYR) so I can receive the timestamp real-time in the MCU...if there is any please let me know. :) \$\endgroup\$ – Chu Apr 23 '19 at 19:11
  • \$\begingroup\$ It's not in the codec. You have read the file and pick the ID3 tags out. Like this ESP project does. \$\endgroup\$ – JRE Apr 23 '19 at 19:25
  • \$\begingroup\$ I see, shame there isn't much information about this yet. But this helps, I will try to dig into that and maybe I can come up with a code. Thanks. \$\endgroup\$ – Chu Apr 23 '19 at 19:27
  • \$\begingroup\$ Your MCU has to read the file and pass the data to the hardware decoder. Just pluck the ID3 lyrics block out before you start passing data blocks to the decoder. \$\endgroup\$ – JRE Apr 23 '19 at 19:34

I once developed an application in which arbitrary events could be triggered by audio watermarks embedded in the sound. You could play the sound through tinny unamplified computer speakers at one end of the conference room table, and my demo box sitting at the other end would turn on LEDs at exactly the correct moments.

It was efficient in the sense that the decoder ran on a tiny 8-bit (6502-based) microcontroller, supported by a simple analog signal processing chain (mic preamp, filter, etc.)

  • \$\begingroup\$ this sounds as a good approach! can you provide some more information about the system you used for encoding and decoding the watermarks? \$\endgroup\$ – Chu Apr 23 '19 at 17:12
  • \$\begingroup\$ Unfortunately, no. The algorithm is proprietary to Verance -- I was working for them at the time. You would have to either license it from them or develop the same concept using a different algorithm. \$\endgroup\$ – Dave Tweed Apr 23 '19 at 17:20
  • \$\begingroup\$ Oddly enough, I built the same type of thing for someone else using near ultrasonic coding. But I wouldn't do it this way for the OP's need - they control the playback from the data source so there's no need to go through an audio chain. If they really want to use a distinct decoder IC, just detecting the start of the output would allow independent timestamps to work, especially if the clock source can be shared. \$\endgroup\$ – Chris Stratton Apr 23 '19 at 17:46
  • \$\begingroup\$ @ChrisStratton: Yes, JRE's approach is better in this case. I just wanted to get watermarking on the table for applications that needed the cues embedded in the audio itself. \$\endgroup\$ – Dave Tweed Apr 23 '19 at 18:03

The quick solution in my world would be to playback from a computer. There are ready computer programs that can synchronize sound on many channels with Midi (normally used for synths) or DMX (very similar to Midi but used for lights).

  • \$\begingroup\$ You can playback the MIDI simultaneously on the esp32 and as a bonus if you map MIDI keyboard inputs to specific colors/lights you can create and edit some pretty robust lighting cue tracks in existing open source MIDI sequencing software. \$\endgroup\$ – nvuono Apr 23 '19 at 18:58
  • 1
    \$\begingroup\$ Yes, through computer it's pretty easy and I've done many times with Max MSP, but the problem is that I need to use this with an MCU. No OS. \$\endgroup\$ – Chu Apr 23 '19 at 19:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.