Developer World Spresense
日本語 中文
Table of Contents

1. Audio Tutorial

The audio library contains a variety of sample sketches, including audio playback and recording.

By taking advantage of the multi-core characteristics of Spresense, codec processing such as audio encoding/decoding is executed by a dedicated core which we call DSP. The audio applications can run without doing the heavy codec processing on the main core.

Audio Tutorial explains how to install DSP files and how to execute various sample sketches.

1.1. Install DSP files

When running the sample, it is necessary to install the DSP file in advance for decoder if use audio playback or encoder if you use audio recorder. This section describes how to install the DSP file.

1.1.1. DSP file type

Table 1. DSP type
DSP file name DSP installer Description

MP3DEC

mp3_dec_installer

Used for MP3 audio playback

MP3ENC

mp3_enc_installer

Used for MP3 audio recording

WAVDEC

wav_dec_installer

Used for WAV (PCM) audio playback

SRC

src_installer

Use for WAV(PCM) audio recording

1.1.2. How to install DSP

There are two ways to install DSP.

  • How to download DSP file and copy directly to microSD card

  • Using the DSP installer sketch

You can use either method when installing to a microSD card. When using the DSP installer, it is possible to install it on the SPI-Flash on the main board in addition to the microSD card on the expansion board.

The DSP needs to be install once, you only need to re install if the DSP file is updated.
1.1.2.1. How to install by copying DSP file to microSD card

Download the latest DSP file from here.

  • MP3DEC

  • MP3ENC

  • WAVDEC

  • SRC

    1. Create a BIN directory on the FAT32 formatted microSD card and copy the downloaded file.

      arduino dsp copy cn
      Figure 1. Copy DSP file

Please insert this microSD card into the expansion board and run it when you want to execute the audio samples.

1.1.2.2. How to install using DSP installer

A DSP installer is provided for each DSP file you install.

  1. From the Arduino IDE, start from File → Examples → Examples for Spresense / Audio → dsp_installer.

    arduino dsp installer cn
    Figure 2. Starting the DSP installer
  2. Select the COM port of Spresense in Tools → Port and execute writing.

  3. When the compilation & writing (Upload) is completed, start the serial monitor.

  4. If you select a baud rate of 115200 bps, a message will appear as shown in the figure below.

  5. Select the number that you want to install and press the send button.
    When installing to a microSD card, insert a FAT32 formatted microSD card into the microSD card slot on the expansion board.

    arduino dsp installer monitor1 en
    Figure 3. Select DSP installation location
  6. If the installation is successful, the message shown below is displayed.

    arduino dsp installer monitor2 en
    Figure 4. DSP installation execution
In the audio playback/recording application program, the location where the DSP file is installed is specified by the argument of the initPlayer() function or the initRecorder() function. By default, a microSD card (/mnt/sd0/BIN) is specified. When changing to SPI-Flash, rewrite to /mnt/spif/BIN.

1.2. High-level interface layer

Spresense Audio library has a layer structure as described in Audio library development guide.
In the high-level interface layer, typical functions with a high degree of abstraction are realized while maintaining the consistency of the entire system.

Below is a sample using this high-level interface layer.

1.2.1. Make a beep

1.2.1.1. Overview

This sample uses the capabilities of the Audio hardware to emit a beep.

Output the sound of the specified frequency with the setBeep() function of the Audio library. This function is very similar to Arduino’s tone() function, but it does not require a piezoelectric buzzer. Sound is output from the headphone terminal of the Spresense extension board.

In this sample, it is not necessary to install the DSP file.

1.2.1.2. Operating environment
  • Spresense Main & Expansion Board

  • Playback headphones or speakers

Connect headphones or active speakers to the headphone jack on the expansion board.

1.2.1.3. Operating procedure
  1. Open the sample sketch by selecting File → Examples → Example for Spresense / Audio → application → beep from Arduino IDE.

arduino audio examples beep

Select the COM port of Spresense with Tools → Port and write to the microcomputer board.

arduino audio examples beep2

The scale of Do-re-mi-fa-so-la-ti-do is output from the headphones.

1.2.1.4. Program description
  • Set the player mode with the setPlayerMode() function.

    theAudio->setPlayerMode(device,          (1)
                            player0bufsize,  (2)
                            player1bufsize); (3)
    1 device: Select the output device, whether to output to speaker headphones (AS_SETPLAYER_OUTPUTDEVICE_SPHP) or I2S (AS_SETPLAYER_OUTPUTDEVICE_I2SOUTPUT).
    This sample uses AS_SETPLAYER_OUTPUTDEVICE_SPHP.
    When using I2S, refer to How to output to I2S.
    2 player0bufsize: Specifies the buffer size of player 0.
    3 player1bufsize: Specifies the buffer size of player 1.
    For details, refer to Player buffer size.
    The buffer is not used if only the beep sound is output, so 0 is specified in this sample.
  • To output the beep sound, use the setBeep() function.

    theAudio->setBeep(enable,       (1)
                       volume,      (2)
                       frequency);  (3)
    1 If you set enable to 1, the beep sound will be output, and if enable is set to 0, the output will stop.
    2 volume: You can change the volume (-90 ~ 0 [dB]).
    3 frequency: Outputs the sound of the specified frequency.

1.2.2. Play MP3 music

1.2.2.1. Overview

Play MP3 music files on the microSD card.

1.2.2.2. Operating environment
  • Spresense Main & Expansion Board

  • microSD card

  • Playback headphones or speakers

This sample uses a microSD card and headphones. Connect headphones or active speakers to the headphone jack on the expansion board.

1.2.2.3. Operating procedure
  1. Install the DSP file MP3DEC for MP3 decoding to the microSD card (see Install DSP files).

  2. Put on the MP3 file you want to play with the file name "Sound.mp3" in the root directory of the microSD card.

  3. Insert the microSD card into the microSD card slot on the expansion board.

  4. Open the sample sketch by selecting File → Examples → Examples for Spresense / Audio → application → player from Arduino IDE.

    arduino audio examples player

Select the COM port of Spresense with Tools → Port and write to the microcomputer board.

When the sketch is uploaded and executed, the MP3 file placed on the microSD card will be played via headphone.
Play the file to the end and stop.

1.2.2.4. Program description
  • Set the clock mode with the setRenderingClockMode() function.

    theAudio->setRenderingClockMode(mode); (1)
    1 mode: Select normal (AS_CLKMODE_NORMAL) or high resolution (AS_CLKMODE_HIRES) or clock mode.
    For MP3 playback, specify AS_CLKMODE_NORMAL.
  • Set the player mode with the setPlayerMode() function.

    theAudio->setPlayerMode(device,    (1)
                             sp_drv);  (2)
    1 device: Select the output device, whether to output to speaker headphones (AS_SETPLAYER_OUTPUTDEVICE_SPHP) or I2S (AS_SETPLAYER_OUTPUTDEVICE_I2SOUTPUT).
    This sample uses AS_SETPLAYER_OUTPUTDEVICE_SPHP.
    When using I2S, refer to How to output to I2S.
    2 sp_drv: Sets the output drive.
    In this sample, specify AS_SP_DRV_MODE_LINEOUT.
    For details, refer to When using the speaker terminal of the expansion board.
  • Initialize the player with the initPlayer() function.

    theAudio->initPlayer(id,         (1)
                         codec,      (2)
                         codec_path, (3)
                         fs,         (4)
                         channel);   (5)
    1 id: Select Player ID.
    This sample uses AudioClass::Player0.
    2 codec: Specify MP3 (AS_CODECTYPE_MP3) as the codec type.
    3 codec_path: Specify the location where the DSP file is installed.
    Specify "/mnt/sd0/BIN" for microSD card and "/mnt/spif/BIN" for SPI-Flash.
    4 fs: Specify the sampling rate.
    MP3 supports 32kHz (AS_SAMPLINGRATE_32000), 44.1kHz (AS_SAMPLINGRATE_44100), 48kHz (AS_SAMPLINGRATE_48000). If you specify AS_SAMPLINGRATE_AUTO, the sampling rate is automatically determined.
    5 channel: Specify monaural (AS_CHANNEL_MONO) or stereo (AS_CHANNEL_STEREO).
  • Write the contents of the MP3 file to the decoder with the writeFrames() function.

    theAudio->writeFrames(id,       (1)
                           myFile); (2)
    1 id: Select Player ID.
    This sample uses AudioClass::Player0.
    2 You can change the playback file by specifying myFile in the argument.
    This sample uses a file called "Sound.mp3".
    Execute this function periodically to supply the stream to the decoder so that the decoding will not be interrupted. When writing the file is completed, AUDIOLIB_ECODE_FILEEND is returned as the return value of the function.
  • Start the player with the startPlayer() function.

    theAudio->startPlayer(id); (1)
    1 id: Select Player ID.
    This sample uses AudioClass::Player0.
    Call it after writing to the decoder with the writeFrames() function.
  • Stop the player with the stopPlayer() function.

    theAudio->stopPlayer(id,      (1)
                          mode);  (2)
    1 id: Select Player ID.
    This sample uses AudioClass::Player0.
    2 You can choose to stop immediately in mode (AS_STOPPLAYER_NORMAL) or wait until the stream supplied to the decoder has finished playing and then stop (AS_STOPPLAYER_ESEND). If called without the mode argument, it works with AS_STOPPLAYER_NORMAL.
  • Adjust the volume with the setVolume(volume) function.

    theAudio->setVolume(volume); (1)
    1 volume: You can change the volume. (-1020 ~ 120)
    1/10 of the specified value becomes the actual volume [dB]. For example, the volume when 15 is specified, is 1.5 [dB].
1.2.2.5. Player buffer size

A buffer size of 160kByte to use for reading from the microSD card is secured by default for each player. This size is such that high resolution WAV audio can be read, and when playing a 48 kHz MP3 file or etc, you can reduce the buffer size used in the argument of the setPlayerMode function. As a guideline, when playing a 192kHz high resolution WAV file, it needs the 160kByte which is default, and it is recommended to set it to about 24kByte when playing an MP3 file of about 128kbps.

Regarding the size, it is a suggestion only. Adjust according to the performance of the microSD card and the processing amount of the application. If you make the buffer too small, you will get errors when reading from the microSD card. If read data underflow occurs, increase the buffer size.
1.2.2.6. How to output to I2S

If you want to switch the output device to I2S, switch the output device with setPlayerMode to AS_SETPLAYER_OUTPUTDEVICE_I2SOUTPUT. The I2S operation mode supports only Slave mode, I2S format mode, 48000Hz, and Stereo.

1.2.2.7. When using the speaker terminal of the expansion board

In the case of line-out using the headphone terminal of the expansion board as the audio output destination, as the drive capability setting of Audio signal, Specify AS_SP_DRV_MODE_LINEOUT as the argument sp_drv of setPlayerMode (default).

When outputting from the speaker terminal of the expansion board, specify AS_SP_DRV_MODE_4DRIVER in the argument sp_drv.

When using the speaker terminal, the board needs to be modified. For details, refer to the following hardware guide.

MP3 files have ID3v2 TAGs (especially large metadata like image data) will cause parse errors by the decoder. Please delete the tag information with some tool.

1.2.3. Play WAV music

1.2.3.1. Overview

Play WAV music files on the microSD card.

1.2.3.2. Operating environment
  • Spresense Main & Expansion Board

  • microSD card

  • Playback headphones or speakers

This sample uses a microSD card and headphones. Connect headphones or active speakers to the headphone jack on the expansion board.

1.2.3.3. Operating procedure
  1. Install the DSP file WAVDEC for WAV decoding to the microSD card (see Install DSP files).

  2. Put on the WAV file you want to play with the file name "Sound.wav" in the root directory of the microSD card.

  3. Insert the microSD card into the microSD card slot on the expansion board.

  4. Open the sample sketch by selecting File → Examples → Examples for Spresense / Audio → application → player_wav from Arduino IDE.

    arduino audio examples player wav
  5. Select the COM port of Spresense with Tools → Port and write to the microcomputer board.

  6. When the sketch is uploaded and executed, the WAV file placed on the microSD card will be played via headphone.
    Play the file to the end and stop.

1.2.3.4. Program description
  • Set the clock mode with the setRenderingClockMode() function.

    theAudio->setRenderingClockMode(mode); (1)
    1 mode: Select normal (AS_CLKMODE_NORMAL) or high resolution (AS_CLKMODE_HIRES) or clock mode.
    This sample uses AS_CLKMODE_NORMAL.
    This function only accepts before calling setPlayerMode() or in ReadyMode.
    To switch between normal and high resolution dynamically, call setReadyMode() and transition to the Ready state, then switch AS_CLKMODE_NORMAL or AS_CLKMODE_HIRES.
  • Set the player mode with the setPlayerMode() function.

    theAudio->setPlayerMode(device,    (1)
                             sp_drv);  (2)
    1 device: Select the output device, whether to output to speaker headphones (AS_SETPLAYER_OUTPUTDEVICE_SPHP) or I2S (AS_SETPLAYER_OUTPUTDEVICE_I2SOUTPUT).
    This sample uses AS_SETPLAYER_OUTPUTDEVICE_SPHP.
    When using I2S, refer to How to output to I2S.
    2 sp_drv: Sets the output drive.
    In this sample, specify AS_SP_DRV_MODE_LINEOUT.
    For details, refer to When using the speaker terminal of the expansion board.
  • Initialize the player with the initPlayer() function.

    theAudio->initPlayer(id,         (1)
                         codec,      (2)
                         codec_path, (3)
                         fs,         (4)
                         channel);   (5)
    1 id: Select Player ID.
    This sample uses AudioClass::Player0.
    2 codec: Specify WAV (AS_CODECTYPE_WAV) as the codec type.
    3 codec_path: Specify the location where the DSP file is installed.
    Specify "/mnt/sd0/BIN" for microSD card and "/mnt/spif/BIN" for SPI-Flash.
    4 fs: Specify the sampling rate.
    5 channel: Specify monaural (AS_CHANNEL_MONO) or stereo (AS_CHANNEL_STEREO).
  • Write the contents of the WAV file to the decoder with the writeFrames() function.

    theAudio->writeFrames(id,      (1)
                          myFile); (2)
    1 id: Select Player ID.
    This sample uses AudioClass::Player0.
    2 You can change the playback file by specifying myFile in the argument.
    This sample uses a file called "Sound.wav".
    Execute this function periodically to supply the stream to the decoder so that the decoding will not be interrupted. When writing the file is completed, AUDIOLIB_ECODE_FILEEND is returned as the return value of the function.
  • Start the player with the startPlayer() function.

    theAudio->startPlayer(id); (1)
    1 id: Select Player ID.
    This sample uses AudioClass::Player0.
    Call it after writing to the decoder with the writeFrames() function.
  • Stop the player with the stopPlayer() function.

    theAudio->stopPlayer(id,      (1)
                          mode);  (2)
    1 id: Select Player ID.
    This sample uses AudioClass::Player0.
    2 You can choose to stop immediately in mode (AS_STOPPLAYER_NORMAL) or wait until the stream supplied to the decoder has finished playing and then stop (AS_STOPPLAYER_ESEND). If called without the mode argument, it works with AS_STOPPLAYER_NORMAL.
  • Adjust the volume with the setVolume(volume) function.

    theAudio->setVolume(volume); (1)
    1 volume: You can change the volume. (-1020 ~ 120)
    1/10 of the specified value becomes the actual volume [dB]. For example, the volume when 15 is specified, is 1.5 [dB].

1.2.4. Play high resolution audio

1.2.4.1. Overview

Play WAV music files of high-resolution sound source on microSD card.

1.2.4.2. Operating environment
  • Spresense Main & Expansion Board

  • microSD card

  • Playback headphones or speakers

This sample uses a microSD card and headphones. Connect headphones or active speakers to the headphone jack on the expansion board.

1.2.4.3. Operating procedure

Install the DSP file WAVDEC for WAV decoding to the microSD card (see Install DSP files).

Put on the WAV file you want to play and rename the file name to "HiResSound.wav" in the root directory of the microSD card.

  1. Insert the microSD card into the microSD card slot on the expansion board.

  2. Open the sample sketch by selecting File → Examples → Examples for Spresense / Audio → application → player_hires from Arduino IDE.

    arduino audio examples player hires
  3. Select the COM port of Spresense with Tools → Port and write to the microcomputer board.

  4. When the sketch is uploaded and executed, the WAV file of high-resolution sound placed on the microSD card will be played via headphone.
    Play the file to the end and stop.

1.2.4.4. Program description
  • Set the clock mode with the setRenderingClockMode() function.

    theAudio->setRenderingClockMode(mode); (1)
    1 mode: Select normal (AS_CLKMODE_NORMAL) or high resolution (AS_CLKMODE_HIRES) or clock mode.
    This sample uses AS_CLKMODE_NORMAL.
    setPlayerMode() can be called only in ReadyMode.
    To switch between normal and high resolution dynamically, call setReadyMode() and transition to the Ready state, then switch AS_CLKMODE_NORMAL or AS_CLKMODE_HIRES.
  • Set the player mode with the setPlayerMode() function.

    theAudio->setPlayerMode(device,    (1)
                             sp_drv);  (2)
    1 device: Select the output device, whether to output to speaker headphones (AS_SETPLAYER_OUTPUTDEVICE_SPHP) or I2S (AS_SETPLAYER_OUTPUTDEVICE_I2SOUTPUT).
    This sample uses AS_SETPLAYER_OUTPUTDEVICE_SPHP.
    When using I2S, refer to How to output to I2S.
    2 sp_drv: Sets the output drive.
    In this sample, specify AS_SP_DRV_MODE_LINEOUT.
    For details, refer to When using the speaker terminal of the expansion board.
  • Initialize the player with the initPlayer() function.

    theAudio->initPlayer(id,         (1)
                         codec,      (2)
                         codec_path, (3)
                         fs,         (4)
                         bitlen,     (5)
                         channel);   (6)
    1 id: Select Player ID.
    This sample uses AudioClass::Player0.
    2 codec: Specify WAV (AS_CODECTYPE_WAV) as the codec type.
    3 codec_path: Specify the location where the DSP file is installed.
    Specify "/mnt/sd0/BIN" for microSD card and "/mnt/spif/BIN" for SPI-Flash.
    4 fs: Specify the sampling rate.
    In this sample, 96kHz (AS_SAMPLINGRATE_96000) is specified.
    5 bitlen: Specify the bit length.
    In this sample, 24bit (AS_BITLENGTH_24) is specified.
    6 channel: Specify monaural (AS_CHANNEL_MONO) or stereo (AS_CHANNEL_STEREO).
  • Write the contents of the WAV file to the decoder with the writeFrames() function.

    theAudio->writeFrames(id,      (1)
                          myFile); (2)
    1 id: Select Player ID.
    This sample uses AudioClass::Player0.
    2 You can change the playback file by specifying myFile in the argument.
    This sample uses a file called "HiResSound.wav".
    Execute this function periodically to supply the stream to the decoder so that the decoding will not be interrupted. When writing the file is completed, AUDIOLIB_ECODE_FILEEND is returned as the return value of the function.
  • Start the player with the startPlayer() function.

    theAudio->startPlayer(id); (1)
    1 id: Select Player ID.
    This sample uses AudioClass::Player0.
    Call it after writing to the decoder with the writeFrames() function.
  • Stop the player with the stopPlayer() function.

    theAudio->stopPlayer(id,      (1)
                          mode);  (2)
    1 id: Select Player ID.
    This sample uses AudioClass::Player0.
    2 You can choose to stop immediately in mode (AS_STOPPLAYER_NORMAL) or wait until the stream supplied to the decoder has finished playing and then stop (AS_STOPPLAYER_ESEND). If called without the mode argument, it works with AS_STOPPLAYER_NORMAL.
  • Adjust the volume with the setVolume(volume) function.

    theAudio->setVolume(volume); (1)
    1 volume: You can change the volume. (-1020 ~ 120)
    1/10 of the specified value becomes the actual volume [dB]. For example, the volume when 15 is specified, is 1.5 [dB].

1.2.5. Play Playlist of music

1.2.5.1. Overview

This sample is a sample sketch for playing music using a playlist.

Plays multiple music files in the microSD card in order. You can perform operations such as play, stop, send to next song, return, repeat play and random play.

1.2.5.2. Operating environment
  • Spresense Main & Expansion Board

  • microSD card

  • Playback headphones or speakers

This sample uses a microSD card and headphones. Connect headphones or active speakers to the headphone jack on the expansion board.

1.2.5.3. Program description

In order to use the playlist function, it is necessary to create a playlist file on a PC beforehand and copy it to the microSD card. The directory structure of the microSD card used in this sample sketch is shown below.

microSD card root directory
|-- BIN/
|   |-- MP3DEC
|   `-- WAVDEC
|-- AUDIO/
|   |-- Sound1.mp3
|   |-- Sound2.mp3
|   |--   :
|   |-- Sound1.wav
|   |-- Sound2.wav
|   |--   :
`-- PLAYLIST/
    `-- TRACK_DB.CSV
  • Install the MP3DEC and WAVDEC DSP binary files in the BIN/ directory (see Install DSP files).

  • Copy the music content files to play in the playlist to the AUDIO/ directory.

  • Copy the playlist file (TRACK_DB.CSV) to the PLAYLIST/ directory.

How to create a playlist file

The format of the playlist file TRACK_DB.CSV is shown below. It is the text file in CSV (comma-separated values) format and in which the information of each file is written in line units.

[filename],[artist],[album],[channel number],[bit length],[sampling rate],[file format]
[filename],[artist],[album],[channel number],[bit length],[sampling rate],[file format]
[filename],[artist],[album],[channel number],[bit length],[sampling rate],[file format]
 :

Please describe the file name, artist name, album name, and codec information (number of channels, bit length, sampling rate) of the music contents. An example of TRACK_DB.CSV is shown below.

Sound1.mp3,Artist1,Album1,2,16,44100,mp3
Sound2.mp3,Artist1,Album1,2,16,44100,mp3
Sound1.wav,Artist2,Album2,2,16,48000,wav
Sound2.wav,Artist2,Album2,2,24,192000,wav

We also provide a simple script tool, mkplaylist.py, for creating playlist files. Please right-click here and download the file. This script uses ffmpeg-python, so please use it with ffmpeg and ffmpeg-python installed.

The usage of mkplaylist.py is shown below.

python mkplaylist.py
usage: python mkplaylist.py dirname [dirname2] [dirname3] ...
usage: python mkplaylist.py -f filename

Generate an audio playlist file named as TRACK_DB.CSV

Please execute by specifying the directory path containing the music file as an argument. When the execution is completed normally, the TRACK_DB.CSV file will be output to the current directory. If the TRACK_DB.CSV file already exists, it will be added to the end of the file.

python mkplaylist.py MyMusic
  • The extension of the music content file name should be .mp3 or .wav.

  • File names containing double-byte code such as Japanese are not supported. Please change the file name to an ASCII code string.

  • The playlist file is written in CSV format, so please do not use a string containing "," comma in filename, artist and album name.

  1. Open the sample sketch by selecting File → Examples → Examples for Spresense / Audio → application → play_playlist from Arduino IDE.

    arduino audio examples player playlist
  2. Select the COM port of Spresense with Tools → Port and write to the microcomputer board.

  3. When you start the serial monitor, the following menu is displayed. Various operations are performed according to the key input from the serial.

=== MENU (input key ?) ==============
p: play  s: stop  +/-: volume up/down
l: list  n: next  b: back
r: repeat on/off  R: random on/off
a: auto play      m,h,?: menu
=====================================
The EEPROM library is used to save setting information such as volume value, random / repeat / auto play, etc. in non-volatile memory. It is a sample application that operates based on the previously preset information at the next startup. Please refer to it when creating your own music player.

1.2.6. Play dual MP3 music

1.2.6.1. Overview

Play two MP3 music files on microSD card at the same time

1.2.6.2. Operating environment
  • Spresense Main & Expansion Board

  • microSD card

  • Playback headphones or speakers

This sample uses a microSD card and headphones. Connect headphones or active speakers to the headphone jack on the expansion board.

1.2.6.3. Operating procedure
  1. Install the DSP file MP3DEC for MP3 decoding to the microSD card (see Install DSP files).

  2. Put the two MP3 files you want to play in the root directory of the microSD card with the file names "Sound0.mp3" and "Sound1.mp3".

  3. Insert the microSD card into the microSD card slot on the expansion board.

  4. Open the sample sketch by selecting File → Examples → Examples for Spresense / Audio → application → dual_players from Arduino IDE.

    arduino audio examples dual players
  5. Select the COM port of Spresense with Tools → Port and write to the microcomputer board.

  6. Once the sketch is uploaded and executed, the two MP3 files placed on the microSD card will be mixed and played from the line-out at the same time. Play each file repeatedly.

1.2.6.4. Program description
  • Set the clock mode with the setRenderingClockMode() function.

    theAudio->setRenderingClockMode(mode); (1)
    1 mode: Select normal (AS_CLKMODE_NORMAL) or high resolution (AS_CLKMODE_HIRES) or clock mode.
    For MP3 playback, specify AS_CLKMODE_NORMAL.
  • Set the player mode with the setPlayerMode() function.

    theAudio->setPlayerMode(device,    (1)
                             sp_drv);  (2)
    1 device: Select the output device, whether to output to speaker headphones (AS_SETPLAYER_OUTPUTDEVICE_SPHP) or I2S (AS_SETPLAYER_OUTPUTDEVICE_I2SOUTPUT).
    This sample uses AS_SETPLAYER_OUTPUTDEVICE_SPHP.
    When using I2S, refer to How to output to I2S.
    2 sp_drv: Sets the output drive.
    In this sample, specify AS_SP_DRV_MODE_LINEOUT.
    For details, refer to When using the speaker terminal of the expansion board.
  • Initialize the player with the initPlayer() function for players 0 and 1 respectively.

    theAudio->initPlayer(id,         (1)
                         codec,      (2)
                         codec_path, (3)
                         fs,         (4)
                         channel);   (5)
    1 id: Select Player ID.
    One specifies AudioClass::Player0 and the other specifies AudioClass::Player1.
    2 codec: Specify MP3 (AS_CODECTYPE_MP3) as the codec type.
    3 codec_path: Specify the location where the DSP file is installed.
    Specify "/mnt/sd0/BIN" for microSD card and "/mnt/spif/BIN" for SPI-Flash.
    4 fs: Specify the sampling rate.
    MP3 supports 32kHz (AS_SAMPLINGRATE_32000), 44.1kHz (AS_SAMPLINGRATE_44100), 48kHz (AS_SAMPLINGRATE_48000). If you specify AS_SAMPLINGRATE_AUTO, the sampling rate is automatically determined.
    5 channel: Specify monaural (AS_CHANNEL_MONO) or stereo (AS_CHANNEL_STEREO).
  • Write the MP3 file contents to the decoder with the writeFrames() function for players 0 and 1, respectively.

    theAudio->writeFrames(id,       (1)
                           myFile); (2)
    1 id: Select Player ID.
    One specifies AudioClass::Player0 and the other specifies AudioClass::Player1.
    2 You can change the playback file by specifying myFile in the argument.
    This sample uses files called "Sound0.mp3" and "Sound1.mp3".
    Execute this function periodically to supply the stream to the decoder so that the decoding will not be interrupted. When writing the file is completed, AUDIOLIB_ECODE_FILEEND is returned as the return value of the function.
  • Start the player with the startPlayer() function for each of players 0 and 1.

    theAudio->startPlayer(id); (1)
    1 id: Select Player ID.
    One specifies AudioClass::Player0 and the other specifies AudioClass::Player1.
    Call it after writing to the decoder with the writeFrames() function.
  • Stop the player with the stopPlayer() function for each of players 0 and 1.

    theAudio->stopPlayer(id,      (1)
                          mode);  (2)
    1 id: Select Player ID.
    One specifies AudioClass::Player0 and the other specifies AudioClass::Player1.
    2 You can choose to stop immediately in mode (AS_STOPPLAYER_NORMAL) or wait until the stream supplied to the decoder has finished playing and then stop (AS_STOPPLAYER_ESEND). If called without the mode argument, it works with AS_STOPPLAYER_NORMAL.
  • Adjust the volume with the setVolume(master, player0, player1) function.

    theAudio->setVolume(master,   (1)
                        player0,  (2)
                        player1); (3)
    1 master: You can change the volume for both players 0 and 1. (-1020 ~ 120)
    2 You can change the volume for player 0 with player0. (-1020 ~ 120)
    3 You can change the volume for player 1 with player1. (-1020 ~ 120)
    1/10 of the specified value becomes the actual volume [dB]. For example, the volume when 15 is specified, is 1.5 [dB]. You can adjust the overall volume with master and the individual volume with player0 and player1.

1.2.7. Listen to the sound input from the microphone

1.2.7.1. Overview

You can hear the sound from the microphone from the headphone jack by using the audio path setting function of Audio HW.

For example, to output digital audio input from I2S to analog, or such as outputting the input from the microphone to I2S it is possible. This function supports an output mechanism that saves power and has very low latency.

1.2.7.2. Operating environment
  • Spresense Main & Expansion Board

  • Playback headphones or speakers

  • Recording microphone

In this sample, headphones and a microphone are used. Connect headphones or active speakers to the headphone jack on the expansion board. Please refer to the following hardware guide for microphone connection.

In this sample, it is not necessary to install the DSP file.

1.2.7.3. Operating procedure
  1. Open the sample sketch by selecting File → Examples → Examples for Spresense Audio → application → through from Arduino IDE.

    arduino audio examples through
  2. The sample sketch outputs the sound input from I2S to the headphone terminal.
    Here, change the sketch so that the sound input from the microphone is output to the headphone jack.
    Save it in an editable location by doing File → Save As.

  3. Change the argument input of the setThroughMode() function from AudioClass::I2sIn to AudioClass::MicIn and save.

  4. Select the COM port of Spresense with Tools → Port and write to the microcomputer board.

  5. Once the sketch is uploaded and executed, you will hear the sound coming from the microphone in your headphones.

1.2.7.4. Program description
  • To set the through mode, use the setThroughMode() function.

    theAudio->setThroughMode(input,      (1)
                             i2s_out,    (2)
                             sp_out,     (3)
                             input_gain, (4)
                             sp_drv);    (5)
    1 Select the input system with input .
    2 Select the source to output to I2S with i2s_out .
    3 Select whether to output to the headphone speaker with sp_out .
    For example, to output the audio input from the microphone to I2S, use
    Specify AudioClass::MicIn for input ,
    Specify AudioClass::Mic in i2s_out ,
    Specify false in sp_out .
    4 Set the microphone gain with input_gain.
    The range is 0 to 21 [dB] for analog microphones and -78.5 to 0 [dB] for digital microphones.
    For an analog microphone, specify an integer value multiplied by 10 for input_gain, for example, 100 when setting 10 [dB].
    For a digital microphone, specify an integer value multiplied by 100 for input_gain, for example -5 when setting -0.05 [dB].
    5 For sp_drv, refer to When using the speaker terminal of the expansion board.

1.2.8. Record in MP3 format

1.2.8.1. Overview

The sound from the microphone is recorded as an MP3 file on the microSD card.

1.2.8.2. Operating environment
  • Spresense Main & Expansion Board

  • Playback headphones or speakers

  • Recording microphone

In this sample, headphones and a microphone are used. Connect headphones or active speakers to the headphone jack on the expansion board. Please refer to the following hardware guide for microphone connection.

1.2.8.3. Operating procedure
  1. Install the DSP file MP3ENC for MP3 encoding to the microSD card (see [_install_dsp_file]).

  2. Open the sample sketch by selecting File → Examples →Examples for Spresense Audio → application → recorder on the Arduino IDE.

    arduino audio examples recorder
  3. Select the COM port of Spresense with Tools → Port and write to the microcomputer board.

  4. Once the sketch is uploaded and executed, recording will start and stop recording after a certain period of time.

  5. The recorded data is saved in the microSD card with the file name "Sound.mp3".
    Take out the microSD card and check the data recorded on the PC.

1.2.8.4. Program description
  • Set the recorder mode with the setRecorderMode() function.

    theAudio->setRecorderMode(input_device, (1)
                              input_gain,   (2)
                              bufsize,      (3)
                              is_digital);  (4)
    1 input_device : Input from the microphone ( AS_SETRECDR_STS_INPUTDEVICE_MIC ) as the input device.
    2 Set the microphone gain with input_gain.
    The range is 0 to 21 [dB] for analog microphones and -78.5 to 0 [dB] for digital microphones.
    For an analog microphone, specify an integer value multiplied by 10 for input_gain, for example, 100 when setting 10 [dB].
    For a digital microphone, specify an integer value multiplied by 100 for input_gain, for example -5 when setting -0.05 [dB].
    3 bufsize : Specify the buffer size of the recorder.
    For details, refer to Recorder buffer size.
    4 If the connected microphone is a digital microphone, specify true in the argument is_digital .
  • Initialize the recorder with the initRecorder() function.

    theAudio->initRecorder(codec,      (1)
                           codec_path, (2)
                           fs,         (3)
                           channel);   (4)
    1 Specify MP3 ( AS_CODECTYPE_MP3 ) as the codec type of codec .
    2 Specify the location where the DSP file is installed with codec_path .
    Specify "/mnt/sd0/BIN" for microSD card and "/mnt/spif/BIN" for SPI-Flash.
    3 Specify the sampling rate with fs .
    MP3 supports 32kHz ( AS_SAMPLINGRATE_32000 ), 44.1kHz ( AS_SAMPLINGRATE_44100 ), 48kHz ( AS_SAMPLINGRATE_48000 ).
    4 channel : Specify monaural ( AS_CHANNEL_MONO ) or stereo ( AS_CHANNEL_STEREO ).
  • Read MP3 encoded data for recording to a file with the readFrames() function.

    theAudio->readFrames(myFile); (1)
    1 The file to be recorded can be changed with the specified myFile.
    This sample write to a file called "Sound.mp3".
    Execute this function periodically to read the encoding result so that the encoding buffer does not overflow.
  • Start the recorder with the startRecorder() function.

  • Stop the recorder with the stopRecorder() function.

1.2.8.5. Recorder buffer size

The default buffer size used for writing to the microSD card is 160kByte. This size is a size that can write high resolution WAV audio enough, but this memory size is too large for playing a 48 kHz MP3 file, so memory is wasted.

Therefore, when recording a 48kHz MP3 file, You can change the buffer size by setRecorderMode. As a guide, the default 160 kByte is when playing a 192 kHz high resolution WAV file, and when recording a 48kHz MP3 file, it is recommended to set it to about 8kByte.

Regarding size, it is a guide only. Adjust according to the performance of the microSD card and the processing amount of the application. If you make the buffer too small, you will get errors when writing to the microSD card. If overflow occurs, increase the buffer size.
Even if audio is obtained, it may not be possible to write all the data due to factors such as the writing performance of the microSD card and the applications that are running. In the case of the current expansion board, the limit is 8ch/48kHz or 2ch/192kHz. It also depends on the speed class of the microSD card, so please use the one with the highest transfer rate possible.

1.2.9. Record in WAV format

1.2.9.1. Overview
1.2.9.2. Operating environment
1.2.9.3. Operating procedure
1.2.9.4. Program description

Retrieve PCM data

Repeat recording and playback

Object interface layer

Play MP3 music

Play dual MP3 music

Low_delay_playback

Record in MP3 format

Record in WAV format

Retrieve PCM data

Spresense Arduino Sensing Library

Let’s try Step Counter on Spresense

An example sketch of Step Counter

The following are the requirements to run the Step Counter sketch.

In this sketch, a physical sensor client using the BMI160 library works with a logical sensor client using the AESM. The acceleration data obtained from BMI160 is input, the activity recognition is performed by using machine learning (AI), and the recognition result is output.

AESM is installed by Tools > Burn Bootloader on Arduino IDE menu. Please refer to Install Bootloader for details.

On the Arduino IDE, select File→ Examples → Sensing under "Examples for Spresense" → application → step_counter and open the example sketch.

arduino stepcounter menu1
arduino stepcounter menu2

Compile this sketch and upload it to the Spresense board.

When you open serial monitor, the pedometer and other information will be displayed in real time.

arduino stepcounter result
tempo

Number of steps per second [Hz]

stride

Stride [cm]
This is fixed value.

speed

Moving speed [m/s]

distance

Moving distance [m]
The option of calculating the distance by fusing GPS positioning data will be supported in the future.
The current released software does not use GPS data.

step

Number of steps

move-type

Activity recognition result (stopping, walking and running)

Try to change the physical sensor of Step Counter

Here, let’s change the physical sensor to another sensor.

  • Sensor board to ROHM’s SPRESENSE-SENSOR-EVK-701.

  • KX122 Arduino Library

    • How to install KX122 Arduino library

      1. Select Clone or download → Download ZIP on GitHub site and download a ZIP file.

      2. Extract the downloaded ZIP file into any folder.

      3. On Arduino IDE menu, open Sketch → Include Library → Add .ZIP Library…​ and select the KX122 folder from the extracted folder.

      4. If Library added to your libraries. is shown on Arduino IDE, the installation is successful.

How to change step_counter.ino is below.

  • Include library

    Change to inclue <KX122.h>.

    #include <BMI160Gen.h>

    to

    #include <Wire.h>
    #include <KX122.h>
    KX122 kx122(KX122_DEVICE_ADDRESS_1F);
  • setup() function

    Change to initialize KX122. The accel_rate is set to 50 Hz by default of KX122 library.

      /* Initialize device. */
    
      BMI160.begin(BMI160GenClass::I2C_MODE, i2c_addr);
    
      /* Set device setting */
    
      BMI160.setAccelerometerRange(accel_range);
      BMI160.setAccelerometerRate(accel_rate);

    to

      Wire.begin();
      kx122.init();
  • loop() function

    Change to get the acceleration data from KX122.

      float x;
      float y;
      float z;
    
      /* Read raw accelerometer measurements from BMI160 */
    
      BMI160.readAccelerometerScaled(x, y, z);
    
      AccelSensor.write_data(x, y, z);

    to

      float acc[3];
    
      /* Read raw accelerometer measurements from KX122 */
    
      kx122.get_val(acc);
    
      AccelSensor.write_data(acc[0], acc[1], acc[2]);

After applying the above modifications, you should be able to work the Step Counter with KX122.

Let’s try to create the unique logical sensor of Step Counter

If you want to implement your own Step Counter algorithm instead of Sony’s provided Step Counter algorithm, create a logic sensor client as follow.

Create a logical sensor class that inherits SensorClient .

For example, implement MyCounterClass in MyCounter.h. Include the header file as needed. For example, SensorClient.h is required, but sensing/logical_sensor/step_counter.h is necessary only if you want to use the StepCounter definition.

Also, as an example, the figure below shows the storage location of generally used code.

diag 99c0d03fa17e569de5628b57e9395653
Figure 5. Example of MyCounter code storage location

Above diagram is an example of Windows environment. The Arduino directory is ~/Arduino in Ubuntu environment and ~/Documents/Arduino in macOS environment.

class MyCounterClass : public SensorClient

Override the following methods of MyCounterClass on MyCounter.cpp .

bool begin(int id,
           uint32_t subscriptions,
           int      rate,
           int      sample_watermark_num,
           int      size_per_sample));

int publish(FAR void* data,
            uint32_t  size_per_sample,
            uint32_t  freq,
            uint32_t  sample_watermark_num,
            uint32_t  timestamp);

int subscribe(sensor_command_data_mh_t& data);
  • Implementation of begin() :

Set initial settings for MyCounter. It is necessary to register a callback function to receive published accelerometer data. The callback function calls MyCounter’s subscribe() and get the data.

Description example
/* Callback function to receive accel sensor data. */

unsigned char mycounter_cb(sensor_command_data_mh_t &data)
{
  /* MyCounter subscribes to the received accel data. */

  return MyCounter.subscribe(data);
}

/* Initial setting of MyCounterClass. */

bool MyCounterClass::begin(int      id,
                           uint32_t subscriptions,
                           int      rate,
                           int      sample_watermark_num,
                           int      size_per_sample)
{
  /* Call super class begin() method. */

  return SensorClient::begin(id,
                             subscriptions,
                             rate,
                             sample_watermark_num,
                             size_per_sample,
                             mycounter_cb);        /* <-- Register callback function. */
}
  • Implementation of subscribe() :

You can create your own Step Counter by subscribing to the accelerometer data and processing it with your own algorithm.

If you use AccelSensorClass which is provided by the library, the data which is publised from accel sensor is 50 samples of acceleration data. The acceleration data is sampled by 50Hz. And the structure of them is data array of x, y, z axis acceleration (the unit is [G]). This structure is defined as st_accel_axis. Call Subscribe() in the base class ( SensorClient ) to get this data.
Next, please apply your original process and publish result of them. In this example, you will publish with SensorResultStepCounter data format. Each parameter of the data format is described here for more details.

Description example
/* Subscribe the accel sensor data which is published from AccelSenseorClass. */

int MyCounterClass::subscribe(sensor_command_data_mh_t& data)
{
  struct st_accel_axis
  {
    float x;
    float y;
    float z;
  } *accel_axis;

  /* Get accel sensor data via subscribe() of super class. */

  accel_axis = static_cast<accel_axis*>(SensorClient::subscribe(data));
  uint32_t  timestamp = data.time;

  /* In this sample, output parameter structure is SensorResultStepCounter. */

  SensorResultStepCounter output;

  /*
    Implement your own algorithm and put a value in output.
   */

  /* Publish processed data to subscriber. */

  publish(&output,
          sizeof(SensorResultStepCounter),
          1, /* 1Hz */
          1, /* 1sample */
          timestamp);
}
The subscribed data is sensor_command_data_mh_t . Please refer to here for this data structure.
  • Implementation of publish() :

publish() send the data calculated by MyCounter.
The StepCounterReader is provided as a sample of the reading application client, and if you use it as it is, the data structure to publish is

SensorResultStepCounter

Note that the parameter exec_result in SensorResultStepCounter should be set to SensorOK to notify result to application.

As mentioned above, when passed in the form of SensorResultStepCounter , publish() of the base class ( SensorClient ) can be used as it is.

SensorResultStepCounter format is below.

/*--------------------------------------------------------------------------*/
/**
 * @struct StepCounterStepInfo
 * @brief the structure of STEP_COUNTER results.
 */
typedef struct {

  float    tempo;     /**< Indicates tempo of walking / jogging calculated
                       *   from the input acceleration data.
                       *   The unit is [Hz].
                       */
  float    stride;    /**< Indicates stride calculated
                       *   from input acceleration data.
                       *   The unit is [cm].
                       */
  float    speed;     /**< Indicates speed of walking / jogging calculated
                       *   from the input acceleration data.
                       *   The unit is [m/s].
                       */
  float    distance;  /**< Indicates cumulative travel distance calculated
                       *   from the input acceleration data.
                       *   The unit is [m].
                       */
  uint32_t step;      /**< Indicates the number of steps calculated
                       *   from the input acceleration data.
                       * The unit is [step].
                       */
  StepCounterMovementType  movement_type; /**<
                                    * Indicates the walking type calculated
                                    * from the input acceleration data.
                                    */
  uint64_t          time_stamp;    /**< Indicates latest timestamp of the
                                    *   acceleration sensor data used.
                                    */
} StepCounterStepInfo;

/*--------------------------------------------------------------------------*/
/**
 * @struct SensorResultStepCounter
 * @brief the structure of sensor result on step counter commands.
 */
typedef struct
{
  SensorExecResult exec_result; /**< Execute resule.  */

  union
    {
      StepCounterStepInfo steps;        /**< Step count.         */
      SensorAssertionInfo assert_info;  /**< Assert information. */
    };
} SensorResultStepCounter;

Let’s try to change the data according to your logic sensor

Of course, you may want to change the resulting sensor data to match your own logic sensor.

In that case, use your defined data type (e.g. MyStepCounterResult) in publish() of MyCounter class.

int MyCounterClass::subscribe(sensor_command_data_mh_t& data)
{
  ..(Omission)..

  MyStepCounterResult output;

  publish(&output,
          sizeof(MyStepCounterResult),
          1, /* 1Hz */
          1, /* 1sample */
          timestamp);
}

Please implement as follows in Application ( StepCountReaderClass ).

subscribe()
{
  MyStepCounterResult* steps = reinterpret_cast<MyStepCounterResult*>(ApplicationSensorClass.subscribe(data));
}

Let’s try to create new sensor client

Let’s try to create new physical sensor client

[T.B.D]

2. Spresense Arduino multi-core environment

This section describes the multi-core programming development environment using Arduino IDE.

In Arduino multi-core environments, please use the MultiCore MP library. The application CPU has total of 6 cores, consisting of 1 MainCore and 5 SubCore. In the normal single core application, the user programs only use MainCore. In the multi-core application environment described here, you can program all cores including SubCore.

MainCore

MainCore is always started at power on.

SubCore(s)

SubCore(s) are launched from MainCore. There are total of 5 SubCores.

In this tutorial, we show two examples with the MultiCore MP library.

Multicore Boot Example

Boot 4 SubCores and perform LED blinking using multicore.
By running this sample, you can understand the procedure to boot SubCore.

Multicore MessageHello Example

Communicate messages between MainCore and SubCore.
By running this sample, you can understand how to communicate between MainCore and SubCore.

2.1. Multicore Boot example

The main steps are:

  1. Open a sketch using Arduino IDE

  2. Select core

  3. Compile & upload

Repeat above for the number of cores you want to program.

Please follow the procedure as below.

  • Launch Arduino IDE from your desktop
    (If there is no icon on your desktop, please launch from the start menu etc.)

    arduino multicore desktop
    In multi-core programming, please note when you launch multiple Arduino IDE windows.
    If you open a new window with File→ New from the Arduino IDE menu, if you switch the menu of Core selection in one window, the setting is reflected to all the windows.
    As a workaround, when you open multiple Arduino IDE windows for every Core, please launch Arduino IDE from the icon on your desktop. Then, menu settings such as Core selection can be set independently for each window.
  • MainCore programming

    • Open a sketch from Arduino IDE menu File → Examples → MultiCore MP → Boot → Main.

      arduino multicore boot0
      arduino multicore boot1 main
    • Explanation for Main.ino sketch

      MP.begin(subid) boot 4 SubCores.

      arduino multicore boot4 main
    • Select MainCore from Arduino IDE menu Tools → Core → MainCore.

      arduino multicore boot2 main

      You can see that MainCore is selected in the status bar.

      arduino multicore boot3 main
      In Arduino IDE 1.8.9 or later, there is an issue, the status bar is not displayed correctly after switching menu.
    • Press the Upload button to do Compile & Upload.

      arduino multicore boot5 main
      You cannot upload multiple core programs from multiple windows at the same time.
      Upload each core one by one so that the upload operation is not executed simultaneously.
    • Size of used memory is displayed on the log of compilation result.

      MainCore always uses 768 KByte of memory.

      arduino multicore boot6 main

      If Upload is successful, MainCore programming is completed.

      • As a test, try to open the serial monitor without SubCore program.
        Since there is no SubCore, you can see that MP.begin(1~4) returns error.

        arduino multicore boot7 serial err

        If the serial monitor is open, uploading from other windows may fail.
        Please close the serial monitor after this trial.

Then we will program the SubCore.

  • SubCore programming

    • Open a sketch from Arduino IDE menu File → Examples → MultiCore MP → Boot → Sub1.

      arduino multicore boot0
      arduino multicore boot1 sub1
    • Explanation for Sub1.ino sketch

      In setup(), by calling MP.begin(), notify MainCore that startup is complete.
      In loop(), output log by MPLog() and blink a LED.

      arduino multicore boot4 sub1
    • Select SubCore 1 from Arduino IDE menu Tools → Core → SubCore 1.

      arduino multicore boot2 sub1

      You can see that SubCore 1 is selected in the status bar.

      arduino multicore boot3 sub1
      In Arduino IDE 1.8.9 or later, there is an issue, the status bar is not displayed correctly after switching menu.
    • Press the Upload button to do Compile & Upload.

      arduino multicore boot5 sub1
      You cannot upload multiple core programs from multiple windows at the same time.
      Upload each core one by one so that the upload operation is not executed simultaneously.
    • Size of used memory is displayed on the log of compilation result.

      You can see that this SubCore sample sketch uses 128 KBytes of memory.
      This size increases or decreases depending on the user program.

      arduino multicore boot6 sub1

      If Upload is successful, SubCore 1 programming is completed.

  • Follow the same procedure for SubCore 2, 3 and 4 and execute Compile & Upload.

    • Open Sub2 sketch, select SubCore 2, and Compile & Upload.

    • Open Sub3 sketch, select SubCore 3, and Compile & Upload.

    • Open Sub4 sketch, select SubCore 4, and Compile & Upload.

  • Operation check

    • Blink 4 LEDs on the Spresense main board.

    • Open the serial monitor, and you can see the log from each core.

      arduino multicore boot7 serial
  • Summary

    • We have learned how to boot SubCore using a simple example sketch.
      Add a call to MP.begin() when you port an existing sketch to multi-core environment.

    • The basic usage, such as Arduino IDE’s Compile & Upload method, is the same as usual in multi-core environments. The main difference is that a menu of core selections has been added in Tools→ Core.

2.2. Multicore MessageHello example

The main steps are the same as Multicore Boot example.
In this example, MainCore and SubCore(s) use a common sketch.

  1. Open a sketch using Arduino IDE (only once)

  2. Select core

  3. Compile & upload

Repeat above for the number of cores you want to program. This example runs all SubCores.

Please follow the procedure as below.

  • Launch Arduino IDE from your desktop
    (If there is no icon on your desktop, please launch from the start menu etc.)

    arduino multicore desktop
    In multi-core programming, please note when you launch multiple Arduino IDE windows.
    If you open a new window with File→ New from the Arduino IDE menu, if you switch the menu of Core selection in one window, the setting is reflected to all the windows.
    As a workaround, when you open multiple Arduino IDE windows for every Core, you always launch Arduino IDE from the icon on your desktop. By doing so, menu settings such as Core selection can be set independently for each window.
  • MainCore programming

    • Open a sketch from Arduino IDE menu File → Examples → MultiCore MP → Message → MessageHello.

      arduino multicore boot0
      arduino multicore hello1
    • Explanation for MessageHello.ino sketch

      We use #ifdef SUBCORE ~ #else ~ #endif to switch between MainCore and SubCore implementations in this sketch.

      • MainCore implementation

        In setup(), MP.begin(subid) boot 5 SubCores
        By calling MP.RecvTimeout(MP_RECV_POLLING), set polling in receiver mode. In this mode, when MP.Recv() is called, Recv() do polling with non-blocking.
        In loop(), MP.Recv() receives the packet address from each SubCore, and outputs the message in the packet.

        arduino multicore hello4 main
      • SubCore implementation

        In setup(), by calling MP.begin(), notify MainCore that startup is complete.
        In loop(), put "Hello" message into packet, and send the packet address by MP.Send().

        arduino multicore hello4 sub
    • Select MainCore from Arduino IDE menu Tools → Core → MainCore.

      arduino multicore hello2 main

      You can see that MainCore is selected in the status bar.

      arduino multicore boot3 main
      In Arduino IDE 1.8.9 or later, there is an issue, the status bar is not displayed correctly after switching menu.
    • Press the Upload button to do Compile & Upload.

      arduino multicore hello5 main
      You cannot upload multiple core programs from multiple windows at the same time.
      Upload each core one by one so that the upload operation is not executed simultaneously.
    • Size of used memory is displayed on the log of compilation result.

      MainCore always uses 768 KByte of memory.

      arduino multicore boot6 main

      If Upload is successful, MainCore programming is completed.

Then we will program the SubCore.

  • SubCore programming

    • Select SubCore 1 from Arduino IDE menu Tools → Core → SubCore 1.

      arduino multicore hello2 sub1

      You can see that SubCore 1 is selected in the status bar.

      arduino multicore boot3 sub1
      In Arduino IDE 1.8.9 or later, there is an issue, the status bar is not displayed correctly after switching menu.
    • Press the Upload button to do Compile & Upload.

      arduino multicore hello5 main
      You cannot upload multiple core programs from multiple windows at the same time.
      Upload each core one by one so that the upload operation is not executed simultaneously.
    • Size of used memory is displayed on the log of compilation result.

      You can see that this SubCore sample sketch uses 128 KBytes of memory.
      This size increases or decreases depending on the user program.

      arduino multicore boot6 sub1

      If Upload is successful, SubCore 1 programming is completed.

  • Follow the same procedure for SubCore 2, 3, 4 and 5 and execute Compile & Upload.

    • Select SubCore 2, and Compile & Upload.

    • Select SubCore 3, and Compile & Upload.

    • Select SubCore 4, and Compile & Upload.

  • Operation check

    • Open the serial monitor.

    • MainCore receives the message sent from each SubCore and outputs them to the serial monitor.

      arduino multicore hello7 serial
  • Summary

    • We have learned how to communicate between MainCore and SubCore.

    • It can communicate by sending the address of message because of the shared memory.

    • You can use a common sketch for multi-core programming.

2.3. Notices

  • See developer guide for the details of MultiCore MP library.

  • If you want to remove SubCore binaries on the Spresense board, please do Install Bootloader. When installing the loader, remove all SubCore binaries and then start installing the loader.

    arduino multicore remove subcore

    Note that SubCore does not run unless you call MP.begin() from MainCore. Therefore, there is no problem even if the old SubCore remains on your Spresense board.

  • If you open multiple Arduino IDE windows for every Core, please launch Arduino IDE from the icon on your desktop. Then, menu settings such as Core selection can be set independently for each window.

  • Use Arduino IDE to open multiple windows and select core is complicated, but you can use command line tools such as arduino-builder and arduino-cli. Also, you can upload the multiple binaries for all cores at once by using flash_writer tool. How to use the command line tools will be updated in this manual later.

3. Tutorial of SignalProcessing Library

The SignalProcessing library has FFT/IIR filters and the following sample sketches are provided for them.

This document explains how to execute these sample sketches.

3.1. PeakDetector

3.1.1. Overview

This sample sketch detects the peak frequency in real time by performing real-time frequency analysis with the FFT library on the sound data input by the microphone. Please refer to Audio library for the Audio function.

In addition, this sample performs with offloading the signal processing library to SubCore. Please see also MultiCore MP Library.

3.1.2. Operating environment

  • Spresense Main & extension Board

  • Microphones for recording

Please refer to the following hardware guide for microphone connection.

This sample is the multi-channels application with 4 channels.
Connect microphones (up to 4@analog, 8@digital) and please adjust them within the application.

3.1.3. Operating procedure

The main operation procedure is as follow:

  1. Compile & upload SubCore for signal processing

  2. Compile & upload MainCore for sound acquisition & application processing

  3. Running the application

If you compile and upload MainCore at first , it will call SubCore after startup, resulting in an error.
If you have already compiled and uploaded to SubCore, the program of SubCore that has already been uploaded will be started after starting MainCore.
3.1.3.1. Compilation & upload of SubCore
  1. Open the sample sketch by selecting File→ Sketch example→ Sketch example for Spresense SignalProcessing→ PeakDetector→ SubFFT from Arduino IDE.

  2. Select Tools→ Core→ SubCore1 (Tools→ Core→ SubCore1) from the Arduino IDE menu.

    In this sample, the offloaded SubCore is SubCore1.

Select the COM port of Spresense with Tools→ Serial Port and write to the microcomputer board.

+ NOTE: The SubCore program waits for instructions from the MainCore after starting.

Now, the preparation on the SubCore is OK.

3.1.3.2. Compilation & upload of MainCore
  1. Open the sample sketch by selecting File→ Sketch example→ Sketch example for Spresense SignalProcessing→ PeakDetector → MainAudio from Arduino IDE.

  2. Select Tools→ Core→ MainCore (Tools→ Core→ MainCore) from the Arduino IDE menu

    Write to the microcomputer board.

If you have selected the serial port for uploading SubCore, you do not need to select it again.

Now, the preparation on the MainCore is also OK.

3.1.3.3. Run the application
  1. After uploading MainCore , open Tools→ Serial Monitor and start the application.

  2. When the application is started, it analyzes the frequency of the sound of each channel input from the microphone and continues to display the peak frequency on the serial monitor.

3.1.4. the Explanation of program

  • The number of input audio channels is specified in the following part of MainAudio.ino.

/* Select mic channel number */
//const int mic_channel_num = 1;
//const int mic_channel_num = 2;
const int mic_channel_num = 4;

NOTE : #define MAX_CHANNEL_NUM 4 on the SubCore means the max number of channels is 4, user can select 1-4 channels dynamically.

This sketch calls FFT.begin(); without arguments at SubCore, so the execution number of channels on SubCore is equal to the number of maximum channels.
If you don’t want the number of execution channels to be equal to the maximum number of channels, Please call FFT.begin (WindoHamming, "The number of execution channels", (FFTLEN / 2));, and specify the number of execution channels.
  • The number of FFT taps is specified as follow in SubFFT.ino.

/* Select FFT length */

//#define FFT_LEN 32
//#define FFT_LEN 64
//#define FFT_LEN 128
//#define FFT_LEN 256
//#define FFT_LEN 512
#define FFT_LEN 1024
//#define FFT_LEN 2048
//#define FFT_LEN 4096
  • The current window function and the number of FFT overlap samples are set to default. The default for the window is `Hamming window' and the overlap size is designed to shift the window by the number of taps/2.

If you want to change this, please change in the SubFFT.ino from

  FFT.begin();

to

  FFT.begin ("window function you want to use", "number of execution channels", "number of samples you want to overlap");

3.2. SoundDetector

3.2.1. Overview

This sample sketch uses the FFT library for real-time frequency analysis of the sound data input by the microphone, and detects whether sound with a certain sound pressure level is input within the specified frequency range. Please refer to Audio library for the Audio function.

In addition, this sample performs with offloading the signal processing library to SubCore. Please see also MultiCore MP Library.

3.2.2. Operating environment

  • Spresense Main & extension Board

  • Microphones for recording

Please refer to the following hardware guide for microphone connection.

This sample is the multi-channels application with 4 channels.
Connect microphones (up to 4@analog, 8@digital) and please adjust them within the application.

3.2.3. Operating procedure

The main operation procedure is as follow:

  1. Compile & upload SubCore for signal processing

  2. Compile & upload MainCore for sound acquisition & application processing

  3. Running the application

If you compile and upload MainCore at first , it will call SubCore after startup, resulting in an error.
If you have already compiled and uploaded to SubCore, the program of SubCore that has already been uploaded will be started after starting MainCore.
3.2.3.1. Compilation & upload of SubCore
  1. Open the sample sketch by selecting File→ Sketch example→ Sketch example for Spresense SignalProcessing→ SoundDetector → SubFFT from Arduino IDE.

  2. Select Tools→ Core→ SubCore1 (Tools→ Core→ SubCore1) from the Arduino IDE menu.

    In this sample, the offloaded SubCore is SubCore1.
  3. Select the COM port of Spresense with Tools→ Serial Port and write to the microcomputer board.

    The SubCore program waits for instructions from the MainCore after starting.

Now, the preparation on the SubCore is OK.

3.2.3.2. Compilation & upload of MainCore
  1. Open the sample sketch by selecting File→ Sketch example→ Sketch example for Spresense SignalProcessing→ SoundDetector→ MainAudio from Arduino IDE.

  2. Select Tools→ Core→ MainCore ( Tools→ Core→ MainCore) from the Arduino IDE menu

    Write to the microcomputer board.

If you have selected the serial port for uploading SubCore, you do not need to select it again.

Now, the preparation on the MainCore is also OK.

3.2.3.3. Run the application
  1. After uploading MainCore, open Tools→ Serial Monitor and start the application.

  2. When the application is started, frequency analysis is performed on the sound of each channel input from the microphone, and if sound with more than specified sound power is input within the specified frequency range, it shows the detection and channel numbernel number like > Found channel 0.

3.2.4. the Explanation of program

  • The number of input audio channels is specified in the following part of MainAudio.ino.

/* Select mic channel number */
//const int mic_channel_num = 1;
//const int mic_channel_num = 2;
const int mic_channel_num = 4;

NOTE : #define MAX_CHANNEL_NUM 4 on the SubCore means the max number of channels is 4, user can select 1-4 channels dynamically.

This sketch calls FFT.begin(); without arguments at SubCore, so the execution number of channels on SubCore is equal to the number of maximum channels.
If you don’t want the number of execution channels to be equal to the maximum number of channels, Please call FFT.begin (WindoHamming, "The number of execution channels", (FFTLEN / 2));, and specify the number of execution channels.
  • The number of FFT taps is specified as follow in SubFFT.ino.

/* Select FFT length */

//#define FFT_LEN 32
//#define FFT_LEN 64
//#define FFT_LEN 128
//#define FFT_LEN 256
//#define FFT_LEN 512
#define FFT_LEN 1024
//#define FFT_LEN 2048
//#define FFT_LEN 4096
  • The current window function and the number of FFT overlap samples are set to default. The default is the window is Hamming window and the overlap size is designed to shift the window by the number of taps/2.

If you want to change this, please change in the SubFFT.ino from

  FFT.begin();

to

  FFT.begin ("window function you want to use", "number of execution channels", "number of samples you want to overlap");
  • The detector settings are as follow:

#define POWER_THRESHOLD 30 // Power
#define LENGTH_THRESHOLD 30 // 20ms
#define INTERVAL_THRESHOLD 100 // 100ms

#define BOTTOM_SAMPLING_RATE 1000 // 1kHz
#define TOP_SAMPLING_RATE 1500 // 1.5kHz
  • Specify the lower limit of frequency range for the detection with BOTTOM_SAMPLING_RATE. Set the Hz value.

  • Specify the upper limit of frequency range for the detection with TOP_SAMPLING_RATE. Set the Hz value.

  • Specify the threshold of the sound power for the detection with POWER_THRESHOLD.

  • If audio in the relevant frequency range continues to be input for more than LENGTH_THRESHOLD time, it is judged that the sound is detected.

  • If it is re-detected within INTERVAL_THRESHOLD after being detected, it is considered to be a part of the previously detected sound. Also, if it is detected more than INTERVAL_THRESHOLD away, another sound is detected and the notification is raised to MainCore again.

3.3. VoiceChanger

3.3.1. Overview

This sample sketch performs FFT / iFFT on the voice data input by the microphone to change the pitch and regenerate the voice data. The LP filter is applied to the generated data to cut off the high frequency noise added due to the lack of phase information, etc., and then output. Please refer to Audio library for the Audio function.

In addition, this sample performs with offloading the signal processing library to SubCore. Please see also MultiCore MP Library.

This sample is provided as a simple FFT/iFFT execution sample. Therefore, it is not good for the sound quality as a pitch shifter.

3.3.2. Operating environment

  • Spresense Main & extension Board

  • Headphones or speakers for playback

  • Microphones for recording

In this sample, headphones and a microphone are used. Please connect headphones or active speakers to the headphone jack on the extension board.
Please refer to the following hardware guide for microphone connection.

The samples provided are for a monorail microphone. If you want more microphones, Connect microphones (up to 4@analog, 8@digital) and please adjust them within the application.
Audio output can only be up to stereo. If the input is 3 channels or more, perform downmix processing before output, convert to 2 channels data.

3.3.3. Operating procedure

The main operation procedure is as follow:

  1. Compile & upload SubCore for signal processing

  2. Compile & upload MainCore for sound acquisition & application processing

  3. Running the application

If you compile and upload MainCore at first , it will call SubCore after startup, resulting in an error.
If you have already compiled and uploaded to SubCore, the program of SubCore that has already been uploaded will be started after starting MainCore.
3.3.3.1. Compilation & upload of SubCore
  1. Open the sample sketch by selecting File→ Sketch example→ Sketch example for Spresense SignalProcessing→ VoiceChanger→ SubFFT from Arduino IDE.

  2. Select Tools→ Core→ SubCore1 (Tools→ Core→ SubCore1) from the Arduino IDE menu.

    In this sample, the offloaded SubCore is SubCore1.
  3. Select the COM port of Spresense with Tools→ Serial Port and write to the microcomputer board.

    The SubCore program waits for instructions from the MainCore after starting.

Now, the preparation on the SubCore is OK.

3.3.3.2. Compilation & upload of MainCore
  1. Open the sample sketch by selecting File→ Sketch example→ Sketch example for Spresense SignalProcessing→ VoiceChanger → MainAudio from Arduino IDE.

  2. Select Tools→ Core→ MainCore ( Tools→ Core→ MainCore) from the Arduino IDE menu

    Write to the microcomputer board.

    If you have selected the serial port for uploading SubCore, you do not need to select it again.

Now, the preparation on the MainCore is also OK.

3.3.3.3. Run the application
  1. After uploading MainCore, open Tools→ Serial Monitor and start the application.

  2. When the application is started, the data in the frequency domain obtained by FFT is processed into the audio data input from the microphone, the data with the changed pitch is reconstructed with iFFT and output.

3.3.4. the Explanation of program

  • The number of input audio channels is specified in the following part of MainAudio.ino.

/* Select mic channel number */
const int mic_channel_num = 1;
//const int mic_channel_num = 2;
In this sample, as #define MAX_CHANNEL_NUM 1 defined on SubCore, it only a monorail channels can be executed.
  • The number of FFT taps is specified as follow in SubFFT.ino.

/* Select FFT length */

//#define FFT_LEN 32
//#define FFT_LEN 64
//#define FFT_LEN 128
//#define FFT_LEN 256
//#define FFT_LEN 512
#define FFT_LEN 1024
//#define FFT_LEN 2048
//#define FFT_LEN 4096
  • This sample regenerates the audio with FFT/iFFT, so it is rectangular window and no overlap.

   FFT.begin(WindowRectangle,MAX_CHANNEL_NUM,0);

If you want to change the algorithm, please change accordingly.

  • The pitch shift amount is set in the following in MainAudio.ino.

   static int pitch_shift = 10;

It can be shifted in the positive or negative direction according to the FFT resolution.
This parameter can be changed even during signal processing.

3.4. HighPassSound / LowPassSound

3.4.1. Overview

This sample sketch outputs the sound data that is perform HPF (high-pass filter) and LPF (low-pass filter) by IIR.

In addition, this sample performs with offloading the signal processing library to SubCore. Please see also MultiCore MP Library.

3.4.2. Operating environment

  • Spresense Main & extension Board

  • Headphones or speakers for playback

  • Microphones for recording

In this sample, headphones and a microphone are used. Please connect headphones or active speakers to the headphone jack on the extension board.
Please refer to the following hardware guide for microphone connection.

The samples provided are for a monorail microphone. If you want more microphones, Connect microphones (up to 4@analog, 8@digital) and please adjust them within the application.
Audio output can only be up to stereo. If the input is 3 channels or more, perform downmix processing before output, convert to 2 channels data.

3.4.3. Operating procedure

The main operation procedure is as follow:

  1. Compile & upload SubCore for signal processing

  2. Compile & upload MainCore for sound acquisition & application processing

  3. Running the application

If you compile and upload MainCore at first, it will call SubCore after startup, resulting in an error.
If you have already compiled and uploaded to SubCore, the program of SubCore that has already been uploaded will be started after starting MainCore.
3.4.3.1. Compilation & upload of SubCore
  1. Open the sample sketch by selecting File→ Sketch example→ Sketch example for Spresense SignalProcessing→ HighPassSound / LowPassSound→ SubFFT from Arduino IDE.

  2. Select Tools→ Core→ SubCore1 (Tools→ Core→ SubCore1) from the Arduino IDE menu.

    In this sample, the offloaded SubCore is SubCore1.
  3. Select the COM port of Spresense with Tools→ Serial Port and write to the microcomputer board.

    The SubCore program waits for instructions from the MainCore after starting.

Now, the preparation on the SubCore is OK.

3.4.3.2. Compilation & upload of MainCore
  1. Open the sample sketch by selecting File→ Sketch example→ Sketch example for Spresense SignalProcessing→ HighPassSound / LowPassSound → MainAudio from Arduino IDE.

  2. Select Tools→ Core→ MainCore ( Tools→ Core→ MainCore) from the Arduino IDE menu

    Write to the microcomputer board.

If you have selected the serial port for uploading SubCore, you do not need to select it again.

Now, the preparation on the MainCore is also OK.

3.4.3.3. Run the application
  1. After uploading MainCore, open Tools→ Serial Monitor and start the application.

  2. When the application is started, the audio data input from the microphone will be the audio data that has been LPF/HPF processed by IIR, and output it from speaker.

3.4.4. the Explanation of program

  • The number of input audio channels is specified in the following part of MainAudio.ino.

/* Select mic channel number */
const int mic_channel_num = 1;
//const int mic_channel_num = 2;
In this sample, as #define MAX_CHANNEL_NUM 1 defined on SubCore, it only a monorail channels can be executed.
  • The cutoff frequency and Q value are set by the following part of SubCore.

   HPF.begin (TYPE_HPF, g_channel, 1000, sqrt (0.5));

or

   LPF.begin (TYPE_LPF, g_channel, 1000, sqrt (0.5));

In this sample, the cutoff frequency is set to 1kHz and the Q value is set to 1 / √2.