Developer World Spresense
日本語 中文
Table of Contents

1. Audio Tutorial

The audio library contains a variety of sample sketches, including audio playback and recording.

By taking advantage of the multi-core characteristics of Spresense, codec processing such as audio encoding/decoding is executed by a dedicated core which we call DSP. The audio applications can run without doing the heavy codec processing on the main core.

Audio Tutorial explains how to install DSP files and how to execute various sample sketches.

1.1. Install DSP files

When running the sample, it is necessary to install the DSP file in advance for decoder if use audio playback or encoder if you use audio recorder. This section describes how to install the DSP file.

1.1.1. DSP file type

Table 1. DSP type
DSP file name DSP installer Description

MP3DEC

mp3_dec_installer

Used for MP3 audio playback

MP3ENC

mp3_enc_installer

Used for MP3 audio recording

WAVDEC

wav_dec_installer

Used for WAV (PCM) audio playback

SRC

src_installer

Use for WAV(PCM) audio recording

1.1.2. How to install DSP

There are two ways to install DSP.

  • How to download DSP file and copy directly to microSD card

  • Using the DSP installer sketch

You can use either method when installing to a microSD card. When using the DSP installer, it is possible to install it on the SPI-Flash on the main board in addition to the microSD card on the extension board.

The DSP needs to be install once, you only need to re install it if the DSP file is updated.
1.1.2.1. How to install by copying DSP file to microSD card

Download the latest DSP file from here.

  • MP3DEC

  • MP3ENC

  • WAVDEC

  • SRC

    1. Create a BIN directory on the FAT32 formatted microSD card and copy the downloaded file.

      arduino dsp copy cn
      Figure 1. Copy DSP file

Please insert this microSD card into the extension board and run it when you want to execute the audio samples.

1.1.2.2. How to install using DSP installer

A DSP installer is provided for each DSP file you install.

  1. From the Arduino IDE, start from File → Examples → Examples for Spresense / Audio → dsp_installer.

    arduino dsp installer cn
    Figure 2. Starting the DSP installer
  2. Select the COM port of Spresense in Tools → Port and execute writing.

  3. When the compilation & writing (Upload) is completed, start the serial monitor.

  4. If you select a baud rate of 115200 bps, a message will appear as shown in the figure below.

  5. Select the number that you want to install and press the send button.
    When installing to a microSD card, insert a FAT32 formatted microSD card into the microSD card slot on the extension board.

    arduino dsp installer monitor1 en
    Figure 3. Select DSP installation location
  6. If the installation is successful, the message shown below is displayed.

    arduino dsp installer monitor2 en
    Figure 4. DSP installation execution
In the audio playback/recording application program, the location where the DSP file is installed is specified by the argument of the initPlayer() function or the initRecorder() function. By default, a microSD card (/mnt/sd0/BIN) is specified. When changing to SPI-Flash, rewrite to /mnt/spif/BIN.

1.2. High-level interface layer

Spresense Audio library has a layer structure as described in Audio library development guide.
In the high-level interface layer, typical functions with a high degree of abstraction are supported while maintaining the consistency of the entire system.

Below is a sample using this high-level interface layer.

1.2.1. Make a beep

1.2.1.1. Overview

This sample uses the capabilities of the Audio hardware to emit a beep.

Output the sound of the specified frequency with the setBeep() function of the Audio library. This function is very similar to Arduino’s tone() function, but it does not require a piezoelectric buzzer. Sound is output from the headphone terminal of the Spresense extension board.

In this sample, it is not necessary to install the DSP file.

1.2.1.2. Operating environment
  • Spresense Main & Extension Board

  • Playback headphones or speakers

Connect headphones or active speakers to the headphone jack on the extension board.

1.2.1.3. Operating procedure
  1. Open the sample sketch by selecting File → Examples → Example for Spresense / Audio → application → beep from Arduino IDE.

arduino audio examples beep

Select the COM port of Spresense with Tools → Port and write to the microcomputer board.

arduino audio examples beep2

The scale of Do-re-mi-fa-so-la-ti-do is output from the headphones.

1.2.1.4. Program description
  • Set the player mode with the setPlayerMode() function.

    theAudio->setPlayerMode(device,          (1)
                            player0bufsize,  (2)
                            player1bufsize); (3)
    1 device: Select the output device, whether to output to speaker headphones (AS_SETPLAYER_OUTPUTDEVICE_SPHP) or I2S (AS_SETPLAYER_OUTPUTDEVICE_I2SOUTPUT).
    This sample uses AS_SETPLAYER_OUTPUTDEVICE_SPHP.
    When using I2S, refer to How to output to I2S.
    2 player0bufsize: Specifies the buffer size of player 0.
    3 player1bufsize: Specifies the buffer size of player 1.
    For details, refer to Player buffer size.
    The buffer is not used if only the beep sound is output, so 0 is specified in this sample.
  • To output the beep sound, use the setBeep() function.

    theAudio->setBeep(enable,       (1)
                       volume,      (2)
                       frequency);  (3)
    1 If you set enable to 1, the beep sound will be output, and if enable is set to 0, the output will stop.
    2 volume: You can change the volume (-90 ~ 0 [dB]).
    3 frequency: Outputs the sound of the specified frequency.

1.2.2. Play MP3 music

1.2.2.1. Overview

Play MP3 music files on the microSD card.

1.2.2.2. Operating environment
  • Spresense Main & Extension Board

  • microSD card

  • Playback headphones or speakers

This sample uses a microSD card and headphones. Connect headphones or active speakers to the headphone jack on the extension board.

1.2.2.3. Operating procedure
  1. Install the DSP file MP3DEC for MP3 decoding to the microSD card (see Install DSP files).

  2. Put on the MP3 file you want to play with the file name "Sound.mp3" in the root directory of the microSD card.

  3. Insert the microSD card into the microSD card slot on the extension board.

  4. Open the sample sketch by selecting File → Examples → Examples for Spresense / Audio → application → player from Arduino IDE.

    arduino audio examples player

Select the COM port of Spresense with Tools → Port and write to the microcomputer board.

When the sketch is uploaded and executed, the MP3 file placed on the microSD card will be played via headphone.
Play the file to the end and stop.

1.2.2.4. Program description
  • Set the clock mode with the setRenderingClockMode() function.

    theAudio->setRenderingClockMode(mode); (1)
    1 mode: Select normal (AS_CLKMODE_NORMAL) or high resolution (AS_CLKMODE_HIRES) or clock mode.
    For MP3 playback, specify AS_CLKMODE_NORMAL.
  • Set the player mode with the setPlayerMode() function.

    theAudio->setPlayerMode(device,    (1)
                             sp_drv);  (2)
    1 device: Select the output device, whether to output to speaker headphones (AS_SETPLAYER_OUTPUTDEVICE_SPHP) or I2S (AS_SETPLAYER_OUTPUTDEVICE_I2SOUTPUT).
    This sample uses AS_SETPLAYER_OUTPUTDEVICE_SPHP.
    When using I2S, refer to How to output to I2S.
    2 sp_drv: Sets the output drive.
    In this sample, specify AS_SP_DRV_MODE_LINEOUT.
    For details, refer to When using the speaker terminal of the extension board.
  • Initialize the player with the initPlayer() function.

    theAudio->initPlayer(id,         (1)
                         codec,      (2)
                         codec_path, (3)
                         fs,         (4)
                         channel);   (5)
    1 id: Select Player ID.
    This sample uses AudioClass::Player0.
    2 codec: Specify MP3 (AS_CODECTYPE_MP3) as the codec type.
    3 codec_path: Specify the location where the DSP file is installed.
    Specify "/mnt/sd0/BIN" for microSD card and "/mnt/spif/BIN" for SPI-Flash.
    4 fs: Specify the sampling rate.
    MP3 supports 32kHz (AS_SAMPLINGRATE_32000), 44.1kHz (AS_SAMPLINGRATE_44100), 48kHz (AS_SAMPLINGRATE_48000). If you specify AS_SAMPLINGRATE_AUTO, the sampling rate is automatically determined.
    5 channel: Specify monaural (AS_CHANNEL_MONO) or stereo (AS_CHANNEL_STEREO).
  • Write the contents of the MP3 file to the decoder with the writeFrames() function.

    theAudio->writeFrames(id,       (1)
                           myFile); (2)
    1 id: Select Player ID.
    This sample uses AudioClass::Player0.
    2 You can change the playback file by specifying myFile in the argument.
    This sample uses a file called "Sound.mp3".
    Execute this function periodically to supply the stream to the decoder so that the decoding will not be interrupted. When writing the file is completed, AUDIOLIB_ECODE_FILEEND is returned as the return value of the function.
  • Start the player with the startPlayer() function.

    theAudio->startPlayer(id); (1)
    1 id: Select Player ID.
    This sample uses AudioClass::Player0.
    Call it after writing to the decoder with the writeFrames() function.
  • Stop the player with the stopPlayer() function.

    theAudio->stopPlayer(id,      (1)
                          mode);  (2)
    1 id: Select Player ID.
    This sample uses AudioClass::Player0.
    2 You can choose to stop immediately in mode (AS_STOPPLAYER_NORMAL) or wait until the stream supplied to the decoder has finished playing and then stop (AS_STOPPLAYER_ESEND). If called without the mode argument, it works with AS_STOPPLAYER_NORMAL.
  • Adjust the volume with the setVolume(volume) function.

    theAudio->setVolume(volume); (1)
    1 volume: You can change the volume. (-1020 ~ 120)
    1/10 of the specified value becomes the actual volume [dB]. For example, the volume when 15 is specified, is 1.5 [dB].
1.2.2.5. Player buffer size

A buffer size of 160kByte to use for reading from the microSD card is secured by default for each player. This size is such that high resolution WAV audio can be read, and when playing a 48 kHz MP3 file or etc, you can reduce the buffer size used in the argument of the setPlayerMode function. As a guideline, when playing a 192kHz high resolution WAV file, it needs the 160kByte which is default, and it is recommended to set it to about 24kByte when playing an MP3 file of about 128kbps.

Regarding the size, it is a suggestion only. Adjust according to the performance of the microSD card and the processing amount of the application. If you make the buffer too small, you will get errors when reading from the microSD card. If read data underflow occurs, increase the buffer size.
1.2.2.6. How to output to I2S

If you want to switch the output device to I2S, switch the output device with setPlayerMode to AS_SETPLAYER_OUTPUTDEVICE_I2SOUTPUT. The I2S operation mode supports only Slave mode, I2S format mode, 48000Hz, and Stereo.

1.2.2.7. When using the speaker terminal of the extension board

In the case of line-out using the headphone terminal of the extension board as the audio output destination, as the drive capability setting of Audio signal, Specify AS_SP_DRV_MODE_LINEOUT as the argument sp_drv of setPlayerMode (default).

When outputting from the speaker terminal of the extension board, specify AS_SP_DRV_MODE_4DRIVER in the argument sp_drv.

When using the speaker terminal, the board needs to be modified. For details, refer to the following hardware guide.

MP3 files have ID3v2 TAGs (especially large metadata like image data) will cause parse errors by the decoder.

Please delete the tag information with some tool.

1.2.3. Play WAV music

1.2.3.1. Overview

Play WAV music files on the microSD card.

1.2.3.2. Operating environment
  • Spresense Main & Extension Board

  • microSD card

  • Playback headphones or speakers

This sample uses a microSD card and headphones. Connect headphones or active speakers to the headphone jack on the extension board.

1.2.3.3. Operating procedure
  1. Install the DSP file WAVDEC for WAV decoding to the microSD card (see Install DSP files).

  2. Put on the WAV file you want to play with the file name "Sound.wav" in the root directory of the microSD card.

  3. Insert the microSD card into the microSD card slot on the extension board.

  4. Open the sample sketch by selecting File → Examples → Examples for Spresense / Audio → application → player_wav from Arduino IDE.

    arduino audio examples player wav
  5. Select the COM port of Spresense with Tools → Port and write to the microcomputer board.

  6. When the sketch is uploaded and executed, the WAV file placed on the microSD card will be played via headphone.
    Play the file to the end and stop.

1.2.3.4. Program description
  • Set the clock mode with the setRenderingClockMode() function.

    theAudio->setRenderingClockMode(mode); (1)
    1 mode: Select normal (AS_CLKMODE_NORMAL) or high resolution (AS_CLKMODE_HIRES) or clock mode.
    This sample uses AS_CLKMODE_NORMAL.
    This function only accepts before calling setPlayerMode() or in ReadyMode.
    To switch between normal and high resolution dynamically, call setReadyMode() and transition to the Ready state, then switch AS_CLKMODE_NORMAL or AS_CLKMODE_HIRES.
  • Set the player mode with the setPlayerMode() function.

    theAudio->setPlayerMode(device,    (1)
                             sp_drv);  (2)
    1 device: Select the output device, whether to output to speaker headphones (AS_SETPLAYER_OUTPUTDEVICE_SPHP) or I2S (AS_SETPLAYER_OUTPUTDEVICE_I2SOUTPUT).
    This sample uses AS_SETPLAYER_OUTPUTDEVICE_SPHP.
    When using I2S, refer to How to output to I2S.
    2 sp_drv: Sets the output drive.
    In this sample, specify AS_SP_DRV_MODE_LINEOUT.
    For details, refer to When using the speaker terminal of the extension board.
  • Initialize the player with the initPlayer() function.

    theAudio->initPlayer(id,         (1)
                         codec,      (2)
                         codec_path, (3)
                         fs,         (4)
                         channel);   (5)
    1 id: Select Player ID.
    This sample uses AudioClass::Player0.
    2 codec: Specify WAV (AS_CODECTYPE_WAV) as the codec type.
    3 codec_path: Specify the location where the DSP file is installed.
    Specify "/mnt/sd0/BIN" for microSD card and "/mnt/spif/BIN" for SPI-Flash.
    4 fs: Specify the sampling rate.
    5 channel: Specify monaural (AS_CHANNEL_MONO) or stereo (AS_CHANNEL_STEREO).
  • Write the contents of the WAV file to the decoder with the writeFrames() function.

    theAudio->writeFrames(id,      (1)
                          myFile); (2)
    1 id: Select Player ID.
    This sample uses AudioClass::Player0.
    2 You can change the playback file by specifying myFile in the argument.
    This sample uses a file called "Sound.wav".
    Execute this function periodically to supply the stream to the decoder so that the decoding will not be interrupted. When writing the file is completed, AUDIOLIB_ECODE_FILEEND is returned as the return value of the function.
  • Start the player with the startPlayer() function.

    theAudio->startPlayer(id); (1)
    1 id: Select Player ID.
    This sample uses AudioClass::Player0.
    Call it after writing to the decoder with the writeFrames() function.
  • Stop the player with the stopPlayer() function.

    theAudio->stopPlayer(id,      (1)
                          mode);  (2)
    1 id: Select Player ID.
    This sample uses AudioClass::Player0.
    2 You can choose to stop immediately in mode (AS_STOPPLAYER_NORMAL) or wait until the stream supplied to the decoder has finished playing and then stop (AS_STOPPLAYER_ESEND). If called without the mode argument, it works with AS_STOPPLAYER_NORMAL.
  • Adjust the volume with the setVolume(volume) function.

    theAudio->setVolume(volume); (1)
    1 volume: You can change the volume. (-1020 ~ 120)
    1/10 of the specified value becomes the actual volume [dB]. For example, the volume when 15 is specified, is 1.5 [dB].

1.2.4. Play high resolution audio

1.2.4.1. Overview

Play WAV music files of high-resolution sound source on microSD card.

1.2.4.2. Operating environment
  • Spresense Main & Extension Board

  • microSD card

  • Playback headphones or speakers

This sample uses a microSD card and headphones. Connect headphones or active speakers to the headphone jack on the extension board.

1.2.4.3. Operating procedure

Install the DSP file WAVDEC for WAV decoding to the microSD card (see Install DSP files).

Put on the WAV file you want to play and rename the file name to "HiResSound.wav" in the root directory of the microSD card.

  1. Insert the microSD card into the microSD card slot on the extension board.

  2. Open the sample sketch by selecting File → Examples → Examples for Spresense / Audio → application → player_hires from Arduino IDE.

    arduino audio examples player hires
  3. Select the COM port of Spresense with Tools → Port and write to the microcomputer board.

  4. When the sketch is uploaded and executed, the WAV file of high-resolution sound placed on the microSD card will be played via headphone.
    Play the file to the end and stop.

1.2.4.4. Program description
  • Set the clock mode with the setRenderingClockMode() function.

    theAudio->setRenderingClockMode(mode); (1)
    1 mode: Select normal (AS_CLKMODE_NORMAL) or high resolution (AS_CLKMODE_HIRES) or clock mode.
    This sample uses AS_CLKMODE_NORMAL.
    setPlayerMode() can be called only in ReadyMode.
    To switch between normal and high resolution dynamically, call setReadyMode() and transition to the Ready state, then switch AS_CLKMODE_NORMAL or AS_CLKMODE_HIRES.
  • Set the player mode with the setPlayerMode() function.

    theAudio->setPlayerMode(device,    (1)
                             sp_drv);  (2)
    1 device: Select the output device, whether to output to speaker headphones (AS_SETPLAYER_OUTPUTDEVICE_SPHP) or I2S (AS_SETPLAYER_OUTPUTDEVICE_I2SOUTPUT).
    This sample uses AS_SETPLAYER_OUTPUTDEVICE_SPHP.
    When using I2S, refer to How to output to I2S.
    2 sp_drv: Sets the output drive.
    In this sample, specify AS_SP_DRV_MODE_LINEOUT.
    For details, refer to When using the speaker terminal of the extension board.
  • Initialize the player with the initPlayer() function.

    theAudio->initPlayer(id,         (1)
                         codec,      (2)
                         codec_path, (3)
                         fs,         (4)
                         bitlen,     (5)
                         channel);   (6)
    1 id: Select Player ID.
    This sample uses AudioClass::Player0.
    2 codec: Specify WAV (AS_CODECTYPE_WAV) as the codec type.
    3 codec_path: Specify the location where the DSP file is installed.
    Specify "/mnt/sd0/BIN" for microSD card and "/mnt/spif/BIN" for SPI-Flash.
    4 fs: Specify the sampling rate.
    In this sample, 96kHz (AS_SAMPLINGRATE_96000) is specified.
    5 bitlen: Specify the bit length.
    In this sample, 24bit (AS_BITLENGTH_24) is specified.
    6 channel: Specify monaural (AS_CHANNEL_MONO) or stereo (AS_CHANNEL_STEREO).
  • Write the contents of the WAV file to the decoder with the writeFrames() function.

    theAudio->writeFrames(id,      (1)
                          myFile); (2)
    1 id: Select Player ID.
    This sample uses AudioClass::Player0.
    2 You can change the playback file by specifying myFile in the argument.
    This sample uses a file called "HiResSound.wav".
    Execute this function periodically to supply the stream to the decoder so that the decoding will not be interrupted. When writing the file is completed, AUDIOLIB_ECODE_FILEEND is returned as the return value of the function.
  • Start the player with the startPlayer() function.

    theAudio->startPlayer(id); (1)
    1 id: Select Player ID.
    This sample uses AudioClass::Player0.
    Call it after writing to the decoder with the writeFrames() function.
  • Stop the player with the stopPlayer() function.

    theAudio->stopPlayer(id,      (1)
                          mode);  (2)
    1 id: Select Player ID.
    This sample uses AudioClass::Player0.
    2 You can choose to stop immediately in mode (AS_STOPPLAYER_NORMAL) or wait until the stream supplied to the decoder has finished playing and then stop (AS_STOPPLAYER_ESEND). If called without the mode argument, it works with AS_STOPPLAYER_NORMAL.
  • Adjust the volume with the setVolume(volume) function.

    theAudio->setVolume(volume); (1)
    1 volume: You can change the volume. (-1020 ~ 120)
    1/10 of the specified value becomes the actual volume [dB]. For example, the volume when 15 is specified, is 1.5 [dB].

1.2.5. Play Playlist of music

1.2.5.1. Overview

This sample is a sample sketch for playing music using a playlist.

Plays multiple music files in the microSD card in order. You can perform operations such as play, stop, send to next song, return, repeat play and random play.

1.2.5.2. Operating environment
  • Spresense Main & Extension Board

  • microSD card

  • Playback headphones or speakers

This sample uses a microSD card and headphones. Connect headphones or active speakers to the headphone jack on the extension board.

1.2.5.3. Program description

In order to use the playlist function, it is necessary to create a playlist file on a PC beforehand and copy it to the microSD card. The directory structure of the microSD card used in this sample sketch is shown below.

microSD card root directory
|-- BIN/
|   |-- MP3DEC
|   `-- WAVDEC
|-- AUDIO/
|   |-- Sound1.mp3
|   |-- Sound2.mp3
|   |--   :
|   |-- Sound1.wav
|   |-- Sound2.wav
|   |--   :
`-- PLAYLIST/
    `-- TRACK_DB.CSV
  • Install the MP3DEC and WAVDEC DSP binary files in the BIN/ directory (see Install DSP files).

  • Copy the music content files to play in the playlist to the AUDIO/ directory.

  • Copy the playlist file (TRACK_DB.CSV) to the PLAYLIST/ directory.

How to create a playlist file

The format of the playlist file TRACK_DB.CSV is shown below. It is the text file in CSV (comma-separated values) format and in which the information of each file is written in line units.

[filename],[artist],[album],[channel number],[bit length],[sampling rate],[file format]
[filename],[artist],[album],[channel number],[bit length],[sampling rate],[file format]
[filename],[artist],[album],[channel number],[bit length],[sampling rate],[file format]
 :

Please describe the file name, artist name, album name, and codec information (number of channels, bit length, sampling rate) of the music contents. An example of TRACK_DB.CSV is shown below.

Sound1.mp3,Artist1,Album1,2,16,44100,mp3
Sound2.mp3,Artist1,Album1,2,16,44100,mp3
Sound1.wav,Artist2,Album2,2,16,48000,wav
Sound2.wav,Artist2,Album2,2,24,192000,wav

We also provide a simple script tool, mkplaylist.py, for creating playlist files. Please right-click here and download the file. This script uses ffmpeg-python, so please use it with ffmpeg and ffmpeg-python installed.

The usage of mkplaylist.py is shown below.

python mkplaylist.py
usage: python mkplaylist.py dirname [dirname2] [dirname3] ...
usage: python mkplaylist.py -f filename

Generate an audio playlist file named as TRACK_DB.CSV

Please execute by specifying the directory path containing the music file as an argument. When the execution is completed normally, the TRACK_DB.CSV file will be output to the current directory. If the TRACK_DB.CSV file already exists, it will be added to the end of the file.

python mkplaylist.py MyMusic
  • The extension of the music content file name should be .mp3 or .wav.

  • File names containing double-byte code such as Japanese are not supported. Please change the file name to an ASCII code string.

  • The playlist file is written in CSV format, so please do not use a string containing "," comma in filename, artist and album name.

  1. Open the sample sketch by selecting File → Examples → Examples for Spresense / Audio → application → play_playlist from Arduino IDE.

    arduino audio examples player playlist
  2. Select the COM port of Spresense with Tools → Port and write to the microcomputer board.

  3. When you start the serial monitor, the following menu is displayed. Various operations are performed according to the key input from the serial.

=== MENU (input key ?) ==============
p: play  s: stop  +/-: volume up/down
l: list  n: next  b: back
r: repeat on/off  R: random on/off
a: auto play      m,h,?: menu
=====================================
The EEPROM library is used to save setting information such as volume value, random / repeat / auto play, etc. in non-volatile memory. It is a sample application that operates based on the previously preset information at the next startup. Please refer to it when creating your own music player.

1.2.6. Play dual MP3 music

1.2.6.1. Overview

Play two MP3 music files on microSD card at the same time.

1.2.6.2. Operating environment
  • Spresense Main & Extension Board

  • microSD card

  • Playback headphones or speakers

This sample uses a microSD card and headphones. Connect headphones or active speakers to the headphone jack on the extension board.

1.2.6.3. Operating procedure
  1. Install the DSP file MP3DEC for MP3 decoding to the microSD card (see Install DSP files).

  2. Put the two MP3 files you want to play in the root directory of the microSD card with the file names "Sound0.mp3" and "Sound1.mp3".

  3. Insert the microSD card into the microSD card slot on the extension board.

  4. Open the sample sketch by selecting File → Examples → Examples for Spresense / Audio → application → dual_players from Arduino IDE.

    arduino audio examples dual players
  5. Select the COM port of Spresense with Tools → Port and write to the microcomputer board.

  6. Once the sketch is uploaded and executed, the two MP3 files placed on the microSD card will be mixed and played from the line-out at the same time. Play each file repeatedly.

1.2.6.4. Program description
  • Set the clock mode with the setRenderingClockMode() function.

    theAudio->setRenderingClockMode(mode); (1)
    1 mode: Select normal (AS_CLKMODE_NORMAL) or high resolution (AS_CLKMODE_HIRES) or clock mode.
    For MP3 playback, specify AS_CLKMODE_NORMAL.
  • Set the player mode with the setPlayerMode() function.

    theAudio->setPlayerMode(device,    (1)
                             sp_drv);  (2)
    1 device: Select the output device, whether to output to speaker headphones (AS_SETPLAYER_OUTPUTDEVICE_SPHP) or I2S (AS_SETPLAYER_OUTPUTDEVICE_I2SOUTPUT).
    This sample uses AS_SETPLAYER_OUTPUTDEVICE_SPHP.
    When using I2S, refer to How to output to I2S.
    2 sp_drv: Sets the output drive.
    In this sample, specify AS_SP_DRV_MODE_LINEOUT.
    For details, refer to When using the speaker terminal of the extension board.
  • Initialize the player with the initPlayer() function for players 0 and 1 respectively.

    theAudio->initPlayer(id,         (1)
                         codec,      (2)
                         codec_path, (3)
                         fs,         (4)
                         channel);   (5)
    1 id: Select Player ID.
    One specifies AudioClass::Player0 and the other specifies AudioClass::Player1.
    2 codec: Specify MP3 (AS_CODECTYPE_MP3) as the codec type.
    3 codec_path: Specify the location where the DSP file is installed.
    Specify "/mnt/sd0/BIN" for microSD card and "/mnt/spif/BIN" for SPI-Flash.
    4 fs: Specify the sampling rate.
    MP3 supports 32kHz (AS_SAMPLINGRATE_32000), 44.1kHz (AS_SAMPLINGRATE_44100), 48kHz (AS_SAMPLINGRATE_48000). If you specify AS_SAMPLINGRATE_AUTO, the sampling rate is automatically determined.
    5 channel: Specify monaural (AS_CHANNEL_MONO) or stereo (AS_CHANNEL_STEREO).
  • Write the MP3 file contents to the decoder with the writeFrames() function for players 0 and 1, respectively.

    theAudio->writeFrames(id,       (1)
                           myFile); (2)
    1 id: Select Player ID.
    One specifies AudioClass::Player0 and the other specifies AudioClass::Player1.
    2 You can change the playback file by specifying myFile in the argument.
    This sample uses files called "Sound0.mp3" and "Sound1.mp3".
    Execute this function periodically to supply the stream to the decoder so that the decoding will not be interrupted. When writing the file is completed, AUDIOLIB_ECODE_FILEEND is returned as the return value of the function.
  • Start the player with the startPlayer() function for each of players 0 and 1.

    theAudio->startPlayer(id); (1)
    1 id: Select Player ID.
    One specifies AudioClass::Player0 and the other specifies AudioClass::Player1.
    Call it after writing to the decoder with the writeFrames() function.
  • Stop the player with the stopPlayer() function for each of players 0 and 1.

    theAudio->stopPlayer(id,      (1)
                          mode);  (2)
    1 id: Select Player ID.
    One specifies AudioClass::Player0 and the other specifies AudioClass::Player1.
    2 You can choose to stop immediately in mode (AS_STOPPLAYER_NORMAL) or wait until the stream supplied to the decoder has finished playing and then stop (AS_STOPPLAYER_ESEND). If called without the mode argument, it works with AS_STOPPLAYER_NORMAL.
  • Adjust the volume with the setVolume(master, player0, player1) function.

    theAudio->setVolume(master,   (1)
                        player0,  (2)
                        player1); (3)
    1 master: You can change the volume for both players 0 and 1. (-1020 ~ 120)
    2 You can change the volume for player 0 with player0. (-1020 ~ 120)
    3 You can change the volume for player 1 with player1. (-1020 ~ 120)
    1/10 of the specified value becomes the actual volume [dB]. For example, the volume when 15 is specified, is 1.5 [dB]. You can adjust the overall volume with master and the individual volume with player0 and player1.

1.2.7. Listen to the sound input from the microphone

1.2.7.1. Overview

You can hear the sound from the microphone from the headphone jack by using the audio path setting function of Audio HW.

For example, to output digital audio input from I2S to analog, or such as outputting the input from the microphone to I2S it is possible. This function supports an output mechanism that saves power and has very low latency.

1.2.7.2. Operating environment
  • Spresense Main & Extension Board

  • Playback headphones or speakers

  • Recording microphone

In this sample, headphones and a microphone are used. Connect headphones or active speakers to the headphone jack on the extension board. Please refer to the following hardware guide for microphone connection.

In this sample, it is not necessary to install the DSP file.

1.2.7.3. Operating procedure
  1. Open the sample sketch by selecting File → Examples → Examples for Spresense Audio → application → through from Arduino IDE.

    arduino audio examples through
  2. The sample sketch outputs the sound input from I2S to the headphone terminal.
    Here, change the sketch so that the sound input from the microphone is output to the headphone jack.
    Save it in an editable location by doing File → Save As.

  3. Change the argument input of the setThroughMode() function from AudioClass::I2sIn to AudioClass::MicIn and save.

  4. Select the COM port of Spresense with Tools → Port and write to the microcomputer board.

  5. Once the sketch is uploaded and executed, you will hear the sound coming from the microphone in your headphones.

1.2.7.4. Program description
  • To set the through mode, use the setThroughMode() function.

    theAudio->setThroughMode(input,      (1)
                             i2s_out,    (2)
                             sp_out,     (3)
                             input_gain, (4)
                             sp_drv);    (5)
    1 Select the input system with input .
    2 Select the source to output to I2S with i2s_out .
    3 Select whether to output to the headphone speaker with sp_out .
    For example, to output the audio input from the microphone to I2S, use
    Specify AudioClass::MicIn for input ,
    Specify AudioClass::Mic in i2s_out ,
    Specify false in sp_out .
    4 Set the microphone gain with input_gain.
    The range is 0 to 21 [dB] for analog microphones and -78.5 to 0 [dB] for digital microphones.
    For an analog microphone, specify an integer value multiplied by 10 for input_gain, for example, 100 when setting 10 [dB].
    For a digital microphone, specify an integer value multiplied by 100 for input_gain, for example -5 when setting -0.05 [dB].
    5 For sp_drv, refer to When using the speaker terminal of the extension board.

1.2.8. Record in MP3 format

1.2.8.1. Overview

The sound from the microphone is recorded as an MP3 file on the microSD card.

1.2.8.2. Operating environment
  • Spresense Main & Extension Board

  • Playback headphones or speakers

  • Recording microphone

In this sample, headphones and a microphone are used. Connect headphones or active speakers to the headphone jack on the extension board. Please also refer to the following hardware guide for microphone connection.

1.2.8.3. Operating procedure
  1. Install the DSP file MP3ENC for MP3 encoding to the microSD card (see Install DSP files).

  2. Open the sample sketch by selecting File → Examples →Examples for Spresense Audio → application → recorder on the Arduino IDE.

    arduino audio examples recorder
  3. Select the COM port of Spresense with Tools → Port and write to the microcomputer board.

  4. Once the sketch is uploaded and executed, recording will start and stop recording after a certain period of time.

  5. The recorded data is saved in the microSD card with the file name "Sound.mp3".
    Take out the microSD card and check the data recorded on the PC.

1.2.8.4. Program description
  • Set the recorder mode with the setRecorderMode() function.

    theAudio->setRecorderMode(input_device, (1)
                              input_gain,   (2)
                              bufsize,      (3)
                              is_digital);  (4)
    1 input_device : Input from the microphone ( AS_SETRECDR_STS_INPUTDEVICE_MIC ) as the input device.
    2 Set the microphone gain with input_gain.
    The range is 0 to 21 [dB] for analog microphones and -78.5 to 0 [dB] for digital microphones.
    For an analog microphone, specify an integer value multiplied by 10 for input_gain, for example, 100 when setting 10 [dB].
    For a digital microphone, specify an integer value multiplied by 100 for input_gain, for example -5 when setting -0.05 [dB].
    3 bufsize : Specify the buffer size of the recorder.
    For details, refer to Recorder buffer size.
    4 If the connected microphone is a digital microphone, specify true in the argument is_digital .
  • Initialize the recorder with the initRecorder() function.

    theAudio->initRecorder(codec,      (1)
                           codec_path, (2)
                           fs,         (3)
                           channel);   (4)
    1 Specify MP3 ( AS_CODECTYPE_MP3 ) as the codec type of codec .
    2 Specify the location where the DSP file is installed with codec_path .
    Specify "/mnt/sd0/BIN" for microSD card and "/mnt/spif/BIN" for SPI-Flash.
    3 Specify the sampling rate with fs .
    MP3 supports 32kHz ( AS_SAMPLINGRATE_32000 ), 44.1kHz ( AS_SAMPLINGRATE_44100 ), 48kHz ( AS_SAMPLINGRATE_48000 ).
    4 channel : Specify monaural ( AS_CHANNEL_MONO ) or stereo ( AS_CHANNEL_STEREO ).
  • Read MP3 encoded data for recording to a file with the readFrames() function.

    theAudio->readFrames(myFile); (1)
    1 The file to be recorded can be changed with the specified myFile.
    This sample write to a file called "Sound.mp3".
    Execute this function periodically to read the encoding result so that the encoding buffer does not overflow.
  • Start the recorder with the startRecorder() function.

  • Stop the recorder with the stopRecorder() function.

1.2.8.5. Recorder buffer size

The default buffer size used for writing to the microSD card is 160kByte. This size is a size that can write high resolution WAV audio enough, but this memory size is too large for playing a 48 kHz MP3 file, so memory is wasted.

Therefore, when recording a 48kHz MP3 file, You can change the buffer size by setRecorderMode. As a guide, the default 160 kByte is when playing a 192 kHz high resolution WAV file, and when recording a 48 kHz MP3 file, it is recommended to set it to about 8 kByte.

Regarding size, it is a guide only. Adjust according to the performance of the microSD card and the processing amount of the application. If you make the buffer too small, you will get errors when writing to the microSD card. If overflow occurs, increase the buffer size.
Even if audio is obtained, it may not be possible to write all the data due to factors such as the writing performance of the microSD card and the applications that are running. In the case of the current extension board, the limit is 8ch/48kHz or 2ch/192kHz. It also depends on the speed class of the microSD card, so please use the one with the highest transfer rate possible.

1.2.9. Record in WAV format

1.2.9.1. Overview

The sound from the microphone is recorded as a WAV file on the microSD card.

1.2.9.2. Operating environment
  • Spresense Main & Extension Board

  • Playback headphones or speakers

  • Recording microphone

In this sample, headphones and a microphone are used. Connect headphones or active speakers to the headphone jack on the extension board. Please refer to the following hardware guide for microphone connection.

1.2.9.3. Operating procedure
  1. Install the DSP file SRC for WAV(PCM) encoding to the microSD card (see Install DSP files).

  2. Open the sample sketch by selecting File → Examples →Examples for Spresense Audio → application → recorder_wav on the Arduino IDE.

    arduino audio examples recorder wav
  3. Select the COM port of Spresense with Tools → Port and write to the microcomputer board.

  4. Once the sketch is uploaded and executed, recording will start and stop recording after a certain period of time.

  5. The recorded data is saved in the microSD card with the file name "Sound.wav".
    Take out the microSD card and check the data recorded on the PC.

1.2.9.4. Program description
  • Set the recorder mode with the setRecorderMode() function.

    theAudio->setRecorderMode(input_device, (1)
                              input_gain,   (2)
                              bufsize,      (3)
                              is_digital);  (4)
    1 input_device : Input from the microphone ( AS_SETRECDR_STS_INPUTDEVICE_MIC ) as the input device.
    2 Set the microphone gain with input_gain.
    The range is 0 to 21 [dB] for analog microphones and -78.5 to 0 [dB] for digital microphones.
    For an analog microphone, specify an integer value multiplied by 10 for input_gain, for example, 100 when setting 10 [dB].
    For a digital microphone, specify an integer value multiplied by 100 for input_gain, for example -5 when setting -0.05 [dB].
    3 bufsize : Specify the buffer size of the recorder.
    For details, refer to Recoder buffer size.
    4 If the connected microphone is a digital microphone, specify true in the argument is_digital .
  • Initialize the recorder with the initRecorder() function.

    theAudio->initRecorder(codec,      (1)
                           codec_path, (2)
                           fs,         (3)
                           channel);   (4)
    1 Specify WAV ( AS_CODECTYPE_WAV ) as the codec type of codec .
    2 Specify the location where the DSP file is installed with codec_path .
    Specify "/mnt/sd0/BIN" for microSD card and "/mnt/spif/BIN" for SPI-Flash.
    3 Specify the sampling rate with fs .
    4 channel : Specify the number of channels(upto 8 channels).
When recording at high resolution (192kHz, 24bit), fs is AS_SAMPLINGRATE_192000 and bitlen is AS_BITLENGTH_24.

+

theAudio->initRecorder(codec,
                       codec_path,
                       fs,
                       bitlen,
                       channel);
  • When recording a WAV file, call the writeWavHeader() function to create a WAV header before executing the audio recording.

    theAudio->writeWavHeader(myFile); (1)
    1 Specify the file to record.
    This sample uses a file called "Sound.wav".
  • The readFrames() function reads WAV (PCM) encoded data for recording to a file.

    theAudio->readFrames(myFile); (1)
    1 You can change the file to be recorded by specifying myFile.
    This sample uses a file called "Sound.wav".
    Please execute this function periodically to read the encoding result so that the encoding buffer does not overflow.
1.2.9.5. About recorder buffer size
1.2.9.6. If recording fails

Depending on the state of the microSD, the manufacturer, etc., recording may fail even with the default buffer size. In that case, secure a larger buffer size for writing.

You can change the buffer size used with setRecorderMode . When performing 8-channel recording, high-resolution stereo recording, etc., increasing the size to about 500 kB will result in fairly stable recording operation.

However, in that case, the memory area for MainCore will be insufficient, so select Tools→ Memory→ 1024KB from the Arduino IDE to secure the memory area.

example.)

/* Select input device as microphone */
  theAudio->setRecorderMode(AS_SETRECDR_STS_INPUTDEVICE_MIC);

/* Select input device as microphone */
  theAudio->setRecorderMode(AS_SETRECDR_STS_INPUTDEVICE_MIC, "microphone gain" ,(500*1024));

1.2.10. Retrieve PCM data

1.2.10.1. Overview

This example captures PCM data input from the microphone.

In this sample, only the beginning of the PCM data is displayed, by using this sample, it is possible to perform frequency analysis processing of PCM raw data and signal processing such as various filters.

1.2.10.2. Operating environment
  • Spresense Main & Extension Board

  • Recording microphone

In this sample, microphones are used. Please refer to the following hardware guide for microphone connection.

If you want to reconfigure the microphone, see Setting of "MIC channel select map".
If you want to build for SDK and integration into the Arduino IDE, please refer to: https://github.com/sonydevworld/spresense-arduino-compatible/blob/master/README.md [How to prepare Arduino environment] .

In this sample, it is not necessary to install the DSP file.

For sound capture and analytics, only 48kHz or 192kHz is supported.
1.2.10.3. Operating procedure
  1. Open the sample sketch by selecting File → Examples →Examples for Spresense Audio → application → pcm_capture on the Arduino IDE.

    arduino audio examples pcm capture
  2. Select the COM port of Spresense with Tools → Port and write to the microcomputer board.

  3. When you start the serial monitor, the PCM data of the audio input from the microphone is displayed.

1.2.10.4. Program description
  • Set the recorder mode with the setRecorderMode() function.

    theAudio-> setRecorderMode(input_device); (1)
    1 input_device : Specify the input from the microphone ( AS_SETRECDR_STS_INPUTDEVICE_MIC ) as the input device.
The microphones gain setting is the default value of 0[db] and the default analog microphones setting.
  • Initialize the recorder with the initRecorder() function.

    theAudio->initRecorder(codec,      (1)
                           codec_path, (2)
                           fs,         (3)
                           channel);   (4)
    1 Specify WAV ( AS_CODECTYPE_PCM ) as the codec type of codec .
    2 Specify the location where the DSP file is installed with codec_path .
    Specify "/mnt/sd0/BIN" for microSD card and "/mnt/spif/BIN" for SPI-Flash.
    3 Specify the sampling rate with fs .
    4 channel : Specify the number of channels(upto 8 channels).
When recording at high resolution (192kHz, 24bit), fs is AS_SAMPLINGRATE_192000 and bitlen is AS_BITLENGTH_24.

+

theAudio->initRecorder(codec,
                       codec_path,
                       fs,
                       bitlen,
                       channel);
  • The readFrames() function reads PCM data for analyzing.

    theAudio-> readFrames(p_buffer, (1)
                          buffer_size, (2)
                          read_size); (3)
    1 Read the PCM data to p_buffer.
    2 Specify the read size.
    3 The size actually read is returned.
    you should reads the PCM data by running this function on a regular basis to prevent the capture buffer from overflowing.
1.2.10.5. About the read size of readFrames

Even if the read buffer size is specified in readFrames, if there is data that can be read internally, only the size of the data will be read. The internal processing is performed by the 768 sample, and even if the buffer size is specified as 2048 , if there is data of the 768 size, the 768 sample will be read.

If you want to read the data up to the buffer size, repeat readFrames until you write the rest, or wait long enough for the data to be ready.

1.2.11. Repeat recording and playback

1.2.11.1. Overview
1.2.11.2. Operating environment
  • Spresense Main & Extension Board

  • microSD card

  • Playback headphones or speakers

  • Recording microphone

In this sample, headphones and a microphone are used. Connect headphones or active speakers to the headphone jack on the extension board. And please refer to the following hardware guide for microphone connection.

1.2.11.3. Operating procedure
  1. Install the DSP file MP3ENC and MP3DEC for MP3 encoding and decoding to the microSD card (see Install DSP files).

  2. Open the sample sketch by selecting File → Examples →Examples for Spresense Audio → application → recorder on the Arduino IDE.

    arduino audio examples rec play
  3. Select the COM port of Spresense with Tools → Port and write to the microcomputer board.

  4. When the sketch is uploaded and executed, the sound from the microphone is recorded and this sound is played back in a cycle of about 20 seconds.
    Record and play 5 times before exiting.

1.2.11.4. Program description
  • The recording and playback functions are the same as other stand-alone samples.

  • By calling the setRecorderMode(), setReadyMode(), and setPlayerMode() functions to switch modes, you can dynamically switch between recording and playback operations.

1.3. Object interface layer

Spresense Audio library has a layer structure as described in Audio Library Development Guide.
The object interface layer allows you to achieve combinations of functions that cannot be achieved with the high-level interface layer.

These samples of this object interface layer are description below.

1.3.1. Play MP3 music

1.3.1.1. Overview

Play MP3 music files on the microSD card.
The function of this sample by the object level interface is the same as the sample Play MP3 music by the high level interface.

1.3.1.2. Operating environment and procedure
  1. Open the sample sketch by selecting File → Examples → Examples for Spresense / Audio → application → player_objif from Arduino IDE.

    For details on the operating environment and operating procedure, please refer to << _mp3_player, Play MP3 music>>.

1.3.1.3. Program description
  • Specify by createStaticPools() the layout of shared memory.

       initMemoryPools ();
       createStaticPools (no); (1)
    1 no: Specify the layout number. In this sample, specify the layout for the playback function (MEM_LAYOUT_PLAYER).
  • Specify the clock mode for rendering by setRenderingClockMode().

    theMixer->setRenderingClkMode(mode); (1)
    1 mode: Select clock mode of the normal mode ( OUTPUTMIXER_RNDCLK_NORMAL ) or high resolution mode ( OUTPUTMIXER_RNDCLK_HIRES ).
    For MP3 playback, specify OUTPUTMIXER_RNDCLK_NORMAL.
Set the HW operating clock before the Mixer HW is powered on, that is, before theMixer→begin().
  • Generate and initialize each Audio library.

    theMixer->begin();                              (1)
    thePlayer->begin();                             (2)
    thePlayer->create(                              (3)
                        id,                         (4)
                        attention_cb);              (5)
    theMixer->create(                               (6)
                        attention_cb);              (5)
    thePlayer->activate(                            (7)
                        id,                         (4)
                        mediaplayer_done_callback); (8)
    theMixer->activate(                             (9)
                        OutputMixer0,               (10)
                        HPOutputDevice,             (11)
                        outputmixer_done_callback); (12)
    1 theMixer→begin
    Initializes the Audio HW block of Mixer and turns on the power.
    (Previously theMixer→activateBaseband.)
    2 thePlayer→begin
    Initialize the Player.
    (It works even if you do not call it to maintain compatibility.)
    3 thePlayer→create
    Generates and initializes the Player software module.
    4 id: Specify the ID of the Player. With Spresense, you can play up to two at the same time. You can specify MediaPlayer::Player0 or MediaPlayer::Player1.
    5 attention_cb: Callback for error or warning notified from each module. Please implement error handling.
    6 theMixer→create
    Generates and initializes the Mixer software module.
    7 thePlayer→activate
    Set the playback operation mode of Player
    8 mediaplayer_done_callback: Callback for error or warning notified from each module. Please implement error handling.
    9 theMixer→activate
    Set the sounding operation mode of Mixer
    10 OutputMixer0: Specify the ID of the Mixer. Please use the same ID as Player.
    11 HPOutputDevice: Set the output destination. HPOutputDevice is an analog output. I2SOutputDevice is the I2S output.
    12 outputmixer_done_callback: Callback for result notification when an event is issued to Mixer.
  • Initialize the Player library.

    thePlayer->initPlayer(id,        (1)
                         codec,      (2)
                         codec_path, (3)
                         fs,         (4)
                         channel);   (5)
    1 id: Select Player ID.
    This sample uses AudioClass::Player0.
    2 codec: Specify MP3 (AS_CODECTYPE_MP3) as the codec type.
    3 codec_path: Specify the location where the DSP file is installed.
    Specify "/ mnt / sd0 / BIN" for a microSD card and "/ mnt / spif / BIN" for SPI-Flash.
    4 fs: Specify the sampling rate.
    MP3 supports 32kHz (AS_SAMPLINGRATE_32000), 44.1kHz (AS_SAMPLINGRATE_44100), 48kHz (AS_SAMPLINGRATE_48000). AS_SAMPLINGRATE_AUTO If specified, the sampling rate will be determined automatically.
    5 channel: Specify monaural (AS_CHANNEL_MONO) or stereo (AS_CHANNEL_STEREO).
  • Use writeFrames() to write the stream of the MP3 file to the decoder.

    thePlayer->writeFrames(id,     (1)
                          myFile); (2)
    1 id: Select Player ID.
    This sample uses AudioClass::Player0.
    2 You can change the playback file by specifying myFile as an argument.
    This sample uses a file called "Sound.mp3".
    Execute this function periodically to feed the stream to the decoder so that the decoding is uninterrupted. When it is written the end of the file, AUDIOLIB_ECODE_FILEEND is returned as the return value.
  • Adjust the volume with setVolume(volume).

    thePlayer->setVolume(volume); (1)
    1 volume: You can change the volume. (-1020 to 120)
    1/10 of the specified value becomes the actual volume [dB]. For example, if you specify 15, the volume is 1.5 [dB].
  • Start the player with the startPlayer().

    thePlayer->startPlayer(id,   (1)
                           mediaplayer_decode_callback (2)
                          );
    1 id: Select Player ID.
    This sample uses AudioClass::Player0.
    Write to the decoder with the writeFrames() function before calling.
    2 mediaplayer_decode_callback: Decode This is a callback that is called in frame units. You need to transfer the data (AsPcmDataParam) to Mixer in the callback. Please implement the code below.
    theMixer->sendData (OutputMixer0,
                        outmixer_send_callback,
                        pcm_param);
On this callback, it is possible to process the PCM that will output before transferring to Mixer.
  • Stop the player with the stopPlayer().

    thePlayer->stopPlayer(id,    (1)
                          mode); (2)
    1 id: Select Player ID.
    This sample uses AudioClass::Player0.
    2 You can choose to stop immediately with mode (AS_STOPPLAYER_NORMAL) or wait until the stream supplied to the decoder finishes playing and then stop (AS_STOPPLAYER_ESEND). When called without the mode argument, it works with AS_STOPPLAYER_NORMAL.
1.3.1.4. Player buffer size

See Play MP3 Music for more information.

1.3.1.5. How to output to I2S

See Play MP3 Music for more information.

1.3.1.6. When using the speaker terminal of the extension board

See Play MP3 Music for more information.

1.3.2. Play dual MP3 music

1.3.2.1. Overview

Play two MP3 music files on microSD card at the same time. The function of this sample by the object level interface is the same as the sample Play dual MP3 music by the high level interface.

1.3.2.2. Operating environment and procedure

Open the sample sketch by selecting File → Examples → Examples for Spresense / Audio → application → dual_players_objif from Arduino IDE.

+ For details on the operating environment and operating procedure, please refer to Play dual MP3 music.

1.3.2.3. Program description

This sample plays MP3 and WAV files, and each player needs to supply Audio ES data independently, in order to avoid effecting each other’s operations.

See Play MP3 Music for the following details of each.

  • Set the layout of shared memory.

  • Initialize each Audio library.

  • Set the clock mode.

  • Initialize the Player library.

  • Write the contents of the MP3/WAV file to the decoder.

  • Start the player.

  • Stop the player.

The clock mode is a setting common to both players. the different resolutions cannot be set.
  • Adjust the volume with setVolume(master, player0, player1).

    thePlayer->setVolume(master,   (1)
                         player0,  (2)
                         player1); (3)
    1 master: You can change the volume for both players 0 and 1. (-1020 ~ 120)
    2 player0: You can change the volume for player 0. (-1020 ~ 120)
    3 player1: You can change the volume for player 1. (-1020 ~ 120)
    1/10 of the specified value becomes the actual volume [dB]. For example, if you specify 15, the volume is 1.5 [dB].

    You can adjust the overall volume with master and the individual volume with player0 and player1.

1.3.3. Low delay Rendering

1.3.3.1. Overview

This is a sample of low-latency playback(Rendering) for cases where you want to make some sounds at high speed for some input, such as a switch push sound, a game, or a musical instrument. Plays audio files (RAW fill only) on the microSD card with low latency.

In order to try low-latency playback, it is designed to be rendering sounds according to the characters entered in the shell. Moreover, in order to support low delay, when it takes time to read an audio file, it is complemented with silence so that playback does not stop.

1.3.3.2. Operating environment
  • Spresense Main & Extension Board

  • microSD card

  • Playback headphones or speakers

In this sample, headphones and a microphone are used. Connect headphones or active speakers to the headphone jack on the extension board.

1.3.3.3. Operating procedure
  1. Save the audio PCM (RAW) files you want to play with the filenames "sound0.raw", "sound1.raw", and "sound2.raw" in the root directory of the microSD card.

    RAW files are PCM files that do not have headers such as WAV headers. If the bit length is 16 bits, the channel data will be interleaved every 16 bits.
    Basically, the read speed from the SD card is the speed-determining speed of this sample, so currently, only 16 bits are supported as shown below.
    If you have no choice but to use 24-bit data, create a 24-bit RAW in a channel-interleaved form by aligning the data in the lower 24 bits with alignment every 32 bits.
  2. Insert the microSD card into the microSD card slot on the extension board.

  3. Open the sample sketch by selecting File → Examples → Examples for Spresense / Audio → application → rendering_objif from Arduino IDE.

  4. Select the COM port of Spresense with Tools → Port and write to the microcomputer board.

  5. Once the sketch is uploaded and executed, it will be waiting for input from the serial monitor. At this time, if you enter "p\n", "Sound0.raw" will be played, if you enter "o\n", "Sound1.raw" will be played, and if you enter "i\n", "Sound2.raw" will be played. Playback will stop at "s\n".

During audio playback, new playback will start as soon as a new playback instruction is received.
1.3.3.4. Program description

SD card read speed is the bottleneck for low latency playback. For this reason, a ring buffer is used to buffer reads from the SD card. In addition, in order to have robustness against read delay under low delay, silence is inserted when read data is missing, so that the sound itself is not stopped.

  • BridgeBuffer class

    This class is for buffering RAW data which we read from the SD card.
    writebuf(File& file, uint32_t size) writes size bytes of data from the file specified by the file object in the SD card to the buffer.
    readbuf(uint8_t * dst) read the data which size is indicated by the return value to the address indicated by dst. Clear the buffer with clearbuf().

  • Specify by createStaticPools() the layout of shared memory.

       initMemoryPools ();
       createStaticPools (no); (1)
    1 no: Specify the layout number. In this sample, specify the layout for the playback function (MEM_LAYOUT_PLAYER).
  • Specify the clock mode for rendering by setRenderingClockMode().

    theMixer->setRenderingClkMode(mode); (1)
    1 mode: Select clock mode of the normal mode ( OUTPUTMIXER_RNDCLK_NORMAL ) or high resolution mode ( OUTPUTMIXER_RNDCLK_HIRES ).
    For low delay Rendering, specify OUTPUTMIXER_RNDCLK_NORMAL.
Set the HW operating clock before the Mixer HW is powered on, that is, before theMixer→begin().
  • Generate and initialize each Mixer library.

    theMixer->begin();                              (1)
    theMixer->create(                               (2)
                        attention_cb);              (3)
    theMixer->activate(                             (4)
                        OutputMixer0,               (5)
                        HPOutputDevice,             (6)
                        outputmixer_done_callback); (7)
    1 theMixer→begin
    Initializes the Audio HW block of Mixer and turns on the power. (Previously theMixer→activateBaseband.)
    2 theMixer→create
    Generates and initializes the Mixer software module.
    3 attention_cb: Callback for error or warning notified from each module. Please implement error handling.
    4 theMixer→activate
    Set the sounding operation modes of Mixer
    5 OutputMixer0: Specify the ID of the Mixer.
    6 HPOutputDevice: Set the output destination. HPOutputDevice is an analog output. I2SOutputDevice is the I2S output.
    7 outputmixer_done_callback: Callback for result notification when an event is issued to Mixer.
  • Adjust the volume with setVolume(volume).

    thePlayer->setVolume(volume); (1)
    1 volume: You can change the volume. (-1020 to 120)
    1/10 of the specified value becomes the actual volume [dB]. For example, if you specify 15, the volume is 1.5 [dB].
Mixer does not have a special event to start. It starts when data (AsPcmDataParam) is sent to Mixer for 3 frames or more.
Mixer does not have a special event for termination. If you set bool is_end in the data (AsPcmDataParam) to be sent to Mixer to ture and send it, the frame will be rendered and the process will end. And the last frame is fade out.
  • About the implementation in outmixer_send_callback outmixer_send_callback is called after one frame has been rendered. In this, the following rendering frame must be supplied to Mixer with the following code.

    theMixer->sendData(OutputMixer0,
                       outmixer_send_callback,
                       pcm_param);

    It is necessary to assign the data to each parameter of the data (AsPcmDataParam pcm_param).

    static bool getFrame(AsPcmDataParam * pcm) {

    In this sample, the assignment process is performed in the above function. For details on the data structure, refer to the API specification in the SDK development document.

    In addition, if the data cannot be read, it is filled with 0 and the pronunciation is continued in this sample.

         if (! getFrame(& pcm_param)) {       : Read from Fifo here
           break;
         }
    
         / * Send PCM * /
         pcm_param.is_end = false;
         pcm_param.is_valid = true;
         if (pcm_param.size == 0) {           : If it cannot be read, the size is 0.
           pcm_param.size = READSIZE;
           pcm_param.sample = pcm_param.size / BYTEWIDTH / CHNUM;
           memset (pcm_param.mh.getPa(), 0, pcm_param.size);         : If it cannot be read, fill in 0 and output.
         }
1.3.3.5. About frame size and buffer size

In this sample, the frame size is READSAMPLE (240). It cannot be set to smaller than this Also, the larger the value, the larger the delay amount.
The buffer size is set to 30 frames at m_buf[30 * READSIZE] in BridgeBuffer. Please adjust accordingly.

It’s support Stereo and 48kHz only, because the playback is performed without internal signal processing to reduce the delay.

1.3.4. Record in MP3 format

1.3.4.1. Overview

The sound from the microphone is recorded as an MP3 file on the microSD card. The function of this sample by the object level interface is the same as the sample Record in MP3 format by the high level interface.

1.3.4.2. Operating environment and procedure
  1. Open the sample sketch by selecting File → Examples → Examples for Spresense / Audio → application → recorder_objif from Arduino IDE.

    For details on the operating environment and operating procedure, please refer to Record in MP3 format.

1.3.4.3. Program description
  • Specify by createStaticPools() the layout of shared memory.

       initMemoryPools ();
       createStaticPools (no); (1)
    1 no: Specify the layout number. In this sample, specify the layout for the recording function (MEM_LAYOUT_RECORDER).
  • Generate and initialize the Recoder library.

    theRecorder->begin(                         (1)
                       attention_cb);           (2)
    theRecorder->activate(                      (3)
                          dev,                  (4)
                          done_callback);       (5)
    1 theRecorder→ begin
    Generates the Recorder software module.
    2 attention_cb: Callback for error warning notified from each module. Please implement error handling.
    3 theRecorder→ activate
    Set the recording operation mode of Recorder.
    4 dev: Set the input device. It can set only microphone devices (AS_SETRECDR_STS_INPUTDEVICE_MIC) now.
    5 done_callback: Callback for result notification when an event is issued to Recorder.
  • Set the recording clock mode with the setCapturingClkMode().

    theRecorder-> setCapturingClkMode(mode); (1)
    1 mode: Select clock mode, normal mode (MEDIARECORDER_CAPCLK_NORMAL), or high resolution mode (MEDIARECORDER_CAPCLK_HIRES).
    For MP3 recording, specify MEDIARECORDER_CAPCLK_NORMAL.
  • Initialize the recorder with the initRecorder().

    theRecorder->init(codec,      (1)
                      channel,    (2)
                      fs,         (3)
                      bits,       (4)
                      bitrate,    (5)
                      path);      (6)
    1 codec: Specify codec type. It is set MP3 (AS_CODECTYPE_MP3).
    2 channel: Specify the number of channels. You can specify 1 or 2.
    3 fs: Specify the sampling rate.
    it support 48kHz (48000) for MP3.
    4 bits: Specify the bit length of the audio data. Specify 16 for MP3.
    5 bitrare: Specify the bit rate of ES data generated by compression. It is set to 96000 in this sample.
    6 path : Specify the location where the DSP files are installed.
    Specify "/mnt/sd0/BIN" for a microSD card and "/mnt/spif/BIN" for SPI-Flash.
  • Read and encode data to MP3 for recording to a file with the readFrames().

    theRecorder->readFrames(ptr,          (1)
                            buffer_size,  (2)
                            size);        (3)
    1 ptr: Specify the start address of the buffer for writing the generated audio stream.
    2 buffer_size: Specify the size of the buffer for writing the generated audio stream.
    3 size: The size of the data actually written out is stored.
    If size is 0, there is no data can read.
  • Start the recorder with the start(). ++

theRecorder->start();
  • Stop the recorder with the stop().

    theRecorder->stop();
1.3.4.4. About recorder buffer size
1.3.4.5. About signal processing

When the ES data write, signal_process(uint32_t size) is called back and the written data can be change. Do not change the data for MP3.

1.3.5. Record in WAV format

1.3.5.1. Overview

The sound from the microphone is recorded as a WAV file on the microSD card. The function of this sample by the object level interface is the same as the sample Record in WAV format by the high level interface.

1.3.5.2. Operating environment and procedure
  1. Open the sample sketch by selecting File → Examples → Examples for Spresense / Audio → application → recorder_wav_objif from Arduino IDE.

    For details on the operating environment and operating procedure, please refer to Record in WAV format.

1.3.5.3. Program description
  • Specify by createStaticPools() the layout of shared memory.

       initMemoryPools ();
       createStaticPools (no); (1)
    1 no: Specify the layout number. In this sample, specify the layout for the recording function (MEM_LAYOUT_RECORDER).
  • Generate and initialize the Recorder library.

    theRecorder->begin(                         (1)
                       attention_cb);           (2)
    theRecorder->activate(                      (3)
                          dev,                  (4)
                          done_callback);       (5)
    1 theRecorder→ begin
    Generates the Recorder software module.
    2 attention_cb: Callback for error warning notified from each module. Please implement error handling.
    3 theRecorder→ activate
    Set the recording operation mode of Recorder.
    4 dev: Set the input device. It can set only microphone devices (AS_SETRECDR_STS_INPUTDEVICE_MIC) now.
    5 done_callback: Callback for result notification when an event is issued to Recorder.
  • Set the recording clock mode with the setCapturingClkMode().

    theRecorder-> setCapturingClkMode(mode); (1)
    1 mode: Select clock mode, normal mode (MEDIARECORDER_CAPCLK_NORMAL), or high resolution mode (MEDIARECORDER_CAPCLK_HIRES).
    Specify MEDIARECORDER_CAPCLK_NORMAL for recording at 48kHz or 16kHz, and specify MEDIARECORDER_CAPCLK_HIRES for recording at 192kHz.
  • Initialize the recorder with the initRecorder().

    theRecorder->init(codec,      (1)
                      channel,    (2)
                      fs,         (3)
                      bits,       (4)
                      bitrate,    (5)
                      path);      (6)
    1 codec: Specify codec type. It is set WAV (AS_CODECTYPE_WAV).
    2 channel: Specify the number of channels. You can specify 1, 2, 4, or 8.
    3 fs: Specify the sampling rate.
    it support 16kHz (16000), 48kHz (48000), or 192kHz (192000) for WAV.
    4 bits: Specify the bit length of the audio data. Specify 16 or 24 for WAV.
    5 bitrare: In the case of WAV, only uncompressed (PCM) is supported, so the bit rate is Don’t care.
    6 path : Specify the location where the DSP files are installed.
    Specify "/mnt/sd0/BIN" for a microSD card and "/mnt/spif/BIN" for SPI-Flash.
  • Read the PCM data for recording to a file with the readFrames().

    theRecorder->readFrames(ptr,          (1)
                            buffer_size,  (2)
                            size);        (3)
    1 ptr: Specify the start address of the buffer for writing the PCM data.
    2 buffer_size: Specify the size of the buffer for writing the PCM data.
    3 size: The size of the data actually written out is stored.
    If size is 0, there is no data can read.
  • Start the recorder with the start(). ++

theRecorder->start();
  • Stop the recorder with the stop().

    theRecorder->stop();
1.3.5.4. About recorder buffer size
1.3.5.5. About the read size of readFrames
1.3.5.6. About signal processing

It is possible to process the write data by calling signal_process(uint32_t size), when it’s recording. Implement signal processing according to the required supply, such as when you want to increase the gain when recording with a digital microphone.

1.3.6. Retrieve PCM data

1.3.6.1. Overview

In this sample, only the beginning of the PCM data is displayed, by using this sample, it is possible to perform frequency analysis processing of PCM raw data and signal processing such as various filters. The function of this sample by the object level interface is the same as the sample Retrieve PCM data by the high level interface.

1.3.6.2. Operating environment and procedure
  1. Open the sample sketch by selecting File → Examples → Examples for Spresense / Audio → application → pcm_capture_objif from Arduino IDE.

    For details on the operating environment and operating procedure, please refer to Retrieve PCM data. In this sample, the standard output of the first 8 data is performed, but please change this, if you want to process.

1.3.6.3. Program description
  • The operation mode is specified at the beginning of this sample program.

    static const int32_t channel_num  = AS_CHANNEL_4CH;                                     (1)
    static const int32_t bit_length   = AS_BITLENGTH_16;                                    (2)
    static const int32_t frame_sample = 384;                                                (3)
    static const int32_t frame_size   = frame_sample * (bit_length / 8) * channel_num;      (4)
    
    static CMN_SimpleFifoHandle simple_fifo_handle;                                         (5)
    static const int32_t fifo_size  = frame_size * 10;                                      (6)
    static uint8_t fifo_buffer[fifo_size];                                                  (7)
    
    
    static const int32_t proc_size  = frame_size * 2;                                       (8)
    static uint8_t proc_buffer[proc_size];                                                  (9)
    1 In this sample, the number of channels is specified. The sound collection operation of 4 channels is specified. 1/2/4/8 channels can be specified.
    2 Specify the bit length of the audio data. In this sample, a bit length of 16 bits is specified. 16/24 can be specified.
    3 Specify the number of samples per frame. In this sample, 320 samples are specified. Specify a number from 240 to 1024.
    4 The number of bytes in one frame is calculated.
    5 A FIFO handle (CMN_SimpleFifoHandle) for reading data is defined.
    6 Specify the size of the FIFO for reading data. In this sample, the size for 10 frames is specified. Please adjust for your application.
    7 The FIFO area for reading data is secured.
    8 Specify the size of the memory area for signal processing. In this sample, the size that is twice the frame size is specified. Specify according to the signal processing you want to perform.
    9 A memory area for signal processing is secured. Please allocate according to the signal processing you want to perform.
  • Specify by createStaticPools() the layout of shared memory.

       initMemoryPools ();
       createStaticPools (no); (1)
    1 no: Specify the layout number. In this sample, specify the layout for the recording function (MEM_LAYOUT_RECORDER).
  • Generate and initialize the Recoder library.

    theFrontEnd = FrontEnd::getInstance();      (1)
    
    theFrontEnd->begin(                         (2)
                       frontend_attention_cb);  (3)
    1 getInstance
    Generates and get instance the FrontEnd software module.
    2 theFrontEnd→ begin
    Start-up the FrontEnd software module.
    3 attention_cb: Callback for error warning notified from each module. Please implement error handling.
  • Set the capturing clock mode with the setCapturingClkMode().

    theFrontEnd-> setCapturingClkMode(mode); (1)
    1 mode: Select clock mode, normal mode (MEDIARECORDER_CAPCLK_NORMAL), or high resolution mode (MEDIARECORDER_CAPCLK_HIRES).
    Specify MEDIARECORDER_CAPCLK_NORMAL for recording at 48kHz, and specify MEDIARECORDER_CAPCLK_HIRES for recording at 192kHz.
  • Activate the sound collection operation with the activate()

    theFrontEnd-> activate(frontend_done_callback); (1)
    1 frontend_done_callback: Specifies the callback for the completion event of each control function.
  • Initialize the sound collection operation with the init().

    theFrontEnd->init(channel_num,     (1)
                      bit_length,      (2)
                      frame_sample,    (3)
                      path,            (4)
                      dst);            (5)
    1 channel_num: Specify the number of channels. You can specify 1/2/4/8.
    2 bit_length: Specify the bit length of the audio data. Specify AS_BITLENGTH_16 or AS_BITLENGTH_24.
    3 frame_sample: Specify the number of frame samples. Set a value between 240 and 1024.
    4 path: Specify the data output method. If you want to get the data by callback to the app, please do AsDataPathCallback.
    5 dst: Specify the data output destination. If you specify AsDataPathCallback, specify the callback function. In this sample, it is specified by dst.cb = frontend_pcm_callback;.
  • Start the recorder with the start().

    theFrontEnd->start();
  • Stop the recorder with the stop().

    theFrontEnd->stop();
  • The processing in the callback function frontend_pcm_callback() is as follows.

    static void frontend_pcm_callback(AsPcmDataParam pcm)
    {
      if (!pcm.is_valid) {                                                                              (1)
        puts("Invalid data !");
        return;
      }
    
      if (CMN_SimpleFifoGetVacantSize(&simple_fifo_handle) < pcm.size) {                                (2)
        puts("Simple FIFO is full !");
        return;
      }
    
      if (CMN_SimpleFifoOffer(&simple_fifo_handle, (const void*)(pcm.mh.getPa()), pcm.size) == 0) {     (3)
        puts("Simple FIFO is full !");
        return;
      }
    
      return;
    }
    1 Check if the acquired data is valid. If it is not valid, do not use it as data.
    2 Check that there is free space in the FIFO for reading data.
    3 If there is free space in the FIFO for reading data, write the data to the FIFO. From FrontEnd Object, PCM data for the size of pcm.size is written to the address indicated by pcm.mh.getPa().
  • About signal processing in application loop

    The actual signal processing is done inside the application loop of loop(). The data in the FIFO for reading is written in the callback is read appropriately in the loop, and execute_aframe is used for frame-by-frame processing.

    bool execute_aframe()
    {
      size_t size = CMN_SimpleFifoGetOccupiedSize(&simple_fifo_handle);                  (1)
    
      if (size > 0) {
        if (size > proc_size) {
          size = (size_t)proc_size;
        }
    
        if (CMN_SimpleFifoPoll(&simple_fifo_handle, (void*)proc_buffer, size) == 0) {    (2)
          printf("ERROR: Fail to get data from simple FIFO.\n");
          return false;
        }
    
        signal_process(size);                                                            (3)
    
      }
    1 Get the size of the data in the FIFO for reading the data.
    2 Copy the data from the FIFO for reading data to the buffer for signal processing.
    3 Perform desired processing such as signal processing. Please implement the contents of signal_process for each application.
1.3.6.4. About signal processing at the time of writing

By performing signal processing (frequency analysis, AI, etc.) in the acquired voice data in signal_process(uint32_t size) you can perform the desired sensing process. ==== About recorder buffer size

2. Spresense Arduino Sensing Library

2.1. Let’s try Step Counter on Spresense

2.1.1. An example sketch of Step Counter

The following are the requirements to run the Step Counter sketch.

In this sketch, a physical sensor client using the BMI160 library works with a logical sensor client using the AESM. The acceleration data obtained from BMI160 is input, the activity recognition is performed by using machine learning (AI), and the recognition result is output.

AESM is installed by Tools > Burn Bootloader on Arduino IDE menu. Please refer to Install Bootloader for details.

On the Arduino IDE, select File→ Examples → Sensing under "Examples for Spresense" → application → step_counter and open the example sketch.

arduino stepcounter menu1
arduino stepcounter menu2

Compile this sketch and upload it to the Spresense board.

When you open serial monitor, the pedometer and other information will be displayed in real time.

arduino stepcounter result
tempo

Number of steps per second [Hz]

stride

Stride [cm]
This is fixed value.

speed

Moving speed [m/s]

distance

Moving distance [m]
The option of calculating the distance by fusing GPS positioning data will be supported in the future.
The current released software does not use GPS data.

step

Number of steps

move-type

Activity recognition result (stopping, walking and running)

2.1.2. Try to change the physical sensor of Step Counter

Here, let’s change the physical sensor to another sensor.

  • Sensor board to ROHM’s SPRESENSE-SENSOR-EVK-701.

  • KX122 Arduino Library

    • How to install KX122 Arduino library

      1. Select Clone or download → Download ZIP on GitHub site and download a ZIP file.

      2. Extract the downloaded ZIP file into any folder.

      3. On Arduino IDE menu, open Sketch → Include Library → Add .ZIP Library…​ and select the KX122 folder from the extracted folder.

      4. If Library added to your libraries. is shown on Arduino IDE, the installation is successful.

How to change step_counter.ino is below.

  • Include library

    Change to inclue <KX122.h>.

    #include <BMI160Gen.h>

    to

    #include <Wire.h>
    #include <KX122.h>
    KX122 kx122(KX122_DEVICE_ADDRESS_1F);
  • setup() function

    Change to initialize KX122. The accel_rate is set to 50 Hz by default of KX122 library.

      /* Initialize device. */
    
      BMI160.begin(BMI160GenClass::I2C_MODE, i2c_addr);
    
      /* Set device setting */
    
      BMI160.setAccelerometerRange(accel_range);
      BMI160.setAccelerometerRate(accel_rate);

    to

      Wire.begin();
      kx122.init();
  • loop() function

    Change to get the acceleration data from KX122.

      float x;
      float y;
      float z;
    
      /* Read raw accelerometer measurements from BMI160 */
    
      BMI160.readAccelerometerScaled(x, y, z);
    
      AccelSensor.write_data(x, y, z);

    to

      float acc[3];
    
      /* Read raw accelerometer measurements from KX122 */
    
      kx122.get_val(acc);
    
      AccelSensor.write_data(acc[0], acc[1], acc[2]);

After applying the above modifications, you should be able to work the Step Counter with KX122.

2.1.3. Let’s try to create the unique logical sensor of Step Counter

If you want to implement your own Step Counter algorithm instead of Sony’s provided Step Counter algorithm, create a logic sensor client as follow.

Create a logical sensor class that inherits SensorClient .

For example, implement MyCounterClass in MyCounter.h. Include the header file as needed. For example, SensorClient.h is required, but sensing/logical_sensor/step_counter.h is necessary only if you want to use the StepCounter definition.

Also, as an example, the figure below shows the storage location of generally used code.

Diagram
Figure 5. Example of MyCounter code storage location

Above diagram is an example of Windows environment. The Arduino directory is ~/Arduino in Ubuntu environment and ~/Documents/Arduino in macOS environment.

class MyCounterClass : public SensorClient

Override the following methods of MyCounterClass on MyCounter.cpp .

bool begin(int id,
           uint32_t subscriptions,
           int      rate,
           int      sample_watermark_num,
           int      size_per_sample));

int publish(FAR void* data,
            uint32_t  size_per_sample,
            uint32_t  freq,
            uint32_t  sample_watermark_num,
            uint32_t  timestamp);

int subscribe(sensor_command_data_mh_t& data);
  • Implementation of begin() :

Set initial settings for MyCounter. It is necessary to register a callback function to receive published accelerometer data. The callback function calls MyCounter’s subscribe() and get the data.

Description example
/* Callback function to receive accel sensor data. */

unsigned char mycounter_cb(sensor_command_data_mh_t &data)
{
  /* MyCounter subscribes to the received accel data. */

  return MyCounter.subscribe(data);
}

/* Initial setting of MyCounterClass. */

bool MyCounterClass::begin(int      id,
                           uint32_t subscriptions,
                           int      rate,
                           int      sample_watermark_num,
                           int      size_per_sample)
{
  /* Call super class begin() method. */

  return SensorClient::begin(id,
                             subscriptions,
                             rate,
                             sample_watermark_num,
                             size_per_sample,
                             mycounter_cb);        /* <-- Register callback function. */
}
  • Implementation of subscribe() :

You can create your own Step Counter by subscribing to the accelerometer data and processing it with your own algorithm.

If you use AccelSensorClass which is provided by the library, the data which is publised from accel sensor is 50 samples of acceleration data. The acceleration data is sampled by 50Hz. And the structure of them is data array of x, y, z axis acceleration (the unit is [G]). This structure is defined as st_accel_axis. Call Subscribe() in the base class ( SensorClient ) to get this data.
Next, please apply your original process and publish result of them. In this example, you will publish with SensorResultStepCounter data format. Each parameter of the data format is described here for more details.

Description example
/* Subscribe the accel sensor data which is published from AccelSenseorClass. */

int MyCounterClass::subscribe(sensor_command_data_mh_t& data)
{
  struct st_accel_axis
  {
    float x;
    float y;
    float z;
  } *accel_axis;

  /* Get accel sensor data via subscribe() of super class. */

  accel_axis = static_cast<accel_axis*>(SensorClient::subscribe(data));
  uint32_t  timestamp = data.time;

  /* In this sample, output parameter structure is SensorResultStepCounter. */

  SensorResultStepCounter output;

  /*
    Implement your own algorithm and put a value in output.
   */

  /* Publish processed data to subscriber. */

  publish(&output,
          sizeof(SensorResultStepCounter),
          1, /* 1Hz */
          1, /* 1sample */
          timestamp);
}
The subscribed data is sensor_command_data_mh_t . Please refer to here for this data structure.
  • Implementation of publish() :

publish() send the data calculated by MyCounter.
The StepCounterReader is provided as a sample of the reading application client, and if you use it as it is, the data structure to publish is

SensorResultStepCounter

Note that the parameter exec_result in SensorResultStepCounter should be set to SensorOK to notify result to application.

As mentioned above, when passed in the form of SensorResultStepCounter , publish() of the base class ( SensorClient ) can be used as it is.

SensorResultStepCounter format is below.

/*--------------------------------------------------------------------------*/
/**
 * @struct StepCounterStepInfo
 * @brief the structure of STEP_COUNTER results.
 */
typedef struct {

  float    tempo;     /**< Indicates tempo of walking / jogging calculated
                       *   from the input acceleration data.
                       *   The unit is [Hz].
                       */
  float    stride;    /**< Indicates stride calculated
                       *   from input acceleration data.
                       *   The unit is [cm].
                       */
  float    speed;     /**< Indicates speed of walking / jogging calculated
                       *   from the input acceleration data.
                       *   The unit is [m/s].
                       */
  float    distance;  /**< Indicates cumulative travel distance calculated
                       *   from the input acceleration data.
                       *   The unit is [m].
                       */
  uint32_t step;      /**< Indicates the number of steps calculated
                       *   from the input acceleration data.
                       * The unit is [step].
                       */
  StepCounterMovementType  movement_type; /**<
                                    * Indicates the walking type calculated
                                    * from the input acceleration data.
                                    */
  uint64_t          time_stamp;    /**< Indicates latest timestamp of the
                                    *   acceleration sensor data used.
                                    */
} StepCounterStepInfo;

/*--------------------------------------------------------------------------*/
/**
 * @struct SensorResultStepCounter
 * @brief the structure of sensor result on step counter commands.
 */
typedef struct
{
  SensorExecResult exec_result; /**< Execute resule.  */

  union
    {
      StepCounterStepInfo steps;        /**< Step count.         */
      SensorAssertionInfo assert_info;  /**< Assert information. */
    };
} SensorResultStepCounter;

2.1.4. Let’s try to change the data according to your logic sensor

Of course, you may want to change the resulting sensor data to match your own logic sensor.

In that case, use your defined data type (e.g. MyStepCounterResult) in publish() of MyCounter class.

int MyCounterClass::subscribe(sensor_command_data_mh_t& data)
{
  ..(Omission)..

  MyStepCounterResult output;

  publish(&output,
          sizeof(MyStepCounterResult),
          1, /* 1Hz */
          1, /* 1sample */
          timestamp);
}

Please implement as follows in Application ( StepCountReaderClass ).

subscribe()
{
  MyStepCounterResult* steps = reinterpret_cast<MyStepCounterResult*>(ApplicationSensorClass.subscribe(data));
}

2.2. Let’s try to create new sensor client

2.2.1. Let’s try to create new physical sensor client

[T.B.D]

3. LTE tutorial

The LTE tutorials explain how to run the various sample sketches.

3.1. Get modem information

3.1.1. Overview

This sample sketch retrieves information about the modem. The sketch will obtain the following information after the modem is turned on:

  • IMEI

  • Firmware Version

3.1.2. Requirements

  • Spresense Main Board

  • Spresense LTE extension board

For instructions on how to connect the boards, please refer to How to connect the Spresense main board to the Spresense LTE extension board.

3.1.3. Instructions

  1. From the Arduino IDE, select File → Examples → LTE → LteTestModem to open the example sketch.

    arduino lte examples testmodem1 en
  2. In Tools → Port select the COM port of Spresense and upload the sketch.

  3. After compiling and uploading are finished, start the serial monitor.

  4. Make sure the baud rate is set to 115200 bps and you should see the message below.

    arduino lte examples testmodem2 en

3.1.4. Application Description

The application consists of three different parts: initialization, setup() and loop().

3.1.4.1. initialization
  • Include the header file for the library used in this sketch.

    // libraries
    #include <LTE.h>  (1)
    1 Include the <LTE.h> header file to use the LTE library to register the modem to the LTE network.
  • Instantiate the class to be used.

    // initialize the library instance
    LTEModem modem;  (1)
    1 Initializes an instance of the LTEModem class.
  • Initialize the variables used in the sketch.

    // IMEI variable
    String IMEI = "";     (1)
    String VERSION = "";
    LTENetworkRatType RAT = LTE_NET_RAT_UNKNOWN;
    1 Define the variables used to store the IMEI number, firmware version and RAT(Radio Access Technology) obtained from the modem.
3.1.4.2. setup()
  • The setup() function sets up the Serial and modem classes.

    void setup() {
      // initialize serial communications and wait for port to open:
      Serial.begin(115200);                          (1)
      while (!Serial) {
          ; /* wait for serial port to connect. Needed for native USB port only */
      }
    
      // start modem test (reset and check response)
      Serial.println("Starting modem test...");
      if (LTE_IDLE == modem.begin()) {               (2)
        Serial.println("modem.begin() succeeded.");
      } else {
        Serial.println("ERROR, no modem answer.");
      }
    }
    1 Setup Serial for displaying messages.
    2 Call the begin() method of LTEModem.
    Calling the begin() method will turn on the modem. When the modem is successfully turned on, the begin() method returns LTE_IDLE.
3.1.4.3. loop()
  • The loop() function retrieves the IMEI number, firmware version and RAT of the modem and displays them in the serial monitor.

    void loop() {
    
      // IMEI of the modem
      IMEI = modem.getIMEI();                 (1)
      Serial.println("IMEI: " + IMEI);
    
      //Firmware version of the modem.
      VERSION = modem.getFirmwareVersion();   (2)
      Serial.println("VERSION: " + VERSION);
    
      // Current RAT of the modem.
      RAT = modem.getRAT();                   (3)
    
      switch (RAT) {
        case LTE_NET_RAT_CATM:
          Serial.println("RAT: LTE-M (LTE Cat-M1)");
          break;
        case LTE_NET_RAT_NBIOT:
          Serial.println("RAT: NB-IoT");
          break;
        default:
          Serial.println("RAT: Unknown type [" + String(RAT) + "]");
          break;
      }
    
      sleep(1);
    }
    1 Get the IMEI number from the modem using the getIMEI() method of LTEModem.
    2 Get the firmware version from the modem using the getFirmwareVersion() method of LTEModem.
    3 Get the RAT set on the modem using the getRAT() method of LTEModem.

3.2. Get network status

3.2.1. Overview

This sample sketch retrieves the LTE network status.
The sketch registers the modem to the LTE network and then retrieves the following information:

  • IP Address

  • Signal strength

  • Network carrier name

3.2.2. Requirements

  • Spresense Main Board

  • Spresense LTE extension board

  • SIM card

For instructions on how to connect the boards, please refer to How to connect the Spresense main board to the Spresense LTE extension board.

This sketch requires a SIM card to connect to the network.
Please check the Confirmed LTE operator list.

3.2.3. Instructions

  1. From the Arduino IDE, select File → Examples → LTE → LteScanNetworks to open the example sketch.

    arduino lte examples scannetworks1 en
  2. Define the APN parameters required to register the modem to the LTE network.
    You will need to modify the values to match your SIM.

    #define APP_LTE_APN "apn" // replace your APN                       (1)
    
    /* APN authentication settings
     * Ignore these parameters when setting LTE_NET_AUTHTYPE_NONE.
     */
    #define APP_LTE_USER_NAME "user"     // replace with your username  (2)
    #define APP_LTE_PASSWORD  "password" // replace with your password  (3)
    1 Please enter the access point name that matches your SIM.
    2 Enter a username that matches your SIM.
    3 Enter a password that matches the SIM you are using.
  3. Set the RAT to use.
    Switch the comment out part according to the SIM you are using.

    /* RAT to use
     * Refer to the cellular carriers information
     * to find out which RAT your SIM supports.
     * The RAT set on the modem can be checked with LTEModemVerification::getRAT().
     */
    
    #define APP_LTE_RAT (LTE_NET_RAT_CATM) // RAT : LTE-M (LTE Cat-M1) (1)
    // #define APP_LTE_RAT (LTE_NET_RAT_NBIOT) // RAT : NB-IoT         (2)
    1 Use this definition when communicating by LTE-M (LTE Cat-M1).
    2 Use this definition when communicating by NB-IoT.
  4. In Tools → Port select the COM port of Spresense and upload the sketch

  5. After compiling and uploading are finished, start the serial monitor.

  6. Make sure the baud rate is set to 115200 bps and you should see the message below.

    arduino lte examples scannetworks2 en

Depending on the environment, network communication may become unstable and may not work properly.
For more details, please refer to the FAQ.

3.2.4. Application Description

The application consists of three different parts: initialization, setup() and loop().

3.2.4.1. initialization
  • Include the header file for the library used in this sketch.

    // libraries
    #include <LTE.h>  (1)
    1 Include the <LTE.h> header file to use the LTE library to register the modem to the LTE network.
  • Define the APN parameters required to register the modem to the LTE network.
    The values have to be replaced with the ones matching your SIM card.

    #define APP_LTE_APN "apn" // replace your APN                       (1)
    
    /* APN authentication settings
     * Ignore these parameters when setting LTE_NET_AUTHTYPE_NONE.
     */
    #define APP_LTE_USER_NAME "user"     // replace with your username  (2)
    #define APP_LTE_PASSWORD  "password" // replace with your password  (3)
    1 Please enter the access point name that matches your SIM.
    2 Enter a username that matches your SIM.
    3 Enter a password that matches the SIM you are using.
  • Instantiate the classes to be used.

    // initialize the library instance
    LTE        lteAccess;        (1)
    LTEScanner scannerNetworks;  (2)
    1 Initializes an instance of the LTE class.
    2 Initializes an instance of the LTEScanner class.
3.2.4.2. setup()
  • The setup() function sets up the Serial and modem classes.

    void setup() {
      // initialize serial communications and wait for port to open:
      Serial.begin(115200);                                                           (1)
      while (!Serial) {
        ; // wait for serial port to connect. Needed for Leonardo only
      }
    
      Serial.println("LTE networks scanner");
    
      while (true) {
    
        /* Power on the modem and Enable the radio function. */
    
        if (lteAccess.begin() != LTE_SEARCHING) {                                     (2)
          Serial.println("Could not transition to LTE_SEARCHING.");
          Serial.println("Please check the status of the LTE board.");
          for (;;) {
            sleep(1);
          }
        }
    
        /* The connection process to the APN will start.
         * If the synchronous parameter is false,
         * the return value will be returned when the connection process is started.
         */
        if (lteAccess.attach(APP_LTE_RAT,
                             APP_LTE_APN,
                             APP_LTE_USER_NAME,
                             APP_LTE_PASSWORD,
                             APP_LTE_AUTH_TYPE,
                             APP_LTE_IP_TYPE) == LTE_READY) {                         (3)
          Serial.println("attach succeeded.");
          break;
        }
    
        /* If the following logs occur frequently, one of the following might be a cause:
         * - APN settings are incorrect
         * - SIM is not inserted correctly
         * - If you have specified LTE_NET_RAT_NBIOT for APP_LTE_RAT,
         *   your LTE board may not support it.
         * - Rejected from LTE network
         */
        Serial.println("An error has occurred. Shutdown and retry the network attach process after 1 second.");
        lteAccess.shutdown();                                                         (4)
        sleep(1);
      }
    }
    1 Setup Serial for displaying messages.
    2 Call the begin() method of the LTE class. Calling the begin() method will turn on the modem and search the network.
    When the network search is successfully started, the begin() method returns LTE_SEARCHING.
    3 Call the attach() method of LTE.
    Register the modem with the LTE network by calling the attach() method.
    When the modem is successfully registered to the LTE network, the attach() method returns LTE_READY.
    4 If the method does not return the expected return value, the LTE method shutdown() will be called.
    This will turn off the modem.
3.2.4.3. loop()
  • The loop() function obtains the IP address, the name of the connected LTE network carrier and the signal strength of the LTE network, and displays them in the serial monitor.

    void loop() {
       // Assigned IP address
      IPAddress address = lteAccess.getIPAddress();         (1)
      Serial.print("IP address: ");
      Serial.println(address);
    
      // currently connected carrier
      Serial.print("Current carrier: ");
      Serial.println(scannerNetworks.getCurrentCarrier());  (2)
    
      // return signal strength
      Serial.print("Signal Strength: ");
      Serial.print(scannerNetworks.getSignalStrength());    (3)
      Serial.println(" [dBm]");
      sleep(1);
    }
    1 Get the IP address assigned by the LTE network with the getIPAddress() method of LTE.
    2 Get the name of the connected LTE network carrier with the LTEScanner’s getCurrentCarrier() method.
    3 The getSignalStrength() method of the LTEScanner retrieves the signal strength of the LTE network.

3.3. Send a HTTP GET request to retrieve server data

3.3.1. Overview

This sample sketch requests data from a server via the LTE network. The sketch will register the modem to the LTE network, send a HTTP GET request to a server and retrieve the result.

3.3.2. Requirements

  • Spresense Main Board

  • Spresense LTE extension board

  • SIM card

For instructions on how to connect the boards, please refer to How to connect the Spresense main board to the Spresense LTE extension board.

This sketch requires a SIM card to connect to the network.
Please check the Confirmed LTE operator list.

3.3.3. Instructions

  1. From the Arduino IDE, select File → Examples → LTE → LteWebClient to open the example sketch.

    arduino lte examples webclient1 en
  2. Define the APN parameters required to register the modem to the LTE network.
    You will need to modify the values to match your SIM.

    #define APP_LTE_APN "apn" // replace your APN                       (1)
    
    /* APN authentication settings
     * Ignore these parameters when setting LTE_NET_AUTHTYPE_NONE.
     */
    #define APP_LTE_USER_NAME "user"     // replace with your username  (2)
    #define APP_LTE_PASSWORD  "password" // replace with your password  (3)
    1 Please enter the access point name that matches your SIM.
    2 Enter a username that matches your SIM.
    3 Enter a password that matches the SIM you are using.
  3. Set the RAT to use.
    Switch the comment out part according to the SIM you are using.

    /* RAT to use
     * Refer to the cellular carriers information
     * to find out which RAT your SIM supports.
     * The RAT set on the modem can be checked with LTEModemVerification::getRAT().
     */
    
    #define APP_LTE_RAT (LTE_NET_RAT_CATM) // RAT : LTE-M (LTE Cat-M1) (1)
    // #define APP_LTE_RAT (LTE_NET_RAT_NBIOT) // RAT : NB-IoT         (2)
    1 Use this definition when communicating by LTE-M (LTE Cat-M1).
    2 Use this definition when communicating by NB-IoT.
  4. In Tools → Port select the COM port of Spresense and upload the sketch.

  5. After compiling and uploading are finished, start the serial monitor.

  6. Make sure the baud rate is set to 115200 bps and you should see the message below.

    arduino lte examples webclient2 en

Depending on the environment, network communication may become unstable and may not work properly.
For more details, please refer to the FAQ.

3.3.4. Application Description

The application consists of three different parts: initialization, setup() and loop().

3.3.4.1. initialization
  • Include the header file for the library used in this sketch.

    // libraries
    #include <LTE.h>  (1)
    1 Include the <LTE.h> header file to use the LTE library to register the modem to the LTE network.
  • Define the APN parameters required to register the modem to the LTE network.
    The values have to be replaced with the ones matching your SIM card.

    #define APP_LTE_APN "apn" // replace your APN                       (1)
    
    /* APN authentication settings
     * Ignore these parameters when setting LTE_NET_AUTHTYPE_NONE.
     */
    #define APP_LTE_USER_NAME "user"     // replace with your username  (2)
    #define APP_LTE_PASSWORD  "password" // replace with your password  (3)
    1 Please enter the access point name that matches your SIM.
    2 Enter a username that matches your SIM.
    3 Enter a password that matches the SIM you are using.
  • Instantiate the classes to be used.

    // initialize the library instance
    LTE lteAccess;     (1)
    LTEClient client;  (2)
    1 Initializes an instance of the LTE class.
    2 Initializes an instance of the LTEClient class.
  • Defines the parameters of the server to connect to.

    // URL, path & port (for example: arduino.cc)
    char server[] = "arduino.cc";                      (1)
    char path[] = "/asciilogo.txt";                    (2)
    int port = 80; // port 80 is the default for HTTP  (3)
    1 URL of the server to connect to
    2 Server path of the HTTP GET request
    3 Port number of the server to connect to
3.3.4.2. setup()
  • The setup() function sets up the Serial and modem classes.

    void setup()
    {
      // initialize serial communications and wait for port to open:
      Serial.begin(115200);                                                           (1)
      while (!Serial) {
          ; // wait for serial port to connect. Needed for native USB port only
      }
    
      Serial.println("Starting web client.");
    
      while (true) {
    
        /* Power on the modem and Enable the radio function. */
    
        if (lteAccess.begin() != LTE_SEARCHING) {                                     (2)
          Serial.println("Could not transition to LTE_SEARCHING.");
          Serial.println("Please check the status of the LTE board.");
          for (;;) {
            sleep(1);
          }
        }
    
        /* The connection process to the APN will start.
         * If the synchronous parameter is false,
         * the return value will be returned when the connection process is started.
         */
        if (lteAccess.attach(APP_LTE_RAT,
                             APP_LTE_APN,
                             APP_LTE_USER_NAME,
                             APP_LTE_PASSWORD,
                             APP_LTE_AUTH_TYPE,
                             APP_LTE_IP_TYPE) == LTE_READY) {                         (3)
          Serial.println("attach succeeded.");
          break;
        }
    
        /* If the following logs occur frequently, one of the following might be a cause:
         * - APN settings are incorrect
         * - SIM is not inserted correctly
         * - If you have specified LTE_NET_RAT_NBIOT for APP_LTE_RAT,
         *   your LTE board may not support it.
         * - Rejected from LTE network
         */
        Serial.println("An error has occurred. Shutdown and retry the network attach process after 1 second.");
        lteAccess.shutdown();                                                         (4)
        sleep(1);
      }
    
      // if you get a connection, report back via serial:
      if (client.connect(server, port)) {                                             (5)
        Serial.println("connected");
        // Make a HTTP request:
        client.print("GET ");                                                         (6)
        client.print(path);
        client.println(" HTTP/1.1");
        client.print("Host: ");
        client.println(server);
        client.println("Connection: close");
        client.println();
      } else {
        // if you didn't get a connection to the server:
        Serial.println("connection failed");
      }
    }
    1 Setup Serial for displaying messages.
    2 Call the begin() method of the LTE class. Calling the begin() method will turn on the modem and search the network.
    When the network search is successfully started, the begin() method returns LTE_SEARCHING.
    3 Call the attach() method of LTE.
    Register the modem with the LTE network by calling the attach() method.
    When the modem is successfully registered to the LTE network, the attach() method returns LTE_READY.
    4 If the method does not return the expected return value, the LTE method shutdown() will be called.
    This will turn off the modem.
    5 Call the connect() method of LTEClient.
    Calling the connect() method connects to the host name and port specified in the arguments.
    The connect() method returns the value 1 when the connection to the server is successfully established.
    6 Send an HTTP request to the server using the print() and println() methods of LTEClient.
3.3.4.3. loop()
  • The loop() function receives the HTTP response and displays the response message in the serial monitor.

    void loop()
    {
      // if there are incoming bytes available
      // from the server, read them and print them:
      if (int len = client.available()) {                (1)
        char buff[len + 1];
        buff[len] = '\0';
        client.read(buff, len);                          (2)
        Serial.print(buff);
      }
    
      // if the server's disconnected, stop the client:
      if (!client.available() && !client.connected()) {  (3)
        Serial.println();
        Serial.println("disconnecting.");
        client.stop();                                   (4)
    
        // do nothing forevermore:
        for (;;)
          sleep(1);
      }
    }
    1 Use the available() method of LTEClient to get the number of bytes available for reading.
    2 Use the read() method of LTEClient to read the data received from the connected server into a buffer and output it in the serial monitor.
    3 Use the connected() method of LTEClient to find out whether the client is connected to the server or not.
    4 Disconnect from the server using the stop() method of LTEClient.

3.4. Checking HTTP Client behavior using the TLS protocol

3.4.1. Overview

This sample sketch demonstrates how to create a HTTP Client that uses the TLS protocol.
The sketch will register the modem to the LTE network, connect to a server using the TLS protocol and use the Arduino HTTP Client library to test the get() and post() methods that it provides.

If the LTE firmware version is RK_03_00_00_00_00_04121_001, the TLS communication API may not work. (Please refer to here )
Please update firmware to refer to the Updater Tool on the download site.

3.4.2. Environment

  • Spresense Main Board

  • Spresense LTE extension board

  • SIM card

  • microSD card

For the instructions on how to connect the hardware, please refer to How to connect the Spresense main board to the Spresense LTE extension board.

This sketch requires a SIM card to connect to the network.
Please check the Confirmed LTE operator list.

3.4.3. Instructions

  1. From the Arduino IDE, select File → Example sketch → Example sketch for Spresense LTE → LteHttpSecureClient to open the example sketch.

    arduino lte examples httpsecureclient1 en
  2. From the Arduino IDE, select Tools → Manage Libraries…​.
    Type ArduinoHttpClient in the search field and install the ArduinoHttpClient by Arduino item in the results.

    arduino lte examples httpsecureclient2 en
  3. Define the APN parameters associated with your SIM card to register the modem to the LTE network.

    #define APP_LTE_APN "apn" // replace your APN                       (1)
    
    /* APN authentication settings
     * Ignore these parameters when setting LTE_NET_AUTHTYPE_NONE.
     */
    #define APP_LTE_USER_NAME "user"     // replace with your username  (2)
    #define APP_LTE_PASSWORD  "password" // replace with your password  (3)
    1 Please enter the access point name that matches your SIM.
    2 Enter a user name that matches your SIM.
    3 Enter a password that matches the SIM you are using.
  4. Set the RAT to use.
    Switch the comment out part according to the SIM you are using.

    /* RAT to use
     * Refer to the cellular carriers information
     * to find out which RAT your SIM supports.
     * The RAT set on the modem can be checked with LTEModemVerification::getRAT().
     */
    
    #define APP_LTE_RAT (LTE_NET_RAT_CATM) // RAT : LTE-M (LTE Cat-M1) (1)
    // #define APP_LTE_RAT (LTE_NET_RAT_NBIOT) // RAT : NB-IoT         (2)
    1 Use this definition when communicating by LTE-M (LTE Cat-M1).
    2 Use this definition when communicating by NB-IoT.
  5. Export the root certificate required for the TLS protocol.
    Please refer to root certificate export procedure for the procedure.

  6. Put the certificate on the microSD card. Make sure that the name of the directory where it is located matches the illustration below:

    microSD:
      └CERTS/
        └httpbin-org.pem
  7. Modify ROOTCA_FILE in the sample sketch to match the name of the file on the microSD card.

    #define ROOTCA_FILE "CERTS/httpbin-org.pem" // Define the path to a file containing CA
                                                // certificates that are trusted.

    Set the file name of the root certificate saved on the microSD card.

  8. Select the COM port of Spresense in Tools → Serial Port and perform the write operation.

  9. After compiling and writing (Upload) is finished, start the serial monitor.

  10. If you select the baud rate 115200 bps you should see log output as shown below.

    arduino lte examples httpsecureclient3 en

If the terminal log does not proceed beyond the indication making GET request, your Spresense LTE extension board firmware may be out of date.
Please update the Spresense LTE board firmware to refer Updater Tool .

Depending on the operating environment, the communication state may become unstable and may not working correctly.
For more details, please check FAQ.

3.4.4. Program Description

The application consists of three different steps: "initialization", "setup()", and "loop()". These steps are explained in detail below.

3.4.4.1. initialization
  • Include header files for the libraries used in this sketch.

    // libraries
    #include <ArduinoHttpClient.h> (1)
    #include <RTC.h>               (2)
    #include <SDHCI.h>             (3)
    #include <LTE.h>               (4)
    1 To use the Arduino HTTP Client library for sending HTTP GET and POST requests, include <ArduinoHttpClient.h>.
    2 Include the header file <RTC.h> to use the RTC library to set the SPRESENSE time.
    3 Include the header file <SDHCI.h> to use the SD library to reference the certificate and private key required for the TLS protocol.
    4 Include the header file <LTE.h> to use the LTE library to register the modem to the LTE network.
  • Define the APN parameters required to register the modem to the LTE network.
    The values must be updated to match those of your SIM card.

    #define APP_LTE_APN "apn" // replace your APN                       (1)
    
    /* APN authentication settings
     * Ignore these parameters when setting LTE_NET_AUTHTYPE_NONE.
     */
    #define APP_LTE_USER_NAME "user"     // replace with your username  (2)
    #define APP_LTE_PASSWORD  "password" // replace with your password  (3)
    1 Please enter the access point name that matches your SIM.
    2 Enter a user name that matches your SIM.
    3 Enter a password that matches your SIM.
  • Define the parameters of the server you are connecting to.

    // URL, path & port (for example: httpbin.org)
    char server[] = "httpbin.org";                       (1)
    char getPath[] = "/get";                             (2)
    char postPath[] = "/post";                           (3)
    int port = 443; // port 443 is the default for HTTPS (4)
    1 URL of the server to connect to
    2 Server path of the HTTP GET destination
    3 Server path of the HTTP POST destination
    4 Port number of the server to connect to
  • Define the parameters for the root certificate, client certificate, and private key required for the TLS protocol.
    The file names must be updated to match those of the files saved in the microSD card.

    #define ROOTCA_FILE "path/to/cafile"   // Define the path to a file containing CA           (1)
                                           // certificates that are trusted.
    #define CERT_FILE   "path/to/certfile" // Define the path to a file containing certificate  (2)
                                           // for this client, if required by the server.
    #define KEY_FILE    "path/to/keyfile"  // Define the path to a file containing private key  (3)
                                           // for this client, if required by the server.
    1 Describe the file name of the root certificate stored on the microSD card.
    2 Describe the file name of the client certificate stored on the microSD card.
    3 Describe the file name of the private key stored on the microSD card.
  • Initialize the instance of the class to be used.

    // initialize the library instance
    LTE lteAccess;
    LTETLSClient tlsClient;
    HttpClient client = HttpClient(tlsClient, server, port); (1)
    SDClass theSD;
    1 Specifies an instance of LTETLSClient when initializing an instance of HttpClient.
3.4.4.2. setup()

This section describes the setup() function step by step.

  • Setup Serial for displaying log messages.

    // initialize serial communications and wait for port to open:
    Serial.begin(115200);
    while (!Serial) {
        ; // wait for serial port to connect. Needed for native USB port only
    }
    
    Serial.println("Starting secure HTTP client.");
  • Mount the microSD card in theSD.begin() method.

    /* Initialize SD */
    while (!theSD.begin()) {
    ; /* wait until SD card is mounted. */
    }
  • Register the modem to the LTE network.

      while (true) {
    
        /* Power on the modem and Enable the radio function. */
    
        if (lteAccess.begin() != LTE_SEARCHING) {                      (1)
          Serial.println("Could not transition to LTE_SEARCHING.");
          Serial.println("Please check the status of the LTE board.");
          for (;;) {
            sleep(1);
          }
        }
    
        /* The connection process to the APN will start.
         * If the synchronous parameter is false,
         * the return value will be returned when the connection process is started.
         */
        if (lteAccess.attach(APP_LTE_RAT,
                             APP_LTE_APN,
                             APP_LTE_USER_NAME,
                             APP_LTE_PASSWORD,
                             APP_LTE_AUTH_TYPE,
                             APP_LTE_IP_TYPE) == LTE_READY) {          (2)
          Serial.println("attach succeeded.");
          break;
        }
    
        /* If the following logs occur frequently, one of the following might be a cause:
         * - APN settings are incorrect
         * - SIM is not inserted correctly
         * - If you have specified LTE_NET_RAT_NBIOT for APP_LTE_RAT,
         *   your LTE board may not support it.
         * - Rejected from LTE network
         */
        Serial.println("An error has occurred. Shutdown and retry the network attach process after 1 second.");
        lteAccess.shutdown();                                          (3)
        sleep(1);
      }
    1 Use the lteAccess.begin() method to start exploring the LTE network. lteAccess.begin() method is called repeatedly until it starts exploring the LTE network.
    2 Use the lteAccess.attach() method to register the modem with the LTE network. The loop is exited when the modem is successfully registered with the LTE network.
    3 If the modem fails to register with the LTE network, use the lteAccess.shutdown() method to turn off the modem, call sleep() to halt execution for about one second and then retry the LTE network search process.

    If your SIM has a PIN lock, please provide the PIN unlock code in the begin() method.

  • Set the SPRESENSE time and output the set time to serial.

    void printClock(RtcTime &rtc)                        (1)
    {
      printf("%04d/%02d/%02d %02d:%02d:%02d\n",
             rtc.year(), rtc.month(), rtc.day(),
             rtc.hour(), rtc.minute(), rtc.second());
    }
    
    void setup()
    {
      :
      // Set local time (not UTC) obtained from the network to RTC.
      RTC.begin();
      unsigned long currentTime;
      while(0 == (currentTime = lteAccess.getTime())) { (2)
        sleep(1);
      }
      RtcTime rtc(currentTime);                         (3)
      printClock(rtc);                                  (4)
      RTC.setTime(rtc);                                 (5)
    }
    1 Define the printClock() function to output the time given in the parameter to serial.
    2 Use the lteAccess.getTime() method to get the local time (in seconds from the source).
    3 Create an instance of RtcTime with the local time.
    4 Outputs the local time to serial using the printClock() function.
    5 Use the RTC.setTime() method to set the local time of SPRESENSE.
3.4.4.3. loop()

This section describes the loop function.

  • Configure the modem with the certificate and private key stored on the microSD card.

    // Set certifications via a file on the SD card before connecting to the server
    File rootCertsFile = theSD.open(ROOTCA_FILE, FILE_READ);       (1)
    tlsClient.setCACert(rootCertsFile, rootCertsFile.available()); (2)
    rootCertsFile.close();
    
    // Remove these commented out if client authentication is required.
    //File certsFile = theSD.open(CERT_FILE, FILE_READ);
    //tlsClient.setCertificate(certsFile, certsFile.available());  (3)
    //certsFile.close();
    //File priKeyFile = theSD.open(KEY_FILE, FILE_READ);
    //tlsClient.setPrivateKey(priKeyFile, priKeyFile.available()); (4)
    //priKeyFile.close();
    1 Use theSD.open() method to create a File instance of the root certificate.
    2 Set the root certificate to the modem by providing the File instance to the tlsClient.setCACert() method.
    3 If a client certificate is required, create a File instance just like the one for the root certificate and use tlsClient.setCertificate() to set it in the modem.
    4 If a private key is required, create a File instance for this one as well and use tlsClient.setPrivateKey() to set it in the modem.
  • Verify the operation of the HTTP GET method.

    // HTTP GET method
    Serial.println("making GET request");
    client.get(getPath);                          (1)
    
    // read the status code and body of the response
    int statusCode = client.responseStatusCode(); (2)
    String response = client.responseBody();      (3)
    
    Serial.print("Status code: ");
    Serial.println(statusCode);                   (4)
    Serial.print("Response: ");
    Serial.println(response);                     (5)
    Serial.println("Wait five seconds");
    sleep(5);
    1 Uses client.get() to connect to the server using the TLS protocol and send the HTTP GET request.
    2 Uses client.responseStatusCode() to receive a response message from the server and extract the status code from it.
    3 Uses client.responseBody() to get the body of the server’s response message.
    4 Outputs the obtained status code to serial.
    5 Outputs the body of the retrieved message to serial.
  • Check the operation of the HTTP POST method.

    // HTTP POST method
    Serial.println("making POST request");
    
    String contentType = "application/x-www-form-urlencoded"; (1)
    String postData = "name=Alice&age=12";                    (2)
    
    client.post(postPath, contentType, postData);             (3)
    
    // read the status code and body of the response
    statusCode = client.responseStatusCode();                 (4)
    response = client.responseBody();                         (5)
    
    Serial.print("Status code: ");
    Serial.println(statusCode);                               (6)
    Serial.print("Response: ");
    Serial.println(response);                                 (7)
    
    // do nothing forevermore:
    for (;;)
    sleep(1);
    1 Specifies the Content-Type of the message to be POSTed.
    2 Specifies the body part of the message to be POSTed.
    3 Uses client.post() to connect to the server using the TLS protocol and send the HTTP POST request.
    4 Uses client.responseStatusCode() to receive the response message from the server and extract the status code from it.
    5 Uses client.responseBody() to get the body of the server’s response message.
    6 Outputs the obtained status code to serial.
    7 Outputs the body of the retrieved message to serial.

3.5. Verifying the NTP Client using the UDP protocol.

3.5.1. Overview

This sample sketch shows how to run the NTP Client using the UDP protocol.
The sketch registers the modem to the LTE network, connects to a NTP server using the UDP protocol and calls the update() method provided by the Arduino NTP Client library.

3.5.2. Requirements

  • Spresense Main Board

  • Spresense LTE extension board

  • SIM card

For instructions on how to connect the boards, please refer to How to connect the Spresense main board to the Spresense LTE extension board.

This sketch requires a SIM card to connect to the network.
Please check the Confirmed LTE operator list.

3.5.3. Instructions

  1. From the Arduino IDE, select File → Examples → LTE → LteNtpClient to open the example sketch.

    arduino lte examples ntpclient1 en
  2. From the Arduino IDE, select Tools → Manage Libraries…​.
    Type NTPClient in the search field and install NTPClient by Fabrice Weinberg in the results.

    arduino lte examples ntpclient2 en
  3. Define the APN parameters required to register the modem to the LTE network.
    You will need to modify the values to match your SIM.

    #define APP_LTE_APN "apn" // replace your APN                       (1)
    
    /* APN authentication settings
     * Ignore these parameters when setting LTE_NET_AUTHTYPE_NONE.
     */
    #define APP_LTE_USER_NAME "user"     // replace with your username  (2)
    #define APP_LTE_PASSWORD  "password" // replace with your password  (3)
    1 Please enter the access point name that matches your SIM.
    2 Enter a username that matches your SIM.
    3 Enter a password that matches the SIM you are using.
  4. Set the RAT to use.
    Switch the comment out part according to the SIM you are using.

    /* RAT to use
     * Refer to the cellular carriers information
     * to find out which RAT your SIM supports.
     * The RAT set on the modem can be checked with LTEModemVerification::getRAT().
     */
    
    #define APP_LTE_RAT (LTE_NET_RAT_CATM) // RAT : LTE-M (LTE Cat-M1) (1)
    // #define APP_LTE_RAT (LTE_NET_RAT_NBIOT) // RAT : NB-IoT         (2)
    1 Use this definition when communicating by LTE-M (LTE Cat-M1).
    2 Use this definition when communicating by NB-IoT.
  5. In Tools → Port select the COM port of Spresense and perform the write operation.

  6. After compiling and writing (Upload) is finished, start the serial monitor.

  7. Make sure the baud rate is set to 115200 bps and you should see the message below.

    arduino lte examples ntpclient3 en

Depending on the environment, network communication may become unstable and may not work properly.
For more details, please refer to the FAQ.

3.5.4. Application Description

The application consists of three different parts: initialization, setup() and loop().

3.5.4.1. initialization
  • Include header files for the libraries used in this sketch.

    // libraries
    #include <NTPClient.h>  (1)
    #include <LTE.h>        (2)
    1 Include the <NTPClient.h> header file to use the Arduino NTP Client library to synchronize with the NTP server.
    2 Include the <LTE.h> header file to use the LTE library to register the modem to the LTE network.
  • Define the APN parameters required to register the modem to the LTE network.
    The values have to be replaced with the ones matching your SIM card.

    // APN data
    #define LTE_APN       "apn"      // replace your APN            (1)
    #define LTE_USER_NAME "user"     // replace with your username  (2)
    #define LTE_PASSWORD  "password" // replace with your password  (3)
    1 Please enter the access point name that matches your SIM.
    2 Enter a username that matches your SIM.
    3 Enter a password that matches the SIM you are using.
  • Define the NTP server name, time offset and time synchronization interval.

    #define NTP_SERVER_NAME "pool.ntp.org"                                                (1)
    #define TIME_OFFSET     (9 * 60 * 60) // time offset in seconds(This example is JST)  (2)
    #define UPDATE_INTERVAL 60000         // update interval in milliseconds              (3)
    1 Hostname of NTP server to synchronize
    2 Time offset value
    3 Time synchronization interval with NTP server
  • Instantiate the classes to be used.

    // initialize the library instance
    LTE lteAccess;                                                                (1)
    LTEUDP ntpUDP;                                                                (2)
    
    // You can specify the time server pool and the offset (in seconds, can be
    // changed later with setTimeOffset() ). Additionally you can specify the
    // update interval (in milliseconds, can be changed using setUpdateInterval() ).
    NTPClient timeClient(ntpUDP, NTP_SERVER_NAME, TIME_OFFSET, UPDATE_INTERVAL);  (3)
    1 Initializes an instance of the LTE class.
    2 Initializes an instance of the LTEUDP class.
    3 Initializes an instance of the NTPClient class.
3.5.4.2. setup()
  • The setup() function sets up the Serial and modem classes, and starts the UDP client.

    void setup()
    {
      // initialize serial communications and wait for port to open:
      Serial.begin(115200);                                                           (1)
      while (!Serial) {
          ; // wait for serial port to connect. Needed for native USB port only
      }
    
      Serial.println("Starting NTP client.");
    
      while (true) {
    
        /* Power on the modem and Enable the radio function. */
    
        if (lteAccess.begin() != LTE_SEARCHING) {                                     (2)
          Serial.println("Could not transition to LTE_SEARCHING.");
          Serial.println("Please check the status of the LTE board.");
          for (;;) {
            sleep(1);
          }
        }
    
        /* The connection process to the APN will start.
         * If the synchronous parameter is false,
         * the return value will be returned when the connection process is started.
         */
        if (lteAccess.attach(APP_LTE_RAT,
                             APP_LTE_APN,
                             APP_LTE_USER_NAME,
                             APP_LTE_PASSWORD,
                             APP_LTE_AUTH_TYPE,
                             APP_LTE_IP_TYPE) == LTE_READY) {                         (3)
          Serial.println("attach succeeded.");
          break;
        }
    
        /* If the following logs occur frequently, one of the following might be a cause:
         * - APN settings are incorrect
         * - SIM is not inserted correctly
         * - If you have specified LTE_NET_RAT_NBIOT for APP_LTE_RAT,
         *   your LTE board may not support it.
         * - Rejected from LTE network
         */
        Serial.println("An error has occurred. Shutdown and retry the network attach process after 1 second.");
        lteAccess.shutdown();                                                         (4)
        sleep(1);
      }
    
      timeClient.begin();                                                             (5)
    }
    1 Setup Serial for message display.
    2 Call the begin() method of the LTE class. Calling the begin() method will turn on the modem and search the network.
    When the network search is successfully started, the begin() method returns LTE_SEARCHING.
    3 Call the attach() method of LTE.
    Register the modem with the LTE network by calling the attach() method.
    When the modem is successfully registered to the LTE network, the attach() method returns LTE_READY.
    4 If the method does not return the expected return value, the LTE method shutdown() will be called.
    This will turn off the modem.
    5 Use the begin() method of NTPClient to connect to the NTP server using the UDP protocol.
3.5.4.3. loop()
  • The loop() function updates the synchronization with the NTP server and displays the synchronization time in the serial output.

    void loop() {
      timeClient.update();  (1)
    
      Serial.println(timeClient.getFormattedTime());
    
      sleep(1);
    }
    1 Use the NTPClient update() method to synchronize the time with the NTP server at the update interval of UPDATE_INTERVAL.

3.6. Publish GNSS positioning information to AWS server

3.6.1. Overview

This sample sketch uses the GNSS Library to publish acquired positioning data to the MQTT broker using the MQTT protocol.
The sketch will register the modem to the LTE network and then connect to the MQTT broker using Arduino MQTT Client library. It starts positioning using the GNSS library and publishes a message containing the location and time to the MQTT broker using the Arduino MQTT Client library.

If the LTE firmware version is RK_03_00_00_00_00_04121_001, the TLS communication API may not work. (Please refer to here )
Please update firmware to refer to the Updater Tool on the download site.

3.6.2. Environment requirements

  • Spresense main board

  • Spresense LTE extension board

  • SIM card

  • microSD card

For instructions on how to connect the boards, please refer to How to connect the Spresense main board to the Spresense LTE extension board.

This sketch requires a SIM card to be able to connect to the network.
Please check the Confirmed LTE operator list.

3.6.3. Procedure

  1. To use AWS IoT you first need to create an AWS account.

  2. For getting started with AWS IoT, please refer to Getting started with AWS IoT Core.

  3. Download the certificate to your PC.
    Download the certificate issued by following the guide describing how to create a thing object. The three files to download are as follows

    • Root certificate:Amazon_Root_CA_1.pem

    • Client certificate:c3c4ff2375.cert.pem

    • Private key:c3c4ff2375.private.key

      These file names are just examples. The actual names of the downloaded files may vary depending on your environment, so please change them accordingly.

  4. Download the certificate to your PC and save it to a microSD card.
    Create a directory named CERTS on the microSD card and put the three files in it.

  5. From the Arduino IDE, select File → Example Sketch → Example Sketch for Spresense LTE → LteGnssTracker to open the example sketch.

    arduino lte examples gnsstracker1 en
  6. From the Arduino IDE, select Tools → Manage Libraries…​.
    Type ArduinoMqttClient in the search field of the dialog and install ArduinoMqttClient by Arduino from the list of results.

    arduino lte examples gnsstracker2 en
  7. Define the APN parameters required to register the modem to the LTE network.
    You will have to modify these values to match your SIM card.

    #define APP_LTE_APN "apn" // replace your APN                       (1)
    
    /* APN authentication settings
     * Ignore these parameters when setting LTE_NET_AUTHTYPE_NONE.
     */
    #define APP_LTE_USER_NAME "user"     // replace with your username  (2)
    #define APP_LTE_PASSWORD  "password" // replace with your password  (3)
    1 Please enter the access point name that matches the SIM you are using.
    2 Enter the user name that matches the SIM you are using.
    3 Enter the password that matches the SIM you are using.
  8. Set the RAT to use.
    Switch the comment out part according to the SIM you are using.

    /* RAT to use
     * Refer to the cellular carriers information
     * to find out which RAT your SIM supports.
     * The RAT set on the modem can be checked with LTEModemVerification::getRAT().
     */
    
    #define APP_LTE_RAT (LTE_NET_RAT_CATM) // RAT : LTE-M (LTE Cat-M1) (1)
    // #define APP_LTE_RAT (LTE_NET_RAT_NBIOT) // RAT : NB-IoT         (2)
    1 Use this definition when communicating by LTE-M (LTE Cat-M1).
    2 Use this definition when communicating by NB-IoT.
  9. Describes the configuration parameters required to connect to the MQTT broker.

    The procedure for checking the MQTT broker name is as follows:

    1. Open the AWS IoT console.

    2. Select Manage and then Things. Select the IoT thing you previously created and select Interact. The endpoint will appear in the HTTPS section of the thing’s details page.

      // MQTT broker
      #define BROKER_NAME "your-MQTT-broker" // replace with your broker                     (1)
      #define BROKER_PORT 8883               // port 8883 is the default for MQTT over TLS.  (2)
      1 Describe the endpoint displayed in the [HTTPS] section.
      2 Specify the port to connect to. The default is 8883 and you can leave it unchanged in this procedure.
  10. Describe the parameters of the root certificate, client certificate, and private key required for the TLS protocol.
    Make sure the file names match those of the files saved on the microSD card.

    #define ROOTCA_FILE "CERTS/Amazon_Root_CA_1.pem"   // Define the path to a file containing CA           (1)
                                           // certificates that are trusted.
    #define CERT_FILE   "CERTS/c3c4ff2375.cert.pem" // Define the path to a file containing certificate     (2)
                                           // for this client, if required by the server.
    #define KEY_FILE    "CERTS/c3c4ff2375.private.key"  // Define the path to a file containing private key (3)
                                           // for this client, if required by the server.
    1 Describe the file name of the root certificate saved in the microSD card.
    2 Describe the file name of the client certificate saved in the microSD card.
    3 Describe the file name of the private key stored in the microSD card.

    The file names used above are just examples. Please replace with the names of the files you actually downloaded.

  11. Describe the topic that the sample application will publish.
    Checking the MQTT messages for a device in the AWS IoT MQTT client.

    // MQTT topic
    #define MQTT_TOPIC "spresense/gnss_tracker" // replace with your topic
  12. Select the COM port used by Spresense in Tools → Serial Port and perform the write operation.

  13. After compiling and flashing the sketch, start the serial monitor.

  14. Set baud rate to 115200 bps.
    When the modem successfully registers with the LTE network and completes the connection to the MQTT broker, the message You’re connected to the MQTT broker! will be displayed.
    After that, when positioning is completed, Position is fixed. will be displayed.
    After the positioning is completed, the device will publish the positioning data every second for a total of 60 times. If successful, a message will be displayed as shown below.

    arduino lte examples gnsstracker3 en

    If the terminal log does not proceed beyond the indication Attempting to connect to the MQTT broker:, your Spresense LTE extension board firmware may be out of date.
    Please update the Spresense LTE board firmware to refer Updater Tool .

    Depending on the operating environment, the communication may become unstable and not working correctly.
    For more details, please the FAQ.

  15. Check the AWS IoT console to see if the message has been published.

  16. Export the messages subscribed to in the AWS IoT console as a CSV file.

  17. The downloaded CSV file will be processed to create GPS logger data. You can use a dedicated tool such as a PC to view and display the trajectory of your movement on a map.

    • Extract only the payload part (from $GPGGA, to ,*xx) from the downloaded CSV. The extracted data should look similar to the example below, and there should be 60 such similar lines in total.

      $GPGGA,053156.05,3525.6925,N,13922.2242,E,1,08,1.1,47.8,M,,M,,*44
    • Check the GPS logger data.
      See GPS logger data checking procedure.

3.6.4. Program Description

The application consists of three functional parts: "initialization", "doAttach()", "setup()", and "loop()". These three operations will be explained in detail.

3.6.4.1. initialization
  • Include the header file for the library used in this sketch.

    // APN data
    #include <RTC.h>               (1)
    #include <SDHCI.h>             (2)
    #include <GNSS.h>              (3)
    #include <LTE.h>               (4)
    #include <ArduinoMqttClient.h> (5)
    1 Include <RTC.h> to use the RTC library to set the SPRESENSE system time.
    2 Include <SDHCI.h> to use the SD library to reference the certificate and private key required for the TLS protocol.
    3 Include <GNSS.h> to use the GNSS library for positioning.
    4 Include <LTE.h> to use the LTE library to register the modem to the LTE network.
    5 Include <ArduinoMqttClient.h> to use the Arduino MQTT Client library to publish messages via MQTT.
  • Define the APN parameters required to register the modem to the LTE network.
    You will need to modify the values according to your SIM.

    #define APP_LTE_APN "apn" // replace your APN                       (1)
    
    /* APN authentication settings
     * Ignore these parameters when setting LTE_NET_AUTHTYPE_NONE.
     */
    #define APP_LTE_USER_NAME "user"     // replace with your username  (2)
    #define APP_LTE_PASSWORD  "password" // replace with your password  (3)
    1 Please enter the access point name that matches your SIM.
    2 Enter the user name that matches your SIM.
    3 Enter the password that matches the SIM you are using.
  • Define the configuration parameters required for the MQTT broker connection.

    // MQTT broker
    #define BROKER_NAME "your-MQTT-broker" // replace with your broker                     (1)
    #define BROKER_PORT 8883               // port 8883 is the default for MQTT over TLS.  (2)
    1 Specify the MQTT broker name.
    2 Specify the port to connect to; the default for MQTT using the TLS protocol is 8883.
  • Define the storage location for the root certificate, client certificate, and private key required for the TLS protocol.

    #define ROOTCA_FILE "path/to/cafile"   // Define the path to a file containing CA           (1)
                                           // certificates that are trusted.
    #define CERT_FILE   "path/to/certfile" // Define the path to a file containing certificate  (2)
                                           // for this client, if required by the server.
    #define KEY_FILE    "path/to/keyfile"  // Define the path to a file containing private key  (3)
                                           // for this client, if required by the server.
    1 Insert the file name of the root certificate.
    2 Insert the file name of the client certificate.
    3 Insert the file name of the private key.

    For example, if you create a CERTS directory and save the certificate with the name cafile, write CERTS/cafile.

    You do not necessarily need to save the file to the microSD card; you can also specify a file saved in SPI-Flash.
    In that case, include <Flash.h> and change the object name from theSD to Flash in the sketch.
    Removes the process of waiting for the mounting to complete in theSD.begin.

  • Describe the topic that the sample application will publish.

    // MQTT topic
    #define MQTT_TOPIC "spresense/gnss_tracker" // replace with your topic
  • Define the interval and number of messages to be published to the MQTT broker.

    // MQTT publish interval settings
    #define PUBLISH_INTERVAL_SEC   1   // MQTT publish interval in sec (1)
    #define MAX_NUMBER_OF_PUBLISH  60  // Maximum number of publish    (2)
    1 Insert the interval (in seconds) determining how often a message is should be published to the MQTT broker.
    2 Insert the number of messages to publish to the MQTT broker. When this limit is reached, the Publish operation will be terminated.
  • Initialize an instance of the classes to be used.

    // initialize the library instance
    LTE lteAccess;
    LTETLSClient client;           (1)
    MqttClient mqttClient(client); (2)
    SDClass theSD;
    SpGnss Gnss;
    1 Initializes an instance of LTETLSClient.
    2 Provide the LTETLSClient instance when initializing an instance of MqttClient.

    By changing from the LTETLSClient class to the LTEClient class, it is possible to operate with plain MQTT.
    Use port number 1883 when connecting using plain MQTT. Also, when using plain MQTT the process of setting up a certificate is no longer necessary.

3.6.4.2. doAttach()

This section describes the doAttach() function.

  • Start the registration process of the modem to the LTE network.

    while (true) {
    
      /* Power on the modem and Enable the radio function. */
    
      if (lteAccess.begin() != LTE_SEARCHING) {                      (1)
        Serial.println("Could not transition to LTE_SEARCHING.");
        Serial.println("Please check the status of the LTE board.");
        for (;;) {
          sleep(1);
        }
      }
    
      /* The connection process to the APN will start.
        * If the synchronous parameter is false,
        * the return value will be returned when the connection process is started.
        */
      if (lteAccess.attach(APP_LTE_RAT,
                            APP_LTE_APN,
                            APP_LTE_USER_NAME,
                            APP_LTE_PASSWORD,
                            APP_LTE_AUTH_TYPE,
                            APP_LTE_IP_TYPE,
                            false) == LTE_CONNECTING) {              (2)
        Serial.println("Attempting to connect to network.");
        break;
      }
    
      /* If the following logs occur frequently, one of the following might be a cause:
        * - APN settings are incorrect
        * - SIM is not inserted correctly
        * - If you have specified LTE_NET_RAT_NBIOT for APP_LTE_RAT,
        *   your LTE board may not support it.
        */
      Serial.println("An error has occurred. Shutdown and retry the network attach preparation process after 1 second.");
      lteAccess.shutdown();                                          (3)
      sleep(1);
    }
    1 Use lteAccess.begin() to start exploring the LTE network. The lteAccess.begin() method will be repeated until it starts exploring the LTE network.
    2 Use lteAccess.attach() to register the modem with the LTE network. Set the sixth argument, synchronous, to false to perform the registration process asynchronously. This is because the GNSS initialization process is done in parallel with the modem being registered. When the modem registration is correctly started, the loop process will be exited.
    3 If the modem LTE network registration fails, use the lteAccess.shutdown() method to turn off the modem, wait for a second or so using the sleep() function and then resume the LTE network search process.
3.6.4.3. setup()

This section describes the setup() function.

  • Setup Serial to display log messages.

    // Open serial communications and wait for port to open
    Serial.begin(115200);
    while (!Serial) {
        ; // wait for serial port to connect. Needed for native USB port only
    }
    
    Serial.println("Starting GNSS tracker via LTE.");
  • Wait for the microSD card to be mounted.

    /* Initialize SD */
    while (!theSD.begin()) {
      ; /* wait until SD card is mounted. */
    }
  • Call the doAttach () function and start the registration process of the modem to the LTE network.

    /* Connect LTE network */
    doAttach();
  • Execute the GNSS initialization process.

    int result;
    
    /* Activate GNSS device */
    result = Gnss.begin();
    assert(result == 0);
    
    /* Start positioning */
    result = Gnss.start();
    assert(result == 0);
    Serial.println("Gnss setup OK");
  • Wait for the modem to register with the LTE network.

      while(LTE_READY != lteAccess.getStatus()) { (1)
        sleep(1);
      }
      Serial.println("attach succeeded.");
    1 Use the lteAccess.getStatus() method to get the status of the modem. Exit the loop when the modem status transitions to LTE_READY.
  • Synchronize the SPRESENSE time with the network and output the set time to the serial output.

    void printClock(RtcTime &rtc)                        (1)
    {
      printf("%04d/%02d/%02d %02d:%02d:%02d\n",
             rtc.year(), rtc.month(), rtc.day(),
             rtc.hour(), rtc.minute(), rtc.second());
    }
    
    void setup()
    {
      :
      // Set local time (not UTC) obtained from the network to RTC.
      RTC.begin();
      unsigned long currentTime;
      while(0 == (currentTime = lteAccess.getTime())) { (2)
        sleep(1);
      }
      RtcTime rtc(currentTime);                         (3)
      printClock(rtc);                                  (4)
      RTC.setTime(rtc);                                 (5)
      :
    }
    1 Define the printClock() function to output the time given in the argument to serial.
    2 Use the lteAccess.getTime() method to get the local time (in seconds from the source).
    3 Create an instance of RtcTime with the local time.
    4 Outputs the local time to serial using the printClock() function.
    5 Use the RTC.setTime() method to set the local time to SPRESENSE.
  • Load the root certificate, client certificate and private key saved in the microSD card and connect to the MQTT broker.

    // Set certifications via a file on the SD card before connecting to the MQTT broker
    File rootCertsFile = theSD.open(ROOTCA_FILE, FILE_READ);    (1)
    client.setCACert(rootCertsFile, rootCertsFile.available()); (2)
    rootCertsFile.close();
    
    File certsFile = theSD.open(CERT_FILE, FILE_READ);          (3)
    client.setCertificate(certsFile, certsFile.available());    (4)
    certsFile.close();
    
    File priKeyFile = theSD.open(KEY_FILE, FILE_READ);          (5)
    client.setPrivateKey(priKeyFile, priKeyFile.available());   (6)
    priKeyFile.close();
    
    Serial.print("Attempting to connect to the MQTT broker: ");
    Serial.println(broker);
    
    if (!mqttClient.connect(broker, port)) {                    (7)
      Serial.print("MQTT connection failed! Error code = ");
      Serial.println(mqttClient.connectError());
      // do nothing forevermore:
      for (;;)
        sleep(1);
    }
    
    Serial.println("You're connected to the MQTT broker!");
    1 Use theSD.open() method to create a File instance of the root certificate.
    2 Set the root certificate to the modem by specifying the File instance of the root certificate in the client.setCACert() method.
    3 Create a File instance of the client certificate using theSD.open() method.
    4 Set the client certificate to the modem by specifying the File instance of the client certificate in the client.setCertificate() method.
    5 Create a File instance of the private key using theSD.open() method.
    6 Set the private key to the modem by specifying the File instance of the private key in the client.setPrivateKey() method.
    7 Use the mqttClient.connect() method to connect to the MQTT broker.
3.6.4.4. loop()

This section explains the loop function.

Wait for the GNSS connection to be established and obtain positioning data.

+

void loop()
{
  /* Check update. */
  if (Gnss.waitUpdate(-1)) {   (1)
    /* Get navData. */
    SpNavData navData;
    Gnss.getNavData(&navData); (2)
1 Wait for the GNSS connection using the Gnss.waitUpdate() function.
2 Get the positioning result using the Gnss.getNavData() function.
  • Determine if the device position is retrieved. If it is, publish it to the MQTT broker.

    bool posFix = ((navData.posDataExist) && (navData.posFixMode != 0)); (1)
    if (posFix) {
      Serial.println("Position is fixed.");
      String nmeaString = getNmeaGga(&navData);                          (2)
      if (strlen(nmeaString.c_str()) != 0) {
        unsigned long currentTime = lteAccess.getTime();
        if (currentTime >= lastPubSec + PUBLISH_INTERVAL_SEC) {          (3)
          // Publish to broker
          Serial.print("Sending message to topic: ");
          Serial.println(topic);
          Serial.print("Publish: ");
          Serial.println(nmeaString);
    
          // send message, the Print interface can be used to set the message contents
          mqttClient.beginMessage(topic);                                (4)
          mqttClient.print(nmeaString);                                  (5)
          mqttClient.endMessage();                                       (6)
          lastPubSec = currentTime;
          numOfPubs++;
        }
      }
    } else {
      Serial.println("Position is not fixed.");
    }
3 Determine if the navData obtained from the Gnss.getNavData() function contains a valid position.
4 Get a GPGGA sentence in NMEA 0183 format using the getNmeaGga() function.
5 Obtain the current time using the lteAccess.getTime() function. Publishing to broker if the obtained time is past the time of the last publish + publish interval.
6 Construct an MQTT message for the target topic name using the mqttClient.beginMessage() function.
7 Use the mqttClient.print() function to set the GPGGA sentence in the MQTT message.
8 Publish the constructed MQTT message to the MQTT broker using mqttClient.endMessage().

3.7. How to export a root certificate

3.7.1. Overview

To access a server using HTTPS or other secure communication via TLS, the root certificate must be downloaded and copied to a location specified by the application.

Spresense supports the following certificates

  • DER encoded binary X.509 (DER)

    Binary certificate file in DER format. The extension is .cer.

  • Base 64 encoded X.509 (PEM)

    ASCII text certificate file in Base 64 format. The extension is .pem / .cer.

The size of the certificate must be less than 1999 bytes. If it exceeds 2000 bytes in PEM text format, please use a certificate in DER binary format.

This section describes how to download a root certificate for server access.

3.7.2. How to download

Follow the steps below to download the file. The following procedure uses Windows/Chrome, but you can follow the same procedure for other OS and browsers to download the file by selecting the supported format.

  1. Open the page of the site you wish to access. + . Open the page you wish to access in your browser. The following is an example of the page you want to access: https://example.com/.

    tutorial lte tls https open
  2. Open the site’s certificate.

    1. Click the Security Communication button displayed in the page URL section.

    2. Open the Security Protection menu.

    3. Open a valid certificate.

      tutorial lte tls certification view en
  3. Open the root certificate.

    1. Open the certificate path (hierarchy).

    2. Selects the Root certificate.

    3. Click View Certificate.

      tutorial lte tls root certification view en
  4. Copy (export) the root certificate.

    1. Select Copy to File…​.

      tutorial lte tls root certification export 1 en
    2. Navigate to the page where you select the format of the export file.

      tutorial lte tls root certification export 2 en
    3. Select the format of the file to be exported.

      Select DER encoded binary X.509 or Base 64 encoded X.509 as supported by Spresense.

      tutorial lte tls root certification export 3 en
    4. Select the export destination.

      tutorial lte tls root certification export 4 en
    5. Execute the export.

      tutorial lte tls root certification export 5 en

By performing the above steps, the root certificate is copied (exported). The copied root certificate can then be copied to the location specified in the application to enable secure communication with Spresense.

4. Spresense Arduino multi-core environment

This section describes the multi-core programming development environment using Arduino IDE.

In Arduino multi-core environments, please use the MultiCore MP library. The application CPU has total of 6 cores, consisting of 1 MainCore and 5 SubCore. In the normal single core application, the user programs only use MainCore. In the multi-core application environment described here, you can program all cores including SubCore.

MainCore

MainCore is always started at power on.

SubCore(s)

SubCore(s) are launched from MainCore. There are total of 5 SubCores.

In this tutorial, we show two examples with the MultiCore MP library.

Multicore Boot Example

Boot 4 SubCores and perform LED blinking using multicore.
By running this sample, you can understand the procedure to boot SubCore.

Multicore MessageHello Example

Communicate messages between MainCore and SubCore.
By running this sample, you can understand how to communicate between MainCore and SubCore.

4.1. Multicore Boot example

The main steps are:

  1. Open a sketch using Arduino IDE

  2. Select core

  3. Compile & upload

Repeat above for the number of cores you want to program.

Please follow the procedure as below.

  • Launch Arduino IDE from your desktop
    (If there is no icon on your desktop, please launch from the start menu etc.)

    arduino multicore desktop
    In multi-core programming, please note when you launch multiple Arduino IDE windows.
    If you open a new window with File→ New from the Arduino IDE menu, if you switch the menu of Core selection in one window, the setting is reflected to all the windows.
    As a workaround, when you open multiple Arduino IDE windows for every Core, please launch Arduino IDE from the icon on your desktop. Then, menu settings such as Core selection can be set independently for each window.
  • MainCore programming

    • Open a sketch from Arduino IDE menu File → Examples → MultiCore MP → Boot → Main.

      arduino multicore boot0
      arduino multicore boot1 main
    • Explanation for Main.ino sketch

      MP.begin(subid) boot 4 SubCores.

      arduino multicore boot4 main
    • Select MainCore from Arduino IDE menu Tools → Core → MainCore.

      arduino multicore boot2 main

      You can see that MainCore is selected in the status bar.

      arduino multicore boot3 main
      In Arduino IDE from 1.8.9 to 1.8.13, there is an issue, the status bar is not displayed correctly after switching menu.
    • Press the Upload button to do Compile & Upload.

      arduino multicore boot5 main
      You cannot upload multiple core programs from multiple windows at the same time.
      Upload each core one by one so that the upload operation is not executed simultaneously.
    • Size of used memory is displayed on the log of compilation result.

      MainCore always uses 768 KByte of memory.

      arduino multicore boot6 main

      If Upload is successful, MainCore programming is completed.

      • As a test, try to open the serial monitor without SubCore program.
        Since there is no SubCore, you can see that MP.begin(1~4) returns error.

        arduino multicore boot7 serial err

        If the serial monitor is open, uploading from other windows may fail.
        Please close the serial monitor after this trial.

Then we will program the SubCore.

  • SubCore programming

    • Open a sketch from Arduino IDE menu File → Examples → MultiCore MP → Boot → Sub1.

      arduino multicore boot0
      arduino multicore boot1 sub1
    • Explanation for Sub1.ino sketch

      In setup(), by calling MP.begin(), notify MainCore that startup is complete.
      In loop(), output log by MPLog() and blink a LED.

      arduino multicore boot4 sub1
    • Select SubCore 1 from Arduino IDE menu Tools → Core → SubCore 1.

      arduino multicore boot2 sub1

      You can see that SubCore 1 is selected in the status bar.

      arduino multicore boot3 sub1
      In Arduino IDE from 1.8.9 to 1.8.13, there is an issue, the status bar is not displayed correctly after switching menu.
    • Press the Upload button to do Compile & Upload.

      arduino multicore boot5 sub1
      You cannot upload multiple core programs from multiple windows at the same time.
      Upload each core one by one so that the upload operation is not executed simultaneously.
    • Size of used memory is displayed on the log of compilation result.

      You can see that this SubCore sample sketch uses 128 KBytes of memory.
      This size increases or decreases depending on the user program.

      arduino multicore boot6 sub1

      If Upload is successful, SubCore 1 programming is completed.

  • Follow the same procedure for SubCore 2, 3 and 4 and execute Compile & Upload.

    • Open Sub2 sketch, select SubCore 2, and Compile & Upload.

    • Open Sub3 sketch, select SubCore 3, and Compile & Upload.

    • Open Sub4 sketch, select SubCore 4, and Compile & Upload.

  • Operation check

    • Blink 4 LEDs on the Spresense main board.

    • Open the serial monitor, and you can see the log from each core.

      arduino multicore boot7 serial
  • Summary

    • We have learned how to boot SubCore using a simple example sketch.
      Add a call to MP.begin() when you port an existing sketch to multi-core environment.

    • The basic usage, such as Arduino IDE’s Compile & Upload method, is the same as usual in multi-core environments. The main difference is that a menu of core selections has been added in Tools→ Core.

4.2. Multicore MessageHello example

The main steps are the same as Multicore Boot example.
In this example, MainCore and SubCore(s) use a common sketch.

  1. Open a sketch using Arduino IDE (only once)

  2. Select core

  3. Compile & upload

Repeat above for the number of cores you want to program. This example runs all SubCores.

Please follow the procedure as below.

  • Launch Arduino IDE from your desktop
    (If there is no icon on your desktop, please launch from the start menu etc.)

    arduino multicore desktop
    In multi-core programming, please note when you launch multiple Arduino IDE windows.
    If you open a new window with File→ New from the Arduino IDE menu, if you switch the menu of Core selection in one window, the setting is reflected to all the windows.
    As a workaround, when you open multiple Arduino IDE windows for every Core, you always launch Arduino IDE from the icon on your desktop. By doing so, menu settings such as Core selection can be set independently for each window.
  • MainCore programming

    • Open a sketch from Arduino IDE menu File → Examples → MultiCore MP → Message → MessageHello.

      arduino multicore boot0
      arduino multicore hello1
    • Explanation for MessageHello.ino sketch

      We use #ifdef SUBCORE ~ #else ~ #endif to switch between MainCore and SubCore implementations in this sketch.

      • MainCore implementation

        In setup(), MP.begin(subid) boot 5 SubCores
        By calling MP.RecvTimeout(MP_RECV_POLLING), set polling in receiver mode. In this mode, when MP.Recv() is called, Recv() do polling with non-blocking.
        In loop(), MP.Recv() receives the packet address from each SubCore, and outputs the message in the packet.

        arduino multicore hello4 main
      • SubCore implementation

        In setup(), by calling MP.begin(), notify MainCore that startup is complete.
        In loop(), put "Hello" message into packet, and send the packet address by MP.Send().

        arduino multicore hello4 sub
    • Select MainCore from Arduino IDE menu Tools → Core → MainCore.

      arduino multicore hello2 main

      You can see that MainCore is selected in the status bar.

      arduino multicore boot3 main
      In Arduino IDE from 1.8.9 to 1.8.13, there is an issue, the status bar is not displayed correctly after switching menu.
    • Press the Upload button to do Compile & Upload.

      arduino multicore hello5 main
      You cannot upload multiple core programs from multiple windows at the same time.
      Upload each core one by one so that the upload operation is not executed simultaneously.
    • Size of used memory is displayed on the log of compilation result.

      MainCore always uses 768 KByte of memory.

      arduino multicore boot6 main

      If Upload is successful, MainCore programming is completed.

Then we will program the SubCore.

  • SubCore programming

    • Select SubCore 1 from Arduino IDE menu Tools → Core → SubCore 1.

      arduino multicore hello2 sub1

      You can see that SubCore 1 is selected in the status bar.

      arduino multicore boot3 sub1
      In Arduino IDE from 1.8.9 to 1.8.13, there is an issue, the status bar is not displayed correctly after switching menu.
    • Press the Upload button to do Compile & Upload.

      arduino multicore hello5 main
      You cannot upload multiple core programs from multiple windows at the same time.
      Upload each core one by one so that the upload operation is not executed simultaneously.
    • Size of used memory is displayed on the log of compilation result.

      You can see that this SubCore sample sketch uses 128 KBytes of memory.
      This size increases or decreases depending on the user program.

      arduino multicore boot6 sub1

      If Upload is successful, SubCore 1 programming is completed.

  • Follow the same procedure for SubCore 2, 3, 4 and 5 and execute Compile & Upload.

    • Select SubCore 2, and Compile & Upload.

    • Select SubCore 3, and Compile & Upload.

    • Select SubCore 4, and Compile & Upload.

  • Operation check

    • Open the serial monitor.

    • MainCore receives the message sent from each SubCore and outputs them to the serial monitor.

      arduino multicore hello7 serial
  • Summary

    • We have learned how to communicate between MainCore and SubCore.

    • It can communicate by sending the address of message because of the shared memory.

    • You can use a common sketch for multi-core programming.

4.3. Notices

  • See developer guide for the details of MultiCore MP library.

  • If you want to remove SubCore binaries on the Spresense board, please do Install Bootloader. When installing the loader, remove all SubCore binaries and then start installing the loader.

    arduino multicore remove subcore

    Note that SubCore does not run unless you call MP.begin() from MainCore. Therefore, there is no problem even if the old SubCore remains on your Spresense board.

  • If you open multiple Arduino IDE windows for every Core, please launch Arduino IDE from the icon on your desktop. Then, menu settings such as Core selection can be set independently for each window.

  • Use Arduino IDE to open multiple windows and select core is complicated, but you can use command line tools such as arduino-builder and arduino-cli. Also, you can upload the multiple binaries for all cores at once by using flash_writer tool. How to use the command line tools will be updated in this manual later.

5. Tutorial of SignalProcessing Library

The SignalProcessing library has FFT/IIR filters and the following sample sketches are provided for them.

This document explains how to execute these sample sketches.

5.1. PeakDetector

5.1.1. Overview

This sample sketch detects the peak frequency in real time by performing real-time frequency analysis with the FFT library on the sound data input by the microphone. Please refer to Audio library for the Audio function.

In addition, this sample performs with offloading the signal processing library to SubCore. Please see also MultiCore MP Library.

5.1.2. Operating environment

  • Spresense Main & extension Board

  • Microphones for recording

Please refer to the following hardware guide for microphone connection.

This sample is the multi-channels application with 4 channels.
Connect microphones (up to 4@analog, 8@digital) and please adjust them within the application.

5.1.3. Operating procedure

The main operation procedure is as follow:

  1. Compile & upload SubCore for signal processing

  2. Compile & upload MainCore for sound acquisition & application processing

  3. Running the application

If you compile and upload MainCore at first , it will call SubCore after startup, resulting in an error.
If you have already compiled and uploaded to SubCore, the program of SubCore that has already been uploaded will start after starting MainCore.
5.1.3.1. Compilation & upload of SubCore
  1. Open the sample sketch by selecting File→ Sketch example→ Sketch example for Spresense SignalProcessing→ PeakDetector→ SubFFT from Arduino IDE.

  2. Select Tools→ Core→ SubCore1 (Tools→ Core→ SubCore1) from the Arduino IDE menu.

    In this sample, the offloaded SubCore is SubCore1.

Select the COM port of Spresense with Tools→ Serial Port and write to the microcomputer board.

+ NOTE: The SubCore program waits for instructions from the MainCore after starting.

Now, the preparation on the SubCore is OK.

5.1.3.2. Compilation & upload of MainCore
  1. Open the sample sketch by selecting File→ Sketch example→ Sketch example for Spresense SignalProcessing→ PeakDetector → MainAudio from Arduino IDE.

  2. Select Tools→ Core→ MainCore (Tools→ Core→ MainCore) from the Arduino IDE menu

    Write to the microcomputer board.

If you have selected the serial port for uploading SubCore, you do not need to select it again.

Now, the preparation on the MainCore is also OK.

5.1.3.3. Run the application
  1. After uploading MainCore , open Tools→ Serial Monitor and start the application.

  2. When the application is started, it analyzes the frequency of the sound of each channel input from the microphone and continues to display the peak frequency on the serial monitor.

5.1.4. the Explanation of program

  • The number of input audio channels is specified in the following part of MainAudio.ino.

/* Select mic channel number */
//const int mic_channel_num = 1;
//const int mic_channel_num = 2;
const int mic_channel_num = 4;

NOTE : #define MAX_CHANNEL_NUM 4 on the SubCore means the max number of channels is 4, user can select 1-4 channels dynamically.

This sketch calls FFT.begin(); without arguments at SubCore, so the execution number of channels on SubCore is equal to the number of maximum channels.
If you don’t want the number of execution channels to be equal to the maximum number of channels, Please call FFT.begin (WindowHamming, "The number of execution channels", (FFTLEN / 2));, and specify the number of execution channels.
  • The number of FFT taps is specified as follow in SubFFT.ino.

/* Select FFT length */

//#define FFT_LEN 32
//#define FFT_LEN 64
//#define FFT_LEN 128
//#define FFT_LEN 256
//#define FFT_LEN 512
#define FFT_LEN 1024
//#define FFT_LEN 2048
//#define FFT_LEN 4096
  • The current window function and the number of FFT overlap samples are set to default. The default for the window is Hamming window and the overlap size is designed to shift the window by the number of taps/2.

If you want to change this, please change it in the SubFFT.ino from

  FFT.begin();

to

  FFT.begin ("window function you want to use", "number of execution channels", "number of samples you want to overlap");

5.1.5. Precautions regarding changing the number of TAPs

When changing the number of FFT TAPs, the following precautions are required regarding the CPU processing performance and SubCore memory usage for real-time processing.

  • If you want to set the number of TAPs to 4096.

If you want to operate on 4 channels, you should change it in SubFFT.ino from

USER_HEAP_SIZE (64 * 1024);

To

USER_HEAP_SIZE (64 * 1024 * 5);

In this case, default SubCore memory will not be enough, so you need to select Tools→ Memory→ 640kB from the Arduino IDE to set the MainCore memory size to 640kB.

If you want to operate on 2 channels, you do not need to change MainCore memory size, and only change it in SubFFT.ino to

USER_HEAP_SIZE(64 * 1024 * 4);
  • If you want to set the number of TAPs to 2048.

You should change it in SubFFT.ino from

USER_HEAP_SIZE (64 * 1024);

To

USER_HEAP_SIZE (64 * 1024 * 4);
  • If you want to set the number of TAPs to 256.

Due to the display performance conditions on the MainCore side, only up to 2 channels can be operated.

5.2. SoundDetector

5.2.1. Overview

This sample sketch uses the FFT library for real-time frequency analysis of the sound data input by the microphone, and detects whether sound with a certain sound pressure level is input within the specified frequency range. Please refer to Audio library for the Audio function.

In addition, this sample performs with offloading the signal processing library to SubCore. Please see also MultiCore MP Library.

5.2.2. Operating environment

  • Spresense Main & extension Board

  • Microphones for recording

Please refer to the following hardware guide for microphone connection.

This sample is the multi-channels application with 4 channels.
Connect microphones (up to 4@analog, 8@digital) and please adjust them within the application.

5.2.3. Operating procedure

The main operation procedure is as follow:

  1. Compile & upload SubCore for signal processing

  2. Compile & upload MainCore for sound acquisition & application processing

  3. Running the application

If you compile and upload MainCore at first , it will call SubCore after startup, resulting in an error.
If you have already compiled and uploaded to SubCore, the program of SubCore that has already been uploaded will start after starting MainCore.
5.2.3.1. Compilation & upload of SubCore
  1. Open the sample sketch by selecting File→ Sketch example→ Sketch example for Spresense SignalProcessing→ SoundDetector → SubFFT from Arduino IDE.

  2. Select Tools→ Core→ SubCore1 (Tools→ Core→ SubCore1) from the Arduino IDE menu.

    In this sample, the offloaded SubCore is SubCore1.
  3. Select the COM port of Spresense with Tools→ Serial Port and write to the microcomputer board.

    The SubCore program waits for instructions from the MainCore after starting.

Now, the preparation on the SubCore is OK.

5.2.3.2. Compilation & upload of MainCore
  1. Open the sample sketch by selecting File→ Sketch example→ Sketch example for Spresense SignalProcessing→ SoundDetector→ MainAudio from Arduino IDE.

  2. Select Tools→ Core→ MainCore ( Tools→ Core→ MainCore) from the Arduino IDE menu

    Write to the microcomputer board.

If you have selected the serial port for uploading SubCore, you do not need to select it again.

Now, the preparation on the MainCore is also OK.

5.2.3.3. Run the application
  1. After uploading MainCore, open Tools→ Serial Monitor and start the application.

  2. When the application is started, frequency analysis is performed on the sound of each channel input from the microphone, and if sound with more than specified sound power is input within the specified frequency range, it shows the detection and channel numbernel number like > Found channel 0.

5.2.4. the Explanation of program

  • The number of input audio channels is specified in the following part of MainAudio.ino.

/* Select mic channel number */
//const int mic_channel_num = 1;
//const int mic_channel_num = 2;
const int mic_channel_num = 4;

NOTE : #define MAX_CHANNEL_NUM 4 on the SubCore means the max number of channels is 4, user can select 1-4 channels dynamically.

This sketch calls FFT.begin(); without arguments at SubCore, so the execution number of channels on SubCore is equal to the number of maximum channels.
If you don’t want the number of execution channels to be equal to the maximum number of channels, Please call FFT.begin (WindowHamming, "The number of execution channels", (FFTLEN / 2));, and specify the number of execution channels.
  • The number of FFT taps is specified as follow in SubFFT.ino.

/* Select FFT length */

//#define FFT_LEN 32
//#define FFT_LEN 64
//#define FFT_LEN 128
//#define FFT_LEN 256
//#define FFT_LEN 512
#define FFT_LEN 1024
//#define FFT_LEN 2048
//#define FFT_LEN 4096
  • The current window function and the number of FFT overlap samples are set to default. The default is the window is Hamming window and the overlap size is designed to shift the window by the number of taps/2.

If you want to change this, please change it in the SubFFT.ino from

  FFT.begin();

to

  FFT.begin ("window function you want to use", "number of execution channels", "number of samples you want to overlap");
  • The detector settings are as follow:

#define POWER_THRESHOLD 30 // Power
#define LENGTH_THRESHOLD 30 // 20ms
#define INTERVAL_THRESHOLD 100 // 100ms

#define BOTTOM_SAMPLING_RATE 1000 // 1kHz
#define TOP_SAMPLING_RATE 1500 // 1.5kHz
  • Specify the lower limit of frequency range for the detection with BOTTOM_SAMPLING_RATE. Set the Hz value.

  • Specify the upper limit of frequency range for the detection with TOP_SAMPLING_RATE. Set the Hz value.

  • Specify the threshold of the sound power for the detection with POWER_THRESHOLD.

  • If audio in the relevant frequency range continues to be input for more than LENGTH_THRESHOLD time, it is judged that the sound is detected.

  • If it is re-detected within INTERVAL_THRESHOLD after being detected, it is considered to be a part of the previously detected sound. Also, if it is detected more than INTERVAL_THRESHOLD away, another sound is detected and the notification is raised to MainCore again.

5.2.5. Precautions regarding changing the number of TAPs

When changing the number of FFT TAPs, the following precautions are required regarding the CPU processing performance and SubCore memory usage for real-time processing.

  • If you want to set the number of TAPs to 4096.

If you want to operate on 4 channels, you should change it in SubFFT.ino from

USER_HEAP_SIZE (64 * 1024);

To

USER_HEAP_SIZE (64 * 1024 * 5);

In this case, SubCore memory will not be enough on default, so you need to select Tools→ Memory→ 640kB from the Arduino IDE to set the MainCore memory size to 640kB.

If you want to operate on 2 channels, you do not need to change MainCore memory size, and only change it in SubFFT.ino from

USER_HEAP_SIZE (64 * 1024);

to

USER_HEAP_SIZE(64 * 1024 * 4);
  • If you want to set the number of TAPs to 2048.

You should change it in SubFFT.ino from

USER_HEAP_SIZE (64 * 1024);

To

USER_HEAP_SIZE (64 * 1024 * 4);

5.3. LowPassSound

5.3.1. Overview

This sample sketch outputs the sound data that is LPF (low-pass filter) by IIR.

In addition, this sample performs with offloading the signal processing library to SubCore. Please see also MultiCore MP Library.

5.3.2. Operating environment

  • Spresense Main & extension Board

  • Headphones or speakers for playback

  • Microphones for recording

In this sample, headphones and a microphone are used. Please connect headphones or active speakers to the headphone jack on the extension board.
Please refer to the following hardware guide for microphone connection.

The samples provided are for a monorail microphone. If you want more microphones, Connect microphones (up to 4@analog, 8@digital) and please adjust them within the application.
Audio output can only be up to stereo. If the input is 3 channels or more, perform downmix processing before output, convert to 2 channels data.

5.3.3. Operating procedure

The main operation procedure is as follow:

  1. Compile & upload SubCore for signal processing

  2. Compile & upload MainCore for sound acquisition & application processing

  3. Running the application

If you compile and upload MainCore at first, it will call SubCore after startup, resulting in an error.
If you have already compiled and uploaded to SubCore, the program of SubCore that has already been uploaded will start after starting MainCore.
5.3.3.1. Compilation & upload of SubCore
  1. Open the sample sketch by selecting File→ Sketch example→ Sketch example for Spresense SignalProcessing→ HighPassSound / LowPassSound→ SubFFT from Arduino IDE.

  2. Select Tools→ Core→ SubCore1 (Tools→ Core→ SubCore1) from the Arduino IDE menu.

    In this sample, the offloaded SubCore is SubCore1.
  3. Select the COM port of Spresense with Tools→ Serial Port and write to the microcomputer board.

    The SubCore program waits for instructions from the MainCore after starting.

Now, the preparation on the SubCore is OK.

5.3.3.2. Compilation & upload of MainCore
  1. Open the sample sketch by selecting File→ Sketch example→ Sketch example for Spresense SignalProcessing→ LowPassSound → MainAudio from Arduino IDE.

  2. Select Tools→ Core→ MainCore from the Arduino IDE menu

    Write to the microcomputer board.

If you have selected the serial port for uploading SubCore, you do not need to select it again.

Now, the preparation on the MainCore is also OK.

5.3.3.3. Run the application
  1. After uploading MainCore, open Tools→ Serial Monitor and start the application.

  2. When the application is started, the audio data input from the microphone will be the audio data that has been LPF/HPF processed by IIR, and output it from speaker.

5.3.4. the Explanation of program

  • The number of input audio channels is specified in the following part of MainAudio.ino.

static const int32_t channel_num  = AS_CHANNEL_MONO;
static const int32_t bit_length   = AS_BITLENGTH_16;
static const int32_t frame_sample = 768;
Only the 16 bit mode is supported for bit length.
The number of channels and the number of frame samples must match the SubCore side.
When outputting audio, it must be converted to interleaved stereo data.
Therefore, in this sample, after signal processing, monaural data is converted to stereo data.
  • The cutoff frequency and Q value are set by the following part of SubCore.

const int   g_channel = 1; /* Number of channels */
const int   g_cutoff  = 1000; /* Cutoff frequency */
const float g_Q       = sqrt(0.5); /* Q Value */

In this sample, the cutoff frequency is set to 1kHz and the Q value is set to sqrt(0.5) .
If nothing is specified with the number of samples, the number of samples in the output frame will be the default value of 768 samples.
If nothing is specified with the sampling rate, the sampling rate will be the default value of 48kHz.
If you do not specify an output format, the default format is per-channel output Planar.

In this sample, sound is collected by FrontEndObject.
Due to HW restrictions, Spresense’s audio function uses 48kHz sampling or 192kHz sampling for all input and output.
When realizing an input / output application that involves audio signal processing, if internal processing is performed at a sampling rate such as 16kHz, resampling with SW is required twice, which requires a lot of processor power and memory.
In the case of real time audio processing using the IIR filter this time, processing at 16KHz is not recommended due to insufficient memory.
  • When acquiring data after signal processing, if the output format is Planar, call get multiple times for the number of channels to acquire the multi channel data.

    for (int i = 0; i < g_channel; i++) {
      cnt = LPF.get(pDst, i);
      /* Channels must be stereo for Mixer */
      if(i == 0) {
        for(int j = 0; j < cnt; j++) {
          out_buffer[pos][j * 2]     = pDst[j];
          out_buffer[pos][j * 2 + 1] = 0;
        }
      }
    }
It is necessary to get areas for each channel to write multi channel data.
In order to output audio, this sample is converted to interleaved stereo data. In this sample, the R channel is generated with 0.

5.4. HighPassSound

5.4.1. Overview

This sample sketch outputs the sound data that is perform HPF (high-pass filter) by IIR.

In addition, this sample performs with offloading the signal processing library to SubCore. Please see also MultiCore MP Library.

5.4.2. Operating environment

  • Spresense Main & extension Board

  • Headphones or speakers for playback

  • Microphones for recording

In this sample, headphones and a microphone are used. Please connect headphones or active speakers to the headphone jack on the extension board.
Please refer to the following hardware guide for microphone connection.

The samples provided are for a monorail microphone. If you want more microphones, Connect microphones (up to 4@analog, 8@digital) and please adjust them within the application.
Audio output can only be up to stereo. If the input is 3 channels or more, perform downmix processing before output, convert to 2 channels data.

5.4.3. Operating procedure

The main operation procedure is as follow:

  1. Compile & upload SubCore for signal processing

  2. Compile & upload MainCore for sound acquisition & application processing

  3. Running the application

If you compile and upload MainCore at first, it will call SubCore after startup, resulting in an error.
If you have already compiled and uploaded to SubCore, the program of SubCore that has already been uploaded will start after starting MainCore.
5.4.3.1. Compilation & upload of SubCore
  1. Open the sample sketch by selecting File→ Sketch example→ Sketch example for Spresense SignalProcessing→ HighPassSound / LowPassSound→ SubFFT from Arduino IDE.

  2. Select Tools→ Core→ SubCore1 (Tools→ Core→ SubCore1) from the Arduino IDE menu.

    In this sample, the offloaded SubCore is SubCore1.
  3. Select the COM port of Spresense with Tools→ Serial Port and write to the microcomputer board.

    The SubCore program waits for instructions from the MainCore after starting.

Now, the preparation on the SubCore is OK.

5.4.3.2. Compilation & upload of MainCore
  1. Open the sample sketch by selecting File→ Sketch example→ Sketch example for Spresense SignalProcessing→ HighPassSound → MainAudio from Arduino IDE.

  2. Select Tools→ Core→ MainCore from the Arduino IDE menu

    Write to the microcomputer board.

If you have selected the serial port for uploading SubCore, you do not need to select it again.

Now, the preparation on the MainCore is also OK.

5.4.3.3. Run the application
  1. After uploading MainCore, open Tools→ Serial Monitor and start the application.

  2. When the application is started, the audio data input from the microphone will be the audio data that has been LPF/HPF processed by IIR, and output it from speaker.

5.4.4. the Explanation of program

  • The number of input audio channels and the number of processing frame samples are specified in the following part of MainAudio.ino.

static const int32_t channel_num  = AS_CHANNEL_STEREO;
static const int32_t bit_length   = AS_BITLENGTH_16;
static const int32_t frame_sample = 240;

In this sample, the number of channels is set to 2ch and the number of processing frame samples is set to 240.

Only the 16 bit mode is supported for bit length.
The number of channels and the number of processing frame samples must match the SubCore side.
In this sample, the number of processing frame samples is 240.
Reducing the number of frame samples reduces the latency of audio processing, but increases the processing load on the CPU.
  • The cutoff frequency and Q value are set by the following part of SubCore.

const int   g_channel = 2; /* Number of channels */
const int   g_cutoff  = 1000; /* Cutoff frequency */
const float g_Q       = sqrt(0.5); /* Q Value */
const int   g_sample  = 240; /* Number of channels */
const int   g_fs      = 48000; /* Sampling Rate */

In this sample, the cutoff frequency is set to 1kHz, the Q value is set to 1 / √2, the number of output frame samples is set to 240, and the sampling rate is set to 48kHz.

And,

   if (! HPF.begin (TYPE_HPF, g_channel, g_cutoff, g_Q, g_sample, IIRClass :: Interleave, g_fs)) {

The output format is specified as Interleave.

In this sample, sound is collected by FrontEndObject.
Due to HW restrictions, Spresense’s audio function uses 48kHz sampling or 192kHz sampling for all input and output.
When realizing an input / output application that involves audio signal processing, if internal processing is performed at a sampling rate such as 16kHz, resampling with SW is required twice, which requires a lot of processor power and memory.
In the case of real time audio processing using the IIR filter this time, processing at 16KHz is not recommended due to insufficient memory.

5.5. VoiceChanger

5.5.1. Overview

This sample sketch performs FFT / iFFT on the voice data input by the microphone to change the pitch and regenerate the voice data. The LP filter is applied to the generated data to cut off the high frequency noise added due to the lack of phase information, etc., and then output. Please refer to Audio library for the Audio function.

In addition, this sample performs with offloading the signal processing library to SubCore. Please see also MultiCore MP Library.

This sample is provided as a simple FFT/iFFT execution sample. Therefore, it is not good for the sound quality as a pitch shifter.

5.5.2. Operating environment

  • Spresense Main & extension Board

  • Headphones or speakers for playback

  • Microphones for recording

In this sample, headphones and a microphone are used. Please connect headphones or active speakers to the headphone jack on the extension board.
Please refer to the following hardware guide for microphone connection.

The samples provided are for a monorail microphone. If you want more microphones, Connect microphones (up to 4@analog, 8@digital) and please adjust them within the application.
Audio output can only be up to stereo. If the input is 3 channels or more, perform downmix processing before output, convert to 2 channels data.

5.5.3. Operating procedure

The main operation procedure is as follow:

  1. Compile & upload SubCore for signal processing

  2. Compile & upload MainCore for sound acquisition & application processing

  3. Running the application

If you compile and upload MainCore at first , it will call SubCore after startup, resulting in an error.
If you have already compiled and uploaded to SubCore, the program of SubCore that has already been uploaded will start after starting MainCore.
5.5.3.1. Compilation & upload of SubCore
  1. Open the sample sketch by selecting File→ Sketch example→ Sketch example for Spresense SignalProcessing→ VoiceChanger→ SubFFT from Arduino IDE.

  2. Select Tools→ Core→ SubCore1 (Tools→ Core→ SubCore1) from the Arduino IDE menu.

    In this sample, the offloaded SubCore is SubCore1.
  3. Select the COM port of Spresense with Tools→ Serial Port and write to the microcomputer board.

    The SubCore program waits for instructions from the MainCore after starting.

Now, the preparation on the SubCore is OK.

5.5.3.2. Compilation & upload of MainCore
  1. Open the sample sketch by selecting File→ Sketch example→ Sketch example for Spresense SignalProcessing→ VoiceChanger → MainAudio from Arduino IDE.

  2. Select Tools→ Core→ MainCore ( Tools→ Core→ MainCore) from the Arduino IDE menu

    Write to the microcomputer board.

    If you have selected the serial port for uploading SubCore, you do not need to select it again.

Now, the preparation on the MainCore is also OK.

5.5.3.3. Run the application
  1. After uploading MainCore, open Tools→ Serial Monitor and start the application.

  2. When the application is started, the data in the frequency domain obtained by FFT is processed into the audio data input from the microphone, the data with the changed pitch is reconstructed with iFFT and output.

5.5.4. the Explanation of program

  • The number of input audio channels and the number of processing frame samples are specified in the following part of MainAudio.ino.

static const int32_t channel_num  = AS_CHANNEL_MONO;
static const int32_t bit_length   = AS_BITLENGTH_16;
static const int32_t frame_sample = 1024;
Only the 16 bit mode is supported for bit length.
The number of channels and the number of processing frame samples must match the SubCore side.
In this sample, the number of processing frame samples is 1024.
Please specify according to the pipeline configuration based on the number of FFT TAPs.
When outputting audio, it must be converted to interleaved stereo data.
Therefore, in this sample, after signal processing, monaural data is converted to stereo data and output.
  • The amount of pitch shift is specified below in MainAudio.ino

static int pitch_shift = -10;

It can be shifted in the positive and negative directions according to the resolution of the FFT.
In this sample, 1024TAP FFT / iFFT processing is performed on the 48kHz sampled audio data, so about 46Hz is changed in one shift.
This parameter can be changed even during signal processing operation.

  • The number of FFT taps is specified as follow in SubFFT.ino.

/* Select FFT length */

//#define FFT_LEN 32
//#define FFT_LEN 64
//#define FFT_LEN 128
//#define FFT_LEN 256
//#define FFT_LEN 512
#define FFT_LEN 1024
//#define FFT_LEN 2048
//#define FFT_LEN 4096
  • This sample regenerates the audio with FFT/iFFT, so it is rectangular window and no overlap.

   FFT.begin(WindowRectangle,MAX_CHANNEL_NUM,0);

If you want to change the algorithm, please change accordingly.

  • The LPF setting value after iFFT is below

const int g_channel = MAX_CHANNEL_NUM; / * Number of channels * /
const int g_cutoff = 1000; / * Cutoff frequency * /
const float g_Q = sqrt (0.5); / * Q Value * /
const int g_sample = 1024; / * Number of channels * /

const int g_max_shift = 30; / * Pitch shift value * /

The cutoff frequency is set to 1kHz and the Q value is set to 1 / √2, but change accordingly.

Since it is a monaural input, the output format is the default Planar.
  • Maximum shift value is specified below.

const int g_max_shift = 30; / * Pitch shift value * /

This sample is set to 30.

  • The output audio data is converted to interleaved stereo data here.

        for (int j = 0; j <cnt; j ++) {
          pOut [pos] [j * 2] = pLpfDst [j];
          pOut [pos] [j * 2 + 1] = 0;
        }

In this sample, the R channel is generated with 0.

In this sample, sound is collected by FrontEndObject.
Due to HW restrictions, Spresense’s audio function uses 48kHz sampling or 192kHz sampling for all input and output.
When supporting an input / output application that involves audio signal processing, if internal processing is performed at a sampling rate such as 16kHz, resampling with SW is required twice, which requires a lot of processor power and memory.
In the case of real time audio processing using the IIR filter this time, processing at 16KHz is not recommended due to insufficient memory.