Spresense Arduino Library Developer Guide

Table of Contents

1. Spresense Arduino Sketch App Development
2. Spresense Arduino Libraries

1. Spresense Arduino Sketch App Development

To quickly get started writing sketches you can use the example files. These examples are included in the Spresense Arduino Library and are available at File > Examples > Spresense.

Many existing Arduino sketches and libraries run on Spresense. Some of these sketches are for different hardware. You will need to modify these sketches. For an overview of the hardware features, see Differences between Spresense and Arduino Uno.

1.1. Functional Differences

The following Arduino language functions behave differently on Spresense and Arduino Uno:

LEDs

The Spresense main board has 4 LEDs: LED0, LED1, LED2, and LED3. LED0 can also be addressed using the name LED_BUILTIN for compatibility with existing sketches.

To control the LEDs, configure the LED as an output using pinMode() and call digitalWrite().

EEPROM

The Spresense doesn’t have EEPROM, but the EEPROM library emulates EEPROM using the flash memory. If you want to handle persistent data, you can use the EEPROM library or the SD card on the extension board.

Serial

The Spresense has two UART serial ports. You can access these ports with the Arduino SDK. The main board’s USB port is addressed as Serial and the logic-level pins DO0 and DO1 are addressed as Serial2.

The main board USB serial port can be used for your application, but be aware that it is also used to provide error messages from the NuttX operating system. These can be helpful in debugging your sketch.

pinMode()

The extension board has pull-ups on all pins. This can alter the logic for some designs.

At startup all pins are pulled up.
When pins are set to input, they are pulled up. This can impact circuits with a pull-down resistor.
When the pin is set to output, digitalWrite() will be set, whether the pin is high or low.

analogRead()

Spresense has 6 dedicated analog pins. These pins are not shared with digital pins and can not be addressed by a number alone. They must be addressed as A0 to A5.

Channel A0 to A3 are slower ADCs with a sample rate of about 65 Hz. Channel A4 and A5 are faster ADC with a sample rate of about 6 kHz.

For advanced users higher ADC performance can be achieved using the Spresense SDK.

analogReference()

analogReference() is not supported. The reference voltage for analog input use for Spresense is fixed.

F()

The Arduino F() macro is defined as Null, the function does nothing. This allows existing code using F() compile without error.

PROGMEM

The Arduino function PROGMEM() is defined as Null, the function does nothing. This allows existing code using PROGMEM() compile without error.

Interrupts - Digital pin

The interrupt number for the digital pin is dynamically assigned within the attachInterrupt() function.

attachInterrupt (pin, ISR, mode);

Where

pin

the pin number, for example D05.

ISR

the ISR to call when the interrupt occurs; this function must take no parameters and return nothing. This function is sometimes referred to as an interrupt service routine.

mode

defines when the interrupt should be triggered. Four constants are predefined as valid values:

LOW	to trigger the interrupt whenever the pin is low,
CHANGE	to trigger the interrupt whenever the pin changes value
RISING	to trigger when the pin goes from low to high,
FALLING	for when the pin goes from high to low.
HIGH	to trigger the interrupt whenever the pin is high.

digitalPinToInterrupt() is defined as an empty macro, to retain compatibility with existing code and can be used in the following way.

attachInterrupt (digitalPinToInterrupt (pin), ISR, mode);

digitalPinToInterrupt (pin) should not be used in the form

x = digitalPinToInterrupt (pin);
attachInterrupt (x, ISR, mode);

A debounce filter has been implemented for digital pin interrupts. This introduces a short delay and prevents a second interrupt occurring within 3 RTC (32.768 kHz) cycles (about 100us) of the first interrupt. Therefore, interrupts may not be acquired for signals which change very rapidly.

Interrupts - Timer

Timer interrupts can be implemented using attachTimerInterrupt(ISR,period). This function uses the same timer resource as tone (), so it can not be used with tone () at the same time.

Where

ISR: the ISR to call when the interrupt occurs; this function must return the next timer period in microseconds. If this function returns 0, the timer stops and it behaves as oneshot timer. The maximum value is about 26 seconds and if it exceeds, an error occurs.
period: defines the timer period in microseconds after the attachTimerInterrupt.

Since this function is called from an interrupt handler, there are restrictions on callable API. For example, you can not use the Analog I / O function from within this function

micros()

The resolution of the micros() timer is approximately 30 microseconds. It is based on a 32768 Hz oscillator.

1.2. Spresense Libraries

The Spresense Arduino Library also provides a range of built-in libraries. These are described in the Spresense Arduino Libraries section.

1.3. Memory Usage

As a feature of Spresense, the compiled sketch is installed into the flash memory. When it is executed, all the data including complied code and read-only data, are copied from the flash memory to the RAM and executed from the RAM.

The maximum RAM available is 768 kilobytes (= 786432 bytes) by default. Which is half of the size of application SRAM of 1.5 MB.

If the program size of sketch exceeds the available maximum RAM size, the compilation will fail and an error will occur.

A memory usage report is generated when you compile a sketch on the Arduino IDE. This report allows you to check the memory consumption. For example:

Sketch uses xxxxxx bytes (xx%) of program storage space. Maximum is 786432 bytes.
Global variables use xxxxxx bytes (xx%) of dynamic memory, leaving xxxxxx bytes for local variables. Maximum is 786432 bytes.

In this report, some terms have different meaning from the original.

program storage space: it means the available maximum RAM size.
dynamic memory: it means the static RAM size used by a sketch.
local variables: it means the remaining RAM size (= program storage space - dynamic memory)

There is a dynamically allocated heap area besides the statically used memory. This heap area is allocated from the remaining RAM called local variables. Therefore, when you program a sketch, it is necessary to leave the remaining RAM size for the heap area. If the size used by a sketch exceeds 75% of the total, a warning message of Low memory available, stability problems may occur. will be generated as a guide. Then, you can review the memory size used by the sketch program.

2. Spresense Arduino Libraries

2.1. Audio Library

Spresense audio library supports audio playback and audio recording, and uses SD card for file storage. This allows simple sketches to make use of the high resolution audio features.

Spresense has a high performance ADC and DAC with full digital amplifier which are controlled by the audio library.

The Spresense has six CPU cores. One of these is used as the application CPU which runs your sketch. The other five are called DSPs. The DSP cores perform audio signal processing, such as decoder and encoder. DSP binaries are provided here for easy and efficient audio application development.

Currently, the high resolution audio supports 24 bit, 192 kHz and it is available for WAV (LPCM).
You need to pre-install the appropriate DSP codec binary file and specify the installation location from your application. Details about installation are described in the DSP codec files section.

The main features of the library are:

Sound recorder
Audio player

The features above support:

Volume control
Balance (L/R Gain)
Beep generation - Please note this produces a tone output to the headphone directly, unlike the Arduino tone() function which produces an output to one of the Arduino header pins.

The audio library allows selection of input and output channel.

Audio Input Channels are:

Analog microphone
Digital microphone
I2S

Audio Output Channel are:

Analog headphone
I2S

DSP codecs are available to support for multiple audio formats such as:

If the MP3 file has ID3v2 TAG, the decoder will generate an error message and is not able to play the audio file. Please remove the tag information with tools such as links:https://www.mp3tag.de/en/download.html[MP3Tag^] if your MP3 file has large metadata such as image data.

WAV (PCM)

Playing of monaural WAV file is not supported yet.

Spresense can record and playback a range of sampling rates and bitrates when using compression.

For more detailed information see Audio Subsystem.

2.1.1. Pre-requisites for Audio

Please prepare these before using the audio library:

Spresense main board
Spresense extension board
Microphone for recording
Headphones for play back
MicroSD card

Spresense extension board has dedicated pins for audio and SD card, so all Arduino header pins remain available for other applications. For simple audio applications analog stereo input and output will be used.

Spresense extension board has a 3.5mm stereo headphone connector and a microphone interface.

For more detailed information on the pinout for the other audio connections please check How to use microphones, hw_docs_en.html.

2.1.2. Example Audio Sketches

The use of the Audio library is demonstrated in a range of examples sketches which can be accessed through the Arduino IDE menu File → Examples → Examples for Spresense Audio → application .

2.1.2.1. Recorder Example

This example requires a SD card and a pair of analog microphones.

First connect the microphone to channel A and B on the extension board, as described in the How to use microphones.

Please refer to Setting of "MIC channel select map" for SDK configuration information on how to use digital microphones.
For information on building SDK and incorporating it into Arduino IDE, see How to prepare Arduino environment

Then make a sub directory /BIN and copy the MP3 or WAV encoder binary files MP3ENC from MP3ENC or SRC from SRC or into the SD card /BIN folder. Alternatively you can also install the binary file into SPI-Flash memory instead of SD card. See DSP codec files for details.

Once the SD card is prepared, mount it in the Spresense SD card slot.

Open the Arduino IDE and select File → Examples → Examples for Spresense Audio → application → recorder .

The audio format to be recorded can be switched with the argument of initRecorder.

When recording a WAV file, it is necessary to call writeWavHeader before creating voice recording to create a WAV header.
See also recorder_wav .

If you want recording with high resolution (192 kHz), you should call setRenderingClockMode with AS_CLKMODE_HIRES before executing setRecorderMode.

If you want to set Mic Gain, please set "input_gain" of setRecorderMode .
Digital is from 0db to -78.5db
Analog is from 21db to -78.5db
These are the valid ranges.

When you upload the sketch audio will be recorded for about 10 seconds. This is saved to "Sound.mp3" in the root of the SD card.

Remove the SD card and put it in your PC to be able to play the recorded audio. You can also, upload the player sketch described below and use that to play back the recorded audio.

If you do no hear any sound, open the serial monitor during recording. The error messages will help you locate the problem. A detailed explanation of the error codes is provided in the Error Information of Audio SubSystem section.

Even if you can acquire sound, it may not be possible to write all the data due to factors such as the writing performance of the SD card and the application that is running.
For example, on the current Spresense extension board the limit is 8ch/48kHz or 2ch/192kHz.
It also depends on the speed class of the SD card. Please use more higher speed class.

The default size of the buffer for writing to the SD card is 160k bytes. This size is a size that can recorde high-resolution WAV audio, and is redundant when recording 48 kHz MP3 files, etc.
If you want to change this size, you set argument of setRecorderMode .
It is recommended to set the 160k bytes which is default when recording high-resolution WAV files, and 8k bytes when recording 48KHz MP3 files.

The size is a guide only, adjust and use according to the performance of the SD card and peformance of your application.
If you make the buffer too small, errors will occur when writing to the SD card. If a write overflow occurs, increase the buffer size.

If the connected microphone is a digital microphone, set "is_digital" of setRecorderMode to "true".

2.1.2.2. Player Example

This example requires a SD card and a pair of headphones.

Connect the headphones to the headphone connector on the extension board .

The SD card should have the MP3 file you want to play. You can use the mp3 file recorder using the recorder sketch or select a pre-recorded mp3 file.

The MP3 file must use a sample rate which is supported by the decoder. The MP3 codec support 32000, 44100 and 48000 Hz. Please use a MP3 sample rate converter if you need to change the sample rate.
The MP3 file should not include an image. Please note, this image usually is the album cover. If you need to remove the image use a ID3 Tag Editor.

Use your PC to rename the file as "Sound.mp3" and put it in the root directory of the SD card.

Then make a sub directory /BIN and copy the MP3 or WAV decoder binary files MP3DEC from MP3DEC or WAVDEC from WAVDEC into the SD card /BIN folder.

Once the SD card is prepared, place it in the Spresense SD card slot.

Open the Arduino IDE and select File → Examples → Examples for Spresense Audio → application → player .

When you upload the sketch you should hear audio from the headphones.

If you want to play high resolution (192 kHz) contents, you should call setRenderingClockMode with AS_CLKMODE_HIRES before you call setPlayerMode.

When switching between high resolution and normal resolution, it is necessary to call setReadyMode once and make a transition to Ready state.

If you want to select the output device to I2S, you can select by setPlayerMode with AS_SETPLAYER_OUTPUTDEVICE_I2SOUTPUT.
The operation mode of I2S corresponds only to Slave mode, I2S format mode, 48000 Hz, Stereo.

The default size of the buffer for reading from the SD card is 160k bytes. This size is a size that can play high-resolution WAV audio , and is redundant when playing 48 kHz MP3 files, etc.
If you want to change this size, you set argument of setPlayerMode .
It is recommended to set the 160k bytes that is default when playing high-resolution WAV files, and 24k bytes when playing 48KHz MP3 files.

The size is a guide only, adjust and use according to the performance of the SD card and the performance of your application.
If you make the buffer too small, an error will occur when reading from the SD card. If read data underflow occurs, increase the buffer size.

When the audio output is line-out from audio connector on Spresense extention board, set "sp_drv" to "AS_SP_DRV_MODE_LINEOUT" of setPlayerMode for the drive capability of the audio signal output from the CXD5247.

When using the output of the Speaker pin that directly uses the amplifier output of the CXD5247, set "sp_drv" to "AS_SP_DRV_MODE_4DRIVER".

The default value is "AS_SP_DRV_MODE_LINEOUT".

If you do not hear any sound, open the serial monitor. The error messages will help you locate the problem. A detailed explanation of the error codes is provided in the Error Information of Audio SubSystem section.

2.1.2.3. Beep Sample (beep)

This sample is an example of playing a beep sound. I will use headphones.

Open the Arduino IDE and select File → Examples → Examples for Spresense Audio → application → beep . When you upload the sketch you should hear beep sounds from the headphones.

Beep sounds requires HW settings to rendering sound. Therefore, you need to call setPlayerMode and change to music playback mode.

2.1.2.4. Playlist Example (play_playlist)

This is an example sketch for play back music using a playlist.

In order to use playlist functions, it is necessary to create a playlist file on PC and copy it into SD card. The directory structure of the SD card used by this example is shown below.

SD card root directory
|-- BIN/
|   |-- MP3DEC
|   `-- WAVDEC
|-- AUDIO/
|   |-- Sound1.mp3
|   |-- Sound2.mp3
|   |--   :
|   |-- Sound1.wav
|   |-- Sound2.wav
|   |--   :
`-- PLAYLIST/
    `-- TRACK_DB.CSV

BIN/ : Copy DSP binary files such as MP3DEC and WAVDEC. Please refer to the DSP installation described later.
AUDIO/ : Copy music files to be played back in the playlist.
PLAYLIST/ : Copy a playlist file ( TRACK_DB.CSV ).

The format of TRACK_DB.CSV is shown below. It is a text file of CSV (comma-separated values) format in which the information of each file is written line by line.

[filename],[artist],[album],[channel number],[bit length],[sampling rate],[file format]
[filename],[artist],[album],[channel number],[bit length],[sampling rate],[file format]
[filename],[artist],[album],[channel number],[bit length],[sampling rate],[file format]
 :

Please describe the file name, artist name, album name and codec information (number of channels, bit length, sampling rate Hz). An example of TRACK_DB.CSV is shown below.

Sound1.mp3,Artist1,Album1,2,16,44100,mp3
Sound2.mp3,Artist1,Album1,2,16,44100,mp3
Sound1.wav,Artist2,Album2,2,16,48000,wav
Sound2.wav,Artist2,Album2,2,24,192000,wav

A simple script tool mkplaylist.py is provided for generating the playlist file. Please right-click here and download file. This script uses ffmpeg-python, so you need to install ffmpeg and ffmpeg-python.

Usage of mkplaylist.py is:

python mkplaylist.py
usage: python mkplaylist.py dirname [dirname2] [dirname3] ...
usage: python mkplaylist.py -f filename

Generate an audio playlist file named as TRACK_DB.CSV

Run the script with argument of directory path including music files. When it is completed successfully, it generates a TRACK_DB.CSV file under the current directory. If the TRACK_DB.CSV file already exists, it adds it to the end of the file.

python mkplaylist.py MyMusic

The extension of music filename should be .mp3 or .wav .
Filenames with 2-byte characters such as Japanese filenames are not supported. Please rename the file with encoding to ASCII.
Do not use string that contains "," comma in filename, artist, album name because the playlist file is written in CSV format.

Open the Arduino IDE and select File → Examples → Examples for Spresense Audio → application → player_playlist . When you upload and run the sketch, you see the following menu on serial monitor. It performs various operations by the key input from serial monitor.

=== MENU (input key ?) ==============
p: play  s: stop  +/-: volume up/down
l: list  n: next  b: back
r: repeat on/off  R: random on/off
a: auto play      m,h,?: menu
=====================================

This example saves the value of last volume and the settings of random, repeat and autoplay keys into non-volatile memory using EEPROM library. So the keys preset information are restored at the next startup.

Please use this example as reference when you create the original music player.

2.1.2.5. Audio Through sample (through)

This sample can achieve power saving and an audio output with very low latency by using the audio HW path settings. For examples, audio digital that input from I2S can be output to analog output, audio data that input from mics can be output to I2S, and so on…

This application does not require a DSP, ie, an SD card, as audio sw signal process do not use.

If you use microphones, you can refer the description in the How to use microphones.

For example, if you want to output to the analog output the sound that input from I2S, set "input" to "I2sIn",
set "i2s_out" to "None"
set "sp_out" to "true".
These are arguments of setThroughMode .

If you want to output to I2S the sound that input from Mic, set "input" to "MicIn",
set "i2s_out" to "Mic"
set "sp_out" to "false".

If you want to set Mic Gain, please set "input_gain" of setThroughMode .
Digital is from 0db to -78.5db
Analog is from 21db to -78.5db
These are the valid ranges.

When using the output of the Speaker pin that directly uses the amplifier output of the CXD5247, set "sp_drv" to "AS_SP_DRV_MODE_4DRIVER".

The default value is "AS_SP_DRV_MODE_LINEOUT".

2.1.2.6. Other composite samples

In addition to the regular samples, there are several complex samples.

dual_players
This is a sample of playing two music simultaneously.
pcm_capture
This sample is only for capturing sound data. Please use for sound signal analysis.
rec_play
It is a sample to play the sound after recording the sound. This sample shows switching operation of the audio mode.
recorder_with_rendering
It is a sample to record while outputting sound.
voice_effector
It is a sample to output while signal processing the input sound.

2.1.3. DSP codec files

It is important that the DSP codec binary files can be accessed from the sketch when using the Audio library. The player and recorder examples put these files on the SD Card and use the location /mnt/sd0/BIN/ in initPlayer() or initRecorder() .

It is also possible to store the codec binary files in the SPI Flash memory on the main board using the DSP installer sketches described below. This uses the location /mnt/spif/BIN/ in initPlayer() or initRecorder(). If you install the DSP codec binary files to the SPI Flash memory, you will need to modify the example sketches accordingly.

Table 1. DSP binaries
DSP file name	installer sketch	Description
MP3DEC	mp3_dec_installer	Used for MP3 audio playback
WAVDEC	wav_dec_installer	Used for WAV (PCM) audio playback
MP3ENC	mp3_enc_installer	Used for MP3 audio recording
SRC	src_installer	Used for PCM audio recording

The DSP can be installed onto the SD card or on the flash. There are two ways to install the DSP binary:

Copy into the SD card using a PC
Use the DSP installer sketch.

It is only necessary to install the DSP binary once. There is no need to reinstall, as long as the files are not overwritten, deleted or the SDK is not updated.

In case of copying the DSP codec binary files to the SD card:

Download the DSP file by clicking the DSP file name in the DSP binaries table. Please make a sub directory /BIN and copy the DSP binary file into the SD card /BIN folder.

In the audio player or recorder application program, it is necessary to specify the DSP installation location of the initPlayer() function or initRecorder() function. When you use the SD card, please specify /mnt/sd0/BIN . In the examples of audio applications, /mnt/sd0/BIN is used.

In case of using the DSP installer.

The DSP installer is launched from Arduino IDE, to open it click on the toolbar menu: File → Examples → Examples for Spresense Audio → dsp_installer . There are different DSP installers available under dsp_installer folder, one for each DSP as shown in the table above. Use DSP binaries table and select the appropriate DSP installer for your design.

Figure 1. Launch DSP Installer

Please open the Serial Monitor , after the sketch is done with compiling and uploading. Make sure you select the baudrate and specify where to install the DSP binary file. Follow the steps below:
1. Open the Serial Monitor
2. Select the Baudrate: For this example the baudrate is 115200
3. Select where to install the binary file: Input [1] for SD Card or [2] for SPI-Flash. If you want to install to the SD card ( option[1] ), please insert the FAT-formatted SD card into the SD card slot on the Spresense extension board.
4. Click on Send button
  
  Figure 2. Select the installation directory
  
  If the installation is successful, you will see this message on the Serial Monitor, as shown in the picture below.
  
  Figure 3. Execute the installation

Please note that, the audio application calls the initPlayer() or initRecorder() function with the argument of the DSP installation directory. If you installed DSP binary to the SD card, you have to specify the /mnt/sd0/BIN . If you installed into the SPI Flash, you have to specify the /mnt/spif/BIN . In the examples of audio applications, /mnt/sd0/BIN is used.

2.1.4. More Information

For more information please check:

2.2. Camera Library

This section explains how to use Camera library of Spresense Arduino Library.

2.2.1. Overview

The Spresense camera library overview is shown as below:

Figure 4. Camera Overview

In Spresense Arduino Library Camera, there are two classes.

One is "theCamera" which is an instance of CameraClass, and the other is CamImage class which is manipulate image from theCamera.

"theCamera" has three major functions:

Video Stream function to get Camera preview image
Setting function to set camera parameters
Capture picture function to get high-resolution JPEG image.

CamImage class is for manipulating an image. The below figure is the overview of the CamImage.

Figure 5. CamImage Overview

CamImage class has two major functions:

To get information of an image from theCamera class
To convert image

Please find the description of supported functions in the next sections.

2.2.2. Video Stream function to get Camera preview image

The viewfinder of a camera shows real-time images shown on a camera.

This real-time image (real-time movie) is called Preview image. "theCamera" has a function to acquire this Preview image frame by frame.

To obtain the Preview image, first determine the image format of the Preview image using the begin() method function. The definition of the begin() method function is as follows.

begin(
    int buff_num=1,
    CAM_VIDEO_FPS fps = CAM_VIDEO_FPS_30,
    int video_width   = CAM_IMGSIZE_QVGA_H,
    int video_height  = CAM_IMGSIZE_QVGA_V,
    CAM_IMAGE_PIX_FMT fmt=CAM_IMAGE_PIX_FMT_YUV422)

All the parameters given to begin() method function determine the parameters of the Preview image. The parameters and the default values are as follow : Number of preview image buffers inside theCamera = 1 Vertical and horizontal size of the image = QVGA (320 x 240) Frame rate of Video (how many frames to acquire per second) = 30 FPS (30 images per second) Image data pixel format = YUV 422

As for the number of internal image buffers, it would be sufficient if you use the default value of 1.

In cases you need to do heavy processing, for example when you are processing large number of images. You need to increase the number of image buffers to be able to do parallel image processing and image acquisition. In such case, by setting the number of internal image buffers to 2, you can perform parallel image processing from the camera while the image frame rate is improved in some cases as a result. This process will consume about 150 KB of buffer memory with QVGA, so please be careful when setting a large number of sheets.

The supported pixel formats for the preview image are YUV422 and JPEG. The only supported image size for YUV422 format is QVGA. The maximum frame rate is related to image size. The larger the image size is, the lower the maximum frame rate is.

Figure 6. The supported frame rate for image size

The begin() method function is the first function to call when using theCamera. When the begin() method function is completed successfully, register the callback function to obtain the Preview image using the startStreaming() method function. The callback function is as follows:

void camera_callback(CamImage img)

The user implements its own function of this type and registers it by using the startStreaming() method function. When "true" is specified as the first argument of startStreaming(), acquisition of the video image for Preview is started, and the registered callback function is called each time the image is acquired. The frequency of acquiring images is determined by the frame rate specified by the begin() method function. The callback function of the next frame will not be called unless the callback function implemented by the user is terminated. To stop the acquisition of the Preview image, call the startStreaming() method function with the first argument of the startStreaming() method function set to false.

2.2.3. Setting function to set Camera parameters

Spresense Camera, the same as any other camera, can set various settings such as color adjustment and brightness.

Table 2. Camera Parameter Control method functions
Method name	Description
setAutoWhiteBalance()	Start and stop automatic white balance of Camera
setAutoISOSensitive()	Start and stop automatic ISO sensitivity of Camera setISOSensitivity()
ISO sensitivity setting with manual ISO sensitivity setting	setAutoWhiteBalanceMode()
Mode setting at automatic white balance	setColorEffect()

These setting method functions can be called at any time after calling the begin() method function.

2.2.4. Capture picture function to get high-resolution JPEG image.

Preview images are low resolution, you can get images with frequency of frame rate as movies. On the other hand, when acquiring data as a photograph, the Spresense Camera can acquire high-resolution JPEG compressed images.

First of all, we will process "to set the film" on theCamera. The method function that does this is setStillPictureImageFormat(). Use this method function to set the image size and pixel format for still images (photographs).

setStillPictureImageFormat(int width, int height, CAM_IMAGE_PIX_FMT fmt = CAM_IMAGE_PIX_FMT_JPEG)

Specify the horizontal and vertical size of the image with the first and second arguments. The third argument specifies the pixel format of the image.

Currently, the image pixel format supports only JPEG.

If you set the setStillPictureImageFormat() method function once, the setting will be effective permanently unless you change the parameters.

After setting the image, at the arbitrary timing, do "push the shutter" process and get the photo data. The method function for that will be takePicture(), which returns an instance of CamImage as a return value. If an error occurs during photography, it returns an empty CamImage. To determine whether an instance of CamImage is empty, you can check it with isAvailable() of CamImage class.

2.2.5. To get informations of an Image from theCamera

The CamImage instance, obtained by callback with startStreaming() or by the return value of takePicture(), contains information on the acquired image. Image information can be obtained using the method function of the CamImage class.

Table 3. Methods to get Informations from CamImage
Method name	Description
isAvailable()	Check availability of this CamImage instance
getWidth()	Get image width in pixels
getHeight()	Get height of image in pixels
getPixFormat()	Get pixel format of image
getImgSize()	Get size of image data in bytes
getImgBuff()	Get buffer address of image data

2.2.6. To convert image

CamImage instances have one image transformation method function.

Table 4. Methods to convert CamImage
Method name	Description
convertPixFormat()	Convert the image from the current pixel format to another pixel format

When you want images to appear on the display, the display’s pixel format is RGB format usually. If your pixel format is different than RGB, use the convertPixFormat() method function to convert pixel format of your image to RGB.

convertPixFormat() converts its own pixel format and overwrites own image data.

Currently, there are the following restrictions. - convertPixformat() supports only two conversion one is from YUV422 to RGB and the other is YUV422 to GRAY.

2.2.7. Explanation by sample code

This section explains how to use the Spresense camera, using the sample code of Camera included in the Spresense Arduino Package.

You can open the sample code from Arduino IDE’s menu bar.

"File" ⇒ "Sketch example" ⇒ "Camera in Spresense sketch example" ⇒ "camera"

/*
 *  camera.ino - One minute interval time-lapse Camera
 *  Copyright 2018 Sony Semiconductor Solutions Corporation
 *
 *  This library is free software; you can redistribute it and/or
 *  modify it under the terms of the GNU Lesser General Public
 *  License as published by the Free Software Foundation; either
 *  version 2.1 of the License, or (at your option) any later version.
 *
 *  This library is distributed in the hope that it will be useful,
 *  but WITHOUT ANY WARRANTY; without even the implied warranty of
 *  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
 *  Lesser General Public License for more details.
 *
 *  You should have received a copy of the GNU Lesser General Public
 *  License along with this library; if not, write to the Free Software
 *  Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
 *
 *  This is a test app for the camera library.
 *  This library can only be used on the Spresense with the FCBGA chip package.
 */

#include <SDHCI.h>
#include <stdio.h>  /* for sprintf */

#include <Camera.h>

#define BAUDRATE                (115200)

SDClass  theSD;
int take_picture_count = 0;


/**
 * Callback from Camera library when video frame is captured.
 */

void CamCB(CamImage img)
{

  /* Check the img instance is available or not. */

  if (img.isAvailable())
    {

      /* If you want RGB565 data, convert image data format to RGB565 */

      img.convertPixFormat(CAM_IMAGE_PIX_FMT_RGB565);

      /* You can use image data directly by using getImgSize() and getImgBuff().
       * for displaying image to a display, etc. */

      Serial.print("Image data size = ");
      Serial.print(img.getImgSize(), DEC);
      Serial.print(" , ");

      Serial.print("buff addr = ");
      Serial.print((unsigned long)img.getImgBuff(), HEX);
      Serial.println("");
    }
  else
    {
      Serial.print("Failed to get video stream image\n");
    }
}

/**
 * @brief Initialize camera
 */
void setup()
{

  /* Open serial communications and wait for port to open */

  Serial.begin(BAUDRATE);
  while (!Serial)
    {
      ; /* wait for serial port to connect. Needed for native USB port only */
    }


  /* begin() without parameters means that
   * number of buffers = 1, 30FPS, QVGA, YUV 4:2:2 format */

  Serial.println("Prepare camera");
  theCamera.begin();


  /* Start video stream.
   * If received video stream data from camera device,
   *  camera library call CamCB.
   */

  Serial.println("Start streaming");
  theCamera.startStreaming(true, CamCB);

  /* Auto white balance configuration */

  Serial.println("Set Auto white balance parameter");
  theCamera.setAutoWhiteBalanceMode(CAM_WHITE_BALANCE_DAYLIGHT);

  /* Set parameters about still picture.
   * In the following case, QUADVGA and JPEG.
   */

  Serial.println("Start streaming");
  theCamera.setStillPictureImageFormat(
     CAM_IMGSIZE_QUADVGA_H,
     CAM_IMGSIZE_QUADVGA_V,
     CAM_IMAGE_PIX_FMT_JPG);
}

/**
 * @brief Take picture with format JPEG per second
 */

void loop()
{
  sleep(1); /* wait for one second to take still picture. */

  /* You can change the format of still picture at here also, if you want. */

  /* theCamera.setStillPictureImageFormat(
   *   CAM_IMGSIZE_HD_H,
   *   CAM_IMGSIZE_HD_V,
   *   CAM_IMAGE_PIX_FMT_JPG);
   */

  /* This sample code can take 100 pictures in every one second from starting. */

  if (take_picture_count < 100)
    {

      /* Take still picture.
      * Unlike video stream(startStreaming) , this API wait to receive image data
      *  from camera device.
      */

      Serial.println("call takePicture()");
      CamImage img = theCamera.takePicture();

      /* Check availability of the img instance. */
      /* If any error was occured, the img is not available. */

      if (img.isAvailable())
        {
          /* Create file name */

          char filename[16] = {0};
          sprintf(filename, "PICT%03d.JPG", take_picture_count);

          Serial.print("Save taken picture as ");
          Serial.print(filename);
          Serial.println("");

          /* Save to SD card as the finename */

          File myFile = theSD.open(filename, FILE_WRITE);
          myFile.write(img.getImgBuff(), img.getImgSize());
          myFile.close();
        }

      take_picture_count++;
    }
}

For this sample please do the setup as follow: - Set the Preview image to QVGA - Set the frame rate to 60 FPS, to acquire 60 frames of data - take a JPEG picture and save it to SDCard.

After doing the setup, please follow these steps: "initial setting", "setup()", "preview callback" and "loop()".

Please find the description of these steps here:

2.2.7.1. Initial setting

/*
 *  camera.ino - One minute interval time-lapse Camera
 *  Copyright 2018 Sony Semiconductor Solutions Corporation
 *
 *  This library is free software; you can redistribute it and/or
 *  modify it under the terms of the GNU Lesser General Public
 *  License as published by the Free Software Foundation; either
 *  version 2.1 of the License, or (at your option) any later version.
 *
 *  This library is distributed in the hope that it will be useful,
 *  but WITHOUT ANY WARRANTY; without even the implied warranty of
 *  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
 *  Lesser General Public License for more details.
 *
 *  You should have received a copy of the GNU Lesser General Public
 *  License along with this library; if not, write to the Free Software
 *  Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
 *
 *  This is a test app for the camera library.
 *  This library can only be used on the Spresense with the FCBGA chip package.
 */

#include <SDHCI.h>
#include <stdio.h>  /* for sprintf */

#include <Camera.h>

#define BAUDRATE                (115200)

SDClass  theSD;
int take_picture_count = 0;

First, when using the Spresense camera library, you need to include the header file <Camera.h>.

#include <Camera.h>

Including this header file makes it possible to use the theCamera instance.

After that, we define variables that are common to the callback function and the loop() function.

int take_picture_count = 0;

The take_picture_count, is a variable which is counting up every second when takePicture() is called. When you use this variable you can control the file name and maximum file number. (In this example, it stops taking picture when picture number is 100.)

2.2.7.2. preview callback

/**
 * Callback from Camera library when video frame is captured.
 */

void CamCB(CamImage img)
{

  /* Check the img instance is available or not. */

  if (img.isAvailable())
    {

      /* If you want RGB565 data, convert image data format to RGB565 */

      img.convertPixFormat(CAM_IMAGE_PIX_FMT_RGB565);

      /* You can use image data directly by using getImgSize() and getImgBuff().
       * for displaying image to a display, etc. */

      Serial.print("Image data size = ");
      Serial.print(img.getImgSize(), DEC);
      Serial.print(" , ");

      Serial.print("buff addr = ");
      Serial.print((unsigned long)img.getImgBuff(), HEX);
      Serial.println("");
    }
  else
    {
      Serial.print("Failed to get video stream image\n");
    }
}

This function is registered on startStreaming() method and it will be called when the preview image is available. This function, check the availability of the CamImage instance at first, and then convert pixel format from YUV422 to RGB565. After the conversion, message of data size and memory address of the image will be sent via Serial. Generally, at this moment, add implementation that the image data will be displayed on a connected LCD monitor like a camera view finder.

2.2.7.3. setup()

/**
 * @brief Initialize camera
 */
void setup()
{

  /* Open serial communications and wait for port to open */

  Serial.begin(BAUDRATE);
  while (!Serial)
    {
      ; /* wait for serial port to connect. Needed for native USB port only */
    }


  /* begin() without parameters means that
   * number of buffers = 1, 30FPS, QVGA, YUV 4:2:2 format */

  Serial.println("Prepare camera");
  theCamera.begin();


  /* Start video stream.
   * If received video stream data from camera device,
   *  camera library call CamCB.
   */

  Serial.println("Start streaming");
  theCamera.startStreaming(true, CamCB);

  /* Auto white balance configuration */

  Serial.println("Set Auto white balance parameter");
  theCamera.setAutoWhiteBalanceMode(CAM_WHITE_BALANCE_DAYLIGHT);

  /* Set parameters about still picture.
   * In the following case, QUADVGA and JPEG.
   */

  Serial.println("Start streaming");
  theCamera.setStillPictureImageFormat(
     CAM_IMGSIZE_QUADVGA_H,
     CAM_IMGSIZE_QUADVGA_V,
     CAM_IMAGE_PIX_FMT_JPG);
}

The setup() initially sets up serial for message display, then calls Camera’s begin() method function. After that, it will enable preview callback function setting and callback with startStreaming(), and set auto white balance mode to day-light by calling setAutoWhiteBalanceMode(). Finally, with setStillPictureImageFormat(), setting for Quad HD size photograph is done.

2.2.7.4. loop()

/**
 * @brief Take picture with format JPEG per second
 */

void loop()
{
  sleep(1); /* wait for one second to take still picture. */

  /* You can change the format of still picture at here also, if you want. */

  /* theCamera.setStillPictureImageFormat(
   *   CAM_IMGSIZE_HD_H,
   *   CAM_IMGSIZE_HD_V,
   *   CAM_IMAGE_PIX_FMT_JPG);
   */

  /* This sample code can take 100 pictures in every one second from starting. */

  if (take_picture_count < 100)
    {

      /* Take still picture.
      * Unlike video stream(startStreaming) , this API wait to receive image data
      *  from camera device.
      */

      Serial.println("call takePicture()");
      CamImage img = theCamera.takePicture();

      /* Check availability of the img instance. */
      /* If any error was occured, the img is not available. */

      if (img.isAvailable())
        {
          /* Create file name */

          char filename[16] = {0};
          sprintf(filename, "PICT%03d.JPG", take_picture_count);

          Serial.print("Save taken picture as ");
          Serial.print(filename);
          Serial.println("");

          /* Save to SD card as the finename */

          File myFile = theSD.open(filename, FILE_WRITE);
          myFile.write(img.getImgBuff(), img.getImgSize());
          myFile.close();
        }

      take_picture_count++;
    }
}

In the loop() function, wait for 1 second by using sleep() function at the beginning. After 1 second, it checks whether the value of the take_picture_count variable exceeds 100. If it does not exceed 100, it enters the photo shooting process. In the shooting process, the takePicture() method function is called, take a picture and assign the obtained picture to the img variable. Then check that if the img variable is available with the isAvailable() function, and save the acquired image to the SD card. Finally, take_picture_count is incremented (by 1) and the loop() function is terminated.

The explanation for using the sample finished here. Let’s create an original camera device using the Spresense camera.

2.3. DNNRT Library

The DNN Runtime library can perform recognition processing using the Deep Neural Network (DNN) using trained models by Neural Network Libraries or Neural Network Console provided by Sony.

DNNRT library has 2 parts. One is DNNRT as a core library, and second is DNNVariable which is for using data I/O.

This section explains how to use DNNRT for recognizing handwritten characters from 0 to 9 (number_recognition.ino) as an example.

2.3.1. Preparation of a trained model

To use the DNNRT, first you need a trained model. This section explains the preparation of the trained model for this example.

2.3.1.1. Installation of Neural Network Console

We use the Neural Network Console (NNC) to create a trained model. You can use cloud version from NNC website.

2.3.1.2. Training

Start up NNC and select the sample project image_recognition.MNIST.LeNet, then it shows the dialog to create a new project. In this tutorial, the name of new project is myproject.
After selecting myproject , editor of NNC is appeared.

Figure 7. Editor
In this tutorial, there is no need to edit the network model. Press Run on the upper right of the edit window to start training. The graph showing the training result will be updated.

Figure 8. Start training
If Training Completed message is displayed on the console at the bottom of the window, training is complete.

Figure 9. Training Completed

TIP

Machine learning takes a very long time.
After training completed, evaluate trained model. Press Run on the upper right of the training window to start evaluation.

Figure 10. Start evaluation
After evaluation completed, evaluation results will be shown. You can confirm that images in x:image and the numbers in y:label are completely matched.

Download this project as NNB file to use DNNRT. Select NNB(NNabla C Runtime file format) from drop down list, and press Download Project to download as result.nnb file.

Figure 11. Download trained model
Rename downloaded trained model file to network.nnb, and copy both the trained model as network.nnb and an image which you want to recognize into SD Card.

TIP

This example can support only PGM (Portable Grayscale Map) file format for the image. If you can’t make PGM file, the example has sample data for a quick trial here: sample PGM image files

You are now ready to run the number_recognition.ino example.

2.3.2. Handwritten character recognition example

The trained model created by image_recognition.MNIST.LeNet takes one image of 28 x 28 size and outputs 10 arrays. These 10 arrays correspond to the numbers recognized by the index, and the probability of each number is output in the array. For example, at the head of an array (index 0), the probability that the input image is the number "0" is output.

Example of displaying the probability of each number

DNNVariable output = dnnrt.outputVariable(0);

Serial.println(output[0]);   // probability 0 of input image.
 ...
Serial.println(output[9]);   // probability 9 of input image.

In number_recognition.ino, the index with the highest probability from the array of output probabilities is serially outputted together with the probability as the recognized number.

You can change the image you want to input in the following line of number_recognition.ino.

  File pgmfile("number4.pgm");

2.4. EEPROM Library

EEPROM is a memory that can hold non-volatile data. This means that the data stored in the EEPROM will be retained while the power is off. The Spresense main board does not have the EEPROM mounted, but an EEPROM is emulated using the SPI Flash memory. This library allows writing and reading to the EEPROM emulated by the SPI Flash.

The API specification of the EEPROM library is documented in the Arduino EEPROM library. Also see README.md for details of advanced features.

The examples for the EEPROM library are available from the menu of Arduino IDE: File → Examples → Examples for Spresense EEPROM. The EEPROM library is compatible with Arduino, so you can use the same example sketches as Arduino.

The default size of the EEPROM is 4000 bytes, that is determined by the value of E2END defined in EEPROM.h.

The maximum capacity of the SPI Flash available from your sketch is 4 Mbytes on the Spresense board. As long as the E2END is the smaller than this maximum capacity, it’s possible to increase the size of the EEPROM by changing the E2END definition.

2.5. eMMC Library

2.5.1. Overview

The eMMC library is a library for accessing eMMC device on the Spresense Add-on board. The eMMC can be accessed from your program or optionally you can configure the system to access the eMMC device via USB port on the Spresense extension board.

The eMMC library has an API structure similar to the Arduino SD library, and it can be easily ported from the existing sketch using the SD library.

A distinctive feature of the eMMC Library is providing the USB MSC (Mass Storage Class) function. You can directly access the file on the eMMC device by connecting to the USB on the Spresense extension board from your PC.

2.5.2. Functions

See API references eMMC Library API for the details.

Function Description

Function	Description
eMMC.begin()	Initialize the eMMC device.
eMMC.beginUsbMsc()	Start USB MSC (Mass Storage Class) function.
eMMC.endUsbMsc()	Stop USB MSC (Mass Storage Class) function.
eMMC.format()	Format the eMMC device with FAT32 filesystem by default.
eMMC.open()	Opens a file on the eMMC device. When the file is opened for writing, it will be created if it doesn’t exist yet. FILE_READ: open the file for reading, starts at the beginning of the file. FILE_WRITE: open the file for reading and writing, starts at the end of the file. This function returns `File` object. See File library for file operations.
eMMC.exists()	Tests whether a file or directory exists on the eMMC device.
eMMC.mkdir()	Create a directory on the eMMC device.
eMMC.rmdir()	Remove a directory from the eMMC device..
eMMC.remove()	Remove a file from the eMMC device.

eMMC.begin()

Initialize the eMMC device.

eMMC.beginUsbMsc()

Start USB MSC (Mass Storage Class) function.

eMMC.endUsbMsc()

Stop USB MSC (Mass Storage Class) function.

eMMC.format()

Format the eMMC device with FAT32 filesystem by default.

eMMC.open()

Opens a file on the eMMC device. When the file is opened for writing, it will be created if it doesn’t exist yet.
FILE_READ: open the file for reading, starts at the beginning of the file.
FILE_WRITE: open the file for reading and writing, starts at the end of the file.
This function returns File object. See File library for file operations.

eMMC.exists()

Tests whether a file or directory exists on the eMMC device.

eMMC.mkdir()

Create a directory on the eMMC device.

eMMC.rmdir()

Remove a directory from the eMMC device..

eMMC.remove()

Remove a file from the eMMC device.

2.5.3. Examples

Four sample sketches using eMMC library are provided here:

Example	Description
format.ino	This is an example to format the eMMC device. Execute it once when you use the eMMC device for the first time.
read_write.ino	This is an example to read and write files on the eMMC device.
UsbMsc.ino	This is a sample for the USB MSC function. If you run this sketch, the eMMC device on the Spresense extension board will be mounted as a drive on the PC and you will be able to access the eMMC device directly from the PC.
UsbMscAndFileOperation.ino	This is an application combining USB MSC function and file operations. For files on the eMMC device, file operation from an application program and the file operation from PC, with USB MSC, must be operated exclusively and can not be used at the same time. In this sample sketch, after acquiring and displaying the file list from the application program the USB MSC function is enabled. You can use the USB MSC function from the PC to operate the files in the eMMC device. Furthermore, by terminating the USB MSC function, file operation is performed again from within the program.

Example

Description

format.ino

This is an example to format the eMMC device. Execute it once when you use the eMMC device for the first time.

read_write.ino

This is an example to read and write files on the eMMC device.

UsbMsc.ino

This is a sample for the USB MSC function. If you run this sketch, the eMMC device on the Spresense extension board will be mounted as a drive on the PC and you will be able to access the eMMC device directly from the PC.

UsbMscAndFileOperation.ino

This is an application combining USB MSC function and file operations.
For files on the eMMC device, file operation from an application program and the file operation from PC, with USB MSC, must be operated exclusively and can not be used at the same time.
In this sample sketch, after acquiring and displaying the file list from the application program the USB MSC function is enabled. You can use the USB MSC function from the PC to operate the files in the eMMC device. Furthermore, by terminating the USB MSC function, file operation is performed again from within the program.

2.6. File Library

2.6.1. Overview

File library is a library commonly used by various storage libraries. File objects are generated by the open() function in the SDHCI, Flash and eMMC libraries. This library provides the file operations for the opened File object.

2.6.2. Functions

See API references File Library API for the details.

When File instance name is myFile, open the file as follow.

#include <SDHCI.h>
SDClass SD;
File myFile;

  myFile = SD.open("test.txt");

The following functions are provided for the myFile object.

Function	Description
myFile.write()	Write data to the file.
myFile.read()	Read from the file.
myFile.peek()	Read a byte from the file without advancing to the next one. That is, successive calls to peek() will return the same value, as will the next call to read().
myFile.available()	Check if there are any data available for reading from the file, and returns the number of bytes available.
myFile.flush()	Ensures that any data written to the file are physically saved to the storage. This is done automatically when the file is closed.
myFile.seek()	Seek to a new position in the file, which must be between 0 and the size of the file (inclusive).
myFile.position()	Get the current position within the file (i.e. the location to which the next byte will be read from or written to).
myFile.size()	Get the size of the file.
myFile.close()	Close the file, and ensure that any data written to it is physically saved to the storage.
myFile.bool()	Tests whether a file or directory exists on the storage.
myFile.name()	Returns the file name.
myFile.isDirectory()	Directories are special kinds of files, this function reports if the current file is a directory or not.
myFile.openNextFile()	Reports the next file or folder in a directory.
myFile.rewindDirectory()	rewindDirectory() will bring you back to the first file in the directory, used in conjunction with openNextFile().

Function

Description

myFile.write()

Write data to the file.

myFile.read()

Read from the file.

myFile.peek()

Read a byte from the file without advancing to the next one. That is, successive calls to peek() will return the same value, as will the next call to read().

myFile.available()

Check if there are any data available for reading from the file, and returns the number of bytes available.

myFile.flush()

Ensures that any data written to the file are physically saved to the storage. This is done automatically when the file is closed.

myFile.seek()

Seek to a new position in the file, which must be between 0 and the size of the file (inclusive).

myFile.position()

Get the current position within the file (i.e. the location to which the next byte will be read from or written to).

myFile.size()

Get the size of the file.

myFile.close()

Close the file, and ensure that any data written to it is physically saved to the storage.

myFile.bool()

Tests whether a file or directory exists on the storage.

myFile.name()

Returns the file name.

myFile.isDirectory()

Directories are special kinds of files, this function reports if the current file is a directory or not.

myFile.openNextFile()

Reports the next file or folder in a directory.

myFile.rewindDirectory()

rewindDirectory() will bring you back to the first file in the directory, used in conjunction with openNextFile().

2.6.3. Examples

A sample sketch using File library are provided here:

Example	Description
read_write.ino	This is a example to read and write files.

Example

Description

read_write.ino

This is a example to read and write files.

2.7. Flash Library

2.7.1. Overview

The Flash library is a library for accessing flash device on the Spresense main board.

The Flash library has an API structure similar to the Arduino SD library, and it can be easily ported from the existing sketch using the SD library.

2.7.2. Functions

See API references Flash Library API for the details.

Function Description

Function	Description
Flash.begin()	Always return true.
Flash.beginUsbMsc()	Start USB MSC (Mass Storage Class) function.
Flash.endUsbMsc()	Stop USB MSC (Mass Storage Class) function.
Flash.format()	Format the Flash device with FAT32 filesystem by default.
Flash.open()	Opens a file on the Flash device. When the file is opened for writing, it will be created if it doesn’t exist yet. FILE_READ: open the file for reading, starts at the beginning of the file. FILE_WRITE: open the file for reading and writing, starts at the end of the file. This function returns `File` object. See File library for file operations.
Flash.exists()	Tests whether a file or directory exists on the Flash device.
Flash.mkdir()	Create a directory on the Flash device.
Flash.rmdir()	Remove a directory from the Flash device..
Flash.remove()	Remove a file from the Flash device.

Flash.begin()

Always return true.

Flash.beginUsbMsc()

Start USB MSC (Mass Storage Class) function.

Flash.endUsbMsc()

Stop USB MSC (Mass Storage Class) function.

Flash.format()

Format the Flash device with FAT32 filesystem by default.

Flash.open()

Opens a file on the Flash device. When the file is opened for writing, it will be created if it doesn’t exist yet.
FILE_READ: open the file for reading, starts at the beginning of the file.
FILE_WRITE: open the file for reading and writing, starts at the end of the file.
This function returns File object. See File library for file operations.

Flash.exists()

Tests whether a file or directory exists on the Flash device.

Flash.mkdir()

Create a directory on the Flash device.

Flash.rmdir()

Remove a directory from the Flash device..

Flash.remove()

Remove a file from the Flash device.

2.7.3. Examples

Four sample sketches using Flash library are provided here:

Example Description

Example	Description
format.ino	This is an example to format the flash device. Although the flash device is formatted and shipped, if it is necessary to re-format, please execute this sketch. The formatting deletes all the user data except for firmwares. EEPROM data created on the flash are also deleted. This sketch uses a serial monitor. Open the serial monitor at a communication rate of 115200 baud. Start formatting by typing `y` from the serial monitor. During formatting, the 4 LEDs will light up, and after formatting is complete, the LEDs will blink. The format takes about 40 seconds. Be careful not to turn off the power during formatting.
read_write.ino	This is an example to read and write files on the Flash device.

format.ino

This is an example to format the flash device. Although the flash device is formatted and shipped, if it is necessary to re-format, please execute this sketch. The formatting deletes all the user data except for firmwares. EEPROM data created on the flash are also deleted.
This sketch uses a serial monitor. Open the serial monitor at a communication rate of 115200 baud.
Start formatting by typing y from the serial monitor.
During formatting, the 4 LEDs will light up, and after formatting is complete, the LEDs will blink.
The format takes about 40 seconds. Be careful not to turn off the power during formatting.

read_write.ino

This is an example to read and write files on the Flash device.

2.8. GNSS Library

The GNSS Library is for controlling the GNSS features built in Spresense and acquiring positioning information. This library is available in the Arduino environment.

GNSS is a built-in feature in the CXD5602 and it does not require any pin allocations or shields. An antenna is also included on the Spresense main board.

Two examples are provided:

gnss.ino is a simple demonstration of how GNSS provides output to the USB serial port. This example uses the GNSS library output format directly. For more information please refer to GNSS.h

gnss_tracker is a more complex tracker, which saves data to the SD card. It converts the output into NMEA format for compatibility with existing GNSS modules and software.

2.8.1. More Information

Please refer to Spresense SDK GNSS documentation for further information for the GNSS functions.

2.9. LowPower Library

LowPower library provides support for power-saving features of Spresense:

The sleep state has two modes: Cold Sleep or Deep Sleep

Deep Sleep

Sleep mode with the lowest power consumption. PMIC (Power Management IC) turns the power on for CXD5247 only, and CXD5602’s power is turned off.

Cold Sleep

Sleep mode with minimum power consumption. In this mode other than CXD5247, CXD5602’s power is turned on as well. Although this mode has more power consumption than Deep Sleep, it can be waken up from Cold Sleep state by GPIO trigger.

About the power consumption during sleeping, when the SD card is inserted on the extension board, the current consumption increases by about 5 mA for the power supply to the SD card.

Get the cause waked-up from the sleep state
Enable/Disable the waked-up cause
Dynamic switching to three clock modes

CLOCK_MODE_156MHz

This is the clock mode with the highest performance. This clock mode is selected at startup. The CPU clock is 156 MHz using PLL.

CLOCK_MODE_32MHz

Reduce power consumption by decreasing the system clock to 32 MHz. When this clock mode is selected, the core voltage of CXD5602 is 0.7V. The CPU clock is 32 MHz using a PLL.

CLOCK_MODE_8MHz

Decrease the system clock to 8 MHz. This clock mode is the lowest power consumption. The core voltage is 0.7 V, and the CPU operates with the internal oscillator without PLL. If the PLL is not used, the power consumption can be further reduced by turning TCXO power off.

In general, the lower the clock, the lower power consumption, but the lower performance. Please note, not all of hardware functions operate in low clock mode. Details will be described later.

For more details of this library, see the API reference LowPower Library API.

2.9.1. The clock mode

By specifying either CLOCK_MODE_156MHz, CLOCK_MODE_32MHz or CLOCK_MODE_8MHz as an argument of clockMode(), the system clock with application CPU is dynamically changed. By lowering the clock, the power consumption can be reduced.

The current consumption at the battery terminals is shown for each clock mode. The measurement conditions are only using the Spresense main board which supplies 3.7V power from the battery connector. The software is in idle state.

Figure 12. Battery current measurement with LowPower

Clock Mode	Battery current
CLOCK_MODE_156MHz	6.68 mA
CLOCK_MODE_32MHz	3.66 mA
CLOCK_MODE_8MHz	3.20 mA
CLOCK_MODE_8MHz + TCXO=PowerOff	1.16 mA

Clock Mode

Battery current

CLOCK_MODE_156MHz

6.68 mA

CLOCK_MODE_32MHz

3.66 mA

CLOCK_MODE_8MHz

3.20 mA

CLOCK_MODE_8MHz + TCXO=PowerOff

1.16 mA

This measurement results include current consumption by the Power-LED on the main board.

Aside from controlling clock mode by the user application, the clock mode may be switched automatically inside the system.

In USB communication function such as USB MSC (Mass Storage), the clock mode will automatically be CLOCK_MODE_156MHz.
In GNSS positioning function, the clock mode will automatically be CLOCK_MODE_156MHz or CLOCK_MODE_32MHz.
During the mounting process of the SD card, it is automatically clocked up to CLOCK_MODE_156 MHz and returned to the previous state after mounting.

Therefore, note that it does not necessarily change to the clock mode specified by the user. The current clock mode can be obtained with the getClockMode() function to check if the clock mode has been changed.

Also, not all functions work in low clock mode. The restrictions are as follow:

If you use the audio functions, set the clock mode to CLOCK_MODE_156 MHz
If you use the camera functions, set the clock mode to CLOCK_MODE_156MHz or CLOCK_MODE_32MHz

2.9.2. Examples

For the basic usage of this library, see the examples of this library through the Arduino IDE menu File → Examples → Examples for Spresense LowPower.

Example	Description
ExternalWakeup	Wake-up by GPIO from cold sleep
TimedWakeup	Wake-up by RTC alarm from cold sleep
TimedWakeupDeep	Wake-up by RTC alarm from deep sleep
Reboot	Reboot the system
WatchdogReboot	Reboot the system by watchdog timer
ClockChange	Switch the clock mode to 156MHz/32MHz/8MHz.

Example

Description

ExternalWakeup

Wake-up by GPIO from cold sleep

TimedWakeup

Wake-up by RTC alarm from cold sleep

TimedWakeupDeep

Wake-up by RTC alarm from deep sleep

Reboot

Reboot the system

WatchdogReboot

Reboot the system by watchdog timer

ClockChange

Switch the clock mode to 156MHz/32MHz/8MHz.

2.10. MultiCore MP Library

2.10.1. Overview

MultiCore MP Library is a library to support multi-core programming in Arduino environment. CPU core that is launched by default is called MainCore, and CPU cores are booted from MainCore are called SubCore. The total number of SubCore are 5, from SubCore 1 to SubCore 5.

You can select MainCore and SubCore 1 ~ 5 by selecting Tools→ Core in Arduino IDE menu. Program on each core and load the binary compiled for each core on the Spresense board and run it. By using the MP library, you can program each core and can work them.

2.10.2. Functions

The MP library mainly provides the following functions:

Start and stop SubCore(s)
Inter-core communication function
Inter-core exclusive control function
Allocate and release shared memory
Exclusive log output function

For details on each API, refer to the API Reference Manual MP Library API.

2.10.2.1. MP class

By including MP.h, an object named MP will be created.

#include <MP.h>

The following table is a list of functions provided by the MP class. Please note some functions are different between MainCore and SubCore, and some functions are not supported by SubCore (*: supported, -: not supported)

Function Description MainCore SubCore

Function	Description	MainCore	SubCore
MP.begin()	Control multi-core boot. (MainCore) Start SubCore with the specified number in the argument. (SubCore) Called without an argument, notifies MainCore that startup is complete. Please refer to Boot SubCore for details.	*	*
MP.end()	Control multi-core shutdown. (MainCore) Stop SubCore with the specified number in the argument. (SubCore) Do nothing. For details, please refer to Shutdown SubCore.	*	*
MP.Send()	Send data to the specified core. (MainCore) Send data to SubCore with the specified number in the argument. (SubCore) Send data to SubCore with the specified number in the argument. Sent to MainCore without argument of subid. For details, refer to Communication between cores.	*	*
MP.Recv()	Receive data from the specified core. (MainCore) Receive data from SubCore with the specified number in the argument. (SubCore) Receive data from SubCore with the specified number in the argument. Received from MainCore without argument of subid. For details, refer to Communication between cores.	*	*
MP.RecvTimeout()	Set the receiver mode and timeout value for Recv(). There are three waiting modes, which are selected by argument. For details, refer to Communication between cores.	*	*
MP.GetRecvTimeout()	Get the value set by RecvTimeout().	*	*
MP.Virt2Phys()	Convert the virtual address specified by the argument to a physical address. Physical address map is common for all cores. On the other hand, virtual address map is different for each core. Since SubCore runs on virtual address, use this function to convert it to a physical address when sending an address to other core. See Application memory for details.	*	*
MP.GetMemoryInfo()	Get the status of application’s available memory. usedMem: Used total memory size (Byte) freeMem: Available total memory size (Byte) largestFreeMem: Continuous maximum available memory size (Byte) See Application memory for details.	*	*
MP.EnableConsole()	Enable input from serial monitor (Serial). Make sure that input from Serial is accepted only by one core. After the system startup, only MainCore is enabled.	*	*
MP.DisableConsole()	Disable input from serial monitor (Serial). If you want to change the Serial input to the specific SubCore, call DisableConsole() from MainCore and call EnableConsole() on the SubCore. For usage, please refer to `Shell` in Examples.	*	*
MP.AllocSharedMemory()	Allocate shared memory and return the first physical address. Allocation and release of shared memory are in 128 Kbyte units. Allocate 128 Kbyte aligned memory from the size specified in the argument. If it cannot allocate due to lack of memory, NULL is returned. For usage, please refer to `SharedMemory` in Examples.	*	-
MP.FreeSharedMemory()	Release the shared memory by passing the address allocated by AllocSharedMemory().	*	-

MP.begin()

Control multi-core boot.
(MainCore) Start SubCore with the specified number in the argument.
(SubCore) Called without an argument, notifies MainCore that startup is complete.
Please refer to Boot SubCore for details.

MP.end()

Control multi-core shutdown.
(MainCore) Stop SubCore with the specified number in the argument.
(SubCore) Do nothing.
For details, please refer to Shutdown SubCore.

MP.Send()

Send data to the specified core.
(MainCore) Send data to SubCore with the specified number in the argument.
(SubCore) Send data to SubCore with the specified number in the argument. Sent to MainCore without argument of subid.
For details, refer to Communication between cores.

MP.Recv()

Receive data from the specified core.
(MainCore) Receive data from SubCore with the specified number in the argument.
(SubCore) Receive data from SubCore with the specified number in the argument. Received from MainCore without argument of subid.
For details, refer to Communication between cores.

MP.RecvTimeout()

Set the receiver mode and timeout value for Recv().
There are three waiting modes, which are selected by argument.
For details, refer to Communication between cores.

MP.GetRecvTimeout()

Get the value set by RecvTimeout().

MP.Virt2Phys()

Convert the virtual address specified by the argument to a physical address. Physical address map is common for all cores. On the other hand, virtual address map is different for each core. Since SubCore runs on virtual address, use this function to convert it to a physical address when sending an address to other core.
See Application memory for details.

MP.GetMemoryInfo()

Get the status of application’s available memory.
usedMem: Used total memory size (Byte)
freeMem: Available total memory size (Byte)
largestFreeMem: Continuous maximum available memory size (Byte)
See Application memory for details.

MP.EnableConsole()

Enable input from serial monitor (Serial).
Make sure that input from Serial is accepted only by one core. After the system startup, only MainCore is enabled.

MP.DisableConsole()

Disable input from serial monitor (Serial).
If you want to change the Serial input to the specific SubCore, call DisableConsole() from MainCore and call EnableConsole() on the SubCore.
For usage, please refer to Shell in Examples.

MP.AllocSharedMemory()

Allocate shared memory and return the first physical address. Allocation and release of shared memory are in 128 Kbyte units. Allocate 128 Kbyte aligned memory from the size specified in the argument. If it cannot allocate due to lack of memory, NULL is returned.
For usage, please refer to SharedMemory in Examples.

MP.FreeSharedMemory()

Release the shared memory by passing the address allocated by AllocSharedMemory().

2.10.2.2. MPMutex class

This provides functions for exclusive control between multiple cores. Create a mutex object by specifying MP_MUTEX_ID number (0 to 10) as an argument.

#include <MPMutex.h>

MPMutex mutex(MP_MUTEX_ID0);

The functions provided by MPMutex are shown below.

Function Description MainCore SubCore

Function	Description	MainCore	SubCore
mutex.Trylock()	Lock mutex. If it can be locked, return 0. If it cannot be locked, return non-zero value. For usage, please refer to `Mutex` in Examples.	*	*
mutex.Unlock()	Unlock mutex locked by Trylock(). If it can be unlocked, return 0. If it cannot be unlocked, return non-zero value. For usage, please refer to `Mutex` in Examples.	*	*

mutex.Trylock()

Lock mutex. If it can be locked, return 0. If it cannot be locked, return non-zero value.
For usage, please refer to Mutex in Examples.

mutex.Unlock()

Unlock mutex locked by Trylock(). If it can be unlocked, return 0. If it cannot be unlocked, return non-zero value.
For usage, please refer to Mutex in Examples.

2.10.2.3. MPLog() function

This is a function to output log in multi-core environment.

Function Description MainCore SubCore

Function	Description	MainCore	SubCore
MPLog()	This is a function macro to output log with exclusive control between cores. The arguments can use the same format as the general printf(). (MainCore) `[Main]` is displayed as the prefix of the log. (SubCore) `[Sub number]` is displayed as the prefix of the log. It is possible to identify per-core log output in multi-core environment. This is also available form an interrupt handler, but the log may be interrupted without exclusive control and displayed.	*	*

MPLog()

This is a function macro to output log with exclusive control between cores. The arguments can use the same format as the general printf().
(MainCore) [Main] is displayed as the prefix of the log.
(SubCore) [Sub number] is displayed as the prefix of the log.
It is possible to identify per-core log output in multi-core environment.
This is also available form an interrupt handler, but the log may be interrupted without exclusive control and displayed.

2.10.2.4. Application memory

The total SRAM memory for the application is 1.5 MBytes (= 1536 KBytes). The memory is divided into 128 KByte x 12 memory tiles. Each CPU and SRAM are connected with bus matrix to units of memory tiles. If only one CPU accesses a tile, the CPU can read and write the SRAM without wait-cycles. (This means the same performance as cache on general CPU architectures). On the other hand, if multiple CPUs access the same memory tile, memory access is slowed down. To prevent the above access conflict, each SubCore is using the dedicated memory tiles and runs on the memory tiles.

The mapping of SRAM 1.5 MByte in Arduino environment is shown below.

Figure 13. Application SRAM

MainCore uses the first 6 tiles (768 KBytes). The remaining 6 tiles (768 KBytes) are shared and used by SubCore and other libraries.

In the figure above, 256 KByte is allocated to SubCore1 and 128 KByte is allocated to SubCore2. SubCore works on a virtual address beginning from address 0. If it is necessary to refer to a common address for inter-CPU communication etc, convert it to a physical address with MP.Virt2Phys().

In addition, the second half of 768 KByte shared memory may be used by various libraries besides SubCore. The used memory size for each library and each usecase is shown below.

Library UseCase Used Memory Size

Library	UseCase	Used Memory Size
Audio, Sensing	For various buffers	256 KByte
Audio	For loading the `MP3DEC` decoder during MP3 playback	128 KByte
Audio	For loading the `MP3ENC` encoder during MP3 recording	128 KByte
Audio	For loading the `WAVDEC` decoder during WAV playback	256 KByte
Audio	For loading the `SRC` sampling rate converter during WAV recording (except for the case of sampling rate 48kHz)	256 KByte
DNNRT	For loading the `dnnrt-mp`	128 KByte

Audio, Sensing

For various buffers

256 KByte

Audio

For loading the MP3DEC decoder during MP3 playback

128 KByte

Audio

For loading the MP3ENC encoder during MP3 recording

128 KByte

Audio

For loading the WAVDEC decoder during WAV playback

256 KByte

Audio

For loading the SRC sampling rate converter during WAV recording (except for the case of sampling rate 48kHz)

256 KByte

DNNRT

For loading the dnnrt-mp

128 KByte

The memory size required for SubCore depends on the user application. When compiling a sketch, the following message is output to the log of compilation result. This value shows the memory size that has been allocated for the SubCore.

In use case of user application, it may not be possible to operate with insufficient memory. You can check the memory usage by using MP.GetMemoryInfo().

Example code is shown below.

{
  int usedMem, freeMem, largestFreeMem;

  MP.GetMemoryInfo(usedMem, freeMem, largestFreeMem);

  MPLog("Used:%4d [KB] / Free:%4d [KB] (Largest:%4d [KB])\n",
        usedMem / 1024, freeMem / 1024, largestFreeMem / 1024);
}

2.10.2.5. Boot SubCore

Call MP.begin(1~5) from MainCore to start a SubCore by using number (1 to 5) as an argument. On the started SubCore side, the completion of boot up is notified by calling MP.begin() without an argument.

2.10.2.6. Shutdown SubCore

Call MP.end(1~5) from MainCore to shutdown a SubCore by using number (1 to 5) as an argument. The specified SubCore is shutdowned.

2.10.2.7. Communication between cores

You can send and receive arbitrary data using the MP.Send() and MP.Recv() functions.
There are 3 modes for waiting for reception, and the mode is set by MP.RecvTimeout(). The default is MP_RECV_BLOCKING.

MP_RECV_BLOCKING (0): The mode for reception to wait forever until it receives data. (Default)
MP_RECV_POLLING (0xffffffff): The mode for polling reception. If there is no received data when Recv() is called, it will be returned immediately. In this mode there is no waiting for reception.
Time value (Other value than above): Set the waiting timeout time (milliseconds). It will wait for the specified time when calling Recv(), and it will be timed out if the received data does not come.

There are two communication methods for MP.Send() and MP.Receive(): passing data and passing address.

Communication with data
- Prototype
  
  int MP.Send(int8_t msgid, uint32_t msgdata, int subid)
  int MP.Recv(int8_t *msgid, uint32_t *msgdata, int subid)
- Argument
  
  msgid: User can define zero or a positive Message ID (8 bit signed data). User cannot use a negative value.
  msgdata: User can define arbitrary Message Data (32 bit data).
  subid: Specify a SubCore number to communicate. If subid is 0 (optional), it communicates with MainCore.
- Return
  
  Returns 0 if successful, non-zero if error.
- Description
  
  Send both Message ID and Message Data by Send() and receive these values by Recv().
  
  Send() runs asynchronously. It has an 8-stage transmit FIFO buffer internally, and Send() function immediately returns 0 after pushing transmission data into the FIFO buffer and before the destination core receives data. If this 8-stage buffer becomes full, Send() function returns -EAGAIN(-11).
  On the other hand, the receiver also has an 8-stage receive FIFO buffer. If this receive buffer becomes full, it is judged as the system abnormal condition and the program assert. If this happens, please check whether the receiving core is working properly or the communication load is too high.
Communication with address
- Prototype
  
  int Send(int8_t msgid, void *msgaddr, int subid)
  int Recv(int8_t *msgid, void *msgaddr, int subid)
- Argument
  
  msgid: User can define zero or a positive Message ID (8 bit signed data). User cannot use a negative value.
  msgdata: User can define arbitrary Message Address (pointer).
  subid: Specify a SubCore number to communicate. If subid is noting, it communicates with MainCore.
- Return
  
  Returns 0 if successful, non-zero if error.
- Description
  
  Send both Message ID and Message Address by Send() and receive these values by Recv(). This function is used when you send only the pointer address of data to communicate. When sending an address from SubCore, the conversion from virtual address to physical address is required. The address conversion of msgaddr is performed inside the Send() function. However, if the msgaddr pointer points the data with address information, you have to convert the address from virtual to physical address.
  
  As mentioned above, Send() is asynchronous. Send() cannot guarantee that the destination core has received the data. Therefore, if the data pointed to by the msgaddr pointer is rewritten immediately after Send(), the received data is broken. To support synchronous communication, you have to implement the communication for acknowledge. Or, it is necessary to add a flag to guarantee data in the data packet.

For usage of Send()/Recv() function, please refer to Message in Examples.

Message/MessageData: Example for data communication
Message/MessageHello: Example for address communication

2.10.2.8. Available libraries from SubCore

All of Arduino core libraries are available from SubCore.

Some Spresense-specific libraries cannot be used from SubCore. If you include them, the following error message is output at compile time.

XXX library is NOT supported by SubCore.

The following libraries are available from SubCore.

Library	MainCore	SubCore	Description
MP	〇	〇
RTC	〇	〇	Alarm function is not available for SubCore.
SPI	〇	〇
Servo	〇	〇
SoftwareSerial	〇	〇
Watchdog	〇	〇
Wire	〇	〇
Audio	〇	-
Camera	〇	-
DNNRT	〇	-
GNSS	〇	-
LowPower	〇	-
LTE	〇	-
Sensing	〇	-
Storage (EEPROM, eMMC, File, Flash, SDHCI)	〇	-

Library

MainCore

SubCore

Description

〇

RTC

〇

Alarm function is not available for SubCore.

SPI

〇

Servo

〇

SoftwareSerial

〇

Watchdog

〇

Wire

〇

Audio

〇

Camera

〇

DNNRT

〇

GNSS

〇

LowPower

〇

LTE

〇

Sensing

〇

Storage (EEPROM, eMMC, File, Flash, SDHCI)

〇

2.10.2.9. SubCore Programming TIPS

How to increase heap size of SubCore program

The memory size required by SubCore is determined automatically at compile time, hereby explains the detailed logic.

The size up to .stack is determined statically at compile time, but the size of .heap depends on the user program. A minimum 32 KBytes of heap memory is assigned, and the final total size is 128 KByte alignment. The heap memory is allocated up to the aligned upper limit.

Figure 14. SubCore program size

By the above rule, a heap area of at least 32 KBytes is assigned. However, the user programs may need more heap space. If you want to increase the heap area, use the USER_HEAP_SIZE() macro on your sketch.

USER_HEAP_SIZE(64 * 1024);

In this case, the heap memory of 64 KByte or more is always assigned.

Compile options to identify SubCore

When building SubCore, -DSUBCORE=core number is specified as compile option. If you want to divide the code by ifdef for each core in common source code, please refer to the following.

#if   (SUBCORE == 1)
// SubCore1 building

#elif (SUBCORE == 2)
// SubCore2 building

#elif (SUBCORE == 3)
// SubCore3 building

#elif (SUBCORE == 4)
// SubCore4 building

#elif (SUBCORE == 5)
// SubCore5 building

#else
// MainCore building

#endif

By applying the following description to user sketch, you can prevent the wrong core from being selected and built or uploaded.

Build error if SubCore is selected in MainCore’s sketch

#ifdef SUBCORE
#error "Core selection is wrong!!"
#endif

Build error if MainCore is selected in SubCore’s sketch

#ifndef SUBCORE
#error "Core selection is wrong!!"
#endif

Build error if the other core is selected in SubCore1’s sketch

#if (SUBCORE != 1)
#error "Core selection is wrong!!"
#endif

Sketch directory structure

You may want to refer to the common header file between MainCore and SubCore.
As a suggestion, for example, you can include a common common.h header file by using the following directory structure.

In your SubSketch.ino,

#include "common.h"

In your MainSketch.ino,

#include "SubSketch/common.h"

Another option is to put a common header file in the place of libraries in your Arduino environment.
If there is the header file under libraries, it is possible to include from any sketch.
You can add the header file into your libraries by specifying the folder containing the header files from the Arduino IDE menu Sketch→ Include Library→ Add .ZIP Library ….

2.10.3. Examples

For basic usage of the MP library, refer to example sketches under Arduino IDE menu File→ Examples→ Examples for Spresense→ MultiCore MP.

Example	Description
Boot	This is an example to boot 4 SubCores. Each launched SubCore controls a separate LED.
Message/MessageData	This is an example to communicate with data between MainCore and SubCore.
Message/MessageHello	This is an example to communicate with address between MainCore and SubCore.
Mutex	This is example for exclusive control using MPMutex between multi-core.
SharedMemory	This is an example to allocate and free the shared memory from MainCore.
Shell	This is an example for SubCore to run shell.
AudioFFT	This is an example for advanced application using SubCore. MainCore captures PCM 4ch data using Audio library and send their data to SubCore. SubCore displays the peak frequency by calculating FFT (Fast Fourier Transform) in real-time. It is possible to display the time series data to Arduino IDE serial plotter.

Example

Description

Boot

This is an example to boot 4 SubCores. Each launched SubCore controls a separate LED.

Message/MessageData

This is an example to communicate with data between MainCore and SubCore.

Message/MessageHello

This is an example to communicate with address between MainCore and SubCore.

Mutex

This is example for exclusive control using MPMutex between multi-core.

SharedMemory

This is an example to allocate and free the shared memory from MainCore.

Shell

This is an example for SubCore to run shell.

AudioFFT

This is an example for advanced application using SubCore. MainCore captures PCM 4ch data using Audio library and send their data to SubCore. SubCore displays the peak frequency by calculating FFT (Fast Fourier Transform) in real-time. It is possible to display the time series data to Arduino IDE serial plotter.

2.11. RTC Library

RTC library controls the Spresense’s Real-Time Clock which is just a monotonically increasing counter with the initial value of 0. RTC is managed as unix time and the initial value of 0 means January 1, 1970 at 00:00:00.

RTC can keep the time during the sleep state such as Deep Sleep and Cold Sleep. When the power is turned off, the RTC counter is reset to the zero value.

When you use this library, please first, call RTC.begin(). It takes about 2 seconds until the RTC XTAL is stable after power is supplied. After returning from RTC.begin(), you can use the various RTC functions.

This library provides some features. For more details of this library, see the API reference RTC Library API.

RTC.setTime() … Set the RTC clock
RTC.getTime() … Get the RTC clock
RTC.attachAlarm() … Register the RTC alarm handler
RTC.detachAlarm() … Unregister the RTC alarm handler
RTC.setAlarm() … Set the RTC alarm
RTC.setAlarmSeconds() … Set the RTC relative alarm of the specified seconds
RTC.cancelAlarm() … Cancel the RTC alarm

For basic usage of this library, see the examples of this library through the Arduino IDE menu File → Examples → Examples for Spresense RTC.

In addition to the RTC library, you can use the POSIX standard APIs such as strftime() and clock_gettime(CLOCK_REALTIME, tp) by adding #include <time.h>.

For more information, refer to NuttX User’s Manual.

Local time has been disabled in the NuttX configuration, so localtime() is not supported.

2.12. SDHCI Library

2.12.1. Overview

The SDHCI library is a library for accessing micro SD on the Spresense extension board. The card can be accessed from your program or optionally you can configure the system to access the SD Card via USB port on the Spresense extension board.

The SDHCI library has an API structure similar to the Arduino SD library, and it can be easily ported from the existing sketch using the SD library.

SD and SDHC cards are supported, but SDXC cards is not supported.
Formats of FAT12, FAT16 and FAT32 are supported, exFAT is not supported.
Capacity of up to 32 GB is supported.

Unlike using a simple SPI interface, Spresense accesses the SD card using dedicated hardware for SD Host Controller Interface, which has the following features:

Since the SD card is connected with a dedicated port, the SPI pins are not required. The SPI pins can be used for other purposes.
It is very fast compared with SPI communication. SDR25 transfer mode, at approximately 25 MB per second, is supported.

A distinctive feature of the SDHCI Library is providing the USB MSC (Mass Storage Class) function. You can directly access the file on the SD card by connecting to the USB on the Spresense extension board from your PC.

2.12.2. Functions

See API references SD card Library API for the details.

SDHCI library doesn’t have an instance of SDClass, therefore please define as below in your sketch.

#include <SDHCI.h>
SDClass SD;

Function Description

Function	Description
SD.begin()	Check if the SD card is mounted. After this function returns `true`, you can call the other APIs. If the SD card is not mounted, this function returns `false` after retrying for 2 seconds.
SD.beginUsbMsc()	Start USB MSC (Mass Storage Class) function.
SD.endUsbMsc()	Stop USB MSC (Mass Storage Class) function.
SD.format()	Format the SD card with FAT32 filesystem by default.
SD.open()	Opens a file on the SD card. When the file is opened for writing, it will be created if it doesn’t exist yet. FILE_READ: open the file for reading, starts at the beginning of the file. FILE_WRITE: open the file for reading and writing, starts at the end of the file. This function returns `File` object. See File library for file operations.
SD.exists()	Tests whether a file or directory exists on the SD card.
SD.mkdir()	Create a directory on the SD card.
SD.rmdir()	Remove a directory from the SD card..
SD.remove()	Remove a file from the SD card.

SD.begin()

Check if the SD card is mounted. After this function returns true, you can call the other APIs.
If the SD card is not mounted, this function returns false after retrying for 2 seconds.

SD.beginUsbMsc()

Start USB MSC (Mass Storage Class) function.

SD.endUsbMsc()

Stop USB MSC (Mass Storage Class) function.

SD.format()

Format the SD card with FAT32 filesystem by default.

SD.open()

Opens a file on the SD card. When the file is opened for writing, it will be created if it doesn’t exist yet.
FILE_READ: open the file for reading, starts at the beginning of the file.
FILE_WRITE: open the file for reading and writing, starts at the end of the file.
This function returns File object. See File library for file operations.

SD.exists()

Tests whether a file or directory exists on the SD card.

SD.mkdir()

Create a directory on the SD card.

SD.rmdir()

Remove a directory from the SD card..

SD.remove()

Remove a file from the SD card.

2.12.3. Examples

Use the SD card formatted with FAT32. Three sample sketches using SDHCI library are provided here:

Example	Description
read_write.ino	This is an example to read and write files on the SD card.
UsbMsc.ino	This is a sample for the USB MSC function. If you run this sketch, the SD card on the Spresense extension board will be mounted as a drive on the PC and you will be able to access the SD card directly from the PC.
UsbMscAndFileOperation.ino	This is an application combining USB MSC function and file operations. For files on the SD card, file operation from an application program and the file operation from PC, with USB MSC, must be operated exclusively and can not be used at the same time. In this sample sketch, after acquiring and displaying the file list from the application program the USB MSC function is enabled. You can use the USB MSC function from the PC to operate the files in the SD card. Furthermore, by terminating the USB MSC function, file operation is performed again from within the program.

Example

Description

read_write.ino

This is an example to read and write files on the SD card.

UsbMsc.ino

This is a sample for the USB MSC function. If you run this sketch, the SD card on the Spresense extension board will be mounted as a drive on the PC and you will be able to access the SD card directly from the PC.

UsbMscAndFileOperation.ino

This is an application combining USB MSC function and file operations.
For files on the SD card, file operation from an application program and the file operation from PC, with USB MSC, must be operated exclusively and can not be used at the same time.
In this sample sketch, after acquiring and displaying the file list from the application program the USB MSC function is enabled. You can use the USB MSC function from the PC to operate the files in the SD card. Furthermore, by terminating the USB MSC function, file operation is performed again from within the program.

2.13. Sensing Library

Spresense has the function of low-power always sensing and the high performance of edge computing. It is possible to create a sensor (called a logic sensor) that can output abstract data that fused various sensor data by signal processing / AI processing, etc., resulting in a smaller amount of data.
The Sensing Library provide a framework to use and create logic sensors by such sensor fusion.

Logic sensors can recognize and signal process that are used Multi Core, which is a feature of Spresense, to perform signal / recognition processing with other cores (hereinafter these cores are called DSP).

Key Features of the Sensing Library

Based on Publish-Subscribe Architecture, it is easy to execute and change the fusions of multiple sensors
It can use data that the sensor driver created by Arduino-supplied ecosystem sensor, and can create logic sensor and subscribed by Arduino application, the same function as SDK can be used from Arduino

Arduino library is basically a wrapper library the functions and APIs possessed by the SDK.
For more information, please refer to the SDK side.

2.13.1. Architecture and data flow

First, here is the Sensor Framework of Spresense.

Please refer to Sensor Fusion Framework for this.

Figure 15. Sensing Library (Publish/subscribe) Architecture

Next, the following is the architecture of the Sensing library for Arduino.

Figure 16. Sensing Library (Publish/subscribe) Architecture

By providing the Sensor Framework part provided by Spresense SDK as a "Sensing" library, the functionality equivalent to the SDK is realized.
For Arduino developers, we provide a framework for creating sophisticated logic sensors and applications using these sensors.
You can also create applications that use logic sensors that better meet your needs by creating logic sensors with a higher degree of abstraction yourself.

Classes of Sensing Library

SensorManagerClass:
It is a wrapper class of the class (Sensor Manager) that manages each sensor client in SDK.
SensorClient:
the base class of sensorclient which is an element of each sensor. Users do not call this class directly.
Inherit this class if you want to create your own logic sensor.
- AccelSensorClass:
  Each sensor client can use acceleration data by publishing the sensor data acquired from the physical sensor client of the acceleration sensor with the sensor sensor driver at Arduino.
- ApplicationSensorClass:
  The application is a sensor client for subscribing sensor data.
  - StepCountReaderClass:
    This is a sensor client for reading data of step counter sensor in application. It is created inheriting from ApplicationSensorClass.
- AesmClass:
  This is a logical sensor client for pedometer algorithm (AESM: Activity Engine and StepMeter) provided by Spresense SDK .
  It use an acceleration sensor.
  Because of logic sensor provided by SDK, publish/subscribe processing from Arduino is unnecessary.

2.13.2. API reference of each classes

It is shown below, the API reference for each classes.

2.13.2.1. SensorManagerClass

It is shown below, the API of SensorManagerClass.

begin

begin();

Begin Sensor Manager. Call it once before using the framework.

end

end();

End Sensor Manager. Call once after using the framework.

2.13.2.2. SensorClient

It is shown below, the API of SensorClient. This class is a base class for each sensor client.

begin

begin(int      id,
      uint32_t subscriptions        = 0,
      int      rate                 = 0,
      int      sample_watermark_num = 0,
      int      size_per_sample      = 0,
      sensor_data_mh_callback_t cb  =NULL):

Call this once only when creating a sensor client.

int id
This is the sensor client ID.
uint32_t subscriptions
Specify the sensor you want to subscribe to. By setting ID-th bit to 1, you can read the data of that ID.
int rate
Specify the sampling rate of sensor data to be input.
int sample_watermark_num
Specify the number of max samples of one process.
int size_per_sample
Specify the size of 1 sample.
sensor_data_mh_callback_t cb
Specify a callback to be called when sensor data is subscribed.

publish

int publish(FAR void* data,
            uint32_t  size_per_sample,
            uint32_t  freq,
            uint32_t  sample_watermark_num,
            uint32_t  timestamp);

This API publishes sensor data acquired/generated by sensor client.

FAR void * data
Specify the address of sensor data to send.
uint32_t size_per_sample
Specify the size of 1 sample.
uint32_t freq
Specify the sampling rate of sensor data to be input.
uint32_t sample_watermark_num
Specify the number of samples in this call.
uint32_t timestamp
Specify the timestamp of the first sample of sensor data to be sent.

int publish(PoolId    id,
            FAR void* data,
            uint32_t  size_per_sample,
            uint32_t  freq,
            uint32_t  sample_watermark_num,
            uint32_t  timestamp);

This API is needed if you define your own memory management area for reduce memory. Currently not used.

int publish(MemMgrLite::MemHandle& mh,
            uint32_t size_per_sample,
            uint32_t freq,
            uint32_t sample_watermark_num,
            uint32_t timestamp);

Calls management of sensor data in the sensor client when using MHandle of MemoryUtil. Currently not used.

int subscribe(sensor_command_data_mh_t& data);

By calling SensorClient subscribe, you can obtain the address where the sensing data is stored. Please use it on SubClass.

The subscribed data type is

sensor_command_data_mh_t

This is defined in Spresense SDK.

typedef struct
{
  sensor_command_header_t header;  /**< command header    */
  unsigned int self: 8;            /**< sender sensor ID  */
  unsigned int time: 24;           /**< time stamp        */
  unsigned int fs: 16;             /**< frequency         */
  unsigned int size: 16;           /**< number of samples */
  MemMgrLite::MemHandle  mh;       /**< mem handle for send data */

  unsigned int get_self(void)
  {
    return self;
  }

} sensor_command_data_mh_t;

Please get it as needed, such as timestamp information.

2.13.2.3. AccelSensorClass

The physical sensor clients for acceleration sensing are also provided as a library. It is made in the inheritance of SensorClient.

begin

bool begin(int id,
           int rate                 = 0,
           int sample_watermark_num = 0);

Begin an accelerometer client.

int id
This is specify the sensor ID. Specify SEN_accelID .
int rate
Specifies the sampling rate of the accelerometer. If use the StepCounter (AESM) provided, please set it to 50 Hz.
int sample_watermark_num
It is the number of transfer samples at one time. You can specify 50 or less. Specify 50 unless you need to make a change.

write_data

int write_data(float x,
               float y,
               float z);

It is a function that actually writes acceleration data. Convert the data acquired from the sensor to G and write it. This interface will be written by one sample. And publishes sensor data to clients, when sensor data is written sample_watermark_num samples.

2.13.2.4. ApplicationSensorClass

Application sensors are sensor clients for passing sensor data to applications. Therefore, only subscribe is implemented.

begin

bool begin(int id,
           uint32_t subscriptions,
           sensor_data_mh_callback_t cb);

int id
This is specify the sensor ID. Specify SEN_app * ID (*: 0 ~ 3) .
uint32_t subscriptions
Specify the sensor you want to subscribe to. Here, by setting the ID-th bit to 1, you can read the data of that ID.
sensor_data_mh_callback_t cb
Specify a callback to be called when sensor data is subscribed.

int subscribe(sensor_command_data_mh_t& data);

By subscribe, data of desired sensor (physical sensor / logical sensor) can be acquired. Please implement according to the application.

2.13.2.5. StepCountReaderClass

One of application sensors, it is an application sensor client prepared to read StepCounter data.

begin

bool begin(int      id,
           uint32_t subscriptions,
           sensor_data_mh_callback_t cb);

int id
This is specify the sensor ID. Specify SEN_app * ID (*: 0 ~ 3) .
uint32_t subscriptions
Specify the sensor you want to subscribe to. Here, since the logic sensor of StepCounter is read, please specify SUBSCRIPTION (SEN_stepcounterID) .
sensor_data_mh_callback_t cb
Specify a callback to be called when sensor data is subscribed.

int subscribe(sensor_command_data_mh_t& data);

By subscribe, data from Step Counter’s logic sensor can be acquired.

The data structure to be notified is as follows.

typedef struct
{
  SensorExecResult exec_result;

  union
    {
      StepCounterStepInfo steps;
      SensorAssertionInfo assert_info;
    };
} SensorResultStepCounter;

Actually StepCounterStepInfo steps; is an entity of sensor data.

It is shown below, the sensor data of StepCounter.

typedef struct {

  float    tempo;
  float    stride;
  float    speed;
  float    distance;
  uint32_t step;
  StepCounterMovementType  movement_type;
  uint64_t          time_stamp;

} StepCounterStepInfo;

2.13.2.6. AesmClass:

This is a sensor client for calling the step counter library, prepared by Sony. This algorithm recommended to use BMI160 that Bosch’s accelerate sensor, but it can use acceleration data if can be got a desired sampling rate from another accel sensor sensor.

begin

bool begin(int      id,
           uint32_t subscriptions,
           int      input_rate,
           int      input_sample_watermark_num,
           int      input_size_per_sample);

　　 * int id
This is specify the sensor ID. Specify SEN_stepcounterID .

uint32_t subscriptions
Specify the sensor you want to subscribe to. Here, since the physical sensor of AccelSensor is read, please specify SUBSCRIPTION (SEN_accelID) .
int rate
Specify the sampling rate of sensor data to be input. Since this algorithm is based on 50Hz, please set it to 50.
int sample_watermark_num
Specify the number of samples in one process. Since this algorithm performs processing in units of one second, set it to 50.
int size_per_sample
There are three float values for x, y and z, so specify 12.

set

int set(uint8_t walking_stride,
  　　    uint8_t running_stride)

It is a function to set the stride. Currently not available.

2.13.3. About the sample sketch

2.13.3.1. Step Counter sample

Here, for a Sensing sample, we provide the sample using StepCounter provided by SONY.

This sample requires to update to the Spresense SDK v1.3.0 or later bootloader.

The Step Counter function in Spresense SDK consists of the following.

Please refer to Sensor Fusion Framework for this.

Figure 17. Sensing Library (Publish/subscribe) Architecture

For Arduino, the part of Sensing library shown in the figure below is provided as a library.

Figure 18. Sensing Library (Publish/subscribe) Architecture

A sample sketch can be implemented by implementing the part that writes the acceleration sensor to the reading framework and the part that reads the generated StepCounter data.

The following is the sequence.

Figure 19. Initialization sequence

Figure 20. Execution sequence

In this way, StepCounter data can be obtained by implementing it.

For details on how to use, please refer to Tutorials.

2.14. Servo Library

The Servo library is one of the available libraries on Spresense board to control the RC Servo.

The Servo library is used the same way as the original Arduino servo library. The Arduino language reference have documented the Arduino Servo library and these instructions can be used for the Spresense Servo library too.

Spresense has 4 pins that can be used by the Servo library. These are the PWM pins on the extension board:

D06 PWM0
D05 PWM1
D09 PWM2
D03 PWM3

Most Servos are designed for a 5V power supply and draw considerable power, so you should power them from a separate supply, not the +5V pin on the Extension board.

The extension board interface voltage is set by the IOREF jumper to 3.3 or 5V. Most radio control servos can use 3.3 or 5V control signals.

2.14.1. Limitations

The Spresense Servo library supports up to 4 Servos.

2.15. Software Serial Library

The Spresense Arduino Library natively supports 2 serial ports: the header pins D0 D1 and the main board USB port. The SoftwareSerial library provides serial communication on any digital pins of the Spresense. The speed is up to 250,000 bps.

The SoftwareSerial library is used the same as the SoftwareSerial Library described in Arduino language reference and is compatible with the examples given on that page.

2.15.1. Limitations

The maximum speed is 250,000 bps.
It can use all the GPIO pins (D0 - D28). TX can use any of the GPIO pins, but there are some limitation in RX pins selection. At the same time, up to 6 pins can be chosen from: 0, 1, 10, 11, 12, 13, 16, 17, 18, 19, 20, 21, 23, 24, 25, 26, 27, 28 and up to 6 pins from: 2, 3, 4, 5, 6, 7, 8, 9, 14, 15, 22.

2.16. SPI Library

The SPI library has been developed to allow the use of SPI devices with Spresense.

The SPI library is used in the same way as SPI Library described in Arduino language reference.

Spresense has multiple SPI interfaces for main board and extension board respectively.

The pins allocated to SPI are shown below.

Table 5. SPI interface pins
SPI instance	Connector	MOSI	MISO	SCK	SS	IO voltage
SPI (or SPI4)	extension board	D11	D12	D13	D10	3.3V or 5V (Jumper switch JP1)
SPI5	main board	D16	D17	D23	D24	1.8V

When SPI.beginTransaction() is called, SPI4 on extension board is used. When you’d like to use SPI5 on the main board, you can use SPI5.beginTransaction() with instance name of SPI5.

2.16.1. Features and Limitations

This Library supports the transfer of 8 bit or 16 bit length.
SPI CS automatically goes low at the start of each transaction and high at the end of each transaction. This make coding easier and is useful for high speed data transfer. If this does not suit your application, or you want to use multiple SPI devices use a different pin for CS and control it using digitalWrite().
16 bit data is sent using Little Endian which the least significant value (bit 0-7) in the sequence is stored first.
See the Hardware Documents How to use the SPI for the maximum speed supported.
For an example of usage in the Arduino IDE see File →Examples → Example for SPRESENSE → SPI.

2.17. Storage Library

2.17.1. Overview

The Storage library defines a base class for various storage libraries as SDHCI, Flash and eMMC.

Basically, it is referred from SDHCI, Flash, eMMC libraries for each device. But you can also use the Storage library directly by specifying an absolute filename including the mount path.

The relationship between various storages and mount path is shown below for reference.

Device	Mount Path
SD card	/mnt/sd0
Flash	/mnt/spif
eMMC	/mnt/emmc

Device

Mount Path

SD card

/mnt/sd0

Flash

/mnt/spif

eMMC

/mnt/emmc

2.17.2. Functions

See API references Storage Library API for the details.

Function Description

Function	Description
Storage.open()	Opens a file with the absolute path on the storage device. When the file is opened for writing, it will be created if it doesn’t exist yet. FILE_READ: open the file for reading, starts at the beginning of the file. FILE_WRITE: open the file for reading and writing, starts at the end of the file. This function returns `File` object. See File library for file operations.
Storage.exists()	Tests whether a file or directory exists on the storage device.
Storage.mkdir()	Create a directory on the storage device.
Storage.rmdir()	Remove a directory from the storage device..
Storage.remove()	Remove a file from the storage device.

Storage.open()

Opens a file with the absolute path on the storage device. When the file is opened for writing, it will be created if it doesn’t exist yet.
FILE_READ: open the file for reading, starts at the beginning of the file.
FILE_WRITE: open the file for reading and writing, starts at the end of the file.
This function returns File object. See File library for file operations.

Storage.exists()

Tests whether a file or directory exists on the storage device.

Storage.mkdir()

Create a directory on the storage device.

Storage.rmdir()

Remove a directory from the storage device..

Storage.remove()

Remove a file from the storage device.

2.17.3. Examples

A sample sketch using Storage library are provided here:

Example	Description
read_write.ino	This is an example to read and write files on the storage device.

Example

Description

read_write.ino

This is an example to read and write files on the storage device.

2.18. Watchdog Library

The Spresense Watchdog library can trigger HW reset by using HW watchdog when system malfunctions from some use cases.

This library contains these functions:

Set the time until HW reset is executed
Acquire the time to reset
Reset time to reset

For example, you can use this library If you want to trigger HW reset, when the time that is enough for finishing all the processes expires in the loop() function.

2.18.1. How to use

Follow these steps to use this Library, also use Watchdog for WatchdogClass instance.

Initialize a watchdog by using Watchdog.begin()
Start to monitor by using Watchdog.start(time)

You need to set time (ms) ⇐ 40000 for timer to trigger HW reset.
Reset timer by using Watchdog.kick() notify the system to keep-alive.

If program is working fine, please execute kick() to notify the system to keep-alive.
When it is not necessary to monitor by Watchdog, please execute Watchdog.stop()

If you do not call Watchdog.stop() or Watchdog.kick() , watchdog timer will expire and will trigger HW reset.

If you want to check the time remaining for trigger HW reset, you can use Watchdog.timeleft().

Watchdog timer will be reset when clock mode change by LowPower Library or insert a SD card.

2.18.2. Sample

You can check Watchdog example sketch at Arduino IDE File > Examples > Watchdog > Watchdog_bite.

2.19. Wire Library

The Wire library allows Spresense to communicate with I2C / TWI devices.

The Wire library is used the same way as the Arduino Wire Library described in the Arduino language reference and is compatible with the examples given on that page.

The pinout I2C for Spresense is:-

D15 I2C_SCL Clock line
D14 I2C_SDA Data line

Caution: The extension board has connectors in the same place as Arduino Uno, but the pins number allocated for I2C are different. Existing Arduino sketches may require modification.

The extension board interface voltage is set by the IOREF jumper to 3.3 or 5V. This sets the interface voltage for I2C. The D14, D15 pins also appear on the main board. This can be used when the extension board is not connected. The interface voltage is then 1.8V along with all other GPIO on the main board.

I2C communication requires the use of pull-up resistors.

The Spresense extension board includes 1k Ohms pull-up resistors so no additional components are required.
The Spresense main board includes 4.7k Ohms pull-up resistors so no additional components are required.

2.19.1. Features and Limitations

The port supports communication at 3 speeds which are defined in the library.

#define TWI_FREQ_100KHZ (100000) // standard mode
#define TWI_FREQ_400KHZ (400000) // fast mode
#define TWI_FREQ_1MHZ (1000000) // fast mode plus

The default is standard mode, 100kHz

The library supports two I2C address lengths.

#define TWI_ADDR_LEN_7_BIT (7)
#define TWI_ADDR_LEN_10_BIT (10)

The default is 7 bit.

Like the original Arduino Wire library this library implementation uses a 32 byte buffer, therefore any communication should be within this limit. Exceeding bytes in a single transmission will just be dropped.