Spresense Header CodeSpresense Header Code

Edge computing with low power consumption

The Spresense development board enables edge solutions with high computing ability and low power consumption.

Spresense SDK Developer Guide

This is old documents. Please move to new documents. https://developer.sony.com/develop/spresense/docs/sdk_developer_guide_en.html

This section explains the contents necessary for creating an application using Spresense SDK.

1. Spresense SDK Overview

This Spresense SDK is the software development kit for the CXD5602 provided by Sony Semiconductor Solutions Corp.

The image below is a block diagram of the CXD5602.

CXD5602 blockdiagram
Figure 1. CXD5602 Internal Block Diagram.

The CXD5602 comprises of four major blocks called domains.

  • Application Domain

    • This block is controlled by the user application.

    • It holds the six CPUs (Cortex®-M4F designed by ARM Co. Ltd.) where the user programs are executed.

  • System IOP Domain

    • This block handles the system startup and manages the power domains inside the CXD5602.

    • One ARM Cortex-M0 processor as controller for the domain.

  • GNSS Domain

    • This blocks handles the satellite positioning functionality.

    • One ARM Cortex-M4F is used as controller for the domain.

  • Sensor Domain

    • This block autonomously performs sensor data acquisition that is connected to the I2C or SPI bus, giving the chip its unique low power continuous sensing capability.

For further information about the CXD5602, please refer to the Sony Semiconductor Solutions Co. Spresense webpage.

Spresense SDK provides software drivers to control the hardware blocks. Spresense SDK uses NuttX as real time operating system which provides functions to utilize the unique features of the CXD5602.

The drivers in Spresense SDK are implemented using the NuttX framework which is similar to the Linux framework.

Spresense’s middleware layer provides an abstraction of Spresense specific functions like Audio, GNSS and ASMP on top of the driver layer.

sdk overview
Figure 2. SDK Overview.

The table below presents the middleware components and drivers provided by the Spresense SDK.

Table 1. Spresense SDK Middleware Overview
Module Name Summary

Audio Middleware

Provides the audio functions. Record and playback in various formats. Supports various data paths. Performs audio processing.

Sensor Framework

Provides sensor data exchange functionality using a Publish-Subscribe architecture

Memory Utility

Provides a fixed-size memory pool function with a reference counter and task synchronization mechanism.

ASMP Framework

Manages 12 tiles of memory, a core feature of CXD5602 memory structure, to distribute the processing load of user programs across the different processor cores.

Power Management

Functions for power save.

GNSS Driver

CXD5602 has a HW subsystem that performs GNSS positioning. This driver interacts with that subsystem and provides the user with functions related to GNSS as a character device.

Read the chapter Spresense SDK functionality for details on each module.

2. License

As with NuttX, Spresense SDK has been released as open source under the 3-Clause BSD license. Detailed license terms are as follows.

3. System boot sequence

Booting of the CXD5602 starts when reset is inactivated. The loader.espk will be loaded into RAM and executed and during this time, control of the CPU is transferred to loader.espk.

Loader.espk load nuttx.spk, this will start the application CPU.

Since nuttx.spk is a binary built with this SDK, the executed contents of nuttx.spk can vary depending on of the settings and built-in applications.

When GNSS is being used in an application, gnss.espk is loaded by loader.espk when the GNSS API is initialized.

Therefore, loader.espk and gnss.espk are essential binaries when using the Spresense board.

For information on how to obtain the binaries, please refer to here.

4. Software configuration management

This chapter gives an overview of the source code of Spresense SDK.

4.1. Repository

Spresense SDK using 2 repositories.

Name Submodule Description

Include BSP, Spresense supported driver, example application source codes.

spresense-nuttx

Clone repository from NuttX. And this is the kernel of Spresense.

4.2. Source Tree

This is Spresense SDK directory structure.

spresense directory structure
Figure 3. Directory structure
Name Description

spresense

Root directory of spresense git

nuttx

Spresense NuttX Kernel

examples

Spresense SDK Example application source codes

sdk

Spresense SDK supported driver, modules

bsp

Spresense SDK BSP

configs

Configuration files for Spresense SDK

drivers

Spresense SDK supported drivers

modules

Spresense SDK supported audio, sensor, etc modules

system

Spresense SDK system tools

tools

Spresense SDK build and configuration tools

5. Development environment

5.1. Prerequisite

You can develop Spresense SDK on Linux/Windows/macOS. Please setup development environment for your appropriate OS.

Supported Operation System

  • Linux (64bit)

    • Ubuntu 16.04 or later

  • Windows

    • 8.1/10

  • macOS

    • High Sierra (10.13) or later

5.1.1. Setup for Linux

  1. Serial Configuration

    Add user to dialout group

    $ sudo usermod -a -G dialout <user-name>

    Please logout and login after running the above command.

  2. Install development tools

    $ wget https://raw.githubusercontent.com/sonydevworld/spresense/master/install-tools.sh
    $ bash install-tools.sh

    And run this command to activate installed tools.

    $ source ~/spresenseenv/setup

    This command must run in every terminal window. If you want to skip this step, please add this command in your ${HOME}/.bashrc.

  3. Clone the Spresense SDK with submodule

    $ git clone --recursive https://github.com/sonydevworld/spresense.git

5.1.2. Setup for Windows

  1. Install MSYS2

  2. Install development tools

    Run MSYS2 MSYS from start menu, and run following commands.

    MSYS2 MinGW 64-bit and MSYS2 MinGW 32-bit is not supported. Please use MSYS2 MSYS.
    $ curl -L https://raw.githubusercontent.com/sonydevworld/spresense/master/install-tools.sh > install-tools.sh
    $ bash install-tools.sh

    And run this command to activate installed tools.

    $ source ~/spresenseenv/setup

    This command must run in every terminal window. If you want to skip this step, please add this command in your ${HOME}/.bashrc.

  3. Clone the Spresense SDK with submodule

    $ git clone --recursive https://github.com/sonydevworld/spresense.git
  4. Install USB serial driver

    User must install before connecting Spresense board to the PC.

    Please download and install from below.

5.1.3. Setup for macOS

  1. Install development tools provided by Apple

    Open the Teminal and run following command.

    $ xcode-select --install
  2. Install Python3

  3. Install development tools

    $ curl -L https://raw.githubusercontent.com/sonydevworld/spresense/master/install-tools.sh > install-tools.sh
    $ bash install-tools.sh

    And run this command to activate installed tools.

    $ source ~/spresenseenv/setup

    This command must run in every terminal window. If you want to skip this step, please add this command in your ${HOME}/.bash_profile.

  4. Clone the Spresense SDK with submodule

    $ git clone --recursive https://github.com/sonydevworld/spresense.git
  5. Install USB serial driver

    User must install before connecting Spresense board to the PC.

    Please download and install from below.

    On High Sierra (10.13) and later, the installation of the driver may be blocked. Please allow this driver from System Preferences Security & Privacy pane. See Apple Technical Note TN2459 "User-Approved Kernel Extension Loading" for more information.

5.2. Flashing bootloader

Correct bootloader is required for using Spresense board properly.

  • The bootloader has to be flashed after purchase.

  • It is needed to accept the End User License Agreement before using the Spresense bootloader binary.

A WARNING might be shown when using tools/config.py or tools/flash.sh with bootloader missing. In this case, please flash the Spresense bootloader into the Spresense board.

WARNING: New loader v1.0.003 is required, please download and install.
         Download URL   : https://developer.sony.com/file/download/eula-binaries-vx.x.x.zip
         Install command:
                          1. Extract loader archive into host PC.
                             ./tools/flash.sh -e <download zip file>
                          2. Flash loader into Board.
                             ./tools/flash.sh -l /home/user/mySpresense/spresense/firmware/spresense -c <port>
  1. Download the EULA binary zip archive.

    Go to Download URL in WARNING description, and download it into your PC.

  2. Extract the EULA binary zip on your PC by executing:

    $ ./tools/flash.sh -e eula-binaries-vx.x.x.zip
  3. Flash Spresense bootloader into Spresense board. Make sure the Spresense board is attached to the PC via the USB cable.

    Execute command that shown in WARNING description.

    $ ./tools/flash.sh -l /home/user/mySpresense/spresense/firmware/spresense -c <port>
  4. Finish.

    After finish, the Spresense board will reboot.

6. Build method

This chapter describes the structure and build procedure of the Spresense SDK. Spresense SDK uses a build system using GNU Make.

It might be handy to have a local clone of the Spresense SDK while reading this section. The Spresense SDK is be cloned from GitHub.

Cloning Spresense SDK via https:
$ git clone --recursive https://github.com/sonydevworld/spresense.git

6.1. Configuration

The NuttX kernel and Spresense SDK are configurable to be able to comply with the requirements for specific applications and hardware configurations.. The Spresense SDK is using Kconfig to handle the configuration, just as the Linux kernel build system does. Please refer to Development environment to get the Kconfig installation instructions.

The configuration is divided into two parts:
  • NuttX kernel configuration

  • SDK configuration

The Spresense SDK provides a set of tools: tools/config.py and tools/mkdefconfig.py to handle the configuration of these two systems.

Usage of tools/config.py
$ cd spresense/sdk
spresense/sdk$ tools/config.py -h
usage: config.py [-h] [-k] [-l] [-m] [-q] [-g] [-d DIR] [-v]
                 [<config name> [<config name> ...]]

Configuration tool

positional arguments:
  <config name>      configuration name

optional arguments:
  -h, --help         show this help message and exit
  -k, --kernel       kernel config
  -l, --list         list default configurations. show kernel defconfigs with
                     --kernel.
  -m, --menuconfig   run config in "menuconfig"
  -q, --qconfig      run config in "qconfig"
  -g, --gconfig      run config in "gconfig"
  -d DIR, --dir DIR  change configs directory
  -v, --verbose      verbose messages
Usage of tools/mkdefconfig.py
spresense/sdk$ tools/mkdefconfig.py -h
usage: mkdefconfig.py [-h] [-k] [-d DIR] [--all] [--verbose] <config name>

Make default config from current config

positional arguments:
  <config name>      configuration name

optional arguments:
  -h, --help         show this help message and exit
  -k                 save kernel configuration
  -d DIR, --dir DIR  change configs directory
  --all              Save SDK and kernel configuration with same name
  --verbose, -v      verbose messages

In the following example tools/config.py tool will load the kconfig-frontends with menuconfig as user interface and load a predefined configuration (described below).

This tool is used for both for the NuttX kernel and Spresense SDK configuration. Selecting what target to configure is done by extra option switches on the command line when executing the tools/config.py script. The -k or --kernel option will affect the NuttX kernel configuration. When the tools/config.py script is called without any switch the Spresense SDK configuration is affected.

e.g.
$ cd spresense/sdk
$ tools/config.py -k -m
The tools/config.py and tools/mkdefconfig.py scripts depend on the current directory structure so make sure that they run from the sdk directory.

6.1.1. menuconfig, qconfig, gconfig

Kconfig have a few different user interfaces. To open them to modify the configuration execute tools/config.py with one of the options in the table below.

Table 2. Setup screen options:

Options

-m or - menuconfig

CUI menu selection using the ncurses library

-q or --qconfig

GUI menu selection using the Qt library

-g or --gconfig

GUI menu selection using the gtk library

Depending on what user interface options that where activated when kconfig-frontend was built some of the options might not be available.

6.1.2. Predefined Configuration (defconfig)

The predefined configuration (defconfig) is a predefined combination of settings required when you want to use a certain function. If you want to use a defconfig, specify the defconfig name as the argument of the tools/config.py tool. It is possible to list the available defconfigs by executing the tools/config.py with the -l option.

List all defconfigs example:
$ tools/config.py -l

Multiple defconfigs can be specified at the same time. If more than one defconfig is specified, the result is a combination of all the defconfigs.

Example of using more than one defconfig:
$ tools/config.py board/spresense device/sdcard
Specifying a defconfig when calling tools/config.py will reset/overwrite the current configuration.

6.1.3. Save settings

It is possible to save custom settings made by the user.

To save the current SDK settings as <config name> use the following command:
$ tools/mkdefconfig.py <config name>

In the example above on the Spresense SDK settings will be saved, but not the NuttX kernel settings. To save the NuttX kernel settings the -k option has to be used. Use the -k option or the --all option. If you specify the --all option, the current settings of both the SDK and the kernel are saved with the specified name.

$ tools/mkdefconfig.py -k <config name>
or
$ tools/mkdefconfig.py --all <config name>
If the name specified in <config name> already exists, a confirmation will be displayed asking if you want to overwrite it.

tools/config.py and tools/mkdefconfig.py apply/save to defconfig files in the folder sdk/configs. The defconfig destination folder can be set with the -d or --dir option.

Example of using ../path/to/configs directory to save the <config name> defconfig to:
$ tools/mkdefconfig.py -d ../path/to/configs <config name>
To use a defconfig in a different path, in this example ../path/to/configs use:
$ tools/config.py -d ../path/to/configs <config name>

Changing the defconfig directory is also supported when using the -k option.

6.2. Build system

The build system is divided into two parts:
  • NuttX kernel

  • Spresense SDK

Each of these two need to be built separately. Starting the build must be done from the sdk directory in both cases.

diag 07e00245f2dae3a9fcfd8f87b42b0208
Figure 4. Build workflow

6.2.1. Building the NuttX kernel

To build the NuttX kernel, type the commands in the following example. Before the build is started, the NuttX kernel have to be configured.

Configure the NuttX kernel via menuconfig:
$ cd spresense/sdk
$ tools/config.py -k release
$ tools/config.py -k -m
# Above commands are to configure the NuttX kernel settings.
$ make buildkernel
Configure the NuttX kernel with default settings:
$ cd spresense/sdk
$ make buildkernel KERNCONF=release
If the settings of the NuttX kernel haven’t been changed or if the source have not been modified, then the NuttX kernel doesn’t have to be rebuilt.

6.2.2. Building the SDK

To build the SDK, type in the commands in the following example. Just as in the NuttX kernel case, the configuration have to be set prior starting the build.

$ tools/config.py default
$ tools/config.py -m
# Above commands are to configure the SDK settings.
$ make

If the build succeeds two files will be created in the sdk directory: nuttx and nuttx.spk.

These two files have the following format:

nuttx

ELF format. To be used when loaded with a debugger.

nuttx.spk

A proprietary file packaged to write to the actual machine.

7. Flashing binaries to the Spresense board

This chapter explains how to flash the following binaries:
  • nuttx.spk

  • loader.espk

  • gnss.espk

7.1. Flash the bootloader

The correct bootloader is required for the Spresense board to function.

Bootloader information
  • The bootloader has to be flashed the very first time the board is used.

  • You have to accept the End User License Agreement to be able to download and use the Spresense bootloader binary.

A WARNING message (example shown below) may be shown when using tools/config.py or tools/flash.sh if the bootloader is missing. In this case you have to flash the Spresense bootloader onto the Spresense board.

WARNING: New loader v1.0.003 is required, please download and install.
         Download URL   : https://developer.sony.com/file/download/spresense-binaries-vx.x.x.zip
         Install command:
                          1. Extract loader archive into host PC.
                             ./tools/flash.sh -e <download zip file>
                          2. Flash loader into Board.
                             ./tools/flash.sh -l /home/user/mySpresense/spresense/firmware/spresense -c <port>
  1. Download the EULA binary zip archive.

    Go to the Download URL in WARNING description which will provide a link to the specific version you need, and download it to your PC.

  2. Extract EULA binary zip in your PC by executing:

    $ ./tools/flash.sh -e spresense-binaries-vx.x.x.zip
  3. Flash Spresense bootloader into Spresense board.

    Execute the command that is shown in WARNING description. (In this example <port> is /dev/ttyUSB0):

    $ ./tools/flash.sh -l ../firmware/spresense -c /dev/ttyUSB0
  4. Finish.

    On installation completion, the Spresense board will reboot.

7.2. Flash the user nuttx.spk image

Use the tools/flash.sh script to flash nuttx.spk to Spresense board.

  1. Connect the USB cable from the Spresense main board to your PC.

  2. Check serial port number by using next command.

    $ dmesg | grep "cp21.*attached"
    [12220.625979] usb 1-1: cp210x converter now attached to ttyUSB0

    In this case, Spresense board use /dev/ttyUSB0 for serial connection.

  3. Type tools/flash.sh for flashing nuttx.spk into Spresense board.

    $ tools/flash.sh -c /dev/ttyUSB0 nuttx.spk
    tutorial nuttx image load
    Figure 5. nuttx.spk flashing
  4. On installation completion, the Spresense board will reboot.

8. Examples

This chapter will explain how to build and execute a simple "hello world" example in using the Spresense SDK.

8.1. Provided examples

The examples in the Spresense SDK are installed as a built-in command in the NuttShell environment. Refer to the README.txt file in the directory of each example for additional details about the required SDK configuration etc.

Example name Description

accel

An example of how to read the accelerometer sensor data

adc

An example of how to read the A/D conversion data

alarm

An example of how to set an RTC alarm

asmp

An example of how to run worker tasks on many cores

audio_player

An example of how to play audio samples

audio_recorder

An example of how to record audio to file

audio_through

An example of how to set the audio-paths from microphone to speaker

camera

An example of how to stream data from the camera to file and to an LCD

colorsensor

An example of how to read the color sensor data

decimator

An example of how to use the decimator

fwupdate

An example of reading and updating the firmware

geofence

An example of how to set up and use a geofence

gnss

An example of how to read GNSS sensor data

gnss_atcmd

A utility application that converts and executes GNSS AT commands as IOCTL commands

gnss_factory

A utility application running the GNSS factory test

gnss_pvtlog

An example of how to use the GNSS PVTLOG

gyro

An example of how to read gyro sensor data

hello

A "hello world" example application in C

helloxx

A "hello world" example application in C++

light

An example of how to read light sensor data

mag

An example of how to read magnetic sensor data

press

An example of how to read barometric pressure sensor data

proximity

An example of how to read proximity sensor data

sixaxis

An example of how to read the gyro and accelerometer sensor data

tilt

An example of how to detect tilting

watchdog

An example of how to configure the watchdog

voice_call

An example of how to set up set up the audio sub-system for voice call

voice_command

An example on how to set up the audio sub-system for voice recognition

8.2. "hello" Example, building and running

8.2.1. Building the "hello" Example

Configuration method

To begin the NuttX kernel and SDK have to be configured:

$ tools/config.py --kernel release
$ tools/config.py board/spresense

The next step is to update the SDK configuration:

$ tools/config.py -m

This command will open the menu configuration view.

Make sure that ["Examples" → "Hello, World! example"] is selected, then save.

spresense hello example configuration1
Figure 6. SDK configuration example.
spresense hello example configuration2
Figure 7. Hello, World! example is selected.
Build method

To start the build the NuttX kernel has to be built first, then the SDK:

Starting the build process:
$ make buildkernel
$ make
Load method

Type tools/flash.sh for flashing nuttx.spk into Spresense board.

$ tools/flash.sh -c /dev/ttyUSB0 nuttx.spk

8.2.2. How to run "hello" Example

Type hello in a serial terminal that is attached to the Spresense board. The example will start after the enter key has been pressed.

spresense hello example execute
Figure 8. Hello sample has executed.

9. Adding a user application

A new user application can be added using the following methods:

  • Using the application template in the examples directory, or

  • Adding the user application to any other directory.

9.1. Add to the examples

The most straightforward way to quickly get started with developing a custom made user application is to use the hello example.

  1. Copy the whole hello folder to a directory with a new name.

  2. Replace all hello strings in Makefile,Make.defs, Kconfig,hello_main.c with the new name chosen in step 1.

  3. Change the file name of hello_main.c to an arbitrary file name. (See Example of adding a user application from scratch).

  4. After replacing all the strings, execute make clean once in the SDK directory.

  5. Enter menuconfig, your new custom made example application should be listed under examples.

Enter menuconfig
$ cd sdk
$ make clean
$ tools/config.py -m
Kconfig menu
Examples --->
  [] My first app example (NEW)
If the Kconfig file have been added or deleted, the config menu menuconfig, can be updated by running make clean in the SDK directory.

Once the new application has been enabled, the SDK is ready to be built by the command:

$ make

After flashing the created nuttx.spk file and restarting the Spresense board, the added application can be executed as a command.

9.1.1. Example of adding a user application from scratch

Kconfig
config EXAMPLES_MYFIRSTAPP
bool "My first app example"
default n
---help---
    Enable the my first app example

if EXAMPLES_MYFIRSTAPP

config EXAMPLES_MYFIRSTAPP_PROGNAME
string "Program name"
default "myfirstapp"
depends on BUILD_KERNEL
---help---
    This is the name of the program that will be used when the NSH ELF
    program is installed.

config EXAMPLES_MYFIRSTAPP_PRIORITY
int "My firstapp task priority"
default 100

config EXAMPLES_MYFIRSTAPP_STACKSIZE
int "My first app stack size"
default 2048

endif

Symbols defined with config <symbol> in the Kconfig file can be referred to in the Makefile with the variable CONFIG_ <symbol>.

Option symbols defined in the Kconfig file must be unique in the SDK. If symbols are duplicated, a warning will be displayed when running the configuration tool.
Makefile
-include $ (TOPDIR) / Make.defs
-include $ (SDKDIR) /Make.defs

# My first app built-in application info

CONFIG_EXAMPLES_MYFIRSTAPP_PRIORITY? = SCHED_PRIORITY_DEFAULT
CONFIG_EXAMPLES_MYFIRSTAPP_STACKSIZE? = 2048

APPNAME = myfirstapp
PRIORITY = $ (CONFIG_EXAMPLES_MYFIRSTAPP_PRIORITY)
STACKSIZE = $ (CONFIG_EXAMPLES_MYFIRSTAPP_STACKSIZE)

# My first app Example

ASRCS =
CSRCS =
MAINSRC = myfirstapp_main.c

CONFIG_EXAMPLES_MYFIRSTAPP_PROGNAME? = Myfirstapp $ (EXEEXT)
PROGNAME = $ (CONFIG_EXAMPLES_MYFIRSTAPP_PROGNAME)

include $ (APPDIR) /Application.mk
APPNAME

The character string set in this variable becomes the command name

PRIORITY

Sets the task priority

STACKSIZE

Sets the task’s stack size

Make.defs
ifeq ($ (CONFIG_EXAMPLES_MYFIRSTAPP), y)
CONFIGURED_APPS + = myfirstapp
endif
CONFIGURED_APPS

The directory set in this variable is subject to build.

9.2. Add to a different directory

It is possible to store the application outside the Spresense SDK repository. The following steps will explain how to add a directory outside of the repository into the build system.

  1. Create arbitrary directories in the same directory as the SDK.

  2. Copy LibTarget.mk,Makefile, Make.defs and Application.mk from the examples directory.

  3. Change the path of the archive in LibTarget.mk.

  4. Change the path of the archive in Application.mk.

  5. Change the file name of the BIN variable in Makefile to the same file name as the file name changed with LibTarget.mk.

  6. Add any application in the newly created directory in the same way as described in Add to the examples.

  7. Execute make in the SDK directory.

9.2.1. Example of adding another directory

In this example, the application myfirstapp is placed in a folder called myapps.

File/directory structure after creating the example.
|-- sdk
|-- examples
└-- myapps
    |-- Application.mk
    |-- LibTarget.mk
    |-- Make.defs
    |-- Makefile
    └-- myfirstapp
        |-- Kconfig
        |-- Make.defs
        |-- Makefile
        └-- myfirstapp_main.c
myapps/LibTarget.mk
# Change directory from examples to myapps
# Change the archive name from libexamples to libmyapps

$(SDKDIR)$(DELIM)..$(DELIM)myapps$(DELIM)libmyapps$(LIBEXT): context
	$(Q) $(MAKE) -C $(dir $@) TOPDIR="$(TOPDIR)" SDKDIR="$(SDKDIR)" $(notdir $@)

lib$(DELIM)libmyapps$(LIBEXT): $(SDKDIR)$(DELIM)..$(DELIM)myapps$(DELIM)libmyapps$(LIBEXT)
	$(Q) install $< $@

EXTLIBS += lib$(DELIM)libmyapps$(LIBEXT)
DELIM

Path delimiter (\ for Windows, / for Linux).

LIBEXT

Library file extension (.a).

myapps/Makefile
MENUDESC = "My Apps"

BIN = libmyapps$(LIBEXT)  # libexamples -> libmyapps

include $(SDKDIR)/Makefile.ext
MENUDESC

String displayed in the configuration menu

myapps/Application.mk (snippet)
ifeq ($(WINTOOL),y)
  BIN = "${shell cygpath -w $(APPDIR)$(DELIM)libmyapps$(LIBEXT)}" # libexamples -> libmyapps
  INSTALL_DIR = "${shell cygpath -w $(BIN_DIR)}"
else
  BIN = $(APPDIR)$(DELIM)libmyapps$(LIBEXT) # libexamples -> libmyapps
  INSTALL_DIR = $(BIN_DIR)
endif

ROOTDEPPATH = --dep-path .

VPATH =

all: .built
.PHONY: clean preconfig depend distclean
.PRECIOUS: $(APPDIR)/libmyapps$(LIBEXT) # libexamples -> libmyapps

10. Debugging

This section will describe which tools and how they can be used in order to do effective debugging of a user application.

10.1. Debugging with Eclipse IDE

10.1.1. Hardware requirement

Mount SWD connector

Please see Hardware Document.

10.1.2. Install OpenOCD

Prebuilt version of OpenOCD has been installed after Development environment, it installed at ${HOME}/spresenseenv.

To use it for Spresense, please copy configuration file sdk/tools/cxd5602.cfg into OpenOCD search directory.

$ cp spresense/sdk/tools/cxd5602.cfg ~/spresenseenv/usr/scripts/target/

On Linux, user must do following instructions to be able to use ICE.

$ sudo cp ~/spresenseenv/usr/contrib/60-openocd.rules /etc/udev/rules.d/
$ sudo udevadm control --reload

And connect your debug adapter.

10.1.3. Install Eclipse

Install Java Runtime Environment (JRE), it is required for Eclipse.

  • On Linux, you can install from apt.

    $ sudo apt install default-jre
  • On Windows/macOS, please download JRE or JDK from Oracle.

Download the latest Eclipse installer from official site (https://www.eclipse.org/downloads/). Please select Eclipse IDE for C/C++ Developers on the installer.

Install CDT
Figure 9. Install Eclipse IDE for C/C++ Developers

After Eclipse installed, please install GNU MCU Eclipse plugin.

  1. Click HelpEclipse Marketplace…​

  2. Input gnu mcu to search box, and press Go

  3. Install GNU MCU Eclipse

Install GNU MCU plugin

10.1.4. Setup the Spresense board

Spresense board will boot automatically after installing the nuttx.spk file, which is generated at the sdk directory after built. This file used for installing into the main board. The nuttx file is also generated at the sdk directory, which is an ELF executable file and used for debugging. However, the nuttx.spk file is stored as nuttx in the local file system on the main board (not an ELF). Before debugging with Eclipse, the nuttx file needs to be removed and the board needs to be in debug mode.

  1. Focus on the console (e.g. minicom).

  2. Press and hold 'r' key to reset the main board.

  3. Enter the updater# prompt on the serial terminal.

  4. Enter the following command to remove the nuttx file:

    updater# rm nuttx
  5. Now the board is in debug mode. The following message should appear in the terminal:

    Waiting for debugger connection..

To return to normal mode and restore the installed program:

$ ./tools/flash.sh nuttx.spk

10.1.5. Debugging (GNU MCU Eclipse plugin)

  1. Click RunDebug Configurations…​.

  2. Select GDB OpenOCD Debugging.

  3. Click New to create a new debug configuration.

  4. Follow the instructions below to setup the debug configuration.

  5. Press the reset switch on the Spresense main board

  6. Press the Debug button to start debugging.

Debug Configuration

Make the following changes in the configuration panes:

  • Main pane

    • C/C++ Application → nuttx

eclipse debug gnu mcu main
Figure 10. Main pane in debug configuration
  • Debugger pane

    • OpenOCD Setup

      • Executable path/path/to/openocd

      • Config options → copy the following command:

        -f interface/cmsis-dap.cfg -f target/cxd5602.cfg
    • GDB Client setup

      • Start GDB Session → Enable

      • Executable name/path/to/arm-none-eabi-gdb

eclipse debug gnu mcu debugger
Figure 11. Debugger pane in debug configuration
  • Startup pane

    • Load Symbols and Executable

      • Load symbols → Enable

      • Load executable → Enable

    • Runtime Options

      • Debug in RAM → Enable

eclipse debug gnu mcu startup
Figure 12. Startup pane in debug configuration

11. Spresense SDK functionality

Detailed description of the functionality provided by Spresense SDK.

11.1. BSP

11.1.1. Overview

BSP (Board Support Package) contains the source code for board specific settings and processing.

NuttX has the following driver software architecture, and the BSP directory contains the software such as Driver (Lower Half), Board specific code and Architecture specific code.

diag 7b3a8011eb336e2a51fcad7d4f0a6496
Figure 13. Device driver hierarchy diagram
Driver (Lower Half)

NuttX has some drivers called Upper Half for standard devices and buses. The Upper Half driver provides the interfaces to the application, protocol processing, etc., the features of the Upper Half are independent of the board but it can not be used by itself. The Lower Half driver has to be implemented specifically for the board.

Spresense SDK provides Lower Half Driver for:

  • I2C (cxd56_i2c.c)

  • PWM (cxd56_pwm.c)

  • RTC (cxd56_rtc_lowerhalf.c)

  • SDIO Device (cxd56_sdhci.c)

  • Serial Device (cxd56_serial.c)

  • SPI (cxd56_spi.c)

  • Timer (cxd56_timer.c)

  • USB Device (cxd56_usbdev.c)

  • Watchdog timer (cxd56_wdt.c)

Please refer to NuttX Device Drivers for the details of NuttX’s driver.

Architecture specific code

Architecture specific drivers include drivers (such as power management) which are designed to use the specific features on the chip.

Board specific code

Board specific code is implemented depending on each board like as pin setting. These can be divided into those that can be the common and those that are completely board dependent. These are respectively located in bsp/board/common and bsp/board/<board name> (<board name> is the name of the corresponding board) .

Board Initialization Process is a part of the board specific code.

11.1.2. Directory Structure

sdk/bsp
|-- board
|   |-- common
|   `-- spresense
|-- include
|   |-- arch
|   |   `-- chip
|   |-- nuttx
|   |   |-- lcd
|   |   |-- sensors
|   |   `-- video
|   `-- sdk
|-- scripts
`-- src
Directory Name Description

bsp/board/common

There are the common code regardless of board.

bsp/board/spresense

The Spresense board specific code like as the pin setting for Spresense board.

bsp/include/arch/chip

The header files for calling the API provided by the chip specific device driver.

bsp/include/arch/nuttx

The header files of the architecture independent device. (This directory name is based on compatibility with the NuttX kernel.)

bsp/include/arch/sdk

The common header files in the Spresense SDK. A SDK configuration file (config.h) is generated into here at the build time.

bsp/scripts

The makefile components and the linker scripts.

bsp/src

The Lower Half drivers and the chip specific device drivers.

11.1.3. Board Initialization Process

diag ea8d94e5046d464d9177095a214ddea1
Figure 14. Startup Sequence

When the main CPU starts up, the NuttX kernel is initialized and the entry point function of SDK is called. By default, nsh (NuttShell) is invoked.

In nsh, the board initialization (boardctl()) is called. In response to this command, boardctl() will call the board-specific implementation of board_app_initialize().

Initialization the board includes the initialization of the features and the device drivers.

If you change CONFIG_SDK_USER_ENTRYPOINT from nsh_main, you need to call boardctl() from your application.

11.2. GPIO/Pin Specification

11.2.1. Software Structure

diag 2ccf26c7a02c2ea94fe313d124e620e3

11.2.2. GPIO Driver Interface

gpioif provides the features below for use in your application.

  • GPIO pin setting

    • Function Mode

    • Input/Output enable

    • Drive Current/Slew rate

    • Pull Up/Down

  • GPIO interrupt setting

    • Level/Edge Trigger

    • Noise Filter

  • GPIO Read/Write

  • GPIO status monitor

See here for the details of API.

GPIO Utility tool

In system tools, GPIO Command utility is provided. If CONFIG_SYSTEM_GPIO=y、gpio command is available from NuttShell.

Usage of gpio command

nsh> gpio
USAGE: gpio command
 stat [<from_pin>] [<end_pin>]
 conf <pin> [-m <0|1|2|3>] [-i] [-H] [-p <0|1|2|3>]
  -m: function mode
  -i: input enable
  -H: Higher drive current/slew rate
  -p: 0=float, 1=pullup, 2=pulldown, 3=buskeeper
 read <pin>
 write <pin> <0|1|-1>

If CONFIG_SYSTEM_GPIO_STATUS=y, it’s possible to display gpio status by gpio stat command.

nsh> gpio stat
-------------------------------------------------------------
( No)PIN NAME          : Mode I/O mA Pull Read IRQ Type NF EN
-------------------------------------------------------------
(  1)PIN_I2C4_BCK      : 1    I/  2  --   1    -1
(  2)PIN_I2C4_BDT      : 1    I/  2  --   1    -1
(  3)PIN_PMIC_INT      : 1    I/  2  --   0    -1
(  4)PIN_RTC_IRQ_OUT   : 0     /  2  --   0    -1
(  5)PIN_AP_CLK        : 0     /  2  --   0    -1
(  6)PIN_GNSS_1PPS_OUT : 0     /  2  --   0    -1
( 17)PIN_SPI0_CS_X     : 1     /  2  --   0    -1
( 18)PIN_SPI0_SCK      : 1    I/  2  --   1    -1
( 19)PIN_SPI0_MOSI     : 0     /  2  --   0    -1
( 20)PIN_SPI0_MISO     : 0     /  2  --   0    -1
  :
(101)PIN_MCLK          : 0     /  2  --   0    -1
(102)PIN_PDM_CLK       : 0     /  2  --   0    -1
(103)PIN_PDM_IN        : 0     /  2  --   0    -1
(104)PIN_PDM_OUT       : 0     /  2  --   0    -1
(105)PIN_USB_VBUSINT   : 1    I/  2  --   1    -1

11.2.3. Pin specification

Pins are organized into groups. Each group of pins has a set of functions, that vary depending on the mode you set. The mode can range from 0 to 3.

For example, for the PWM0,1 pins:

  • If you select Mode0, the PWM0 and PWM1 pins both function as GPIO pins.

  • If you select Mode1, the PWM0 and PWM1 pins both function as PWM pins.

You cannot change the function of one pin in a group without changing the others. For example, you cannot make the PWM0 pin function as a GPIO without also changing PWM1 to PWM.

For all pins, the initial mode after a CPU reset is Mode0(GPIO).

Pin List
  • Pin Name is defined as PIN_XXX in sdk/bsp/include/arch/chip/pin.h.

  • WLCSP is the 100-pin package. Some pins are removed.

  • FCBGA is the 185-pin fully featured package.

  • ModeX describes the pin role for Modes 0-3.

Table 3. List of pins that the SDK controls.
Pin Name Arduino
compatible
Pin Name
WLCSP FCBGA Mode0 Mode1 Mode2 Mode3

I2C4_BCK

*

*

GPIO

I2C#4
(CXD5247)

I2C4_BDT

*

*

PMIC_INT

*

*

GPIO

PMIC
Interrupt

PMIC Interrupt
(OpenDrain)

RTC_IRQ_OUT

*

GPIO

RTC_IRQ_OUT

RTC_IRQ_OUT
(OpenDrain)

AP_CLK

*

*

GPIO

AP_CLK

PMU_WDT

PMU_WDT
(OpenDrain)

GNSS_1PPS_OUT

*

GPIO

GNSS_1PPS_OUT

CPU_WDT

CPU_WDT
(OpenDrain)

SPI0_CS_X

*

*

GPIO

UART#1
(Console)

SPI#0

SPI0_SCK

*

*

SPI0_MOSI

*

GPIO

I2C#2

SPI0_MISO

*

SPI1_CS_X

*

*

GPIO

SPI#1
(SPI-Flash)

SPI#0

SPI1_SCK

*

*

SPI1_IO0

*

*

SPI1_IO1

*

*

SPI1_IO2

*

*

GPIO

SPI1_IO3

*

*

SPI2_CS_X

*

*

GPIO

SPI#2
(HostIF)

UART#0
(HostIF)

I2C#3
(HostIF)

SPI2_SCK

*

*

SPI2_MOSI

D04

*

*

GPIO

SPI2_MISO

D08

*

*

HIF_IRQ_OUT

D02

*

*

GPIO

HIF_IRQ_OUT

HIF_IRQ_OUT
(OpenDrain)

GNSS_1PPS_OUT

HIF_GPIO0

*

GPIO

GPS_EXTLD

SEN_IRQ_IN

D22

*

*

GPIO

SEN_IRQ_IN

SPI3_CS0_X

*

*

GPIO

SPI3_CS0_X

SPI3_CS1_X

D07

*

*

GPIO

SPI3_CS1_X

SPI3_CS2_X

*

*

GPIO

SPI3_CS2_X

SPI3_SCK

*

*

GPIO

SPI#3
(Sensor)

SPI3_MOSI

*

*

SPI3_MISO

*

*

I2C0_BCK

D15

*

*

GPIO

I2C#0
(Sensor)

I2C0_BDT

D14

*

*

PWM0

D06

*

*

GPIO

PWM#0,1

PWM1

D05

*

*

PWM2

D09

*

*

GPIO

PWM#2,3

I2C#1
(Sensor)

PWM3

D03

*

*

IS_CLK

*

GPIO

CMOS Image Sensor

IS_VSYNC

*

IS_HSYNC

*

IS_DATA0

*

IS_DATA1

*

IS_DATA2

*

IS_DATA3

*

IS_DATA4

*

IS_DATA5

*

IS_DATA6

*

IS_DATA7

*

UART2_TXD

D01

*

*

GPIO

UART#2

UART2_RXD

D00

*

*

UART2_CTS

D27

*

*

UART2_RTS

D28

*

*

SPI4_CS_X

D10

*

*

GPIO

SPI#4

SPI4_SCK

D13

*

*

SPI4_MOSI

D11

*

*

SPI4_MISO

D12

*

*

EMMC_CLK

D23

*

*

GPIO

eMMC

SPI#5

EMMC_CMD

D24

*

*

EMMC_DATA0

D16

*

*

EMMC_DATA1

D17

*

*

EMMC_DATA2

D20

*

*

GPIO

EMMC_DATA3

D21

*

*

SDIO_CLK

*

GPIO

SDIO

SPI#5

SDIO_CMD

*

SDIO_DATA0

*

SDIO_DATA1

*

SDIO_DATA2

*

GPIO

SDIO_DATA3

*

SDIO_CD

*

GPIO

SDIO
(Card)

SDIO_WP

*

SDIO_CMDDIR

*

GPIO

SDIO

SDIO_DIR0

*

SDIO_DIR1_3

*

SDIO_CLKI

*

GPIO

SDIO
(Card)

I2S0_BCK

D26

*

*

GPIO

I2S#0

I2S0_LRCK

D25

*

*

I2S0_DATA_IN

D19

*

*

I2S0_DATA_OUT

D18

*

*

I2S1_BCK

LED0

*

GPIO

I2S#1

I2S1_LRCK

LED1

*

I2S1_DATA_IN

LED2

*

I2S1_DATA_OUT

LED3

*

MCLK

*

*

GPIO

MCLK
(Audio)

PDM_CLK

*

*

GPIO

PDM
(Audio)

PDM_IN

*

*

PDM_OUT

*

*

USB_VBUSINT

*

*

GPIO

USB VBUS Interrupt

Pin Configuration

Pin mode configuration is implemented in device the drivers. For example, the pin mode setting is performed in I2C or SPI driver, the correct pin mode is selected for the I2C or SPI function as appropriate. Therefore, there is no need for the user application to change pin mode directly.

Only when pins are used as Mode0(GPIO) is it appropriate to use APIs defined in gpioif.

Board Specific Pin Pull and Drive Current Setting

Pin pull up and down settings and drive current setting are defined in sdk/bsp/src/chip/cxd5602_pinconfig.h

The setting is Hi-Z floating by default, and drive current of the most pins is set to 2mA.

If you would like to change these default setting you do not need to modify cxd5602_pinconfig.h directly. You can select the following configuration CONFIG_BOARD_CUSTOM_PINCONFIG=y and update sdk/bsp/board/spresense/include/board_pinconfig.h.

In case of Spresense board,

/* Customize from default to the board specific pin configuration
 * The default pin configurations are defined in sdk/bsp/src/chip/cxd5602_pinconfig.h.
 *  Mode: shared pin function mode
 *  ENZI: 1=Input Enable, 0=Input Disable
 *  4mA : Drive Current 1=4mA, 0=2mA
 *  Pull: 0=HiZ floating, PINCONF_PULLUP, PINCONF_PULLDOWN
 *                                                                      M  E     P
 *                                                  P                   o  N  4  u
 *                                                  i                   d  Z  m  l
 *                                                  n                   e  I  A  l
 */
#undef PINCONF_UART2_CTS
#define PINCONF_UART2_CTS                   PINCONF(PIN_UART2_CTS,      1, 1, 0, PINCONF_PULLDOWN)

#undef PINCONF_SPI4_CS_X
#undef PINCONF_SPI4_SCK
#undef PINCONF_SPI4_MOSI
#define PINCONF_SPI4_CS_X                   PINCONF(PIN_SPI4_CS_X,      1, 0, 1, 0)
#define PINCONF_SPI4_SCK                    PINCONF(PIN_SPI4_SCK,       1, 0, 1, 0)
#define PINCONF_SPI4_MOSI                   PINCONF(PIN_SPI4_MOSI,      1, 0, 1, 0)

#undef PINCONF_PWM0
#undef PINCONF_PWM1
#undef PINCONF_PWM2
#undef PINCONF_PWM3
#define PINCONF_PWM0                        PINCONF(PIN_PWM0,           1, 0, 1, 0)
#define PINCONF_PWM1                        PINCONF(PIN_PWM1,           1, 0, 1, 0)
#define PINCONF_PWM2                        PINCONF(PIN_PWM2,           1, 0, 1, 0)
#define PINCONF_PWM3                        PINCONF(PIN_PWM3,           1, 0, 1, 0)
  • Pull down of UART2_CTS

  • Change the drive current of SPI4 from 2mA to 4mA

  • Change the drive current of PWM from 2mA to 4mA

11.3. Audio Subsystem

11.3.1. General

The CXD5602 SoC has an Audio Subsystem to support audio. It has the following functions:

  • Audio Codec hardware (AD/DA, DNC, DEQ, etc.) control

  • Audio Player function

  • Audio Recorder function

  • Bluetooth-related function (for BT-A2DP)

  • Filter functions (eg bandpass filter for voice calls etc.)

This document describes software to control the audio functions that can be implemented on the CXD5602 hardware. Please refer to Spresense Hardware Documents for Audio Hardware.

The current Firmware does not support Bluetooth functions (for BT-A2DP) and Sound Effector functions (eg band pass filters for voice calls etc)

11.3.2. About layer structure

The stack diagram of the audio subsystem is shown below.

Audio Sub-system Stack Diagram
Figure 15. Audio Sub-system Stack diagram

The audio subsystem has three major layers:

Audio Manager (High Level API)

The Audio Manager is the highest layer and controls at the highest level of abstraction. It coordinates the whole system.

Object Layer (ObjectLevel API)

The Object Layer provides simpler functions to control the audio system. You can create more flexible applications by using the Component Layer API.

Component Layer (Low Level API)

The Component Layer controls the audio system at the lower level of abstraction. Audio processing, with high degree of flexibility, can be achieve by configuring processing with a combination of Component Layer processing blocks.

11.3.3. High Level API

You can use the high-level Audio Utility Libraries to control the Audio Manager. The following is a stack diagram of the Audio Subsystem:

Control System by Command send and receive

You can control the Audio Subsystem using the high-level API Command System. To send a command to the Audio Subsystem, use AS_SendAudioCommand ; the command format is AudioCommand . To receive results from the Audio Subsystem, call AS_ReceiveAudioResult . The result format is AudioResult .

The command is a synchronous command. After issuing the command, the next command can not be issued until the result is returned. When using the High Level API via Audio Manager , please program so that send / receive procedures are paired with one command unit.
Audio HighLevel Command System
Figure 16. Audio High-Level API Command System
About control data format (command format)

The command object is 1 word (4 bytes) 'AudioCommandHeader' It is a data structure that starts with, and then adds as many parameter areas as necessary. Because the command object is based on a word unit, it consists of integral multiple of 1 word (32 bits, 4 bytes). Set 0 in the reserved field of the command object.

typedef struct
{
  AudioCommandHeader header;
  union
  {
    Command Parameters (Payload)
    ...
    ...
  };
} AudioCommand;
Command header

The first one word (4 bytes) of all command objects has the following format. This one word is called the command header ('AudioCommandHeader').

typedef struct
{
  uint8_t reserved;
  uint8_t sub_code;
  uint8_t command_code;
  uint8_t packet_length;
} AudioCommandHeader;
packet_length

Indicates the length of the command object including the command header. All command objects are composed of an integer number of words (4 bytes), and the value specified by packet_length is one quarter of the word length of the command packet, that is, the byte length of the command object.

command_code

Code specific to the command. The value 0x00 is not used. See Command list for the list of commands.

sub_code

This is a code to identify the target to be set and controlled in each command.

Notification data format (result format)

The result object is a data structure starting with 1 word (4 bytes) 'AudioResultHeader' followed by as many parameter areas as necessary. Since the result object is based on a word unit, it consists of an integral multiple of 1 word (32 bits, 4 bytes). Ignore the reserved field of the result object.

The first 1 word (4 bytes) of all result objects has the following format. This one word is called the result header ('AudioResultHeader').

typedef struct
{
  AudioResultHeader header;
  union
  {
    Result Parameters (Payload)
    ...
    ...
  };
} AudioResult;
Result header

The first one word (4 bytes) of all result objects has the following format. This one word is called the result header ('AudioCommandHeader').

typedef struct
{
  uint8_t reserved;
  uint8_t sub_code;
  uint8_t result_code;
  uint8_t packet_length;
} AudioResultHeader;
packet_length

Indicates the length of the result object including the result header. All result objects consist of an integer number of words (4 bytes), and the value specified by packet_length is the word length of the result object, that is, 1/4 of the byte length of the result object.

result_code

It is a code to identify the result type. See Result list for the list of results.

sub_code

The same value as the sub_code of the executed command is entered.

State Transition

The High Level API has several states. The state transition diagram is shown below.

diag 1cd1ed11837767c5451b2aedcd69ede8

The explanation of each mode is as follow.

  • PowerOff state

It is the state immediately after the object of the audio subsystem is generated and started up. If you do not use audio, transition to this state will make the power consumption in the audio block almost 0.

Transition only to the Ready state by the AUDCMD_POWERON command.

  • Ready state

I am preparing to turn on the audio block and operate the audio function in the operation mode. In this state, the power consumption has not decreased, but since the IO / analog is running, it is possible to perform mode transition promptly.

The state transition is as follow.

The AUDCMD_SETPOWEROFFSTATUS command allows you to enter the PowerOff state.

The AUDCMD_SETPLAYERSTATUS command allows you to enter the Player state.

The AUDCMD_SETRECORDERSTATUS command allows you to enter the Recorder state.

The AUDCMD_SETBASEBANDSTATUS command allows you to enter the Baseband state.

  • Player state

It is a state to realize the function to decode the compressed audio file from storage such as SD card and the network such as WiFi / LTE, and to sound to AnalogOut and I2S. It has two substates in the state: PlayerReady state and PlayerActive state.

The PlayerReady state is the state of music playback stop. AUDCMD_PLAYPLAYER Transit to PlayerActive and perform music playback operation.

The PlayerActive state is the state during music playback. AUDCMD_STOPPLAYER Transit to PlayerReady and stop music playback.

AUDCMD_SETREADYSTATUS Transition to the Ready state.

  • Recorder status

It is a state that compresses the voice data input from Mic and I2S, realizes the function of exporting to the storage such as SD card, and sending it to a communication network such as WiFi / LTE.

Within the state it has two substates, the RecorderReady state and the RecorderActive state.

The RecorderReady state is the state of audio recording stop. AUDCMD_STARTREC Transitions to RecorderActive and performs voice recording operation.

The RecorderActive state is the state during voice recording. AUDCMD_STOPREC Transit to RecorderReady and stop recording voice.

AUDCMD_SETREADYSTATUS Transition to the Ready state.

  • Baseband state

It is a state that realizes the function of internally processing the sound data input from Mic or I2S, and outputting it to AnalogOut or I2S. Within the state it has two substates, the BasebandReady state and the BasebandActive state.

The BasebandReady state is the state of audio input / output stop. AUDCMD_STARTBB Transit to BasebandActive and start voice input / output operation.

The RecorderActive state is a state during audio input / output operation. AUDCMD_STOPBB Transit to BasebandReady and stop audio input / output operation. AUDCMD_SETREADYSTATUS Transition to the Ready state.

The current Firmware does not support BaseBand State.
Command list

The lists of each command are as follows. You can request each function for the audio subsystem by specifying each command ID into the Command header.

General or common Command

Common commands in any state. You can call it from any state.

Table 4. General or common Command List
Command Name Command ID Response Result Detail

GetStatus

0x01

NotifyStatus

Get the current state

For details, refer to the Doxygen file below.

Baseband Initialize Command

Command to initialize Baseband HW. You can only call from the Ready state.

Table 5. Baseband Initialize Command List
Command Name Command ID Response Result Detail

InitMicGain

0x53

InitMicGainCmplt

Set the microphone gain

InitI2SParam

0x54

InitI2SCmplt

Configure I2S

InitOutputSelect

0x56

InitOutputSelectCmplt

Configure the device to be sounded

InitClearStereo

0x58

InitClearStereoCmplt

Set the clear stereo function

initRenderClk

0x5c

SetRenderingClkCmplt

Switch HiResolution setting

For details, refer to the Doxygen file below.

Baseband Set Command

This command is for setting Baseband HW. It can be called from a state other than the PowerOff state.

Table 6. Baseband Set Command List
Command Name Command ID Response Result Detail

SetVolume

0x59

SetVolumeCmplt

Perform volume setting for pronunciation

SetVolumeMute

0x5a

SetVolumeMuteCmplt

Set mute of sound volume

SetBeep

0x5b

SetBeepCmplt

BEEP Set the sound

For details, refer to the Doxygen file below.

Player Command

This command controls player. You can call it from Player state.

Table 7. Player Command List
Command Name Command ID Response Result Detail

InitPlayer

0x21

InitPlayCmplt

Sets the playback information of the player

StartPlayer

0x22

PlayCmplt

Decodes from the beginning of the buffer

StopPlayer

0x23

StopPlayCmplt

Stops Player regardless of buffer state

ClkRecovery

0x24

ClkRecoveryComplete

Fine adjustment of the sound output time

SetDecoderGain

0x25

SetDecoderGainComplete

Multiply the sound output level by Gain for L / R respectively

For details, refer to the Doxygen file below.

Recorder Command

This command controls the Recorder. You can call it from Recorder state.

Table 8. Recorder Command List
Command Name Command ID Response Result Detail

InitRecorder

0x31

InitRecCmplt

Initializes the voice recording function

StartRecorder

0x32

RecCmplt

Start audio recording

StopRecorder

0x33

StopRecCmplt

Stop audio recording

For details, refer to the Doxygen file below.

State Transition Command
Table 9. State Transition Command List
Command Name Command ID Response Result Detail

PowerOn

0x71

StatusChanged

Transit to Ready state

SetPowerOffStatus

0x72

StatusChanged

Transits to Power Off state

SetBaseBandStatus

0x73

StatusChanged

Transit to Baseband state

SetPlayerStatus

0x75

StatusChanged

Transit to Player state

SetRecorderStatus

0x76

StatusChanged

Transitions to Recorder state

SetReadyStartus

0x77

StatusChanged

Transit to Ready state

SetThrouStartus

0x78

StatusChanged

Audio path Transit to the through state

For details, refer to the Doxygen file below.

Result list

There are the lists of result from the audio susbsystem. You will be notified the result ID that stored in the Result header.

General or Common Result
Table 10. General or common result List
Result Name Command ID Trigger Command Detail

NotifyStatus

0x01

GetStatus

Notify the current state

Baseband Initialize Result
Table 11. Baseband Initialize Result List
Result Name Command ID Trigger Command Detail

InitMicGainCmplt

0x53

InitMicGain

Notify completion of the microphone gain setting

InitI2SCmplt

0x54

InitI2SParam

Notify completion of the I2S setting

InitOutputSelectCmplt

0x56

InitOutputSelect

Notify completion of the device setting to be sounded

InitClearStereoCmplt

0x58

InitClearStereo

Notify completion of the clear stereo setting

SetRenderingClkCmplt

0x5c

InitRenderClk

Notify completion to be switched to the Hi-Resolution setting

Baseband Set Result
Table 12. Baseband Set Result List
Result Name Command ID Trigger Command Detail

SetVolumeCmplt

0x59

SetVolume

Notify completion of the volume settings for pronunciation

SetVolumeMuteCmplt

0x5a

SetVolumeMute

Notify completion of the mute setting of the sound volume

SetBeepCmplt

0x5b

SetBeep

Notify completion of the beep sound setting

Player Result
Table 13. Player Result List
Result Name Command ID Trigger Command Detail

InitPlayCmplt

0x21

InitPlayer

Notify completion of the playback setting

PlayCmplt

0x22

StartPlayer

Notify the playback to be beginned

StopPlayCmplt

0x23

StopPlayer

Notify the playback to be stopped

ClkRecoveryComplete

0x24

ClkRecovery

Notify completion of the fine adjustment of the sound ouuput time

SetDecoderGainComplete

0x25

SetDecoderGain

Notify completion of the L / R gain controls

Recorder Result
Table 14. Recorder Result List
Result Name Command ID Trigger Command Detail

InitRecCmplt

0x31

nitRecorder

Notify completion to be initialized the voice recording

RecCmplt

0x32

StartRecorder

Notify the recorder to be beginned

StopRecCmplt

0x33

StopRecorder

Notify the recorder to be stopped

State Transition Result
Table 15. State Transition Result List
Result Name Command ID Trigger Command Detail

StatusChanged

0x71

PowerOn SetPowerOffStatus SetBaseBandStatus SetPlayerStatus SetRecorderStatus SetReadyStartus

Notify completion of the state transition

Error Notification Result
Table 16. Error Notification Result List
Result Name Command ID Trigger Command Detail

ErrorResponse

0xf1

Notify the error

ErrorAttention

0xf2

Notify the internal error

Details of the command packets
GetStatus

Get the state of the audio subsystem

  • Header

AudioCommandHeader
{
  uint8_t reserved      : 0x00
  uint8_t sub_code      : 0x00
  uint8_t command_code  : 0x01
  uint8_t packet_length : 0x02
}
  • Parameters

None

InitMicGain

Set the microphone gain.
It’s possible to set the gain control individually by each microphone.

  • Header

AudioCommandHeader
{
  uint8_t reserved      : 0x00
  uint8_t sub_code      : 0x00
  uint8_t command_code  : 0x53
  uint8_t packet_length : 0x05
}
  • Parameters

InitI2SParam

Configure the I2S input and output settings.

  • Header

AudioCommandHeader
{
  uint8_t reserved      : 0x00
  uint8_t sub_code      : 0x00
  uint8_t command_code  : 0x54
  uint8_t packet_length : 0x03
}
  • Parameters

InitOutputSelect

Configure the input setting

  • Header

AudioCommandHeader
{
  uint8_t reserved      : 0x00
  uint8_t sub_code      : 0x00
  uint8_t command_code  : 0x56
  uint8_t packet_length : 0x02
}
  • Parameters

InitClearStereo

Configure the clear stereo setting.

  • Header

AudioCommandHeader
{
  uint8_t reserved      : 0x00
  uint8_t sub_code      : 0x00
  uint8_t command_code  : 0x58
  uint8_t packet_length : 0x02
}
  • Parameters

initRenderClk

Set the Hi-Resolution mode.

  • Header

AudioCommandHeader
{
  uint8_t reserved      : 0x00
  uint8_t sub_code      : 0x00
  uint8_t command_code  : 0x5c
  uint8_t packet_length : 0x02
}
  • Parameters

SetVolume
  • Header

AudioCommandHeader
{
  uint8_t reserved      : 0x00
  uint8_t sub_code      : 0x00
  uint8_t command_code  : 0x59
  uint8_t packet_length : 0x03
}
  • Parameters

SetVolumeMute
  • Header

AudioCommandHeader
{
  uint8_t reserved      : 0x00
  uint8_t sub_code      : 0x00
  uint8_t command_code  : 0x5a
  uint8_t packet_length : 0x02
}
  • Parameters

SetBeep
  • Header

AudioCommandHeader
{
  uint8_t reserved      : 0x00
  uint8_t sub_code      : 0x00
  uint8_t command_code  : 0x5b
  uint8_t packet_length : 0x03
}
  • Parameters

InitPlayer
  • Header

AudioCommandHeader
{
  uint8_t reserved      : 0x00
  uint8_t sub_code      : 0x00
  uint8_t command_code  : 0x21
  uint8_t packet_length : 0x09
}
  • Parameters

StartPlayer
  • Header

AudioCommandHeader
{
  uint8_t reserved      : 0x00
  uint8_t sub_code      : 0x00
  uint8_t command_code  : 0x22
  uint8_t packet_length : 0x02
}
  • Parameters

StopPlayer
  • Header

AudioCommandHeader
{
  uint8_t reserved      : 0x00
  uint8_t sub_code      : 0x00
  uint8_t command_code  : 0x23
  uint8_t packet_length : 0x02
}
  • Parameters

ClkRecovery
  • Header

AudioCommandHeader
{
  uint8_t reserved      : 0x00
  uint8_t sub_code      : 0x00
  uint8_t command_code  : 0x24
  uint8_t packet_length : 0x02
}
  • Parameters

SetDecoderGain
  • Header

AudioCommandHeader
{
  uint8_t reserved      : 0x00
  uint8_t sub_code      : 0x00
  uint8_t command_code  : 0x25
  uint8_t packet_length : 0x02
}
  • Parameters

InitRecorder
  • Header

AudioCommandHeader
{
  uint8_t reserved      : 0x00
  uint8_t sub_code      : 0x00
  uint8_t command_code  : 0x31
  uint8_t packet_length : 0x0a
}
  • Parameters

StartRecorder
  • Header

AudioCommandHeader
{
  uint8_t reserved      : 0x00
  uint8_t sub_code      : 0x00
  uint8_t command_code  : 0x32
  uint8_t packet_length : 0x02
}
  • Parameters

None

StopRecorder
  • Header

AudioCommandHeader
{
  uint8_t reserved      : 0x00
  uint8_t sub_code      : 0x00
  uint8_t command_code  : 0x33
  uint8_t packet_length : 0x02
}
  • Parameters

None

PowerOn
  • Header

AudioCommandHeader
{
  uint8_t reserved      : 0x00
  uint8_t sub_code      : 0x00
  uint8_t command_code  : 0x71
  uint8_t packet_length : 0x02
}
  • Parameters

SetPowerOffStatus
  • Header

AudioCommandHeader
{
  uint8_t reserved      : 0x00
  uint8_t sub_code      : 0x00
  uint8_t command_code  : 0x72
  uint8_t packet_length : 0x02
}
  • Parameters

None

SetBaseBandStatus
  • Header

AudioCommandHeader
{
  uint8_t reserved      : 0x00
  uint8_t sub_code      : 0x00
  uint8_t command_code  : 0x73
  uint8_t packet_length : 0x09
}
  • Parameters

SetPlayerStatus
  • Header

AudioCommandHeader
{
  uint8_t reserved      : 0x00
  uint8_t sub_code      : 0x00
  uint8_t command_code  : 0x75
  uint8_t packet_length : 0x06
}
  • Parameters

SetRecorderStatus
  • Header

AudioCommandHeader
{
  uint8_t reserved      : 0x00
  uint8_t sub_code      : 0x00
  uint8_t command_code  : 0x76
  uint8_t packet_length : 0x04
}
  • Parameters

SetReadyStartus
  • Header

AudioCommandHeader
{
  uint8_t reserved      : 0x00
  uint8_t sub_code      : 0x00
  uint8_t command_code  : 0x77
  uint8_t packet_length : 0x02
}
  • Parameters

None

SetThroughStatus
  • Header

AudioCommandHeader
{
  uint8_t reserved      : 0x00
  uint8_t sub_code      : 0x00
  uint8_t command_code  : 0x78
  uint8_t packet_length : 0x02
}
  • parameters

None

Details of the result packets
NotifyStatus

Notify the response of GetStatus.

  • Header

AudioResultHeader
{
  uint8_t reserved      : 0x00
  uint8_t sub_code      : 0x00
  uint8_t result_code   : 0x01
  uint8_t packet_length : 0x02
}
  • Parameters

typedef struct
{
  uint8_t vad_status;
  uint8_t reserved1;
  uint8_t sub_status_info;
  uint8_t status_info;
} NotifyStatus;
vad_status

The current VAD (Voice Activity Detection) status.

sub_status_info

The current sub-status.

status_info

The current status.

InitMicGainCmplt

Notify the response of InitMicGain.

  • Header

AudioResultHeader
{
  uint8_t reserved      : 0x00
  uint8_t sub_code      : 0x00
  uint8_t result_code   : 0x53
  uint8_t packet_length : 0x02
}
  • Parameters

None

InitI2SCmplt
  • Header

AudioResultHeader
{
  uint8_t reserved      : 0x00
  uint8_t sub_code      : 0x00
  uint8_t result_code   : 0x54
  uint8_t packet_length : 0x02
}
  • Parameters

None

InitOutputSelectCmplt
  • Header

AudioResultHeader
{
  uint8_t reserved      : 0x00
  uint8_t sub_code      : 0x00
  uint8_t result_code   : 0x56
  uint8_t packet_length : 0x02
}
  • Parameters

None

InitClearStereoCmplt
  • Header

AudioResultHeader
{
  uint8_t reserved      : 0x00
  uint8_t sub_code      : 0x00
  uint8_t result_code   : 0x58
  uint8_t packet_length : 0x02
}
  • Parameters

None

SetRenderingClkCmplt
  • Header

AudioResultHeader
{
  uint8_t reserved      : 0x00
  uint8_t sub_code      : 0x00
  uint8_t result_code   : 0x5c
  uint8_t packet_length : 0x02
}
  • Parameters

None

SetVolumeCmplt
  • Header

AudioResultHeader
{
  uint8_t reserved      : 0x00
  uint8_t sub_code      : 0x00
  uint8_t result_code   : 0x59
  uint8_t packet_length : 0x02
}
  • Parameters

None

SetVolumeMuteCmplt
  • Header

AudioResultHeader
{
  uint8_t reserved      : 0x00
  uint8_t sub_code      : 0x00
  uint8_t result_code   : 0x5a
  uint8_t packet_length : 0x02
}
  • Parameters

None

SetBeepCmplt
  • Header

AudioResultHeader
{
  uint8_t reserved      : 0x00
  uint8_t sub_code      : 0x00
  uint8_t result_code   : 0x5b
  uint8_t packet_length : 0x02
}
  • Parameters

None

InitPlayCmplt
  • Header

AudioResultHeader
{
  uint8_t reserved      : 0x00
  uint8_t sub_code      : 0x00
  uint8_t result_code   : 0x21
  uint8_t packet_length : 0x02
}
  • Parameters

None

PlayCmplt
  • Header

AudioResultHeader
{
  uint8_t reserved      : 0x00
  uint8_t sub_code      : 0x00
  uint8_t result_code   : 0x22
  uint8_t packet_length : 0x02
}
  • Parameters

None

StopPlayCmplt
  • Header

AudioResultHeader
{
  uint8_t reserved      : 0x00
  uint8_t sub_code      : 0x00
  uint8_t result_code   : 0x23
  uint8_t packet_length : 0x02
}
  • Parameters

None

ClkRecoveryComplete
  • Header

AudioResultHeader
{
  uint8_t reserved      : 0x00
  uint8_t sub_code      : 0x00
  uint8_t result_code   : 0x24
  uint8_t packet_length : 0x02
}
  • Parameters

None

SetDecoderGainComplete
  • Header

AudioResultHeader
{
  uint8_t reserved      : 0x00
  uint8_t sub_code      : 0x00
  uint8_t result_code   : 0x25
  uint8_t packet_length : 0x02
}
  • Parameters

None

InitRecCmplt
  • Header

AudioResultHeader
{
  uint8_t reserved      : 0x00
  uint8_t sub_code      : 0x00
  uint8_t result_code   : 0x31
  uint8_t packet_length : 0x02
}
  • Parameters

None

RecCmplt
  • Header

AudioResultHeader
{
  uint8_t reserved      : 0x00
  uint8_t sub_code      : 0x00
  uint8_t result_code   : 0x32
  uint8_t packet_length : 0x02
}
  • Parameters

None

StopRecCmplt
  • Header

AudioResultHeader
{
  uint8_t reserved      : 0x00
  uint8_t sub_code      : 0x00
  uint8_t result_code   : 0x33
  uint8_t packet_length : 0x02
}
  • Parameters

None

StatusChanged
  • Header

AudioResultHeader
{
  uint8_t reserved      : 0x00
  uint8_t sub_code      : 0x00
  uint8_t result_code   : 0x7f
  uint8_t packet_length : 0x02
}
  • Parameters

typedef struct
{
  uint8_t changed_status;
  uint8_t reseved1;
  uint8_t reseved2;
  uint8_t reseved3;
} StatusChangedParam;
changed_status

The status after transition.

ErrorResponse

Returns this response if the command error occurs.
The details of error are stored in the parameter area.

  • Header

AudioResultHeader
{
  uint8_t reserved      : 0x00
  uint8_t sub_code      : 0x00
  uint8_t result_code   : 0xf1
  uint8_t packet_length : 0x02
}
  • Parameters

typedef struct
{
  uint32_t error_code;
  uint16_t reserved1;
  uint8_t  sub_module_id;
  uint8_t  module_id;
  uint32_t error_sub_code;
  uint32_t reserved2;
  uint32_t ErrorCommand[];
} ErrorResponseParam;
error_code

The kind of error that are described in Error Code List.

module_id

The module ID with error. These are defined in Module ID List.

sub_module_id

Not used.

error_sub_code

The kind of error sub code which are different for each command.

ErrorCommand[]

Not used.

ErrorAttention

Notify the internal error.

  • Header

AudioResultHeader
{
  uint8_t reserved      : 0x00
  uint8_t sub_code      : 0x00
  uint8_t result_code   : 0xf2
  uint8_t packet_length : 0x0e
}
  • Parameters

typedef struct
{
  uint32_t reserved1;
  uint8_t  error_code;
  uint8_t  cpu_id;
  uint8_t  sub_module_id;
  uint8_t  module_id;
  uint32_t error_att_sub_code;
  uint32_t reserved2;
  uint16_t line_number;
  uint8_t  task_id;
  uint8_t  reserved3;
  uint8_t  error_filename[];
} ErrorAttentionParam;
error_code

The kind of error.

module_id

The module ID with error.

error_att_sub_code

The details of error, are described in Attention Code List.

line_number

The line number with error.

error_filename

The file name with error. The maximum length of the file name is 32 bytes.

Memory management and inter-task synchronization
Memory Manager Library

AudioSubSystem manages the data area to be used with a special management method. A library called MemoryManager in MemoryUtility secures the required memory area as a fixed-size memory pool according to the layout information of the memory_layout file. You can have multiple Layout information in one file, and you can secure the necessary memory for each function by specifying the Layout number.

This memory layout is to be freely decided according to the needs of Application, but please note that the minimum necessary resources are available for each function.

For details, refer to the library description of Memory Manager.

Also, please refer to the explanation of each example for necessary Layout according to each function.

Each object in AudioSubSystem creates an instance of MemHandle that points to a segment of the required memory area, thereby securing a memory segment linked to it and using it.

Get memHandle

By passing an instance of this MemHandle to the object located next to the data pipeline, the next object can be used by the next object, and if it becomes unnecessary, by destroying this instance , The memory area is freed.

Instances can be copied, secured instances are secured while necessary objects are required for their own memory, and even if they are no longer needed, they can be secured / released securely even if they are discarded at their own timing.

Copy memHandle

When the use of a segment becomes unnecessary, an object executed asynchronously is implicitly referenced by destroying the instance without being conscious of memory management.

Use memHandle

This makes it easy to manage memory between asynchronous objects. When all references are gone, release the memory.

Release memHandle

For Memory Layout, you need to prepare a header file group to use Memory Layout in advance. These header files are created by creating a Memory Layout definition file (mem_layout.conf) and using the tool.

How to use the tool: python3 mem_layout.conf layout_header fence_header pool_header

The explanation of each argument is as follow.

mem_layout.conf

Memory Layout definition file

layout_header

Variable constant values are header files output as macros of C language header file in which memory fence address of

fence_header

FixedArea is output header file where various definitions of

pool_header

PoolArea are output

Message Library

In order to use this memory management mechanism, it is necessary to send and receive class objects between tasks. In order to realize this, we have prepared a message library that allows sending and receiving class instances to the task synchronization mechanism, which AudioSubSystem uses.

By adding the ID of the sending / receiving destination to the object in each task, it is possible to send to the task you want to send and receive.

For example, in the case of sending, the object is sent to the ID of the destination, and the task of the receiving side will be received only when a transmission request of your ID occurs. Until reception, the task sleeps and waits.

By doing this, AudioSubSystem is doing object design by event driven.

For Message, it is necessary to prepare a header file group for using Message in advance. These header files are created by creating a MessageQueueLayout definition file (msgq_layout.conf) and using the tool.

How to use the tool: python3 msgq_layout.conf area_address area_size id_header pool_header

The explanation of each argument is as follow.

msgq_layout.conf

Message Layout definition file

Address of area_address

Message area

area_size

Message Size of area

id_header

MessageQueueID File in which the macro is output

File in which definition of pool_header

MessageQueuePool is output

For details, refer to the explanation of the Message Library.

Simple FIFO Library

When audio data is exchanged between AudioSubSystem and user application FIFO is used. This FIFO is a simple FIFO and there is nothing special to mention.

For details, refer to the library description of Simple FIFO.

Details of each function
Audio Player

The Audio Subsystem has an audio player function with sound effects.

The following is a simple data flow diagram of the audio player:

Audio player dataflow
Figure 17. Audio Player Data Flow

When Audio SubSystem operates with PlayerMode, user Application inputs ES data into FIFO. When Player is started with a certain level of accumulation, this ES data will be consumed in accordance with the playing time. Unless this FIFO underflow, audio data will play without interruption.

Player can generate two instances. Audio decoded by each is Output Mixer, and it mixes and sounds.

The data flow communicates via message sending internally. Message communication has an ID for each client. In the case of Audio Player, the ID will be as follow for the Layout example:

User Application                       : MSGQ_AUD_APP
Audio Manager                          : MSGQ_AUD_MNG
Audio Player (Main Sound)              : MSGQ_AUD_PLY
Audio Player (Effect Sound)            : MSGQ_AUD_SFX
Output Mixer                           : MSGQ_AUD_OUTPUT_MIX
Audio DSP                              : MSGQ_AUD_DSP
Rendering Component(Main Sound)        : MSGQ_AUD_RND_PLY
Rendering Component(Effect Sound)      : MSGQ_AUD_RND_SFX
Rendering Component Sync(Main Sound)   : MSGQ_AUD_RND_PLY_SYNC
Rendering Component Sync(Effect Sound) : MSGQ_AUD_RND_SFX_SYNC
Post Filter (Channel0)                 : MSGQ_AUD_PFDSP0
Post Filter (Channel1)                 : MSGQ_AUD_PFDSP1
MSGQ_AUD_RND_PLY / SFX_SYNC will be deleted.
Audio Player Message ID
Figure 18. Audio Player Message ID

In addition, the data area of ​​each data is as follows.

ES Data (Main Sound)                             : DEC_ES_MAIN_BUF_POOL
ES Data (Effect Sound)                           : DEC_ES_SFX_BUF_POOL
PCM Data (Main Sound)                            : REND_PCM_BUF_POOL
PCM Data (Effect Sound)                          : REND_PCM_SFX_BUF_POOL
Audio Decoder DSP Command                        : DEC_APU_CMD_POOL
SamplingRateConverter Work Buffer (Main Sound)   : SRC_WORK_BUF_POOL
SamplingRateConverter Work Buffer (Effect Sound) : SRC_WORK_SFX_BUF_POOL
Post Filter PCM Data (Channel0)                  : PF0_PCM_BUF_POOL
Post Filter PCM Data (Channel1)                  : PF1_PCM_BUF_POOL
Post Filter DSP Command (Channel0)               : PF0_APU_CMD_POOL
Post Filter DSP Command (Channel1)               : PF1_APU_CMD_POOL
Audio Player Pool ID
Figure 19. Audio Player Pool ID

These IDs must be specified when generating.

How to use
Preparation

The following software components control the Audio Subsystem:

  • AudioManager

  • playerObject

  • OutputmixObject

  • RenderComponent

To access these components in an application, call their respective create functions.

Create AudioManager

In order to enable AudioManager, you need to call AS_CreateAudioManager(AudioSubSystemIDs) . In the AudioSubSystemIDs structure, specify the MsgQueID defined in the Message Library Configuration. Specify 0xFF for an unused object.
AudioAttentionCb, specify a callback function for asynchronous notification. If NULL is specified, no notification is made.

static void attention_callback(const ErrorAttentionParam *attparam)
{
  ...
}

AudioSubSystemIDs ids;

ids.app         = MSGQ_AUD_APP;
ids.mng         = MSGQ_AUD_MNG;
ids.player_main = MSGQ_AUD_PLY;
ids.player_sub  = MSGQ_AUD_SFX;
ids.mixer       = MSGQ_AUD_OUTPUT_MIX;
ids.recorder    = 0xFF;
ids.effector    = 0xFF;
ids.recognizer  = 0xFF;

AS_CreateAudioManager(ids, attention_callback);
Create MediaPlayerObject

In order to enable MediaPlayerObject you need to call AS_CreatePlayerMulti (AsPlayerId, AsCreatePlayerParam_t).
For AsPlayerId, specify one of the two players (AS_PLAYER_ID_0, AS_PLAYER_ID_1 of AsPlayerId). To use two players, please create respectively.
In AsCreatePlayerParam_t, you need to specify MsgQueID defined in Message Library Configuration and PoolId defined in Memory Manager Configuration.
AudioAttentionCb, specify a callback function for asynchronous notification. Please register when you are not using. If you register a callback function with AS_CreateAudioManager or if you specify NULL in AS_CreatePlayerMulti, notification is not done.

static void attention_callback_from_player0(const ErrorAttentionParam *attparam)
{
  ...
}

static void attention_callback_from_player1(const ErrorAttentionParam *attparam)
{
  ...
}

AsCreatePlayerParam_t cparam;

cparam.msgq_id.player = MSGQ_AUD_PLY;
cparam.msgq_id.mng    = MSGQ_AUD_MNG;
cparam.msgq_id.mixer  = MSGQ_AUD_OUTPUT_MIX
cparam.msgq_id.dsp    = MSGQ_AUD_DSP;
cparam.pool_id.es     = DEC_ES_MAIN_BUF_POOL;
cparam.pool_id.pcm    = REND_PCM_BUF_POOL;
cparam.pool_id.dsp    = DEC_APU_CMD_POOL;
cparam.pool_id.src_work = SRC_WORK_BUF_POOL;

bool act_rst = AS_CreatePlayerMulti(AS_PLAYER_ID_0, &cparam, attention_callback_from_player0);

cparam.msgq_id.player = MSGQ_AUD_SFX;
cparam.msgq_id.mng    = MSGQ_AUD_MNG;
cparam.msgq_id.mixer  = MSGQ_AUD_OUTPUT_MIX
cparam.msgq_id.dsp    = MSGQ_AUD_DSP;
cparam.pool_id.es     = DEC_ES_SFX_BUF_POOL;
cparam.pool_id.pcm    = REND_PCM_SFX_BUF_POOL;
cparam.pool_id.dsp    = DEC_APU_CMD_POOL;
cparam.pool_id.src_work = SRC_WORK_SFX_BUF_POOL;

act_rst = AS_CreatePlayerMulti(AS_PLAYER_ID_1, &cparam, attention_callback_from_player1);
Create OutputMixerObject

In order to enable OutputMixerObject you need to call AS_CreateOutputMixer(AsCreateOutputMixParam_t).
In AsCreateOutputMixParam_t, you need to specify MsgQueID defined in AsMessage Library Configuration and PoolId defined in Memory Manager Configuration.
AudioAttentionCb, specify a callback function for asynchronous notification. Please specify when not using AudioManager. If you register a callback function with AS_CreateAudioManager or if you specify NULL in AS_CreatePlayerMulti, notification is not done.

static void attention_callback_from_outputmixer(const ErrorAttentionParam *attparam)
{
  ...
}

AsCreateOutputMixParam_t cparam;

cparam.msgq_id.mixer                   = MSGQ_AUD_OUTPUT_MIX;
cparam.msgq_id.render_path0_filter_dsp = MSGQ_AUD_PFDSP0;
cparam.msgq_id.render_path1_filter_dsp = MSGQ_AUD_PFDSP1;
cparam.pool_id.render_path0_filter_pcm = PF0_PCM_BUF_POOL;
cparam.pool_id.render_path1_filter_pcm = PF1_PCM_BUF_POOL;
cparam.pool_id.render_path0_filter_dsp = PF0_APU_CMD_POOL;
cparam.pool_id.render_path1_filter_dsp = PF1_APU_CMD_POOL;

result = AS_CreateOutputMixer(&output_mix_act_param, attention_callback_from_outputmixer);
Create RendererComponent

In order to enable RendererComponent you need to call AS_CreateRenderer(AsCreateRendererParam_t).
In AsCreateRendererParam_t, you need to specify MsgQueID defined in Message Library Configuration and PoolId defined in Memory Manager Configuration.
If your application uses only one MediaPlayer, you need to set the third and fourth arguments to 0xFF as shown below.

AsCreateRendererParam_t renderer_act_param;
cparam.msgq_id.dev0_req  = MSGQ_AUD_RND_PLY;
cparam.msgq_id.dev0_sync = MSGQ_AUD_RND_PLY_SYNC;
cparam.msgq_id.dev1_req  = MSGQ_AUD_RND_SFX;
cparam.msgq_id.dev1_sync = MSGQ_AUD_RND_SFX_SYNC;

result = AS_CreateRenderer(&renderer_act_param);
In the future, the Generate function in HighLevel API will only be AudioManager.
Initialize and Status change

When the necessary objects are created, initialization process for setting the Audio HW, turning on power supply, change of operation mode, etc. is carried out in order to perform Player operation.

Please use the following commands to do the initialization:

Power on Audio SubSystem

In order to enable the audio block, issue the AUDCMD_POWERON, PowerOnParam command to turn on the power and change the state of the Audio Sub system to the Ready state.
Enable_sound_effect must be fixed to AS_DISABLE_SOUNDEFFECT.

  AudioCommand command;
  command.header.packet_length = LENGTH_POWERON;
  command.header.command_code  = AUDCMD_POWERON;
  command.header.sub_code      = 0x00;
  command.power_on_param.enable_sound_effect = AS_DISABLE_SOUNDEFFECT;
  AS_SendAudioCommand(&command);
Initialize output device

After performing PowerOn and transitioning to the Ready state, use the AUDCMD_INITOUTPUTSELECT, InitOutputSelectParam command to select the output destination from Mixer.

The set up for output_device_sel is as follow:

output_device_sel
  AS_OUT_OFF: Output OFF
  AS_OUT_SP: Output from Speaker
  AS_OUT_I2S: Output from I2S
Setup commands example:
  AudioCommand command;
  command.header.packet_length = LENGTH_INITOUTPUTSELECT;
  command.header.command_code  = AUDCMD_INITOUTPUTSELECT;
  command.header.sub_code      = 0x00;
  command.init_output_select_param.output_device_sel = AS_OUT_SP;
  AS_SendAudioCommand(&command);
AUDCMD_INITI2SPARAM is not supported. Please change the setting of I2S from Kconfig.
Set speaker driver mode

To enable the driver of the digital amplifier for the speakers use these commands: AUDCMD_SETSPDRVMODE, SetSpDrvModeParam.

The setting of mode indicating driving ability is as follows.

For details on how to use speakers, see How to use speakers in the hardware documentation.

mode
  AS_SP_DRV_MODE_LINEOUT : Driving ability weakest. for Line-out.
  AS_SP_DRV_MODE_1DRIVER : Driving ability weaker. for headphone out.
  AS_SP_DRV_MODE_4DRIVER : Driving ability strongest. for speaker out.
Setup commands example:
  AudioCommand command;
  command.header.packet_length = LENGTH_SETSPDRVMODE;
  command.header.command_code  = AUDCMD_SETSPDRVMODE;
  command.header.sub_code      = 0x00;
  command.set_sp_drv_mode.mode = AS_SP_DRV_MODE_LINEOUT;
  AS_SendAudioCommand(&command);
Change to Player Status

Use AUDCMD_SETPLAYERSTATUS, SetPlayerStsParam command to change the state of AudioSubSystem to Player state.

The setting of each parameter is as follow:

active_player
  AS_ACTPLAYER_MAIN : Play only player 0
  AS_ACTPLAYER_SUB  : Play only player 1
  AS_ACTPLAYER_BOTH : Mix player 0 and player 1 and play
input_device
  AS_SETPLAYER_INPUTDEVICE_RAM:: Input from RAM (fixed)
ram_handler

Specify a pointer to Handle information of SimpleFifo.

simple_fifo_handler

  Specify hands acquired by CMN_SimpleFifoInitialize ().

callback_function

  Callback that PlayerObject notifies of events read from SimpleFifo. The size of the read data will be notified.

notification_threshold_size

  Please specify the number of PlayerObjects which need to be read before callback notification is performed. You will be notified when reading is over for the size specified here.
  If 0 is specified, it will be notified each time PlayerObject reads it.

Setup commands example:
  AsPlayerInputDeviceHdlrForRAM input0_ram_handler;
  input0_ram_handler.simple_fifo_handler         = &input0_handle;
  input0_ram_handler.callback_function           = input0_device_callback;
  input0_ram_handler.notification_threshold_size = 0;

  AsPlayerInputDeviceHdlrForRAM input1_ram_handler;
  input1_ram_handler.simple_fifo_handler         = &input1_handle;
  input1_ram_handler.callback_function           = input1_device_callback;
  input1_ram_handler.notification_threshold_size = 0;
  AudioCommand command;
  command.header.packet_length = LENGTH_SET_PLAYER_STATUS;
  command.header.command_code = AUDCMD_SETPLAYERSTATUS;
  command.header.sub_code = 0x00;
  command.set_player_sts_param.active_player         = AS_ACTPLAYER_BOTH;
  command.set_player_sts_param.player0.input_device  = AS_SETPLAYER_INPUTDEVICE_RAM;
  command.set_player_sts_param.player0.ram_handler   = &input0_ram_handler;
  command.set_player_sts_param.player0.output_device = 0x00;
  command.set_player_sts_param.player1.input_device  = AS_SETPLAYER_INPUTDEVICE_RAM;
  command.set_player_sts_param.player1.ram_handler   = &input1_ram_handler;
  command.set_player_sts_param.player1.output_device = 0x00;
  AS_SendAudioCommand(&command);
When using player 1, please activate AS_PLAYER_ID_1 by calling AS_CreatePlayerMulti(AsPlayerId, AsCreatePlayerParam_t, AudioAttentionCb).
Set volume

When Speaker is set as output, you can set the volume with AUDCMD_SETVOLUME, SetVolumeParam. The setting of each parameter is as follow:

Please note volume can not be changed in I2S.

input1_db

This sets up the volume of player 0. Use a 10-fold integer value to set dB. The setting range is from -1020 (-102.0 dB) to 120 ( 12.0 dB) with step width 5 (0.5 dB).

input2_db

This sets up the volume of player1. The setting range is the same as input1_db.

master_db

This sets up the volume for using both player 0 and player 1. The setting range is the same as input1_db.

  AudioCommand command;
  command.header.packet_length = LENGTH_SETVOLUME;
  command.header.command_code  = AUDCMD_SETVOLUME;
  command.header.sub_code      = 0;
  command.set_volume_param.input1_db = 0;    /* 0.0dB */
  command.set_volume_param.input2_db = 0;    /* 0.0dB */
  command.set_volume_param.master_db = -200; /* -20.0dB */
  AS_SendAudioCommand(&command);
diag ab6be6ebf643316e68d2560a6e3e3b5b
Figure 20. Initialization sequence
Initialize and Start player

This section describes the music playback initialization and start sequence.

Player initialization

AUDCMD_INITPLAYER, PlayerCommand, AsInitPlayerParam to initialize the playback.

player_id

AsPlayerId Sets the ID of the instance. There are two instances as shown below, please set either one.

Instance number setting value

0

AS_PLAYER_ID_0

1

AS_PLAYER_ID_1

codec_type

Please set the codec type of the playback content to MP3 or WAV as shown here:

Codec type Setting value

MP3

AS_CODECTYPE_MP3

WAV

AS_CODECTYPE_WAV

bit_length

Sets the bit length per sample of playback content to 16 bit and 24bit as shown here:

bit length set value

16

AS_BITLENGTH_16

24

AS_BITLENGTH_24

NOTE:

channel_number

Sets the number of channels of playback content to mono (1ch) or stereo (2ch) as shown here:

Number of channels Setting value

1

AS_CHANNEL_MONO

2

AS_CHANNEL_STEREO

sampling_rate

Sets the sampling frequency of the playback content. The setting value that can be set differs for each codec type as shown here:

Sampling frequency Set value Corresponding codec type

16kHz

AS_SAMPLINGRATE_16000

MP3,WAV

32kHz

AS_SAMPLINGRATE_32000

MP3,WAV

44.1kHz

AS_SAMPLINGRATE_44100

MP3,WAV

48kHz

AS_SAMPLINGRATE_48000

MP3,WAV

88.2kHz

AS_SAMPLINGRATE_88200

WAV

96kHz

AS_SAMPLINGRATE_96000

WAV

176.4kHz

AS_SAMPLINGRATE_176400

WAV

192kHz

AS_SAMPLINGRATE_192000

WAV

Automatic detection

AS_SAMPLINGRATE_AUTO

MP3

AS_SAMPLINGRATE_AUTO is used to automatically determine the sampling frequency from Syntax stream syntax. Currently, it support only MP3.
For the high resolution sampling rate, such as, AS_SAMPLINGRATE_88200 , AS_SAMPLINGRATE_96000 , and AS_SAMPLINGRATE_176400 , use the DSP with DualCore and more working area. When trying to perform Dual Decode, 384 kB is required for the DSP area. If necessary, change the SDK configuration and change the DSP area.
dsp_path

Specify the absolute path that stores Decoder’s DSP binary image, you can use up to 24 characters.

Command setting example
  AudioCommand command;
  command.header.packet_length = LENGTH_INIT_PLAYER;
  command.header.command_code  = AUDCMD_INITPLAYER;
  command.header.sub_code      = 0x00;
  command.player.player_id                  = AS_PLAYER_ID_0;
  command.player.init_param.codec_type      = AS_CODECTYPE_MP3;
  command.player.init_param.bit_length      = AS_BITLENGTH_16;
  command.player.init_param.channel_number  = AS_CHANNEL_STEREO;
  command.player.init_param.sampling_rate   = AS_SAMPLINGRATE_48000;
  command.player.init_param.dsp_path        = "/mnt/sd0/BIN";
  AS_SendAudioCommand(&command);
Start Player

AUDCMD_PLAYPLAYER, PlayerCommand will start playing. When music playback is started, it starts to read the compressed audio data from the FIFO.
For this reason, please input a sufficient amount of compressed audio data to the FIFO before music playback starts.

If you do not enter a sufficient amount of data in the FIFO at the start, Underflow will occur immediately after starting, and audio playback will stop.
player_id

AsPlayerId Set the ID of the instance. AUDCMD_INITPLAYER initialized instance ID Please set.

Instance number setting value

0

AS_PLAYER_ID_0

1

AS_PLAYER_ID_1

  AudioCommand command;
  command.header.packet_length = LENGTH_PLAY_PLAYER;
  command.header.command_code  = AUDCMD_PLAYPLAYER;
  command.header.sub_code      = 0x00;
  command.player.player_id     = AS_PLAYER_ID_0;
  AS_SendAudioCommand(&command);
diag 7e47356b50f2845c4cd105cf47495f2c
Figure 21. Player State sequence
Stop playback

This section describes the sequence of audio playback stop.

Stop Player
player_id

AsPlayerId Set the ID of the instance. AUDCMD_PLAYPLAYER Please specify the instance ID you want to stop playback with. It must be the same as started instance ID.

Instance number setting value

0

AS_PLAYER_ID_0

1

AS_PLAYER_ID_1

stop_mode

AsStopPlayerStopMode Sets the stop mode. There are three types of stop modes: normal stop, ES end stop, and forced stop.
At ES end stop, all data contained in the FIFO will be played and stopped at the stop request time.
Normal stop will stop immediately at the timing of the stop request and the contents of the FIFO will remain.
Forced stop will be published internally when error occurred in Audio Subsytem. Application has no need to use this mode.

Stop mode Setting value

Normal stop

AS_STOPPLAYER_NORMAL

Wait ES end

AS_STOPPLAYER_ESEND

Forced stop

AS_STOPPLAYER_FORCIBLY

  AudioCommand command;
  command.header.packet_length = LENGTH_STOP_PLAYER;
  command.header.command_code  = AUDCMD_STOPPLAYER;
  command.header.sub_code      = 0x00;
  command.player.player_id     = AS_PLAYER_ID_0;
  command.player.stop_param.stop_mode = AS_STOPPLAYER_NORMAL;
  AS_SendAudioCommand(&command);
diag 04921c65915d8e75e2586c91fcea81be
Figure 22. Stop sequence
Build Configurations

In order to use the function of AudioPlayer please follow these instructions:

$> cd sdk
$> tools/config.py -m

You need to open the Config menu and configure the following Config.

Select options as shown below:

:(Select audio player application)
[CXD65xx Configuration]
  [SDIO SD Card]                 <= Y (If use SD card)
  [Audio]                        <= Y
    [Audio baseband config settings]
      [CXD5247 settings]
        [X'tal frequency of the CXD5247] <= 49.152Mhz (If DVT board)

[SDK audio]                      <= Y
  [Audio Utilities]
    [Audio Player]               <= Y
      [Playlist manager]         <= Y (If use PlayList)

[Memory Manager]                 <= Y
  [Memory Utilities]             <= Y

[ASMP]                           <= Y
Error Attentions and Approach

This section shows a list of audio playback error attentions and suggested action to remove the error. See Error Information of Audio SubSystem for more details.

ID Attention Code Attention Level Approach

0x05

AS_ATTENTION_SUB_CODE_SIMPLE_FIFO_UNDERFLOW

WARNING

This is because AudioSubSystem could not read playback data. Increase the CPU priority of tasks that write playback data to Simple FIFO.

0x0D

AS_ATTENTION_SUB_CODE_MEMHANDLE_ALLOC_ERROR

ERROR

This is due to insufficient number of segments in the data area. Decrease the priority of tasks other than AudioSubSystem or increase the number of segments in the data area.

0x0F

AS_ATTENTION_SUB_CODE_TASK_CREATE_ERROR

ERROR

This is due to insufficient heap space. Expand the heap area.

0x18

AS_ECODE_DSP_VERSION_ERROR

ERROR

Due to the different version of the DSP binary. Please update the DSP binary image with the file "sdk/modules/audio/dsp".

0x1A

AS_ATTENTION_SUB_CODE_STREAM_PARSER_ERROR

ERROR

Sync word was not found in the playback file. Make sure that the playback file matches the specified codec.

0x21

AS_ATTENTION_SUB_CODE_ALLOC_HEAP_MEMORY

WARNING

Heap area was used instead of pool area. Please confirm that pool area (SRC_WORK_BUF_POOL) of work buffer of sampling rate converter is set.

AS_ATTENTION_SUB_CODE_SIMPLE_FIFO_UNDERFLOW When this occurs audio playback stops and playback error occurs. If this happens, immediately issue the AsStopPlayerParam command and change to the playback stop state.
Be sure to clear FIFO after transition to play stop. If the application does not do so noise will be generated.
DSP install
DSP binary image install

Store the DSP binary image in the path set in Kconfig. The binary image is in sdk/modules/audio/dsp .

Table 17. Binary image required for audio player according to configuration:
Image size

MP3DEC

61 kbyte

WAVDEC

32 kbyte

When playing high resolution sampling rate, make sure you use 2 cores (192 kB per core) DSP.

Audio Player Example

This section will show you an audio player example to playback the music using a simple sampling application .

Preparation
Build Configurations (kconfig)

In build configuration setup Audio player example to Y to use the sample programs for Audio Player.

(audio player:)
[Examples]
  [Audio player example] <= Y

Alternatively,

$> cd sdk $> tools/config.py examples/audio_player

Audio, Logical sensor example and other multiple samples can not be selected at the same time. If you select more than one, a compile error will appear.
Memory and Message Utility Configurations and Layout

To set the inter-task communication library (Message Library) and the memory management library (Memory Manager) as follow:

Message Library Configuration

It is necessary to define the MessageQueue which is necessary when using the AudioPlayer function. Definition is done in the MessageQueueLayout definition file, and you can generate a header file to be included in the code with the tool.

In the example of Audio Player, do as follow:

$>cd examples/audio_player/config
$>python3 msgq_layout.conf 0x000fe000 0x1000 msgq_id.h msgq_pool.h

The description contents of the MessageQueueLayout definition file (msgq_layout.conf) are as follow:

MsgQuePool
 # ID,                        n_size  n_num  h_size  h_nums
  ["MSGQ_AUD_MNG",            88,      4,    0,      0],
  ["MSGQ_AUD_APP",            40,      2,    0,      0],
  ["MSGQ_AUD_DSP",            20,      5,    0,      0],
  ["MSGQ_AUD_PFDSP0",         20,      5,    0,      0],
  ["MSGQ_AUD_PFDSP1",         20,      5,    0,      0],
  ["MSGQ_AUD_PLY",            48,      5,    0,      0],
  ["MSGQ_AUD_SFX",            48,      5,    0,      0],
  ["MSGQ_AUD_OUTPUT_MIX",     48,      8,    0,      0],
  ["MSGQ_AUD_RND_PLY",        32,     16,    0,      0],
  ["MSGQ_AUD_RND_PLY_SYNC",   16,      8,    0,      0],
  ["MSGQ_AUD_RND_SFX",        32,     16,    0,      0],
  ["MSGQ_AUD_RND_SFX_SYNC",   16,      8,    0,      0],

The explanation of each parameter is as follow:

Parameter Description

ID

Specify the name of the message queue pool ID as a character string beginning with "MSGQ_".

n_size

Number of bytes of each element of the normal priority queue (8 or more and 512 or less). Specify fixed header length (8 bytes) + parameter length as a multiple of 4.

n_num

Number of elements of normal priority queue (1 or more and 16384 or less).

h_size

Number of bytes (0 or 8 to 512 inclusive) for each element of the high priority queue. Specify 0 when not in use.

h_num

Number of elements in the high priority queue (0 or 1 to 16384 or less). Specify 0 when not in use.

For each ID, see Audio Player Message ID of Audio Player Functions.

Since n_size is the optimum value, please do not change.

There is no need to change n_num, but when using the AudioPlayer function with other Application, it may be necessary to increase the value considering the load.

Use h_size and h_nums when you want to process the AudioPlayer function preferentially.

Refer to examples/audio_player/config/msgq_layout.conf for details on each definition.
      If the settings change, please use the tool to generate a new header file.
Memory Manager (Intelligent Fix Pool) Configuration

It is necessary to define the MemoryLayout (pool) which is necessary when using the AudioPlayer function.
Definition is done in the MemoaryLayout definition file, and it is possible to generate a header file to be included in the code with the tool.

In the example of Audio Player, do as follow:

$>cd examples/audio_player/config
$>python3 mem_layout.conf mem_layout.h fixed_fence.h pool_layout.h

The contents of the MemoaryLayout definition file (mem_layout.conf) are as follow:

FixedAreas
FixedAreas
 # name,                  device,     align,        size,         fence
  ["AUDIO_WORK_AREA",     "AUD_SRAM", U_STD_ALIGN,  0x0003e000,   False], # Audio work area
  ["MSG_QUE_AREA",        "AUD_SRAM", U_STD_ALIGN,  0x00001000,   False], # message queue area
  ["MEMMGR_WORK_AREA",    "AUD_SRAM", U_STD_ALIGN,  0x00000200,   False], # MemMgrLite WORK Area
  ["MEMMGR_DATA_AREA",    "AUD_SRAM", U_STD_ALIGN,  0x00000100,   False], # MemMgrLite DATA Area

The explanation of each parameter is as follow:

Parameter Description

name

area name (name starting with uppercase letters and ending with "_AREA", uppercase letters, numbers, _ can be used)

device

Device name of MemoryDevices to reserve space

align

Start alignment of the region. Specify a multiple of MinAlign (= 4) except 0 size

size of the region. Specify a value of a multiple of 4 except 0

fence

The purpose of each name is as follow:

AUDIO_WORK_AREA

AudioSubSystem

MSG_QUE_AREA

MessageQueue (fixed name). Do not exceed the size of (MSGQ_END_DRM - MSGQ_TOP_DRAM) of msgq_id.h.

MEMMGR_WORK_AREA

Work area used by Memory Manager (fixed name, fixed size)

MEMMGR_DATA_AREA

Data area used by Memory Manager (fixed name, fixed size)

Make sure that the total size of each name does not exceed the size of the shared memory secured by mpshm_init(), mpshm_remap().

Fixed Areas can not be customized
PoolAreas
PoolAreas
 # name,                    area,              align,        pool-size,                    seg,                        fence
  ["DEC_ES_MAIN_BUF_POOL",  "AUDIO_WORK_AREA", U_STD_ALIGN,  U_DEC_ES_MAIN_BUF_POOL_SIZE,  U_DEC_ES_MAIN_BUF_SEG_NUM,  True ],
  ["REND_PCM_BUF_POOL",     "AUDIO_WORK_AREA", U_STD_ALIGN,  U_REND_PCM_BUF_POOL_SIZE,     U_REND_PCM_BUF_SEG_NUM,     True ],
  ["DEC_APU_CMD_POOL",      "AUDIO_WORK_AREA", U_STD_ALIGN,  U_DEC_APU_CMD_POOL_SIZE,      U_DEC_APU_CMD_SEG_NUM,      True ],
  ["SRC_WORK_BUF_POOL",     "AUDIO_WORK_AREA", U_STD_ALIGN,  U_SRC_WORK_BUF_POOL_SIZE,     U_SRC_WORK_BUF_SEG_NUM,     True ],
  ["PF0_PCM_BUF_POOL",      "AUDIO_WORK_AREA", U_STD_ALIGN,  U_POF_PCM_BUF_SIZE,           U_POF_PCM_BUF_SEG_NUM,      True ],
  ["PF1_PCM_BUF_POOL",      "AUDIO_WORK_AREA", U_STD_ALIGN,  U_POF_PCM_BUF_SIZE,           U_POF_PCM_BUF_SEG_NUM,      True ],
  ["PF0_APU_CMD_POOL",      "AUDIO_WORK_AREA", U_STD_ALIGN,  U_DEC_APU_CMD_POOL_SIZE,      U_DEC_APU_CMD_SEG_NUM,      True ],
  ["PF1_APU_CMD_POOL",      "AUDIO_WORK_AREA", U_STD_ALIGN,  U_DEC_APU_CMD_POOL_SIZE,      U_DEC_APU_CMD_SEG_NUM,      True ],

The explanation of each parameter is as follow:

Parameter Description

name

pool name (name starting with uppercase letters and ending with "_POOL", upper case letters, numbers, _ can be used)

area

Area name of FixedArea to be used as pool area. The area must be located in the RAM

align

Starting alignment of the pool. Specify a multiple of MinAlign (= 4) except 0

pool - size

size of the pool. A value of a multiple of 4 except 0. In the Basic pool, segment size * number of segments In the final area of ​​each area, RemainderSize indicating the remaining size can be specified.

seg

Number of segments. Specify a value between 1 and 255

fence

Specify whether the fence is valid or invalid. This item is ignored when UseFence is False

The purpose of each name is as follow:

DEC_ES_MAIN_BUF_POOL

Buffer area for storing input data for player 0

REND_PCM_BUF_POOL

Buffer area for output of decoded data for player 0

DEC_APU_CMD_POOL

Command area for DSP (Decoder)

SRC_WORK_BUF_POOL

Work buffer area for DSP(SamplingRateConverter)

PF0_PCM_BUF_POOL

Buffer area for PostFilter 0

PF1_PCM_BUF_POOL

Buffer area forPostFilter1

PF0_APU_CMD_POOL

Command area for PostFilter 0

PF1_APU_CMD_POOL

Command area for PostFilter1

Refer to examples/audio_player/config/mem_layout.conf for details of each definition.
      If the setting changes, please use the tool to generate a new header file.
Table 18. Only support the following format
sampling rate PCM bit length channel number CPU frequency lock

mp3

16kHz / 32kHz 44.1kHz / 48kHz

16bit

1ch / 2ch

High voltage

wav (Low Power)

16kHz / 32kHz 44.1kHz / 48kHz

16bit

1ch/2ch

Low voltage

wav

48kHz / 88.4kHz / 96kHz / 176.4kHz / 196kHz

16bit/24bit

1ch/2ch

High voltage

DSP and Sound files
  • Preparation

    Music file

    To run the audio player application, you need a music file. Create an AUDIO/ directory on the root directory of the SD card and copy music files to this directory.

    Playlist

    To play music files, you need a playlist. This is a file that contains track databases in CSV data format. You can add new tracks to this file or delete tracks from this file. Create a PLAYLIST/ directory on the root directory of the SD card and copy the TRACK_DB.CSV files to this directory.

    The track database has this format: [filename],[artist],[album],[channel number],[bit length],[sampling rate],[file format]

   ABC.mp3,artist1,album1,2,16,44100,mp3
The example is a premise to use Playlist. Only the first line of the list is played.
Using SPI-Flash instead of SD Card

If you want to use SPI-Flash for the file system, please change the path specification of playback file, Playlist, DSP binary image of example to "/mnt/spif/*". Also copy the playback file, Playlist, DSP binary image to the modified path.

How to execute

Start the player application from NuttShell.

nsh> player

player The application starts and the following log is displayed.

Start Audio Player example

Playback of the first file of PlayList starts.

If the SD card can not be recognized, the following error log is displayed. Check the status of sd card.

Error: /mnt/sd0/AUDIO directory path error. check the path!
Error: app_open_contents_dir() failure.
Exit AudioPlayer example

If PlayList can not be recognized, the following error log is displayed. Please check that the path of PlayList is correct.

Track db(playlist) /mnt/sd0/PLAYLIST/TRACK_DB.CSV open error. check paths and files!
/mnt/sd0/PLAYLIST/alias_list_alltrack.bin cannot opened.

If PlayFile can not be recognized, the following error log is displayed. Please check whether path has File or whether the PlayList matches the File name.

Error: /mnt/sd0/AUDIO/***.mp3 open error. check paths and files!
Error: app_start_player() failure.

If you do not set the pool area(SRC_WORK_BUF_POOL) of work buffer for sampling rate converter, the following warning log is displayed. Heap space is used instead of pool area and fragmentation may occur. Please Set SRC_WORK_BUF_POOL with AS_CreatePlayerMulti.

Attention: module[5] attention id[1]/code[33] (objects/media_player/media_player_obj.cpp L****)

After 10 seconds playback, the Player application will end.

Exit AudioPlayer example
Audio Recorder functions

A simple data flow of Audio Recorder is shown below.

Audio Recorder Dataflow
Figure 23. Audio Recorder Dataflow.

When the Audio SubSystem operates with RecorderMode, the User Application needs to prepare a FIFO for storing audio data. When recording audio data starts, audio data accumulates in this FIFO after a certain period of operation. This audio data is encoded in the specified compression format and should be read out from the FIFO appropriately so that it does not overflow from the FIFO so that continuous audio data can be acquired.

Although Recorder can capture two inputs from the HW, at the present time, it does not support the function to generate two instances and record two lines.

The User Application implements the Recorder application by processing this voice according to the requirements of each system (for example, exporting it to Storage, sending it to the Connectivity module, cloud processing, etc.).

The inside of the data flow communicates with Messages. Each message communication has an ID for each client. In the case of Audio Recorder, based on the example Layout in the example, the ID will be as follows.

User Application           : MSGQ_AUD_APP
Audio Manager              : MSGQ_AUD_MNG
Audio Frontend             : MSGQ_AUD_FRONTEND
Audio Recorder             : MSGQ_AUD_RECORDER
Audio Capture Component    : MSGQ_AUD_CAP
Audio DSP                  : MSGQ_AUD_DSP

※MSGQ_AUD_CAP_SYNC will be deleted.

Audio Recorder Message ID
Figure 24. Audio Recorder Message ID

In addition, the data area of ​​each data is as follow.

PCM (Input) Data Buffer    : INPUT_BUF_POOL
ES Data Buffer (for DSP)   : ES_BUF_POOL
PreProcess DSP command     : PRE_APU_CMD_POOL
Audio Encoder DSP Command  : ENC_APU_CMD_POOL
Audio Recorder Pool ID
Figure 25. Audio Recorder Pool ID

These IDs must be specified when generating.

How to use
Preparation

"AudioManager", "MicFrontendObject", "MediaRecorderObject", "CaptureComponent" are software components designed to control audio subsystems and implement an Audio Recorder.

Create AudioManager

To enable AudioManager you need to call AS_CreateAudioManager(AudioSubSystemIDs).
In the AudioSubSystemIDs structure, specify the MsgQueID defined in the Message Library Configuration. Specify 0xFF for an unused object.
AudioAttentionCb, specify a callback function for asynchronous notification. If NULL is specified, no notification is made.

static void attention_callback(const ErrorAttentionParam *attparam)
{
  ...
}

AudioSubSystemIDs ids;

ids.app         = MSGQ_AUD_APP;
ids.mng         = MSGQ_AUD_MNG;
ids.player_main = 0xFF;
ids.player_sub  = 0xFF;
ids.micfrontend = MSGQ_AUD_FRONTEND
ids.mixer       = 0xFF;
ids.recorder    = MSGQ_AUD_RECORDER;
ids.effector    = 0xFF;
ids.recognizer  = 0xFF;

AS_CreateAudioManager(ids, attention_callback);
Create MicFrontendObject

In order to enable MicFrontendObject you need to call AS_CreateMicFrontend(AsCreateMicFrontendParam_m). You must specify MsgQueID defined in Message Library Configuration and PoolId defined in Memory Manager Configuration. The "dsp" in MsgQueID, and "output" "dspcmd" in PoolId are effective only when preprocess will works. If not, set NULL_POOL is allowed. AudioAttentionCb, specify a callback function for asynchronous notification. Please register when you are not using AudioManager. If you register a callback function with AS_CreateAudioManager or if you specify NULL in AS_CreateMicFrontend, notification is not sent.

static void attention_callback_from_frontend(const ErrorAttentionParam *attparam):qa
{
  ...
}

AsCreateMicFrontendParam_t cparam;

cparam.msgq_id.micfrontend = MSGQ_AUD_FRONTEND;
cparam.msgq_id.mng         = MSGQ_AUD_MNG;
cparam.msgq_id.dsp         = MSGQ_AUD_PREDSP;
cparam.pool_id.capin       = INPUT_BUF_POOL;
cparam.pool_id.output      = PREPROC_BUF_POOL; /* When you don't use preprocess, set to NULL_POOL */
cparam.pool_id.dspcmd      = PRE_APU_CMD_POOL; /* When you don't use preprocess, set to NULL_POOL */

result = AS_CreateMicFrontend(&cparam, attention_callback_from_frontend);
Create MediaRecorderObject

In order to enable MediaRecorderObject you need to call AS_CreateMediaRecorder (AsActRecorderParam_t). You must specify MsgQueID defined in Message Library Configuration and PoolId defined in Memory Manager Configuration.
AudioAttentionCb, specify a callback function for asynchronous notification. Please register when you are not using AudioManager. If you register a callback function with AS_CreateAudioManager or if you specify NULL in AS_CreateMediaRecorder, notification is not sent.

static void attention_callback_from_recorder(const ErrorAttentionParam *attparam)
{
  ...
}

AsActRecorderParam_t cparam;

cparam.msgq_id.recorder      = MSGQ_AUD_RECORDER;
cparam.msgq_id.mng           = MSGQ_AUD_MNG;
cparam.msgq_id.dsp           = MSGQ_AUD_DSP;
cparam.pool_id.input         = INPUT_BUF_POOL;
cparam.pool_id.output        = ES_BUF_POOL;
cparam.pool_id.dsp           = ENC_APU_CMD_POOL;

result = AS_CreateMediaRecorder(&cparam, attention_callback_from_recorder);
Activate CaptureComponent

In order to enable CaptureComponent, you need to call AS_CreateCapture(). You must specify MsgQueID defined in Message Library Configuration and PoolId defined in Memory Manager Configuration. If your application does not use the second channel, you need to set the third and fourth arguments to 0xFF .

AsActCaptureParam_t capture_act_param;

cparam.msgq_id.dev0_req  = MSGQ_AUD_CAP;
cparam.msgq_id.dev0_sync = MSGQ_AUD_CAP_SYNC;
cparam.msgq_id.dev1_req  = 0xFF;
cparam.msgq_id.dev1_sync = 0xFF;

result = AS_CreateCapture(&cparam);
Initialize and Change state

When the necessary objects have been generated, initialization processing such as audio HW setting, power on, change of operation mode, etc. is performed in order to perform Recorder operation.

It can be realized by issuing the following commands in order.

Power on Audio SubSystem

To turn on the audio block, issue the AUDCMD_POWERON, PowerOnParam command to turn on the power and change the state of the Audio Sub system to the Ready state.

enable_sound_effect

The enable_sound_effect is fixed to AS_DISABLE_SOUNDEFFECT.

AS_DISABLE_SOUNDEFFECT::SoundEffect invalid
Command setting example
AudioCommand command;
command.header.packet_length = LENGTH_POWERON;
command.header.command_code  = AUDCMD_POWERON;
command.header.sub_code      = 0x00;
command.power_on_param.enable_sound_effect = AS_DISABLE_SOUNDEFFECT;
AS_SendAudioCommand(&command);
Initialize Mic Gain

AUDCMD_INITMICGAIN Set Mic’s Gain.

init_mic_gain_param[]

In the case of an analog microphone, the value obtained by multiplying the dB value by 10. It can be set in the range of 0 (0.0 dB) to 210 (21.0 dB) in multiples of 5. The default value is 0.0 dB.  + In the case of a digital microphone, the value obtained by multiplying the dB value by 100. It can be set within the range of -7850 (-78.50 dB) to 0 (0.00 dB). The default value is -78.50 dB.
If you do not want to change the value of Gain, please specify 'AS_MICGAIN_HOLD'.

Command setting example
AudioCommand command;
command->header.packet_length = LENGTH_INITMICGAIN;
command->header.command_code  = AUDCMD_INITMICGAIN;
command->header.sub_code      = 0;
command->init_mic_gain_param.mic_gain[0] = 210;
command->init_mic_gain_param.mic_gain[1] = 210;
command->init_mic_gain_param.mic_gain[2] = 210;
command->init_mic_gain_param.mic_gain[3] = 210;
command->init_mic_gain_param.mic_gain[4] = AS_MICGAIN_HOLD;
command->init_mic_gain_param.mic_gain[5] = AS_MICGAIN_HOLD;
command->init_mic_gain_param.mic_gain[6] = AS_MICGAIN_HOLD;
command->init_mic_gain_param.mic_gain[7] = AS_MICGAIN_HOLD;
AS_SendAudioCommand(&command);
Refer to "examples/audio_recorder/config/msgq_layout.conf" for the definition of details.

Each element of mic_gain[] corresponds to the ID of the microphone. The ID of the microphone is set by the value of "MIC channel select map" of Config. The default setting is the analog mic 1/2/3/4.

Below is the information of configuration.

Setting of "MIC channel select map"
[CXD56xx Configuration]
  [Audio]
    [Audio baseband config settings]
      [CXD5247 settings]
        (0xFFFF4321) MIC channel select map

The value of "MIC channel select map" indicates the ID of the MIC every 4 bits. The relationship between the element of mic_gain and the bit field of "MIC channel select map" is as follow.

element of mic_gain [7] [6] [5] [4] [3] [2] [1] [0]

bit field

31 - 28

27 - 24

23 - 20

19 - 16

15 - 12

11 - 8

7 - 4

3 - 0

The relationship between the value (ID) of "MIC channel select map" and the type of microphone is as follow.

HEX value (ID) Microphone type

0x1

CXD5247 Analog microphone 1

0x2

CXD5247 Analog microphone 2

0x3

CXD5247 Analog microphone 3

0x4

CXD5247 Analog microphone 4

0x5

CXD5247 Digital microphone 1

0x6

CXD5247 Digital microphone 2

0x7

CXD5247 Digital microphone 3

0x8

CXD5247 Digital microphone 4

0x9

CXD5247 Digital microphone 5

0xA

CXD5247 Digital microphone 6

0xB

CXD5247 Digital microphone 7

0xC

CXD5247 Digital microphone 8

Please set the microphone to use from element 0 in order. It is not possible to skip element numbers. Mixing of analog mic and digital microphone is not supported. To set the analog microphone please set element 0-3. If the element is an even number, it is the L channel, and if the element is odd, it becomes the R channel.

Set Preprocess Type

Set Preprocess type by AUDCMD_SETMFETYPE command.
By this function, you can apply user customizable signal processing to recording audio data.

If you don’t call AUDCMD_SETMFETYPE command, Preprocess will be through.

Following diagram indicates the position of signal processing by your original DSP.

User customizable signal processing will works on highlighted Customproc and UserCustmDSP.
You can write signal process code which works on the UserCustomDSP.

Audio recorder preprocess
Figure 26. Audio Recorder Preprocess

This command requires following parameter.

preproc_type
AsMicFrontendPreProcThrough    : Preprocess through
AsMicFrontendPreProcUserCustom : Preprocess will be user original signal process
Command setting example
AudioCommand command;
command.header.packet_length = LENGTH_SETMFETYPE;
command.header.command_code  = AUDCMD_SETMFETYPE;
command.header.sub_code      = 0x00;
command.set_mfetype_param.preproc_type = AsMicFrontendPreProcUserCustom;
AS_SendAudioCommand(&command);
Change to Recorder Status

AUDCMD_SETRECORDERSTATUS Transitions the state of AudioSubSystem to Recorder state.

input_device

Specify the input device to be recorded. It is necessary to match with the microphone type set with AUDCMD_INITMICGAIN.

AS_SETRECDR_STS_INPUTDEVICE_MIC_A  : CXD5247: analog microphone
AS_SETRECDR_STS_INPUTDEVICE_MIC_D  : CXD5247: digital microphone
AS_SETRECDR_STS_INPUTDEVICE_I2S_IN : I2S: I2S input
input_device_handler

It is fixed at 0 at this time.

output_device

Specify the output destination device of the encoded Elementary stream (ES) data.
At the moment only RAM device output is supported.

AS_SETRECDR_STS_OUTPUTDEVICE_RAM : Output to RAM device
output_device_handler

Specify the handler of Simple FIFO where output (Encoded ES data) is stored.
simple_fifo_handler is obtained with CMN_SimpleFifoInitialize().

Command setting example
AudioCommand command;
command.header.packet_length = LENGTH_SET_RECORDER_STATUS;
command.header.command_code  = AUDCMD_SETRECORDERSTATUS;
command.header.sub_code      = 0x00;
command.set_recorder_status_param.input_device          = AS_SETRECDR_STS_INPUTDEVICE_MIC_A;
command.set_recorder_status_param.input_device_handler  = 0x00;
command.set_recorder_status_param.output_device         = AS_SETRECDR_STS_OUTPUTDEVICE_RAM;
command.set_recorder_status_param.output_device_handler = &s_recorder_info.fifo.output_device;
AS_SendAudioCommand(&command);
Start Recording

The audio recording start sequence.

Init recorder information

AUDCMD_INITREC, RecorderCommand, AsInitRecorderParam to set the recording operation.

sampling_rate
AS_SAMPLINGRATE_8000  : 8kHz
AS_SAMPLINGRATE_16000 : 16kHz
AS_SAMPLINGRATE_48000 : 48kHz
channel_number
AS_CHANNEL_MONO   : Monoral
AS_CHANNEL_STEREO : Stereo
AS_CHANNEL_4CH    : 4ch
AS_CHANNEL_6CH    : 6ch
AS_CHANNEL_8CH    : 8ch
bit_length
AS_BITLENGTH_16 : 16bit
AS_BITLENGTH_24 : 24bit
codec_type
AS_CODECTYPE_MP3  : MP3
AS_CODECTYPE_LPCM : LinearPCM
bitrate

Valid only when MP3 encoding

AS_BITRATE_8000   : 8000
AS_BITRATE_16000  : 16000
AS_BITRATE_24000  : 24000
AS_BITRATE_32000  : 32000
AS_BITRATE_40000  : 40000
AS_BITRATE_48000  : 48000
AS_BITRATE_56000  : 56000
AS_BITRATE_64000  : 64000
AS_BITRATE_80000  : 80000
AS_BITRATE_96000  : 96000
AS_BITRATE_112000 : 112000
AS_BITRATE_128000 : 128000
AS_BITRATE_144000 : 144000
AS_BITRATE_160000 : 160000
AS_BITRATE_192000 : 192000
AS_BITRATE_224000 : 224000
AS_BITRATE_256000 : 256000
AS_BITRATE_320000 : 320000
dsp_path

Specify the absolute path that stores the Decoder’s DSP image. Maximum of 24 characters.

The combination of input device and number of channels is limited.
Input number of channels

Mic

1 ch (Monoral), 2 ch (Stereo), 4 ch (* 1), 6 ch (* 2), 8 ch (* 2)

I2S

2 ch (Stereo)

  • (* 1. LPCM only)

  • (* 2. Only when using LPCM and DigitalMic)

The combination of Codec, bit length, sampling frequency, and bit rate is limited.
Codec bit length sampling frequency bit rate

MP3

16 bit

16 kHz

8000 (*1), 16000 - 160000                      

 

48 kHz

32000 - 320000

LPCM

16 bit

16 kHz, 48 kHz

-            

24 bit (*2)

16 kHz, 48 kHz, 192 kHz (*2)

-

  • (*1. when 1ch is specified)

  • (*2. Designated HiResolution mode)

Command setting example
AudioCommand command;
command.header.packet_length = LENGTH_INIT_RECORDER;
command.header.command_code  = AUDCMD_INITREC;
command.header.sub_code      = 0x00;
command.recorder.init_param.sampling_rate  = s_recorder_info.file.sampling_rate;
command.recorder.init_param.channel_number = s_recorder_info.file.channel_number;
command.recorder.init_param.bit_length     = AS_BITLENGTH_16;
command.recorder.init_param.codec_type     = AS_CODECTYPE_OPUS;
command.recorder.init_param.bitrate        = AS_BITRATE_8000;
command.recorder.init_param.computational_complexity = AS_INITREC_COMPLEXITY_0
command.recorder.init_param.dsp_path       = "/mnt/sd0/BIN";
diag 84dcd546cab2122e96ffa509711d1d6f
Figure 27. Recorder initialization sequence diagram
Start Recorder

AUDCMD_STARTREC, RecorderCommand Recording starts.

Once you start recording, the Audio System writes the ES data to the FIFO. In order to correctly record audio data, it is necessary to read the written data before the FIFO overflows. Data writing will be notified, so please read it in response to this event as appropriate.

For details, please refer to "examples/audio_recorder/config/msgq_layout.conf".
When the FIFO is Full, the Audio System can not write, and discards audio data that can not be written. Therefore, if this occurs during recording the audio data will be discontinuous audio.
Command setting example
AudioCommand command;
command.header.packet_length = LENGTH_START_RECORDER;
command.header.command_code  = AUDCMD_STARTREC;
command.header.sub_code      = 0x00;
AS_SendAudioCommand(&command)
diag f68e7e55de14066644e8f556d64c9a26
Figure 28. Recorder sequence
Stop Rec
Stop Recorder

AUDCMD_STOPREC, RecorderCommand Stop recording

Recorder encodes up to the audio data that was captured when receiving the stop instruction.

Command setting example
AudioCommand command;
command.header.packet_length = LENGTH_STOP_RECORDER;
command.header.command_code  = AUDCMD_STOPREC;
command.header.sub_code      = 0x00;
AS_SendAudioCommand(&command)
diag eec82697477342d6fdecc763d5ef27a3
Figure 29. Stop recording sequence
Build Configurations

To use AudioRecorder’s features

$>cd sdk/
$>tools/config.py -m

You need to open the Config menu and set the following Config.

Select options in below:

:(Select audio recorder)
[CXD56xx Configuration]
  [SDIO SD Card]                 <= Y (If using the SD card)
  [Audio]                        <= Y
    [Audio baseband config settings]
      [CXD5247 settings]
        [X'tal frequency of the CXD5247] <= 49.152MHz (If DVT board)

[SDK audio]                      <= Y
  [Audio Utilities]
    [Audio Recorder]             <= Y

[Memory Manager]                 <= Y
  [Memory Utilities]             <= Y

[ASMP]                           <= Y
Error Attentions and Approach

This section shows a list of audio recording error attentions and suggested action to remove the error. See Error Information of Audio SubSystem for more details.

ID Attention Code Attention Level Approach

0x06

AS_ATTENTION_SUB_CODE_SIMPLE_FIFO_OVERFLOW

WARNING

This is because AudioSubSystem could not write recording data to SimpleFIFO. Increase the CPU priority of tasks that read recording data to Simple FIFO.

0x0D

AS_ATTENTION_SUB_CODE_MEMHANDLE_ALLOC_ERROR

ERROR

This is due to insufficient number of segments in the data area. Decrease the priority of tasks other than AudioSubSystem or increase the number of segments in the data area.

0x0F

AS_ATTENTION_SUB_CODE_TASK_CREATE_ERROR

ERROR

This is due to insufficient heap space. Expand the heap area.

0x18

AS_ATTENTION_SUB_CODE_DSP_VERSION_ERROR

ERROR

Due to the different version of the DSP binary. Please update the DSP binary image with the file "sdk/modules/audio/dsp".

DSP install
DSP binary image install

Store the DSP binary image in the path set in Kconfig. The binary image is in sdk/modules/audio/dsp .

Table 19. Binary image required for audio recorder according to configuration
Image size

MP3ENC

111 kbyte

SRC (Sampling Rate Converter)

21 kbyte

Although LPCM does not require compression processing, loading of the DSP of Sampling Rate Converter (SRC) is necessary because frequency conversion processing is necessary.
Audio Recorder Example

There is a simple Recorder Example, and you can use to check the operation of Recorder.

Preparation
Build Configuration (kconfig)

To use the sample program of Audio Recorder, please make the following settings.

Read config of audio_recorder.

$>cd sdk/
$>tools/config.py examples/audio_recorder

Make sure Audio recorder is enabled.

$>tools/config.py -m

(audio recorder:)
[Examples]
  [Audio recorder example] <= Y
Audio & Logical sensor example and other multiple samples can not be selected at the same time. If you select more than one, a compile error will appear.
Only support the following format:

For details, see the Start Rec item.

Memory Utility Configurations and Layout
Message Library Configuration

It is necessary to define the MessageQueue that is required when using the AudioRecorder function. The definition is done in the MessageQueueLayout definition file, and you can generate a header file to be included in the code with the tool.

example of audio recorder is performed as follows.

$>cd examples/audio_recorder/config
$>python3 msgq_layout.conf 0x000fe000 0x1000 msgq_id.h msgq_pool.h

The description contents of the MessageQueueLayout definition file (msgq_layout.conf) are as follows.

MsgQuePool
 # ID,                           n_size  n_num  h_size  h_num
  ["MSGQ_AUD_MNG",               88,     3,     0,      0],
  ["MSGQ_AUD_APP",               40,     2,     0,      0],
  ["MSGQ_AUD_DSP",               20,     5,     0,      0],
  ["MSGQ_AUD_RECORDER",          48,     5,     0,      0],
  ["MSGQ_AUD_CAP",               24,    16,     0,      0],
  ["MSGQ_AUD_CAP_SYNC",          16,     8,     0,      0],

The explanation of each parameter is as follow.

Parameter Description

ID

The name of the message queue ID is specified by a character string beginning with "MSGQ_".

n_size

Number of bytes of each element of the normal priority queue (8 or more and 512 or less). Specify fixed header length (8 bytes) + parameter length as a multiple of 4.

n_num

Number of elements of normal priority queue (1 or more and 16384 or less).

h_size

Number of bytes (0 or 8 to 512 inclusive) for each element of the high priority queue. Specify 0 when not in use.

h_num

Number of elements in the high priority queue (0 or 1 to 16384 or less). Specify 0 when not in use.

Usage of each ID is as follow.

MSGQ_AUD_MNG

Used to receive commands of Audio Manager

MSGQ_AUD_APP

Application to receive command response

MSGQ_AUD_DS

Used for receiving responses from DSP (Decoder)

MSGQ_AUD_RECORDER

Used to receive MediaRecorderObject command

MSGQ_AUD_CAP

Used to receive CaptureComponent commands

MSGQ_AUD_CAP_SYNC

Used for internal synchronization processing of CaptureComponent

Refer to examples/audio_recorder/config/msgq_layout.conf for details on each definition.
      If the settings change, please use the tool to generate a new header file.
Memory Manager (Intelligent Fix Pool) Configuration

It is necessary to define the MemoryLayout(pool) when using the AudioRecorder function.
Definition is done in the MemoaryLayout definition file, and it is possible to generate a header file to be included in the code with the tool.

example of audio recorder is performed as follows.

$>cd examples/audio_recorder/config
$>python3 mem_layout.conf mem_layout.h fixed_fence.h pool_layout.h

The contents of the Memory Layout definition file (mem_layout.conf) are as follow.

FixedAreas
FixedAreas
 # name,                  device,     align,        size,         fence
  ["AUDIO_WORK_AREA",     "AUD_SRAM", U_TILE_ALIGN, 0x0003c000,   False],
  ["MSG_QUE_AREA",        "AUD_SRAM", U_MSGQ_ALIGN, 0x00003140,   False],
  ["MEMMGR_WORK_AREA",    "AUD_SRAM", U_STD_ALIGN,  0x00000200,   False],
  ["MEMMGR_DATA_AREA",    "AUD_SRAM", U_STD_ALIGN,  0x00000100,   False],

The explanation of each parameter is as follow.

Parameter Description

name

area name (name starting with uppercase letters and ending with "_AREA", uppercase letters, numbers, _ can be used)

device

Device name of MemoryDevices to reserve space

align

Start alignment of the region. Specify a multiple of MinAlign(= 4) except 0

size

Size of the region. Specify a value of a multiple of 4 except 0

fence

Specify whether fence is enabled or disabled (This item is ignored when UseFence is False)

The purpose of each name is as follow.

AUDIO_WORK_AREA

Used by AudioSubSystem

MSG_QUE_AREA

Used by MessageQueue (fixed name). Do not exceed the size of (MSGQ_END_DRM - MSGQ_TOP_DRAM) of msgq_id.h.

MEMMGR_WORK_AREA

Work area used by Memory Manager (fixed name, fixed size)

MEMMGR_DATA_AREA

Data area used by Memory Manager (fixed name, fixed size)

Make sure that the total size of each name does not exceed the size of the shared memory secured by mpshm_init(), mpshm_remap().

Fixed Areas can not be customized
PoolAreas
PoolAreas
 # name,                    area,              align,        pool-size,  seg, fence
  ["ES_BUF_POOL",           "AUDIO_WORK_AREA", U_STD_ALIGN,  0x00008700, 5,   True],
  ["INPUT_BUF_POOL",        "AUDIO_WORK_AREA", U_STD_ALIGN,  0x00003000, 2,   True],
  ["ENC_APU_CMD_POOL",      "AUDIO_WORK_AREA", U_STD_ALIGN,  0x000000FC, 3,   True],
  ["SRC_APU_CMD_POOL",      "AUDIO_WORK_AREA", U_STD_ALIGN,  0x000000FC, 3,   True],

The explanation of each parameter is as follow.

Parameter Description

name

pool name (name starting with uppercase letters and ending with "_POOL", upper case letters, numbers, _ can be used)

area

Area name of FixedArea to be used as pool area. The area must be located in the RAM

align

Starting alignment of the pool. Specify a multiple of MinAlign(= 4) except 0

pool - size

size of the pool. A value of a multiple of 4 except 0. In the Basic pool, segment size * number of segments

seg

Number of segments. Specify a value between 1 and 255

fence

Specify whether the fence is valid or invalid. This item is ignored when UseFence is False

The purpose of each name is as follow.

ES_BUF_POOL

Storage buffer area of ​​the result of encoding input speech

INPUT_BUF_POOL

Input storage buffer area for audio data to be recorded

ENC_APU_CMD_POOL

Encoder Communication with DSP command buffer area

SRC_APU_CMD_POOL

SRC Communication with DSP Command buffer area communication

Refer to examples/audio_recorder/config/mem_layout.conf for details on each definition.
      If the setting changes, please use the tool to generate a new header file.
How to execute

Start the recorder application from NuttShell.

nsh> recorder

The Audio Recorder application starts and recording starts.
The recording file is "/mnt/sd0/REC/YYMMDD_HHMMSS.mp3".

Start AudioRecorder example

After recording for 10 seconds, the Audio Recorder application will terminate.

Exit Audio Recorder example
Audio Through Functions

The simple data flow of Audio Through is shown below:

Audio Through Dataflow
Figure 30. Audio Through Dataflow

When Audio SubSystem operates in Through Mode, the User Application can set up data flow without CPU intervention.

I2S or MIC can be specified as the data input source. Speaker or I2S can be specified as the data output destination.

In addition, User Application can set two data flows. By using MIXER, you can mix two input data into one.

For setting to User Application, communicate command with Message. Message communication has an ID for each client. In the case of Audio Through, the ID is as follow:

User Application           : MSGQ_AUD_APP
Audio Manager              : MSGQ_AUD_MNG
Audio Through Message ID
Figure 31. Audio Through Message ID

These IDs must be specified.

Audio HW internal dataflow

In the Audio Through, inside the Audio HW is the data flow shown in Audio HW internal dataflow.
There are I2S In, MIC In, Mixer Out as input sources of data flow.
There are Mixer In1, Mixer In2, I2SOut as output destinations of the data flow. When Mixer In1 or Mixer In2 is set as the output destination, it outputs from MixerOut to Speaker Out.

diag 65e983773cddcc639db0941b79e4fdb8
Figure 32. Audio Through internal dataflow

The relationship between configurable input source and output destination is as follow:

Table 20. Audio Through dataflow combination
Input source Output destination

I2S In

Mixer In1, Mixer In2

MIC In

Mixer In1, Mixer In2, I2S Out

Mixer Out

I2S Out
Speaker Out (always output)

How to use
Preparation

"AudioManager" is a software component designed to control audio subsystem and support audio through.

Therefore, in order to support Audio Through, it is necessary to call the generation function of the following objects in advance.

Activate AudioManager

In order to enable AudioManager, you need to call AS_CreateAudioManager (AudioSubSystemIDs).
In AudioSubSystemIDs, you need to specify the MsgQueID defined in Message Library Configuration.
Items specified with 0xFF will not be used.
AudioAttentionCb, specify a callback function for asynchronous notification. If NULL is specified, no notification is made.

static void attention_callback(const ErrorAttentionParam *attparam)
{
  ...
}

AudioSubSystemIDs ids;

ids.app         = MSGQ_AUD_APP;
ids.mng         = MSGQ_AUD_MNG;
ids.player_main = 0xFF;
ids.player_sub  = 0xFF;
ids.mixer       = 0xFF;
ids.recorder    = 0xFF;
ids.effector    = 0xFF;
ids.recognizer  = 0xFF;

AS_CreateAudioManager(ids, attention_callback);
Initialize and Status change

When necessary objects are generated, initialization processing such as audio HW setting, power on, change of operation mode, etc. is performed in order to perform Audio Through operation.

This can be performed by issuing the following commands in the order below:

Power on Audio SubSystem

To turn on the audio block, issue the AUDCMD_POWERON, PowerOnParam command, also change the state of the Audio Sub system to the Ready state.
Enable_sound_effect must be fixed to AS_DISABLE_SOUNDEFFECT.

  AudioCommand command;
  command.header.packet_length = LENGTH_POWERON;
  command.header.command_code  = AUDCMD_POWERON;
  command.header.sub_code      = 0x00;
  command.power_on_param.enable_sound_effect = AS_DISABLE_SOUNDEFFECT;
  AS_SendAudioCommand(&command);
Initialize output device

After performing PowerOn and transitioning to the Ready state, use the AUDCMD_INITOUTPUTSELECT, InitOutputSelectParam command to select the output destination from Mixer.

The setting of output_device_sel is as follow:。

output_device_sel
   AS_OUT_OFF: Output OFF
   AS_OUT_SP:  Output from Speaker
   AS_OUT_I2S: Output from I2S
If you select AS_OUT_I2S to control the HW power supply, it will not use Speaker as output. If you want to use both I2S and Speaker with Audio Through, select AS_OUT_SP.
Command setting example
  AudioCommand command;
  command.header.packet_length = LENGTH_INITOUTPUTSELECT;
  command.header.command_code  = AUDCMD_INITOUTPUTSELECT;
  command.header.sub_code      = 0x00;
  command.init_output_select_param.output_device_sel = AS_OUT_SP;
  AS_SendAudioCommand(&command);
Change to Through Status

AUDCMD_SETTHROUGHSTATUS The state of AudioSubSystem is transited to the Through state by the command.

  1. Command setting example

  AudioCommand command;
  command.header.packet_length = LENGTH_SET_THROUGH_STATUS;
  command.header.command_code = AUDCMD_SETTHROUGHSTATUS;
  command.header.sub_code = 0x00;
  AS_SendAudioCommand(&command);
Set volume

When setting Speaker as output, you can set the volume with AUDCMD_SETVOLUME, SetVolumeParam. The setting of each parameter is as follow:

Volume can not be changed in I2S.

input1_db

Volume of MIXER 1. Set dB with a 10-fold integer value. The range can be set from -1020 (-102.0 dB) to 120 (+ 12.0 dB) with step size 5 (0.5 dB).

input2_db

Volume of MIXER 2. The range is the same as input1_db.

master_db

The volume after mix of MIXER 1 and MIXER 2. The range is the same as input1_db.

  AudioCommand command;
  command.header.packet_length = LENGTH_SETVOLUME;
  command.header.command_code  = AUDCMD_SETVOLUME;
  command.header.sub_code      = 0;
  command.set_volume_param.input1_db = 0;    /* 0.0dB */
  command.set_volume_param.input2_db = 0;    /* 0.0dB */
  command.set_volume_param.master_db = -200; /* -20.0dB */
  AS_SendAudioCommand(&command);
Initialize Mic Gain

Please refer to Recorder Init Mic Gain.

The microphone that can be used is a combination of CXD5247 analog microphone 1, CXD5247 analog microphone 2, CXD5247 digital microphone 1 or CXD5247 digital microphone 2 combination.
You can change the microphone by changing the Layout, the same as Recorder Init Mic Gain.
diag 346f1f3ab068a122193bd7ec36b1f684
Figure 33. Initial sequence
Start Audio Through

By setting the data flow path, data input / output is started.

Set Through Path

You can set two data paths at the same time with AUDCMD_SETTHROUGHPATH, AsSetThroughPathParam, AsThroughPath. The setting of each parameter of each data path is as follow:

en

Enable / disable the data path.

true: Enabled
false: Disable
in

Set the data input source.

AS_THROUGH_PATH_IN_MIC : MIC As input source
AS_THROUGH_PATH_IN_I2S1 : I2SA s input source
AS_THROUGH_PATH_IN_MIXER : Mixer Out As input source
MIC refers to CXD5247 analog microphone 1 and CXD5247 analog microphone 2 for analog microphone. In the case of a digital microphone, it refers to CXD5247 digital microphone 1 and CXD5247 digital microphone 2. I2S refers to I2S0.
out

Set the data output destination.

AS_THROUGH_PATH_OUT_MIXER1 : Mixer In1 As output destination
AS_THROUGH_PATH_OUT_MIXER2 : Mixer In2 As output destination
AS_THROUGH_PATH_OUT_I2S1 : I2S As output destination
I2S refers to I2S0
command example 1
  AudioCommand command;
  command.header.packet_length = LENGTH_SET_THROUGH_PATH;
  command.header.command_code  = AUDCMD_SETTHROUGHPATH;
  command.header.sub_code      = 0x00;
  command.set_through_path.path1.en  = true; (1)
  command.set_through_path.path1.in  = AS_THROUGH_PATH_IN_MIC; (2)
  command.set_through_path.path1.out = AS_THROUGH_PATH_OUT_I2S1; (3)
  command.set_through_path.path2.en  = true; (4)
  command.set_through_path.path2.in  = AS_THROUGH_PATH_IN_I2S1;   (5)
  command.set_through_path.path2.out = AS_THROUGH_PATH_OUT_MIXER1; (6)
  AS_SendAudioCommand(&command);
1 Activate the data path 1 setting
2 Data path 1 uses MIC In as the input source
3 Data path 1 uses I2S Out as the output destination
4 Activate the data path 2 setting
5 Data path 2 uses I2S In as the input source
6 Data path 2 uses Mixer In 1 as the output destination
diag cd6d142421022eb676e8d1fbc5aad9bf
Figure 34. command example 1 dataflow
command example 2
  AudioCommand command;
  command.header.packet_length = LENGTH_SET_THROUGH_PATH;
  command.header.command_code  = AUDCMD_SETTHROUGHPATH;
  command.header.sub_code      = 0x00;
  command.set_through_path.path1.en  = true; (1)
  command.set_through_path.path1.in  = AS_THROUGH_PATH_IN_MIC; (2)
  command.set_through_path.path1.out = AS_THROUGH_PATH_OUT_MIXER1; (3)
  command.set_through_path.path2.en  = true; (4)
  command.set_through_path.path2.in  = AS_THROUGH_PATH_IN_MIXER;  (5)
  command.set_through_path.path2.out = AS_THROUGH_PATH_OUT_I2S1; (6)
  AS_SendAudioCommand(&command);
1 Activate the data path 1 setting
2 Data path 1 uses MIC In as the input source
3 Data path 1 uses Mixer In 2 as the output destination
4 Activate the data path 2 setting
5 Data path 2 uses Mixer Out as the input source
6 Data path 2 uses I2S Out as the output destination
diag 9e31aab8ad24a88571c9d1f443a4db31
Figure 35. command example 2 dataflow
command example 3
  AudioCommand command;
  command.header.packet_length = LENGTH_SET_THROUGH_PATH;
  command.header.command_code  = AUDCMD_SETTHROUGHPATH;
  command.header.sub_code      = 0x00;
  command.set_through_path.path1.en  = true; (1)
  command.set_through_path.path1.in  = AS_THROUGH_PATH_IN_MIC; (2)
  command.set_through_path.path1.out = AS_THROUGH_PATH_OUT_MIXER2; (3)
  command.set_through_path.path2.en  = false; (4)
  AS_SendAudioCommand(&command);
1 Activate the data path 1 setting
2 Data path 1 uses MIC In as the input source
3 Data path 1 uses Mixer In 2 as the output destination
4 Invalidate the setting of data path 2
diag 7d633c524d51c4368fc35ca75018a85d
Figure 36. command example 3 dataflow
command example 4
  AudioCommand command;
  AudioResult result;
  command.header.packet_length = LENGTH_SET_THROUGH_PATH;
  command.header.command_code  = AUDCMD_SETTHROUGHPATH;
  command.header.sub_code      = 0x00;
  command.set_through_path.path1.en  = true; (1)
  command.set_through_path.path1.in  = AS_THROUGH_PATH_IN_I2S1; (2)
  command.set_through_path.path1.out = AS_THROUGH_PATH_OUT_MIXER1; (3)
  command.set_through_path.path2.en  = true; (4)
  command.set_through_path.path2.in  = AS_THROUGH_PATH_IN_I2S1;  (5)
  command.set_through_path.path2.out = AS_THROUGH_PATH_OUT_MIXER2; (6)
  AS_SendAudioCommand(&command);

  AS_ReceiveAudioResult(&result); (7)

  command.header.packet_length = LENGTH_SET_THROUGH_PATH;
  command.header.command_code  = AUDCMD_SETTHROUGHPATH;
  command.header.sub_code      = 0x00;
  command.set_through_path.path1.en  = true;
  command.set_through_path.path1.in  = AS_THROUGH_PATH_IN_MIXER; (8)
  command.set_through_path.path1.out = AS_THROUGH_PATH_OUT_I2S1; (9)
  command.set_through_path.path2.en  = false;
  AS_SendAudioCommand(&command);
1 Activate the data path 1 setting
2 Data path 1 uses I2S In as the input source
3 Data path 1 uses Mixer In 1 as the output destination
4 Activate the data path 2 setting
5 Data path 2 uses Mic In as the input source
6 Data path 2 uses Mixer In 1 as the output destination
7 Receive the result
8 Data path 1 uses Mixer Out as the input source
9 Data path 1 uses I2S Out as the output destination
diag 37574f27f4d427b95cf63048a3e8a9d8
Figure 37. command example 4 dataflow
diag d2ac51c70637e66fcb4730f137d5b455
Figure 38. Player State sequence
Build Configurations

To use the function of AudioThrough.

$>cd sdk
$>tools/config.py -m

With Config menu you need to configure the following.

Select options in below:

[CXD65xx Configuration]
  [Audio]                        <= Y

[SDK audio]                      <= Y
Error Attentions and Approach

A list of warnings when playing music and the corrective actions are as follow. See Error Information of Audio SubSystem for details.

ID Attention Code Attention Level Approach

0x0F

AS_ATTENTION_SUB_CODE_TASK_CREATE_ERROR

ERROR

This is due to insufficient heap space. Expand the heap area.

AS_ATTENTION_SUB_CODE_SIMPLE_FIFO_UNDERFLOW If this occurs, audio playback stops and playback errors will occur. When this happens, immediately issue the AsStopPlayerParam command and change to the playback stop state.
Be sure to clear FIFO after transition to play stop. If you do not noise will be generated.

11.3.4. Object Level API

The Audio function APIs provide simpler functions than the High Level API.
You can create more flexible applications by using the Object Level API.

About Usecase

Below are some examples of use cases using the Object Level API.

Usecase 1

This is a case to decode sound output of Audio data using MediaPlayerObject and OutputMixerObject .
PCM data from MediaPlayerObject can be processed by Application and sent to OutputMixerObject .

diag 2b47196fdad4a2157bc395e0bccdc9dd
Usecase 2

In this case PCM data placed in RAM is sent to OutputMixerObject without using MediaPlayerObject .
Since it does not have to be decoded, it is suitable for playing back sound sources that require promptness such as sound effects.

diag 91d463806ce512abdb8a0e45884fa4ee
Usecase 3

Using MediaRecorderObject to process the recorded MIC / I2S input with Application .
This is a case of sending to speaker or I2S output via OutputMixer .

diag 9f05800bf1c0f5f6ed11092cdb699d7d
MediaPlayerObject

MediaPlayerObject performs decode management of the audio data and output of the decoding result PCM.
Two players can be used at the same time, and they are controlled individually by parameter of PlayerID in each API.

Application passes ES data to MediaplayerObject via a buffer called Simple FIFO.
If the buffer underflows, playback stops. Application design is required to prevent this condition occurring.
When decoding is completed, MediaPlayerObject notifies MemoryHandle of PCM data.

diag f4b4ec84d21ebaf098aadeca6c6e0294

It is also possible to send PCM data to the OutputMixerObject . In this case there is no response to Application .
However, as a matter of course, it is necessary to generate and start OutputMixerObject .
(Default is a Callback response to Application as shown above.)

diag bcaf7c36bef0baccb48dd3e7ffb724d4
Functions
Create

You must make a call when using MediaPlayerObject.
This will create an instance and transition to wait for MsqQueue.

API
bool AS_CreatePlayer(AsPlayerId id, FAR AsCreatePlayerParam_t *param);
Activate

Activate MediaPlayerObject. You must call it.
Set the handle of the buffer to pass Audio data from Application to MediaPlayerObject here.

API
bool AS_ActivatePlayer(AsPlayerId id, FAR AsActivatePlayer &actparam);
Init

Set the information of the content to be played and initialize the MediaPlayerObject.
It is necessary to call each time the content information () changes.
(
Codec type, bit length, sampling frequency)

API
bool AS_InitPlayer(AsPlayerId id, FAR AsInitPlayerParam &initparam);
Play

Start playback.
After playback starts, the decoding result (PCM data) will continue to respond.
Application needs to supply Audio data to the buffer specified during Activate.
If the supply can not keep up, the buffer will underflow and playback stops.

You can choose whether to send PCM data of decoded result to Application or directly to OutputMixerObject.

API
bool AS_PlayPlayer(AsPlayerId id, FAR AsPlayPlayerParam &playparam);
Stop

Stop playback.
You can choose to stop after the data remaining in the Audio data buffer is completely recovered at the time of calling this API
You can choose to stop immediately. In the former case, it may take time to stop.

Also, depending on the timing of calls, notification of decoding completion may be received after call due to misplacement.

API
bool AS_StopPlayer(AsPlayerId id, FAR AsStopPlayerParam &stopparam);
Set Gain

Set the playback volume gain as L / R respectively. It can be set from Init until Deactivate.
If you do not call it, 100% is the default volume. (There is no dependency relation with the volume setting of BaseBand)

API
bool AS_SetPlayerGain(AsPlayerId id, FAR AsSetGainParam &gainparam);
Next Request

Requests the next Decode process (playback continuation process).
Please make a continuous call when the decode result data buffer is empty.
If you do not call this API, subsequent decoding processing will not be done and the sound will stop or break.

(* Call is unnecessary when PCM data transmission destination is directly OutputMixerObject.)

API
bool AS_RequestNextPlayerProcess(AsPlayerId id, FAR AsRequestNextParam &nextparam);
Deactivate

Deactivate MediaPlayerObject. To use MediaPlayerObject again, you need to start from Acitvate again.

API
bool AS_DeactivatePlayer(AsPlayerId id, FAR AsDeactivatePlayer &deactparam);
Delete

Delete the task, instance of MediaPlayerObject.
To use MediaPlayerObject again we need to do it again from Create.

API
bool AS_DeletePlayer(AsPlayerId id);
Sequence

Simple sequence of MediaPlayerObject.

diag de42f2839a5ec2f36d8cbf3e88b270e9
OutputMixerObject

OutputMixerObject manages sending PCM data (Rendering).
Application sends PCM data to OutputMixerObject via MemoryHandle.
When Rendering is completed, OutputMixerObject responds with a callback.

diag bfaf0f6425a975302f975fae14b46c76
Functions
Create

You must make a call when using OutputMixerObject.
This will create an instance and transition to wait for MsqQueue.

API
bool AS_CreateOutputMixer(FAR AsCreateOutputMixParam_t *param);
Activate

Activate OutputMixerObject. You must call it.
Select output destination (speaker or I2S) at this time.

API
bool AS_ActivateOutputMixer(uint8_t handle, FAR AsActivateOutputMixer &actparam);
Send Data

Send PCM data. When transmission is completed, the response is called back.

Please pass the following data before transmission of data passed in this API is completed.
If it not done in time, the DMA transfer underflows and stops.

Also, PCM data is actually sent when this API is called more than once.

diag e865af4ec3091ea6423c59bbae562a9d
API
bool AS_SendDataOutputMixer(FAR AsSendDataOutputMixer &sendparam);
Frame Term Control

Adjust the transmission time of PCM data.
It can be fine-tuned longer or shorter than the recording time of actual PCM data.
It is mainly used for adjusting the remaining buffer amount (under / overflow suppression) for PCM data.
(* Small noise may occur at adjustment execution.)

API
bool AS_FrameTermFineControlOutputMixer(uint8_t handle, FAR AsFrameTermFineControl &ftermparam);
Deactivate

Deactivate OutputMixerObject.

API
bool AS_DeactivateOutputMixer(uint8_t handle, FAR AsDeactivateOutputMixer &deactparam);
Delete

Delete the task, instance of OutputMixerObject.
In order to use OutputMixerObject again, it is necessary to start from Create again.

API
bool AS_DeleteOutputMix(void);
Sequence

Simple sequence of OutputMixerObject.

diag e6b6ca25a20e5f056b6d21ff90c4d565
MediaRecorderObject

MediaRecorderObject manages the encoding of Audio data and outputs the encoding result ES.

It encodes the input (PCM data) from the captured MIC or IS 2 and sequentially stores it in the Simple FIFO buffer.
Application can extract ES data by reading the FIFO. The extraction must be done in time, since overwriting stops recording. The application needs to be designed to prevent this condition.

diag 75593d76f238c730ff85bb837905911b
Functions
Create

You must make a call when using MediaRecorderObject.
This will create and instance and transition to wait for MsqQueue.

API
bool AS_CreateMediaRecorder(FAR AsActRecorderParam_t *param);
Activate

Activate MediaRecorderObject.
At this time, we pass the handle of the buffer used to pass the encoded audio data.

API
bool AS_ActivateMediaRecorder(FAR AsActivateRecorder *actparam);
Init

Initialize MediaRecorderObject by setting information on the content to be recorded.
It can not be changed dynamically during recording.

API
bool AS_InitMediaRecorder(FAR AsInitRecorderParam *initparam);
Start

Recording starts based on the information set in Init.
Encoded ES data is sequentially stored in the buffer for Audio specified at the time of Activate.
It is necessary for the application to extract ES data from the Audio buffer.

API
bool AS_StartMediaRecorder(void);
Stop

Stop recording.
It corresponds only to immediate stop.

API
bool AS_StopMediaRecorder(void);
Deactivate

Deactivate MediaRecorderObject.

API
bool AS_DeactivateMediaRecorder(void);
Delete

Delete the MediaRecorderObject.

API
bool AS_DeleteMediaRecorder(void);
Sequence
diag bfb9793025abdfe09300941239eeaec2

11.3.5. Low Level API

The low level API is under development.

11.3.6. Preprocess and UserCustomDSP

Preprocess is a architecture which can apply user defined signal process to recording audio data. The signal process is done by UserCustomDSP. Here explains the detail of architecture.

Codes of framework

If you would like to add your signal process, you should create UserCustomDSP first. A minimum set of source code for UserCustomDSP is provided as framework by SDK. You should fetch them and build with your source code, and you can create UserCustomDSP.

Pleaser refer Audio Recorder Tutorial for how to create UserCustomDSP.
diag 25a03e75844acdce8cd8a80f2463a143
Figure 39. Layout of framework codes
Relation between framework codes and user codes

Relation between framework codes and user codes should follow class diagram. You will edit "User Edit DSP Codes" inside diagram. (You don’t need to edit framework codes.)

diag c0a77694f54fbc5efa5835e772fdd6b7
Figure 40. Relation between framework coeds and user codes

When you edit user codes, you should follow these points.

  1. A Command format between UserCustomDSP and user codes are able to define freely, but it must inherit CustomprocCommand::CmdBase which is defined in framework. Because, format must follow Command format.

    diag ba7132addde99c08512c28d8e26d74e0
    Figure 41. Command format between UserCustomDSP and user codes

    For example, write as follow.

    struct InitPram : public CustomprocCommand::CmdBase
    {
      uint8_t ch_num;
      ...
    }
  2. A class which you write user codes ( UserProc in Class Diagram) must inherit CustomprocDspUserProcIf (pure abstract class) of framework, and override the methods to implement.

    class UserProc : public CustomprocDspUserProcIf
    {
      void init(CustomprocCommand::CmdBase*);
      ...
    }
UserCustomDSP sequence

In AudioRecorder function, a UserCustomDSP which is created with framework codes runs as follow sequence. Please refer to Set Preprocess type for role of UserCustomDSP in Audio Recorder.

As follow diagram, Init , Set commands are sent to UserCustomDSP when the user codes calls for API. On the other hand, Exec , Flush commands are sent to UserCustomDSP by AudioSubsystem internally every time capture audio data. (You don’t need to send Exec , Flush commands from user codes.)

Brief sequence.

  1. AUDCMD_INITMFE, Init is sent to UserCustomDSP. It will used to notify some parameters to init UserCustomDSP. For example, ch number, bit length, and more.

  2. AUDCMD_STARTREC, Start capturing audio data and send it to UserCustomDSP with Exec . Apply signal process to them by UserCustomDSP.

  3. AUDCMD_SETMFE, Set is sent to UserCustomDSP. You can send anytime even if after Start was sent. For examples, set filter coefficient.

  4. AUDCMD_STOPREC, Stop capturing audio data and send Flush after Exec of last frame. Flush is used for flush of delayed samples.

diag 9cb6961167b7baf310fe936368f25b68
Figure 42. Sequence between user codes and UserCustomDSP

UserCustomDSP is expected to have state transition like follow diagram. The trigger of transition will Init , Exec , Flush , Set command. (The framework codes "doesn’t" implement state transition. It is "expected" to implement.)

diag 4f47087566d9182079e9a3ddc34d7fc0
Figure 43. UserCustomDSP state transition

11.3.7. Error Information of Audio SubSystem

Overview

The High Level API of Audio SubSystem has an interface for the sending and receiving data for commands and results, and there are two kinds of errors.

If there is a problem with the command issued for SubSystem, "ErrorResponse" result is returned instead of completion response to command event. This error is called a response error.

Also, if an error occurs inside the system, an "ErrorAttention" event will be generated from inside the system and notified as a result. It is called an attention error.

To perform troubleshooting identify the error type as response error or attention error, and read the appropriate parts of the following sections.

Response Error

When you want to control the SubSystem, and there is a status violation or a parameter mistake, a response of the completion response to each command will not occur, and an "ErrorResponse" result will be generated as a response.

diag ff9482ae53f8a6bfe412ef83b0634937

For the data format of the "ErrorResponse", see result format and result packet details.

"Error Response" has a "Error Code", and "Error Code" means that what kind of error occurred.

For a list of Error Code, please refer to Error Code List.

Attention Error

An "ErrorAttention" result is generated for errors while processing inside the Audio Sub System (not command processing). In order to receive this event, it is necessary to register the callback function with the AS_CreateAudioManager.

diag 36459fc0406ab90c4f0684f216826c8f

For the data format of the "ErrorAttention" result, see Result format and Error Attention

This error is caused by a control failure such as the supply of ES(Elementary Stream) being interrupted at the time of rendering operation, the ES write buffer overflowing at the time of recording operation, a system error such as depletion of memory resources, delay of real time processing, etc. These are fatal errors, so please implement error handling based on the type of error that occurred.
Also, by changing the program implementation, errors should be reduced or eliminated.

For the list of attention codes, please refer to here.

Attention Level

A level of importance is specified for the attention notification, and the processing method for recovery is dependent on the severity of the error.

Table 21. Attention Level
Level value Description

FATAL

0x03

A system call error, etc. that is not recoverable and requires a reset to restore.

ERROR

0x02

It is an error in the operation of which the audio system can not continued with the internal error (queue Full / Empty, DSP load / unload etc). It is possible to recover by returning the system to the initial state (Ready state).

WARN

0x01

There is a possibility that the operation is abnormal, such as encode / decode error, data underflow / overflow, etc. There is a possibility that abnormality has occurred in voice data, etc., but operation can continue.

Error Code List

Below is a list of "Error Code" added to "Error Response".

AS_ECODE_STATE_VIOLATION
code

0x01

description

A status violation has occurred in Audio Sub System.
Since the requested command does not match the state in AudioSubSystem, it can not accept the command.
Please make a transition to the correct state with a command to change the state and then re-send the command.

For the state transition in AudioSubSystem see below.

state transition

AS_ECODE_PACKET_LENGTH_ERROR
code

0x02

description

The packet length parameter of the command packet is incorrect.
The packet length is specified in units of Word (4 bytes). It is not a byte unit.
Make sure that the correct value is set.

AS_ECODE_COMMAND_CODE_ERROR
code

0x03

description

An unknown command was received.
Check the command ID.

AS_ECODE_COMMAND_NOT_SUPPOT
code

0x04

description

The command you sent is not valid.
In order to use the command, it is necessary to enable the corresponding function in the configuration.

For details on configuring Audio Player and Audio Recorder, refer to the following.

Audio Player
Audio Recorder

AS_ECODE_AUDIO_POWER_ON_ERROR
code

0x05

description

The Audio power supply can not be turned on. There is a possibility that the power is already on.

AS_ECODE_AUDIO_POWER_OFF_ERROR
code

0x06

description

The Audio power can not be turned off. There is a possibility that the power has already been turned off.

AS_ECODE_DSP_LOAD_ERROR
code

0x07

description

Can not load / start the DSP.

Make sure that the path to the DSP file is set correctly.
Please refer to the following for setting the path.

AudioPlayer Player initialization
AudioRecorder Init recorder information

Also, there is a possibility that the required DSP is not placed in the defined path.
AudioPlayer requires MP3DEC, AACDEC, WAVDEC, OPUSDEC (depending on the type of playback file).
AudioRecorder requires MP3ENC, OPUSENC, SRC (depending on recording file type).
These are located in sdk/modules/audio/dsp/ .

AS_ECODE_DSP_UNLOAD_ERROR
code

0x08

description

DSP can not be terminated / unloaded.
It may already be unloaded or may not have been loaded.

AS_ECODE_DSP_VERSION_ERROR
code

0x09

description

The version of DSP is different from what AudioSubStyle expects and can not be used.
Please use the DSP sdk/modules/audio/dsp/ that is included in the SDK package you are using.

AS_ECODE_SET_AUDIO_DATA_PATH_ERROR
code

0x0a

description

Path setting (input / output) of Audio data is wrong.
Please confirm that the following parameters are correctly set.

AudioRecorder is Init recorder information
BaseBand is SetBaseBandStatusParam
Through mode is AsSetThroughPathParam

AS_ECODE_CLEAR_AUDIO_DATA_PATH_ERROR
code

0x0b

description

Audio data path setting can not be cleared.

AS_ECODE_NOT_AUDIO_DATA_PATH
code

0x0c

description

The input device or output device is not enabled.
It is necessary to transition to BaseBandStatus or ThroughStatus.

The transition to BasebandStatus is SetBasebandStatus
The transition to ThroughStatus is SetThroughStatus

AS_ECODE_DECODER_LIB_INITIALIZE_ERROR
code

0x0d

description

Initialization error is being reported from Decoder DSP.

AS_ECODE_ENCODER_LIB_INITIALIZE_ERROR
code

0x0e

description

An initialization error is being reported from the Encoder DSP.

AS_ECODE_FILTER_LIB_INITIALIZE_ERROR
code

0x0f

description

Initialization error is being reported from Filter DSP. .

AS_ECODE_COMMAND_PARAM_CODEC_TYPE
code

0x11

description

The type of the specified Audio Codec is incorrect.

For the codec supported by AudioSubSystem, refer to the following.

AudioPlayer Player initialization
AudioRecorder Init recorder information

AS_ECODE_COMMAND_PARAM_CHANNEL_NUMBER
code

0x13

description

The specified number of channels is incorrect, or an error occurs in combination with other parameters.
Both AudioPlayer and AudioRecorder have limitations on combination of allowed parameters such as Codec type and number of channels.

For details of each restriction, please refer to the following.

AudioPlayer Player initialization
AudioRecorder Init recorder information

AS_ECODE_COMMAND_PARAM_SAMPLING_RATE
code

0x14

description

The specified sampling frequency is incorrect, or an error occurs in combination with other parameters.
Both AudioPlayer and AudioRecorder have limitations on combination of allowed parameters such as Codec type and sampling frequency.

In the case of the AudioPlayer, information on the sampling frequency included in the header of the audio data and this error also occurs if the specified sampling frequency is different.

For details of each restriction, please refer to the following.

AS_ECODE_COMMAND_PARAM_BIT_RATE
code

0x15

description

The specified bit rate is incorrect, or an error occurs in combination with other parameters.
AudioRecorder has limitations on combinations of allowed parameters such as Codec type and bit rate.

Please refer to the following for details of restriction.

Init recorder information

AS_ECODE_COMMAND_PARAM_BIT_LENGTH
code

0x16

description

The specified bit length is incorrect, or an error occurs in combination with other parameters.

For details of each restriction, please refer to the following.

AudioPlayer Player initialization
AudioRecorder Init recorder information

Also, if you specify 24 bits for bit length, you need to switch to HiReso mode.

For details on switching modes, see Init Rendering Clock.

AS_ECODE_COMMAND_PARAM_COMPLEXITY
code

0x17

description

The compression rate of the specified OPUS encoding is incorrect.

For the parameters, please refer to the following.+
Init recorder information

AS_ECODE_COMMAND_PARAM_ACTIVE_PLAYER
code

0x18

description

Invalid Player specification is enabled.
It can not transit to Player state.

For the parameters, please refer to the following.

Change to Player Status

AS_ECODE_COMMAND_PARAM_INPUT_DEVICE
code

0x19

description

The specified input device is incorrect.

For the parameters, please refer to the following.

Change to Player Status
Change to Recorder Status

AS_ECODE_COMMAND_PARAM_OUTPUT_DEVICE
code

0x1A

description

The specified output device is incorrect.

For the parameters, please refer to the following.

Change to Player Status
Change to Recorder Status

AS_ECODE_COMMAND_PARAM_INPUT_HANDLER
code

0x1B

description

The specified input device handle is incorrect.
Make sure that Handle and callback function of SimpleFIFO are set correctly.

For the parameters, please refer to the following.

Change to Player Status

AS_ECODE_COMMAND_PARAM_CONFIG_TABLE
code

0x1F

description

The coefficient table address of the MFE or MPP filter is incorrect.

AS_ECODE_COMMAND_PARAM_WITH_MFE
code

0x20

description

The activation specification parameter of MFE is incorrect.
In the first place it is possible that an incorrect value is set or MFE is disabled even though the VoiceCommand is valid.
(* MFE must also be enabled whenever you activate VoiceCommand.)

For the parameters, please refer to the following.

Set Baseband Status

AS_ECODE_COMMAND_PARAM_WITH_MPP
code

0x21

description

The parameter to activate MPP is invalid. (MPP is not supported yet.)

AS_ECODE_COMMAND_PARAM_INPUT_DB
cod

0x28

description

The mute specification parameter is incorrect.

For the parameters, please refer to the following.
Set Volume Mute

AS_ECODE_DMAC_INITIALIZE_ERROR
code

0x2B

description

Failed to initialize voice capture / rendering.

AS_ECODE_DMAC_READ_ERROR
code

0x2C

description

Audio data acquisition failed.

AS_ECODE_CHECK_MEMORY_POOL_ERROR
code

0x2E

description

MemoryPool check error occurred during Object Create in AudioSubSystem.
It is possible that the memory pool ID is not passed to the Create API correctly or the MemoryPool is created incorrectly.

Please refer here for specification of memory pool ID.

Create Media Player
Create Media Recorder

For details on creating MemoryPool, please refer to here.

AudioPlayer PoolAreas

AS_ECODE_SIMPLE_FIFO_UNDERFLOW
code

0x2F

description

At the start of Audio playback, the Simple FIFO used as the ES buffer underflows.
When starting playback, AudioSubSystem immediately extracts ES data from SimpleFIFO and starts decoding. So in the application, it is necessary to put ES data in the FIFO before playback starts.

diag 15d95d65cefb8f57d246cca6f7135ead
AS_ECODE_SET_MIC_GAIN_ERROR
code

0x30

description

The microphone input gain setting value is wrong.
The setting range is different between the analog microphone and the digital microphone.

For the parameters, please refer to the following.
Initialize Mic Gain

AS_ECODE_SET_OUTPUT_SELECT_ERROR
code

0x32

description

The specified output destination setting is incorrect.

InitOutputSelect

AS_ECODE_INIT_CLEAR_STEREO_ERROR
code

0x33

description

Clear Stereo setting error occurred.

AS_ECODE_SET_VOLUME_ERROR
code

0x34

description

The specified playback volume is incorrect.
Volume setting value is limited. For details of the parameters, refer to the following.

Set Volume

AS_ECODE_SET_VOLUME_MUTE_ERROR
code

0x35

description

Volume setting to AudioDriver failed.

AS_ECODE_SET_BEEP_ERROR
code

0x36

description

The specified beep parameters are incorrect.
The volume / frequency setting value range has limits. For details of the parameters, refer to the following.

AS_ECODE_QUEUE_OPERATION_ERROR
code

0x37

description

AudioSubSystem internal queue operation (push, pop) error occurred.

AS_ECODE_COMMAND_PARAM_RENDERINGCLK
code

0x39

description

Parameter error in HiReso mode setting.
For the parameters that can be set, refer to the following.

SetRenderingClock

AS_ECODE_SET_RENDERINGCLK_ERROR
code

0x3A

description

HiReso mode setting error.
In order to use the HiReso mode, it is necessary to set the audio clock to 49.152 Mhz in the configuration.

$>cd sdk
$>tools/config.py -m
[CXD56xx Configuration]
  [Audio]
    [Audio baseband config settings]
      [CXD5247 settings]
        [X'tal frequency of the CXD5247] <- 49.152Mhz
Attention Code List

Below is a list of "Attention Code" added to "ErrorAttention".

AS_ATTENTION_SUB_CODE_DMA_UNDERFLOW
Code

0x01

Attention Level

ERROR

Description

DMA transfer underflow occurred.
The transfer request to DMA may be slower than the data transfer speed of DMA. Or the DMA transfer may be stopped.
The transfer speed of DMA is 48000 samples / second in the normal mode and 192000 samples / second in the HiReso mode.

Error Handling

If this error occurs, application task processing may have priority over real-time processing.
Please review the priority of each task.

AS_ATTENTION_SUB_CODE_DMA_OVERFLOW
Code

0x02

Attention Level

ERROR

Description

DMA transfer overflow occurred.
The transfer request to DMA may be faster than the data transfer speed of DMA. Or DMA transfer may not be started.

The transfer speed of DMA is 48000 samples / second in the normal mode and 192000 samples / second in the HiReso mode.

Error Handling

In the case of recording operation, the master clock of Audio supplied by HW may be stopped. Is the clock supplied correctly? Please Confirm. If it occurs during playback operation, check that the correct clock is being sent.

AS_ATTENTION_SUB_CODE_DMA_ERROR
Code

0x03

Attention Level

FATAL, ERROR

Description

An error is being responded from the DMA hardware.
It may have requested transfer start before sending a transfer request to DMA.

Error Handling

It is an error that can not normally occur.
Please contact SDK development member if this error occurs.

AS_ATTENTION_SUB_CODE_SIMPLE_FIFO_UNDERFLOW
Code

0x05

Attention Level

WARNING

Description

Simple FIFO underflow occurred during AudioPlayer operation.
Simple FIFO is used as a buffer to pass ES data from Application to AudioSubSystem.
The application can not keep up with the data extraction speed of AudioSubSystem.

Error Handling

Adjust the task priority so that buffer supply from the application can be made fast enough.
Please increase the buffer size so that data input can be made in time.

AS_ATTENTION_SUB_CODE_SIMPLE_FIFO_OVERFLOW
Code

0x06

Attention Level

WARNING

Description

An overflow of SimpleFIFO occurred while AudioRecorder was running.
Simple FIFO is used to buffer EncodedES data from AudioSubSystem to application.
The application does not follow the data input speed of AudioSubSystem.+

Error Handling

Adjust the task priority so that application buffer data extraction can be made in time.
Please increase the buffer size so that data extraction can be made in time.

AS_ATTENTION_SUB_CODE_ILLEGAL_REQUEST
Code

0x07

Attention Level

ERROR

Description

An illegal event was received inside AudioSubSystem.
There is a possibility that the control sequence is abnormal.

Error Handling

It is an error that can not normally occur.
Please contact SDK development member if this error occurs.

AS_ATTENTION_SUB_CODE_INTERNAL_STATE_ERROR
Code

0x08

Attention Level

ERROR

Description

Status error in MediaPlayer.

error handling

It is an error that can not normally occur.
Please contact SDK development member if this error occurs.

AS_ATTENTION_SUB_CODE_UNEXPECTED_PARAM
Code

0x09

Attention Level

ERROR

Description

The command parameter is incorrectly.

Error Handling

Check the command parameters.

AS_ATTENTION_SUB_CODE_QUEUE_POP_ERROR
Code

0x0A

Attention Level

ERROR

Description

An internal queue POP error has occurred.

Error Handling

It is an error that can not normally occur.
Please contact SDK development member if this error occurs.

AS_ATTENTION_SUB_CODE_QUEUE_PUSH_ERROR
Code

0x0B

Attention Level

ERROR

Description

An internal queue PUSH error has occurred.

Error Handling

It is an error that can not normally occur.
Please contact SDK development member if this error occurs.

AS_ATTENTION_SUB_CODE_QUEUE_MISSING_ERROR
Code

0x0C

Attention Level

ERROR

Description

The internal queue is unintentionally empty.

Error Handling

It is an error that can not normally occur.
Please contact SDK development member if this error occurs.

AS_ATTENTION_SUB_CODE_MEMHANDLE_ALLOC_ERROR
Code

0x0D

Attention Level

ERROR

Description

Acquisition of memory handle failed.
All memory handles are in use or the handle ID is incorrect.

Error Handling

If the number of memory handle steps is insufficient, please review the memory pool setting.
Please refer to the following for the setting method. (An example of AudioPlayer setting)

Player MemoryPool overview
Player MemoyPool definitions

AS_ATTENTION_SUB_CODE_MEMHANDLE_FREE_ERROR
Code

0x0E

Attention Level

ERROR

Description

Failed to release memory handle.
The handle you tried to release may have already been released.

Error Handling

It is an error that can not normally occur.
Please contact SDK development member if this error occurs.

AS_ATTENTION_SUB_CODE_TASK_CREATE_ERROR
Code

0x0F

Attention Level

ERROR

Description

Failed to generate a task to be used inside AudioSubSystem.
There is a possibility that the number of task generation in the whole system has reached the upper limit.

Error Handling

Increase the task creation suck limit or reduce the tasks used in the application.

To increase the upper limit of the task generation number, you can set it by config.

$>cd sdk
$>cd tools/config.py -k -m
[RTOS Features]
  [Tasks and Scheduling]
    [Max number of tasks] <- Set this.
AS_ATTENTION_SUB_CODE_RESOURCE_ERROR
Code

0x10

Attention Level

ERROR

Description

Failed to create / delete instances used within AudioSubSystem.
It is possible that you called the Create API, Delete API of Object / Component twice.
Or the heap area may be short.

Error Handling

It is an error that can not normally occur.
Please contact SDK development member if this error occurs.

AS_ATTENTION_SUB_CODE_DSP_LOAD_ERROR
Code

0x12

Attention Level

ERROR

Description

Failed to load DSP binary.
The DSP binary file may not be in the specified folder path.

Error Handling

Check the existence of file or check the specification of folder path.
For details on specifying the folder path, refer to the following.

AudioPlayer Player initialization
AudioRecorder Init recorder information

AS_ATTENTION_SUB_CODE_DSP_UNLOAD_ERROR
Code

0x13

Attention Level

ERROR

Description

DSP unload failed. It may already be unloaded or may not have been loaded at all.

Error handling

It is an error that can not normally occur.
Please contact SDK development member if this error occurs.

AS_ATTENTION_SUB_CODE_DSP_EXEC_ERROR
Code

0x14

Attention Level

ERROR, WARNING

Description

An error occurred during execution of processing in the DSP.
Although it does not immediately affect the operation of AudioSubSystem
the result of the decoding / encoding / filter may not be normal.
(There is a possibility that the sound is temporarily distorted, the data drops out, etc.)

Error Handling

In playback operation, the audio data may be corrupted.
Please check the audio data.

AS_ATTENTION_SUB_CODE_DSP_ILLEGAL_REPLY
code

0x16

Attention Level

ERROR

Description

A response command packet from the DSP is corrupted.

Error handling

It is an error that can not normally occur.
Please contact SDK development member if this error occurs.

AS_ATTENTION_SUB_CODE_DSP_VERSION_ERROR
Code

0x18

Attention Level

ERROR

Description

A DSP version error occurred. Can not load DSP.

Error Handling

Please use the DSP binary packed in the SDK.
The DSP binary is in sdk/modules/audio/dsp .

AS_ATTENTION_SUB_CODE_BASEBAND_ERROR
Code

0x19

Attention Level

ERROR

Description

A register setting error occurred in Audio Driver.

Error Handling

It is an error that can not normally occur.
Please contact SDK development member if this error occurs.

AS_ATTENTION_SUB_CODE_STREAM_PARSER_ERROR
Code

0x1A

Attention Level

ERROR

Description

ES data analysis error occurred in AudioPlayer.
Player initialization parameters and ES data analysis results are different.

Error Handling

Please check whether ES data passed to AudioSubSystem is wrong or corrupted.

AS_ATTENTION_SUB_CODE_DSP_LOG_ALLOC_ERROR
Code

0x1E

Attention Level

ERROR

Description

Failed to acquire DSP log buffer.
When multiple DSPs are in use, there is a possibility that the remaining capacity of the log buffer area is insufficient.

Error Handling

The buffer area for DSP log can not be set from application or config.

However, if the DSP log is unnecessary, you can turn off the logging function itself by config.

$>cd sdk
$>cd tools/config.py -k -m
[Audio Utilities]
  [Audio component menu]
    [dsp debug dump] <- Set to "No".
AS_ATTENTION_SUB_CODE_DSP_ASSETION_FAIL
Code

0x1F

Attention Level

ERROR

Description

An error occurs inside the DSP and it is impossible to continue processing.

Error Handling

The system reset is necessary for recovery.

AS_ATTENTION_SUB_CODE_DSP_SEND_ERROR
Code

0x20

Attention Level

ERROR

Description

Failed to send the command to DSP.

Error Handling

It is an error that can not normally occur.
There may be a problem inside the SDK.

Module ID List

AudioSubSystem ID list of modules to be used internally.
It is notified with Attention callback along with attention code, and it judges which module caused an error.

AS_MODULE_ID_AUDIO_MANAGER

Audio Manager ID.

AS_MODULE_ID_AUDIO_DRIVER

Audio Baseband Driver Module ID.

AS_MODULE_ID_INPUT_DATA_MNG_OBJ

Input Data Manager Object ID.

AS_MODULE_ID_MEDIA_RECORDER_OBJ

Media Recorder Object ID.

AS_MODULE_ID_OUTPUT_MIX_OBJ

Output Mix Object ID.

AS_MODULE_ID_PLAYER_OBJ

Player Object ID.

AS_MODULE_ID_RECOGNITION_OBJ

Recognition Object ID.

AS_MODULE_ID_SOUND_EFFECT_OBJ

Sound Effect Object ID.

AS_MODULE_ID_CAPTURE_CMP

Capture Component ID.

AS_MODULE_ID_DECODER_CMP

Decoder Component ID.

AS_MODULE_ID_ENCODER_CMP

Encoder Component ID.

AS_MODULE_ID_FILTER_CMP

Filter Component ID.

AS_MODULE_ID_RECOGNITION_CMP

Recognition Component ID.

AS_MODULE_ID_RENDERER_CMP

Renderer Component ID.

AS_MODULE_ID_POSTFILTER_CMP

Postfilter Component ID.

11.3.8. Use the libraries

The audio subsystem requires the following libraries:

11.4. Camera

11.4.1. Overview

CXD5602 has the 8-bit parallel Camera I/F, and can be connected to Camera module with such I/F. Currently, Spresense support Camera module which mount Sony ISX012. Camera module has the I/F for module control in addition to I/F for data. I/F for ISX012’s control is I2C.

HW overview is as follow:

camera hw overview
Figure 44. Camera HW Overview

CISIF, which is Camera I/F block in CXD5602, bridges 8-bit parallel I/F and internal bus of CXD5602. In addition, ISX012 camera module use I2C bus.

This chapter provides an overview of how to control camera module connected to this Camera I/F by Spresense SDK.

Camera driver I/F of Spresense SDK is similar to V4L2 which is well-known on Linux systems. Therefore, V4L2 application code is reusable easily. This I/F is called V4S(Video for Spresense).

V4S provides abstract APIs of camera functions by using standard file system interface(open, close and ioctl, ..). These APIs enable application not to care about camera device.

camera v4s overview
Figure 45. V4S SW Overview

V4S provides two virtual streams. One is video stream for Camera preview. The other is still picture stream.

camera v4s dataflow
Figure 46. V4S Dataflow

Application can get image data for each streams according to the following procedure:

  • prepare a buffer.

  • Enqueue the buffer to the driver’s queue by VIDIOC_QBUF.

  • Dequeue the buffer from the driver’s queue by VIDIOC_DQBUF.

The buffer need to be aligned on 32-bit boundary.

Control stream is specified by v4l2_buf_type:

This V4L2_BUF_TYPE_STILL_CAPTURE is V4S-specific parameter.

The outline from V4S initialization to image data capture is as follow:

diag 9939b4c77a21183c907defec4d850616
Figure 47. V4S Sequence Overview

11.4.2. State Transition

Because V4S manages the states for each streams, application controls each streams in parallel.

About getting image from camera device, V4L2_BUF_TYPE_STILL_CAPTURE control has priority, that is, V4L2_BUF_TYPE_VIDEO_CAPTURE stream stop getting image from VIDIOC_TAKEPICT_START to VIDIOC_TAKEPICT_STOP.

diag 5129cebde75b7318003607628b7187db

11.4.3. V4S supported ioctl command

Table 22. V4S supported ioctl command
分類 command purpose Spresense-specific

Buffer Control

VIDIOC_REQBUFS

Initialize buffer control field in driver.

Add parameter v4l2_buf_mode mode to give ring structure to buffers.

VIDIOC_QBUF

Enqueue buffer which application prepared.

V4L2 compliance

VIDIOC_DQBUF

Dequeue buffer which has image data.

V4L2 compliance

VIDIOC_CANCEL_DQBUF

Cancel VIDIOC_DQBUF.

Spresense-specific command

Stream Control

VIDIOC_STREAMON

Start stream.

V4L2 compliance

VIDIOC_STREAMOFF

Stop stream.

V4L2 compliance

VIDIOC_TAKEPICT_START

Start taking still pictures.

Spresense-specific command

VIDIOC_TAKEPICT_STOP

Stop taking still pictures.

Spresense-specific command

Check range of frame setting

VIDIOC_ENUM_FMT

Check supported pixel formats by camera device.

V4L2 compliance

VIDIOC_ENUM_FRAMESIZES

Check supported image sizes by camera device.

Add parameter v4l2_buf_type to check for each streams.

VIDIOC_ENUM_FRAMEINTERVALS

Check supported frame intervals by camera device.

Add parameter v4l2_buf_type to check for each streams.

VIDIOC_TRY_FMT

Check pixel format and image size passed from application are valid

V4L2 compliance

Change frame setting

VIDIOC_S_FMT

Set pixel format and image size.

V4L2 compliance

VIDIOC_S_PARM

Set frame interval with numerator and denominator of fraction.

V4L2 compliance

Check range of camera setting

VIDIOC_QUERYCTRL

Check range of camera setting.

V4L2 compliance

VIDIOC_QUERY_EXT_CTRL

Check range of camera setting. This is the extended API of VIDIOC_QUERYCTRL and encompass VIDIOC_QUERYCTRL.

V4L2 compliance

VIDIOC_QUERYMENU

About the camera setting items which take discrete values, get the discrete values.

V4L2 compliance

Get current value of camera setting

VIDIOC_G_CTRL

Get current value of camera setting.

V4L2 compliance

VIDIOC_G_EXT_CTRLS

Get current value of camera setting. This is the extended API of VIDIOC_G_CTRL and encompass VIDIOC_G_CTRL.

V4L2 compliance

Change camera setting

VIDIOC_S_CTRL

Change camera setting

V4L2 compliance

VIDIOC_S_EXT_CTRLS

Change camera setting This is the extended API of VIDIOC_S_CTRL and encompass VIDIOC_S_CTRL.

V4L2 compliance

VIDIOC_DO_HALFPUSH

Do camera setting when shutter button is half-pushed.

Spresense-specific command

11.4.4. ISX012 Proprietary Specifications

JPEG + YUV4:2:2 format

Application can get image data of a frame both with JPEG format and with YUV422 format by setting ioctl(VIDIOC_S_FMT) as follow:

  • parameter pixelformat = V4L2_PIX_FMT_JPEG_WITH_SUBIMG

  • parameter subimg_pixelformat = V4L2_PIX_FMT_UYVY

For example, application can take a picture and display it without JPEG decoder.

diag 0095045efa480f9d7235ffe2a678c280
Figure 48. JPEG + YUV4:2:2 format

11.4.5. Restrictions of image size and frame rate

Valid image size in VIDIOC_S_FMT and valid frame interval(reciprocal number of frame rate) in VIDIOC_S_PARM are related to each other. For example, when using larger image size, frame interval must be larger(frame rate must be smaller).

Supported ranges in Spresense + ISX012 environment are as follow:

ISX012 supported frame rates are 120 / 60 / 30 / 15 / 7.5 / 6 / 5. In the following diagram of YUV4:2:2 format case,

  • ≦ QVGA : application use all of 120FPS to 5FPS, because max FPS=120.

  • > QVGA : application use 60FPS to 5FPS(can not use 120FPS), because max FPS=60.

diag e20be42c17fe936f4b3ca31aee2cb114
Figure 49. YUV4:2:2 format case
diag 2d164fbd359fa17a6c4988de576bdaef
Figure 50. JPEG format case
diag 1415c4a641b352130a344b6b54cad2cc
Figure 51. JPEG + YUV4:2:2 format case

In JPEG + YUV4:2:2 format, YUV4:2:2 image size must be 96x64 to WQVGA(400x240). In this range, the above frame rate restriction is not affected.

11.4.6. Sample Code

11.5. DNN Runtime

11.5.1. DNN Runtime Overview

The DNN Runtime library can perform recognition processing using the Deep Neural Network (DNN) using trained models by Neural Network Libraries or Neural Network Console provided by Sony.

User must create the trained model file (nnb file format) by Neural Network Console. See Preparation of a trained model to create it.

dnnrt overview en
Figure 52. DNN Runtime Overview

See Neural Network Libraries and Neural Network Console Official sites

And DNN Runtime Library uses NNabla C Runtime.

11.5.2. Sample code

This sample code is handwritten number recognition using the trained model file created at Preparation of a trained model section, and it was created from image_recognition.MNIST.LeNet sample network model.

The trained model created by image_recognition.MNIST.LeNet takes one image of 28 x 28 size and outputs 10 arrays. These 10 arrays correspond to the numbers recognized by the index, and the probability of each number is output in the array. For example, at the head of an array (index 0), the probability that the input image is the number "0" is output.

See DNNRT example README for more details.

diag 41897df473adcc0797c696868be4a848
Figure 53. DNNRT sequence

11.6. GNSS

The Spresense board has a Global Navigation Satellite System (GNSS) receiver that calculates its current position, velocity and time. To use this feature, you need to connect the GNSS chip to an antenna, such as the one on the Spresense Main Board.

11.6.1. Key Features

The embedded GNSS receiver on the Spresense board supports:

  • The GPS and GLONASS, QZSS(Michibiki) GNSS systems.

  • The satellite-based augmentation systems (SBAS) capabilities of each GNSS systems.

  • Asynchronous position processing and notification.

  • Geo-fencing (detecting whether the receiver is leaving a specified area).

Your application can control these features using POSIX file functions such as open, close, read, seek, and ioctl. For example, the application can open the GNSS device file '/dev/gps', call the ioctl command, and use the read function to retrieve position data.

GNSS has several ioctl commands, which are described in this section.

  • For NuttX, ioctl commands have three arguments.

  • For GNSS, the second parameter, "req", is a GNSS command, and the third parameter, "arg", is input/output data.

11.6.2. GNSS Configuration

To use the GNSS receiver, enable the device by setting CONFIG_CXD56_GNSS to Y.

[System Type]
  [CXD56xx Peripheral Support]
    [GNSS device] (CXD56_GNSS)  = Y

To enable conversion of the GPS position data to the widely-used NMEA data format, enable the NMEA conversion library /ref gnss_utilities_nmea, by setting CONFIG_CXD56_GNSS, CONFIG_LIBM, and CONFIG_GPSUTILS_CXD56NMEA_LIB to Y.

[System Type]
  [CXD56xx Peripheral Support]
    [GNSS device] (CXD56_GNSS) = Y
[Library Routines]
  [Standard Math library] (LIBM) = Y
[Application Configuration]
  [GPS Utilities]
    [Support CXD56xx gnss NMEA convert library] (GPSUTILS_CXD56NMEA_LIB) = Y

If GPS signals are not available (e.g., at many indoor locations), you can use example GPS position data for test and development purposes. To enable the example GPS position data, set CONFIG_CXD56_GNSS and CONFIG_EXAMPLES_GNSS for the GNSS position example.

[System Type]
  [CXD56xx Peripheral Support]
    [GNSS device] (CXD56_GNSS) = Y
[Application Configuration]
  [Examples]
    [GNSS position example] (EXAMPLES_GNSS) = Y

If GPS signals are not available (e.g., at many indoor locations), you can use example the GNSS command emulator for test and development purposes. To use this capability, enable the GNSS receiver and NMEA conversion library, as shown previously. Also enable CONFIG_EXAMPLES_GNSS_ATCMD for the GNSS command emulator example.

[System Type]
  [CXD56xx Peripheral Support]
    [GNSS device] (CXD56_GNSS) = Y
[Library Routines]
  [Standard Math library] (LIBM) = Y
[Device Drivers]
  [USB Device Driver Support]
    [USB Modem (CDC/ACM) support] = Y
[Application Configuration]
  [GPS Utilities]
    [Support CXD56xx gnss NMEA convert library] (GPSUTILS_CXD56NMEA_LIB) = Y
[Application Configuration]
  [Examples]
    [GNSS CXD5603 @command emulator example] (EXAMPLES_GNSS_ATCMD) = Y

To test the GNSS receiver hardware, you can perform a factory test.

To enable a factory test:

  • Use CONFIG_CXD56_GNSS to enable the GNSS receiver.

  • Enable CONFIG_EXAMPLES_GNSS_FACTORY for the GNSS factory test example data.

  • Select the EXAMPLES_GNSS_FACTORY_SVID according to the test environment.

To start a factory test, set [Application entry point] to gnss_factory_test.

To interpret the test results, multiply the values of cn and doppler results by 1000000.

[System Type]
  [CXD56xx Peripheral Support]
    [GNSS device] (CXD56_GNSS) = Y
[Application Configuration]
  [Examples]
    [GNSS FACTORY test] (EXAMPLES_GNSS_FACTORY) = Y
    [FACTORY TEST svid] (EXAMPLES_GNSS_FACTORY_SVID) = 1
[RTOS Features]
  [Tasks and Scheduling]
    [Application entry point]
      set 'gnss_factory_test'
To display the GNSS status on an E-ink display, see the following example code:spresense/examples/gnss_factory/README.txt

You can configure the following settings:

  • Disable power control of the LNA device from GNSS device driver controls if a Low-Noise Amplifier (LNA) receives power through the BSP layer. Set this to N to ensure that the LNA device power can be controlled through BSP.

  • You can reduce power consumption by stopping the GNSS module while it is not needed. To enable this capability, set Enable GNSS HOT Sleep to Y.

  • The GNSS backup file name and GNSS CEP file name to specify the binary files the GNSS uses to back up data and calculate circular error probable (CEP). Change the file name according to the file system the SDK uses.

[System Type]
  [CXD56xx Peripheral Support]
    [GNSS setting]
      [Disable power control of the LNA device from GNSS device driver] = N
      [Enable GNSS HOT Sleep] = N
      [GNSS backup file name] = '/mnt/spif/gnss_backup.bin'
      [GNSS CEP file name] = '/mnt/vfat/gnss_cep.bin'

The GNSS device driver notifies the application using POSIX’s poll or signal mechanism every time the receiver outputs position data. You can independently configure the maximum number of poll waits and signal receivers as shown below. The default values are 4 for poll and 3 for signal. For details, see Position Calculation Notification.

[System Type]
  [CXD56xx Peripheral Support]
    [GNSS setting]
      [GNSS max poll waiters] = 4
      [GNSS max signal receivers] = 3

11.6.3. Device Control

This section describes the control commands for GNSS positioning.

The ioctl commands that control GNSS are listed in ioctl commands. Multiple applications can simultaneously open a GNSS device as a file. The GNSS device processes ioctl commands in the order it receives them. GNSS should arbitrate the commands issued from multiple applications. Otherwise, GNSS might run with unintended settings.

Startup

At startup, select the type of satellite used for positioning by calling CXD56_GNSS_IOCTL_SELECT_SATELLITE_SYSTEM. Set the positioning cycle by calling CXD56_GNSS_IOCTL_SET_OPE_MODE.

When performing a hot start, specify the current position and time. See Using GNSS Backup Data for information on configuring warm start and hot start.

Start Positioning

You can specify the start mode by calling CXD56_GNSS_IOCTL_START. The following table describes the start modes.

These values are general reference values. They do not guarantee performance.
Start mode IOCTL command parameter TTFF (*1) Information to use

Cold Start

CXD56_GNSS_STMOD_COLD

> 45 sec

Discard all current position, time, and satellite orbital information. Start positioning from the beginning.

Warm Start

CXD56_GNSS_STMOD_WARM

> 20 sec

Use the current position, time, and almanac. Do not use ephemeris.

Hot Start

CXD56_GNSS_STMOD_HOT

>= 1 sec

Use the current position, time, almanac, and ephemeris.

(*1) Time To First Fix

If the data required for a “warm start” or “hot start” are not available, or it is a long time has passed since the last operation, the GNSS begins a “cold start”. “Cold start” will use the position data that is available.

Stop Positioning

To stop positioning, call CXD56_GNSS_IOCTL_STOP. It takes 200 to 300 milliseconds to stop the positioning.

Position Calculation Notification

There are two types of notifications:

  • Poll method

  • Signal method

To make the application poll the GNSS for its calculated position, use CONFIG_CXD56_GNSS_NPOLLWAITERS to set the number of tasks that must elapse before sending each poll request to the GNSS.

To have the GNSS signal its position to the application, call the ioctl command CXD56_GNSS_IOCTL_SIGNAL_SET with the data cxd56_gnss_signal_setting_s set to GNSS device descriptor for fd, 1 for enable, to any signal number for signo and to CXD56_GNSS_SIG_GNSS for gnsssig. Use this command to specify which signal the GNSS issues on each event, such as positioning. The application can make it wait with sigwaitinfo. If it is not necessary to receive a signal, set the enable param to 0 and call the ioctl command CXD56_GNSS_IOCTL_SIGNAL_SET. Your application can receive signals with a specified number of tasks by setting the configuration value CONFIG_CXD56_GNSS_NSIGNALRECEIVERS.

The application example program shows how to switch the notification method with CONFIG_EXAMPLES_GNSS_USE_SIGNAL.

11.6.4. Faster Positioning

This section explains how to use backup data and other information to fix the position quickly using hot start mode.

The diagram below shows the standard hot start flow to issue ioctl commands.

diag 85475f0cb676af49d3bd3e42a6b3a2c4
Figure 54. Standard flow for hot start using backup data
Using GNSS Backup Data

The receiver position, ephemeris, almanac, TCXO offset, and other information required for a hot start are included in the backup data. Also, using ioctl commands, you can save this backup data to a file on flash memory or other file system. When the backup data is saved, it is restored to RAM when the system boots from a power off state. The GNSS subsystem can then start positioning using a hot start.

Power Mode States and Backup Data

The following list describes how backup data is handled in each power mode state:

  • While in a power on state, GPS positioning is running and the backup data is saved.

  • While in a sleep state, power is supplied to the backup RAM and the real-time clock. It stays in the condition required for a hot start.

  • While in a deep sleep state, the backup data is not saved. The real-time clock is kept in PMIC.

  • While in a power off state, the backup data and the real-time clock operation are lost.

Saving Backup Data

To save backup data, the application sends the ioctl command CXD56_GNSS_IOCTL_SAVE_BACKUP_DATA. The GNSS device saves backup data to a file with the name specified by CONFIG_CXD56_GNSS_BACKUP_FILENAME.

To preserve the flash memory, be careful not to save backup data too frequently. You should save data only before system power off or deep sleep.

Invalid Backup Data

In rare cases, saved backup data might be corrupted. The system rejects invalid data using checksum calculation before it initializes data. The GNSS will start positioning using cold start instead of hot start.

Expired Data

The ephemeris and almanac have an expiration time limit. Expired data is ignored and the GNSS will use “cold start”.

GPS Time

To perform a hot start, set the current time to within 60 seconds. There are two ways to set the time:

  • Use the GPS time stored in RTC_GPS.

    RTC_GPS stores the GPS time of the most recent positioning. When positioning stops, RTC_GPS is updated with the current GPS time. The error of the GPS time is the error of the RTC clock itself. If the period from positioning stop to positioning restart is short, the GPS time has less error and can be used for a hot start.

    When you turn on the system power supply, RTC_GPS indicates the time "0h 6 - Jan - 1980". After positioning is performed once, RTC_GPS begins counting based on the GPS time. If no time setting has been done, the positioning calculation is performed based on RTC_GPS.

  • Call the ioctl command CXD56_GNSS_IOCTL_SET_TIME from the application.

Current Location

To perform a hot start, set the current location. If the application does not have the current location and does not set it on the GNSS device, the device calculates it based on the last position at which a hot start was performed.

To set the current location on the GNSS device, the application calls the ioctl command CXD56_GNSS_IOCTL_SET_RECEIVER_POSITION_ELLIPSOIDAL or CXD56_GNSS_IOCTL_SET_RECEIVER_POSITION_ORTHOGONAL.

11.6.5. Accurate Positioning

This section describes the position data field used to perform accurate positioning.

Receiver Position Data Fields

The cxd56_gnss_receiver_s structure contains fields relating to receiver positioning.

Number of Satellites
  • numsv

  • numsv_tracking

  • numsv_calcpos

  • numsv_calcvel

The fields numsv, numsv_tracking, numsv_calcpos, and numsv_calcvel are the number of visible satellites, the number of tracking satellites, the number of satellites used for position calculation, and the number of satellites used for speed calculation. The visible satellite is expected to be present in the sky from the satellite orbit information, but it actually includes obstacles and satellites that have not received the signal. Signals that weak reception and lack accuracy are tracking and are not used for calculation of positioning or speed measurement. Positioning accuracy depends not only on the strength of the signal as described later but also on the arrangement of satellites, so it can not be said unconditionally, but in general it is often possible to continue positioning with an error of several meters if the number of numsv_calcpos is kept at 6 or more It seems.

DOP for Position and Velocity

The cxd56_gnss_receiver_s structure contains the following fields:

  • posDop is the dilution of precision (DOP).

  • velIdx is the DOP for velocity.

The attributes of the posDop and velIdx fields are:

  • Type: cxd56_gnss_dop_s

  • Members: all members are of type float:

    • pDop: overall DOP

    • hDop: horizontal DOP

    • vDop: vertical DOP

    • ewDop: East-West DOP

    • nsDop: North-South DOP

  • Units: None

DOP specifies the effect of satellite geometry. velIdx is calculated by multiplying the velocity DOP by the weight coefficient.

The lower the value of DOP, the more precise the geometry is. The apparent direction of the satellite that appears in the sky may be biased by several factors. These factors include a narrow view of the sky, for example from a city street, or when few satellites are operating in the area.

When pDOP is less than 2, the satellite is considered to be uniformly present enough that a geometrically correct position can be calculated. When pDOP exceeds 5, it is difficult to calculate the correct position. When you use positioning with a large DOP over an extended time, your use case should assume low precision positioning.

Position Error Accuracy

The cxd56_gnss_receiver_s structure contains the posAcc field. The attributes of this field are:

  • Type: cxd56_gnss_var_s

  • Members: both members are of type float:

    • hVar: horizontal error accuracy

    • vVar: vertical error accuracy

  • Units: Meters

The value of posAcc is the square root of the error covariance between position and velocity. It represents the effect of noise and other inaccuracies on the position filter.

PosAcc represents the standard deviation of the position error. Satellite signal noise, multipath effect, and DOP affect the position error accuracy. If the error accuracy is higher than the system requirement, use a more sensitive antenna to improve this value, and wait for the use of the positioning results until the satellite number increases, so it is necessary to improve the reception performance.

Satellite Position Data

The cxd56_gnss_sv_s structure contains fields relating to receiver positioning.

Signal Strength

The cxd56_gnss_sv_s structure contains the sigLevel field. The attributes of this field are:

  • Type: float

  • Members: None

  • Units: dBHz

This value represents the carrier-to-noise density (C/N) of the GNSS signal in dBHz. It is also called carrier-to-noise ratio (CNR). This represents signal reception strength. C/N is different for each satellite. The greater the value, the more stable the GNSS positioning can be. If a noise source or obstacle is near the receiver, the signal value is reduced, and stable positioning is not possible. For stable positioning, the GNSS should receive five or more satellite signals at 30 dBHz or higher.

Multi-GNSS

As the number of satellites to be acquired and tracked increases as described above, the positioning accuracy tends to improve. A GNSS device can simultaneously use multiple satellite systems and increase the number of satellite signals to be used for positioning calculations.

  • GLONASS

Like the GPS, it is a Russian positioning satellite system covering the world. Position accuracy can be lower than in the case of GPS alone when GPS satellite alone can secure sufficient number of satellites and positioning calculation is performed by mixing signals of the GLONASS system with inferior accuracy 70 m compared with the standard accuracy of 20 m in the GPS system. Please note that some antennas do not support the frequency of GLONASS.

  • QZSS-L1C/A

It is a satellite signal transmitted from Japan’s Michibiki, the Quasi-Zenith Satellite System(QZSS). Since this signal is compatible with GPS L1C/A, it seems as if there are more GPS satellites. It is called this a supplement function of GPS by Michibiki. Michibiki is a 4 operational satellites (as of 2018), its orbit covering East Asia and Oceania mainly in Japan, so there is no complementary effect in other areas.

Augmentation

Positioning accuracy can be improved by using augmentation signals transmitted from WAAS or MICHIBIKI’s QZSS-L1S. The augmentation signal is based on SBAS format, and a calculation parameter for improving the accuracy of positioning calculation updated every few minutes. When using a augmentation signal, positioning with no augmentation information mixed GLONASS can not be done.

  • WAAS

WAAS is a valid augmentation satellites in United States(CONUS), Hawaii, Puerto Rico, Alaska, Canada, and Mexico. Improve the accuracy of the pseudo range (distance between satellites and receivers) obtained from GPS satellite signals.

  • QZSS-L1S

QZSS-L1S is effective only in the range of Japan. Improve the accuracy of the pseudo range (distance between satellites and receivers) obtained from the GPS satellite signal and QZSS-L1C/A. Satellites with a low elevation angle (elevation mask at 20 degrees as of September 2018) will not receive augmentation information and can not be used for positioning.

Select positioning and augmentation satellite systems

GNSS device is assumed to perform positioning with the following positioning satellite system and combination of augmentation signals. It is able to set which satellite system to use with CXD56_GNSS_IOCTL_SELECT_SATELLITE_SYSTEM .

The following positioning error (Accuracy) is a reference value under general good conditions. It varies depending on your system and environment.
GPS GLONASS QZSS-L1C/A WAAS QZSS-L1S 95% Accuracy Effective place

x

<5m

under the open sky

x

x

<5m

in East Asia and Oceania

x

x

<7.8m

in the city

x

x

x

<7.8m

in the city of East Asia and Oceania

x

x

<2m

in North America

x

x

x

<2m

in Japan

11.6.6. Short message delivery

GNSS devices can receive disaster (disaster / crisis) reports such as disaster information and crisis management information sent from QZSS Michibiki.

Applications can signal notifications from GNSS devices asynchronously by setting signal notifications on GNSS devices. For disaster notification signal notification setting, cxd56_gnss_signal_setting_s which sets the file descriptor of the GNSS device in field fd, 1 for enable, arbitrary signal number for gnsssig, and CXD56_GNSS_SIG_SBAS for gnsssig Please call the ioctl function as an argument of the IOCTRL command CXD56_GNSS_IOCTL_SIGNAL_SET to associate the signal with the disaster report.

When the GNSS device receives SBAS message type 43 (Meteorological Agency Disaster Prevention Information) or 44 (Optional Information), it issues the associated signal. By signal processing with sigwaitinfo you can read the SBAS message that caused the notification by reading cxd56_gnss_sbasdata_s, CXD56_GNSS_READ_OFFSET_SBAS as offset in the data buffer and reading the GNSS device.

11.6.7. Utilities

NMEA Converter

The gpsutils library converts binary format position data read from CXD56xx GNSS device to NMEA format.

First, register the storage buffer management and output callback functions in the library. Position data read from the GNSS device is defined by the cxd56_gnss_sv_s structure. The conversion function converts this position data to NMEA format.

To enable this conversion, set CONFIG_MLIB to Y.

For details, see gnss_nmea and the application example NMEA Output.

11.6.8. Geofencing

Key Features

You can use geofencing to send a notification regarding the location of the receiver relative to a predefined region.

  • Each region is defined by latitude, longitude, and radius.

  • You can define up to twenty regions.

geofence notification
Figure 55. Geofencing transition notification

There are three types of notifications: "ENTER", "DWELL", and "EXIT". The distance between the device and the center of each region is calculated every at positioning update. You can define the dwelling period.

After defining the region, the GNSS core continues to monitor transitions. When GNSS core recognizes a state change, it sends a notification to the application.

Configuration

To enable geofencing, set GNSS device and Geofence Support to Y.

[System Type]
  CXD56xx Peripheral Support -->
    [*] GNSS device  = Y
    [*]   Geofence Support  = Y
Add Region and Setup Options

Applications open /dev/geofence to use the Geofence feature.

To add a Region and set the Geofence options, use the following ioctl commands from /dev/geofence.

CXD56_GEOFENCE_IOCTL_ADD

This command adds regions. You can define up to twenty regions. Each region is defined by:

  • Latitude

  • Longitude

  • Radius

All north latitude and east longitude values are positive. However, do not use a plus sign ("+") prefix with these values. All south latitude or west longitude values are negative. Important: Use a minus sign ("-") prefix with these values. Specify radius values in meters.

CXD56_GEOFENCE_IOCTL_SET_MODE

This command defines the dead zone and the dwelling period. Dwelling period is the duration that receiver is in the DWELL state.

State Transition
geofence transition
Figure 56. Geofence State Transition
Dead Zone

To reduce excess notifications due to position fluctuations, you can define a dead zone within a circular boundary.

The dead zone is defined as follows:

  • The ENTER boundary defines the dead zone’s inner circle

  • The EXIT boundary defines the dead zone’s outer circle

The state does not change as a device moves inside a dead zone.

geofence deadzone
Figure 57. Geofence Dead Zone
Read Geofence Transition Data

You can determine a Geofence state change using the poll method. After you add a region, the Geofence status is always notified of the position. After that, the Geofence status is notified only when the state is updated.

The transition data structure is cxd56_geofence_status_s. For more information, see the application example Geofencing Transitions.

11.6.9. PVTLog

Key Features

PVTLog is a function that logs position, velocity, and time information in the GNSS core. PVT stands for position, velocity, and time.

You can store up to 170 logs in the GNSS core. When the specified number of logs is stored, the application is notified using the signal method. For example, if you log PVT every 15 minutes, GNSS can store logs for 42 hours continuously independent of the application.

For more information on logging data, see the cxd56_pvtlog_s structure.

Configuration

When you use PVTLog, set GNSS device support to Y.

[System Type]
  CXD56xx Peripheral Support -->
    [*] GNSS device  = Y
Start and Stop Log Store

To start log store, use the ioctl command CXD56_GNSS_IOCTL_PVTLOG_START. This command sets the log store interval to the specified number of notifications. The log store interval is longer than the position interval. Logging starts when position starts.

To stop log store, use the ioctl command CXD56_GNSS_IOCTL_PVTLOG_STOP.

Notification

When the specified number of logs are saved in the log store, the application is notified by the signal method using CXD56_GNSS_SIG_PVTLOG.

After this notification, the application must read log data before the next log is saved. The log data in the GNSS core is deleted each time a new log is created.

The application can read log data at the offset location CXD56_GNSS_READ_OFFSET_PVTLOG.

The log data in the GNSS core is located in backup RAM. Data is stored there before logging begins is not deleted. To start a new log store, start logging after calling the delete command CXD56_GNSS_IOCTL_PVTLOG_DELETE_LOG.

For more information on PVT logging, see the application example PVTLog.

11.6.10. Application Examples

Notify Positioning

This example demonstrates GNSS positioning notification. It notifies using the poll or signal method. The method can be selected using the config settings CONFIG_EXAMPLES_GNSS_USE_SIGNAL or CONFIG_EXAMPLES_GNSS_USE_POLL.

Introduction

The gnss_atcmd is a sample application for evaluating GNSS functions on the Spresense SDK.

If you send a command via a serial port from a host such as a PC, the result of the NMEA sentences are outputted to that serial port. The command specification and usage of the application will be described in this document later.

@Command specification

This section explains the specifications of commands sent from the host.

The gnss_atcmd application receives a command sent from a host such as a PC, and returns the processing result to the host. NMEA is not output during the period from the command transmission until the result is sent. Please do not issue another command before the response message (Done or Err) is returned. The time from sending the command to get the command response, which depends on the type of command and the state, may take about 5 seconds in the worst case. When timeout is detected by the host controller, please expect a timeout for 5 seconds.

Command format

Command format is shown below. After sending "@" (at mark), send a command string and argument, and send a carriage return and a line feed code at the end.

@Command [Argument0] [Argument1]...<CR><LF>
Command

string of 4 characters or less. See command list for more details.

Argument

number in decimal. If the beginning of string is "0x", it represents a hexadecimal number. Depending on the command, multiple arguments are taken.

<CR><LF>

Represents CR(Carriage Return) and LF (Line Feed)

Normal response format

Normal response format is shown below. [Sent command string] and "Done" are replied.

[Command] Done<CR><LF>
Command

Received command string

<CR><LF>

Represents CR(Carriage Return) and LF (Line Feed)

Error response format

Error response format shows below. [Sent command string], "Err" and error code are replied.

[Command] Err ErrorCode<CR><LF>
Command

Received command string

ErrorCode

negative value of error code

<CR><LF>

Represents CR(Carriage Return) and LF (Line Feed)

Command sequence

If @GCD command is issued, NMEA sentences will be replied periodically after "Done" is replied.

diag 91a83a0494f9aa2169cbe3644536a9ae

If @VER command is issued, "Done" will be replied after the version information.

diag aecaf2c20740a1432f3725e6be277c10
Command list

Below is a list of commands that gnss_atcmd can process.

Command Argument Description

@AEXT

-

Stop application. Please execute this command while positioning is stopped.

@BSSL

bit mask of NMEA sentence to output

Of NMEA sentences defined in the NMEA 0183 (ver 4.00) standard, specify the NMEA sentence to output as a bit mask value to an argument. In the initial state, 0xef is set to the NMEA mask.

NMEA Bit Description

$xxGGA

0

Global Positioning System Fix Data

$xxGLL

1

Geographic Position - Latitude / Longitude

$xxGSA

2

GNSS DOP and Active Satellites

$xxGSV

3

Detailed information of satellites in view

$xxGNS

4

Fix status

$xxRMC

5

Recommended Minimum Specific GNSS Data

$xxVTG

6

Course Over Ground & Ground Speed

$xxZDA

7

Time & Date

$QZQSM

14

Disaster Crisis report message (QZSS Original Sentence)

xx means talker ID as below:

  • GP:Positioned only by GPS satellite

  • GL:Positioned only by GLONASS satellite

  • QZ:Positioned only by QZS satellite

  • GN:Positioned by multiple satellite systems

Example of command:

Set Output Only for $xxGGA

@BSSL 0x1<CR><LF>

Configure Output for $xxRMC and $QZQSM

@BSSL 0x4020<CR><LF>

@BUP

-

Save ephemeris and other information into flash. The saved data will be automatically restored the next started-up time. Please execute this command while positioning is stopped.

@GCD

-

Forces cold start positioning and outputs NMEA sentences periodically.
The output NMEA sentence depends on the NMEA mask.

@GNS

bit mask of selected satellites

Select the bit mask of satellites used for positioning.
e.g.) GPS positioning is 0x1, Hybrid is 0x3

Bit Satellite

0

GPS

1

GLONASS

2

SBAS(WAAS) reinforcement

3

QZSS L1C/A complement(Michibiki)

4

Reserved

5

QZSS L1S reinforcement(Michibiki)

Example of command:

Select GPS

@GNS 0x1<CR><LF>

Select GPS+GLONASS+QZSS L1C/A

@GNS 0xb<CR><LF>

@GPOE

<latitude[degree]>
<latitude[minute]>
<latitude[second]>
<longitude[degree]>
<longitude[minute]>
<longitude[second]>

Set present location in ellipsoidal coordinates.

Example of command:

north latitude 35°37’09”,east longitude 139°43’51”

@GPOE 35 37 09 139 43 51<CR><LF>

north latitude 33°07’19”,west longitude 117°19’18”

@GPOE 33 07 19 -117 19 18<CR><LF>

@GSR

-

If possible, start positioning using hot start and outputs NMEA sentences periodically.
The output NMEA sentence depends on the NMEA mask.

@GSTP

-

Stop positioning

@GSW

-

Forces warm start positioning and outputs NMEA sentences periodically.
The output NMEA sentence depends on the NMEA mask.

@GTIM

<Year>
<Month>
<Day>
<Hour>
<Minute>
<Second>

Set UTC time

Example of command:

2018/2/1 13:30'30"

@GTIM 2018 02 01 13 30 30<CR><LF>

2018/7/10 00:00'00”

@GTIM 2018 07 10 00 00 00<CR><LF>

@VER

-

Return the version number of all zero.

Usage of application

This chapter shows an example of operation of the gnss_atcmd application.

Configuration

In order to run this application, enable the following SDK Configuration.

CONFIG_CXD56_GNSS=y
CONFIG_GPSUTILS_CXD56NMEA_LIB=y
CONFIG_EXAMPLES_GNSS_ATCMD=y

These configurations can be automatically enabled by executing the following command.

$ ./tools/config.py examples/gnss_atcmd

In addition, the port used for command input/output can be switched by the configuration GNSS Command IO.

│ Prompt: GNSS Command IO
│   Location:
│     -> Examples
│       -> GNSS CXD5603 @command emulator example (EXAMPLES_GNSS_ATCMD [=y])
  • Example uses USB CDC tty : Use USB port on the extension board (default)

  • Example uses STDINOUT for nsh debug UART : Use USB port on the main board (UART1)

  • Example uses UART ttyS0 : Use USB port on the main board (UART1)

  • Example uses UART ttyS1 : Not supported

  • Example uses UART ttyS2 : Use UART port on the main or extension board (UART2)

After the gnss_atcmd application is executed via nsh prompt, input commands as mentioned above.

e.g. Start positioning using cold start and stop positioning

The application starts positioning using cold start with selecting GPS, Glonass and QZSS L1C/A as satellites. NMEA sentences are outputted for a short period of time. After that, the application stops positioning and exits.

nsh> gnss_atcmd       (Start application)
@GNS 0x0b↵         (Select satellites)
@GCD↵             (Cold Start positioning)

----- <NMEA output> ----

@GSTP↵           (Stop positioning)
@AEXT↵           (Finish application)
nsh>

e.g. Start positioning using hot start

At first, the application starts positioning using cold start with selecting GPS, Glonass and QZSS L1C/A as satellites. After positioning, the application stops the GNSS while keeping the power supply of Spresense. At next, the application starts positioning using hot start. Hot start will be fixed after a few seconds of starting positioning.

nsh> gnss_atcmd       (Start application)
@GNS 0x0b↵         (Select satellites)
@GCD↵             (Cold Start positioning)

----- <NMEA output> ----

@GSTP↵            (Stop positioning)
@AEXT↵           (Finish application)

nsh> gnss_atcmd       (Start application)
@GSR↵             (Hot Start positioning)

----- <NMEA output> ----

@GSTP↵           (Stop positioning)
@AEXT↵           (Finish application)
nsh>

e.g. Start positioning using hot start across Spresense power cycle

At first, the application starts positioning using cold start with selecting GPS, Glonass and QZSS L1C/A as satellites. After positioning, Spresense’s power is turned off. At next power cycle, Spresense is turned power on, and the application set current UTC time and present location. The application starts positioning using hot start. Hot start will be fixed after a few seconds of starting positioning.

nsh> gnss_atcmd       (Start application)
@GNS 0x0b↵         (Select satellites)
@GCD↵             (Cold Start positioning)

----- <NMEA output> ----

@GSTP↵            (Stop positioning)
@BUP↵              (Backup into flash)
@AEXT↵           (Finish application)

----- <Power OFF> -----

----- <Power ON> -----

nsh> gnss_atcmd       (Start application)
@GTIM 2018 11 09 02 45 05↵   (Set UTC time)
@GPOE 35 39 31 139 44 05↵  (Set present location)
@GSR↵             (Hot Start positioning)

----- <NMEA output> ----

@GSTP↵           (Stop positioning)
@AEXT↵           (Finish application)
nsh>

[[Output of Michibiki(QZSS) QZQSM sentences]] === Output of Michibiki(QZSS) QZQSM sentences

Select QZSS L1 / CA as the satellite mask, activate the QZQSM sentence with the NMEA mask, and start positioning. If the reception conditions are good, the first QZQSM sentence will be output in less than 10 seconds, and will be output every 4 seconds thereafter.

nsh> gnss_atcmd

@GNS 0x29↵

@BSSL 0x40ef↵

@GCD↵

$GPGGA,000001.00,,,,,0,00,,,,,,,*49 $GNGLL,,,,,000001.00,V,N*55
…​
$GNZDA,000008.00,06,01,1980,,*77 $QZQSM,56,9AADF260540002C3F2587F8B101962082C41A588ACB1181623500011439023C*7B $GPGGA,000009.00,,,,,0,00,,,,,,,*41
…​

Geofencing Transitions

11.7. ASMP Framework

11.7.1. General

The Spresense ASMP framework is designed for multi-core architecture processors, based on NuttX. It defines two tasks:

  • Supervisor task (runs on the main CPU)

  • Worker task (runs on the coprocessor)

diag 4cd7c8ce4ac20e4203abeee402a5cf73
Figure 58. ASMP Framework Relationship Diagram

The Worker task executes on the coprocessor, independently from the main CPU.

The Supervisor task uses the ASMP framework API to control the lifecycle of the Worker task operations. For details, see MP Task.

The ASMP framework provides the following functions for the Supervisor and Worker tasks to communicate with each other.

MP message queue is a message exchange interface similar to the POSIX message queue.

diag d2717059fdc2bcbb3223b39c19a30144
Figure 59. MP Message Queue

MP mutex provides a mechanism for exclusive control similar to pthread mutex.

diag e14bf772ca9c7d753d2e5553217419ec
Figure 60. MP Mutex

MP shared memory provides a memory area shared by each task.

diag bbeb95a67ff807e65635e69b7ab7b9b1
Figure 61. MP Shared Memory

11.7.2. Configuration

The ASMP framework requires a memory area outside the kernel management for shared memory between tasks that can be controlled by the ASMP shared memory size option. If the framework is not used, the full 1.5 MB memory can be used by the kernel.

Set the ASMP Shared memory size setting in memory blocks of 128 KB (0x20000).
[ASMP] = Y
  [ASMP shared memory size]  = 0x100000

11.7.3. Worker Task Startup Sequence

diag bd220a430c2c8af28b5ce6ab978b7bb5
Figure 62. Worker Task Startup Flow
  1. Call mptask_init to generate a Worker task.

  2. Prepare a message queue and allocated shared memory if necessary. Bind it to the Worker task using mptask_bindobj.

  3. Execute the Worker task by calling mptask_exec.

  4. Call mptask_destroy to stop the Worker task.

  5. If a message queue and shared memory was created, discard it.

For details, please refer to Supervisor Task example and Worker Task example.

11.7.4. Building a Worker Task

The Worker task loads and executes a file in ELF format. As the Worker task executes on the coprocessor, it is not able to make direct calls to the kernel or SDK API.

The layout of the ELF file supported by ASMP framework is as follows:

diag 2300eb5e02a824d83361c67ee26eeb61
Figure 63. ELF file layout for Worker task

Each program’s start address must be 0x00000000 and each section must be contiguous. The SDK provides compiler options, linker options and linker scripts necessary to generate ELF files for the Worker task. These are defined in sdk / Make.defs as CELFFLAGS and LDRAWELFFLAGS respectively. For specific build rules please refer to ASMP Worker Task 'Hello World' example.

In order to use ASMP framework with the Worker task, a library for the Worker task has to be generated before building the Worker task. This library contains the necessary code (blue colored regions in the figure) for configuring the Worker task.

Refer to Worker Task Makefile and Worker Task Library Makefile for sample code to generate a library for the Worker task.

11.7.5. Sample Code

11.8. Sensor Control Unit

11.8.1. Overview

The Sensor Control Unit (SCU) can receive data from sensor devices connected by the SPI and I2C buses. It is designed to reduce processor load and power consumption.

Your application can control the following SCU operations:

  • Automatically receives sensor data from the sequencer.

  • Performs decimation processing for received sensor data.

  • Applies IIR filters.

  • Notifies the application of events.

The SCU has the following components:

  • One SPI bus

  • Two I2C buses

  • Eight sequencers

  • Two decimators

The following diagram shows the functional relationship of the SCU with other components:

diag f05c91093debc9654558029c9c5d4693
Figure 64. SCU block diagram

11.8.2. Configuration

  CXD56xx Configuration  --->
    Sensor Control Unit (SCU)  --->
      Sequencer Sampling Predivider  = 64 (default) (1)
      SCU clock mode                 = RCOSC (default) (2)
      SCU32K clock source            = RTC (default) (3)
      SCU Decimator assignments (4)
        ...
        (Decimator supported drivers)
        ...
      SCU Debug                      = N (default)
      DMAC support                   = Y (default)
1 Sequencer Sampling Predivider sets the sequencer sampling rate.
2 SCU clock mode sets the clock to run.
3 SCU32K clock source sets the source of the SCU sampling base clock at 32768Hz.
4 You can assign the SCU Decimator assignments option to a decimator-supported sensor. You can enable three or more sensors. open() fails if you call it when two decimators are in use.

11.8.3. Sequencer

The SCU has two sequencer types:

  • The normal sequencer receives sensor data periodically and saves it to FIFO.

  • SCU uses a sequencer device driver that reads from FIFO.

diag c1785ba91bc9f1ad5a61bda8b03d2997
Figure 65. Sequencer block diagram

Each sequencer’s FIFO is assigned a configurable size using SCUIOC_SETFIFO. The total FIFO memory is 40KB. Your application must set this before using the SCU supported driver.

ret = ioctl(fd, SCUIOC_SETFIFO, sizeof(struct three_axis_s) * 128);

The sequencer chooses the sensor data by sampling the rate independent of the sensor device. Your application must set the sequencer sampling rate using SCUIOC_SETSAMPLE.

ret = ioctl(fd, SCUIOC_SETSAMPLE, 2);

SCUIOC_SETSAMPLE is the binary clock divider.

To calculate the sequencer sampling rate:

Sequencer sampling rate = 32768 / CONFIG_CXD56_SCU_PREDIV / (2 ^ n)

For example, to set the sequencer sampling rate to 128 Hz:

  • Set CONFIG_CXD56_SCU_PREDIV to 64 (the default value)

  • Set SCUIOC_SETSAMPLE to 2

The sequencer sampling rate is different from the sensor sampling rate. Your application must set separate values for each rate.

The SCU driver signals the application when the sampling data in the sequencer FIFO reaches the watermark value SCUIOC_SETWATERMARK.

For example, to make the SCU signal every 1 second:

  • Set the sampling rate to 128 Hz

  • Set the watermark to 128

wm.signo = CONFIG_EXAMPLES_MAG_SIGNO;
wm.ts = &ts;
wm.watermark = 128;

ret = ioctl(fd, SCUIOC_SETWATERMARK, (unsigned long)(uintptr_t)&wm);

The siginfo data structure contains the timestamp of the watermark signal.

11.8.4. Decimator

The decimator is a sequencer feature that performs decimation processing. Your application must handle sensor data from only one sensor device. The decimator has three FIFOs for storing sensor data.

diag 0b3cd224d7ba24a3341d235eafb78bcc
Figure 66. Decimator block diagram
Decimation Processing

The decimator processes sample data at the sequencer sampling rate. The decimation process applies a CIC filter. Your application can set the decimation parameters using SCUIOC_SETDECIMATION using the decimation_s data structure:

struct decimation_s dec;

dec.ratio = 1;
dec.leveladj = SCU_LEVELADJ_X1;
dec.forcethrough = 0;

ret = ioctl(d->fd, SCUIOC_SETDECIMATION, (unsigned long)(uintptr_t)&dec);

The ratio member specifies the pick up ratio against the sampling rate in exponents of 2. For example, if the sampling rate is 64 Hz and ratio is set to 1 ( 64 / (2 ^ 1) ), the effective sampling rate is 32 Hz. Setting ratio to 0 makes no change to the sampling rate, but the decimation process applies CIC filter logic to the sample data. To disable the CIC filter, set forcethrough to 1.

Amplification

Decimation can amplify sampling data using the leveladj member. Amplified sampling data is saturated to its data width.

leveladj can use these constants:

Supported devices

The decimator requires sensor driver support. Currently supported sensors:

  • KX022

  • BMI1422GMV

11.8.5. Sensor Data Processing

The SCU has sensor data processing for taking sampling data. The sensor data processing unit contains:

  • An IIR filter

  • An event detector

Your application can specify 1 to 3 sensor data processing units. Each unit is connected to a FIFO.

IIR Filter

The SCU can use two IIR filters for sampling data.

Your application can set the IIR filters to any combination of positions. FIFO is the sequencer’s decimator FIFO. Event Detector is an SCU function.

The IIR filter position settings are shown in the following diagram:

diag ad9db3a7d27d9a0f8bdd7ae0a7ff047d
Figure 67. IIR filter path
  • A - Apply SCU to FIFO and event detector

  • F - Apply SCU only to FIFO

  • E - Apply SCU only to event detector

You can apply the two IIR filters as follows:

  • Both filters to FIFO and event detector

  • One filter to both FIFO and event detector, one to FIFO

  • One filter to both FIFO and event detector, one to event detector

  • Both filters to FIFO

  • Both filters to FIFO and Event detector

  • Both filters to event detector

IIR filter position settings are defined by the enumeration filter_pos_e.

The IIR filter can be configured using coefficients:

IIR filter block diagram
Figure 68. IIR filter block diagram

Each coefficient is a 34-bit fixed point number in s2.31 format. IIR filter coefficients are structured as 32-bit (h) and 8-bit (l) values in the iir_coeff_s data structure:

The coefficient structure is shown in the following diagram:

struct_iir_coeff_s
Figure 69. struct iir_coeff_s h, l

Bit 31 of h is S, which specifies the coefficient’s sign: 0 = plus, 1 = minus.

Bits 30 to 29 of h are the coefficient’s integer value.

Bits 28 to 0 of h and bits 7 to 6 of l are the coefficient’s fractional part.

Event Detector

The event detector is a monitoring feature of sensor data processing. Your application can determine the timing of the sensor data.

The event detector counts input sensor data as high or low compared to a configured threshold. The event detector can handle 16 bits per sample and 1-, 2-, or 3-axis data. If the input is 2- or 3-axis data, it normalizes this data before counting. If the counted value reaches the threshold, the event detector sends an interrupt. Your application receives this event as a signal with a timestamp.

The normalization calculation depends on the number of axes:

  • For 1 axis (X):

  norm = abs(x)
  • For 2 axes (X, Y):

 norm = max(abs(x), abs(y)) * 123 / 128 + min(abs(x), abs(y)) * 51 / 128
  • For 3 axes (X, Y, Z):

  L1   = max(abs(x), abs(y)) * 120 / 128 + min(abs(x), abs(y)) * 49 / 128
  norm = max(L1, abs(z)) + min(L1, abs(z)) * 44 / 128

You configure the event detector with the scuev_notify_s data structure. The rise and fall members control event detection. Each of these members have:

  • threshold

  • count0

  • count1

The SCU notifies the application when the input data count reaches count0 + count1.

The event detector requires IIR filter output. You must configure the IIR filter before you configure the event detector.

The event detector works according to these rules:

  • Count values that are greater than the specified threshold.

  • If the count reaches count0, begin to count.

  • If the input data falls below the specified threshold before reaching count0, stop and reset the count.

  • If the total count reaches count0 + count1, send a rise event.

  • If count1 is zero, send a notification when the count reaches count0.

  • If the or count0 is 0, ignore the configuration.

scu event
Figure 70. Event detection

11.8.6. Restrictions

The decimator, IIR Filter, and event detector can be programmed only for the following data formats:

  • 16-bit data per sample

  • 1, 2, or 3 elements

The decimator can accept any data format, but processes only data duplication when it is given data other than the above formats.

11.8.7. For Driver Developers

Developers of SCU-supported drivers must:

  • Understand sequencer instructions

  • Configure the sensor device using a special transfer API

  • Open the sequencer and connect with the sensor driver

  • Read sampling data from the sequencer

  • Pass ioctl commands to the sequencer

Understanding Sequencer Instructions

The SCU sequencer sends and receives data through SPI and I2C buses using a specific instruction format. Specify sequencer instructions as an array of uint16_t values. Use the SCU_INST_SEND and SCU_INST_RECV macros to set sequencer instruction values.

This series of instructions tells the sequencer how to transfer data with the sensor device. Include the special termination indicator SCU_INST_LAST at the end of the instruction array.

Examples of instruction arrays:

inst[0] = SCU_INST_SEND(0x00);                  // Send register address 0x00
inst[1] = SCU_INST_RECV(2) | SCU_INST_LAST;     // Read 0x00 and 0x01 address data, and indicate the last of instructions
Configure Sensor Device with Transfer API

Initialize the driver for the target device. The SCU supported driver must use scu_spitransfer or scu_i2ctransfer to access device registers. Use these functions with the sequencer instruction macro SCU_INST_SEND, as shown in this register write access example:

Ex. Register write access function
static void sensor_putreg8(FAR struct sensor_dev_s *priv, uint8_t regaddr, uint8_t regval)
{
  uint16_t inst[2];

  /* Send register address and set the value */

  inst[0] = SCU_INST_SEND(regaddr);
  inst[1] = SCU_INST_SEND(regval) | SCU_INST_LAST;

  scu_i2ctransfer(priv->port, priv->addr, inst, 2, NULL, 0);
}
The SPI and I2C bus contolled by SCU can be accessed directly. If you access them from the SCU and sensor driver through the same bus at the same time, it causes a bus conflict. Use the APIs as shown above to avoid conflicts.
Open/Close Sequencer

The SCU driver handles sequencer object operations. Each sensor driver must include a sequencer object. In this example, a sequencer object is stored in device data:

g_seq = seq_open(MAG_SEQ_TYPE, SCU_BUS_I2C0);
if (!g_seq)
  {
    return -ENOENT;
  }
priv->seq = g_seq;

The decimator can be read from three FIFOs. Create three device files to read data from different applications. In this case, you must call seq_open() once, because the decimator replicates sensor data to three FIFOs. Count the number of references to avoid multiple initializations.

if (g_refcnt == 0)
  {
    int ret;
    ret = sensor_seqinit(priv);
    if (ret < 0)
      {
        return ret;
      }
  }
else
  {
    /* Set existing sequencer */
    priv->seq = g_seq;
  }
g_refcnt++;

The SCU can sleep when no sequencers are running. The SCU sleep function supports seq_open() and seq_close(). Call these functions from the open() and close() driver interfaces respectively. The driver can also use them to control device power mode and reduce power consumption.

Read Sampling Data from Sequencer

Set the instructions for the sequencer to receive sensor data using seq_setinstruction:

Examples of reading a sample data of SENSOR_BYTESPERSAMPLE from SENSOR_DATA_REGISTER:

static const uint16_t g_sensorinst[] =
{
  SCU_INST_SEND(SENSOR_DATA_REGISTER),
  SCU_INST_RECV(SENSOR_BYTESPERSAMPLE) | SCU_INST_LAST,
};
seq_setinstruction(priv->seq, g_sensorinst, itemsof(g_sensorinst));

Sensor data is sent to the FIFO that is assigned to the sequencer. The driver can read from its FIFO using seq_read(). The driver reads sensor data and stores it in the application buffer:

len = seq_read(priv->seq, priv->id, buffer, len);
Pass ioctl Commands to Sequencer

To pass SCU ioctl commands from sensor drivers to the sequencer, use seq_ioctl(). Call seq_ioctl() from each driver’s ioctl command if it is a valid SCU IO command. Use the _SCUIOCVALID() macro to determine whether the command is valid.

For example:

if (_SCUIOCVALID(cmd))
{
  /* Redirect SCU commands */
  ret = seq_ioctl(priv->seq, priv->id, cmd, arg);
}

11.9. Memory Utility Libraries

11.9.1. Overview

The media & sensor processing functions requires safe handling of big memory chunks without causing leakage in the form of fragmentation. For this reason Spresense SDK provides a memory utility library.

The library consists of the following three libraries, "Memory Manager", "Message Library", "Simple FIFO".

  • The "Memory Manager" manages memory between tasks.

    • It has a fixed size memory pool library that implicity manages segment areas.

  • "Message Library" is a library for sending and receiving class objects between tasks in order to use "Memory Manager".

  • "Simple FIFO" is the basic FIFO used to transfer data between applications and a framework.

The use of these libraries is premised on media processing & sensor processing. Details are described in the following chapters.

11.9.2. Memory Manager

This chapter will explain the function of the "Memory Manager".

Overview

"Memory Manager" is a fixed-size memory pool that manages implicit acquisition and release of memory. It is possible to acquire and release the memory as necessary by defining the memory area as a memory pool and securing needed segments in it.

To avoid memory fragmentation a pool type, number of segments and allocation memory area can be defined.

This allocation information is called memory_layout, this layout can be switched according to the needs of the application and securing the necessary memory for each function.

This memory_layout can be freely decided and created according to the needs of the application. How to create the memory_layout is described in Configurations and Generate.

Acquiring the needed memory segment is done by creating a MemHandle object and calling the API to clam the segment. Claimed segments this way are guaranteed as long as the MemHandle object isn’t destroyed. This is also guaranteed in the case that multiple users claim and release MemHandle asynchronously until all users have released the MemHandle.

Mechanism of segment acquisition and release

Each instance of MemHandle need to point to the a segment of the required memory area. By calling the API that will acquire the segment area will secure the memory segment that is associated with the handle.

When the segment has been acquired, the reference counter of the segment is increased by 1. As a result, the segment area is in use and managed so that it can not be used from other requests.

Get memHandle

The reference counter will also be incremented by 1 by the operator = and the Copy Constructor.

By copying this instance of MemHandle to the object located next to the data pipeline and passing it, you can refer to segments attached to "MemHandle" and the area is guaranteed.

Copy memHandle

When MemHandle isn’t needed anymore, the reference counter will be decremented by -1 by the Destructor. When the reference counter is down to 0, all linked memory segments will be released.

Use memHandle

This mechanism enables it to safely allocate and free shared memory asynchronously between different tasks.

Release memHandle

To use memory_layout a header file group has to be prepared in advance. These header files are created by creating a Memory Layout definition file (mem_layout.conf) and using a dedicated tool(mem_layout.py).

APIs

The interface of "Memory Manager" is as follow. For details, see memutils_memory_manager.

Functions of the Manager class
Initializing the MemMgrLite library
Function
static err_t Manager::initFirst(void* lib_area, uint32_t area_size)
argument
void * lib_area: Data area used by the library
uint32_t area_size: size of area (in bytes)
Return value
ERR_OK: initialization succeeded
ERR_STS: This function is executed more than once
ERR_DATA_SIZE: area_size is less than sizeof(MemMgrLite::Manager)
ERR_ADR_ALIGN: lib_area is not a multiple of 4
Description
Initialize the entire library.
When the fence function is effective, initialize the fence of FixedArea.
Before using other APIs in this library,
// once with a single CPU agreed in advance
to execute this function only.

The argument lib_area specifies the address of the data area for the library.
If the address is not a multiple of 4, ERR_ADR_ALIGN is returned.
The area to be specified is an area having a permanent life. In terms of performance, SRAM
High speed memory is desirable.

The area_size argument specifies the size of lib_area in units of bytes.
If the size is less than sizeof (MemMgrLite::Manager), ERR_DATA_SIZE is returned.
Since the size required for the current implementation is (4 + 4 * NUM_MEM_POOLS), a minimum of 12 bytes
The maximum is 1028 bytes.
//If you use the optional dynamically generated pool, you can add an additional (8 + 5 * NUM_DYN_POOLS)
A size rounded up to a multiple of 4 is required.

//When using this library with multi-core (shared pool) support, this area is it must be accessible from all CPUs.
//Since ARM TCM, MIPS ScratchPad, etc. are CPU local memory, multicore it can not be specified for lib_area at the time of support.
  1. Usage example 1 An example of defining and using the MemMgrLite data area in the memory layout definition file is shown below.

//S_ASSERT (MEMMGR_DATA_AREA_SIZE> = sizeof(MemMgrLite::Manager));
void * lib_data_va = MemMgrLite::translatePoolAddrToVa(MEMMGR_DATA_AREA_ADDR);
if(MemMgrLite::Manager::initFirst(lib_data_va, MEMMGR_DATA_AREA_SIZE))! = ERR_OK)
  {
    / * Error handling * /;
  }
  1. Usage example 2 Here is an example that uses the data area of MemMgrLite defined by a static variable.
    Since area can be secured with sizeof, there is no worry about size shortage. Note that the area is defined as an array of uint32_t to guarantee 4-byte alignment.

static uint32_t s_lib_data [sizeof (MemMgrLite :: Manager) / sizeof (uint32_t)];
if(MemMgrLite::Manager::initFirst(s_lib_data, sizeof (s_lib_data))! = ERR_OK)
  {
    / * Error handling * /;
  }
Initialization per CPU
function
static err_t Manager::initPerCpu (void * lib_area)
argument
void* lib_area / * Data area used by library * /
Return value
ERR_OK: initialization succeeded
ERR_STS: This function is executed more than once, or initFirst() is not executed yet
Description
Initialize each CPU library.
The argument lib_area specifies the same address as specified in Manager::initFirst().

If Manager::initFirst() is not executed yet, or if this API is executed again on a CPU that has already executed this API, ERR_STS is returned.

//When using this library with multi-core (shared pool) support, all CPUs Wait for Manager::initFirst () to complete execution (in a platform-specific manner)
//It is necessary to execute this function.
  1. Usage example 1 An example of using the data area of MemMgrLite defined in the memory layout definition file is shown below.

void * lib_data_va = MemMgrLite :: translatePoolAddrToVa (MEMMGR_DATA_AREA_ADDR);
if (MemMgrLite :: Manager :: initPerCpu (lib_data_va))! = ERR_OK)
  {
    / * Error handling * /;
  }
  1. Usage example 2 Here is an example that uses the data area of MemMgrLite defined by a static variable.

if (MemMgrLite::Manager::initPerCpu (s_lib_data))! = ERR_OK)
  {
    / * Error handling * /;
  }
Generating a static memory pool
function
static err_t Manager::createStaticPools(NumLayout layout_no,
void * work_area,
uint32_t area_size,
const PoolAttr * pool_attr)
argument
NumLayout layout_no: Memory layout number (0 origin)
void * work_area: Work area for static memory pool operation
uint32_t area_size: size of the work area (in bytes)
const PoolAttr * pool_attr: Address of memory layout
Return value
ERR_OK: initialization succeeded
ERR_STS: initPerCpu() has not been executed or this function has been executed
ERR_ADR_ALIGN: work_area is not a multiple of 4
ERR_DATA_SIZE: area_size is less than maximum work area size
Description
Generate a memory pool group with the specified layout number.
To change the memory layout, once with Manager::destroyStaticPools()
Discarding the memory pool group and then executing this function again.

If Manager :: initPerCpu () is not executed yet,or if this function has already been executed, ERR_STS is returned.

The argument layout_no specifies the number of the memory layout to be used.

The argument work_area specifies the address of the work area for static memory pool operation.
If the address is not a multiple of 4, ERR_ADR_ALIGN is returned.
The specified area exists until Manager::destroyStaticPools() is called must be an area. In terms of performance, high-speed memory such as SRAM is desirable.

The area_size argument specifies the size of the work area for static memory pool operations.
Macro defined for each layout number MEMMGR_Lx_WORK_SIZE (x is a layout number)
Specify the above values.
The maximum work area size in the entire layout is defined by the macro MEMMGR_MAX_WORK_SIZE.
If the size of the work area is insufficient, ERR_DATA_SIZE is returned.

//When using this library with multi-core (shared pool) support, this area is it must be accessible from all CPUs.
//Since ARM TCM, MIPS ScratchPad, etc. are CPU local memory, multicore it can not be specified when supporting.
Spresense can select only SRAM.
Destroying (releasing) static memory pool
function
static void Manager :: destroyStaticPools ()
argument
None
Return value
None
Description
Discard the static memory pool generated by Manager::createStaticPools().
When static memory pool is not created, do nothing.
If Manager::initPerCpu() has not been executed, "debug_assert" it.

When the memory pool is destroyed, the segment release omission is checked.
If release leakage is detected, "assert".

When the fence check function is valid, a fence check of the static memory pool is performed and if fence failure is detected, "assert" it.
Example of use
/* Discard static memory pool */
MemMgrLite::Manager::destroyStaticPools();
Get the current memory layout number
function
static NumLayout Manager :: getCurrentLayoutNo()
argument
None
Return value
Currently set memory layout number
If not set, BadLayoutNo
Description
Get the current memory layout number.
The initial value of the memory layout number is BadLayoutNo.
When you call Manager::createStaticPools(), the value specified by the argument is set.
Calling Manager::destroyStaticPools() will reset it to its initial value.
Example of use
/* Get the current memory layout number */
MemMgrLite::NumLayout layout_no = MemMgrLite::Manager::getCurrentLayoutNo();
if (layout_no! = MemMgrLite :: BadLayoutNo)
  {
    printf ("Current memory layout number:% d \ n", layout_no);
  }
else
  {
    printf ("Memory layout number not set \ n");
  }
Acquiring Memory Segment in Use
function
static uint32_t Manager::getStaticPoolsUsedSegs(MemHandle * mhs, uint32_t num_mhs)
//static uint32_t Manager::getUsedSegs(PoolId id, MemHandle * mhs, uint32_t num_mhs)
argument
PoolId id: Pool ID
MemHandle * mhs: Memory handle array storing the memory segment in use
uint32_t num_mhs: Number of elements of the array
Return value
Number of segments stored in memory handle array
Description
Store the segment in use in the memory pool in the memory handle array.
The reference count of the stored segment increases by one.
When the number of segments in use is larger than the value specified by the argument num_mhs, the processing is terminated at the time of storing the segment.
getStaticPoolsUsedSegs() targets the entire static pool of the current memory layout.
//getUsedSegs() targets static or dynamic memory pools with the specified ID.

The argument mhs specifies an empty memory handle array that does not hold memory segments.
If the argument id is not a valid pool ID, "debug_assert".
"debug_assert" if argument mhs or num_mhs is 0.

Before destroying the memory pool, these APIs can be used to specify details of unreleased segments
It assumes a purpose of acquiring information.

If you only know the number of segments in use, the following one is more efficient.
Manager::getPoolNumSegs(id) - Manager::getPoolNumAvailSegs(id)
Example of use
const uint 32 _ t MaxSegInfo = 8;
MemHandle mhs [MaxSegInfo];

uint32_t num_used = Manager::getStaticPoolsUsedSegs(mhs, MaxSegInfo);
for (uint32_t i = 0; i<num_used; ++i)
  {
    printf ("ID =% u SegNo =% u VA =% 08x Size =% u RefCnt =% u \ n"
             mhs[i].getPoolId(), mhs[i].getSegNo(), mhs[i].getVa(),
             mhs [i].getSize(), mhs[i].getRefCnt());
  }
Acquire various information of memory pool
Function
static bool Manager::isPoolAvailable(PoolId id)
static PoolType Manager::getPoolType(PoolId id)
static PoolAddr Manager::getPoolAddr(PoolId id)
static PoolSize Manager::getPoolSize(PoolId id)
static NumSeg Manager::getPoolNumSegs(PoolId id)
static NumSeg Manager::getPoolNumAvailSegs(PoolId id)
static bool Manager::isPoolFenceEnable(PoolId id)
static LockId Manager::getPoolLockId(PoolId id)
Argument
PoolId id /* memory pool ID (1 origin) */
Return value
See Description
Description
isPoolAvailable() returns whether or not the specified pool ID is valid in the current memory layout.

getPoolType(), getPoolAddr(), getPoolSize(), getPoolNumSegs()
Returns pool type, address, size, number of segments specified at creation time.

getPoolNumAvailSegs() returns the current number of free segments.

isPoolFenceEnable() returns the fence specification specified at pool creation time.
This function is defined only when fence function is enabled (UseFence = True).

//getPoolSpinLockId() returns the spin lock ID specified at pool creation time.
This function is defined only when multicore support is enabled (UseMultiCore = True).

Since specifying an invalid pool ID will cause a NULL pointer access,
Confirm effectiveness and use.
Example of use
if (MemMgrLite::Manager::isPoolAvailable(my_pool_id))
  {
    printf ("type =% d addr =% 08x size =% 08x num_seg =% u avail_seg =% u",
             MemMgrLite::Manager::getPoolType(my_pool_id),
             MemMgrLite::Manager::getPoolAddr(my_pool_id),
             MemMgrLite::Manager::getPoolSize(my_pool_id),
             MemMgrLite::Manager::getPoolNumSegs(my_pool_id),
             MemMgrLite::Manager::getPoolNumAvailSegs(my_pool_id));
# ifdef USE_MEMMGR_FENCE
    printf("fence =% u", MemMgrLite::Manager::isPoolFenceEnable(my_pool_id));
# endif
//# ifdef USE_MEMMGR_MULTI_CORE
//printf("spl_id =% u", MemMgrLite::Manager::getPoolSpinLockId(my_pool_id));
//# endif
  }
Functions of the MemHandle class
Constructor / Destructor
function
MemHandle::MemHandle() // default constructor
MemHandle::MemHandle(PoolId id, size_t size) // segment allocate constructor
MemHandle::MemHandle(const MemHandle & mh) // copy constructor
MemHandle::~MemHandle() // destructor
argument
PoolId id: Pool ID
size_t size: Request size (in bytes)
const MemHandle & mh: Source memory handle
Return value
None
Description
Creates or destroys an instance of MemHandle class.

The argument id specifies the pool ID from which the memory segment is acquired.
The acquisition result is determined by isAvail() or isNull().

The size argument specifies the requested size.
Actually, this argument is used only for comparison with segment size, and segment size
If a value exceeding the specified value is specified, it is "debug_assert".

The argument mh specifies the copy source memory handle.
When the copy source memory handle holds a memory segment
The reference count of the memory segment increases by one.

The destructor of the instance holding the memory segment is stored in the
Decrement the reference count by one.
Example of use
MemMgrLite::MemHandle mh(MY_POOL_ID, sizeof(MyData));
if(mh.isNull())
  {
    /* Error handling */;
  }
Assignment operator
function
MemHandle::MemHandle& operator = (const MemHandle & mh)
argument
const MemHandle& mh: Source memory handle
Return value
Reference to itself
Description
Assigns an instance when its value is different from the value of the copy source.
In the same case do not do anything.

If a memory segment is held, it is necessary to set memory segment
Decrement the reference count by one.

When the copy source memory handle holds a memory segment
Increment the reference count of memory segments by 1 after assignment.
Example of use
MemMgrLite::MemHandle mh; /* default constructor */
mh = src_mh; /* call operator = () */
Memory segment acquisition
function
err_t MemHandle::allocSeg(PoolId id, size_t size, MemHandleProxy& proxy)
argument
PoolId id: Pool ID
size_t size: Request size(in bytes)
MemHandleProxy& proxy: Memory segment reference
Return value
ERR_OK: Successful acquisition
ERR_DATA_SIZE: size exceeds segment size
ERR_MEM_EMPTY: There are no segments available
Description
Get memory segment from the specified memory pool.
The argument id specifies the pool ID from which the memory segment is acquired.

The size argument specifies the requested size.
Actually, this argument is used only for comparison with segment size, and if a value exceeding the specified value is specified, ERR_DATA_SIZE is returned.

If there is no segment that can be acquired, ERR_MEM_EMPTY is returned.
Example of use
MemMgrLite::MemHandle mh;
if (mh.allocSeg(MY_POOL_ID, sizeof(MyData))! = ERR_OK)
  {
    /* Error handling */;
  }
Explicit memory segment release
function
void MemHandle::freeSeg()
argument
None
Return value
None
Description
Explicitly free memory segments without relying on destructors.
If the memory segment is not held, do nothing.
When holding a memory segment, the reference count of the memory segment is set to decrease by 1 and reinitialize the instance.
Acquire various information of memory segment
function
bool MemHandle::isAvail()
bool MemHandle::isNull()
bool MemHandle::isSame(const MemHandle & mh)
PoolId MemHandle::getPoolId()
NumSeg MemHandle::getSegNo()
PoolAddr MemHandle::getAddr()
PoolSize MemHandle::getSize()
SegRefCnt MemHandle::getRefCnt()
argument
const MemHandle & mh: comparison memory handle
Return value
See Description
Description
isAvail() returns whether or not the instance holds a memory segment.

isNull() returns whether or not an instance holds a memory segment.

isSame() returns whether or not the instance and argument mh are the same value.

If getPoolId() holds a memory segment for the instance,returns the ID of the pool to which the segment belongs.
If it does not hold a segment, it returns NullPoolId.

If getSegNo() holds the memory segment for the instance, returns the segment number (1 origin) in the pool to which the segment belongs.
If it does not hold a segment, it returns NullSegNo.

If getSegAddr() holds the memory segment for the instance, returns the address of the segment.
If it does not hold a segment, it is "debug_assert".

If getSegSize() holds the memory segment for the instance, returns the size of the segment.
If it does not hold a segment, it is "debug_assert".

If getRefCnt() holds memory segments for the instance, returns the reference count of the segment.
If it does not hold a segment, it is "debug_assert".
Defining user constants

If you want to add user’s unique definitions, you would be recommand to define that starts with "U_" in the MemoryLayout file.

Example of writing user-defined constants

  # Start with "U_" so that it does not overlap with the definition in the script and let's only capital letters, numbers and "_"
  # When defined with a name that begins with "U_MEM_", macros with the same name are output to the header file

  U_STD_ALIGN = 8 # standard alignment
  U_MSGQ_ALIGN = 64 # message queue area alignment
Device definition of Memory

Define various Memory devices. These definitions are used to define fixed areas. It is output as a name_ADDR macro and a name_SIZE macro in the header file.

Memory device definition description example
MemoryDevices.init (
  # name ram addr size
  ["AUD_SRAM", True, 0x000c0000, 0x00040000],
  None # end of definition
)

The explanation of each parameter is as follow.

name device name (3 or more characters, starting with upper case letters, capital letters, numbers, "_" can be used)

ram

True if the device is RAM. Otherwise False.

addr

address of the region. Specify a value of a multiple of 4 except 0.

size

size of the region. Specify a value of a multiple of 4 except 0.

Definition of fixed area

Define an area for each memory device. The start address of each area is determined by the cumulative size, align, fence for each device.
It is output as a name_ALIGN, name_ADDR, name_SIZE macro in the header file.

Description example of fixed area definition

FixedAreas.init (
  # name, device, align, size, fence
  ["AUDIO_WORK_AREA", "AUD_SRAM", U_STD_ALIGN, 0x0003e000, False], # Audio work area
  ["MSG_QUE_AREA", "AUD_SRAM", U_STD_ALIGN, 0x00001000, False], # message queue area
  ["MEMMGR_WORK_AREA", "AUD_SRAM", U_STD_ALIGN, 0x00000200, False], # MemMgrLite WORK Area
  ["MEMMGR_DATA_AREA", "AUD_SRAM", U_STD_ALIGN, 0x00000100, False], # MemMgrLite DATA Area
  None # end of definition
)

The explanation of each parameter is as follow.

Parameter Description

name

area name (name starting with uppercase letters and ending with "_ AREA", uppercase letters, numbers, _ can be used)

device

Device name of MemoryDevices to reserve space

align

Start alignment of the region. Specify a multiple of MinAlign (= 4) except 0.

size

size of the region. Specify a value of a multiple of 4 except 0.

fence

Specify whether the fence is valid or invalid. (This item is ignored if UseFence is False)

Definition of PoolLayout

Define the memory pool layout. The start address of each pool area is determined by the cumulative size, align, fence for each fixed area.
In the header file, pool ID and NUM_MEM_POOLS, NUM_MEM_LAYOUTS and Lx_name_ALIGN, Lx_name_ADDR, Lx_name_SIZE, Lx_name_NUM_SEG, Lx_name_SEG_SIZE are output as a macro. (x is the layout number)
If the fence is valid, the Lx_name _ L _ FENCE and L x _ name _ U _ FENCE macros are also output.

Description example of .PoolLayout definition

# Definition for player
U_MAIN_BUF_SIZE = 1024
U_MAIN_BUF_SEG_NUM = 4
U_MAIN_BUF_POOL_SIZE = U_MAIN_BUF_SIZE * U_MAIN_BUF_SEG_NUM

U_PCM_BUF_SIZE = 1024
U_PCM_BUF_SEG_NUM = 8
U_PCM_BUF_POOL_SIZE = U_PCM_BUF_SIZE * U_PCM_BUF_SEG_NUM

# Definition for recorder
U_REC_BUF_SIZE = 2048
U_REC_BUF_SEG_NUM = 6
U_REC_BUF_POOL_SIZE = U_REC_BUF_SIZE * U_REC_BUF_SEG_NUM

PoolAreas.init (
  [# layout 0 for Player
    # [name, area, align, pool-size, seg, fence]
      ["MAIN_BUF_POOL", "AUDIO_WORK_AREA", U_STD_ALIGN, U_MAIN_BUF_POOL_SIZE, U_MAIN_BUF_SEG_NUM, True],
      ["PCM_BUF_POOL", "AUDIO_WORK_AREA", U_STD_ALIGN, U_PCM_BUF_POOL_SIZE, U_PCM_BUF_SEG_NUM, True],
    None # end of each layout
  ], # end of layout 0

  [# layout 1 for Recorder
    # [name, area, align, pool-size, seg, fence]
      ["REC_BUF_POOL", "AUDIO_WORK_AREA", U_STD_ALIGN, U_REC_BUF_POOL_SIZE, U_REC_BUF_SEG_NUM, True],
    None # end of each layout
  ], # end of layout 1

  None # end of definition
)

Up to three memory pools can be defined in different areas.

Example to define PoolLayout for Audio and PoolLayout for Sensor.
# Definition for player
U_MAIN_BUF_SIZE = 1024
U_MAIN_BUF_SEG_NUM = 4
U_MAIN_BUF_POOL_SIZE = U_MAIN_BUF_SIZE * U_MAIN_BUF_SEG_NUM

U_PCM_BUF_SIZE = 1024
U_PCM_BUF_SEG_NUM = 8
U_PCM_BUF_POOL_SIZE = U_PCM_BUF_SIZE * U_PCM_BUF_SEG_NUM

# Definition for recorder
U_REC_BUF_SIZE = 2048
U_REC_BUF_SEG_NUM = 6
U_REC_BUF_POOL_SIZE = U_REC_BUF_SIZE * U_REC_BUF_SEG_NUM

# Sensor
U_SENSOR_DSP_CMD_SIZE       = 0x300
U_SENSOR_DSP_CMD_SEG_NUM    = 8
U_SENSOR_DSP_CMD_POOL_SIZE  = U_SENSOR_DSP_CMD_SIZE  * U_SENSOR_DSP_CMD_SEG_NUM

U_SENSOR_DATA_BUF_SIZE      = 0x300
U_SENSOR_DATA_BUF_SEG_NUM   = 8
U_SENSOR_DATA_BUF_POOL_SIZE = U_SENSOR_DATA_BUF_SIZE * U_SENSOR_DATA_BUF_SEG_NUM

# section 0
PoolAreas.init(
  [ # layout 0 for Player
    #[ name,            area,              align,        pool-size,            seg,                 fence]
     ["MAIN_BUF_POOL",  "AUDIO_WORK_AREA", U_STD_ALIGN,  U_MAIN_BUF_POOL_SIZE, U_MAIN_BUF_SEG_NUM,  True ],
     ["PCM_BUF_POOL",   "AUDIO_WORK_AREA", U_STD_ALIGN,  U_PCM_BUF_POOL_SIZE,  U_PCM_BUF_SEG_NUM,   True ],
    None # end of each layout
  ], # end of layout 0

  [ # layout 1 for Recorder
    #[ name,           area,              align,        pool-size,            seg,                fence]
     ["REC_BUF_POOL",  "AUDIO_WORK_AREA", U_STD_ALIGN,  U_REC_BUF_POOL_SIZE,  U_REC_BUF_SEG_NUM,  True ],
     None # end of each layout
  ], # end of layout 1

  None # end of definition
)

# section 1
PoolAreas.init(
  [ # layout 0 for Sensor
    #[ name,                     area,               align,       pool-size,                   seg,                        fence]
     ["SENSOR_DSP_CMD_BUF_POOL", "COMMON_WORK_AREA", U_STD_ALIGN, U_SENSOR_DSP_CMD_POOL_SIZE,  U_SENSOR_DSP_CMD_SEG_NUM,   False],
     ["SENSOR_DATA_BUF_POOL",    "COMMON_WORK_AREA", U_STD_ALIGN, U_SENSOR_DATA_BUF_POOL_SIZE, U_SENSOR_DATA_BUF_SEG_NUM,  False],
     None # end of each layout
  ], # end of layout 0

  None # end of definition
)

The explanation of each parameter is as follow.

Parameter Description

name

pool name (name starting with uppercase letters and ending with "_ POOL", upper case letters, numbers, _ can be used)

area

Area name of FixedArea to be used as pool area. Place the area in the RAM.

align

Starting alignment of the pool. Specify a multiple of MinAlign (= 4) except 0.

pool-size

size of the pool. A value of a multiple of 4 except 0. Segment size * Number of segments. In the final area of each area, you can specify RemainderSize indicating the remaining size.

seg

Number of segments. Specify a value between 1 and 255 inclusive.

fence

Specify whether the fence is valid or invalid. This item is ignored if UseFence is False.

How To Generate

The way to create a Layout file using "mem_layout.py" is as follow.

python3 mem_layout.conf layout_header fence_header pool_header

Ex) python3 mem_layout.conf mem_layout.h fixed_fence.h pool_layout.h

The explanation of each argument of "mem_layout.py" is as follow.

Table 23. Layout file creation tool argument
Parameter Description

mem_layout.conf

Memory layout definition file

layout_header

header file in which various constant values are output as macros

fence_header

Header file to output FixedArea’s memory fence address. It is a file used by "Memory Manager" and should not be used by users.

pool_header

Header file to which various definitions of PoolArea are output. It is a file used by "Memory Manager", so please do not use it by users.

11.9.3. Message Library

Mainly, when it is necessary to send and receive Class Instance between Tasks like "Memory Manager", since it can not be used the system calls provided by the OS, we can realize by using "Message Library". I will explain this "Message Library" here.

General

"Message Library" is an inter-task synchronization library that can send and receive Class Instance between tasks.

This library explicitly specifies the destination as an ID and sends it. In addition, the receiving side waits for a transmission event and realizes event driven operation.

Message Sequence

In addition, this message has Type, and it is judged by type and judged what kind of case the received data has instance or not, and if the instance is what is it.

However, it is necessary to correctly implement the copy constructor for the class you want to send and receive.
Message ID and Type
Message ID

Message ID is statically created according to the following How to write a Message Layout File configuration. For details of Configuration, please refer to that.

After confirming the ID, in the loop of the task,

  MsgQueId id = XX; // Assign the created ID to a variable "id".

  MsgQueBlock * que;
  MsgPacket * msg;

  err_t err = MsgLib::referMsgQueBlock (id, &que);
  err = que->recv(TIME_FOREVER, &msg);

By waiting for the Massage event, event-driven processing with Message ID = XX can be realized.

Conversely, when sending an event,

  MsgQueId send_id = XX; // Assign ID that be sent to a variable "send_id".
  MsgQueId ret_id = XX; // Assign ID that will return to a variable "self_id".

  Object instance; // Class and Instance you want to send.

  instance (); // Construction.

  err_t err = MsgLib::send<Object>(send_id, MsgPriNormal, MSG_TYPE, ret_id, instance);

In this way, assign the ID you want to send as send_id, the ID you wish to reply to ret_id, create an instance of the class (Object) you want to send, and send it with MsgLib::send.

the ID is created according to the application and explicitly used while designating the data path of transmission/reception.

Message Type

In order to extract the appropriate Instance from the received Message Packet, the Message Library uses the Message Type to determine what kind of object was sent. For this reason, Message Type is added to all Message Packet.

This Message Type is defined in "sdk/modules/include/memutils/message/message_type.h".

The Message Type takes the following data structure.

diag 788d4c0f09923d010e16c0d729677b6e

The description of each field is as follow.

Field Name Field Description

REQ

[15]

Indicates whether the message is a request or a response. 0 is the response, 1 is the request.

MSG_USER

[14-12]

User information of the message. You can specify a system that uses messages with a value between 0 and 7. However, since 6 is reserved by AudioSubSystem and 7 by SensorSubSystem, it is actually possible to use values from 0 to 5.

MSG_CATEGORY

[11-8]

Message category information. Categories can be specified with values from 0 to 15.

MSG_SUB_TYPE

[7-0]

Message subtype information. You can specify a message type within a category with a value from 0 to 255.

The defined ID is as follow.

MSG_TYPE_REQUEST

Indicates the direction of the message.

D15 Description

0

response

1

request

MSG_USER

Specify the system using the message. The IDs of Audio and Sensor are currently reserved.

D14 - D12 Description

0 - 5

reserved

6

Audio Sub System

7

Sensor Sub System

MSG_CATEGORY

Please freely define within each system.

For example, in the case of Audio,
"sdk/modules/include/audio/audio_message_types.h"

For Sensor,
"sdk/modules/include/sensing/sensor_message_types.h"

It is defined in.

MSG_SUB_TYPE

Please freely define within each system.

For example, in the case of Audio,
"sdk/modules/audio/include/commmon/audio_interanl_message_types.h"

For Sensor,
"sdk/modules/include/sensing/sensor_message_types.h"

It is defined in.

The message type can be operated even if the same value is defined when the ID is different, but for debuggability and modifiability, it is better to be made everything unique.

We will extract Instance from Message Packet with the following implementation.

  MsgQueBlock * que;
  MsgPacket * msg;

  err_t err = MsgLib::referMsgQueBlock(id, &que);
  err = que->recv(TIME_FOREVER, &msg);

  if (msg->getType() == MSG_TYPE) { // Check that the message type is as expected or not.

     Object instance = msg->moveParam<Object>(); // get an instance of type Object from Message packet.

  }
APIs

The interface of "Message Library" is as follow. For details, see memutils_message.

Functions of the MsgLib class
Initializing the message library
function
static err_t MsgLib::initFirst()
argument
None
Return value
ERR_OK: initialization succeeded
ERR_STS: This function has been executed
Description
Initialize the message library as a whole.
Before using other APIs of this library, execute this function only once with a single CPU agreed in advance.
Initialization per CPU
function
static err_t MsgLib::initPerCpu()
argument
None
Return value
ERR_OK: initialization succeeded
ERR_STS: MsgLib::initFirst() is not executed yet
Description
Initialize the message library for each CPU.
Initialization of various areas for each CPU and counting semaphores (OS environment only).

All the CPUs using this library must wait to complete the execution of MsgLib::initFirst() and  after that you need to execute this function.

If MsgLib::initFirst() has not been executed, it is "debug_assert".
If this API is executed again on the CPU that has already executed this API,it is "debug_assert".
Exiting Message Library
function
static err_t MsgLib::finalize()
argument
None
Return value
ERR_OK: initialization succeeded
ERR_STS: MsgLib::initFirst() is not executed yet
Description
Terminate the message library.
Clear internal management information.
Acquiring Message Queuing Initialization State
function
static bool MsgLib::isInitComplete(MsgQueId id)
argument
MsgQueId id: Message queue ID
Return value
true: Initialized (CPU that owns the queue, initialization has been completed)
false: Uninitialized
Description
Get the message queue initialization status.
If the message cube lock indicated by the argument id does not exist, it is "assert".
If MsgLib::initFirst () has not been executed, it is "assert".
Sending a message packet
function
static err_t MsgLib::send(MsgQueId dest, MsgPri pri, MsgType type, MsgQueId reply)

template<typename T> static err_t MsgLib::send(MsgQueId dest, MsgPri pri, MsgType type, MsgQueId reply, const T& param)

static err_t send(MsgQueId dest, MsgPri pri, MsgType type, MsgQueId reply, const void* param, size_t param_size)

static err_t MsgLib::sendIsr(MsgQueId dest, MsgPri pri, MsgType type, MsgQueId reply)

template<typename T>
  static err_t MsgLib::sendIsr(MsgQueId dest, MsgPri pri, MsgType type, MsgQueId reply, const T& param)

static err_t sendIsr(MsgQueId dest, MsgPri pri, MsgType type, MsgQueId reply, const void* param, size_t param_size)
argument
MsgQueId dest: Destination message queue ID
MsgPri pri: Message priority
MsgType type: Value for identifying the message type
MsgQueId reply: Reply destination message queue ID
const T & param: Message parameters
const void * param: address to message parameter
size_t param_size: parameter size
Return value
ERR_OK: Send successful
ERR_QUE_FULL: There is no space in the message queue
Description
The message packet is stored in the queue indicated by the dest and pri arguments.
send() is the task context, sendIsr() is used in the non-task context.
When using parameters with sendIsr(), in order to minimize interrupt handling, as much as possible do not use class instances, minimize parameter size.

If the size of the argument param is too large to be stored in the queue,  it is "assert".
There are no queues indicated by the dest and pri arguments, or the initialization is completed if not, it is "assert".
When owning another CPU and designating a queue without spin lock sharing with its own CPU, it is "assert".

If sendIsr() specifies a shared queue, it is "assert".
This is to prevent spin lock waiting in non-task context.
Notify message reception
function
static void MsgLib::notifyRecv(MsgQueId dest)
argument
MsgQueId dest: notification destination message queue ID
Return value
None
Description
It notifies reception of a message from another CPU or task.
This function is assumed to be called from an  communication interrupt handler. the

If the lock indicated by the dest argument does not exist or the initialization has not been completed, It is "assert",
Functions of the MsgQueBlock class
Get reference to message cube lock
function
static MsgQueBlock& MsgLib::referMsgQueBlock(MsgQueId id)
argument
MsgQueId id: Message queue ID
Return value
Message cube lock reference
Description
Get reference to message cube lock.
If message cube lock indicated by argument id does not exist or initialization is not completed, it is "assert".
Receiving message packets
function
MsgPacket* MsgQueBlock::recv(uint32_t ms)
argument
uint32_t ms: Reception wait time(ms) or TIME_POLLING or TIME_FOREVER
Return value
Non-NULL: Pointer to message packet in message queue
NULL: Reception timeout
Description
Waits to receive a message packet for the specified time.
Unlike MsgLib::send (), simultaneous recv() to the same queue by multiple tasks is not supported.
TIME_POLLING is defined as 0 and TIME_FOREVER is defined as -1.

If the previously received message packet has not been destroyed by MsgQueBlock::pop(), It is "assert",
If the message queue owns another CPU, It is "assert",
Destroying a message packet
function
void MsgQueBlock::pop()
argument
None
Return value
None
Description
Remove the message packet from the message queue.
If there is a parameter in the message packet, you must discard the parameters  in advance  by MsgPacket::moveParam() or MsgPacket::popParam().
Simultaneous pop() of the same queue by multiple tasks is not supported.

If there is no discard target message packet, it is "assert".
If the parameter length of the message packet to be discarded is other than 0, it is "assert".
If the message queue owns another CPU, it is "assert".
Acquiring number of message packets
function
uint16_t MsgQueBlock::getNumMsg(MsgPri pri) const
argument
MsgPri pri: Message queue priority
Return value
Number of message packets
Description
Get the number of message packets in the message queue.
If the value of the arg argument is invalid, it is "assert".
Acquiring number of message packets that can be stored
function
uint16_t MsgQueBlock::getRest(MsgPri pri) const
argument
MsgPri pri: Message queue priority
Return value
Number of message packets that can be stored
Description
Get the number of message packets that can be stored in the message queue.
Unused queues always return 0.
If the argument pri is invalid, it is "assert".
Functions of the MsgPacket class
Acquiring message type ID
function
MsgType MsgPacket::getType() const
argument
None
Return value
Message type ID
Description
Get the type ID of the message packet.
Acquiring message reply queue ID
function
MsgQueId MsgPacket::getReply() const
argument
None
Return value
Reply queue ID
Description
Get the reply queue ID of the message packet.
Acquiring Message Parameter Length
function
uint16_t MsgPacket::getParamSize() const
argument
None
Return value
Message parameter length. 0 is returned when there is no parameter.
Description
Get the parameter length of the message packet.
Destructive take out message parameters (movement)
function
template <typename T> T MsgPacket :: moveParam ()
argument
None
Return value
Message parameters
Description
Retrieve the parameter of the message packet (move).
if "sizeof(T)! = MsgPacket::getParamSize()" is, it is "assert".
This API is equivalent to the following processing.

  T param = MsgPacket::peekParam<T>(); /* Make a copy of the parameter with the left side unreferenced */
  MsgPacket::popParam<T>(); /* Discard parameter */
  return param; /* Return copy of parameter */


By calling this API, the message parameter length is changed to 0.
Note that references to the message parameters and pointers are invalid.
Acquiring message parameter reference
function
template <typename T> const T& MsgPacket::peekParam() const
template <typename T> const T& MsgPacket::peekParamOther() const
argument
None
Return value
See const in message parameter
Description
Get a reference to the parameter of the message packet.
peekParam() is the same type as at the time of transmission, peekParamOther() is a different type (For example, the type of the header of the parameter).

peekParam () is assert, if "sizeof(T)! = MsgPacket::getParamSize()".
peekParamOther () is assert, if "sizeof(T) > MsgPacket::getParamSize()".

If you get the return value of these APIs as non-referenced, you can get a copy of the parameters.
Destruction of message parameters
function
template <typename T> void MsgPacket::popParam()

void MsgPacket::popParamNoDestruct()
argument
None
Return value
None
Description
Discards the parameter of the message packet.
popParam() calls the destructor of the parameter.
Since popParamNoDestruct() does not call the destructor, the destructor it can only be used for message packets that do not exist.
Unless there is a special reason, use popParam ().

popParam ()is assert, if "sizeof(T) != MsgPacket::getParamSize().

By calling these APIs, the message parameter length is changed to 0.
Note that references to the message parameters and pointers are invalid.
Configurations and Generate

Here we explain how to write and create the "Message Library" Layout file.

Layout information is described in Python in "msgq_layout.conf" (name of the file can change) With the tool "msgq_layout.py", two headers in C ++ language, "msgq_id.h" "msgq_pool.h" will be generated. Users can use "Message Library" by including this header.

How to write a Message Layout File

"msgq_layout.conf" defines the user constants of "Message Library", defines the message queue pool, specifies the debug value, and checks the message parameters. The explanation of each is shown below.

Defining user constants

If you want to add user’s unique definitions, you would be recommand to define that starts with "U_" in the Message Layout file.

Example of writing user-defined constants

  # User-defined constants must be the names of uppercase letters and numbers beginning with "U_"
  # If defined with a name beginning with "U_MSGQ_", it is also output as a define macro in msgq_id.h

  U_HEADER_SIZE = 8 # Message packet header size
Defining a Message Queue Pool

You can define a message queue pool using user defined constants.

Description example of message queue pool definition

U_MSG_SIZE = 16
U_MSG_NUM = 8

MsgQuePool = [
 # ID, n_size n_num h_size h_nums
  ["MSGQ_USER_APP", U_MSG_SIZE, U_MSG_NUM, 0, 0],
  ["MSGQ_DSP_CMD", 256, 10, U_MSG_SIZE, U_MSG_NUM],
  None # end of user definition
 # end of MsgQuePool

The explanation of each parameter is as follow.

Parameter Description

ID

Specify the name of the message queue pool ID as a character string beginning with "MSGQ_". "MSGQ_NULL", "MSGQ_TOP", "MSGQ_END" are reserved and therefore prohibited.

n_size

Number of bytes of each element of the normal priority queue (8 or more and 512 or less). Specify fixed header length (8 bytes) + parameter length as a multiple of 4.

n_num

Number of elements of normal priority queue (1 or more and 16384 or less).

h_size

Number of bytes (0 or 8 to 512 inclusive) for each element of the high priority queue. Please specify 0 when not in use.

h_num

Number of elements in the high priority queue (0 or 1 to 16384 or less). Please specify 0 when not in use.

Although there is also a designation of debugging value and checking of message parameters, please do not use it.
MsgFillValueAfterPop = 0x00
MsgParamTypeMatchCheck = false
How To Generate

The way to create a Layout file using "msgq_layout.py" is as follow.

python3 msgq_layout.conf start_addr size id_header pool_header

Ex) python3 msgq_layout.conf 0x000fe000 0x1000 msgq_id.h msgq_pool.h

The explanation of each argument of "msgq_layout.py" is as follow.

Table 24. Layout file creation tool argument
Parameter Description

msgq_layout.conf

Message queue layout definition file

start_addr

Address of message area

size

Area size in bytes

id_header

File to which message queue ID macro is output

pool_header

File in which definition of message queue pool is output

Sequence
diag 562e8fa1190620dc9f471906aec6c25e

11.9.4. Simple FIFO

General

Simple FIFO This library supports one-writer and one-reader access without any exclusive control. Some exclusive access control is required to support multi-writer/reader outside of this library.

This library supports access from multi-processor inserting proper data-sync-barriers and data-memory-barriers. It makes sure the order of update data contents and WP/RP for the purpose.

APIs

For details, refer to Doxygen. A list of APIs is as follow.

  1. Simple FIFO API List

API name

Description

CMN_SimpleFifoInitialize

Initialization of Simple FIFO. Sets the memory area used for FIFO.

CMN_SimpleFifoOffer

Push data to the FIFO. Copy processing uses memcpy ().

CMN_SimpleFifoOfferWithSpecificCopier

Push data to the FIFO. Copy processing is prepared by the user side. We are assuming copy using DMA.

CMN_SimpleFifoOfferContinuous

Push data to the FIFO. It copies to the continuous area by the specified size. If there is no contiguous area for the specified size, push is not performed. Copy processing uses memcpy().

CMN_SimpleFifoOfferContinuousWithSpecificCopier

Push data to the FIFO. It copies to the continuous area by the specified size. If there is no contiguous area for the specified size, push is not performed. Copy processing is prepared by the user side. We are assuming copy using DMA.

CMN_SimpleFifoPoll

Pops the data from the FIFO. Pop data will be deleted from the FIFO. Copy processing uses memcpy().

CMN_SimpleFifoPollWithSpecificCopier

Pops the data from the FIFO. Pop data will be deleted from the FIFO. Copy processing is prepared by the user side. We are assuming copy using DMA.

CMN_SimpleFifoPeekWithOffset

Acquires address information of specified size data from Offset specified from the beginning of the FIFO. It is used when you want to refer to the FIFO data without deleting it.

CMN_SimpleFifoPeek

Acquires address information of data of the specified size from the beginning of the FIFO. It is used when you want to refer to the FIFO data without deleting it. **

CMN_SimpleFifoClear

Clear FIFO Read / Write pointer to empty state.

CMN_SimpleFifoGetVacantSize

Get the free size of the FIFO.

CMN_SimpleFifoGetOccupiedSize

Get the FIFO usage size.

CMN_SimpleFifoGetExtInfo

Acquires the extended information of the FIFO set by CMN_SimpleFifoInitialize().

CMN_SimpleFifoGetDataSizeOfPeekHandle

Get the number of elements of address information acquired by CMN_SimpleFifoPeek().

CMN_SimpleFifoCopyFromPeekHandle

Acquires data from address information acquired by CMN_SimpleFifoPeek(). Copy processing uses memcpy().

CMN_SimpleFifoCopyFromPeekHandleWithSpecificCopier

Get data from address information obtained by CMN_SimpleFifoPeek(). Copy processing is prepared by the user side. We are assuming copy using DMA.

Sequence

It is a simple sequence of Simple FIFO.

diag 62882cd0b3968f64772df34bd581e49f

11.9.5. Code Examples

These libraries are used with audio features. So, their samples are audio samples.

  • examples/audio_player

  • examples/audio_recorder

11.10. Power Management

11.10.1. Overview

Power Management is a module to achieve power saving which is a characteristic feature of the Spresense SDK. It manages the power domain and clock in the CXD5602 and supports various sleep modes.

11.10.2. Power State

The Spresense SDK supports three sleep modes in order to save power: Deep Sleep, Cold Sleep, and Hot Sleep.

Deep Sleep

The CXD5602 is in a state where all power domains are turned off and only the CXD5247 PMIC (Power Management IC) is turned on. In this state, the CXD5247 has the lowest power consumption.

Cold Sleep

In this state, the necessary minimum power domains are turned on in the CXD5602. Compared to Deep Sleep, this state can be awakened by triggers of various factors, such as an alarm timer, USB insertion/removal, or the changes of the GPIO signal.

Hot Sleep

This state is entered after the OS has been idle for a while without being interrupted. Since SRAM state is retained during this state, it is not necessary to reload the program from SPI-Flash, and it can be awakened at high speed.

In the current SDK version, Hot Sleep is not supported.
Power State Transition

The power state transition diagram is shown below.

Power State Diagram
Figure 71. Power State Diagram
State Description

PowerOff

No power supply as battery detached.

Battery Check

Monitors the voltage level of the power supply. When the power supply is 3.4V or higher, the CXD5602 boots and enters the run state.

Run

Normal running state.

Idle

The OS is idle and waits for an interrupt (WFI).

Hot Sleep

When the OS is in idle state and stays idle for longer than the threshold time, this state is entered. During Hot Sleep state, SRAM data is retained.

Cold Sleep

Only the necessary minimum power supply of CXD5602 is turned on, all other power supplies such as SRAM are turned off.

Deep Sleep

The CXD5602 is turned off and only the CXD5247 PMIC (Power Management IC) is turned ON.

The Spresense SDK supports the following states:

Power Domain Hierarchical Structure

The inside of the CXD5602 chip is divided into several power domains with a hierarchical structure. The relationship between various sleep modes and power domains is shown below.

Power Domain Hierarchical Structure

11.10.3. Power State Control API

The Spresense SDK has the cause of boot-up or wake-up (called as boot cause) and the mask for controlling whether to enable or disable the boot cause (called as boot mask).

boot cause

Indicates the cause of boot-up or wake-up from the various sleep modes.

boot mask

Indicates the mask to be enabled or disabled by the boot cause.

boot cause and boot mask have the common bit flag structure, and each factor is represented by one bit. If 1 is set in the corresponding bit of boot mask, it will be enabled as the wake-up factor and if set to 0 it will be disabled. When in Sleep state, it wakes up from Sleep state when an event of wake-up factor allowed by boot mask occurs. At that time, the boot cause is reflected in boot cause.

The boot cause and boot mask are defined as follows.

  • Boot Type is POR (Power-On-Reset), Reboot, the wake-up from Deep Sleep or Cold Sleep state.

  • Maskable is enabled as the boot cause only for Yes. No is disabled as the boot cause.

boot cause / boot mask Boot Type Maskable Description

PM_BOOT_POR_NORMAL

POR Boot

No

Battery or power turned on

PM_BOOT_POR_DEADBATT

POR Boot

No

Battery or power supply is 3.4 V or higher

PM_BOOT_WDT_REBOOT

Reboot

No

Rebooted by system watchdog

PM_BOOT_WDT_RESET

Reboot

No

Reset by a CXD5602 watchdog

PM_BOOT_DEEP_WKUPL

Deep Boot

Yes

Detected the WKUPL signal(*2)

PM_BOOT_DEEP_WKUPS

Deep Boot

Yes (*1)

Detected the WKUPS signal(*2)

PM_BOOT_DEEP_RTC

Deep Boot

Yes (*1)

Expired RTC Alarm

PM_BOOT_DEEP_USB_ATTACH

Deep Boot

No

Connected USB(*2)

PM_BOOT_DEEP_OTHERS

Deep Boot

No

Not used

PM_BOOT_COLD_SCU_INT

Cold Boot

Yes

Detected SCU interrupt

PM_BOOT_COLD_RTC

Cold Boot

Yes

Expired RTC Alarm

PM_BOOT_COLD_RTC_ALM0

Cold Boot

Yes

Expired RTC Alarm0 (This is used by SDK)

PM_BOOT_COLD_RTC_ALM1

Cold Boot

Yes

Expired RTC Alarm1 (Not used)

PM_BOOT_COLD_RTC_ALM2

Cold Boot

Yes

Expired RTC Alarm2 (Not used)

PM_BOOT_COLD_RTC_ALMERR

Cold Boot

Yes

Expired RTC Alarm Error (Not used)

PM_BOOT_COLD_GPIO

Cold Boot

Yes

Detected GPIO interrupt

PM_BOOT_COLD_SEN_INT

Cold Boot

Yes

Detected sensor interrupt (SEN_INT)

PM_BOOT_COLD_PMIC_INT

Cold Boot

Yes

Detected interrupt from PMIC (CXD5247)

PM_BOOT_COLD_USB_DETACH

Cold Boot

Yes

Disconnected USB(*2)

PM_BOOT_COLD_USB_ATTACH

Cold Boot

Yes

Connected USB(*2)

(*1) Both PM_BOOT_DEEP_WKUPS and PM_BOOT_DEEP_RTC can not be disabled.
(*2) Not supported on the Spresense board.

Regarding the power state control, the following Power Management APIs are provided:

  • Get the boot cause.

  • Get the boot mask.

  • Enable / disable the boot cause.

  • Enter the sleep mode.

  • Reboot.

up_pm_get_bootcause()
Function Prototype
uint32_t up_pm_get_bootcause(void);
Description
  • Returns the boot cause.

up_pm_get_bootmask()
Function Prototype
uint32_t up_pm_get_bootmask(void);
Description
  • Returns the boot mask.

    • Enables all boot causes by default.

    • Resets the boot mask at Deep Sleep or Power-On-Reset. The boot mask is kept during hot sleep and cold sleep.

up_pm_set_bootmask()
Function Prototype
uint32_t up_pm_set_bootmask(uint32_t mask);
Description
  • Enables the boot cause specified in the argument.

  • Returns the updated boot mask.

up_pm_clr_bootmask()
Function Prototype
uint32_t up_pm_clr_bootmask(uint32_t mask);
Description
  • Disables the boot cause specified in the argument.

  • Returns the updated boot mask.

Example
#include <arch/chip/pm.h>

  uint32_t bootmask;

  bootmask = up_pm_get_bootmask(); // Get the current bootmask
  printf("bootmask=0x%08x\n", bootmask);

  bootmask = up_pm_clr_bootmask(PM_BOOT_COLD_USB_DETACH); // Disable wakeup by USB detached
  printf("bootmask=0x%08x\n", bootmask); // Display the updated bootmask
up_pm_sleep()
Function Prototype
int up_pm_sleep(enum pm_sleepmode_e mode);
Description
  • Enters the sleep mode specified in the argument.

  • Doesn’t come back from the function call.

Arguments
Argument mode Description

PM_SLEEP_DEEP

Enter Deep Sleep

PM_SLEEP_COLD

Enter Cold Sleep

By calling up_pm_sleep(), the OS enters Sleep mode as a single CXD5602 chip. To control Sleep mode with the board instead of the chip, the function board_power_off() needs to be implemented in the board specific code of BSP.

The Sleep transition API implemented in the BSP is shown below. This can also be controlled from the poweroff command in NuttShell.

Function Prototype
int board_power_off(int status)
Arguments
Argument status Description NuttShell command

BOARD_POWEROFF_DEEP

Enter Deep Sleep

poweroff

BOARD_POWEROFF_COLD

Enter Cold Sleep

poweroff --cold

up_pm_reboot()
Function Prototype
int up_pm_reboot(void);
Description
  • Reboots the system.

  • Doesn’t come back from the function call.

If you want to control the reboot with board, you can implement board_reset() function in board specific code of BSP.

Function Prototype
int board_reset(int status);

The board_reset() can also be controlled from the reboot command on NuttShell.

11.10.4. Sleep Mode

Deep Sleep
Features:
  • The CXD5602 is turned off.

  • The CXD5247 enters Sleep mode, and the Core voltage and I/O voltage supplied to the CXD5602 are turned off.

    • If your system has an RTC XTAL, it retains the RTC time.

    • The GPO value of CXD5247 switch is retained during Deep Sleep.

      • If the GPO is not needed to be turned on during Deep Sleep, it is recommended to turn it off before entering Deep Sleep to minimize power consumption.

    • CXD5247 Load Switch is turned OFF.

    • Make sure that current does not leak from the peripheral device to the CXD5602 since the I/O of the CXD5602 is turned OFF.

Power Consumption
  • The power consumption of the battery is about several uA at the battery end. However, this value depends on board design. Since the CXD5602 is powered off, design the circuit board carefully so that there is no leakage current from the peripheral device to the CXD5602.

About the power consumption during sleeping, when the SD card is inserted on the Spresense extension board, the current consumption increases by about 5 mA for the power supply to the SD card.
Sleep Conditions
  • Call up_pm_sleep(PM_SLEEP_DEEP) or board_poweroff(BOARD_POWEROFF_DEEP).

  • Assert the WKUPL signal for more than three seconds.

The Spresense board doesn’t have the WKUPL terminal, therefore transition to Deep Sleep by the signal can not be used.
Wakeup Conditions

When any wake-up condition occurs, the program will be loaded from SPI-Flash, so it takes about the same time as starting by Power-On-Reset.

  • PM_BOOT_DEEP_WKUPL : Assert the WKUPL signal for more than three seconds.

  • PM_BOOT_DEEP_WKUPS : Assert the WKUPS signal.

  • PM_BOOT_DEEP_RTC : A RTC alarm expired.

  • PM_BOOT_DEEP_USB_ATTACH : Connected the USB port to a power source.

    • Normally, when the USB cable is connected, you can not enter the Deep Sleep state. However, if you set CONFIG_BOARD_USB_DISABLE_IN_DEEP_SLEEPING=y, you can disable the USB function and enter the Deep Sleep state. At this time, it can not wake up with a USB connection as a trigger.

The Spresense board doesn’t have the WKUPL and WKUPS terminals, therefore wake-up from Deep Sleep by the signals can not be used.
Cold Sleep
Features:
  • Only the PMU power domain in the CXD5602 is ON.

    • CXD5602 I/O pin is activated.

    • Backup SRAM are retained.

  • CXD5247 is the normal running state.

    • If your system has an RTC XTAL, it retains the RTC time.

    • The GPO signal is retained.

    • The Load Switch signal is retained.

Power Consumption
  • The power consumption of the battery is approximately several hundred uA at the battery end. This current value depends on board design.

About the power consumption during sleeping, when the SD card is inserted on the Spresense extension board, the current consumption increases by about 5 mA for the power supply to the SD card.
Sleep Conditions
  • Call up_pm_sleep(PM_SLEEP_COLD) or board_poweroff(BOARD_POWEROFF_COLD).

Wakeup Conditions

When any wake-up condition occurs, the program will be loaded from SPI-Flash, so it takes about the same time as starting by Power-On-Reset.

  • PM_BOOT_COLD_SCU_INT : A SCU interrupt is asserted.

  • PM_BOOT_COLD_SEN_INT : A sensor interrupt is asserted.

  • PM_BOOT_COLD_PMIC_INT : A interrupt from CXD5247 is asserted.

    • WKUPS signal is asserted.

    • Low Battery notification is asserted.

  • PM_BOOT_COLD_GPIO : A GPIO interrupt is asserted.

  • PM_BOOT_COLD_RTC_ALM0 : A RTC Alarm expired.

  • PM_BOOT_COLD_USB_ATTACH : Connected the USB port.

  • PM_BOOT_COLD_USB_DETACH : Disconnected the USB port.

The Spresense board does not support wake-up from Cold Sleep by attached/detached USB.
Hot Sleep
In the current SDK version, Hot Sleep is not supported.
Features
  • The CXD5602 is almost equal to the normal running state, but the CPU application on which NuttX is running, is turned off.

    • However, SRAM is retained during Hot Sleep.

  • CXD5247 is in the normal running state.

Power Consumption
  • The power consumption of the battery can be lowered to at most a few hundred uA at the battery end, but it depends strongly on what is operating as an application.

Sleep Conditions
  • If CONFIG_CXD56_HOT_SLEEP=y, Hot Sleep is enabled.

    • With the CONFIG parameter below, you can change the condition to transition to Hot Sleep state.

      • CONFIG_CXD56_HOT_SLEEP_WAIT_COUNT (milliseconds)

      • CONFIG_CXD56_HOT_SLEEP_THRESHOLD (milliseconds)

    • Acquiring and releasing wakelock.

      • Hot Sleep control is performed with the wakelock mechanism to be described later. As long as at least one wakelock is acquired, transition to Hot Sleep is prohibited.

Power Manager transits to the Hot Sleep state with the following algorithm:

  1. When the NuttX OS is idle, Power Manager counts the time to stay idle.

    • If any interrupt occurs, the system leaves the idle state and the counter is reset.

  2. If the counter exceeds the value defined by CONFIG_CXD56_HOT_SLEEP_WAIT_COUNT, Power Manager will calculate by predicting the following idle time.

  3. If this predicted idle time > CONFIG_CXD56_HOT_SLEEP_THRESHOLD has been exceeded, the system will enter the Hot Sleep state.

    • When any wakelock is acquired, the system is prohibited from entering the Hot Sleep state.

Wakeup Conditions
  • Any interrupt to NuttX CPU is asserted.

  • UART1 Rx interrupt.

  • UART2 Rx interrupt.

  • SCU (Sensor Control Unit) Watermark interrupt.

  • USB VBUS (Connected) interrupt.

  • USB VBUS Negative (Disconnected) interrupt.

  • PMIC (Low Battery Notification or WKUPS) interrupt.

  • Any GPIO interrupt(s) that was registered and used before entering Hot Sleep.

  • Inter-CPU communication from other CPUs to NuttX CPU.

  • NuttX OS timer event interrupt.

11.10.5. Power Saving Control

The SDK provides the following power-saving control features:

  • A CPU working frequency control.

  • A CPU power down control.

CPU Working Frequency Control

The SDK provides the following frequency states:

Mode CPU Clock CXD5602 Core Voltage

HV (High Voltage) Mode

PLL 156MHz

1.0 V

LV (Low Voltage) Mode

PLL 32MHz

0.7 V

RCOSC Mode

RCOSC approximately 8MHz

0.7 V

By default, the SDK uses the high voltage mode and cannot transition to another mode. To release the CPU frequency lock, apply the following configuration settings:

When the dynamic clock control is disabled, it always operates in HV mode. To enable the power saving function by dynamic clock control, please set the following configuration.

 [CXD56xx Configuration]
   [Power Management]
     [Dynamic clock control] <= Y

When Dynamic clock control is enabled, the system will start clocking in HV mode and then clock down to RCOSC mode. In steady state, it operates in RCOSC mode.

If the system requires high performance temporarily, it is possible to change the clock dynamically using the Frequency Lock mechanism.

Although the operating clock rises in the order of RCOSC mode, LV mode, HV mode, but the power consumption also increases accordingly. The frequency lock mechanism acquires LV lock when switching to LV mode, and HV lock when switching to HV mode. Releasing the acquired lock returns to the original clock state. As long as any one of the clients acquires a lock of the higher clock mode, the system operates in a higher clock mode. As an example, if both LV lock and HV lock have been acquired, it will be in HV mode. When the HV lock is released, it switches to LV mode, the next higher clock mode. Then, when the LV lock is also released, it returns to the RCOSC mode.

  • Acquire HV lock.

      struct pm_cpu_freqlock_s lock;
    
      lock.flag = PM_CPUFREQLOCK_FLAG_HV;
      up_pm_acquire_freqlock(&lock);
  • Release HV lock.

      up_pm_release_freqlock(&lock);
  • Acquire LV lock.

      struct pm_cpu_freqlock_s lock;
    
      lock.flag = PM_CPUFREQLOCK_FLAG_LV;
      up_pm_acquire_freqlock(&lock);
  • Release LV lock

      up_pm_release_freqlock(&lock);
CPU power down control

The SDK provides a function to automatically transition to Hot Sleep when the application CPU is in the idle state.

In the current SDK version, Hot Sleep is not supported.

By default, the Hot Sleep function is disabled. The following settings are required to enable this function:

 [CXD56xx Configuration]
   [Power Management]
     [Hot Sleep] <= Y
       [Hot Sleep wait counter] <= 20
       [Hot Sleep threshold] <= 1000
       [Enable GNSS Hot Sleep] <= Y

When CONFIG_CXD56_HOT_SLEEP=y, it automatically enters the Hot Sleep state in the OS idle state. By setting CONFIG_CXD56_GNSS_HOT_SLEEP=y, it enables the Hot Sleep function of the GNSS CPU.

In the idle state, it automatically enters the Hot Sleep state, but it can suppress the transition to the Hot Sleep state by the wakelock mechanism.

  • Acquire wakelock.

    struct pm_cpu_wakelock_s lock;
    
    lock.count = 0;
    up_pm_acquire_wakelock(&lock);
  • Release wakelock.

    up_pm_release_wakelock(&lock);

11.11. Sensor Fusion Framework

11.11.1. General

CXD5602 has a function to constantly sensing with low power consumption. Then, the sensor fusion function can be supported by multiple cores and memory.

SDK provides a framework for easily supporting these.

The sensor fusion framework consists of "Sensor Manager" and "Sensor Client" for each sensor.

  • "Sensor Manager" controls multiple "Sensor Client" based on the Publish-Subscribe Architecture and delivers Published Sensor Data from "Sensor Manager".

  • "Sensor Client" consists on of "Physical Sensor", "Logical sensor" and "Sensor Application". "Physical Sensor" controls drivers that control various "Sensor Devices". "Logical Sensor" supports a highly functional sensor by fusing data from each "Sensor Device". "Sensor Application" is required for Application to receive data.

The Architecture diagram of Sensor Framework is as follow.

Sensor Framework(Publish/Subscribe) Architecture
Figure 72. Sensor Framework (Publish/Subscribe) Architecture

11.11.2. Sensor Manager

"Sensor Manager" registers and manages multiple "Sensor Client" and properly distributes sensor data.

General

"Sensor Client" registered in "Sensor Manager" send the acquired data to "Sensor Manager". "Sensor Manager" delivers the received data to "Sensor Client" that requesting subscribe.

In addition, it provides a framework that can power off the "Sensor Client" which is not subscribed.

APIs

"Sensor Manager" provides the following APIs. The "Sensor Manager" interface is controllable to issue commands in data format called packets. Register, delete, send and receive "Sensor Client".

Table 25. Sensor Manager API
APIs Description Corresponding API Corresponding packet

Register

Resister a Sensor Client to Sensor Manager as subscriber.

SS_SendSensorResister

sensor_command_register_t

Release

Unregister the Sensor Client from Sensor Manager.

SS_SendSensorRelease

sensor_command_release_t

SendData

Sender function to Sensor Manager without MemHandle.

SS_SendSensorData

sensor_command_data_t

SendData(MH)

Sender function to Sensor Manager with MemHandle.

SS_SendSensorDataMH

sensor_command_data_mh_t

SendSetPower

Set power status of sensors.

SS_SendSensorSetPower

sensor_command_power_t

Sequence
diag f9fe639e23da44120823c9c25a4e1096
Figure 73. Sensor Manager Sequence
For MemHandle please refer to Memory Manager Library.

11.11.3. Logical Sensors

General

"Logical sensor" is composed of "Sensor Client" for creating sensor data obtained from various physical sensors based on some kind of signal processing algorithm, to create highly functional sensor data.
There are several ways of implementing allocation of actual algorithms as follow.

  • Implement it as a task on NuttX.

  • Implement on the DSP using the asmp framework.

  • Implement using the encrypted DSP provided by each vendor.

A logical sensor task on NuttX

Implement as module or task on NuttX.
Especially, in the case of processing which is not heavy or it is not frequent, this type of implementation can be used which will not consume memory and CPU resources.

Table 26. Samples of Logical Sensors on NuttX
Contents Sample provider

Barometer

From sensor vender

TAP Gesture (Tapping Detector)

From SDK

A logical sensor on DSP by asmp

In the case where the user independently creates an algorithm of "Logical Sensor" and needs to offload the processing to the DSP side (for example, heavy processing), it is implemented on the DSP side using the ASMP framework.
In such case, "Logical Sensor" can be supported by multicore processing by implementing "Logical Sensor" with supervisor task and sending processing request from the supervisor task to the Worker task on the DSP for processing.

Table 27. Samples of Logical Sensors on DSP
Contents Sample provider

""

From SDK

SuperVisor and Worker
Figure 74. SuperVisor and Worker
For ASMP framework please refer to ASMP Framework.
A logical sensor on DSP with encryption

Each solution vendor provides various logical sensor algorithms. In that case, each vendor can provide the function based on each contract. without disclosing the code.
In this case, it can be supported by loading the build and encrypted binaries on the ASMP framework. Encrypted DSP is provided by each vendor.

Table 28. Samples of Logical Sensors on DSP with encryption
Contents Sample provider

AESM (Activity Engine and Step Meter)

From SDK

SuperVisor and Encrypted Worker
Figure 75. SuperVisor and Encrypted Worker

11.11.4. Logical Sensor API

Each Logical sensor provides the following APIs.

Table 29. Logical Sensor API
APIs Description

Create

Create a class instance to communicate with workers.

Open

Load library and boot up as worker task.

Write

Send data to worker task.

Close

Destroy worker task.

These requests are defined as the following events and are used for sending and receiving with the Worker task on the DSP.
For details, refer to each supervisor.

Event Description

Init

Initialization event.

Exec

Execution event.

Flush

Terminal event.

11.11.5. Details of each Logical Sensors

11.11.6. Step Counter

The configuration diagram of "Step Counter" is shown below.

Configuration diagram of Step Counter
Figure 76. Configuration diagram of Step Counter
Supervisor

Supervisor is a framework for "Logical Sensor". It provides several APIs for controlling the Worker on the Supervisor’s DSP.

APIs

"Step Counter" provides the following 5 APIs.

Table 30. Step Counter API
APIs Description Corresponding API

Create

Create StepCounterClass instance.

StepCounterCreate

Open

Load StepCounter library and boot up as worker task.

StepCounterOpen

Write

Send data to StepCounter worker task.

StepCounterWrite

Close

Destroy StepCounter worker task.

StepCounterClose

Set

Set setting to StepCounter library .

StepCounterSet

Data format for Step Counter

"Step Counter" requires "Accelerometer" data in the format specified as follow.
"Application" will send these data to the worker with Write API.

Data format of Accelerometer
Figure 77. Data format for Step Counter
Result of sensing

"Step Counter" outputs the following pedometer information every 1 second. These information are stored in StepCounterStepInfo.

Table 31. Step Counter result
Information Unit Note

Tempo

Hz

Take a value of 1, 2, 3 Hz. Even when it is in the stopped state, it is 1 Hz or more.

Stride length

cm

It becomes a fixed value in walking and running respectively.

Speed

m/s

Distance

m

Step count

-

Behavior recognition

-

It indicates stopping/walking/running condition.

How to use
Preparation

It controls "Step Counter" with "Sensor Manager" framework designed to control multiple "Sensor Client".

Therefore, in order to control "Step Counter", you need to enable "Sensor Manager" in advance.

Activate Sensor Manager

To enable "Sensor Manager" you need to call SS_ActivateSensorSubSystem(MsgQueId, api_response_callback_t) .

MsgQueId must specify the MsgQueID defined in Message Library Configuration.

api_response_callback_t specifies a callback function for asynchronous notification. If NULL is specified, notification is not performed.

static void sensor_manager_api_response(unsigned int code,
                                        unsigned int ercd,
                                        unsigned int self)
{
  ...
}

SS_ActivateSensorSubSystem(MSGQ_SEN_MGR, sensor_manager_api_response);
Register to Sensor Manager

After "Sensor Manager" is activated, in order for "Step Counter" to know the sensing result of "Accelerometer", "Step Counter" registers "Accelerometer" as "Sensor Client" which request "Subscribe".
Then, specify a callback function that processes "Subscribe". In the callback function, call StepCounterWrite.

Also, in order for "Application" to know the sensing result of "Step Counter", "Application" registers "Step Counter" as "Sensor Client" which request "Subscribe".
Then, specify a callback function that processes "Subscribe". In the callback function, do processing to display or notify the result.

bool step_counter_receive_data(sensor_command_data_mh_t& data)
{
  StepCounterWrite(sp_step_counter_ins, &data);

  return true;
}

bool step_counter_recieve_result(sensor_command_data_mh_t& data)
{
  bool ret = true;
  FAR SensorCmdStepCounter *result_data =
    reinterpret_cast<SensorCmdStepCounter *>(data.mh.getVa());
  if (SensorOK == result_data->result.exec_result)
    {
      if (result_data->exec_cmd.cmd_type ==
            STEP_COUNTER_CMD_UPDATE_ACCELERATION)
        {
           ...
        }
    }
    return ret;
}

sensor_command_register_t reg;

reg.header.code   = ResisterClient;
reg.self          = stepcounterID;
reg.subscriptions = (0x01 << accelID);
reg.callback      = NULL;
reg.callback_mh   = &step_counter_receive_data;

SS_SendSensorResister(&reg);

reg.header.code   = ResisterClient;
reg.self          = app0ID;
reg.subscriptions = (0x01 << stepcounterID);
reg.callback      = NULL;
reg.callback_mh   = &step_counter_recieve_result;
SS_SendSensorResister(&reg);
diag ae8a4ac55b476a0df674d172603fa41a
Figure 78. Register sequence
Create and Open

After completing preparations, "Step Counter" will be generated and activated.

To generate "Step Counter" you need to call StepCounterCreate(PoolId).

For PoolId, you need to specify the ID defined in "Memory Manager Configuration".
As a return value, a pointer to the instance of "Step Counter" is returned.

FAR StepCounterClass *step_counter_instance;
step_counter_instance = StepCounterCreate(SENSOR_DSP_CMD_BUF_POOL);

To activate "Step Counter" you need to call StepCounterOpen(FAR StepCounterClass*).
For StepCounterClass *, you need to specify a pointer to the instance of "Step Counter" which is the return value of StepCounterCreate.

StepCounterOpen(step_counter_instance);
diag 1b72b5399858368398d27c07008facb0
Figure 79. Create and Open sequence
Set stride

The initial value of stride when sensing is 60 cm in walking condition and 80 cm in running condition.

To change this initial value, you need to call StepCounterSet(FAR StepCounterClass*, StepCounterSetting*).
For StepCounterClass *, you need to specify a pointer to the instance of "Step Counter" which is the return value of StepCounterCreate.
For StepCounterSetting *, please set the stride length of walking state and running state in step_length in unit cm. The maximum stride is 250 cm.
step_mode is fixed "STEP_COUNTER_MODE_FIXED_LENGTH"

StepCounterSetting set;
set.walking.step_length = 70; /* Set stride to 70 cm */
set.walking.step_mode   = STEP_COUNTER_MODE_FIXED_LENGTH;
set.running.step_length = 90; /* Set stride to 90 cm */
set.running.step_mode   = STEP_COUNTER_MODE_FIXED_LENGTH;
StepCounterSet(step_counter_instance, &set);
diag a016bdc761fe89c1780f645ec3970ee0
Figure 80. Set sequence
Start Sensing

"Step Counter" sends data to the worker on the DSP via StepCounterWrite at the timing of "Subscribe" of "Accelerometer" and performs sensing processing.
The sequence at this time is shown below.

diag 824cab537bddc049b902fdb13bc6bacd
Figure 81. Sensing sequence of Accelerometer publish
Close

To disable "Step Counter", you need to call StepCounterClose(FAR StepCounterClass*).
For StepCounterClass *, you need to specify a pointer to the instance of "Step Counter" which is the return value of StepCounterCreate.

StepCounterClose(step_counter_instance);
diag f824d2e23fb7b0a136d620b2bb1fa517
Figure 82. Close sequence
Release from Sensor Manager

After disable "Step Counter", cancel registration to "Sensor Manager".
In this case, please suspend the operation of "Sensor Client" which subscribes to "Step Counter" in advance.

sensor_command_register_t reg;

rel.header.code = ReleaseClient;
rel.self        = stepcounterID;
SS_SendSensorRelease(&rel);
Build Configurations

In order to use the function of "Step Counter" please follow these instructions:

$>cd sdk
$>tools/config.py -m

You need to open the Config menu and configure the following Config.

Select options in below:

[CXD56xx Configuration]
  [I2C0] <= Y
  [Sensor Control Unit] <= Y
[Memory manager] <= Y
  [Memory Utilities]
    [Memory manager] <= Y
    [Message] <= Y
[Drivers]
  [Sensor Drivers] <= Y
    [Bosch BMI160 Sensor support] <= Y
      [SCU Sequencer] <= Y
[Sensing]
  [Sensing manager] <= Y
  [Step counter] <= Y
[ASMP] <= Y
Worker

Worker runs on other Core and analyzes sensor data sent from Supervisor.
And it returns the processing result (number of steps, state.) of the requested event.

The worker of "Step Counter" needs the following data for analysis.

・ Accelerometer data(32Hz、32samples/1sec)
APIs

The "Step Counter" Worker provides the following 3 APIs.

Table 32. Step Counter Worker API
APIs Description

Init

Initialize for AESM(Activity Engine and Step Meter) library.

Exec

Execute calculate on sensor data on AESM library.

Flush

End execute process of AESM library.

These APIs are called depending on the event type and command type contained in packets sent from Supervisor.

The relationship is as follow.

Init
SensorDspCmd dsp_cmd;
dsp_cmd.header.sensor_type     = StepCounter;
dsp_cmd.header.event_type      = InitEvent;
Exec
SensorDspCmd dsp_cmd;
dsp_cmd.header.sensor_type     = StepCounter;
dsp_cmd.header.event_type      = ExecEvent;
dsp_cmd.exec_aesm_cmd.cmd_type = AESM_CMD_UPDATE_ACCELERATION;
Flush
SensorDspCmd dsp_cmd;
dsp_cmd.header.sensor_type     = StepCounter;
dsp_cmd.header.event_type      = FlushEvent;
Step Counter Example

There is "Step Counter" example as a sample application of "Step Counter". Here, we will explain how to use it.

Preparation
Build Configurations (kconfig)

In build configuration setup Step counter sensor example to Y to use the sample programs for "Step Counter".

Select options in below:

[Examples]
  [Step counter sensor example] <= Y

Alternatively, use the default configuration of "Step Counter".

$ ./tools/config.py examples/step_counter
Memory and Message Utility Configurations and Layout

To set the inter-task communication library (Message Library) and the memory management library (Memory Manager) as follow:

Message Library Configuration

It is necessary to define the MessageQueue which is necessary when using the "Step Counter" function. Definition is done in the MessageQueueLayout definition file, and you can generate a header file to be included in the code with the tool.

In the example of "Step Counter", do as follow:

$>cd examples/step_counter/config
$>python3 msgq_layout.conf 0x000fe000 0x1000 msgq_id.h msgq_pool.h

The description contents of the MessageQueueLayout definition file (msgq_layout.conf) are as follow:

MsgQuePool
 # ID,             n_size  n_num    h_size  h_nums
 ["MSGQ_SEN_MGR",  40,     8,       0,      0],

The explanation of each parameter is as follow:

Parameter Description

ID

Specify the name of the message queue pool ID as a character string beginning with "MSGQ_".

n_size

Number of bytes of each element of the normal priority queue (8 or more and 512 or less). Specify fixed header length (8 bytes) + parameter length as a multiple of 4.

n_num

Number of elements of normal priority queue (1 or more and 16384 or less).

h_size

Number of bytes (0 or 8 to 512 inclusive) for each element of the high priority queue. Specify 0 when not in use.

h_num

Number of elements in the high priority queue (0 or 1 to 16384 or less). Specify 0 when not in use.

Since n_size is the optimum value, please do not change.

There is no need to change n_num, but when using the "Step Counter" function with other Application, it may be necessary to increase the value considering the load.

Use h_size and h_nums when you want to process the "Step Counter" function preferentially.

Refer to examples/step_counter/config/msgq_layout.conf for details on each definition.
      If the settings change, please use the tool to generate a new header file.
Memory Manager (Intelligent Fix Pool) Configuration

It is necessary to define the MemoryLayout (pool) which is necessary when using the "Step Counter" function.
Definition is done in the MemoaryLayout definition file, and it is possible to generate a header file to be included in the code with the tool.

In the example of "Step Counter", do as follow:

$>cd examples/step_counter/config
$>python3 mem_layout.conf mem_layout.h fixed_fence.h pool_layout.h

The contents of the MemoaryLayout definition file (mem_layout.conf) are as follow:

FixedAreas
FixedAreas
 # name,                  device,     align,        size,         fence
  ["SENSOR_WORK_AREA",    "SHM_SRAM", U_STD_ALIGN,  0x0001e000,   false],
  ["MSG_QUE_AREA",        "SHM_SRAM", U_STD_ALIGN,  0x00001000,   false],
  ["MEMMGR_WORK_AREA",    "SHM_SRAM", U_STD_ALIGN,  0x00000200,   false],
  ["MEMMGR_DATA_AREA",    "SHM_SRAM", U_STD_ALIGN,  0x00000100,   false],

The explanation of each parameter is as follow:

Parameter Description

name

area name (name starting with uppercase letters and ending with "_AREA", uppercase letters, numbers, _ can be used)

device

Device name of MemoryDevices to reserve space

align

Start alignment of the region. Specify a multiple of MinAlign (= 4) except 0

size

size of the region. Specify a value of a multiple of 4 except 0

fence

Specify whether fence is enabled or disabled (This item is ignored when UseFence is false)

The purpose of each name is as follow:

SENSOR_WORK_AREA

Sensor fusion uses

MSG_QUE_AREA

MessageQueue (fixed name). Do not exceed the size of (MSGQ_END_DRM - MSGQ_TOP_DRAM) of msgq_id.h.

MEMMGR_WORK_AREA

Work area used by Memory Manager (fixed name, fixed size)

MEMMGR_DATA_AREA

Data area used by Memory Manager (fixed name, fixed size)

Make sure that the total size of each name does not exceed the size of the shared memory secured by mpshm_init(), mpshm_remap().

PoolAreas
PoolAreas
 # name,                     area,               align,       pool-size,                   seg,                        fence
 ["SENSOR_DSP_CMD_BUF_POOL", "SENSOR_WORK_AREA", U_STD_ALIGN, U_SENSOR_DSP_CMD_POOL_SIZE,  U_SENSOR_DSP_CMD_SEG_NUM,   false],
 ["ACCEL_DATA_BUF_POOL",     "SENSOR_WORK_AREA", U_STD_ALIGN, U_ACCEL_DATA_BUF_POOL_SIZE,  U_ACCEL_DATA_BUF_SEG_NUM,   false],

The explanation of each parameter is as follow:

Parameter Description

name

pool name (name starting with uppercase letters and ending with "_POOL", upper case letters, numbers, _ can be used)

area

Area name of FixedArea to be used as pool area. The area must be located in the RAM

align

Starting alignment of the pool. Specify a multiple of MinAlign (= 4) except 0

pool size

size of the pool. A value of a multiple of 4 except 0. In the Basic pool, segment size * number of segments

seg

Number of segments. Specify a value between 1 and 255

fence

Specify whether the fence is valid or invalid. This item is ignored when UseFence is false

The purpose of each name is as follow:

SENSOR_DSP_CMD_BUF_POOL

send/receive buffer area with Worker

ACCEL_DATA_BUF_POOL

Accelerometer data buffer area

Refer to examples/step_counter/config/mem_layout.conf for details of each definition.
      If the setting changes, please use the tool to generate a new header file.

11.12. JPEG Decoder

11.12.1. Overview

Spresense JPEG Decoder is based on libjpeg library developed by IJG.

JPEG Decoder API is compliant with original libjpeg, however there are customized features for Spresense which are explained in this chapter.

In the following document, libjpeg instance refers to the struct jpeg_decompress_struct type variable which libjpeg application must prepare.

11.12.2. Customized features for Spresense

Output format(color space)

All formats supported by original libjpeg have at least 24 bits/pixel (also called bits-per-pixel or just bpp). Spresense support the following 16 bpp formats: Cb/Y/Cr/Y(YUV4:2:2) which requires less memory to be decoded. The following defines are valid in Spresense. Application have to set the out_color_space member of the libjpeg instance to one of the defines in the list below to specify what format to decode.

○ means that it is supported.
× means that it is not supported.

Table 33. output format(color space)
define meaning bits/pixel Spresense original

JCS_CbYCrY

Cb/Y/Cr/Y(YUV4:2:2)

16

×

JCS_YCbCr

Y/Cb/Cr(YUV)

24

×

JCS_RGB

sRGB

24

×

JCS_CMYK

C/M/Y/K

32

×

JCS_YCCK

Y/Cb/Cr/K

32

×

Unit of reading decoded result

In addition to reading by the line which original libjpeg support, Spresense support reading by the MCU which has the smaller size.

Table 34. reading decoded result API
API meaning Spresense original

jpeg_read_scanlines

line unit

jpeg_read_mcus

MCU unit

×

MCU or Minimum Coded Unit is the block of JPEG compression unit. The size is basically 8×8pixels and depends on compression and decode parameters. Compression parameters are set in JPEG header and decode parameters are passed from application. Application can check the MCU size from libjpeg instance after jpeg_start_decompress() as follow:

  • 1MCU width = output_width /MCUs_per_row (total width / total number of MCUs in width direction)

  • 1MCU height = output_height/MCU_rows_in_scan(total height / total number of MCUs in height direction)

MCU size do not depend on image size.
Therefore, the larger the image size becomes, the more remarkable the difference between the above two units become as the following examples.

Table 35. examples of unit pixel number: decode JPEG file taken in Spresense Camera
unit QVGA case HD case 5M case

line

320

1280

2560

MCU

128

128

128

JPEG data input methods

Spresense support file descriptor in addition to file pointer and buffer similar to original libjpeg.

Table 36. JPEG data input API
API meaning Spresense original

jpeg_stdio_src

file pointer

jpeg_fd_src

file descriptor

×

jpeg_mem_src

buffer

This file descriptor needs to be readable JPEG data by read() function.
For example, support file descriptor by socket() function, besides the file descriptor of regular file by open() function. (Of course, in socket descriptor case, JPEG data need to be sent from peer directly.)

Error Handling

In both original libjpeg and Spresense, libjpeg API execution task is exited by exit() function when errors happen.
In order to avoid exit task, original libjpeg show an example which use setjmp/longjmp. Spresense on the other hand, can not use this method because Spresense(NuttX) does not support setjmp/longjmp. Spresense will support some alternative error handling methods to setjmp/longjmp in the future.

libjpeg error handling premise that error handler do not return to the caller. If error handler return to the caller, libjpeg operation is not guaranteed.
Table 37. End method of error handler
End method of error handler meaning Spresense original

exit

exit libjpeg API execution task