Technical Application Notes
Click an item to read from the list below.

Technical Application Notes


 

 

Using Spinnaker on ARM and Embedded Systems

Applicable products

  • Spinnaker® SDK version 1.5 or later
  • FLIR USB3 machine vision cameras
10910

Application note description

With the release of version 1.5 of the Spinnaker SDK, you can program and operate FLIR USB3 machine vision cameras on an ARM device in a Linux environment. This application note explains the components and steps that are necessary to get started with Spinnaker and ARM as well as the limitations of using Spinnaker on an ARM device. It provides a summary and instructions for streaming FLIR USB3 machine vision cameras using Spinnaker on ARM-based embedded boards. It includes examples of some of the more commonly used embedded boards: ODROID-XU4, and NVIDIA Jetson TX1 and TX2. The benchmark results show that embedded boards are able to support high speed machine vision applications.

Note: Testing is ongoing. Wherever possible, limitations have been noted; however, as more testing is completed this information may change. All possible configurations of ARM and cameras may not experience the same results.

Preparing for use

Before you use your camera, we recommend that you are aware of the following resources available from our downloads page:

  • Getting Started Manual for the camera—provides information on installing components and software needed to run the camera.
  • Technical Reference for the camera—provides information on the camera’s specifications, features and operations, as well as imaging and acquisition controls.
  • Firmware updates—ensure you are using the most up-to-date firmware for the camera to take advantage of improvements and fixes.
  • Tech InsightsSubscribe to our monthly email updates containing information on new knowledge base articles, new firmware and software releases, and Product Change Notices (PCN).

Supported System Configuration

Before installing Spinnaker, you must have the following prerequisites:

  • Odroid XU4, Jetson TX1/TX2, or other OMAP4 device with an ARMv7 Cortex processor (including Cortex-A7, -A8, -A9, -A15… etc).
  • A FLIR USB3 camera: Blackfly® S, Blackfly®, Chameleon®3, Flea®3, or Grasshopper®3
  • Other FLIR machine vision cameras (FireWire, USB2, or CameraLink) are NOT supported, although we have tested GigE cameras to stream on select ARM boards.
  • An external power supply such as a powered USB hub or GPIO power cable.
  • Linux. The recommended version is Ubuntu 16.04

Configuring the Operating System and Installing the Required Libraries

For Spinnaker to run on a Linux Ubuntu system, the following dependencies must be installed:

  • Ubuntu 16.04

user$: sudo apt-get install libraw1394-11 libusb-1.0-0

Note: The raw1394 module that is installed with the libraw1394-8 package may not load after a reboot, causing a Spinnaker bus event error and failure to start an application. To fix, add raw1394 to the /etc/modules file. If problems persist, add video1394 as well.

Installing the Spinnaker SDK

To install the Spinnaker SDK:

  1. Download Spinnaker SDK for ARM from our Downloads page.
  2. Copy your spinnaker-<version>_arm.tar.gz package on a network or USB drive then copy it to a location on the ARM board.
  3. Untar the installation package:
    tar xvfz spinnaker-<version>_arm.tar.gz
  4. Run the install script:
    cd spinnaker-<version>_arm/
    sudo sh install_spinnaker_arm.sh
  5. Follow the instructions of the script. This installs all the Spinanaker libraries, example code, sample applications, and documentation. Additionally, the script prompts you to configure udev so that devices can be used by a particular user. If you choose to configure devices, the script changes permissions on the nodes by overwriting the default Ubuntu permissions and giving the user full read and write access to the device nodes.
  6. Restart your board for the user permissions to take effect.

Compiling the Examples

The Spinnaker SDK includes a number of example applications to help get you started in programming common API tasks. Example files are installed under /usr/src/.

We suggest you copy the extracted folder and sub-folders to a location with write access.

To compile the examples, install the GNU C++ (g++) compiler that is included with the build-essential package:

user$ sudo apt-get install build-essential

To compile a specific example, run the makefile located in the example directory. Binaries are copied to the bin directory, and libraries are copied to the lib directory. For example:

user$ cd <copied folder>/Spinnaker/src/Acquisition
user$ make

You can also use a cross compiler to build examples for the target architecture. A cross compiler that works well is CodeSourcery.

Alternatively, you can set up an ARM emulator using QEmu, install Ubuntu on it, and build your software on the virtual machine.

Limitations Using ARM

Linux users do not have access to Microsoft Windows-only technologies such as:

  • DirectShow
  • Cognex AIK
  • Twain
  • Managed .NET API
  • ActiveX

Spinnaker on an ARM device does not support:

Note: Spinnaker on an ARM device has limited color processing options.

Additional ARM device limitations:

  • The speed of the processor affects the maximum available frame rate. The maximum frame rates achieved with a Windows configuration may not be available.

Viewing Images and Videos

We suggest the following tools for image and video viewing. FLIR does not officially endorse these tools.

For image viewing:

For video viewing:

  • VLC media player

For working with Glade files:

Removing Spinnaker

Uninstall by manually removing the Spinnaker files, as in the following example:

user$: sudo sh
remove_spinnaker-arm.sh

Delete any extracted files or newly compiled files on your system.

Streaming on Embedded Systems

Note: The GS3-U3 was powered externally via the GPIO. The BFS-U3 was powered via the USB3 interface.

System Configuration for ODROID-XU4

ODROID-XU4 Board Specification

CPU

Exynos5422 Cortex-A15 – 2 GHz Quad and Cortex-A7 1.4 GHz Quad

GPU

ARM Mali-T628 MP6

RAM

2 GB LPDDR3

Host Adapter Driver

exynos-xhci

Operating System

Ubuntu 16.04.2 LTS with 3.10.105-141 Linux kernel

Software

Spinnaker 1.5.0.27 SDK Hard Float

Camera and Firmware

GS3-U3-23S6M (firmware 2.25.3.0)
BFS-U3-51S5C (firmware 1704.0.13.0)

Getting Started on ODROID XU4

  1. Follow Hardkernel’s Getting Started wiki to install Ubuntu 16.04 image on to the eMMC/microSD card.
  2. Download Spinnaker ARM hard float package (spinnaker-1.5.0.27-armhf-pkg.tar.gz or newer).
  3. Extract Spinnaker ARM hard float package.
  4. Retrieve dependency using command:
    sudo apt-get install libraw1394-11 libusb-1.0-0
  5. Install the Spinnaker deb files from the extract Spinnaker ARM directory using command:
    sudo sh install install_spinnaker_arm.sh

The default Ubuntu image supports images up to 2 MB. To support larger images, increase the USBFS to 1000 MB.

  1. Install git package using command sudo apt-get install git (may have to update the package list using sudo apt-get update beforehand)
  2. Download the kernel source code:
    $ git clone --depth 1 https://github.com/hardkernel/linux -b odroidxu3-3.10.y
    $ cd linux
  3. Modify ../linux/drivers/usb/core/devio.c from:
    static unsigned usbfs_memory_mb = 16
    to:
    static unsigned usbfs_memory_mb = 1000
  4. Compile and install the new kernel:
    $ make odroidxu3_defconfi
    $ make -j9
    $ sudo make modules_install
    $ sudo cp -f arch/arm/boot/zImage /media/boot
    $ sudo cp -f arch/arm/boot/dts/exynos5422-odroidxu3.dtb /media/boot

Benchmarks for ODROID XU4

These results are from a console application that continuously captures images using GS3-U3-23S6M and BFS-U3-51S5C-C. The benchmark using the console application is shown below.

GS3-U3-23S6M

Requested
Frame
Rate

Processed
Frame
Rate

CPU
Usage

Bandwidth

1280 x 960 Raw8

202 FPS

202 FPS

16%

234 MB/s

1920 x 1200 Raw8

163 FPS

163 FPS

13%

328 MB/s

BFS-U3-51S5C

Requested
Frame
Rate

Processed
Frame
Rate

CPU
Usage

Bandwidth

2448 x 2048 BayerRG8

75 FPS

75 FPS

11%

358 MB/s

System Configuration for NVIDIA Jetson TX1

NVIDIA Jetson TX1 Specification

CPU

Cortex A57

GPU

Maxwell GPU with 256 CUDA cores

RAM

4 GB

Host Adaptor Driver

xhci-hcd

Operating System

Ubuntu 16.04 Linux tegra-ubuntu 3.10.96-tegra

Software

Spinnaker 1.5.0.37 Linux ARM64

Camera and Firmware

GS3-U3-23S6M (firmware 2.25.3.0)
BFS-U3-51S5C (firmware 1704.0.13.0)

Getting Started on NVIDIA Jetson TX1

  1. Follow NVIDIA’s Quick Start Guide file to install the latest 64-bit LT4 release package. L4T release package ->Tegra X1->Jetson TX1 R24.2.1 - September 2016 was used for this application note.
    Note: This step overwrites your existing file system.
  2. Install 64-bit Spinnaker ARM by downloading the latest Spinnaker 64-bit ARM and follow the readme file for installation instructions. Spinnaker 1.5.0.27 SDK - ARM64 - Ubuntu 16.04 (64-bit) — 08/22/2017 was used for this application note.
  3. Install the g++ compiler in order to install the Spinnaker SDK’s C++ samples:
    sudo apt-get update
    sudo apt-get install build-essential

The steps above allow you to stream your USB3 camera at a maximum image size of 2 MB. To acquire images greater than 2 MB in resolution, add the following to the APPEND line:

usbcore.usbfs_memory_mb=1000

to this file:

/boot/extlinux/extlinux.conf

Benchmarks for NVIDIA Jetson TX1

These are the results from a console application that continuously captures images using GS3-U3-23S6M and BFS-U3-51S5C-C. The resulting benchmark using the console application is shown below.

GS3-U3-23S6M

Requested
Frame
Rate

Processed
Frame
Rate

CPU
Usage

Bandwidth

1280 x 960 Raw8

202 FPS

202 FPS

~32%

236 MB/s

1920 x 1200 Raw8

163 FPS

155 FPS

~30%

340MB/s

BFS-U3-51S5C

Requested
Frame
Rate

Processed
Frame
Rate

CPU
Usage

Bandwidth

2448 x 2048 BayerRG8

75 FPS

75 FPS

~37%

358 MB/s

System Configuration for NVIDIA Jetson TX2

NVIDIA Jetson TX2 Specification

CPU

Cortex A57

GPU

Pascal GPU with 256 CUDA cores

RAM

8 GB

Host Adaptor Driver

xhci-hcd

Operating System

Ubuntu 16.04 Linux tegra-ubuntu 4.4.38-tegra

Software

Spinnaker 1.5.0.27 Linux ARM64

Camera and Firmware

GS3-U3-23S6M (firmware 2.25.3.0)
BFS-U3-51S5C (firmware 1704.0.13.0)

Getting Started on the NVIDIA Jetson TX2

  1. Follow NVIDIA’s Quick Start Guide file to install the latest 64-bit Linux 4 Tegra release package.64-bit Linux for Tegra R28.1 was used for this application note.
    Note: This step overwrites your existing file system.
  2. Install 64-bit Spinnaker ARM by downloading the latest Spinnaker 64-bit ARM and follow the readme file for installation instructions. Spinnaker 1.5.0.27 SDK - ARM64 - Ubuntu 16.04 (64-bit) — 08/22/2017 was used for this application note.

The steps above allow you to stream your USB3 camera at a maximum image size of 2 MB. To acquire images greater than 2 MB in resolution, add the following to the APPEND line:

usbcore.usbfs_memory_mb=1000

to this file:

/boot/extlinux/extlinux.conf

Benchmarks for the NVIDIA Jetson TX2

These are the results from a console application that continuously captures images using GS3-U3-23S6M and BFS-U3-51S5C-C. The resulting benchmark using the console application is shown below.

GS3-U3-23S6M

Requested
Frame
Rate

Processed
Frame
Rate

CPU
Usage

Bandwidth

1920 x 1200 Raw8

163 FPS

163 FPS

~45%

358 MB/s

BFS-U3-51S5C

Requested
Frame
Rate

Processed
Frame
Rate

CPU
Usage

Bandwidth

2448 x 2048 BayerRG8

75 FPS

75 FPS

~50%

358 MB/s

Troubleshooting

Image transfer fails to start when image size is bigger than 2 MB

Cause: The USBFS buffer size is too small (16 MB by default). Use the following command to check usbfs_memory_mb size:

cat /sys/module/usbcore/parameters/usbfs_memory_mb

Solution: Increase the memory by adding the following to the APPEND line:

usbcore.usbfs_memory_mb=1000

to this file:

/boot/extlinux/extlinux.conf

If this method fails to set the memory limit, run the following command:

sudo sh -c 'echo 1000 > /sys/module/usbcore/parameters/usbfs_memory_mb'

Camera is detected but cannot stream

Cause: The USB3 bus power provided by the ARM board is not sufficient to power the camera.

Solution: Power the USB3 camera externally using a GPIO cable.