Technical Application Notes
Click an item to read from the list below.

Technical Application Notes


 

 

Streaming Cameras on Embedded Systems

Subject

Technical Application Note (TAN2014009): Streaming Cameras on Embedded Systems

10519

Applicable Product(s)

  • FLIR USB3 Vision cameras (excluding Blackfly S)
  • FLIR GigE Vision cameras (excluding Blackfly S)

 

Application Note Description

This technical application note provides a summary and instructions for streaming FLIR machine vision cameras using FlyCapture2 on ARM-based embedded boards. It includes examples of some of the more commonly used embedded boards: ODROID-XU; Samsung Exynos 5250 Arndale; and NVIDIA Jetson TK1, TX1, TX2 and DRIVE PX. The benchmark results show that embedded boards are able to support high-speed machine vision applications. See Getting Started with FlyCapture2 and ARM for information on using FlyCapture2 on ARM devices in a Linux environment.

 

System Configuration

 

These boards do not support Power over Ethernet (PoE), therefore all GigE models were powered externally via the GPIO. All GS3-U3 models were powered externally via the GPIO. The CM3 model was powered via the USB3 interface.

 

  ODROID-XU Board Specification
CPU Exynos5 Octa Cortex™-A15 - 1.6 GHz quad core
GPU PowerVR SGX544MP3
RAM 2 Gbyte LPDDR3
Host Adaptor Driver exynos-xhci
Operating System Ubuntu 13.10 with 3.4.67 linux kernel
Software FlyCap 2.6.3.3 Linux ARM
Camera and Firmware GS3-U3-23S6M (firmware 2.3.3.0)

 

  Samsung Exynos 5250 Arndale Specification
CPU Exynos5 Octa Cortex™-A15 - 1.6 GHz quad core
GPU ARM Mali T-604
RAM 2 Gbyte DDR3
Hard Drive SanDisk Extreme II SSD 480GB (SDSSDXP-480G)
Hard Drive Enclosure Oyen Digital MiniPro 2.5-inch eSATA 6 Gb/s, USB 3.1 hard drive enclosure
Host Adaptor Driver xhci-hcd
Operating System Arndale raring server release 13.09 with 3.12 Linux kernel
Software FlyCap 2.6.3.3 Linux ARM
Camera and Firmware GS3-U3-23S6M (firmware 2.3.3.0)

 

  NVIDIA Jetson TK1 Specification
CPU Tegra K1 SOC
GPU Kepler GPU with 192 CUDA cores
RAM 2 Gbyte
Host Adaptor System xhci-hcd
Operating System Ubuntu 14.04
Software FlyCap 2.6.3.3 Linux ARM
Camera and Firmware GS3-U3-23S6M (firmware 2.3.3.0)
GS3-PGE-23S6C (firmware 1.6.3.0)
FL3-GE-03S1M (firmware 1.26.3.0)

  NVIDIA Jetson TX1 Specification
CPU Cortex A57
GPU Maxwell GPU with 256 CUDA cores
RAM 4 Gbyte
Host Adaptor System xhci-hcd
Operating System Ubuntu 14.04
Software FlyCap 2.9.3.43 Linux ARM
Camera and Firmware GS3-U3-23S6M (firmware 2.22.3.0)
GS3-PGE-23S6M (firmware 1.10.3.0)

  NVIDIA Jetson TX2 Specification
CPU Cortex A57
GPU Pascal GPU with 256 CUDA cores
RAM 8 Gbyte
Host Adaptor System xhci-hcd
Operating System Ubuntu 16.04
Software FlyCap 2.11.3.121 Linux ARM
Camera and Firmware GS3-U3-23S6M (firmware 2.25.3.0)
GS3-PGE-23S6M (firmware 1.13.3.0)
BFLY-PGE-23S6M (firmware 1.61.3.0)

 

  NVIDIA DRIVE PX Specification
CPU Tegra X1 64-bit processor
GPU Maxwell GPU with 256 CUDA cores
RAM 4 Gbyte
Host Adaptor System xhci-hcd
Operating System Ubuntu 15.04 with 3.18.21-tegra-g06aec38
Linux NVIDIA kernel
Software FlyCap 2.9.3.43 Linux ARM64
Camera and Firmware BFLY-PGE-50S5C-C (firmware 1.53.3.0)
BFLY-PGE-23S6M-C (firmware 1.40.3.0)
CM3-U3-50S5M-CS (firmware 1.10.3.0)
Note: To achieve maximum bandwidth:
GigE cameras require a Gigabit adapter connected directly to an on-board USB port.
USB3 cameras require a custom kernel as described in NVIDIA DRIVE PX Getting Started.

  

System Setup

ODROID-XU Getting Started

1. To boot from the microSD card, ensure boot media SW1-1 is set to OFF and SW2-2 is set to ON.

2. Download xUbuntu 13.10 image on a Linux PC running Ubuntu.

3. Unpack the files. For an .xz extension, use xz -d image.img.xz on cmd line.

4. Insert the microSD card in the Ubuntu PC.

Note: Windows users can use Windows tool Win32DiskImager.exe to write an image to the microSD card.

5. Clear the destination media by entering dd if=/dev/zero of=/dev/sdX bs=4M

Note: /dev/sdX where X denotes the microSD card’s file storage.

6. Flash your image: sudo dd if=image.img of=/dev/sdX bs=4M

7. Enter sync before removing the media to make sure that all data is written.

8. Download FlyCapture2 ARM hard float package (flycapture.2.6.3.2_armhf.tar.gz or newer).

9. Retrieve dependency using command sudo apt-get install libglademm-2.4-dev

10. Extract FlycCapture2 ARM hard float package and run flycap2-conf as root.

11. Reboot ODROID-XU.

12. In the terminal, change to FlyCapture2 ARM hard float directory’s bin folder.

13. Run the command export LD_LIBRARY_PATH=./../lib

14. Start FlyCapture2 using command ./FlyCap2

Samsung Exynos 5250 Arndale Getting Started

1. Download the Linaro image for Arndale. This configuration used Version 13.11.

2. Insert the microSD card in the Ubuntu PC.

3. Write the image to a microSD card.

Note: Windows users can use Windows tool Win32DiskImager.exe to write an image to the microSD card.

4. Download the FlyCapture2 ARM hard float package (flycapture.2.6.3.2_armhf.tar.gz or newer).

5. Extract the Flycapture2 ARM hard float package and run flycap2-conf as root.

6. Reboot.

7. In the terminal, change to Flycapture2 ARM hard float directory’s bin folder.

8. Run the command export LD_LIBRARY_PATH=./../lib

9. Start FlyCapture2 using command ./FlyCap2

NVIDIA Jetson TK1 Getting Started

 

Make sure you have at least 15 GB of free space on your Linux machine. This file copying process takes approximately one hour. The flashing process ends with 'Reset the board to boot from internal eMMC.'

1. Follow NVIDIA’s readme file and boot up the pre-installed Ubuntu 14.04 on TK1.

2. By default, Jetson TK1 does not provide support for USB3 cameras. To enable USB3 support:

a. Go to https://developer.nvidia.com/linux-tegra-rel-19 and download both of these tar-balls:

• Driver Package: Jetson TK1

• Sample file system

b. On the Linux PC, unzip the tar files by running the command:

export RELEASE_NAME=Tegra124_Linux_R19.3.0_armhf.tbz2
sudo tar xpf ${RELEASE_NAME}
cd Linux_for_Tegra/rootfs/
sudo tar xpf ../../Tegra_Linux_Sample-Root-Filesystem_R19.3.0_armhf.tbz2
cd ../

Note: It’s important to use sudo to extract these files.

c. In the extracted Linux_for_Tegra folder, edit the jetson-tk1.conf file by uncomment the line (ODMDATA=0x6209C000) for USB2 and comment in the line for USB3 (ODMDATA=0x6009C000).

d. Enter sudo ./apply_binaries.sh

e. Connect the TK1 to the Linux machine using the micro-USB cable that comes with the TK1.

f. On the TK1, hold down the Recovery button and press the Reset button once. Wait two seconds and release the Recovery button.

g. Verify that the TK1 is connected to the Linux PC by issuing the lsusb command on the Linux PC and look for a connected usb device called NVIDIA Corp.

h. Copy the Linux_for_Tegra folder unzipped in step 2 to your Linux machine.

i. In the Linux machine, enter sudo ./flash.sh -S 14GiB jetson-tk1 mmcblk0p1

3. Reboot TK1 by pressing the Reset button.

The steps above will allow you to stream your USB3 camera at a maximum image size of 2 MB. To acquire images greater than 2 MB in resolution, add the following to the APPEND line:

usbcore.usbfs_memory_mb=1000

to this file:

/boot/extlinux/extlinux.conf

NVIDIA Jetson TX1 Getting Started

  1. Follow NVIDIA’s Quick Start Guide file to install the 64-bit Linux 4 Tegra R24.1 package.
    Note: This step overwrites your existing file system.
  2. Install 64-bit FlyCapture2 ARM by downloading the latest FlyCapture2 64-bit ARM and follow the readme file for installation instructions.
  3. Install the g++ compiler in order to install the FlyCapture2 SDK’s C++ samples

sudo apt-get update
sudo apt-get install build-essential

The steps above allow you to stream your USB3 camera at a maximum image size of 2 MB. To acquire images greater than 2 MB in resolution, add the following to the APPEND line:

usbcore.usbfs_memory_mb=1000

to this file:

/boot/extlinux/extlinux.conf

NVIDIA Jetson TX2 Getting Started

  1. Follow NVIDIA’s Quick Start Guide file to install the 64-bit Linux 4 Tegra R24.1 package.
    Note: This step overwrites your existing file system.
  2. Install 64-bit FlyCapture2 ARM by downloading the latest FlyCapture2 64-bit ARM and follow the readme file for installation instructions.

The steps above allow you to stream your USB3 camera at a maximum image size of 2 MB. To acquire images greater than 2 MB in resolution, add the following to the APPEND line:

usbcore.usbfs_memory_mb=1000

to this file:

/boot/extlinux/extlinux.conf

NVIDIA DRIVE PX Getting Started

To obtain maximum bandwidth with DRIVE PX:

  • USB3 cameras require the latest version of a custom kernel from GitHub, as described below.
  • GigE cameras require a USB 3.1-to-Ethernet adapter connected directly to an on-board USB port.

For USB3 cameras:

  1. Get the latest kernel from GitHub:

    # Clone source repo for mainline kernel
    git clone https://github.com/torvalds/linux.git
    # Get list of tags to determine appropriate release (I settled on v4.7.0-rc7 for specific technical reasons).

    git tag -l

    # Checkout specific tagged revision to local branch.
    # git checkout tags/<tag_name> -b <branch_name>
    git checkout tags/v4.7-rc7 -b nv_custom_branch

  2. To acquire images greater than 2 MB in resolution, modify the os_args parameter of the KERNEL_PRIMARY partition in:

    vibrante-<platform|ver>-foundation/utils/scripts/bootburn/<cfg_file>

    Add the following to os_args:

    usbcore.usbfs_memory_mb=1000
  3. After the kernel is built, flash the custom kernel onto the board. Because the process depends on the board and kernel version, consult the NVIDIA Vibrante Linux Development guide at /home/nvidia/PDK_Docs/ for specific instructions.
    Note: You only want to re-flash the kernel, not the complete file system. To avoid flashing the complete file system, add the "-s" option (skip filesystem) to the bootburn.sh command:

    "./utils/scripts/bootburn/bootburn.sh -b e2379b00a -s"

    The sample in the documentation does not use the -s.

For GigE and USB3 cameras:
(Note: Connect the GigE camera to the PC with a USB-to-Ethernet adapter.)

  1. Install the g++ compiler in order to install the FlyCapture2 SDK's C++ samples:

    sudo apt-get update
    sudo apt-get install build-essential

  2. Install 64-bit FlyCapture2 ARM by downloading the latest FlyCapture2 64-bit ARM and follow the README file for installation instructions.

Benchmark Results

The following tables benchmark the performance of running selected USB3 or GigE cameras on the following boards: ODROID-XU; Samsung Exynos 5250 Arndale, and NVIDIA Jetson TK1 sand TX1, and DRIVE PX. 

ODROID-XU

These results are from a console application that continuously captures images using GS3-U3-23S6M. The benchmark using the console application is shown below. 

 

Console application for ODROID-XU

 

  Requested Frame Rate Processed Frame Rate CPU Usage Bandwidth
1280 x 960 Raw8 200 FPS 200 FPS 22% 234 MB/s
1792 x 960 Raw8 200 FPS 200 FPS 28% 328 MB/s

GS3-U3-23S6M benchmark results

 

Currently, ODROID-XU officially supports up to Linux kernel 3.4.67. Expect the CPU usage on the ODROID-XU board to reduce by 50% when running Linux kernel 3.6 or higher.


Samsung Exynos 5250 Arndale

Using FlyCapture2 to enumerate camera

 

These are the results from a console application that continuously captures images using GS3-U3-23S6M.

 

 

Console application for Arndale

This is a typical solution for an in-field system solution where the embedded board is portable and capable of performing basic processing as well as image/data saving. The benchmark results using the console application are shown below. 

 

  Requested Frame Rate Processed Frame Rate CPU Usage Bandwidth Image Save Speed
1280 x 960 Raw8 200 FPS 200 FPS 24% 234 MB/s 190 MB/s
1792 x 960 Raw8 200 FPS 200 FPS 28% 328 MB/s 170 MB/s
1920 x 1200 Raw8 162 FPS 162 FPS 29% 356 MB/s 176 MB/s

GS3-U3-23S6M benchmark results

 

GS3-U3-23S6M running at 1280 x 960 resolution, 24% CPU usage

GS3-U3-23S6M running at 1280 x 960 resolution, Raw8 pixel format, approximately 200 FPS

 

The speed of image saving is dependent on the hard drive. Typical hard drive write speed is advertised with sequential write speed. When benchmarking hard drives, one should use a file size similar to the image size.


GS3-U3-23S6M running at 1280 x 720 resolution, Raw8 pixel format, 24 FPS

NVIDIA Jetson TK1 

These are the results from a console application that continuously captures images using GS3-U3-23S6M, GS-PGE-23S6C, and FL3-GE-03S1M. The resulting benchmark using the console application is shown below. 

Console application for NVIDEA Jetson TK1

 

    Requested Frame Rate Processed Frame Rate CPU Usage Bandwidth
1280 x 960
Raw8
200 FPS 200 FPS 24% 234 MB/s
1920 x 1200
Raw8
162 FPS 162 FPS 30% 356 MB/s

GS3-U3-23S6M benchmark results

 

    Requested Frame Rate Processed Frame Rate CPU Usage Bandwidth
1920 x 1200
Raw8
45 FPS 45 FPS 25% 103 MB/s

GS3-PGE-23S6C benchmark results

 

    Requested Frame Rate Processed Frame Rate CPU Usage Bandwidth
640 x 480
Raw8
120 FPS 120 FPS 24% 36 MB/s

Flea3-GE-03S1M benchmark results

 

NVIDIA Jetson TX1 

These are the results from a console application that continuously captures images using Grasshopper3 GS3-U3-23S6M and Grasshopper3 GS-PGE-23S6M. The resulting benchmark using the console application is shown below. In our console application, we enabled maximum packet size supported on the Gigabit host controller, which is 9000 bytes.

Console application for NVIDEA Jetson TX1

    Requested Frame Rate Processed Frame Rate CPU Usage Bandwidth
1280 x 960
Raw8
201 FPS 201 FPS 27% 234 MB/s
1920 x 1200
Raw8
163 FPS 163 FPS 27% 356 MB/s

GS3-U3-23S6M benchmark result using NVIDIA Jetson TX1

 

    Requested Frame Rate Processed Frame Rate CPU Usage Bandwidth
1920 x 1200
Raw8
45 FPS 45 FPS 30% 104 MB/s

GS3-PGE-23S6M benchmark result using NVIDIA Jetson TX1

NVIDIA Jetson TX2 

These are the results from a console application that continuously captures images using Grasshopper3 GS3-U3-23S6M, Grasshopper3 GS-PGE-23S6M, and Blackfly BFLY-PGE-23S6M. The resulting benchmark using the console application is shown below. In our console application, for GigE cameras we enabled maximum packet size supported on the Gigabit host controller, which is 9000 bytes.

Console application for NVIDEA Jetson TX1

    Requested Frame Rate Processed Frame Rate CPU Usage Bandwidth
1920 x 1200
Raw8
163 FPS 163 FPS 25% 375 MB/s

GS3-U3-23S6M benchmark result using NVIDIA Jetson TX1

 

    Requested Frame Rate Processed Frame Rate CPU Usage Bandwidth
1920 x 1200
Raw8
45 FPS 46 FPS 25% 105 MB/s

GS3-PGE-23S6M benchmark result using NVIDIA Jetson TX2

    Requested Frame Rate Processed Frame Rate CPU Usage Bandwidth
1920 x 1200
Raw8
41 FPS 41 FPS 25% 94 MB/s

BFLY-PGE-23S6M benchmark result using NVIDIA Jetson TX2

NVIDIA DRIVE PX 

These are the results from a console application that continuously captures images using BLFY-PGE-23S6M, BFLY-PGE-50SM-C and CM3-U3-50SM. The resulting benchmark using the console application is shown below.

Console application for NVIDEA DRIVE PX

    Requested Frame Rate Processed Frame Rate CPU Usage Bandwidth
2448 x 2048
Mono16
18 FPS 18 FPS 80% 85 MB/s

BFLY-PGE-23S6M benchmark results

 

    Requested Frame Rate Processed Frame Rate CPU Usage Bandwidth
2448 x 2048
Mono16
9 FPS 9 FPS 80% 92 MB/s

BFLY-PGE-5056M benchmark results

    Requested Frame Rate Processed Frame Rate CPU Usage Bandwidth
1920 x 1200
Mono16
19 FPS 19 FPS 40% 23 MB/s

CM3-U3-5056M benchmark results

Troubleshooting

Image Transfer Fails to Start when Image Size is Bigger than 2 MB

Cause: The USBFS buffer size is too small (16 MB by default). Use the following command to check usbfs_memory_mb size.

cat /sys/module/usbcore/parameters/usbfs_memory_mb

Solution: Increase the memory by adding the following to the APPEND line:

usbcore.usbfs_memory_mb=1000
to this file:
/boot/extlinux/extlinux.conf

If this method fails to set the memory limit, run the following command:
sudo sh -c 'echo 1000 > /sys/module/usbcore/parameters/usbfs_memory_mb'

Camera is Detected but Cannot Stream

Cause: The USB3 bus power provided by the ARM board is not sufficient to power the camera.

Solution: Power the USB3 camera using a GPIO cable.

GigE Camera Frame Rates are extremely low on Jetson TK1, TX1, or TX2 board

Cause: The socket buffer size by default is too low.

Solution: Increase socket buffer size by typing in the following in Terminal:

sudo sysctl –w net.core.rmem_max=33554432
sudo sysctl –w net.core.wmem_max=33554432
sudo sysctl –w net.core.rmem_default=33554432
sudo sysctl –w net.core.wmem_default=33554432