Quantcast Robot Lends Helping Hand In Kitchen: Dragonfly Application Story
Case Studies
Click an item to read from the list below.
Case Studies
Perfect Book
Quality Kegs
Sweet Success

Robot Lends Helping Hand In Kitchen: Dragonfly Application Story

Collaborative Research Center robot


The Collaborative Research Center 588 (SFB 588) "Humanoid Robots—Learning and Cooperating Multimodal Robots" was established by the Deutsche Forschungsge-meinschaft (DFG) in 2001 in Karlsruhe, Germany. The Center focuses on the development of humanoid robots that can safely coexist with humans, perceive and interpret sensory data, and act and interactively communicate with humans within built-for-human environments. In particular, the group addresses multimodal human-humanoid interaction and human-humanoid cooperation in order to demonstrate robot tasks in a kitchen environment as a prototypical human-centered scenario.

The Research Center’s development efforts resulted in ARMAR III, a humanoid robot consisting of seven sub systems: sensor head with flexible neck, left and right arms with seven degrees-of-freedom, left and right five-finger hands, torso and mobile platform. The tasks of loading and unloading a dishwasher and stocking a refrigerator were used as benchmarks for the highest requirements for perception and action abilities for the robot. To perform such tasks in a household environment, ARMAR III had to be able to recognize objects of interest and localize them with a high degree of accuracy for grasping.

Head Start

The team, led by Prof. Rüdiger Dillmann, built a seven degrees-of-freedom head and neck system for ARMAR. Each of the head’s two eyes consist of a pair of Dragonfly® Extended Head 640x480 color cameras. The camera pairs are equipped with one wide and one narrow angle lens. This simple method is used to mimic human foveal vision. The imaging system facilitates simple visual-motor behaviors such as tracking and saccadic motions towards salient regions, as well as more complex visual tasks such as hand-eye coordination. The head features human-like characteristics in motion and response (the neck and eyes have human-like speed and range of motion). The robot head is also part of a related European project by PACO-PLUS, studying how cognitive capabilities emerge in artificial robot systems.

Scene Analysis

Scene left camera


The Collaborative Research team developed object recognition and localization systems for two classes of kitchen objects: those that could be segmented globally, and those with enough texture to allow the application of methods using local texture features.

To simplify the problem of segmentation and demonstrate how the robot could handle a complicated task like loading and unloading a dishwasher, colored plastic dishes were selected as target objects. Through the recognition and localization process, potential colored regions are segmented and matched in the left and right camera image, rejecting regions outside the areas of interest. Remaining regions are then matched based on their appearance in terms of gradients with the entries in the database. By combining stereo vision with the information about the orientation of the object that was stored with its view, it is possible to determine the full 6D pose with respect to the object’s 3D model at frame rate.



The second class of objects used includes textured objects such as tetra pack drink boxes, boxes containing any kind of food, and bottles similar to those found in a fridge. The research team developed a system based on local texture features, combining stereo vision, SIFT features, a kd-tree with best-bin-first search and a generalized Hough transform.

To address the visual perception requirement, the group chose the Dragonfly camera from Point Grey Research because of the availability of an extended head version—essential to building a lightweight eye unit. They also selected the Dragonfly camera because of its ability to generate color images, automatically synchronize multiple cameras on the same 1394 bus, and its compatibility with Linux software.


Designing a robot that could coexist with humans, interactively communicate with them and usefully manipulate objects within a built-for-humans environment required enormous collaboration between research efforts and scientific disciplines such as learning theory, artificial intelligence, human-machine interaction, mechatronics, biomechanics and computational neuroscience, but SFB 588 was well equipped for the job.

About the University of Karlsruhe and SFB 588

The University of Karlsruhe, founded in 1825, is the oldest technical university in Germany. In 2006, it succeeded in the first round of the Excellence Initiative launched by the German federal and state governments and was elected as one of the three Centers of Excellence in Germany. This success was based on their concept for the future—a central element of which was the foundation of the Karlsruhe Institute of Technology (KIT) together with the Forschungszentrum Karlsruhe.

SFB 588 includes the University of Karlsruhe, the partners Forschungszentrum Karlsruhe (FZK), Forschungszentrum Informatik (FZI) and Fraunhofer-Institut für Informations-und Datenverarbeitung (IITB). The SFB 588 group first unveiled ARMAR III at CeBIT 2006 in Hannover, Germany.