Vision Guided Pick and Place with RoboDK

June 27, 2022

At Stray Robots, our goal is to enable cutting-edge robotics and computer vision capabilities for everyone. That’s why we partnered up with RoboDK, who make robot simulation and offline programming software, on this project to show you how easy it is to build automatic pick-and-place solutions using the Stray computer vision toolkit and RoboDK.

In this post we’ll go through the process of setting up a simulation environment, training a detection model, and running the whole system together. The great thing about RoboDK is that once you have a working simulation it is super easy to deploy the controller and other logic on a real robot environment.

The specific example we will use in this post is detecting boxes on a conveyor belt, picking them, and moving them onto another conveyor belt, all autonomously. We created a detailed tutorial with example data and models so you can skip to running the full example system right away. The only prerequisite is some basic python knowledge and you are good to go. You can find the tutorial here.

Setting up a robot environment

First, we set up a simulation environment in RoboDK. To do this, we go shopping in the RoboDK library to find all the components we need for our robot cell.

As a robot, we choose the ABB CRB 15000 robot arm. We take two conveyor belts from the library and place it on both sides of the robot. Finally, we insert a RoboDK cardboard box to serve as a canonical box model in the simulation.

We then write some Python code to program the logic of our simulation. In this example, we want a box to be spawned at one end of the conveyor belt and have boxes move down the conveyor belt, unless they are lifted off the belt. If the box makes it to the end of the conveyor belt, we simply spawn it back at the beginning. We continue this loop ad-infinitum. The task of the robot is to move boxes from one conveyor belt to the other.

To make the task more challenging, each time we spawn the box, we randomly change it’s height, width and depth as well as randomize the starting position and angle at which the box starts on the conveyor belt. This way, our robot can’t blindly assume a picking position and we actually have to compute this using computer vision.

You can find the full simulation setup in the repository, which you can clone and open up in RoboDK.

Collecting a computer vision dataset and training a model

To achieve automatic detection of boxes passing by the conveyor belt, we use learning-based methods that need annotated training data for fitting the detection model parameters. The specific parts we are trying to detect on the box are the top corners, which can be used to determine the pick location.

To produce the example data and annotations, we use the Stray CLI and Studio. The tutorial has instructions for collecting RGB-D scans of boxes arriving on the conveyor belt. We capture the boxes from multiple viewpoints, as the learning based algorithm needs quite a few examples to learn. This way we both get many examples at once and we get a diverse set of viewpoints to make the learned model robust to small changes in the images.

The scanning setup.

After the scans have been collected, they are processed into a 3D representation which are needed in the annotation process. The 3D representations of the boxes are easily produced using the Stray CLI. The tutorial uses 20 such scans, which is enough to create a working model in the example environment.

Once the 20 scans have been processed, they can be opened in the Stray Studio GUI for annotation. In the annotation view we use the rectangle primitive to mark the top of the box. The benefit of using the rectangle is that the corner positions of the box can easily be determined and the lengths of the edges stay consistent.

The Stray Studio GUI for 3D annotation.

Training the learning-based picking model is very simple, clear instructions have been provided in the tutorial. The model training iterates through the collected dataset 100 times, which is enough to learn a model that effectively detects the boxes. The robot controller makes use of this trained model in determining the pick location.

Running the full system

The tutorial provides a simple example script for running the full robot environment with the trained example model. The model first detects the corners of the box arriving on the conveyor belt and then determines the picking location by computing the center of the box and combining with the measured depth from the RGBD camera mounted on the robot to determine the 3D pick location. As the box is constantly moving, we additionally use the speed of the conveyor belt to compute the exact pick location further down the conveyor belt.

We want to highlight that the system is able to account for boxes of various sizes as, we can detect boxes of unknown size and combining with the depth capability of the camera.

After the picking location has been calculated, the robot controller moves the suction cup gripper of the robot to that location and picks up the box. The box is then moved to another conveyor belt on the other side. The video below shows how the full system looks like in action:

Getting started

The Stray computer vision tools are available for everyone to use, free of charge. For a RoboDK license, visit their website. RoboDK offers a free trial of the software which you can use to go through the tutorial discussed in this blog post.

If you have any projects requiring 3D computer vision, don't hesitate to reach out to, we would love to talk to you!

Other Blog Posts