The sensor

Overview of the application

This application is a Windows XP console program. Its objective is to obtain the position and attitude of a flying object in the control space and make them available to other applications.
The measurements are calculated from the images obtained from the cameras in the laboratory. Each image is delivered to the application independently at a rate of 30fps. The XYZ coordinates and pitch, yaw, roll angles of the helicopter are transmitted to any other application that might need them through TCP port 20248.
The structure of the sensor application was designed in the beginnings of this project but its features have evolved over time. The structure of the application is shown in the figure below.

Strucure of the sensor application

It is a multithreaded application with four threads executing concurrently. Thread 1 and 2 are the image processing threads. Each one of them carries out exactly the same processing on the left or right images obtained from the cameras. The data they produce are sent to the triangulation thread, which calculates the 3D position and attitude. After that, thread 4 sends through TCP port 20248 (or UDP, both versions exist but TCP is more reliable) a string of text with the following format:

<n>tick number<x>position<y>position<z>position<p>pitch<w>yaw<r>roll<k>reliability<t>time

tick number: integer representing the current frame number
x,y,z position: floats representing helicopter position wrt camera 1 reference frame
pitch,yaw,roll: floats representing helicopter attitude (euler angles) in degrees 
reliability: confidence level of the data, where 0 is no confidence and 1 is absolute confidence
time: time stamp

This process is repeated until the user presses the return key.

Image processing

IR leds positions and labels

The processing carried out to detect the helicopter has evolved over time, too. First attempts worked with background subtracted images. The remaining object could be easily segmented and it was blobbed to extract its moments. The CoG was useful to approximate the position of the helicopter. Nevertheless, the shape variability of the blob depending on the attitude of the helicopter made it too difficult to obtain any further infromation from it.

The current sensor application is much more accurate and simple due to the use of infrared markers on the helicopter and filters in the cameras. The figure on the right shows the actual led distribution. They are placed in the middle of the longitudinal axis of the helicopter making a right-angled triangle. The images thus obtained are roughly binarized and the remainig connected pixels correspond to the leds of the helicopter. The images below show the process of binarization.

Image with IR filter

Binarized image

The only problem left is the identification of each led, i.e. the correct labeling of leds 1, 2 and 3 frame after frame. In order to be able to make an accurate identification, the helicopter needs to be initially placed so that led 2 lies under led 1 (which is a natural starting position).

After that, images are acquired continuously and three different cases are considered. The first case is when 3 leds are detected. In this case each led is labeled according to the shortest distance from the previous labeling (see figure 1, below). The second case is when 2 leds are detected and one is missing due to occlusions or alignments. In this case, a first order approximation of the motion of leds is considered and they are labeled according to the shortest distance from their estimated position (see figure 2, below). The confidence variable decreases inversely proportiortional to the number of frames the sensor application is in this situation. The third case is when one led or no leds are detected at all. This case usually happens only when the helicopter gets completely out of the control space. The confidence variable is zero.

figure 1

figure 2

Hardware arrangement

The IR markers of the helicopter were first active markers, composed by IR leds. Everey marker was a turret with three leds disposed at 120° to each other and a current limiting resistor. The three turrets were powered by an additional battery. The overall arrangement added an extra weight of 50gr (approx.) to the helicopter, which was too heavy for it to fly properly during more that 2 or 3 minutes.

Helicopter with active markers

Close view of the front turrets

Close view of the rear turret

Passive marker arrangement

The active arrengement was substituted by a passive one. A large array of leds was placed arround the camera and the turrets were replaced by cylinders of very high gain reflective tape. The landing skid was also replaced by a carbon fiber frame to avoid the lower marker hit the ground when landing. This arrangement is lighter and allows the helicopter to fly properly for about 10 minues.

Go back