FireWire Vision Tools A simple MATLAB interface to FireWire cameras

FireWire Vision Tools
A simple MATLAB interface to
FireWire cameras
Selec t objec t to be trac ked...
100
90
80
70
60
50
40
30
20
10
20
40
60
80
100
F. Wörnle, March 2006
FireWire Vision Tools
A simple MATLAB interface to FireWire cameras
1
FireWire Vision Tools
A simple MATLAB interface to FireWire cameras
Contents
1
Introduction
4
2
The FireWire Vision Interface
6
2.1
Supported video formats
6
2.2
Image capture commands
7
2.3
Image processing commands
9
2.4
Selecting colour ranges
13
2.5
Simulink interface
13
2.6
Simple stereoscopic vision
16
------------------------------------------------------------------------------------------------
Appendices
18
Appendix A
The MATLAB script yuv2rgb
20
Appendix B
The MATLAB script yuyv2rgb
21
Appendix C
The camera configuration file
22
Appendix D
A test program for capProc
23
Appendix E
Recompiling the drivers
24
2
FireWire Vision Tools
A simple MATLAB interface to FireWire cameras
3
FireWire Vision Tools
1.
A simple MATLAB interface to FireWire cameras
Introduction
This document describes a small collection of drivers which have been written
to provide direct access from within MATLAB to FireWire cameras. The
commands of the toolbox make use of MATLAB’s standard C-Mex interface
and/or Simulink S-Functions. Live images can thus be processed directly from
the MATLAB command prompt as well as from within Simulink block
diagrams.
A few of the provided drivers include rudimentary image processing
capabilities. It is possible to classify image contents according to a
programmable colour range and/or object size criteria. Furthermore, the
algorithm readily returns the coordinates of the centroid of any detected
object. This feature is particularly useful in machine vision based robotics
applications including the automatic tracking of coloured objects (e. g.
RoboCup, etc.). Finally, a set of commands allow for two cameras to be
connected to the same firewire bus. This can be used to implement and
experiment with simple stereoscopic vision systems. The relevant commands
have the label ‘Stereo’ as part of the filename.
The drivers of this toolbox have been written for and work best with “UniBrain
fire-i” cameras (www.unibrain.com). Access to the low-level Windows Driver
Model (WDM) stream class drivers is commonly provided through the
DirectX/DirectShow framework. The complexity of the DirectX/DirectShow
framework is considerable. This contribution therefore makes use of a
universal driver for firewire cameras, the ‘1394 Digital Camera Driver’
(http://www.cs.cmu.edu/~iwan/1394/) developed at the Robotics Institute of
Carnegie Mellon University (http://www.ri.cmu.edu/). This driver provides a
simple to use Application Programming Interface (API) which should work with
the majority of firewire cameras.
The classification algorithms are those of the Color Machine Vision project
(CMVision), a project conducted by the CORAL group at the Carnegie Mellon
School of Computer Science (http://www.cs.cmu.edu). This software has been
published under the GNU General Public License (GPL) and can be obtained
from http://www-2.cs.cmu.edu/~jbruce/cmvision/ (CMVision project page).
CMVision provides “a simple, robust vision system suitable for real time
robotics applications” by means of “global low level colour vision at video
rates without the use of special purpose hardware”.
To install and use the FireWire Vision Tools toolbox, extract the contents of
the zip to a local folder. The _bin folder of the distribution should be added to
the MATLAB path. Obtain the 1394 Digital Camera Driver (see above). Install
this driver for each of the cameras to be used with this toolbox. Installation
instructions for this driver are included with the driver distribution.
The FireWire Vision Tools are distributed as ‘Free Software’ under the terms
of the GNU General Public License Agreement. Users are therefore given the
right to copy, re-distribute and/or modify the source code to suit their needs.
4
FireWire Vision Tools
A simple MATLAB interface to FireWire cameras
Comments and bug reports are always welcome. Please direct your feedback
to the following address:
Frank Wornle (frank.wornle@adelaide.edu.au)
The University of Adelaide
School of Mechanical Engineering
1 March 2006
5
FireWire Vision Tools
A simple MATLAB interface to FireWire cameras
2.
The FireWire Vision Tools interface
2.1
Supported video format
The Imaging Source FireWire cameras can be configured to produce digitized
image information in a multitude of formats. Two popular video standards are
currently supported by FireWire Vision Tools:
- RGB24
Each pixel is represented by a colour triplet of three
consecutive bytes: Red, Green, Blue.
- YUV
This version of the toolbox assumes that the camera can
be setup to send data in YUV411 format. The latter
encodes groups of 4 adjacent pixels using 6 bytes (i. e.
1.5 bytes/pixel). Each pixel carries its own luminance
byte (Y1 – Y4) but the colour information (chrominance,
Cr, Cb or, equivalently, U, V) is the same for all 4 pixels.
Compared to YUV422 (=YUYV → 4 bytes for 2 pixels, or
2 bytes/pixel) this amounts to a saving of half a byte
every pixel. The total transmission bandwidth therefore
reduces by 0.5/2 = 0.25 (25 %). Compared to RGB (3
byte per pixel) the saving is 1.5/3 = 0.5 (50 %). This is
the reason why the UniBrain fire-i camera transmits
YUV411 images at 30 frames per second (fps),
compared to the 15 fps for RGB.
Note:
(1)
The camera used during the development of this toolbox is a UniBrain
fire-i. This camera can be set up to work with a resolution of 640 x 480.
The maximum frame rate is 15 fps in RGB mode or 30 fps in YUV411
mode.
(2)
The CMVision colour detection algorithms work in the YUYV colour
space (= YUV422). The drivers therefore have to convert a transmitted
RGB image / YUY411 image to the required YUVY format. An outline
of the underlying algorithms can be found in Appendices A and B.
(3)
For more details about YUV411 and conversion routines to other
formats see astronomy.swin.edu.au/~pbourke/dataformats/yuv.
6
FireWire Vision Tools
2.2
A simple MATLAB interface to FireWire cameras
Image capture commands
The following list describes the commands which can be used from the
MATLAB command line to acquire visual information and return it to the
workspace in form of a cell array.
2.2.1 capImage(0)
This command acquires a single frame of the camera. The call-up parameter
defines the video format to be used (0 = RGB); the command returns an n x n
x 3 cell array containing the corresponding Red, Green and Blue values.
The following line can be used to display an RGB image:
>> image(capImage(0));
50
100
150
200
250
300
350
400
450
100
200
300
400
500
600
640 x 480 pixel RGB image
The following lines cause MATLAB to continuously acquire RGB images and
display them on screen (the program stops after 100 frames). Upon
7
FireWire Vision Tools
A simple MATLAB interface to FireWire cameras
completion of 100 cycles, the grabber is switched off by calling the driver with
call-up parameter ‘-1’.
% start camera
for(i=1:100)
image(capImage(0));
drawnow
end
% stop grabber
capImage(-1);
2.2.3 capImage(-1)
A call to capImage with call-up parameter ‘-1’ stops the data acquisition (not
necessary, but it can be used to reduce the current drawn by the camera).
The next call to capImage will re-initialize the device. This has to be done
when switching from RGB mode (15 fps) to YUV mode (30 fps) or back.
2.2.4 capImage(1)
Captures YUV411 images and returns them to MATLAB. The transmission of
a YUV411 image requires less bandwidth than that of a regular RGB image
(YUV411 use 6 bytes to represent four pixels: U, Y1, Y2, V, Y3, Y4). The
function performs all required graphic format conversions (YUV411 →
YUV422 → RGB) before finally returning an RGB image in MATLAB format.
Note:
The camera settings are commonly read from a file cameraconfig.txt, which
should be in the folder the drivers are called from. If this file cannot be found,
the following hard coded default values are used. These values have been
found using ‘1394CameraDemo’, the demo program that comes with the CMU
camera driver. They work well in our indoor environment with its specific
lighting conditions. For valid ranges, please consult the documentation of your
camera.
Shutter speed:
Saturation level:
White balance:
Gain:
Auto exposure:
Brightness:
6
90
96, 69 (fixed → automatic adjustment switched off)
87
511 (fixed → automatic adjustment switched off)
304 (fixed → automatic adjustment switched off)
8
FireWire Vision Tools
2.3
A simple MATLAB interface to FireWire cameras
Image processing commands
The commands in this section combine image acquisition with the data
classification algorithms of the CMVision project (see: The CORAL group,
http://www-2.cs.cmu.edu/~jbruce/cmvision/).
2.3.1 capProc()
This command can be used to detect and track objects. A number of
CMVision algorithms are used to classify the acquired image data. Clusters
with the requested size and/or colour qualities are being detected and made
available to MATLAB in form of a cluster information structure. This structure
include among other things the position and size of a rectangular box around
the cluster, as well as the coordinates of its centroid. The location of the
centroid can directly be used in tracking applications.
50
100
150
200
250
300
350
400
450
100
200
300
400
500
600
Detecting 3 different colours - Cluster boundary box and cluster centroid
The above example demonstrates the situation for a relatively selective colour
range (only a few shades of each detected colour are accepted). A boundary
box has been plotted to reveal the size of the detected cluster. The small star
denotes the location of the centroid.
9
FireWire Vision Tools
A simple MATLAB interface to FireWire cameras
To learn about the admissible colour ranges, the command reads the
specified colour file (e.g. ‘color.txt’). This text file contains colour attributes for
up to 32 different objects to be detected. The general structure of the colour
file is shown below:
[Colors]
(255,128, 0)
(255,255, 0)
( 0, 0,255)
(255,255,255)
(255, 0,255)
(160, 0,160)
( 0,160, 0)
( 0, 0, 0)
( 0, 0, 0)
( 0, 0, 0)
( 0, 0, 0)
( 0, 0, 0)
( 0, 0, 0)
( 0, 0, 0)
( 0, 0, 0)
( 0, 0, 0)
(255,128, 0)
(255,255, 0)
( 0, 0,255)
(255,255,255)
(255, 0,255)
(160, 0,160)
( 0,160, 0)
0.5000
0.6000
0.6000
0.0000
0.0000
0.0000
0.0000
0.0000
0.0000
0.0000
0.0000
0.0000
0.0000
0.0000
0.0000
0.0000
0.0000
0.0000
0.0000
0.0000
0.0000
0.0000
0.0000
2
3
3
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
Ball
Yellow_Team
Blue_Team
White
Marker_Pink
Marker_Purple
Marker_Green
C08
C09
C10
C11
C12
C13
C14
C15
C16
Ball_2
Yellow_Team_2
Blue_Team_2
White_2
Marker_Pink_2
Marker_Purple_2
Marker_Green_2
[Thresholds]
( 7:175, 50:150,160:200)
( 47:120, 5:80, 130:200)
( 76:112,110:190, 67:128)
(130:255, 81:131,125:178)
( 50:181,102:135,190:222)
(103: 96,118:140,144:166)
( 67:134, 96:129, 85:124)
( 0: 0, 0: 0, 0: 0)
( 0: 0, 0: 0, 0: 0)
( 0: 0, 0: 0, 0: 0)
( 0: 0, 0: 0, 0: 0)
( 0: 0, 0: 0, 0: 0)
( 0: 0, 0: 0, 0: 0)
( 0: 0, 0: 0, 0: 0)
( 0: 0, 0: 0, 0: 0)
( 0: 0, 0: 0, 0: 0)
( 0: 0, 0: 0, 0: 0)
( 86:221, 35: 79,102:150)
( 0: 0, 0: 0, 0: 0)
( 0: 0, 0: 0, 0: 0)
( 0: 0, 0: 0, 0: 0)
( 0: 0, 0: 0, 0: 0)
( 0: 0, 0: 0, 0: 0)
10
FireWire Vision Tools
A simple MATLAB interface to FireWire cameras
The section [Colors] includes four columns:
(1)
A user defined RGB colour triplet which can be used to visualize the bits
of a detected cluster. This can be useful to validate the settings of a
particular colour detection range. An example with four different detected
regions is shown below.
220
200
180
160
140
120
100
80
60
40
20
50
100
150
200
Setting all pixels of a detected cluster to its default colour
(2)
The merge density parameter is assigned a value between 0 and 1. It
defines a colour density threshold beyond which two individual clusters
are being merged into one bigger cluster. This parameter therefore
allows control of the granularity of detected clusters within an acquired
image.
(3)
Colour ID; this parameter can be used to identify clusters of a particular
colour quality.
(4)
Colour name; this optional name can be used to link a detected object to
a clear text identifier.
The section [Thresholds] contains the RGB colour ranges which are used to
detect a particular cluster.
For further details about the CMVision algorithms, please refer to the
CMVision manuals (http://www-2.cs.cmu.edu/~jbruce/cmvision/).
capProc can be invoked with one or two call-up parameters and one or two
return parameters:
11
FireWire Vision Tools
A simple MATLAB interface to FireWire cameras
The first call-up parameter should always be set to 1 to make the camera
acquire YUV images (required by the CMVision algorithms). The optional 2nd
parameter is a vector of colour IDs to be detected. When omitted, the
command only detects objects defined by the first entry in the colour file.
capProc always returns the result of the image processing step (clusters,
centroids, colours, names, ...). Additionally, the corresponding image data can
be returned in an optional 2nd output parameter (RGB). An example of how to
use capProc() can be found in appendix C.
2.3.2 capClassify()
The command capClassify can be used to find appropriate settings for a
particular parameter in the colour file. All pixels of a detected cluster are set to
their default colour (1st column in section [Colors]).
220
200
180
160
140
120
100
80
60
40
20
50
100
150
200
Finding the correct settings of colour range and density threshold
2.3.3 imgProc()
The command imgProc can be used to test the validity of a chosen colour
definition. This command takes a previously stored RGB image as input, as
well as the colour filename and an optional vector of colour IDs. The
command returns a list of all detected regions together with their size, coordinates of the centroid, etc. See the test program in folder /imgProc for
further details.
12
FireWire Vision Tools
2.4
A simple MATLAB interface to FireWire cameras
Selecting colour ranges
Object tracking by colour detection relies on the correct definition of
appropriate RGB colour ranges. Too wide a range will lead to an unwanted
detection of objects with similar colour qualities; on the other hand, too narrow
ranges might cause the algorithm to return without having detected anything.
To determine appropriate RGB ranges, a number of small MATLAB m-files
have been provided. These ‘training’ programs allow a user to select an object
to be tracked from either a still image (trainStill.m) or a live stream of images
(trainCamera.m). Both programs return the colour characteristics of the
selected object. The thus established values (ranges of Y, U, V) can directly
be copied to a colour definition file.
Additional information on the issue of colour training can be found in the
document ‘Train_UserManual.doc’ which has been included in the present
distribution of the toolbox.
Training the camera to ‘see’ yellow…
2.5
Simulink interface
Object tracking can also be done in Simulink. A suitable SFunction block has
been designed and can be found in folder /SFcapProc. This block analyses
the current image of the camera and returns the coordinates (x, y) of the first
detected object of each specified colour. The block mask allows specification
of the colour definition file. The (optional) display of image information can be
enabled using the ‘display’ check box.
13
FireWire Vision Tools
A simple MATLAB interface to FireWire cameras
Block parameters: colour definition file and (optional) display mode
A Simulink based vision processing system
The display of images can be switched to ‘classify’ mode. This reveals which
objects the camera currently associates with the provided colour definitions.
Each detected object is coloured in the associated display colour (see colour
definition file, section 2.3.1). A boundary box is drawn around each selected
object and the centroid is displayed.
14
FireWire Vision Tools
A simple MATLAB interface to FireWire cameras
450
400
350
300
250
200
150
100
50
100
200
300
400
500
600
Classify mode: Detected objects are coloured, undetected objects are black
450
400
350
300
250
200
150
100
50
100
200
300
400
500
600
Regular display mode: Detected objects have centroids and boundary boxes
15
FireWire Vision Tools
2.6
A simple MATLAB interface to FireWire cameras
Simple stereoscopic vision
Two or more UniBrain fire-i cameras can be daisy-chained to transmit images
on the same firewire bus interface. This can be used to provide a simple
MATLAB interface for experiments with stereoscopic vision. The driver
command ‘capImageStereo’ assumes that two cameras are connected to the
same bus interface. The following command sequence illustrates how two
images can be captured by ‘capImageStereo’ or captured and processed by
‘capProcStereo’ (uses a common colour definition file) or ‘capProcStereo2’
(uses two separate colour definition files). Note that all cameras are usually
slightly different – in the figure below, the brightness property of camera 1 is
larger (187) than that of camera 2 (87). The camera settings can be adjusted
iteratively until one and the same colour detection specification works equally
well with both cameras. Alternatively, ‘capProcStereo2’ could be used:
[out1, out2, im1, im2] = capProcStereo2(0, 'c1.txt', 'c2.txt');
subplot(1,2,1)
image(im1); title('first camera'); axis image; drawnow
subplot(1,2,2)
image(im2); title('second camera'); axis image; drawnow
A simple stereoscopic vision system
Note that UniBrain fire-i cameras do not provide for synchronized sampling.
Therefore, strictly speaking, this is not stereoscopic vision. However, if the
objects to be tracked are not moving too fast, this method is an easy way to
experiment with stereoscopic vision systems. More information about
synchronized image acquisition using firewire cameras can be found in a
document that comes with the UniBrain fire-i camera. A copy of this document
can be found at:
http://www.mecheng.adelaide.edu.au/robotics_novell/WWW_Devs/FireWireVi
sionTools/FireWire_and_IIDC_cameras.pdf
16
FireWire Vision Tools
A simple MATLAB interface to FireWire cameras
17
FireWire Vision Tools
A simple MATLAB interface to FireWire cameras
Appendices
18
FireWire Vision Tools
A simple MATLAB interface to FireWire cameras
19
FireWire Vision Tools
A simple MATLAB interface to FireWire cameras
Appendix A – The MATLAB script yuv2rgb
% converts rgb data to yuv
(FW-04-03)
function dst = rgb2yuv(src)
% ensure this runs with rgb images as well as rgb triples
if(length(size(src)) > 2)
%
r
g
b
rgb image ([r] [g] [b])
= double(src(:,:,1));
= double(src(:,:,2));
= double(src(:,:,3));
elseif(length(src) == 3)
%
r
g
b
rgb triplet ([r, g, b])
= double(src(1));
= double(src(2));
= double(src(3));
else
% unknown input format
error('rgb2yuv: unknown input format');
end
%
y
u
v
convert...
= floor(0.3*r + 0.5881*g + 0.1118*b);
= floor(-0.15*r - 0.2941*g + 0.3882*b + 128);
= floor(0.35*r - 0.2941*g - 0.0559*b + 128);
% ensure valid range for uint8 values
y(y > 255) = 255;
y(y < 0)
= 0;
u(u > 255) = 255;
u(u < 0)
= 0;
v(v > 255) = 255;
v(v < 0)
= 0;
% generate output
if(length(size(src)) > 2)
% yuv image ([y] [u] [v])
dst(:,:,1) = uint8(y);
dst(:,:,2) = uint8(u);
dst(:,:,3) = uint8(v);
else
% yuv triplet ([y, u, v])
dst = uint8([y, u, v]);
end
20
FireWire Vision Tools
A simple MATLAB interface to FireWire cameras
Appendix B – The MATLAB script yuyv2rgb
% converts yuv data to rgb
(FW-04-03)
function dst = yuv2rgb(src)
% ensure this runs with yuv images as well as yuv triples
if(length(size(src)) > 2)
%
y
u
v
yuv image ([y] [u] [v])
= double(src(:,:,1));
= double(src(:,:,2));
= double(src(:,:,3));
elseif(length(src) == 3)
%
y
u
v
yuv triplet ([y, u, v])
= double(src(1));
= double(src(2));
= double(src(3));
else
% unknown input format
error('yuv2rgb: unknown input format');
end
% convert...
u = 2*u - 256;
v = 2*v - 256;
r = y + v;
g = floor(y - 0.51*v - 0.19*u);
b = y + u;
% ensure valid range for uint8 values
r(r > 255) = 255;
r(r < 0)
= 0;
g(g > 255) = 255;
g(g < 0)
= 0;
b(b > 255) = 255;
b(b < 0)
= 0;
% generate output
if(length(size(src)) > 2)
% rgb image ([r] [g] [b])
dst(:,:,1) = uint8(r);
dst(:,:,2) = uint8(g);
dst(:,:,3) = uint8(b);
else
% rgb triplet ([r, g, b])
dst = uint8([r, g, b]);
end
21
FireWire Vision Tools
A simple MATLAB interface to FireWire cameras
Appendix C – The camera configuration file
The camera configuration file (cameraconfig.txt) has the structure shown
below. The [Format] section outlines the meaning of the entries for each
camera. This structure is fixed and should not be modified (the driver files
simply ignore this section). The entries in sections [Camera_1] and
[Camera_2] define the parameters listed in the format section (in that order).
The default values have been found using the configuration panel of the demo
program that comes with the driver (1394CameraDemo). The values work in
our indoor environment, but any slight change in the lighting conditions may
require a re-adjustment.
[Format]
Shutter (6)
Saturation (90)
White Balance (96 69)
Gain (87)
Auto Exposure (511)
Brightness (304)
[Camera_1]
6
90
96 69
187
511
304
[Camera_2]
6
90
96 69
87
511
304
22
FireWire Vision Tools
A simple MATLAB interface to FireWire cameras
Appendix D – A test program for capProcs
% test program to continuously capture pictures from the camera
function
test(varargin)
% run variable is global -> stop button callback can reach this
global run;
run = 1;
% check inputs of function 'test'
if(isempty(varargin))
mode = 0;
% YUV411, 30 fps
fname = 'colors.txt';
% default filename
scan4col = [];
elseif(nargin == 1)
mode = varargin{1};
fname = 'colors.txt';
% default filename
scan4col = [];
elseif(nargin == 2)
mode = varargin{1};
fname = varargin{2};
scan4col = [];
else
mode = varargin{1};
fname = varargin{2};
scan4col = varargin{3};
end
if(run)
% open figure window and create stop button
figure
title('Click STOP to stop data acquisition...');
stop_h = uicontrol('Style', 'pushbutton', 'String', 'STOP',...
'Position', [20 200 50 50], 'Callback', 'zeroGlobalRun');
while(run)
[out, R] = capProc(mode, fname, scan4col);
image(R);
if(~isempty(out))
hold on;
% no regions detected
nCol = length(out);
% number of colours with valid regions
for(j = 1:nCol)
kk = out(j).nRegions;
col = out(j).Colour;
while(kk)
cx = out(j).Regions(1,kk);
% centroid
cy = out(j).Regions(2,kk);
x = out(j).Regions(3,kk);
% boundary box
y = out(j).Regions(4,kk);
w = out(j).Regions(5,kk);
h = out(j).Regions(6,kk);
patch([x x+w x+w x],[y y y+h y+h],col);
plot(cx,cy,'w*');
kk = kk - 1;
end
end
hold off
end
drawnow
end
% delete 'freeze' button
delete(stop_h);
end
23
FireWire Vision Tools
A simple MATLAB interface to FireWire cameras
Appendix E – Recompiling the drivers
The file structure of this toolbox includes a ‘development’ folder (_devlpmt).
Within this folder, each of the commands has its own subfolder.
To recompile a command change the current directory to the corresponding
subfolder, e.g.
>> cd _devlpmt/capProc
>> dir
.
..
R14mexopts.bat
Readme.txt
capProc.cpp
capProc.dll
cc.m
cmvision.cpp
cmvision.h
colors.txt
test.m
zeroGlobalRun.m
>>
Edit capProc.cpp to reflect the changes as desired, e.g.
/* camera settings ... */
theCamera.SetShutter(6);
theCamera.SetSaturation(90);
theCamera.SetWhiteBalance(96, 69);
theCamera.SetGain(87);
// don’t switch auto exposure to manual
//theCamera.SetAutoExposure(511);
theCamera.SetBrightness(304);
Save capProc.cpp and recompile this command by typing (at the MATLAB
prompt):
>> cc
Finally, copy the newly created DLL (capProc.dll) to the ‘_bin’ folder of the
toolbox. Subsequent calls to capProc() will now make use of the modified
driver.
24