Vision-based Control of an Industrial Robot Arm

Chapter 1
Introduction
Oneithersidetherestoodgoldand
silvermastiffswhichVulcan,with
hisconsummateskill,hadfashioned
expresslytokeepwatchoverthe
palaceofkingAlcinous;sotheywere
immortalandcouldnevergrowold.
Homer,Odyssey7.78-94
In this introductory chapter we will first see a brief historical excursus,
thatgivesusthebackgroundandthemotivationbehindtheprojectthisreport
isabout;itfollowsadescriptionoftheprojectitselfandanoverviewofthe
structureofthereport.
1.1 Background
Mankindhadstartedbeingfascinatedbyrobotics
1
thousandsofyearsago:
already in Greek mythology (precisely in Iliad and also in the successively
written Odyssey, both attributed to Homer) the god of fire and volcanoes
Hephaestus created golden mechanical servants [2] to protect the palace
of king Alcinous. Over the following centuries, an impressive number of
mechanicaldevicesthatcouldbeconsideredtheancestorsofmodernrobots
havebeenbuilt. Somenoteworthyexamplesofthesedevicesarethedigesting
duck
2
(figure1.1(a))orthetea-servingdoll(figure1.1(b)). Inthelastdecades,
technologygavetothehumankindthepossibilitytodramaticallyimprove
thesedevicesandsciencefictionalwriterswerespeculatingmoreandmoreon
howcomplexandfuturisticthesedevicescouldbecomeinthefuture,because
1
Thetermrobotderivesfromtheslavicwordrobota(forcedlabor),anditbecamepopular
after having been coined by the playwright Karel
ˇ
Capek in his play “R.U.R.” (acronym for
Rossum’sUniversalRobots),whichpremieredin1921[1].
2
AnautomatoncreatedbyJacquesdeVaucansonin1739consistinginamechanicalduck
capableofeatinggrainanddigestingit[3].
1
2 CHAPTER1. INTRODUCTION
(a) TheautomatonDigestingDuck (b) Amechanicaltea-servingdoll
Figure1.1: Famousautomata
somethingspreviouslyconsideredmythorfictionhadactuallyalreadycame
topass.
Ontheotherhand,modernrobotsareconsiderablydifferentfromwhatit
wasexpectedbythesewritersinthepast: mostrobotsareusedinindustries
for highly repetitive and burdensome tasks, while anthropomorphic and
domestic robots are mostly object of research and not something actually
commerciallyavailable(therearesomeremarkableexceptions,liketheseveral
hundredthousandsvacuumcleaningrobots-themostfamousbeing IROBOTS
Roomba,figure1.2-workinginthisexactmomentinresidentialhouses).
The success of industrial robots in factories is easy to understand: it is
not conceptually hard to design and build a robot much stronger than the
humanbody(intermsofmaximumweightliftable,velocityandresistance
tofatigue),andintheseenvironmentstoadaptthesurroundingstotherobot
isusuallypossible[4];nevertheless,actionswhichrequireabasicdegreeof
flexibility, awareness of the surrounding environment and dexterity (all of
these took for granted by humans) are devilishly complex for a robot and
thiscanexplainwhywedonothaveyetanthropomorphousrobotassistants
helpingusaround. Thiscanbejustifiedconsideringtheremarkablecomplex-
ityofthehumanbrainandthehumansenseswhichgrantsusthecapability
tosolveproblems,thinknewstrategiesandadapttothesurroundingsinan
1.2. RESEARCHPROBLEMDESCRIPTION 3
Figure1.2: IROBOTSRoomba
astonishinglyefficientway.
Tocreatearobotwhichisabletobeeffectiveinanunknownenvironment
(whichcannotbecontrivedatreasonablecoststofittherobot)andtoexecute
tasksrequiringahighdexterity,abigcomputationalpowerandhighlyper-
formingsensorsarebothneeded. Thistworequirementsarefulfilledmore
and more overtime: we can now buy for few cents an amount of computa-
tionalpowerthatcostedseveralthousandsdollarsbackintheseventiesanda
similarpatterncanbeidentifiedinmostthecomponentsarobotneedstowork
properly[5]. Thisevolutioninthecomponentsisgrantingusthepossibilityto
makecommerciallyconvenienttodesignandproduceontheonehandrobots
whichstillworkintheindustriesbutaremoreflexibleandlesstaskspecific
thantheirancestors,andontheotherhandrobotsthatcansafelyinteractwith
humansinanunstructuredenvironment.
Thesenseofvisionplaysanimportantroleinourcapabilityofinteracting
withourenvironmentandtoperformdexteroustasksandtheobjectofthis
projectistodesignandbuildavisionsystemallowingarobotarmtoachieve
someresultsinthisarea,i.e.,bouncinganunconstrainedball. Itisworthyto
highlighthowsuchavisionsystemrequiresacomputationalpowerthatwas
notwidelyavailableonlyacoupleofdecadesago.
1.2 Researchproblemdescription
Theobjectofthisprojectistodesignandbuildastereoscopicvisionsystem
capableoftrackingafast-movingobjectandpredictingitsfuturepositions,
4 CHAPTER1. INTRODUCTION
enabling a UR5 UNIVERSAL ROBOTS arm (figure 1.2) to interact with it. A
colour-basedobjectdetectionhasbeenusedandthepositionin3Dcoordinates
hasbeenestimatedwithtwodifferentstrategies,comparedinthisreport: one
isageometricaltriangulation,theotheroneisaneuralnetworktrainedfor
this purpose; a curve fitting algorithm predicted the future position of the
trackedobjectandapaddlemovedbytherobotarmisorientedandpositioned
according to a strategy meant to stabilize the bouncing. The robot motion
hasbeenperformedwithdifferentstrategies,oneoftheminvolvingtheneed
of the inverse kinematics of the arm, which was calculated using a neural
network.
1.3 Structureofthereport
Thisreportisstructuredasfollows: inChapter2somerelatedworkwillbe
summarized;Chapter3consistsinabriefreferencetosomenoteworthytheory
usedtocarryonthetestsandtowritethereportitself;Chapter4describesthe
equipmentused;inChapter5thereisadetaileddescriptionofthealgorithm
written to carry out the task (object tracking and prediction, motion of the
robot);Chapter6providesthereaderwithinformationabouttheprocedures
usedtocalibratethecamerasandtotraintheneuralnetworksusedtocalculate
theinversekinematicsandsolvethestereoscopicproblem,andpresentsalsoa
Figure1.3: UR5 UNIVERSALROBOTSArm
1.3. STRUCTUREOFTHEREPORT 5
comparisonbetweenthesestrategies;Chapter7presentstheresultsobtained
and the issues encountered; Chapter 8 summarizes the conclusion we had
came to after the work behind this project; some appendices, finally, are
included; they consist in additional results and theory that the interested
reader can make reference to if he wants more detail. A comprehensive
bibliographyisincludedalso,forthereader’sconvenience.
A CD is bundled with the present report; its content is described in ap-
pendixD.
Chapter 2
Related work
IfIhaveseenfurtheritisby
standingonyeshouldersofGiants
IsaacNewton
In the introductory chapter we have seen, among the other things, the
backgroundbehindthisproject,thatprovidesthemotivationforit. Butbefore
startinganewproject,itisalsoimportanttoexploretherelatedliterature,to
have an idea of the state of the art in the key parts of the project itself, that
mightinspiredifferentapproachestotheonesthatcouldhavebeenfollowed
without such a research. Since this project is made up by several parts, an
exploration of the related work for each part has been performed. In the
followingsectionstheresultsofthisstudyarereported.
2.1 RobotVision
One of the most important senses in a lot of different biological species
(especiallyinthemostevoluteones)isthevisionandthisistruealsoinmany
robots,thatuseitfordifferentreasons;thecapabilityofarobottocopewithan
unstructuredenvironmentcantakesignificantadvantageofavisionsystem:
visualinformationcanbeuseddirectlyusedtoperformclosed-loopposition
control of the robot [4] and usually for this purpose multiple cameras are
used.
Avisionsystemcanbedefinedasaremotesensorgatheringinformation
fromaportionoftheenvironmentinacontactlessfashion[6].
Thefirstchallengeistoextractmeaningfulinformationfromthepictures;
this process is called image segmentation and it consists in partitioning the
imageintomultiplesegments(e.g.,isolatingobjectsorboundaries). Thiscan
bedoneinmanydifferentwaysandamethodusedinseveralapplications
(andinthisproject,aswewillseein§5.1)iscoloursegmentation[7].
Noise and errors due to several factors (e.g., lens distortion) can lead
to unacceptable inaccuracy of position and orientation estimation; when a
7
8 CHAPTER2. RELATEDWORK
sequence of images is available, the accuracy can be enhanced using the
extended Kalman filter. Siciliano et al. [8] proposed an algorithm based on
Kalmanfilterforthepositionandorientationestimationinrealtimeofmoving
objectsusingmultiplecameras; oneofthedrawbacksofthismethodisthe
big computational power required, which could be a problem in real time
applicationsbutthisishardlyanissuewiththefastgrowingcapabilitiesof
microprocessors, which are significantly reducing the computational time
requiredtorunthealgorithm.
2.2 Neuralnetworksusedinrobotics
Neural networks are the artificial counterpart of their remarkable anal-
ogous in biology; since animals have noteworthy abilities in a very wide
rangeofdifferenttasks,ithasbeennaturalforresearcherstotrytousethese
networksinrobotics,forseveraldifferentpurposes.
Neuralnetworkscanbeafundamentalpartofthevisionsystemofarobot,
inmanydifferentways[9];e.g.,incomputerstereo-vision,aswewillseein
§3.2, the knowledge of some parameters relative to the cameras (described
in §6.1) is needed to perform the triangulation
1
; several methods exist to
computetheseparameters,oneofthembeingtheusageofaneuralnetwork
trained for this goal [10]; Ruichek and Postaire [11] proposed to solve the
‘correspondenceproblem’
2
usinganeuralnetwork;aneuralnetworkcanalso
beusedtosolvethestereoscopicproblem(seeDo[12]andXingetal.[13]),as
wewillseein§6.2.
To drive a robot in joint space, the knowledge of its inverse kinematics is
neededand,whentherobotisrelativelycomplex(evenarobotarmcanbe,if
ithasasufficientnumberofdegreesoffreedom),thisproblemcanbedifficult,
since it involves equations which are strongly non-linear and which often
donothaveanuniqueoranalyticalsolution[14];inthisprojecttosolvethis
problemwasalsonecessaryanditwasdecidedtofollowanapproachsimilar
to the one proposed by Tejomurtula and Kak [15], consisting in training a
feedforwardneuralnetworkforthispurpose.
Anothercrucialissueinroboticsisthetrajectorycontrol,andthisproblem
canbecomereallycomplexfornon-trivialrobots,sinceitinvolvestheknowl-
edgeoftheinverse-dynamicsmodelofthearm, andcontrolstrategiesthat
canbetrickytodevelop;Miyamotoetal.[16]proposedforthispurposethe
usageofafeedbackneuralnetwork,andtheresultwasthat,oncethenetwork
learnedhowtocontrolsomesimpleandslowmovements,itwasthenableto
generalizethisknowledgeandperformmorecomplexandfastmotions.
1
Theoperationconsistinginconvertingstereo-pairimagesfromthetwocamerasin3D
worldcoordinates.
2
Thisisaproblempresentinstereoscopicvision,itconsistsinmatchingfeaturesextracted
fromtwoimagesthatareprojectionsofthesameentityinthe3Dworld.
2.3. ROBOTSUSEDFORDEXTEROUSTASKS 9
2.3 Robotsusedfordexteroustasks
The idea of performing dexterous tasks (e.g., play table tennis) with a
robot has fascinated researchers all over the world, as a challenge both in
machinevisionandcontrolfields[17]. Thesetasksoftenrequirearobustand
high-speedvisionsystemandfastandprecisecontrolofrobotarm(e.g.,Li
proposedahighspeedvisionsystemconsistingintwocamerasworkingat
highframeratetrackingaballandpredictingitsfuturepositions[18];Acosta
etal.proposedsimilarsystem,aping-pongrobotplayer,thatusesinsteada
25Hzpairofcameras[19]).
Atypicalexampleofsuchadexteroustaskisthesystemconsistingina
bouncingballoveravibratingpaddle;itisaverysimplesystemwithavery
complexdynamicalbehaviorandforthisreasonithasbeenstudiedbymany
authors(e.g.,Holmes[20]andTufillaroetal.[21]). Severalapproacheshave
beenusedovertimetobounceanunconstrainedball: e.g.,someauthors[22]
usedamemory-basedlearningapproachtojugglewiththeball;others[23]
usedablindjugglertobounceanunconstrainedball(theycouldavoidusing
camerasorothersensorsbyshapingthepaddleconveniently,preciselythey
triedtominimizethe H
2
norm
3
bygivingtothevibratingsurfaceacurvature
theycalculateintheirreport).
Amethodsimilartotheonedescribedinthepreviouslineswillbeusedin
thisproject,butwithimportantdifferences,biggestofthembeingtheusageof
cameras. Thesameauthorscreatedlateranimprovedversionoftheirjuggler
(figure2.1),abletojuggletheballbackandforthalongahorizontaldistance
ofabout1mandaverticalapexof1.1m[25].
2.4 Summary
Inthischapterwesawsomeworkrelatedwiththisproject,whichinspired
the author in exploring new possibilities in the research for the solution of
problemsencounteredoveritsdevelopment.
Inthenextchapterinstead,sometheoreticalbasicsessentialforthisproject
willbepointedout,forabetterunderstandingofwhatcomesafterwards.
3
Thestandard H
2
normofasystemcanbeconsideredastheRMSresponseoftheoutput
signalwhentheinputisaunitvariancewhitenoise[24].
10 CHAPTER2. RELATEDWORK
Figure2.1: Thependulumjuggler[25]