Commanding Grasping Robot through Virtual Reality and Simulated Wrenches

Chapter 1
Introduction
Literature proposes a multitude of studies involving the use of robots for teleoper-
ation in industrial and medical applications. Recent studies propose the manipulation
of objects through multi-agent interaction on the same object through miniaturized tac-
tile devices [1, 2]. These solutions rise to technological development aimed at providing
technical and scientic advancement in the eld of remote manipulation [3]. Despite
the numerous recent developments, the robot teleoperation still requires a direct vision
of the environment, through cameras, in which the robotic part provides occlusion and
operational problems (e.g time delay, high cognitive burden to control the robot [53])
which aect many applications in both medical and industrial elds [4,5].
In this study is proposed a dierent strategy by introducing an innovative experimental
setting aimed to overcome the cited problems.
More in detail, a new experimental system is proposed to create a more suitable virtual-
to-real environment interconnection in order to eciently support the communication of
the data from the real part, where the robot agent operates, to the virtual one monitored
by the user. The real environment is approximated with a completely virtual one. In
this hypothesis, several physical models can be manipulated by a robotic system with
which they can interact through preset mathematical models.
However, the virtual counterpart, replicates the information necessary for the user to view
and control the physical parts and the robot. The operator himself interacts towards the
real environment through the information provided from the virtual environment where
the robotic part is omitted by design, in order to minimize the occlusion and visual
issues.
1.1 The Development of the Project
Virtual reality provides the possibility to receive a visual feedback and allows the
user to train with the surrounding environment.
However, challenges are still present due to the dierence between the poor quality and
quantity of haptic sensation obtained through virtual interaction and the full sensation
obtained through physical interaction [11].
The human can perceive in the physical world more properties such as stiness, rough-
ness, temperature of the objects, or deformation of objects.
Discrepancies regarding the perception of the physics could lead to wrong decisions from
the user side.
To grasp virtual objects is considered one of the most complex tasks in Virtual Reality
1
since many parameters such as object geometry, hand kinematics, collider shape, mate-
rial and physic properties have to be considered.
Realizing a realistic virtual grasp is a complex problem since the virtual environment
cannot physically constrain real hand motion [12].
The virtual environment, created with Unity’s software, consists of objects to grasp, the
avatar of the user’s hand, and a sphere that encloses the object touching the outermost
points of its mesh. The sphere is considered the shell of the object: the reference system
of the center of mass of the object and of the sphere coincide. Also, the sphere has the
same mass as the real object to grasp.
User’s hand is tracked through motion capture softwares and hardware systems. The rst
tracking system includes a grounded setup with cameras using the software Optitrack
Motive. The second methodology concerns a portable system that includes CyberGlove,
Leap Motion Controller and IMU.
The user establishes contact forces with the sphere which will be perceived through hap-
tic devices that provide a feedback of the generalized forces applied.
Through the grasping theory, the wrench at the center of mass of the sphere is deter-
mined and it is assumed to be equal to the object’s wrench. Known the wrench at the
center of mass of the object, through the virtual reality information on its motion, it is
possible to replicate it in real world through any robotic arm that has already grabbed
the object.
To corroborate this idea, it has been developed a hardware and software network for the
interaction between virtual reality and robot grasping in the real world. To resume, in
virtual reality, the goal is to study an algorithm aimed at simplifying grasping through
the use of the sphere.
As last analysis, the experimental evaluations concerns the grasping of light and small
objects, with homogeneous density, in the real world by applying specic forces to the
points of contact with the end-eector.
1.2 Overview
The following study is organized into ve chapters.
In Chapter 2 it is reviewed the state-of-the-art related to teleoperation interfaces in
human-robot cooperations, haptic feedback technologies and virtual reality applications.
The experimental setup and the system overview are presented in Chapter 3 with par-
ticular references at the creation of the virtual scenario, dierent tracking systems of
the user’ s hand and the teleoperation of the robot with virtual reality interface. Then,
experimental evaluations, in terms of grasping light and small objects and manipulation,
will be developed at the end of Chapter 3.
Finally, a conclusion and brief discussion with proposed further improvements are pre-
sented in Chapter 4. The most important scripts can be viewed in the Appendix.
2
Chapter 2
Related Works
In this chapter it is reported the state-of-the-art related to human-robot cooperation
exploiting teleoperation interfaces, haptic feedback technologies and virtual reality tech-
niques.
2.1 Teleoperation
During the last years, Physical Human-Robot Interaction (pHRI) has gained lot
of popularity due to the importance of combining the high adaptability of the human
partners with the endurance and accuracy of the robots in order to accomplish common
human tasks.
Teleoperation remains a dominant control paradigm for human-robot collaboration in
unknown or unstructured environments to perform a cooperative task [14]. It is the
capability to control remotely a robotic manipulator and, according to [7], an extension
of a person’s sensing.
A telerobot is a robot, hooked up to a haptic interface, controlled remotely by a human,
regardless of the degree of autonomy of the robot. Telerobotic systems are divided in
two parts [17]: the local site with the human and elements to support the system’s
connection with the robotic manipulator, and the remote site, which contains the robot
and sensors to transmit back the information. The human operator (master) controls the
motion and/or the forces of the robot (slave). Time delay, accuracy of the actuators are
parameters that must be taken into account when the purpose is to realize a real-time
communication in the master-slave system.
The telerobot requires commands from the human operator (Figure 2). The control
architectures (Figure 1) can be splitted in three main classes:
 Direct Control: the motion of the slave robot is directly controlled by the user
with the master interface. The robotic system has no autonomy;
 Shared Control: it is based on sensory feedback at the remote site, by which com-
mands were rened autonomously providing the telerobot with a modest sensory
intelligence. The human operator provides path commands by using, for example,
a kinesthetic feedback device. The overall task is divided between the human and
the telerobot;
 Supervisory Control: intelligence is distributed between man and machine. Part
of the task can be implemented directly on the robotic manipulator independently
3
while high level task planning must be decided by the human. The telerobot is
locally autonomous.
Fig. 1: Control architectures in tele-
operation
Fig. 2: Telerobotics
To control remotely the robot, several methodologies have been applied such as hand
gesture recognition based either on visual detection or internal sensor detection. The
latter is more reliable since visual detection system suers from many drawbacks such
as sensitivity to light, changing distance, hand motion modeling complexity, and posi-
tion [26, 27]. Some common human-robot interfaces are joysticks, keyboards, wireless
mouse [55] and arm, inertial sensors. An example is the study [54] where a joystick
controller transmits the corresponding velocity and orientation command to the mobile
robot. Nevertheless, there is a disadvantage in the use of these technologies as they
require an intense training phase by the human operator [28]. Another technique has
been explored in the study [29] where it has been highlighted problems regarding the
control of a high-dimensional system, such as the Kinova JACO Arm (Kinova, CA), with
a low-dimensional input such as a joystick. Switching between multiple control modes
leads to 17.4% of time loss and cognitive load.
In a teleoperated system, investigations have been made on how the visualization of
data is extremely important to evaluate the performance of the system. In literature,
there many examples where robot have been employed as telerobots. Important areas of
application concerning telerobotics are surgery, space exploration, military, telepresence,
underwater maintenance, disposal of radioactive substances, cleaning of hostile environ-
ments, remote inspection, rescue lives in hazardous environments [30]. Among these, the
following elds have achieved important results:
 Surgery: In 1985, the rst surgical robot was developed. It was a robotic arm
called Puma 560 that was used for neurosurgical biopsies and the transurethral
resection of the prostate [57] .
The surgical telerobots, e.g da Vinci robot (Intuitive Surgical, U.S), were made
for accuracy and delicacy for operations on human bodies. The da Vinci robotic
system is capable to reproduce a stereo laparoscopic video to the robot’s console
window during surgery. This allows the surgeon to operate on a patient remotely
using a camera and robotic arms to manipulate tools with more steady robot’s
hands [19]. Progresses have been made including sensors technology, communica-
tions optimization, input control devices, augmented reality and visual feedback.
Researches have involved both computer interfaces and robotics either to reduce fa-
tigue, restore hand-eye coordination and improve dexterity of human surgeons [56].
There are still investigations concerning force feedback and haptic systems. In Fig-
ure 3 it is shown the da Vinci robot;
4
Military: improvements are noticeable in eld of target acquisition and recon-
naissance. Teleoperated robots with thermal imaging technology allowed faster
detection of people [22]. Another application is the use of the Kraft force feedback
manipulator arms to clear unexploded bombs;
 Space exploration: Robonaut2 (Dextrous Robotics Laboratory, NASA (Texas))
in Figure 4 is an application of teleoperation in space with the purpose to improve
the performance of astronaut’s tasks that implied risks such as cosmic radiations
[23]. Space explorations include Free-ying telerobots which, due to their easy
mobility, are most suitable for a wide range of mobile sensor tasks [58]. Smart
SPHERES is a free-ying robot that performs intravehicular activities such as
interior environmental surveys [59];
 Industrial eld : typical applications of teleoperation tasks are repair operations,
security infrastructure, remote power generation, transmission applications, remote
platforms over low-bandwidth communications links [24, 25]. One of the most
common industrial telerobot is KUKA (KUKA, Germany), a 6-degree-of-freedom
robotic arm;
 Telepresence: the user perceives an association with the remote task since there
is an ideal sensing of sucient information such that the human operator is phys-
ically and personally embedded within the remote site [34]. The immersion in
the environment may provide more eective performance through the control in-
terface. Telepresence approach has covered a great role especially in the eld of
social robotics such as assisting older people. For example, the PARO robot (AIST,
Japan) is designed for therapeutic interventions with older adults [60].
Fig. 3: da Vinci Robot
Fig. 4: Robonaut2, advance remote
robotic control and task development
for future exploration missions
2.2 Haptic Feedback
An important element involved in the teleoperation is the haptic feedback. Hap-
tics played an important role in teleoperation from the early 1950s [21]. During the
decades, the progress of haptic feedback systems has increased, involving manipulators
up to impressive desktop devices, used to support real-time interaction with 3D visual
simulations, or Virtual Reality [35].
According to [31,32], the function of haptic interface is to replicate the touch experience
concerning the perception of an environment using mechatronic devices that are able to
exert controllable forces on the user with one or more degrees of freedom. It generally
5