Thesis_Roel_Kuiper.

Thesis_Roel_Kuiper.
GRABBING ROCK IN DEEP SEA MINING
THE EFFECT OF OFFERING NATURAL FORCE FEEDBACK
AND HAPTIC SHARED CONTROL ON RATE AND FORCE
CONTROL OF A DEEP SEA MINING SUSPENDED GRAB
R.J. Kuiper
Delft, 15th of February 2012
SECTION:
SPECIALIZATION:
STUDENT NO.:
MECHANICAL ENGINEERING
BIOMECHANICAL DESIGN
1175408
Prof.dr. F.C.T. van der Helm (ME)
Dr.ir. D.A. Abbink (ME)
Ir. J.C.L. Frumau
Ir. S. Tamsma
GRABBING ROCK IN DEEP SEA MINING
Copyright © 2012 by Seatools bv.
All rights reserved. Complying with all applicable copyright laws is the responsibility of the user. No part of
this document may be reproduced, stored in a retrieval system, or transmitted in any form or by any means,
electronic, mechanical, photocopying, recording, or otherwise, without express written permission of
Seatools bv.
If, however, your only means of access is electronic, permission to print one copy is hereby granted.
In no event will Seatools bv. or its employees be liable for any consequential, incidental or indirect damages
(including damages for loss of business profits or business information, business interruption, and the like),
arising out the use of or inability to use this document, even if Seatools bv. has been advised of the
possibility of such damages.
Information in this document is subject to change without notice.
15-02-2012
2/71
PREFACE
PREFACE
I have accomplished a double master degree at the Delft University of Technology, one for Mechanical
Engineering and one for Offshore and Dredging Engineering. This document describes the thesis part for my
Mechanical Engineering master with the specialisation Biomechanical Design, which focused on the design
and evaluation of different feedback mechanisms to control a suspended grab for excavating rock in a deep
sea mining process. This document is written according to the agreement between Seatools bv. and the Delft
University of Technology. The confidential information is removed from this document and this version is
only for publicly access in the digital archives of Delft University of Technology. More detailed information of
the experimental results and design details is accessible by contacting Seatools bv.
This document is written with the support of David Abbink, Jan Frumau and Frans van der Helm through
several discussions guiding the thesis and constructing feedback of the draft report. Great support for
conducting the experiments is given to me from Sieds Tamsma and David Abbink. Both the TU Delft and
Seatools bv. greatly supported the research with materials and knowledge.
3/71
R.J. Kuiper
Biomechanical part of thesis
GRABBING ROCK IN DEEP SEA MINING
15-02-2012
4/71
ABSTRACT
ABSTRACT
High concentrations of common and rare-earth metals are found on the seafloor in rock material named
seafloor massive sulphides deposits, at depths up to 2000 meters. A proposition for excavating these
deposits at such great depths is to use a large suspended grab, controlled by teleoperation. Such a mining
operation has as goal to excavate the maximum amount of minerals at the minimal amount of time.
Environmental uncertainties prohibit the use of full automation, inducing the need of an operator controlling
the process by teleoperation. General problems that are expected to occur with teleoperation of a large
machine on great depths are operator’s performance and situation awareness, which can reduce operational
efficiency and might even lead to failure when not undertaking appropriate responses in critical situations.
Therefore optimization of the grasping teleoperation is needed to increase the entire excavation performance
and decrease periods of operational downtime due to incorrect execution of the operator.
Two approaches from literature seem promising for improving the human-machine interface of remote
grasping: ’natural force feedback’ (i.e., reflecting system forces to the operator during grasping, creating
transparency in the teleoperation), or ‘haptic shared control’ (i.e., sharing the control through forces on the
input device, therefore guiding the operator based on constraints in the environment). Natural force
feedback has been shown in literature to increase situation awareness and performance of a teleoperation by
reflecting information of the environment to the operator. Forces of the hydraulic actuators for excavating
rock material with a grab are used for natural feedback to the operator. Haptic shared control is shown to
increase task performance and decrease control effort of a teleoperation with position control. Artificial
forces are used to guide the operator to control the process efficiently and prevent incorrect responses in
critical situations. Combining natural force feedback and haptic shared control could be an optimal solution to
increase grasping performance on a deep sea mining teleoperation.
The goal of this thesis is to determine the improvements for the operator using these two feedback
mechanisms for controlling a deep sea mining grabbing process. An experimental setup has been developed
to validate the hypotheses of improved performance and situation awareness using haptic feedback. The
experimental setup consists of two joysticks for controlling the velocity and force of the hoisting cable and
clamshells of the simulated process, based on a comparable dredging application. A visual projection of the
process is displayed to the operator, with additional visual information such as position and forces of the
hoisting cable and clamshells, for optimal control of the process. Both haptic feedback mechanisms are
applied separately and combined as experimental conditions during the experiment and compared with a
baseline condition without haptic feedback. The experiments are conducted using two types of trials:
‘performance trials’ (i.e., trials conducted at normal operational conditions) and ‘catch trials’ (i.e., trials at
difficult conditions which cannot always be fully completed), to determine improvements in excavation
production, control effort and responses in critical situations.
The experimental results for performance trials showed no improvements in performance, neither offering
natural force feedback as haptic shared control, based on excavation production results. Haptic shared
control however did show a reduction of control effort during performance trials, based on total summation
angles of control inputs and subjective workload. When offering natural force feedback, reduction of control
effort only occurred when it was combined with haptic shared control. Natural force feedback indicated an
increase of situation awareness based on reduction of the maximal heeling angle of the machine at ground
contact, preventing the machine to fall over during catch trials. When offering haptic shared control, a
reduction of heeling angle only occurred when also offering natural force feedback. This implies that natural
force feedback improved the situation awareness at critical situations, resulting in less production loss and
damaging of the system. Haptic shared control on the other hand reduced the control effort, resulting in an
increased attention of the operator during the entire process, due to a less demanding task.
5/71
R.J. Kuiper
Biomechanical part of thesis
GRABBING ROCK IN DEEP SEA MINING
CONTENTS
PREFACE __________________________________________________________________ 3
ABSTRACT _________________________________________________________________ 5
CONTENTS _________________________________________________________________ 6
1
INTRODUCTION _________________________________________________________ 9
1.1
DEEP SEA MINING PROCESS DESCRIPTION______________________________________ 9
1.1.1 SITUATION OVERVIEW ____________________________________________________________ 9
1.1.2 CONTROLLING A GRAB ___________________________________________________________ 10
1.1.3 PRINCIPLE HYDRAULIC ACTUATORS __________________________________________________ 11
1.1.4 TASK DECOMPOSITION ___________________________________________________________ 12
1.2
TELE-OPERATION _____________________________________________________ 14
1.2.1 NATURAL FORCE FEEDBACK ________________________________________________________ 15
1.2.2 HAPTIC SHARED CONTROL ________________________________________________________ 17
1.3
GRASPING WITH TELE-OPERATION __________________________________________ 20
1.3.1 GRASPING WITH NATURAL FORCE FEEDBACK ____________________________________________ 21
1.3.2 GRASPING WITH HAPTIC SHARED CONTROL ____________________________________________ 21
1.4
PROBLEM STATEMENT AND GOAL ___________________________________________ 22
1.5
HYPOTHESES ________________________________________________________ 22
1.6
APPROACH __________________________________________________________ 23
2
EXPERIMENT METHOD ___________________________________________________ 25
2.1
PROCESS DESCRIPTION _________________________________________________ 25
2.1.1 SUBJECTS ____________________________________________________________________ 25
2.1.2 EXPERIMENTAL SETUP ___________________________________________________________ 26
2.1.3 TASK DESCRIPTION _____________________________________________________________ 26
2.2
EXPERIMENT DESCRIPTION _______________________________________________ 27
2.2.1 EXPERIMENTAL CONDITIONS _______________________________________________________ 28
2.2.2 MEASURED VARIABLES ___________________________________________________________ 29
2.2.3 DATA ANALYSIS AND STATISTICS ____________________________________________________ 30
3
EXPERIMENTAL SETUP DESIGN ____________________________________________ 31
3.1
MECHANICAL DESIGN___________________________________________________ 31
3.2
COMPONENTS ________________________________________________________ 33
3.2.1 MECHANICAL __________________________________________________________________ 33
3.2.2 ELECTRICAL___________________________________________________________________ 33
15-02-2012
6/71
CONTENTS
3.3
SOFTWARE DESIGN ____________________________________________________ 34
3.3.1 GENERAL OVERVIEW ____________________________________________________________ 34
3.3.2 VISUAL INTERFACE ______________________________________________________________ 35
3.3.3 REAL-TIME CONTROLLER __________________________________________________________ 36
3.3.4 VIRTUAL GRAB SIMULATION _______________________________________________________ 38
3.3.5 CONTROL PANEL _______________________________________________________________ 42
4
EXPERIMENT RESULTS ___________________________________________________ 45
4.1
GENERAL RESULTS _____________________________________________________ 45
4.1.1 TIME TRACE RESULTS____________________________________________________________ 46
4.1.2 SELF-ASSESSMENT RESULTS _______________________________________________________ 48
4.2
PERFORMANCE TRIALS __________________________________________________ 50
4.2.1 PERFORMANCE AND SYSTEM RESULTS _________________________________________________ 50
4.2.2 CONTROL EFFORT RESULTS ________________________________________________________ 51
4.3
CATCH TRIALS _______________________________________________________ 54
4.3.1 CATCH TRIAL OF CRITICAL SITUATION ________________________________________________ 54
4.3.2 CATCH TRIAL OF HARD SOIL _______________________________________________________ 57
4.3.3 CATCH TRIAL OF REACTION TIME ___________________________________________________ 59
5
DISCUSSION __________________________________________________________ 61
6
CONCLUSION __________________________________________________________ 63
7
RECOMMENDATIONS ____________________________________________________ 65
ABBREVIATIONS ___________________________________________________________ 67
NOMENCLATURE ___________________________________________________________ 67
REFERENCES ______________________________________________________________ 69
7/71
R.J. Kuiper
Biomechanical part of thesis
GRABBING ROCK IN DEEP SEA MINING
15-02-2012
8/71
INTRODUCTION
1 INTRODUCTION
New technologies such as mobile phones and laptops are rapidly increasing the demand for rare-earth
metals such as tellurium, selenium and rubidium. Also the prizes are rising for common metals such as gold,
silver and copper. This induces the need to look for new material resources. A solution for new material
resources is found by collecting deposits on the seafloor at depths of 2000m, through a process called deep
sea mining. Excavating these deposits from the seafloor is a challenging task, due to large depth and high
costs of such a process resulting in the need for a large production rate. A promising option for excavating
mineral deposits is the use of a suspended grab. Full autonomous operation is not feasible due to several
factors, such as unknown reactions of rock sediments in deep waters and positioning uncertainties. The
human operator therefore needs to be included in the process, which requires an intuitive human-machine
interface in order to control the excavation process by teleoperation. A description of a deep sea mining
process with a suspended grab for excavating rock materials is given in paragraph 1.1. The general principle
of teleoperation and two haptic feedback mechanisms as found in literature is described in paragraph 1.2.
The difference of a grasping teleoperation as used for controlling a suspended grab is described in paragraph
1.3. Paragraph 1.4 describes the problem statement for this thesis, paragraph 1.5 the hypothesis and
paragraph 1.6 the approach of this thesis to verify the hypothesis.
1.1 Deep Sea Mining Process Description
An overview of a deep sea mining process using a suspended grab is given in paragraph 1.1.1 and the main
control parameters in paragraph 1.1.2. Paragraph 1.1.3 gives a brief overview of the principle of a hydraulic
actuator and the transition of force and velocity of such a system. A decomposition of an entire deep sea
mining task is given in paragraph 1.1.4.
1.1.1 Situation Overview
Excavation of rock material from the seafloor for a deep sea mining process is a large technical difficulty due
to the size and forces of the process. The production has to be large scale due to the offshore location and
therefore costs of such a process. Due to this scale, the size of machinery and encountered forces are large
and difficult to operate. A promising way for excavation is the use of a large hydraulic grab, suspended from
a ship with a hoisting cable as shown in Figure 1. The grab can translate in horizontal direction with the use
of thrusters, full autonomous in normal conditions. These thruster forces can be neglected when the grab
has ground contact, due to the weight of the grab and forces of the closing clamshells. A deep sea mining
process as shown in Figure 1 has not yet been performed. A comparable operation with a suspended grab
has been applied for a dredging operation at a water depth of about 120m (Es 2004 [10]).
~ 2000 m
Figure 1: Situation Overview of a deep sea mining process
9/71
R.J. Kuiper
Biomechanical part of thesis
GRABBING ROCK IN DEEP SEA MINING
In Figure 2 are the degrees of freedom shown of a suspended grab with hydraulic actuated clamshells, which
can move vertical by changing the length of the suspended cable and horizontal by actuating the thrusters
on the grab. The heading of the grab can also be controlled by actuating the thrusters, however the heeling
and pitch angle cannot be controlled, but are stable when free hanging on the hoisting cable due to the low
center of gravity in comparison with the suspension point. The clamshell angles are both actuated with
hydraulic cylinders and are individually controlled. Almost all degrees of freedom are in local coordinates,
except for the vertical position and the heeling and pitch angle due to the hoisting cable force.
φ
ψ
θ
x
z
y
α
Figure 2: Degrees of freedom of a suspended grab, shown in coordinates of control
Degrees of freedom:
 Heeling angle φ
 Pitch angle θ
 Heading angle ψ
 Sideways translation x
 Forward translation y
 Vertical translation z
 Clamshell angle α
1.1.2 Controlling a Grab
Controlling a suspended grab during teleoperation is done with the use of joysticks; controlling both the
grabbing force from the clamshells, as the cable hoisting force from the winch as shown in Figure 3 a) and
b). The winch drum is located on the deck of the ship and has a large delay for controlling the cable hoisting
force due to the size of the drum and length of the cable.
Winch
drum
FW
FG
FW
Frame
Clamshell
Winch
joystick
Grab
joystick
FG
Figure 3: a) Schematic view of control parameters of actuators on slave. b) Schematic view of control
parameters of joysticks on master
15-02-2012
10/71
INTRODUCTION
Controlling the cable force is used to lower or uplift the grab from the seabed and to prevent the grab to
tumble over during ground contact. The closing force of both clamshells is controlled with the use of one
joystick, because the grabbing force can only occur between both clamshells. Both joysticks for controlling
the cable and grab forces will eventually be combined in a single joystick with two functions, rotating and
grasping. Separation of both joysticks is chosen for this thesis to simplify the mechanical setup. The overview
of the control system of the human operating the grab by tele-operation is shown in Figure 4. The red
arrows indicate the feedback of the joystick on the human hand, force, velocity and position of the joystick
angle. The blue lines from the controller indicate the current on the electric motors, actuating haptic
feedback to the human. The green dotted line represents the visualisation feedback to the human operator,
displaying the remotely controlled operation. The environment acts on the grab behaviour and back, the
controller has different settings, from the control panel for different conditions. The human operator gets
instructions to control the process.
Instructions
Left
Hand
Winch
Joystick
Be
Human
Operator
Right
Hand
Gab
Joystick
Hydraulic
Valve
Cable
Length
Environment
Behavior
Grab
Hydraulic
Valve
Clamshell
Angle
Settings
Controller
Eyes
Sensors
Visualization
Hardware Master
Subject
Virtual Slave
Figure 4: System control overview for tele-operating a remotely controlled grab. Red lines indicate feedback on
human hand, blue lines current on electric motors, green lines visual feedback and orange lines system inputs.
1.1.3 Principle Hydraulic Actuators
A deep sea mining excavation process with a suspended grab consists of two main components, the winch
drum and clamshells as described in paragraph 1.1.1 and shown in Figure 3 a). These two components
control the two main parameters, cable force and cutting force, with hydraulic actuators. The winch drum is
actuated with hydraulic motors and the clamshells with hydraulic cylinders. Both hydraulic actuators are
based on the same principle of compressed hydraulic fluid forcing the actuator to move with a controlled
velocity and force. The amount of force and velocity of the actuator is controlled with a valve, restricting the
incoming flow as shown in Figure 5.
Fvol, Uvol
Avalve
Qpump
Qvol
Vvol, Avol
Apipe
Figure 5: Schematic view of principle of hydraulic volume with controlled inflow
11/71
R.J. Kuiper
Biomechanical part of thesis
GRABBING ROCK IN DEEP SEA MINING
The flow through the valve as shown in Figure 5 is defined as a ratio of area as show in Eq. 3.
Eq. 3
The velocity of the volume level is defined as flow Q over area A as shown in Eq. 2.
Eq. 2
The calculated velocity in Eq. 2 is based on a non-loading condition of the hydraulic actuator. A constant
loading condition is given in Figure 6, describing the principle of compression of a fluid for changing volume
at applied pressure.
P
V 0 , P0
P
P
P
P
P
Figure 6: Principle of uniform compressed hydraulic volume.
Elasticity of a fluid as shown in Figure 6 is expressed with the bulk modulus, the reciprocal of compressibility,
relating specific volume and differential pressure (George and Barber 2007 [11]). The definition of bulk
modulus is given in Eq. 1 (Totten 1999 [36]).
Eq. 1
⁄
Reversing Eq. 1 gives an equation for the pressure of the fluid, shown in Eq. 5. The volume of the fluid itself
is defined as the integral of the inflow of the fluid.
Eq. 5
∫
The force of the hydraulic volume as shown in Figure 5 is defined as pressure over area, as shown in Eq. 4.
(
)
Eq. 4
Sub conclusion
Eq. 2 and Eq. 4 give the definitions of velocity and force related to the control valve area as shown in Figure
5. The combination of both equations therefore gives for controlling the inflow a transition of controlling the
velocity to rate of loading, when the hydraulic actuator changes in loading condition. This results in a velocity
controlled task for the operator when no contact is made and converts into a force controlled task during
contact. This means a velocity task for controlling the winch drum when the grab is free hanging and
controlling the cable force when the grab has contact with the seabed. For controlling the clamshells, the
velocity is directly controlled after breaking rock material and force controlled before the breaking point of
rock material.
1.1.4 Task Decomposition
A deep sea mining process consists of multiple tasks to be conducted, based on various fields of expertise.
Evidently the entire process starts with exploration of the seafloor determining the location of the minerals.
15-02-2012
12/71
INTRODUCTION
After exploration an operating area is chosen and a ship is positioned for optimal production. Subsequently
the following process steps are required for excavation of mineral deposits using a suspended hydraulic grab.
 Lower grab from ship to deep water.
 Planning of optimal excavation locations.
 Horizontal position grab on excavation location, while rejecting disturbances.
 Lower grab for contact with seafloor, using maximal impact for initial rock breaking.
 Closing the clamshells and controlling the grab force during cutting of the material
and stabilizing the grab using cable forces.
 Uplift the grab from the seafloor.
 Translate the grab to a central collection point on the seafloor.
 Discharge the rock material at the collection point.
 Vertical transport of the material to the ship.
 Process the material on deck of the ship.
 Transport the material using bulk carriers.
In the enclosed blue area are the tasks listed that are the focus of this research, controlled by tele-operation.
These tele-operations could be improved by offering natural force feedback to inform the operator of the
environmental forces and haptic shared control with attractive guiding forces to assist the operator. In Figure
3 b) are the two joysticks shown to control this teleoperation, a grab joystick for closing the clamshells and a
winch joystick for controlling the cable forces. The natural feedback forces to be offered on the grabbing
joystick can be based on the cutting forces from the hydraulic cylinders, for the winch joystick the cable
forces of the winch drum can be used. Guiding forces on the winch joystick can be used to assist the
operator for controlling the correct cable force. The cable force has to be high enough for tension in cable to
avoid tumbling over. The cable force also has to be not too high to remain vertical forces of from the weight
of the grab on seabed for cutting the rock material. This required vertical force reduces when the clamshells
are more closed also the required cable force for stable position of the grab has to increase when closing the
grab. Figure 7 shows the change in degrees of freedom during transition of free hanging and ground
contact. During free hanging the thrusters can translate the grab in the horizontal plane and change the
heading angle. However during ground contact the thuster forces are limited compared to the contact and
cutting forces, approximately 5% as calculated in part 1 of the thesis. Therefore during contact the cable and
cutting forces dominate the behaviour of the grab and when incorrect controlled they can cause a large
heeling angle of the grab, possibly damaging the machine. When the ground reaction forces are uneven, the
cable force actuated with the winch has to stabilize the grab. However the cable forces have to be as low as
possible to increase the vertical component of the cutting forces to break the rock material, without causing
slack in the cable resulting in late response of the cable forces. Guiding forces on the control inputs could
therefore help the operator to correctly control the winch. When the clamshells are almost closed the cable
forces have to be increased to prevent instability of the grab, which can be informed with guiding forces to
the operator as well. Natural force feedback gives the operator information of the acting forces and improves
the awareness of the operator of the current situation, hence improving the correct control.
13/71
R.J. Kuiper
Biomechanical part of thesis
GRABBING ROCK IN DEEP SEA MINING
ψ
x
Free hanging
y
z
φ
z
Ground contact
α
Figure 7: Change of grab movement during contact transition
1.2 Tele-operation
A deep sea mining operation consists of a suspended grab hanging from a ship as shown in Figure 1. Rock is
excavated from the seabed, by controlling the position and cutting force of the grab remotely from the ship,
using teleoperation. Teleoperation is remotely sensing and manipulating a device through means of
communication using artificial sensors and actuators (Sheridan 1989 [33]). A high level of tele-presence
describes the operation when a tele-operator feels physical present at the remote location, using ideal
communication of the environment (Sheridan 1989 [33]). Difficulties in creating a high level of tele-presence
are time delays, accuracy and limiting sensory feedback mechanism. These time-delays and low accuracies
can even cause instabilities in controlling a teleoperation (Hannaford 1989 [14], Niemeyer and Slotine 1991
[26], Sayers et al. 1998 [31]). A method for increasing the level of tele-presence is the use of augmented
reality, the addition of a virtual environment displayed to the operator based on sensory information
(Milgram et al. 1995 [24]). In Figure 8 is a master-slave system shown, where the operator’s movements on
the input device (master) are mimicked by the remote controlled device (slave), combined called telemanipulator (Sheridan 1989 [33]).
master
slave
Figure 8: Schematic view of a teleoperation with visual feedback (adapted from Sheridan 1989 [33], pp. 2)
15-02-2012
14/71
INTRODUCTION
Teleoperation using only visual feedback would be applied for a deep sea mining process, controlling a
suspended grab as shown in Figure 1. Haptic feedback could improve this teleoperation and therefore
increase the level of telepresence. Two types of haptic feedback mechanisms to improve the human-machine
interface are found in literature; natural force feedback, described in paragraph 1.2.1 and haptic shared
control, described in paragraph 1.2.2.
1.2.1 Natural Force Feedback
Natural force feedback reflects forces acting on the slave part of a teleoperation to the operator, based on
sensor information of the environment. Transparency in the teleoperation is the reflection of perceived
inertia, damping and stiffness of the slave to the master as schematically shown in Figure 9 b) (Hannaford
1989 [13], Hannaford 1991 [15], Yokokohji 1992 [40], Abbot and Okamura 2003 [2]). The schematic
representation as shown in Figure 9 b) resembles the experimental setup of Abbot and Okamura as shown in
Figure 9 a), to determine the effect of different control architectures based on position and force feedback
for applying forbidden regions in a teleoperation (Abbot and Okamura 2003 [2]).
slave
k
master
m
c
Figure 9: a) A 1 DoF Transparent Feedback Mechanism experiment for contact probing (adapted from Abbot
and Okamura 2003 [2], pp. 2). b) Schematic representation of 1 DoF transparent feedback mechanism with
artificial mechanical properties; mass m, stiffness k and damping c.
A transparent master-slave system as shown in Figure 9 a) consists of several dynamic properties, such as
mass, stiffness and damping as shown in Figure 9 b) and displayed to the operator using force feedback.
The dynamic properties are based on the dynamics of the object, master and slave device and the operator
himself, as shown in Figure 10 (Yokokohji 1992 [40]).
Figure 10: a) Schematic view of dynamics in transparent teleoperation of master and slave arms, operator and
object (adapted from Yokokohji 1992 [40], pp. 850).
Scaling
In a deep sea mining process the forces acting on the slave part of the teleoperation, the hydraulic grab,
cannot be directly reflected to the operator due to the size of these forces. These forces therefore need to be
scaled down to the capability of a human operator. This can be done using a static geometric scaling factor
or to add a dynamic part to this scaling factor as shown in Figure 11 (Kaneko et al. 1997 [20]). Scaling due
to geometric differences is often needed, but scaling due to maximum capable force differences can also
occur. The maximum occurring forces of a hydraulic grab is limited by the available hydraulic power of the
15/71
R.J. Kuiper
Biomechanical part of thesis
GRABBING ROCK IN DEEP SEA MINING
machine and can therefore be scaled down using the maximal occurring hydraulic pressure of the actuators
for both winch and grab control as shown in Figure 3.
Figure 11: Transparent Feedback Application with impedance shaping scale factors (adapted from Kaneko et al.
1997 [20], pp. 71).
Control Architectures
Traditional teleoperation systems with natural force feedback consist of two channel system architectures,
which can be position-position, position-force and force-force systems (Hannaford 1989 [13], Hannaford
1991 [15], Yokokohji 1992 [40], Abbot and Okamura 2003 [2]). Lawrence proposed a multivariable system
architecture for achieving transparency and optimizing the results (Lawrence 1993 [21]). He introduces the
use of four-channel data transmission between master and slave; velocities and forces in both directions as
shown in Figure 12. Combined with this feedback mechanism he designed a tool for quantifying
teleoperation system performance and stability under time delay. Where the tools for quantifying stability
and performance relates to Niemeyer and Slotine of force reflecting systems operating under time delay
(Niemeyer and Slotine 1991 [26]). Zhu and Salcudean used the four-channel data transmission structure to
achieve transparency in their system when using rate and force control instead of position control (Zhu and
Salcudean 1995 [41]).
15-02-2012
16/71
INTRODUCTION
Figure 12: Transparent feedback control algorithm using a four channel structure, depicted with gain C1-4 of
forces and velocities both ways (adapted from Lawrence 1993 [21], pp. 626).
Offering natural force feedback has shown improvements of the situation awareness of the tele-operator
(Endsley 1995 [9]). Considerable amount of research has been done to improve the quality of natural force
feedback, but perfect transparency has not been realized yet. Commercially available master-slave systems
often employ non-optimized controllers, such as 2-channel force feedback, which inherently limits
transparency. Still, even rough indications of contact forces can benefit tele-operation processes, although
marginally (Wildenbeest et al. 2010 [39]).
Sub Conclusion
Offering natural force feedback to reflect slave forces on the master creates transparency in the system,
increasing the operator’s level of tele-presence. The reflected forces need scaling for a deep sea mining, due
to the high operating forces. The forces reflected using a two way feedback structure would be sufficient and
could improve the situation awareness, however is still limited and never completely transparent.
1.2.2 Haptic Shared Control
Natural force feedback as described in paragraph 1.2.1 reflects naturally occurring system forces to the
operator, based on sensor information. The quality of this force feedback method is therefore as good as the
sensor feedback with its time delays. Inaccurate sensor information and environmental uncertainties also
prevent full automation for a deep sea mining process. The necessity for some form of manual control was
already described by Tomovic due to limitations of full control in uncertain environments (Tomovic 1969
[35]). Not all events can be captured using a mathematical decision model in real-life situations with sensory
and environmental inaccuracies. Unexpected events can occur, which cannot always be foreseen, or even
sensor failure can occur. This will render autonomous control inapplicable, and will require the human
operator to suddenly take over control, which is difficult when he is not used to control the operation
manually (Endsley 1995 [9]). Shared control is therefore developed as a combination of manual control and
17/71
R.J. Kuiper
Biomechanical part of thesis
GRABBING ROCK IN DEEP SEA MINING
automation, a continuous shifting of authority as shown in Figure 13. The first found literature of shared
control is by Hayati and Venkataraman, where a task-level sharing is applied (Hayati and Venkataraman
1989 [17]). The planning of the controlled path is shared with an autonomous controller and manual input
from an operator.
Direct
Control
Shared
Control
Autonomous
Control
Figure 13: Schematic view of principle of haptic shared control, for continuous shift of authority.
Shared control is mainly applied in two ways, input-mixing shared control and haptic shared control. The
most common and widely used manner is input-mixing, where the controller takes over the operators tasks
in critical situations and disconnecting the controller inputs from the process. The disconnection of the
control inputs creates difficulties for the operator to understand the process and the control actions. Haptic
shared control on the other hand provides forces to the operator, when the operator diverts from the
controllers optimal path for instance and keeping the connection of the control input to the process. The
application of mixed inputs has been applied on a wide range of tele-operations (Hayati and Venkataraman
1989 [17]). However it does not seems the most promising solution for optimizing the human-machine
interface because it is not keeping the human in the loop and simply takes over control, therefore greatly
reducing the operator’s situation awareness (Endsley 1995 [9]). The automatic controller is more detectible
when applying a haptic shared control, this can be done using virtual fixtures or to apply artificial guiding
forces to the operator.
Virtual Fixtures
Rosenberg, introducing Virtual Fixtures (VF) (Rosenberg 1992 [29]), has done one of the first steps in
designing the sharing of control of an operator and automatic controller. The application of VF is a fairly
simple concept for controlling a path with a master-slave combination. Where the optimal trajectory is
calculated and certain boundaries are applied to guide the operator over the path as shown in Figure 14.
Figure 14: Principle of Virtual Fixtures for trajectory control (adapted from Rosenberg 1992 [29], pp. 15, 20).
The virtual fixtures presented by Rosenberg were virtual walls in space to guide the human operator, this
was felt as a hard wall. Abbot and Okamura further looked into this hardness of the wall and presented it as
Forbidden-Region Virtual Fixtures (FRVF). They experimented with hard and soft virtual fixtures to guide the
operator (Abbot and Okamura 2003 [2]). They also experimented with virtual fixtures at the master or at the
slave side of the device. They concluded that only a hard VF at the master side is giving poor results and
that the best results came from a soft VF at both ends for their 1 DoF system as shown in Figure 9 a).
15-02-2012
18/71
INTRODUCTION
Artificial Guiding Forces
When applying a soft virtual fixture to the system, the operator is guided to stay in the correct zone of
operation. With the use of a soft virtual fixture, the operator can even be guided to the most optimal path
(Sayers and Paul 1994 [30], Abbot and Okamura 2003 [2], Bettini et al. 2004 [5]). A hard VF would make it
impossible to deviate from that path and therefore almost leaving no control to the operator. A guiding force
can therefore help the operator to stay on the path, but allows deviation in case of overruling by the
operator. The guiding control can also be split up into an attractive or repulsive control as illustrated in
Figure 15 (Prada and Payandeh 2009 [28]). The soft virtual fixtures are all repulsive forces, applying forces
to the operator to stay out of certain region. But when the repulsive field is all around the optimum path and
directs the operator down the path, an attractive force is applied. The attractive force is directing the
operator over the path. When the operator does not hold the master device, the controller would even
operate completely autonomous.
Figure 15: a) Principle of 2 DoF repulsive artificial guiding, (adapted from Prada and Payandeh 2009 [28], pp.
121). b) Principle of 2 DoF attractive artificial guiding (adapted from Prada and Payandeh 2009 [28], pp. 127).
Attractive guidance forces are mostly applied when offering haptic shared control with guidance on a teleoperating task (Boessenkool et al. 2011 [7], Mulder et al. [25]). This is proven to increase the performance
of an operator when conduction free-air movements, however for contact or rotational tasks this did not
show any improvements (Boessenkool et al. 2011 [7]). Also attractive guidance is usually applied in position
controlled tasks rather than rate controlled. However the research of Abbink showed a rate or even
acceleration controlled task with a similar guidance feedback for controlling the throttle of a car, which is
focussed on the relative velocity between two vehicles and therefore a sort of rate controlled task (Abbink
2006 [1]).
Control Architectures
Haptic shared control is a continuous combination of manual control and automation as shown in Figure 13.
Both the human operator and automation control the teleoperation by applying forces on the master input
device. The human operator therefore directly notices the actions of the automation and is able to confirm its
actions or disagree. This creates a continuous shifting of authority of controlling the teleoperation. Virtual
fixtures creates a virtual contact comparable with natural force feedback, but can be disagreed by the human
operator. Artificial guiding forces are applied continuous and therefore give a continuous feedback of the
optimal path to the operator. The level of continuous feedback forces depend on the diversion of slave from
the optimal path. When combining the level of control action into these feedback forces, a stiffness feedback
is created (Hogan 1984 [19], Abbink et al. 2008 [2]). This is best described with an application in the
automotive sector for throttle control of a vehicle (Abbink 2006 [1]), and to apply this for a rate controlled
task of a deep sea mining process with hydraulic actuators as described in paragraph 1.1.3. Artificial guiding
forces are applied on the gas pedal of the car for assisting the driver to keep a safe driving distance from the
next vehicle in front and prevent collision. The level of force feedback is calculated from the time headway as
shown in Figure 16 a). This feedback force can be applied purely depending on the time headway, or on a
combination of the pedal depression creating a stiffness feedback as shown in Figure 16 b). The use of
stiffness feedback rather than force feedback is useful for controlling a deep sea mining application with a
joystick due to the bidirectional input device, which would be instable with direct force feedback.
19/71
R.J. Kuiper
Biomechanical part of thesis
GRABBING ROCK IN DEEP SEA MINING
Figure 16: a) Car Following Application of Shared control (adapted from Abbink 2006 [1], pp. 19). b) Schematic
representation of force and stiffness feedback mechanism (adapted from Abbink et al. 2008 [2], pp. 7).
The design of the guiding forces to be offered to the operator using stiffness feedback can be best shown
with the example of throttle control of Abbink (Abbink et al. 2008 [2]). A Driver Support System (DSS) is
developed as automatic controller to measure time headways and apply forces on the gas pedal. The forces
on the gas pedal are detected by the human operator, which decides to confirm or disagree with these
actions based on visual feedback of the situation as shown in Figure 17.
Figure 17: Shared Control Algorithm for car following with guiding haptic force feedback (adapted from Abbink
2006 [1], pp. 7)
Sub Conclusion
Virtual fixtures in a deep sea mining process could prevent the operator to operate out of capabilities of the
machine and prevent incorrect control actions of the operator. Attractive artificial guiding forces could assist
the human operator in continuous operation to increase the performance of the operation and reduce control
effort by guiding the operator to the optimal control. Attractive artificial guiding is up to now only applied for
position tasks, when the controller can determine an optimal path. Therefore applying attractive artificial
guiding forces for a rate and force task of a deep sea mining excavation operation needs to be investigated.
For the cable forces controlled with the winch drum as shown in Figure 3 a), the application of haptic shared
control could improve the teleoperation of a deep sea mining operation by guiding the optimal cable force to
apply.
1.3 Grasping with Tele-operation
One of the main processes during a deep sea mining operation is cutting rock material from the seabed. This
will be done using a large-scale hydraulic actuated suspended grab to cut the rock material as shown in
15-02-2012
20/71
INTRODUCTION
Figure 1. The rock material will be cut by closing both clamshells of the grab using a single joystick with a
grasping motion as shown in Figure 3 b). Grasping with teleoperation can also be improved using haptic
feedback to improve the level of tele-presence as described in paragraph 1.2. Most of the research for haptic
feedback is applied on tele-operations controlled with arm movements, which differs for a grasping
teleoperation fully controlled with hand movements and forces. Several research of grasping with natural
force feedback is described in paragraph 1.3.1 and the use of haptic shared control for grasping in paragraph
1.3.2.
1.3.1 Grasping with Natural Force Feedback
The use of natural force feedback is besides contact forces also applied on grasping forces, to increase the
transparency of the system during a grasping task. Transparency in a grasping teleoperation can be
expressed with the teleoperation stiffness and damping (Christiansson et al. 2008 [8]). Research has been
done to determine the effect of stiffness and damping for performance of discriminating objects during
teleoperation as shown in Figure 18 a) (Christiansson et al. 2008 [8]). No improvements of performance for
object discriminating was found for improving quality of feedback, when tele-operator stiffness higher than
environmental stiffness. Natural force feedback is as well applied for manipulation of objects using dexterous
teleoperation, as shown in Figure 18 b). Using an instrument glove witch an exoskeleton attached, the
operator could feel the applied forces of the slave robot to hold and manipulate objects. Experimental results
did not show an improvement in task performance for the addition of natural force feedback due to
imperfect force transparency (Turner et al. 2000 [38]).
Figure 18: a) Discriminating object with teleoperation (adapted from Christiansson 2008 [8], pp. 1253). b)
Grasping of objects using teleoperation (adapted from Turner et al. 2000 [38], pp. 1).
Sub Conclusion
The use of natural force feedback to increase situation awareness for grasping teleoperation did not show an
increase in task performance. However some indication was found of improving learning curve and reduction
of control errors compared to manual control. Natural force feedback applied on the grasping teleoperation
of a deep sea mining process controlling the cutting force could therefore improve the situation awareness of
the process and reduce control errors in controlling the cable force.
1.3.2 Grasping with Haptic Shared Control
Shared control is also applied for manipulation of objects using dexterous teleoperation as shown in Figure
19 a). The automated controller assisted the operator to hold the objects with the correct grasping force
using input-mixing shared control (Griffin et al. 2003 [12]). The automated controller intervened when
incorrect grasping forces were applied, combined with visual and auditory feedback. Due to intervention of
the applied grasping force, the target window for desired forces could be enlarged as shown in Figure 19 b).
This resulted in improved task performance using shared control for a grasping task in combination with
visual and auditory feedback.
21/71
R.J. Kuiper
Biomechanical part of thesis
GRABBING ROCK IN DEEP SEA MINING
Figure 19: a) Schematic representation of Grasping Objects with Shared Control (adapted from Griffin et al. 2003
[12], pp. 1). b) Methods of feedback to the operator for grasping with shared control (adapted from Griffin et al.
2003 [12], pp. 3).
Sub Conclusion
Besides input-mixing shared control, no literature was found describing haptic shared control, guiding or
restricting the operator during a grasping task. However shared control did show improvements in grasping
task performance. Haptic shared control could alert the operator when cutting forces are too low or high and
guide the operator to apply the correct forces. This can be applied when clamshells are closed and no high
cutting force is necessary.
1.4 Problem Statement and Goal
Realizing a deep sea mining process is not been done so far due to technical difficulties and uncertainties of
economic feasibility. However at this moment a number of remaining technical difficulties for this process
such as the level of rock cutting forces and vertical lift mechanism of the material are being researched. This
thesis focusses on improving the human-machine interface for controlling deep sea mining excavation
process by applying haptic feedback to the operator. Controlling a deep sea mining excavation process by
tele-operation using only a standard human-machine interface with visual feedback is difficult for the
operator. The operator’s situation awareness is limited by the feedback of the process, therefore making a
tele-operation of a large machine on great depths a complex procedure. Consequently this reduces the
process efficiency and can even cause control errors in critical processes. The aim of this research is to
improve the human-machine interface by offering haptic feedback to the operator, by offering ‘natural force
feedback’ and by offering ‘haptic shared control’. Both have never been designed for deep sea tele-operated
manipulation and not applied for a rate and force controlled task.
1.5 Hypotheses
It is hypothesised that natural force feedback will not affect performance, but will increase situation
awareness for a teleoperation of deep sea mining using a grab. This will result in reduced control errors that
would otherwise cause incorrect behaviour of the grab and increased response time in critical processes. It is
also hypothesised that haptic shared control will improve the teleoperation performance and reduce the
control effort of the teleoperation. This will result in increased production results for rock excavation and
reduced control angles during deep sea mining processes in normal conditions. The summary of the
15-02-2012
22/71
INTRODUCTION
hypotheses for the effect of both haptic feedback mechanisms on the performance is given in Table 1,
control effort in Table 2 and situation awareness in Table 3.
Conditions
No SC
SC
No NFF
NFF
0
+
+
++
Table 1: Hypotheses of increase of performance results for all conditions
Conditions
No SC
SC
No NFF
NFF
0
+
0
+
Table 2: Hypotheses of reduction of control effort results for all conditions
Conditions
No SC
SC
No NFF
NFF
0
0
+
+
Table 3: Hypotheses of increase of situation awareness results for all conditions
1.6 Approach
The human-machine interface can be improved by applying haptic feedback, thereby it will increase the
performance and situation awareness of the operator. Two haptic feedback mechanisms as described in
paragraph 1.2 are stated in the hypotheses to improve the teleoperation. The hypotheses are tested with
experiments to measure the improvements of each feedback mechanism for performance, effort and
responses in critical situations. The experimental method is given in chapter 2, describing two types of tasks
that were conducted by the subjects during the experiments at different conditions. Chapter 3 describes the
design of the mechanical and software parts of the experimental setup. The results of the experiments are
shown in chapter 4, which are further discussed in chapter 5. The conclusions of all hypotheses based on the
experimental results are given in chapter 6. Chapter 7 gives further recommendations regarding the
conducted experiments and validations of the stated hypotheses.
23/71
R.J. Kuiper
Biomechanical part of thesis
GRABBING ROCK IN DEEP SEA MINING
15-02-2012
24/71
EXPERIMENT METHOD
2 EXPERIMENT METHOD
Excavating rock material during a deep sea mining process using a suspended grab as shown in Figure 1 in
paragraph 1.1.1 is a difficult procedure. Full automation is not feasable due to environmental uncertainties
and sensor limitations. Controlling the process manual by teleoperation could be improved by using haptic
feedback. Two haptic feedback mechanisms to improve the human-machine interface were found in
literature and described in paragraph 1.2; natural force feedback and haptic shared control. Paragraph 1.1.1
described the deep sea mining process with a suspended grab and the control parameters as shown in
Figure 3. An experimental setup is developed to validate the hypotheses as described in paragraph 1.5 for
improvements of haptic feedback mechanisms for a deep sea mining process. The experiment consists of a
human operator conducting a virtual deep sea mining task using two haptic joysticks and a display as shown
in Figure 20 a) and b).
Display
Grab
Handle
Winch
Handle
F
W
F
EM Button
G
Armrests
Figure 20: a) Schematic view of experimental setup with a display and two joystick handles. b) Overview of
experimental setup with two joysticks and display.
A description of the process of the experiment is given in paragraph 2.1, describing the experimental setup
and task to be conducted by the subjects. In paragraph 2.2 is a description of the experiment given, for
conditions and measured variables.
2.1 Process Description
An experiment was conducted to validate the hypotheses of improvements of haptic feedback mechanisms
for a deep sea mining process. The participating subjects are described in paragraph 2.1.1. A general
description of the used experimental setup is given in paragraph 2.1.2, which is described in more detail in
chapter 3. A description of the task is given in paragraph 2.1.3.
2.1.1 Subjects
Ten subjects in the range of 24 to 27 participated in the experiment, nine male and one female. None of the
subjects had previous experience with teleoperation systems besides some gaming experience. All subjects
were right handed and participated voluntarily without financial compensation for their time and effort. Only
a small suiting prize was promised beforehand and given to the top three of best performing subjects, to
increase the competitive behaviour for optimal concentration. Before conduction the experiment each subject
25/71
R.J. Kuiper
Biomechanical part of thesis
GRABBING ROCK IN DEEP SEA MINING
was given a written instruction that explained the experiment and setup to be controlled. After a short
demonstration of the setup, each subject had at least ten practice trials, more if requested and included all
conditions. A brief study was performed to determine the learning curve of the subjects for controlling this
experimental setup and no improvements were found beyond just a few trials.
2.1.2 Experimental Setup
The experimental setup for controlling a deep sea mining process and control variables as shown in Figure 3,
consist of two joysticks and a display as shown in Figure 20 a). The setup consists of an electro-mechanical
master to be used as input device for the human operator to control the virtual slave robot. The master
device consists of two force feedback joysticks and armrests as shown in Figure 20 b). Each joystick is
actuated with an electronic planetary-geared motor, connected with cables and spring to create series-elastic
actuation. Using two incremental angular encoders for each joystick, the position and force on the joystick
can be measured. The joystick for controlling the grab force is a one directional joystick to control the closing
force, combined with a button on top of the joystick the clamshells can be opened. On both joysticks were
hall-sensors attached to detect contact of the operator, wearing magnets attached to the hand to prevent
uncontrolled movement of the grab. The behaviour of the virtual slave robot is calculated with a
mathematical model describing the kinetics of the machine and forces of the environment on the machine. A
real-time controller applies feedback forces on the master device based on parameters gained from the
virtual model, depending on the experimental conditions. The real-time controller runs on an industrial
computer dedicated to real-time computing, connected to the sensors and actuators of the master device.
The display in front of the operator shows visual information about the deep sea mining process gained from
the virtual model. This visual feedback is shown to the operator in all experimental conditions. The visual
feedback contains besides information of the behaviour of the grab also several control parameters, such as
optimal cable force and predictions of ground contact and closed clamshells. Both the virtual model and the
visualization run on two separate computers with a Windows XP operating system.
2.1.3 Task description
Each subject has to operate the system as fast as possible, when operating safely. This is stimulated by
recording real-time the production of the task and previous tasks, displayed to the operator. A penalty of 300
seconds is given for a task when the subject makes an error during execution of the task. These errors can
consist of a high heeling angle of the grab or a late response during mechanical failure. These penalties were
applied to increase the attention level of the subject and add a competitive factor to the experiment, where
the best results were awarded with a small prize. These time penalties were not taken into account for
further analysis besides the number of errors made. An overview of the situation of excavating rock using a
suspended grab is shown in Figure 21.
Figure 21: Situation overview of grab excavating rock on seafloor.
The excavation procedure using a suspended grab is shown in Figure 21, consisting of multiple stages.
15-02-2012
26/71
EXPERIMENT METHOD
In Figure 3 are the main control parameters shown to operate the deep sea mining process, also shown in
Figure 20. The different stages for operating a deep sea mining excavation process with a grab mainly
consists of the vertical transport and cutting process, as listed in Table 4 with the estimated task completion
time.
Description
Estimated task time
Setting down the empty grab on the seabed
Starting the grabbing process
Controlling the grabbing process
Stopping the cutting process
Uplifting the grab with a full grab
5
1
30
1
5
Table 4: Task description of the conducted experiment for a grab with estimated task completion times
For each condition the subject had to execute six tasks, in random order for each condition. For each subject
the executed tasks were randomized identical for each condition, only the order of conditions was changed
for each subject. The six tasks to be executed per condition consist of three performance tasks to complete
and three catch trials as listed in Table 5. The three performance trials are conducted for each experimental
condition in normal situations and normal ground forces. The first catch trial is to see how the subject reacts
under difficult situations, to see if the experimental conditions improve or confusing the operator during
difficult situations. The difficult situation mainly consist of highly fluctuating ground forces due to small chip
fractures, causing the grab not to penetrate the soil and merely scraping the surface. During the second
catch trial the ground forces are exceeding the grab’s capabilities, causing an incomplete cutting process.
This task is added to increase the attention level of the subject and not expecting a full completion, but
measuring response of the operator. The last catch trial consists of normal ground forces, but a mechanical
failure after varying times and is used to measure the reaction times to this failure. This mechanical failure is
only shown using visual feedback, indicating the level of attention of the operator and mental load of the
process. All these six tasks are randomized for each condition in the same order for each subject.
Task
Category
Description
1-3
4
Performance
Catch trial
Normal conditions
Critical situation due to difficult soil, causing rotation of grab
5
6
Catch trial
Catch trial
Hard soil overall, unable to complete tha task and causing rotation
Reaction time of mechanical failure of pressure or temperature
Table 5: Different types of task description during experiment
The presented data in the results is reordered, in the order of 1-6 as described in Table 5. The shown
ground pattern, displayed to the operator was also randomized. The ground pattern had no influence on the
behaviour of the grab and was solely meant to mask the different tasks, so there was no recognition of the
events.
2.2 Experiment Description
The experiment is conducted with the experimental setup as shown in Figure 20 to validate the hypotheses
for improving the human-machine interface for a deep sea mining process. The different experimental
conditions used during this experiment are described in paragraph 2.2.1 and the measured variables in
paragraph 2.2.2.
27/71
R.J. Kuiper
Biomechanical part of thesis
GRABBING ROCK IN DEEP SEA MINING
2.2.1 Experimental Conditions
The experiment was developed to validate the hypotheses as stated in paragraph 1.5. Two methods for
haptic feedback are described for improving the human-machine interface of a deep sea mining
teleoperation; natural force feedback and haptic shared control. Both methods are applied in four different
combinations for experimental conditions as listed in Table 6.
Condition
Description
Abbreviation
E.C.
E.C.
E.C.
E.C.
Direct Control (baseline)
Haptic Shared Control
Natural Force Feedback
Both Feedback Mechanisms
DC
DC-SC
FF
FF-SC
0
1
2
3
Table 6: Experimental Conditions
The baseline condition represents the current control application as applied in comparable situations of
controlling a suspended grab. Visual feedback is applied during all four conditions, also showing the feedback
parameters used for haptic feedback as described in paragraph 2.1.2 and described in detail in paragraph
3.3.2. The design of the haptic feedback mechanisms is based on stiffness feedback, adapted from Abbink
which again was based on the impedance control of Hogan (Hogan 1984 [19], Abbink et al. 2008 [2]). The
magnitude of the feedback force F for stiffness feedback depends on the angle of the joystick θ, therefore at
increasing joystick angle an increasing feedback force is applied as shown in Figure 22 a). The total feedback
force Fmax determines the stiffness of the feedback and is adjusted depending on the environmental forces
acting on the grab. The Fmax is determined using sensor feedback of the hydraulic actuators of their
operating pressure, causing the actuation force. Therefore the natural force feedback of the grabbing
joystick depends on the hydraulic pressure in the clamshell hydraulic cylinders and the winch joystick on the
hydraulic pressure of the winch hydraulic motors pulling the hoisting cable. The haptic shared control
feedback forces depend on a shifting of the equilibrium point dθ of the stiffness feedback as shown in Figure
22 b) based on the literature of Mulder for guiding a steering wheel (Mulder, Abbink and de Boer 2008 [25]).
Furthermore virtual fixtures are applied as shared control method limiting the joystick angle when maximal
velocity of the actuator is reached, reducing the control effort. The application of feedback mechanisms is
described in more detail in paragraph 3.3.3.
F
F
Fmax
dθ
θ
θ
Figure 22: a) Stiffness feedback mechanism for haptic feedback on a joystick for natural force feedback. b) Shift
of equilibrium point of joystick angle for haptic shared control using stiffness feedback.
The parameters used for haptic feedback are shown in Figure 23 for both joysticks with natural force
feedback shown in blue and haptic shared control shown in red. The warning on the grab joystick uses a
vibrating feedback force when applying an input force which is unnecessary after closing of the clamshells.
15-02-2012
28/71
EXPERIMENT METHOD
Hydraulic
Actuator Forces
Continuous
Forces
Winch
Joystick
Contact
Seafloor
Shift of
Equilibrium
Minimal
Cable Force
Hydraulic
Actuator Forces
Continuous
Forces
Clamshell Angle
Closed
Warning
Gab
Joystick
Figure 23: Decision model of the intelligent controller
2.2.2 Measured Variables
To analyse the effect of the applied haptic feedback mechanisms, a great amount of variables were
measured of the human operator and virtual grab model. The variables were measured at different sample
rates, depending on the rate of fluctuation of the variables to reduce the load of network communication.
Several metrics were calculated based on the measured variables and categorised in four groups as listed in
Table 7. The first three categories of the calculated metrics are measured objectively, except for the selfassessment category, which are subjective metrics. All metrics are used for analysing the three performance
trials as listed in Table 5. For the first catch trial only the performance, system and control effort metrics are
used. To analyse the second catch trial, only production time, frame angle and control effort is used. For the
last catch trial none of the mentioned metrics are used and a new metric reaction time was used, only
applicable for this tasks.
Performance
System
Control effort
Self-assessment
Time
Frame Angle
Summation
Joystick Angle
Overall Workload
Volume
Cable Length
Difference
Moving Joystick
Angle
Overall Rating
Production
Force Setpoint
Deviation
Applied Joystick
Force
Table 7: Evaluation metrics for evaluating experiment results in four categories.
To determine the subjective workload, the subject was asked to answer six brief questions at the end of
every conducted task. These questions are based on Hart and Staveland’s NASA Task Load Index (TLX)
method to determine the subjective workload of an operation (Hart and Staveland 1988 [16]). The six
questions are described in Table 8, which were asked for each six task, at all four conditions.
29/71
R.J. Kuiper
Biomechanical part of thesis
GRABBING ROCK IN DEEP SEA MINING
Title
Abbreviation
Description
Mental Demand
MD
How mentally demanding was the task?
Physical
Demand
PD
How physically demanding was the task?
Temporal
Demand
TD
How hurried or rushed was the pace of
the task?
Performance
PE
How successful were you in accomplishing
what you were asked to do?
Effort
EF
How hard did you have to work to
accomplish your level of performance?
Frustration
Level
FR
How insecure, discouraged, irritated,
stressed, and annoyed were you?
Table 8: Questionnaire
Each question had to be answered by means of rating from one to twenty as shown in Figure 24. The total
workload consists of a weighted average of all six questions for each task, with an equal weigh factor for all
questions. The overall rating was asked in the end of the entire experiment for each subject, rating each
experimental condition with the similar scale as shown in Figure 24.
Trial Number
Very Low
Very High
Figure 24: Example question rating
2.2.3 Data Analysis and Statistics
The statistical data is displayed with the use of a box plot (McGill et al. 1978. The statistical data for each
condition in the boxplots consist of the median (white circle), upper and lower quartile (blue/red tick line),
extreme values (blue/red thin line), confidence interval of mean value (black triangles) and outliers (blue/red
circles). Some of the data shows a wide confidence interval, exceeding the upper and lower quartile, due to
the small sample size. The median of the baseline condition (grey dotted line) indicates the reference point
of the data. First a one-way analysis of variance (ANOVA) is used for determining statistical significance in
difference of means. A two-way ANOVA was chosen for correcting of variations between subjects, using (••)
for p≤0.01, (•) for p≤0.05 and (-) for p>0.05. The independent variables used for the statistical analysis are
the experimental conditions as listed in Table 6 of the haptic feedback selection. The measured variables as
listed in Table 7 of the evaluation metrics are the dependent variables, the outcome depending on the
experimental conditions.
15-02-2012
30/71
EXPERIMENTAL SETUP DESIGN
3 EXPERIMENTAL SETUP DESIGN
The experimental setup as shown in Figure 25 mainly consists of a display and two joysticks to operate a
deep sea mining procedure, one joystick controlling the winch and one the grab. Both joysticks are one
dimensional, controlling a velocity parameter of the process. The display shows a side view of the position of
the grab and seafloor. Also position, velocity and force data of the controlled parameters and various
additional parameters are shown to control the process.
MMI Computer
Connecting Hub
Extra Display for
Virtual Model
MMI Display
Recording Computer
Virtual Model &
Control Panel
Computer
Master Device
Real-time
Controller
Figure 25: Complete overview of experimental setup with all connected hardware for controlling the setup.
The operator has to place both arms on the armrests and his hands around the sticks to control the process.
The electronics for applying force feedback on the joysticks is placed on the same base in front of the
operator, combined with a real-time controller placed in between the joysticks. The mechanical design is
further described in more detail in paragraph 3.1. An overview and list of used mechanical and electrical
components is given in paragraph 3.2. A detailed overview of the software structure, design of the visual
interface, real-time controller and the mathematical model of virtual slave robot is given in paragraph 3.3.
3.1 Mechanical Design
The mechanical part of the experimental setup is designed to apply a constant high level of force on a
joystick instead of vibrations as used in commercial available joysticks. Also an adjusted design was needed
to create a grasping motion for controlling the cutting force of the grab, for eventually combining both
joysticks. The mechanical principle of both joysticks consists of two pulleys connected with cables and
springs as shown in Figure 26 a). The joystick handle is connected to one pulley and an incremental encoder,
measuring the angular position as shown in Figure 26 b). An electrical geared motor is connected to the
other pulley and an encoder. The relative measured distance of both angles represents the elongation of
both springs, which have pretension to prevent slack in cables. This elongation is used in combination with
the calibrated stiffness of the spring, to measure the force on the joystick handles.
31/71
R.J. Kuiper
Biomechanical part of thesis
GRABBING ROCK IN DEEP SEA MINING
Hand rest
E-motor
Springs
Encoders
Figure 26: a) Schematic view of mechanical principle for a single joystick. b) View of experimental setup for the
grabbing joystick.
Both pulleys are connected on steel axes, mounted with bearings in aluminium tubing for ensuring alignment
of both axes. Two aluminium frames connect the two encoders on both axes, mounted with springs for
alignment of the fragile incremental encoders. The joystick for controlling the winch drum is shown in Figure
27 a) and the clamshells is controlled with the joystick design as shown in Figure 27 b). Both joysticks are
almost identical in design as shown in the figure. The main difference is the static pole attached on the base
frame for the grabbing joystick as shown in Figure 27 b). The winch joystick has a slightly larger stroke,
allowing bidirectional control.
Figure 27: a) View of mechanical design of joystick handle for controlling the winch drum. b) Mechanical design
of the grasping joystick for controlling the movement of the clamshells.
Both bases contain some drilled mounting holes for assembly purposes and are mounted on a base plate for
alignment of the joysticks and armrest to operate the controls. The electronics for controlling the electrical
motors and sensors also are mounted on this base plate as shown in Figure 28 a) and b), including the realtime controller in the middle.
Figure 28: a) Design drawing of mechanical setup. b) View of master device of experimental setup.
15-02-2012
32/71
EXPERIMENTAL SETUP DESIGN
3.2 Components
Most of the applied components in the experimental setup are mainly supplied on loan base for cost
reduction. Reduction of design time was hereby also achieved because of pre-designs for the original
applications of the components. The main mechanical components such as the joystick handles and base
frames were specially designed and fabricated at Delft University of Technology (DUT), which also supplied
many components. Seatools bv. supplied the industrial real-time computer, laptops and remaining electrical
components for the setup. A general overview of all the used components and connections of the
experimental setup is shown in Figure 29.
Emergency
Button
Motor
Controller
Display
MMI
Computer
Angular
Encoder
E-motor
Virtual Slave &
Control Panel
Computer
Real-time
Computer
Joysticks
Fan
Sub
Frame
Recording
Power
Supply
Arm Rest
Computer
Ethernet
Hub
Relays &
Fuses
Wiring
Figure 29: Schematic view of mechanical and electrical components of experimental setup
3.2.1 Mechanical
The mechanical components as shown in Figure 29 of the experimental setup are summarized listed in Table
9.
Description
Supplier
Details
Pulleys
Springs
Frame
DUT
DUT
Fabricated
Special pre-designed cable pulleys and cables
Inclusive modified spring stops and adjusters
Frames, handles and attachments
Table 9: Description of mechanical components of the experimental setup and their suppliers.
3.2.2 Electrical
The electrical components of the experimental setup are listed in Table 10. The real-time computer is
powered trough the 24V power supply, also powering the motor control board. The real-time computer uses
an analogue output to send a control signal to the motor control board, controlling the DC electrical motors
with a pulse width modulation signal. The emergency button switches off the motor control board and
therefore disabling the electrical motors, but the real-time computer remains powered.
33/71
R.J. Kuiper
Biomechanical part of thesis
GRABBING ROCK IN DEEP SEA MINING
Description
Supplier
Manufacturer
Details
Electrical motor
Angular encodes
Real-time computer
Signal IO
Motor control board
Power Supply
Cooling Fan
Emergency button
Relays and wiring
DUT
DUT
Seatools
Seatools
DUT
Seatools
Seatools
Seatools
Seatools
Maxon
Scancon
Bachmann
Bachmann
AMC
Puls
-
RE35 90W 24V DC, GP32 planetary gearbox
2RMHF 7500 pulses incremental encoder
MX213, 200 MHz
AIO288, DIO232 and 2xCNT204
2x az12a8ddc
24V, 10A, 288 W
Including contact detection hall sensors
Table 10: Description of electrical components of the experimental setup and their suppliers.
3.3 Software Design
Several software applications are designed for this experimental setup. An interface is designed for
visualizing the process parameters to the operating and enabling communication of the operator to adjust
settings of the process, using a Man-Machine Interface (MMI) as described in paragraph 3.3.2. A local
controller is designed to operate the master device and apply the correct forces on the joysticks, as
described in paragraph 3.3.3. The behaviour of the slave is calculated using a mathematical model as
described in paragraph 3.3.4. All these software part communicate through the real-time computer, using a
Standard Variable Interface (SVI) as described in the general overview in paragraph 3.3.1. A control panel is
designed to apply the correct settings and experimental conditions, as described in paragraph 3.3.5.
3.3.1 General Overview
The general layout of the software structure is shown in Figure 30. This figure shows the three main
software components connected with the SVI layer on the real-time computer; The MMI, virtual slave and
local controller. Also two less important software components are shown, the control panel to adjust settings
and the recorder for offline analyses.
real-time
computer
laptop
Real-time
controller
MMI
laptop
Virtual
Slave
mechanical
Master
laptop
SVI
Recorder
Control
Panel
Figure 30: Software structure of the experimental setup
Three laptops are connected with the real-time computer using a hub as shown in Figure 29. Two extra
displays were needed to display the MMI to the operator and one for the virtual model. Three laptops were
needed for this setup as shown in Figure 30 and Figure 25, because of the loading and related stability of the
software components. Due to the limited time to develop this setup, the MMI was programmed using Matlab
GUI, causing unnecessary large process time running the program due to its Java machine. The virtual
model on the other hand consists of just a large amount of computations, causing a large process time and
15-02-2012
34/71
EXPERIMENTAL SETUP DESIGN
therefore the need to split these processes. The control panel was designed to run parallel on the same
laptop as the virtual slave process. A third laptop was needed to run the recording program, to insure a
stable recording of the measured variables.
3.3.2 Visual Interface
The operator is presented with a display that shows visual information about the deep sea mining process.
The interface contains the position and some control parameters of this process, as shown in Figure 31. On
the right bottom of the figure a top view of the grab is shown with several targets displayed, scheduled for
excavation. These targets are linked to the tasks to be performed and help to execute them in the correct
order. In the right top of the figure a side view of the operation is shown, to give a general overview of the
process. The red dotted line represents the virtual start/finish line of the task, to compare experiment results
and production calculation. On the left top of the figure is shown the relative position of the clamshells to the
grab’s frame. For each clamshell also the rotating velocity and hydraulic cutting force is shown in bar graphs.
Some extra numerical data is shown in this field, indicating the progress of the excavation process. The
button labelled with ‘open grab’ indicates whether or not the clamshells are controlled to be opened. This
opening function is activated using the hardware button on the grasping joystick, but can also be activated in
the MMI.
Clamshell data
System data
Frame data
Control data
Winch data
Automation data
Side view
Target planning
Figure 31: Visual interface for controlling a deep sea mining process
The top middle part of the figure shows an artificial horizon, displaying the rotations of the grab’s frame and
two clocks display the heading and depth of the frame. Three bar graphs indicate the winch data, hydraulic
cable force, velocity of the winch and the relative length. In Figure 32 is shown the principle of this relative
35/71
R.J. Kuiper
Biomechanical part of thesis
GRABBING ROCK IN DEEP SEA MINING
cable length as a difference in positions. The relative length variable displays whether the cable is stretched
or has slack, indicating the expected reaction time of the cable force and frame displacement. The winch
cable force bar also has an indication of the automatic controller showing the optimal cable force, displayed
with the red triangle.
+
Lcable
dL = Zframe - Lcable
zframe
Figure 32: Relative cable length
In the left middle part of Figure 31 is shown some system data of the virtual grab, valve position, pump
pressures and temperatures. On the right of this system data are some indicators, showing a five second
prediction of some control actions. The bottom part of the interface displays some data of the experimental
setup, such as connections and motor actuations.
3.3.3 Real-time Controller
The real-time controller determines the forces acting on the mechanical setup. The controller uses the
angular incremental encoders and the setpoint data from the virtual model to control the electrical motors of
the setup. The controller consists of two parts as shown in Figure 33 b), one for the winch and one for the
grab joystick, both almost identical. The main difference between both controller parts is the bidirectional
movement and therefore switching of the force direction and the detection of the mid position. The in- and
outputs of both control parts is read and written to the SVI layer on the real-time controller as shown in
green in Figure 33 a).
Figure 33: a) Top level of control model as implemented in Matlab Simulink. b) Control of the real-time
controller, showing two control parts for the grasping and winch joystick.
The general overview of a control part is shown in Figure 34, where the grabbing control part is displayed.
The control part contains a logic part, checking sensor signals, directions, temperatures and standby
functions. The calibration part initializes the incremental encoders and has an automatic calibrations function.
The length and forces of the springs and force on the handles is calculated in the setup part of the control,
for offline analysis and not for controlling the forces. The force setpoint part of the control calculates the
required force based on the joystick angle, settings of the experiment and the setpoint from the virtual grab
simulation. The output control to the motor is calculated in the controller with the use of feed forward,
calculated from the setpoint. This is included with output limitations, protections and calibration of the output
motor signal. The feed forward control is found the most stable control instead of several applied closed loop
controls, based on angular feedback from the motors and joysticks. The calculated forces of the springs with
15-02-2012
36/71
EXPERIMENTAL SETUP DESIGN
their hysteresis in combination with the backlash in the gears mainly caused this instability of the closed loop
controls.
Figure 34: A Matlab Simulink representation of the grabbing control part of the real-time controller
The mathematical model of the virtual slave sends the main setpoint for transparent feedback to the control
model, based on measurements of the virtual hydraulic pressure. The setpoint for haptic shared control also
comes from the virtual grab model, based on several conditions of the system such as cable force and
predicted ground contact. The actual force setpoint for motor actuation of the joystick is calculated based on
these setpoints as shown in Figure 35. The setpoint is based on a minimal static force, a spring force
calculated using the angle of rotation and an end force. The control force is not used during this experiment
and was added for calibration purposes. The vibration forces are added to the force setpoint with some
limiting checks finally for ensuring safety of the system.
Figure 35: Matlab Simulink representation of force setpoint calculation, depending on different components
The method of applied feedback mechanisms is schematically shown in Figure 36 a) for the winch joystick
and in Figure 36 b) for the grab joystick, displaying the return force against the joystick angles. The minimal
returning force (Fmin) is present in all conditions to return the joysticks to their initial position. Variable spring
stiffness is applied for natural feedback, adjusted proportional with the slave stiffness. The end of the stroke
is marked with a force barrier in combination with a vibration alarm, when shared control is enabled. The
winch feedback is identical to the grab feedback, only the winch feedback is applied in both directions and
the grab feedback only has one direction.
37/71
R.J. Kuiper
Biomechanical part of thesis
GRABBING ROCK IN DEEP SEA MINING



Figure 36: a) Illustration of the force feedback mechanisms of the winch joystick. b) Illustration of the force
feedback mechanisms of the grasping joystick.
In Figure 37 is the actual shared control feedback mechanism shown, which essentially calculates the optimal
steering angle, and then shifts the equilibrium point of the joystick to that angle. A changed force is applied
due to the shift of equilibrium point, if the joystick remains in the same position, causing a pull to this new
point. Contact detection using hall sensors inhibits the joysticks to rotate when the operator does not hold
the joystick. Furthermore to induce the need for the operator to participate in the movement, is the applied
new control position always slightly less than required. The shifting of the equilibrium point guides the
operator to the correct control position of the joysticks and the force end barrier acts as a virtual fixture of
the joystick.


Figure 37: Illustration of the mechanism of shifted equilibrium point of the continuous haptic shared control
feedback mechanisms for the winch joystick.
3.3.4 Virtual Grab Simulation
Using an actual slave robot during this experiment was not an option and therefore a virtual slave robot had
to be designed. A mathematical simulation model is developed for calculating the behaviour of this virtual
grab. Figure 38 shows the top level view of this simulation for the behaviour of a grab. The in- and outputs
of the simulation are level-2 S-functions for implementing the communication protocol with the real-time
controller, using a special developed script by Seatools bv. for establishing this communication using the
Transmission Control Protocol (TCP).
15-02-2012
38/71
EXPERIMENTAL SETUP DESIGN
Figure 38: Top level of Matlab Simulink representation of the virtual grab model.
The general in- and outputs to and from the virtual grab simulation are given in Figure 39. The control panel
transfers setting to the virtual slave for experimental conditions and task numbers to be executed. The user
controls the master device, which sends the angular values to the controller which again sends its calibrated
values to the virtual grab simulation that controls hydraulic valves. The haptic feedback settings to the
controller are set by the virtual grab simulation and the visual feedback on the display is based on the
behaviour of the virtual grab from the simulation.
Master
θW-M θW-JS
θG-M θG-JS
Real-Time
Controller
θW FW
θG FG
Virtual
Slave
IW IG
Control
Panel
ECsel Grnr
Trnr
θWeq FWmax
MMI
FGmax
XW UW FW CW XG UG FG CG
XF UF FF Ppumps Tpumps
Figure 39: Schematic view of communication of the virtual grab model.
The calculation of the behaviour of the virtual grab is subdivided into physical separated parts, illustrated in
Figure 40. The generated ship motions after heave compensation induce a change of position of the winch
drum. The control signal from the winch joystick adjusts the position of a hydraulic valve, controlling the
velocity and force of the winch drum. The cable force acts backwards on the winch drum but is also
dependent on the relative winch and frame position, causing a spring force.
39/71
R.J. Kuiper
Biomechanical part of thesis
GRABBING ROCK IN DEEP SEA MINING
Figure 40: A Matlab Simulink representation of the virtual grab simulation model, subdivided into physical
separated parts.
Figure 41 illustrates the calculation of the frame position from the various forces acting on it. The frame
calculation mainly consists of a six degree of freedom (DoF) mass with four forces translating and rotating it.
The thruster force is controlled with an automatic controller for holding the horizontal position and rotation
around the z-axis constant during free hanging. The cable force is decomposed in a vertical and horizontal
component in combination with momentum acting on the frame. This calculation is done using rotation
matrices from the Euler angles for converting the inertial components to body components. This conversion
is done for all four forces.
Figure 41: A Matlab Simulink representation of the calculations of the grabs frame dynamic behaviour.
The position of the frame is combined with the relative translation of the clamshells. The translation of the
clamshells is subdivided into a hydraulics part and translation part, comparable to the winch calculation. The
control signal from the grab joystick manipulates a hydraulic valve as well. The clamshells are actuated with
hydraulic cylinders consisting of changing compressible hydraulic volumes. The hydraulic flow can be
calculated from a simple pump simulation in combination with this valve position. The forces of the
15-02-2012
40/71
EXPERIMENTAL SETUP DESIGN
clamshells are acting on the ground causing the rock material to break. The subdivision of the calculations
for reaction of the ground forces is shown in Figure 42 a). The simulated ground pattern as also shown in
the visualization, is first calculated for the position of the ground level. From this position and the clamshell
positions, contact can be determined. When contact is enabled, the calculation holds contact for stability of
simulation. Contact is broken if vertical forces exceed a threshold for a certain time. The reacting ground
forces are modelled as a spring-damper for both teeth positions of the clamshells in both horizontal as
vertical direction as shown in Figure 42 b). Therefore the ground forces are in equilibrium with the frame and
clamshell dynamics, so mainly gravity and inertial reactions. The cutting forces are calculated as the resulting
sum of all ground forces and clamshell forces. These cutting forces are used to determine the breaking of
rock chips, using a vector of thresholds to determine fracturing of the rock material. When a threshold of
rock forces is exceeded, the contact position of the rock is changed at a limited rate.
FL
kh,ch
kh,ch
kv,cv
FR
kv,cv
XL
XR
Figure 42: a) A Matlab Simulink representation of ground contact and force calculations. b) Schematic
representation of mass-spring system for ground contact forces for changing ground positions.
The setpoints of the feedback forces for the mechanical setup are calculated from all the calculated positions
and forces of the virtual grab. The setpoint for the natural force feedback is simply proportional to the
hydraulic pressure of the actuator. The calculation of the equilibrium point for shared control on the winch
joystick is shown in Figure 43 as previously described and shown in Figure 37 in paragraph 3.3.3. The
setpoint is calculated based on contact with ground, cable force, failure of components and the position of
the clamshells. These values are based on a five second prediction calculated with the velocity of these
values. The calculation of the setpoint for the cable force is based on the prediction of contact, rotation of
the frame and closing of the clamshells. This setpoint is also transferred to the visualization to help the
operator controlling the process.
Figure 43: Calculation of equilibrium point for winch shared control
41/71
R.J. Kuiper
Biomechanical part of thesis
GRABBING ROCK IN DEEP SEA MINING
Non-linear control output
The virtual grab is controlled with the use of two joysticks; one for rotating the winch drum and therefore
controlling the cable force, and one joystick for rotating the clamshells and therefore the grab force. Both
joysticks operate a hydraulic valve of the virtual slave, therefore controlling the velocity or force when there
is load on the actuator. The winch joystick handle has a non-linear output character to the given rotation
angle of the joystick, as shown in Figure 44 a). This increases the sensitivity of the control while still enabling
full speed of the controlled actuator.
FI
FH
FG
FV

Figure 44: a) Non-linear winch joystick output b) Non-linear grab force c) decomposed grab forces.
The output grab force has a non-linear character as shown in Figure 44 b), due to the geometry of the
clamshells rotating on the frame. The output force can be decomposed into a vertical and horizontal
component, as shown in Figure 44 c). The vertical component is limited by the weight and dynamic effects of
the entire grab and is therefore relative small. The horizontal force increases when the closing angle
increases; therefore the available grab force increases while closing the grab.
3.3.5 Control Panel
A simple control panel is designed for determining the settings for the experiment and overview the setup,
shown in Figure 45. An overview of the main parameters of the virtual grab simulation is shown real-time
low frequent on the left of the panel. This also includes a check of the selected ground model, reaction
forces and feedback mechanism. On the right of the panel is an overview shown of all the experimental
setup parameters. The right bottom of the panel contains the selection of the force feedback mechanisms
and task selection options. The centre of the panel contains the parameters and selections of the real-time
control model.
15-02-2012
42/71
EXPERIMENTAL SETUP DESIGN
Figure 45: Control Panel
43/71
R.J. Kuiper
Biomechanical part of thesis
GRABBING ROCK IN DEEP SEA MINING
15-02-2012
44/71
EXPERIMENT RESULTS
4 EXPERIMENT RESULTS
The experiment as described in chapter 2 was conducted to validate the hypotheses as described in
paragraph 1.5. The hypotheses state improvements of performance for offering haptic shared and/or natural
force feedback, reduction of control effort for when offering haptic shared control and increasing situation
awareness when offering natural force feedback. The experiment was conducted at four experimental
conditions as described in paragraph 2.2.1.; baseline condition direct control (DC), haptic shared control (DCSC), natural force feedback (FF), both feedback mechanisms (FF-SC). The general results of the experiment
are presented in paragraph 4.1 for the overall production results for all performance trials. The general
results are also further elucidated by presenting several time traces of the experiment, and by presenting
subjective results. The results during the three performance trials are analysed in more detail in paragraph
4.2, using the all described metrics as listed in Table 7 in paragraph 2.2.2. Statistical results of the catch
trials with a reduced number of metrics are given in paragraph 4.3, only given for relevant metrics as
described in paragraph 2.2.2.
4.1 General Results
Since the goal of the experiment for each subject is to excavate as much rock volume as fast as possible, the
general performance is therefore defined as the gained rock volume divided over the time to complete, as
previously described in paragraph 2.2.2. The individual results of the three performance trials for each
subject for all four experimental conditions are shown in Figure 46 a), with the mean of all three
performance trial per subject per condition and standard deviation. The means of all subjects per condition
are separately shown in Figure 46 b) in combination with the overall mean of all subjects. However no
conclusions can be drawn directly from both Figure 46 a) and b) due to the spread in results.
Figure 46: a) Mean and standard deviation of individual production results of performance tasks only, for each
subject per experimental condition. b) Mean of production results for each subject of performance task and
overall mean, for each experimental condition.
Mainly during catch trials time penalties were given for incorrect control of the virtual slave robot, based on
large heeling angles or late response to critical situations as described in paragraph 2.1.3. The number of
time penalties for each task at each condition, summated for all subjects are given in Figure 47. The first
three tasks are the performance trials and the last three are catch trials as described in Table 5 in paragraph
2.1.3.
45/71
R.J. Kuiper
Biomechanical part of thesis
GRABBING ROCK IN DEEP SEA MINING
Figure 47: Results of number of time penalties per task for each condition, as a summation of all subjects.
The statistical significance of all results of the metrics as listed in Table 7 in paragraph 2.2.2 are shown in
Table 11, using (••) for p≤0.01, (•) for p≤0.05, (-) for p>0.05 for level of significance. The general
performance results do not shown a statistical difference, only the reduced frame angle results for natural
force feedback. The control effort however does show some statistical significant reduction. The subjective
workload is significant reduced for almost all conditions and subject rating increased for all conditions. The
results workload and rating are given for all six tasks. The results for frame angle, cable length difference
and force deviation are given for the catch trial tasks. The other performance and control effort results are
given for the three performance tasks.
Time
Volume
Production
Frame Angle
Cable Length Difference
Force Setpoint Deviation
Summation Joystick Angle
Moving Joystick Angle
Applied Joystick Force
Overall Workload
Overall Rating
Reaction Time
Trial
DC-SC
FF
FF-SC
P1-3
-
-
•
P1-3
-
-
-
P1-3
-
-
-
C1
-
•
•
C1
-
-
-
C1
-
-
-
P1-3
••
-
••
P1-3
-
-
•
P1-3
••
••
••
P&C1-3
-
•
••
P&C1-3
••
••
••
C3
-
-
-
Table 11: Two-way ANOVA results for all evaluation metrics compared to the baseline conditions DC, using (••)
for p≤0.01, (•) for p≤0.05 and (-) for p>0.05. For each metric the corresponding trial is given, performance
trials (P), catch trials (C), the number indicates the corresponding trial number.
Several results of control input and virtual slave responses using time traces are given in paragraph 4.1.1.
Results of the self-assessment as described in Table 8 in paragraph 2.2.2 are given in paragraph 4.1.2.
4.1.1 Time Trace Results
The time trace results of one subject for all three performance tasks at every applied feedback mechanism
are shown in Figure 48. This subject had the second best overall production results and was operating the
system with the highest subtlety. The three subfigure of Figure 48 a) display the winch joystick angle over
time for all conditions at each task. The winch joystick is controlling the rotation of the winch drum and
therefore the cable tension during ground contact. The three subfigures of Figure 48 b) show the applied
cable force and given setpoint of cable force for each condition at each task.
15-02-2012
46/71



EXPERIMENT RESULTS
Figure 48: a) Time trace results of winch joystick rotation angles of subject nr.6 during all three performance
tasks. b) Time trace results of cable tension force of the virtual slave robot simulation for all three performance
tasks.
The time traces as shown in Figure 48 display different control actions and responses of cable forces of the
virtual slave robot, even when controlling an identical task under different conditions. A more clear view of a
single task of a typical control action with several response parameters of the virtual slave robot is shown in
Figure 49 a) and b). The time trace is shown for subject nr.6 during performance task 1 using direct control
(DC) in Figure 49 a) and haptic shared control (DC-SC) in Figure 49 b). Both figures show the control action
compared to response of slave robot simulation for heeling angle of frame, force of cable and setpoint for
cable force. In Figure 49 b) is shown also the setpoint for the haptic shared control feedback.











Figure 49: a) Time trace results of performance task 1 for subject nr.6 using direct control (DC) as experimental
condition, showing the control action of joystick angles and reaction of the slave robot simulation of relative
cable length, heeling angle and cable force with controller setpoint. b) Time trace results of performance task 1
for subject nr.6 with Haptic Shared control feedback (DC-SC) showing identical parameters, including the
feedback parameter for the winch joystick.
47/71
R.J. Kuiper
Biomechanical part of thesis
GRABBING ROCK IN DEEP SEA MINING
Control actions using the joystick angles can be decomposed into a low and high-frequent control movement
as shown in Figure 50, using a break-frequency of 0.1 Hz. Due to this breakup in frequency, the overall
control movement of the subject is split from the short term control corrections of the subject. The highfrequent movements are an indication of the control effort of the subject to control the process.








Figure 50: Decomposition of low and high-frequent control movement of joystick angles for subject nr.6 using
haptic shared control (DC-SC) condition during performance task 1.
The time trace results show a large variation of control of the process, also within the same task as shown in
Figure 48. It is therefore very difficult to draw conclusions from looking only at this time trace data. A
statistical representation of the data is more effective to display an overall subtle trend in data results.
4.1.2 Self-Assessment Results
The statistical results of all six questions from the TLX assessment are shown in Figure 51. Almost all
questions indicate a reduction of effort compared to the baseline condition. The haptic shared control (SC)
conditions seem to decrease the effort more than natural force feedback (FF). Due to the small sample size
of ten subjects these reductions are not always significant.
Figure 51: Self-assessment of all six workload metrics, shown for all four experimental conditions in each graph.
A two-way ANOVA is performed on all results to state the statistical significance for equal mean corrected for
in between subject differences, given in Table 12. Not all results have statistical significant (p≤0.05) different
means.
15-02-2012
48/71
EXPERIMENT RESULTS
Condition
DC
:
DC
:
DC
:
DC-SC :
FF
:
DC-SC
FF
FF-SC
FF-SC
FF-SC
MD
PD
•
-
TD
-
PE
•
•
-
EF
•
-
FR
••
-
•
•
Table 12: Two-way ANOVA results for difference in means of all six self-assessment questions, using (••) for
p≤0.01, (•) for p≤0.05 and (-) for p>0.05 significance level.
The overall workload is the combination of all six questions with even weighing. Each subject was also asked
to give a rating of the entire experience of the experimental condition. These statistical results are shown in
Figure 52. The results indicate an increase in overall rating for all conditions compared to the baseline
condition (DC) and a decrease in workload. The difference in the mean of results are enlarged for both
conditions with haptic shared control applied (SC).
Figure 52: Overall workload and rating results,
The statistical significance level of the difference in means as shown in Figure 52 is listed in Table 13. The
results show for almost all conditions a statistical significant (p≤0.05) difference in means compared to the
baseline condition.
Condition
DC
:
DC
:
DC
:
DC-SC :
FF
:
DC-SC
FF
FF-SC
FF-SC
FF-SC
Overall
Rating
Overall
Workload
••
-
••
•
••
••
-
-
••
••
Table 13: Two-way ANOVA results of overall self-assessment, using (••) for p≤0.01, (•) for p≤0.05 and (-) for
p>0.05 significance level.
49/71
R.J. Kuiper
Biomechanical part of thesis
GRABBING ROCK IN DEEP SEA MINING
4.2 Performance Trials
The six tasks to be executed for each condition consist of three performance tasks and three catch trials as
described in Table 5 of paragraph 2.1. During the three performance tasks per condition, the subject was
operating the system under normal conditions. Results of these tasks are given in the time and statistical
domain to analyse the difference of feedback mechanisms. Paragraph 4.2.1 shows the results for the
performance and system evaluation metrics, paragraph 4.2.2 the results of the control effort metrics. The
evaluation metrics of the self-assessment is only given in paragraph 4.1.2 for all six tasks combined as
general results, due to the subjective nature of the questions.
4.2.1 Performance and system results
This paragraph shows the statistical performance and system results of the performance trials as described
in the list of evaluation metrics in Table 7 in paragraph 2.2.2.
Performance Results
The statistical results for production time, production volume and resulting production of all three
performance trials are shown in Figure 53. The results only show a slight increase of time to complete for
both feedback mechanisms FF-SC and therefore show a slight decrease of production results as well.
Figure 53: Production results absolute and normalized with the control condition median
The statistical significance level of the difference in means of the results as shown in Figure 53, is given in
Table 14. The results show hardly any significant difference in means compared to the baseline condition DC.
Condition
DC
:
DC
:
DC
:
DC-SC :
FF
:
DC-SC
FF
FF-SC
FF-SC
FF-SC
Time
Vol
Prod
-
-
-
-
-
-
•
-
-
•
-
-
•
-
-
Table 14: Two-way ANOVA results of performance data, using (••) for p≤0.01, (•) for p≤0.05 and (-) for p>0.05
significance level.
15-02-2012
50/71
EXPERIMENT RESULTS
System Data Results
The heeling angle of the grab’s frame is an indication of the effective level of control of the operator. If the
cable force in combination with the rock cutting force at critical moments is not correct controlled, the
heeling angle will increase. A warning is given to the operator when exceeding a minimum level of of heeling
absolute and a maximum angle for causing an error, as described in 2.1.3. The attention of the operator can
be measured by the magnitude of slack of the cable. The magnitude of deviation from the given optimal
cable force setpoint is an indication of the correct response of the operator for controlling the process. These
three described variables are shown in Figure 54. The maximum heeling angle of the grab's frame increases
slightly when haptic feedback is applied, but the spread in data is very large compared to the differences.
The maximum slack of the cable has a large spread as well and a very small difference in means. The
deviation of cable force from the controller setpoint is almost the same for all conditions combined as well
with a large spread in data.
Figure 54: Results for maximum heeling angle of frame, maximum slack of cable and average deviation of cable
force from setpoint. The orange dotted line in the frame angle results shows the warning level of heeling.
The levels of statistical significance of the results as given in Figure 54 are listed in Table 15. Only the frame
angle shows statistical difference in means, however it indicates an increase in mean.
Condition
DC
:
DC
:
DC
:
DC-SC :
FF
:
DC-SC
FF
FF-SC
FF-SC
FF-SC
Frame
Angle
Diff
Lcable
Diff
Fcable
••
-
-
••
-
-
-
-
-
-
-
-
-
-
-
Table 15: Two-way ANOVA results of system data, using (••) for p≤0.01, (•) for p≤0.05 and (-) for p>0.05
significance level.
4.2.2 Control Effort Results
This paragraph described the total summed joystick angle during a conducted task and high-frequent moving
angle, indicating the control effort of the subject.
51/71
R.J. Kuiper
Biomechanical part of thesis
GRABBING ROCK IN DEEP SEA MINING
Sum Angle Results
The sum angle is the average joystick rotation angle during an entire task, indicating the amount of joystick
usage per condition. The statistical results are shown in Figure 55 for each individual joystick and an average
of both joystick angles, shown for all three performance tasks for every experimental condition. The results
indicate a large reduction of joystick usage for the grab joystick, when applying haptic feedback. The use of
the grab joystick especially reduces when applying haptic shared control (SC). The cause of this reduction is
the warning vibration on the grab joystick handle, when the clamshells are closed and a closing force is
unnecessary. This effect is shown with a time trace for a subject with and without shared control in Figure
49 in paragraph 4.1.1. The operator holds on to the grab joystick during the entire task without feedback.
With shared control the warning prevents the operator for unnecessary action with the joystick.
Figure 55: Average of rotation angle of joystick during entire task of the grab, winch and both joysticks average
angles for all four conditions.
The level of significance is listed in Table 16, for the difference in means of the summed joystick angles as
shown in Figure 55. The grabbing joystick angle and total angle show a statistical significant difference in
means compared to the baseline condition DC.
Condition
DC
:
DC
:
DC
:
DC-SC :
FF
:
DC-SC
FF
FF-SC
FF-SC
FF-SC
Grab
Sum Ang
Winch
Sum Ang
Total
Sum Ang
••
-
••
•
-
-
••
-
••
-
-
-
•
-
••
Table 16: Two-way ANOVA results of total summed joystick angle, using (••) for p≤0.01, (•) for p≤0.05 and (-)
for p>0.05 significance level.
Move Angle Results
Another indication for the operator’s effort is the movement of the rotation angle and therefore the average
of the high frequent rotation. The time trace results in Figure 50 in paragraph 4.1.1 show the decomposition
of both joystick angles in a low and high frequent part, with a 0.1 Hz distinction. The statistical results for
the average of this high frequent movement of the joystick angles are shown in Figure 56, for both joysticks
individual and combined for all four conditions. The results do not show a noticeable difference for changing
conditions, except when both feedback mechanisms are combined (FF-SC).
15-02-2012
52/71
EXPERIMENT RESULTS
Figure 56: Statistical results of average high-frequent joystick movement of the grab, winch and both joystick
angles for all four conditions.
Hardly any statistical level of significance is calculated for the results as given in Figure 56, which is shown in
Table 17.
Condition
DC
:
DC
:
DC
:
DC-SC :
FF
:
DC-SC
FF
FF-SC
FF-SC
FF-SC
Grab
Mov
Angle
Winch
Mov
Angle
Total
Mov
Angle
-
-
-
-
-
-
-
•
-
-
-
-
-
-
Table 17: Two-way ANOVA results of high-frequent moving joystick angle, using (••) for p≤0.01, (•) for p≤0.05
and (-) for p>0.05 significance level.
Applied Force Results
The statistical results of the applied forces on the joysticks are shown in Figure 57, as an objective indication
of the physical load of the subject. The results clearly indicate the differences of applied forces for the
feedback mechanisms. During the baseline condition (DC) hardly any force was applied besides a static small
spring load as a return force to the initial position. The haptic shared controller (SC) shows an increased
feedback force level, as explained in paragraph 3.3.3 and shown in Figure 36 a) and b).
53/71
R.J. Kuiper
Biomechanical part of thesis
GRABBING ROCK IN DEEP SEA MINING
Figure 57: Statistical results of applied joystick force
The levels of significance are given in Table 18 for the applied joystick force as shown in Figure 57, all
statistical significant different compared to the baseline condition DC where hardly any joystick force was
applied.
Condition
DC
:
DC
:
DC
:
DC-SC :
FF
:
DC-SC
FF
FF-SC
FF-SC
FF-SC
Grab
Fapplied
Winch
Fapplied
Total
Fapplied
••
••
••
••
••
••
••
••
••
-
-
-
••
••
••
Table 18: Two-way ANOVA results of applied joystick force, using (••) for p≤0.01, (•) for p≤0.05 and (-) for
p>0.05 significance level.
4.3 Catch Trials
Besides the three performance trials, three catch trials were conducted as well as described in paragraph
2.1.3 and listed in Table 5. The first catch trial was essentially more just a very difficult task to complete, due
to hard soil conditions causing fast changing high reaction forces to the grab, statistical results shown in
paragraph 4.3.1. However not all subjects completed the task due to the difficult conditions, resulting in zero
production volume. The second trial was to increase the attention level of the subject and not expected to
complete the task, due to very hard soil conditions exceeding the grab’s capabilities, shown in paragraph
4.3.2. The third catch trial was to determine the reaction time of the operator after a mechanical failure, only
visual noticeable. Only the statistical results of the reaction times are given in paragraph 4.3.3 of the last
catch trial.
4.3.1 Catch Trial of Critical Situation
For this catch trial only the statistical results for performance, system data and a reduced number of results
of control effort are given, as relevant information.
15-02-2012
54/71
EXPERIMENT RESULTS
Performance Results
The statistical production results during this critical situation are shown in Figure 58, for time to complete,
production volume and overall production. The results show a slight decrease in time to complete for the
haptic shared control conditions (SC) and an increase for the natural force feedback conditions (FF). The
large spread of the results is due to the small sample size and large variety in completion of task, due to
abortion of the tasks of several subjects.
Figure 58: Statistical results of performance data for the catch trial of a critical situation, for all four
experimental conditions.
Hardly any statistical level of significance is calculated for the results as given in Figure 58, which is listed in
Table 19. Only the production results for both feedback mechanisms (FF-SC) compared to the baseline
conditions (DC) show a statistical significant difference.
Condition
DC
:
DC
:
DC
:
DC-SC :
FF
:
DC-SC
FF
FF-SC
FF-SC
FF-SC
Time
Vol
Prod
-
-
-
-
-
-
-
-
•
•
-
-
-
-
-
Table 19: Two-way ANOVA results of performance data for the catch trial of a critical situation, using (••) for
p≤0.01, (•) for p≤0.05 and (-) for p>0.05 significance level.
System Data Results
The maximum heeling angle of the frame, slack of cable and deviation to cable force setpoint are shown in
Figure 59 for the critical situation catch trial. The results show a decrease of maximal heeling angle of the
grabs frame when natural force feedback (FF) is applied.
55/71
R.J. Kuiper
Biomechanical part of thesis
GRABBING ROCK IN DEEP SEA MINING
Figure 59: Maximum frame heeling angle, maximum cable slack and difference of cable force for the catch trial
of a critical situation, for all four experimental conditions.
Only the results of the maximal heeling angle of the grabs frame shows statistical significant difference for all
applied conditions compared to the baseline, as listed in Table 20. The results in Figure 59 for heeling angle
show clearly a large reduction when applying natural force feedback during this difficult task, indicating a
much improved situation awareness.
Condition
DC
:
DC
:
DC
:
DC-SC :
FF
:
DC-SC
FF
FF-SC
FF-SC
FF-SC
Frame
Angle
Diff
Lcable
Diff
Fcable
-
-
-
•
-
-
•
-
-
•
-
-
-
-
-
Table 20: Two-way ANOVA results of system data for the catch trial of a critical situation, using (••) for p≤0.01,
(•) for p≤0.05 and (-) for p>0.05 significance level.
Control Effort Results
Statistical results for the average joystick rotation angle and the high-frequent movement of the rotation
angle are shown in Figure 60. Both sum and moving angle do show a slight decrease of control effort
compared to the baseline condition. The results for the applied joystick forces are comparable to the
performance trials and therefore irrelevant and not shown for this task.
15-02-2012
56/71
EXPERIMENT RESULTS
Figure 60: Statistical results of the average total sum angle and moving angle of both joystick angles, for all
four experimental conditions.
None of the shown data in Figure 60 has a statistical significant different mean compared to the baseline
condition (DC).
4.3.2 Catch Trial of Hard Soil
During this catch trial, very hard soil conditions were applied for the virtual slave robot simulation, exceeding
the grab’s capabilities. A reduced number of performance, system data and control effort results are given,
due to some irrelevant metrics for this catch trial.
Performance and System Results
The results of time to complete the task and maximum heeling angle of the grabs frame are given in Figure
61. The evaluation metric production is not shown for this catch trial, because no gained volume was
possible with these soil conditions. The results do not show a clear indication of changed reactions for this
trial using different feedback mechanisms.
57/71
R.J. Kuiper
Biomechanical part of thesis
GRABBING ROCK IN DEEP SEA MINING
Figure 61: Statistical results for the time to complete the task and maximal heeling angle of the grabs frame, for
all four experimental conditions.
None of the shown data in Figure 61 has a statistical significant different mean compared to the baseline
condition (DC).
Control Effort Results
Statistical results of the control effort of the joystick angles for the catch trial with hard soil are shown Figure
62, which is similar to Figure 60 of the catch trial of a critical situation.
Figure 62: Statistical results of the average total sum angle and moving angle of both joystick angles, for all
four experimental conditions.
The difference of the results shown in Figure 62 to the critical situation is that most of these results do show
a statistical difference in reduction of control effort, as listed in Table 21.
15-02-2012
58/71
EXPERIMENT RESULTS
Condition
DC
:
DC
:
DC
:
DC-SC :
FF
:
DC-SC
FF
FF-SC
FF-SC
FF-SC
Total
Sum
Angle
Total
Mov
Angle
•
-
•
•
••
-
-
-
•
-
Table 21: Two-way ANOVA results of control effort data for the catch trial of hard soil, using (••) for p≤0.01, (•)
for p≤0.05 and (-) for p>0.05 significance level.
4.3.3 Catch Trial of Reaction Time
This catch trial consisted of a mechanical failure during operation of the task with normal soil conditions, as
described in paragraph 2.1.3. Statistical results of the reaction time to this failure are shown in Figure 63 for
all four experimental conditions. A slight decrease for natural force feedback mechanism (FF) is shown,
compared to the baseline condition (DC). The spread in data is very large because of the small sample size.
Figure 63: Statistical results of the reaction times to mechanical failure during the catch trial, shown for all four
experimental conditions.
None of the shown data in Figure 63 has a statistical significant different mean compared to the baseline
condition (DC).
59/71
R.J. Kuiper
Biomechanical part of thesis
GRABBING ROCK IN DEEP SEA MINING
15-02-2012
60/71
DISCUSSION
5 DISCUSSION
Controlling a deep sea mining process to excavate SMS deposits of the seafloor by teleoperation is a complex
procedure and has not been done so far. A promising way for excavation is the use of a large hydraulic grab,
suspended from a ship as described in paragraph 1.1.1, using hydraulic actuators as described in paragraph
1.1.3. Controlling a suspended grab by teleoperation using a standard human-machine interface with only
visual feedback is difficult for the human operator (Es 2004 [10]). Literature describes two types of haptic
feedback mechanisms to improve the human-machine interface for controlling the deep sea mining
excavation process, natural force feedback as described in paragraph 1.2.1 and haptic shared control as
described in paragraph 1.2.2. Both methods have shown improvements for controlling a teleoperation for
position control (FF, Wildenbeest et al. 2010 [39]; SC, Abbink 2006 [1], Abbink et al. 2008 [2], Boessenkool
et al. 2011 [7]), however their usefulness is still unknown when applied on rate and force control when used
for a deep sea mining process (Zhu and Salcudean 1995 [41]).
In this thesis an experimental setup is developed to determine the effect of haptic feedback mechanisms on
the performance and effort of controlling a deep sea mining operation. The experiment consists of a human
operator conducting a virtual deep sea mining task using two haptic joysticks and a display as shown in
Figure 20 in chapter 2. The haptic joysticks of the experimental setup applied force feedback to the human
operator with additional information to control the operation. The design of the haptic joysticks however
appeared to have a relative short stroke for controlling the process compared to industrial joysticks,
sometimes inducing non-continuous control. The relative short design of the joysticks was caused by the
limiting motor torque of the given electric motors in combination with the given cable spring design. The
information details of force feedback were therefore also more difficult to be noticed correctly due to the low
feedback forces and short sticks. The conducted experiment consisted of six different tasks for every
experimental condition. An extra set of experiments with one subject were conducted to determine if
variation in outcome would be reduced when constantly conducting a single identical task instead of six
different tasks. However the results had an unchanged outcome for conducting only identical tasks,
indicating the task in general to be difficult to conduct due to the absence of a clear optimal control path.
Two types of tasks were conducted during the experiment to determine the effect of the haptic feedback
conditions; performance tasks to determine the influence on the production and control effort, and catch trial
tasks to determine the influence on critical parameters and reaction times. The general production results of
the performance tasks as shown in Figure 46 and showed no difference for changing haptic feedback
conditions. The reduction of the number of time penalties for incorrect control resulting in a critical control
error, as shown in Figure 47, indicates an increase in situation awareness when offering natural force
feedback. The time traces results as shown in Figure 48 and in paragraph 4.1.1, indicate the unclear optimal
control path which occurred due to a lots of variation of control inputs for operating identical tasks under
different conditions. These results also display the difficulty of operating the process with the small stroke
joysticks, resulting in bang-bang control, as shown in Figure 48 a). The subjective results of the subjects’
self-assessment showed a clear reduction in workload and increase in overall rating when applying haptic
feedback mechanisms as shown in Figure 52. This indicates improvement of controlling the process even
when production results did not show any change.
The production results of the performance trials as defined as time to complete and production volume, as
shown in Figure 53 show hardly any statistical difference for varying conditions. However the control effort
as shown in Figure 55 and Figure 56 for joystick angles does indicate a reduction when using haptic shared
control conditions. The results of the catch trial for conducting a task in a critical situation displayed in Figure
59, showed reduction of the maximal heeling angle of the grab for natural force feedback conditions. This
indicates an improvement of the situation awareness for applying transparency in the teleoperation. The
reduction in control effort when applying haptic shared control for the critical situation catch trial as shown in
61/71
R.J. Kuiper
Biomechanical part of thesis
GRABBING ROCK IN DEEP SEA MINING
Figure 60, is much less in comparison with the performance trials. On the other hand the catch trial for hard
soil conditions as shown in Figure 62 did indicate a reduction of control effort when applying haptic shared
control. However the catch trial of hard soil as is displayed in Figure 61 did not show a reduction of time to
complete, which would be a reasonable explanation for a reduction of control effort. The reaction time
during the mechanical failure catch trial displayed in Figure 63 showed a slight improvement for natural force
feedback, but which was not significant.
The summarised results of improved performance are listed in Table 22 and are compared with the
hypothesis, based on production results during performance trials and catch trials. The effect as stated in the
hypothesis for increase of performance is not measured, only during the difficult catch trials more production
is measured when combining both haptic feedback mechanisms.
Performance
No NFF
NFF
HYP
PR-PT
PR-CT
HYP
PR-PT
0
0
0
0
0
0
+
+
0
+
0
+
No HSC
HSC
PR-CT
Table 22: Results summery of increase of performance for all conditions. In grey are stated the hypothesis and
green the results validating the hypothesis, red indicated the hypothesis partly rejected (Hypothesis / Results).
Results are given for the hypothesis (HYP), production during performance trials (PR-PT) and production during
catch trials (PR-CT)
The summarized results of the reduced control effort are listed in Table 23 and are based on total summed
control angle and high frequency movement of the joystick angle during performance trials. The results give
an indication of the verification of the hypotheses when reduced control effort when offering haptic shared
control.
Control
Effort
No NFF
NFF
HYP
SUM
HF
WL
HYP
SUM
HF
WL
0
0
0
0
0
0
0
+
+
0
0
+
+
+
+
+
No HSC
HSC
Table 23: Results summery of reduction of control effort for all conditions during performance trials, given for
the hypothesis (HYP), joystick summation angle (SUM), high frequency joystick angles (HF) and subjective
workload (WL).
The summarised results of the improved situation awareness listed in Table 24, based on heeling angle at
difficult conditions and reaction times to mechanical failures during catch trials. The results indicate the
verification of the hypothesis for increased situation awareness for offering natural force feedback.
Situation
Awareness
No NFF
HYP
ANG
RT
NFF
HYP
ANG
RT
No HSC
HSC
0
0
0
0
0
+
+
0
+
+
+
0
Table 24: Results summery of increase of situation awareness for all conditions during catch trials, given for the
hypothesis (HYP), frame heeling angle (ANG) and reaction time to mechanical failure (RT).
In conclusion the experimental results indicate that haptic shared control does not seem to increase the
performance of an operation. However it does reduce the control effort, resulting in a less demanding task
for the operator and therefore the task is easier to complete. On the other hand natural force feedback
showed improvements in situation awareness, resulting in less control errors during operation. This implies
that human operators pay more attention to the process due to the short term increased situation awareness
and long term less demanding tasks when both natural force feedback and haptic shared control are applied,
which results in less errors causing production loss or damaging the system.
15-02-2012
62/71
CONCLUSION
6 CONCLUSION
Derived from the experimental results it can be concluded that controlling a deep sea mining process can be
improved by offering haptic feedback to the operator, thereby informing the environmental forces by offering
natural force feedback and guiding the operator to an optimal control of the process by offering haptic
shared control with attractive guiding forces.
The experimental results during normal task completion indicate an increase in situation awareness of the
subject when natural force feedback was applied. This can be seen in a reduction of control errors resulting
in a reduction of a critical heeling angle of the grab. Also the reaction time during a mechanical failure and
the given time penalties for incorrect control were reduced when natural force feedback was applied.
However the results of the experiment during a task at normal conditions did not show an increase in
performance when applying haptic shared control. Also the results for time to complete the task or the
gained production results did not show improvements. On the other hand the control effort did decrease for
controlling a normal task and for conducting a difficult task. The subjective overall workload and overall
rating results showed improvements for both feedback mechanisms.
To summarize the conclusions of this thesis the following statements can be made.
 Natural force feedback showed an increase in situation awareness, causing a reduction of critical
heeling angle of the grab. It also showed a reduction of the reaction time during mechanical failure
and a reduction of incorrect control resulting in critical situations.
 Haptic shared control showed no increase in performance at normal conditions, but did show a
reduction of control effort at normal and difficult conditions.
 Combining natural force feedback and haptic shared control showed reduction of subjective workload
and rating.
Therefore for this experiment it can be concluded that when controlling a deep sea mining tele-operation
using rate and force control, offering natural force feedback improves the situation awareness and haptic
shared control reduces the control effort of the human operator. However improvements of the operation
performance were not found for either haptic feedback mechanisms. Nonetheless this implies that less
production loss will occur due to incorrect control and damage of the system when combining the haptic
feedback mechanisms for a deep sea mining tele-operation.
63/71
R.J. Kuiper
Biomechanical part of thesis
GRABBING ROCK IN DEEP SEA MINING
15-02-2012
64/71
RECOMMENDATIONS
7 RECOMMENDATIONS
The described experiments did not completely give clear results for all situations, and only partly validated
the hypotheses. Limitations of the experimental setup are partly the cause of these outcomes, therefore a
list of recommendations is given for further experiments.

It is recommended to use a different mechanical principle of actuating the joysticks as shown in Figure
64 a), thereby enabling high dynamic actuation and removing the limitations due to springs. By direct
coupling of motor and joystick without springs it will enable higher actuation forces, not limited by the
springs and removing backlash of the gearbox.

Another suggestion would be to use longer handles of the mechanical setup as shown in Figure 64 b),
for more control and sensitivity of forces. The downside of this is a larger required motor force, due to
the increase momentum.
Handle
Stroke
Handle
+
Emotor
+
+
Figure 64: a) Schematic view of recommendation for improved design for mechanical principle of the
experimental setup. b) Schematic view for improved dimensions of the experimental setup

65/71
It is also recommended to use one ground model for all tasks to improve learning and to increase
distinction between feedback methods, despite the fact that a quick look of this method did not show
any mayor differences as previously described.
R.J. Kuiper
Biomechanical part of thesis
GRABBING ROCK IN DEEP SEA MINING
15-02-2012
66/71
ABBREVIATIONS
ABBREVIATIONS
ANOVA
DoF
DC
DC-SC
DSS
DUT
FF
FF-SC
FRVF
MMI
SC
SMS
SVI
TCP
TLX
VF
UIC
NOAA
HMI
EC
MD
PD
TD
PE
EF
FR
HYP
PR-PT
PR-CT
SUM
HF
WL
ANG
RT
P1-3
C1-3
:
:
:
:
:
:
:
:
:
:
:
:
:
:
:
:
:
:
:
:
:
:
:
:
:
:
:
:
:
:
:
:
:
:
:
:
ANalysis Of VAriance
Degree of Freedom
Direct Control
Direct Control using haptic Shared Control feedback
Driver Support System
Delft, University of Technology
natural Force Feedback
both natural Force Feedback with haptic Shared Control feedback
Forbidden-Region Virtual Fixtures
Man-Machine Interface
Shared Control feedback mechanism
Seafloor Massive Sulphides
Standard Variable Interface
Transmission Control Protocol
Task Load Index
Virtual Fixtures
University of Illinois at Chicago
National Oceanic and Atmospheric Administration
Human Machine Interface
Experimental Condition
Mental Demand
Physical Demand
Temporal Demand
PErformance
EFfort
Frustration level
Hypotheses
PRoduction during Performance Trials
PRoduction during Catch Trials
SUMmation angle of joystick control
High Frequency
subjective Work Load
machine heeling ANGle
Reaction Time
Performance trials number 1 to 3
Catch trails number 1 to 3
NOMENCLATURE
Avalve
Apipe
B
Bfluid
67/71
:
:
:
:
Hydraulic valve area
Hydraulic pipe area
Bulk modulus
Hydraulic fluid bulk modulus
R.J. Kuiper
Biomechanical part of thesis
GRABBING ROCK IN DEEP SEA MINING
ch, cv
dθ
dL, dLcable
F
FI
FH, FV
FG
Fw
Fvol
Fmax
Fl, Fr
Fmin
Frequired
Fend
Fwinch
Fset
kh, kv
P0
Qcyl
Qvol
Qpump
Uvol
V0
Vfluid
Vvol
X
Y
Z
α
∆P
∆V
θgrab
θwinch
θSC,winch
θ
θ
θframe
θLF,grab
θHF,grab
θwinch
θLF,winch
θHF,winch
θvibrate
θequalibrium
φ
ψ
:
:
:
:
:
:
:
:
:
:
:
:
:
:
:
:
:
:
:
:
:
:
:
:
:
:
:
:
:
:
:
:
:
:
:
:
:
:
:
:
:
:
:
:
:
:
Horizontal and vertical damping coefficient
Change of equilibrium angle
Differential cable length
Force
Impact force
Horizontal and vertical force component
Resulting grab from clamshells
Resulting winch force
Force of hydraulic actuator
Maximum force
Left and right cutting force
Minimal feedback force
Required feedback force?
End feedback force
Winch force
Force setpoint
Horizontal and vertical spring coefficient
Initial hydraulic pressure
Hydraulic cylinder inflow
Hydraulic inflow
Hydraulic pump flow
Hydraulic fluid velocity
Hydraulic initial volume
Hydraulic fluid volume
Hydraulic volume
Sideways movement of grab
Forward movement of grab
Vertical movement of grab
Clamshell angle relative to grab
Hydraulic differential pressure
Hydraulic differential volume
Grab joystick angle
Winch joystick angle
Shared control angle for the winch joystick
Angle
Pitch angle of grab
Frame heeling angle
Low frequent grab joystick control angle
High frequent grab joystick control angle
Winch joystick angle
Low frequent winch joystick control angle
Low frequent winch joystick control angle
Joystick vibration angle
Equilibrium joystick angle
Heeling angle of grab
Heading angle of grab
15-02-2012
68/71
REFERENCES
REFERENCES
[1]
David A. Abbink. Neuromuscular analysis of haptic gas pedal feedback during car following. Ph.D.
Thesis TU Delft, 2006.
[2]
David A. Abbink, Erwin R. Boer, and Mark Mulder. Motivation for continuous haptic gas pedal feedback
to support car following. IV’08 Conference, 2008.
[3]
J.J. Abbott and A.M. Okamura. Virtual fixture architectures for telemanipulation. IEEE International
Conference on Robotics and Automation, 2003
[4]
Berk, L.J.M. van den, van den Berk Consultancy; Recources of the Sea; 2009 van den Berk
Consultancy. 2009.
[5]
Bettini, P. Marayong, S. Lang, A.M. Okamura, and G. Hager. Vision assisted control for manipulation
using virtual fixtures. IEEE Transactions on Robotics, 2004.
[6]
J.W. van Bloois, IHC Merwede, J.C.L. Frumau, Seatools bv.; Deep Sea Mining, The New Horizon for
Dredging Technology; 2009, Offshore Technology Conference. 2009.
[7]
H. Boessenkool, D. Abbink, C. Heemskerk, F.C.T. van der Helm. Haptic shared control improves teleoperated task performance towards performance in direct control. World Haptics Conference (WHC),
IEEE, pp. 433-438. 2011
[8]
G. A. V. Christiansson, R. Q. van der Linde, and F. C. T. van der Helm. The influence of teleoperator
stiffness and damping on object discrimination. IEEE TRANSACTIONS ON ROBOTICS, VOL. 24, NO. 5,
2008.
[9]
M.R. Endsley. Toward a Theory of Situation Awareness in Dynamic Systems. Human Factors, Vol. 37,
No. 1, pp. 32-64. 1995
[10] B. van Es, N. Pille, R. de Vlaming, and J. de Vries. Construction of wellhead protection glory holes for
white rose project, Canada. Terra et Aqua, No. 95, pp. 22-32. 2004.
[11] H.F. George, and A. Barber. What is bulk modulus and when is it important. Hydraulics and
Pneumatics, Lubrizol Corp., Penton Media, Inc., System Design pp. 34-39. 2007
[12] Weston B. Griffin, William R. Provancher, and Mark R. Cutkosky. Feedback strategies for shared
control in dexterous telemanipulation. Intelligent Robots and Systems Conference, 2003.
[13] B. Hannaford. A Design Framework for Teleoperators with Kinesthetic Feedback. IEEE Transactions on
Robotics and Automation, Vol. 5, No. 4, pp. 426-434. 1989.
[14] B. Hannaford, Stability and Performance Tradeoffs in Bi-Lateral Telemanupulation. Proceedings, 1989
International Conference on Robotics and Automation, pp. 1764-1767. 1989.
[15] B. Hannaford, L. Wood, D.A. McAffee, and H. Zak. Performance evaluation of a six-axis generalized
force-reflecting teleoperator. IEEE Trans. Syst., Man, Cybernetics. 1991.
69/71
R.J. Kuiper
Biomechanical part of thesis
GRABBING ROCK IN DEEP SEA MINING
[16] S.G. Hart, and L. E. Staveland. Development of NASA-TLX (Task Load Index): Results of empirical and
theoretical research. Human Mental Workload, 139-183 Elsevier Science. 1988.
[17] S. Hayati and S.T. Venkataraman. Design and implementation of a robot control system with traded
and shared control capability. IEEE International Conference on Robotics and Automation, p. 13101315, 1989.
[18] P. Hoagland, S. Beaulieu, M.A. Tivey, R.G. Eggert, C. German, L. Glowka, and J. Lin. Deep sea mining
of seafloor massive sulfides. Marine Policy, Elsevier Ltd., 2009.
[19] N. Hogan. Impedance Control: An Approach to Manipulation. IEEE, American Control Conference, pp.
304-313. 1984
[20] K. Kaneko, H. Tokashiki, K. Tanie, and K. Komoriya. Impedance shaping based on force feedback
bilateral control in macro-micro teleoperation system. IEEE International Conference on Robotics and
Automation, 1997.
[21] D.A. Lawrence. Stability and transparency in bilateral teleoperation. IEEE Trans. on Robotics and
Automation vol 9. no 5. pp. 624-637, 1993.
[22] I. Lipton. Mineral resource estimate solwara 1 project bismarck sea papua new guinea. Golder
Associates Pty Ltd, 2008.
[23] R. McGill, J.W. Tukey, and W.A. Larsen. Variations of Box Plots. The American Statistician, February
1978, Vol. 32, No 1, pp. 12-16. 1978.
[24] P. Milgram, A. Rastogi, and J.J. Grodsk. Telerobotic control using augmented reality. IEEE
International Workshop on Robot and Human Communication, 1995.
[25] M. Mulder, D. A. Abbink, and E. R. Boer. The effect of haptic guidance on curve negotiation behavior
of young, experienced drivers. Conference Proceedings - IEEE International Conference on Systems,
Man and Cybernetics, pp. 804–809. 2008
[26] G. Niemeyer and J. E. Slotine. Stable adaptive teleoperation. IEEE Journal of Oceanic Engineering vol.
16, 1991.
[27] S. Petersen and M.D. Hanningron. Modern seafloor massive sulfide deposits: Distribution, types of
deposits and origin. Leibniz-Institute of Marine Sciences, Deep-Sea Minerals and Mining Conference
2008, 2008.
[28] R. Prada and S. Payandeh. On study of design and implementation of virtual fixtures. Virtual Reality,
13(2), 117-129, 2009.
[29] L.B. Rosenberg. The use of virtual fixtures as perceptual overlays. Technical Report, Armstrong
Laboratory, 1992.
[30] C.P. Sayers, and R.P. Paul. An operator interface for teleprogramming employing synthetic fixtures.
Presence Cambridge Mass, Vol. 3, Issue 4, pp. 309-320. 1994
15-02-2012
70/71
REFERENCES
[31] C.P. Sayers, R.P. Paul, L.L. Withcomb, and D.R. Yoerger. Teleprogramming for Subsea Teleoperation
Using Acoustic Communication. IEEE Journal of Oceanic Engineering, Vol 23, No. 1, pp. 60-71. 1998
[32] S.D. Scott. Lessons from land for modern seafloor volcanogenic massive sulfides. Scotiabank Marine
Geology Research Laboratory, Deep-Sea Minerals and Mining Conferenece 2008, 2008.
[33] T.B. Sheridan. Telerobotics. IFAC World Congress, Munich, 1989.
[34] M.K. Tivey. Generation of seafloor hydrothermal vent fluids and associated mineral deposits.
Oceanography 20, 1: pp. 50-65, 2007.
[35] R. Tomovic. On man-machine control. Automatica Vol 5. pp 401-404, 1969.
[36] G.E. Totten. Handbook of Hydraulic Fluids Technology. CRC Press, 1999
[37] S.A. Trebilcock. Development of world’s first seafloor massive sulphide mining system. Nautilus
Minerals Inc., Deep-Sea Minerals and Mining COnference 2008, 2008.
[38] M.L. Turner, R.P. Findley, W.B. Griffin, M.R. Cutkosky, and D.H. Gomez. Development and testing of a
telemanipulation system with arm and hand motion. ASME Dynamic Systems and Control Division. pp.
1-7, 2000.
[39] J.G.W. Wildenbeest, D.A. Abbink, C.J.M. Heemskerk, F.C.T. van der Helm. Improving the Quality of
Haptic Feedback Yields Only Marginal Improvements in Teleoperated Task Performance. Msc thesis,
Delft Universisty of Technology. 2010
[40] Y. Yokokohji, and T. Yoshikawa. Bilateral Control of Master-Slave Manipulators for Ideal Kinesthetic
Coupling. Proceedings of the IEEE Conference on Robotics and Automation, pp. 849-858. 1992.
[41] M. Zhu and S.E. Salcudean. Achieving transparency for tele-operator systems under position and rate
control. International Conference on Intelligent Robots and Systems , 1995.
71/71
R.J. Kuiper
Biomechanical part of thesis
Was this manual useful for you? yes no
Thank you for your participation!

* Your assessment is very important for improving the work of artificial intelligence, which forms the content of this project

Download PDF

advertisement