Multi-annual roadmap

Multi-annual roadmap
Robotics 2020
Multi-Annual Roadmap
For Robotics in Europe
Call 1 ICT23 – Horizon 2020
Initial Release B 15/01/2014
Rev B: TRL Examples updated; expanded roadmap structure section; table of contents added; text
formatting changes.
1. Introduction .............................................................................................................. 1 1.1 Reading the Roadmap .......................................................................................................................................... 2 1.2 Why read this document? .................................................................................................................................. 2 1.3 Roadmap Structure ............................................................................................................................................... 3 1.4 Background .............................................................................................................................................................. 5 1.4.1. Introduction to the PPP ............................................................................................................................. 5 1.4.2. Introduction to Horizon 2020 ................................................................................................................ 6 2. Markets and Applications ........................................................................................ 7 2.1.1. Application Domains .................................................................................................................................. 7 2.2 Manufacturing Domain ....................................................................................................................................... 9 2.2.1. Domain Overview ......................................................................................................................................... 9 2.2.2. Current and Future Opportunity ........................................................................................................... 9 2.2.3. Barriers to Market ....................................................................................................................................... 9 2.2.4. Key Market Data ........................................................................................................................................ 10 2.2.5. Relationship to other Domains and Markets ................................................................................ 10 2.2.6. Europe’s Place in the Market ................................................................................................................ 10 2.2.7. Key Stakeholders ....................................................................................................................................... 10 2.2.8. Current Key Projects ................................................................................................................................ 11 2.2.9. European Products ................................................................................................................................... 12 2.2.10. Production ................................................................................................................................................. 13 2.2.11. Food .............................................................................................................................................................. 13 2.2.12. SME Manufacturing ............................................................................................................................... 15 2.2.13. Soft Products ............................................................................................................................................ 16 2.2.14. Craft and Bespoke .................................................................................................................................. 16 2.2.15. Key System Ability Targets ................................................................................................................ 17 2.2.16. Key Technology Targets ...................................................................................................................... 22 2.2.17. Technology Combinations .................................................................................................................. 26 2.2.18. Product Visions ....................................................................................................................................... 28 2.3 Agriculture Domain ........................................................................................................................................... 29 2.3.2. Agriculture ................................................................................................................................................... 38 2.3.3. Forestry ......................................................................................................................................................... 38 2.3.4. Fisheries ........................................................................................................................................................ 39 2.3.5. Key System Ability Targets ................................................................................................................... 40 2.3.6. Key Technology Targets ......................................................................................................................... 44 2.3.7. Technology Combinations ..................................................................................................................... 47 2.4 Civil Domain .......................................................................................................................................................... 50 2.4.2. Civil Infrastructure ................................................................................................................................... 58 2.4.3. Search and Rescue .................................................................................................................................... 59 2.4.4. Environment ............................................................................................................................................... 60 2.4.5. Law Enforcement ...................................................................................................................................... 61 2.4.6. Emergency Services ................................................................................................................................. 61 2.4.7. Science Support .......................................................................................................................................... 62 2.4.8. Key System Ability Targets ................................................................................................................... 64 2.4.9. Key Technology Targets ......................................................................................................................... 67 2.5 Commercial Domain .......................................................................................................................................... 71 3. System Abilities ..................................................................................................... 74 3.1 Configurability ..................................................................................................................................................... 75 3.2 Adaptability ........................................................................................................................................................... 78 3.3 Interaction Capability ....................................................................................................................................... 81 3.4 Dependability ....................................................................................................................................................... 86 3.5 Motion Capability ................................................................................................................................................ 90 3.6 Manipulation Ability .......................................................................................................................................... 94 3.7 Perception Ability ............................................................................................................................................... 97 3.8 Decisional Autonomy ...................................................................................................................................... 104 3.9 Cognitive Ability ................................................................................................................................................ 107 3.10 MAR Technology Readiness Levels (TRLs) ......................................................................................... 117 4. Technologies ....................................................................................................... 120 4.2 Systems Development .................................................................................................................................... 121 4.3 Human Robot Interaction ............................................................................................................................. 130 4.4 Mechatronics ...................................................................................................................................................... 137 4.4.2. Mechanical Systems ............................................................................................................................... 140 4.4.3. Sensors ......................................................................................................................................................... 142 4.4.4. Actuators ..................................................................................................................................................... 145 4.4.5. Power Supply and Management ....................................................................................................... 147 4.4.6. Communications ...................................................................................................................................... 148 4.4.7. Materials ..................................................................................................................................................... 150 4.4.8. Control ......................................................................................................................................................... 154 4.5 Perception ............................................................................................................................................................ 159 4.5.2. Sensing ......................................................................................................................................................... 162 4.5.3. Interpretation ........................................................................................................................................... 164 4.6 Navigation ............................................................................................................................................................ 166 4.7 Cognition .............................................................................................................................................................. 171 1. Introduction
This Multi-Annual Roadmap (MAR) is a companion to the Strategic Research
Agenda (SRA) providing a greater level of technical and market detail.
It will be updated annually as priorities, technologies and strategic developments
shape European research development and innovation (R&D&I). The annual update
will follow a process that utilises the expertise within Topic Groups formed by
euRobotics aisbl and seeks open consultation.
The priorities for R&D&I funding, including near market activities, will be derived from
the MAR as a part of the annual review cycle.
Robotics is a diverse field and this roadmap relies on expert opinion in each domain
and technical field to provide and verify the information within it. The annual review
process will examine each key technical and market area to ensure material is
brought up to date at least once per annum. This document and its associated Wiki
provide a framework through which this additional information can be accessed.
You, the reader, are encouraged to engage with this process and to contribute your
knowledge to the content of this document. It will then reflect and sustain a live
discourse on the current state of robotics technology. You can do this by joining
euRobotics and then contributing to the Wiki and the associated Topic Groups.
1.1 Reading the Roadmap
Each person will read this document, and the Strategic Research Agenda, with a
different perspective. In creating this resource the aim has been to take these
different perspectives into account.
This document provides a detailed analysis of the development and application of
robotics technology. It reviews robotics technology, market place and the business
and academic infrastructure needed to create a viable robotics market in Europe.
The Roadmap provides a focal point of contact with the European Robotics
community allowing both experts and non-experts to to engage with the community
and understand and access information and resources.
1.2 Why read this document?
Do you work in an industry or service sector where you think
robotics technology can be applied?
Then you may wish to start by identifying your particular market sector and working
through the applications to uncover the types of robots and technologies that might
be applicable to your market.
Are you a researcher trying to understand the level of capability of
a particular robotics technology?
Then you may wish to start by examining the technology clusters to find the
technology you are interested in and then exploring the current and expected future
capability and its impact on applications. You may also be interested in the general
system abilities of robots to understand how the technology you are interested in
might impact on these abilities.
Are you a researcher who believes that they have a technology
that could be of use to the robotics community?
Then you may wish to start by looking at the technology clusters to see if your
technology can be fitted in. This may give you new ideas, or help you identify others
providing similar technology. It may also lead you to which application domains may
be the most likely to exploit your type of technology.
Are you are a policy maker trying to understand the European
robotics community?
Then you may need to read the Strategic Research Agenda to gain a background
understanding of robotics and its application. If you have already done this then you
may find the sections in this document on markets useful in order to understand
potential areas of application.
Are you involved in financing or managing startups and wish to
understand the opportunities in robotics?
Then you may want to look at the different market domains and see where you can
find opportunities, or may be you can identify a new area of application. You may
also wish to examine the different technology sectors to see where current
development is taking place or examine the current set of research priorities.
Are you a potential user of robotics technology and wish to
understand the general capability level of robots?
Then you should examine the Abilities section and gain an understanding of what
can be achieved with current technology and what might still lie in the future.
Similarly you should examine the market domains so understand how robots are
being applied in different industries and what the future might hold.
1.3 Roadmap Structure
The roadmap is divided into three main sections:
Market Domains (mentioned in Call 1 ICT23 - 2014)
System Abilities
Technology clusters
There is considerable cross linkage between the information in each of these
sections. Familiarity with the framework of description in the Strategic Research
Agenda will be critical to understanding the content of each section.
1.3.1. Market Domains
This section details each of the Market Domains referenced in Call 1 (ICT23 - 2014).
It provides background detail about each domain, and describes the currently
perceived opportunities in that domain. Each section also outlines the requirements
for technical capability and systems ability levels needed to impact on products.
In each section there is a brief overview of Europe’s place in the domain, the types of
stakeholder within Europe and a list of sub-domains and their opportunities. Key
barriers to deployment are also outlined.
This background information should be sufficient to identify how a proposal might
impact on a domain, or sub-domain, and indicate the dependence of a domain on
particular technologies or system abilities.
1.3.2. System Abilities
Each of the system abilities is described in detail without reference to particular
domains or applications. Abilities capture the generic characteristics of whole
Each ability is also defined in terms of levels, each domain or more likely each type
of application will have requirements for levels of ability.
Each ability level will in turn require an integration of multiple technologies to achieve
a level of ability. Each technology will need to reach a capability, in the context of the
task, in order to achieve a particular ability level.
Each ability section also lists key targets whereby achieving that target is expected
to have a significant impact on markets and applications. Given that the objective of
R&D&I investment is to raise these ability levels it is important to identify current
ability levels and to identify the ability levels targeted at completion.
1.3.3. Technology Clusters
The different robotics technologies are divided into clusters, as detailed in the
Strategic Research Agenda. In each technology section a brief overview of the
techniques and methods used is given and an outline of the scope of the technology.
It is inevitable that there will be some overlap at the boundaries between
The technology sections also detail step changes in technical capability that are
expected to have an impact on markets and applications together with a review of
benchmarks and metrics that may be able to indicate the current state of the art.
Reviewing this impact and targeting step changes with broad impact will be an
important part of achieving strategic objectives.
In many cases step changes in technology depend on progress in other technologies
and in turn these step changes impact on other technologies and system abilities.
There is also a brief indication of the impact on domains.
1.3.4. Targets and TRLs
In both the ability and technology sections targets are set. In the domain sections
requirements are given for ability levels and technology capability. The goal of the
closer to market funding in Horizon 2020 is to bring about step changes and a rise in
ability levels that will impact on the market by bridging the gap between requirement
and capability.
It is also important to measure the Technology Readiness Level of each technical
step or ability level within each application or domain. To implement a technology in
a product, by definition, requires that technology to be at TRL9. However raising a
technology to that level will first require the development of techniques to bring it to
TRL2, 3, 4, etc. It is also possible that this may only be achieved within a narrow
domain of application. Although is some cases step changes will be generic and
impact across many areas of application. In the context of assessing impact and the
value of research it is important to understand that Ability Levels, Technology Steps
and TRL levels all measure different aspects of progress.
1.4 Background
The companion to this document, the Strategic Research Agenda (SRA), provides a
high level strategic overview of the European robotics community and its objectives.
It also provides a descriptive framework for robotics its market, technology and robot
types. This framework of description underlies this roadmap.
The SRA also acts as an introduction to the European robotics community for nonrobotics specialists, policy makers, entrepreneurs and industries intending to use
robotics technology or work within the robotics market.
This document, the Multi-Annual Roadmap (MAR), is a more detailed technical guide
that identifies expected progress within the community and provides an analysis of
medium to long term research and innovation goals.
Through the actions of euRobotics aisbl with inputs from the Topic Groups set up by
euRobotics this document will be updated annually to maintain its relevance.
This document aims to provide the following:
Further details of the applications and markets outlined in the SRA.
Background and progress targets for the technologies outlined in the SRA.
Basic information about the Public Private Partnership (PPP) and the Horizon
2020 instruments.
An overview of potential impact on market domains of step changes in
technical and system ability.
An overview of applications and targets for progress in each area.
An overview of the contribution robotics technology can make to the European
societal challenges.
This Roadmap will be reviewed and updated during the lifetime of Horizon 2020 for
the following reasons:
To reflect changes in strategic priority within the European robotics
To reflect market and technology advances.
To reflect changes in European strategy and direction.
1.4.1. Introduction to the PPP
The robotics Public Private Partnership (PPP) between the European Commission,
the public side, and the robotics community, represented by euRobotics aisbl,
provides the mechanism for implementing strategy within Europe.
euRobotics aisbl provides a forum for the robotics community to develop strategy
and promote its needs both in terms of technology and resource to the Commission
so that funding might be directed to ensure a vibrant and successful industry
develops to the benefit of Europe and its citizens.
The European Commission and euRobotics aisbl members have joint responsibility
for setting and prioritising R&D&I goals. This change combined with the shift in
emphasis of Horizon 2020 closer to market led activities, and the establishment of
instruments to support this, is reflected in the strategic emphasis of the robotics
This document sets out an aligned roadmap covering technical and application
objectives. It represents a consensus viewpoint for the direction robotics must take in
the coming decade.
1.4.2. Introduction to Horizon 2020
Horizon 2020 is the eighth European Framework Program. It builds on the success
of the seventh Framework Program (FP7) while attaching greater importance to
innovation and wealth creation resulting from research. Robotics continues to
provide a strategic focus within Horizon 2020.
The shift in focus within Horizon 2020 closer to market is illustrated by the
introduction of new funding instruments that enable near to market collaborative
2. Markets and Applications
Robotics technology can be deployed in a wide range of different market domains.
Each domain has its own needs and requirements. These must be captured and
assessed in order to direct R&D&I funding where it will have the greatest impact.
The robotics market place is also complex involving a diverse range of opportunities.
Organisations may create value by concentrating on specific end applications,
supplying different types of robot, modules, sub-systems, tools, or providing services
within the market. It also includes dedicated supply chains, design services, and
research and development organisations. Providing a coherent categorisation of the
potential in each type of market is an important step in evaluating the potential for
robotics and robotics technology.
The SRA provides an overview of the major application domains and the Roadmap
provides a breakdown of the different areas of activity. This illustrates the opportunity
for innovation and provides a basis for identifying linkage between current and future
technology capability and market impact.
Within the Roadmap this market characterisation needs to be accessible to different
observers. Observers from outside of the robotics community need to be able to
understand the potential impact of robotics technology in their own market sector.
Observers from the robotics community need to understand their context within the
internal and external markets.
Each market domain will present barriers, both technical and non-technical.
Identification of these barriers will be the key to maximising the impact of R&D&I
In order to fully develop a viable market in Europe each possible domain where
robotics technology can be applied must be fully explored so that new markets are
not left undiscovered.
2.1.1. Application Domains
Markets can be presented as a series of individual market domains clustered under
a set of high level categories. Each high level category representing a similar type of
market opportunity.
These clusters are based on a number of common characteristics which broadly
apply to a class of market domains.
These characteristics are:
The business model used to deliver and deploy robotics within the specific
The types of end user
The broad legal infrastructure that applies to the domain.
Based on these characteristics the high level market domains are:
Transport and Logistics
Under each of these categories are a collection of individual sub-domains that
characterise the activity within each domain.
{Note: This release of the MAR is designed to cover Call 1 (ICT23) of Horizon 2020
and as such only elaborates the domains detailed in the first call}
2.2 Manufacturing Domain
2.2.1. Domain Overview
Robot technology has become the backbone of may large scale manufacturing
industries. In order to compete globally manufacturing must be both competitive and
agile. Robots are the key drivers of flexibility and competitiveness and will be
instrumental in bringing manufacturing back to Europe.
As the pressure to automate moves beyond the traditional manufacturing industries
such as automotive and electronics, the need for flexibility in these automation
systems grows, particularly for SME manufacturers. Meeting these needs will require
new technologies and new working practices.
As Europe strives to increase the value added by manufacturing (back to 20% of
EU’s GDP by 2020) it will be competing not just with low-wage economies, but also
highly automated economies. Leadership in robotics will be a key differentiator in
driving up the productivity of Europe’s manufacturing base.
2.2.2. Current and Future Opportunity
The current market for robotics technology in manufacturing is concentrated on large
scale manufacturing industries that have high levels of automation. However it is
widely recognised that the impact of robotics technology on manufacturing must
widen its base to address a broader range of manufacturing. For example by
addressing SME manufacturing, systems able to handle soft materials and millimetre
scale assembly operations amongst others.
New automation concepts such as Human Robot Collaboration (HRC) and CyberPhysical Systems (CPS) are recognised as having the potential to impact and
revolutionise the production landscape. Increasing the flexibility of industrial robots
and providing automation systems that provide faster more intuitive configuration are
important goals for future production systems.
Robotics technology will impact on these areas in the medium term;
lean and agile manufacturing,
miniaturised assembly,
introduction of Cyber-physical production systems (CPS) for example the
“Industrie 4.0” programme in Germany,
introduction of intuitive and adaptive manufacturing systems including intuitive
programming and tasking,
deployment of Dual-arm, lightweight, low-cost compliant manipulators,
increased cooperation with humans including physical cooperation,
novel business models and deployment strategies.
2.2.3. Barriers to Market
The application of robotics technology to manufacturing is a dynamically developing
domain. For European manufacturing industry to thrive amongst global competitors,
it is necessary to overcome various barriers to growth:
User awareness of robotics technology capabilities
User concerns about system complexity
Cost of ownership and return on investment
Flexibility and adaptation of systems to changing needs.
2.2.4. Key Market Data
The annual World Robotics Report of the International Federation of Robotics (IFR)
provides a comprehensive overview of the robotics business worldwide, showing
breakdowns in geographical regions as well as in application areas.
2.2.5. Relationship to other Domains and Markets
Within a European context there are strong connections and synergies with the
“Factories of the Future” PPP and EFFRA, the “European Factories of the Future
Research Association” (
Within the market domains defined in the SRA Manufacturing will impact on the
production of goods in all other domains. However the strongest linkages are with
the Robot Markets and in particular the market for robot arms and the markets for
Systems Development tools. With the advent of smart manufacturing robots these
linkages will expand to encompass user interface systems and wide area sensing.
2.2.6. Europe’s Place in the Market
Europe presently has a leading role in industrial robotics, supplying the world
market,1 but this position is vulnerable. Aside from well-established Japanese
suppliers, new companies are entering the European market.
The typical business model of the established suppliers of industrial robots is to work
closely together with system integrators. In this way, the suppliers concentrate on
the technology of the robot manipulator and controller and the application-related
know-how resides mostly with smaller companies doing the integration work.
This method of doing business works well across many market domains, ranging
from food & beverage to automotive. Future markets may need to review and adapt
this way of working to accommodate new boundary conditions. Examples could
include application rental agreements, pay-on-production, equipment leasing
arrangements, etc.
Finally, the larger of the equipment manufacturers are actually “global players”,
supplying not only the European markets, but also markets abroad.
2.2.7. Key Stakeholders
There are a significant number of European based companies that have a global
reach in the manufacturing sector. In addition there are significant end users of large
scale manufacturing systems within Europe. Europe also has a high proportion of
SME manufacturer end users and there is an open market within Europe to exploit
these strengths. In addition to the robotics suppliers there is also a well proven
network of service companies that install and configure systems.
The strong market for manufacturing and for robotics technology has been supported
by outstanding research and academic organisations distributed throughout Europe.
There is a strong research base and extensive opportunity for technology transfer.
See web site, visited 2013-0909.
This is a well established market with a well defined structure, however there will
need to be awareness of the disruptive nature of new technology in smart
manufacturing such that market shares can be maintained over time.
2.2.8. Current Key Projects
The following projects funded under FP7 have the potential to impact on this domain.
Flexible Skill Acquisition and Intuitive Robot Tasking for
Mobile Manipulation in the Real World
Highly Customisable and Flexible Packaging Station for
Mid-to-Upper Sized Electronic Consumer Goods Using
Industrial Robots
Knowledge, Awareness and Prediction of Man, Machine,
Material, and Method in Manufacturing
Smart robotics for high added value footwear industry
Plug-and-Produce Components and Methods for Adaptive
Control of Industrial Robots Enabling Cost Effective, High
Precision Manufacturing in Factories of the Future
New Machine Functionalities Through Process Dynamic
Stability Control
Advanced Intelligent Machine Adaptive Control System
Hierarchical and Adaptive Smart Components for Precision
Production Systems Application
The Toolkit for Building Low Cost Robot Co-Workers in
Assembly Lines
Plug and Produce Joint Interface Modules
Efficient and Precise 3D Integration of Heterogeneous
Microsystems from Fabrication to Assembly
AUTOnomous co-operative machines for highly
RECONfigurable assembly operations of the future
The Productive Robot Apprentice
Autonomous Robotic System for Thermo-Graphic
Detection of Cracks
Miniaturised Robotic systems for holistic in-situ Repair and
maintenance works in restrained and hazardous
Mobile Robots for Inspection and Maintenance Activities in
Extensive Industrial Plants
Parallel Cable Robotics for Improving Maintenance and
Logistics of Large-Scale Products
Plug&Play robots for smart factories
Development of flexible, sustainable and automated
platform for high accuracy manufacturing operations in
medium and large complex components using spherical
robot and laser tracker on overhead crane
2.2.9. European Products
The maturity of this market and the strength of European companies in the global
market mean that there are a significant number of products designed and produced
in Europe. These products are being augmented by smaller lighter more compact
manufacturing solutions suited to SME manufacture.
Manufacturing Sub-Domains:
2.2.10. Production Sub-Domain Overview
Mass production systems in the aerospace, automotive, electronics and domestic
appliance sectors have been a cornerstone of the robotics market for several
decades. This industrial robotics sector is an important and major source for revenue
and investment. The market is mature and well understood. Sales are mainly to
larger manufacturing operations and most often represent repeat orders for faster,
better more efficient assembly robots. Current Opportunity
The push to increase employment and increase competitiveness will open the
market for increased automation. European companies already operate in a global
market and maintaining their current market share will require R&D&I investment. Future Opportunity
It is widely acknowledged that this sector will expand through the integration of
service robotic technologies and through the deployment of robots into novel areas
of manufacturing, into SME manufacturing and into areas of manufacturing that
require more complex materials handling such as the food industry. Key Market Data
The IFR report on World Robotics provides an overview of the key market sectors
that use robots in production. The main markets are:
Electronics assembly
Automotive parts manufacture and automotive assembly
General production of metal, rubber or plastic parts.
Food processing
Production in SMEs now accounts for a significant proportion of the manufacturing in
Europe and represents a new market for the application of robotics technology.
2.2.11. Food Sub-Domain Overview
Increasing concern about food cost, traceability and security have impacted on all
aspects of the food chain in the last decade. There has been considerable interest in
the application of robotics technology to different aspects of the food production
industry, from farming to the preparation of food for consumption. Current Opportunity
Many applications for robotics technology have been proposed in the food
preparation industry, with new applications typically concentrating on areas where
there is a high level of manual labour or where there is a need for responsive
production with a fast turn-round or where contamination is a significant risk.
Areas considered include deboning meat, the preparation of ready-meals and the
packaging of delicate products. There is already considerable automation in many
areas of the food production industry where the uniformity of product and high
volumes can justify the investment. Where there is a significant variation in raw
materials and a high preparation overhead, or where the speed of processing is
limited by human factors these are areas that have attracted robotic solutions.
These applications often present significant manipulation and quality control
challenges where exact qualities of additives and flavourings must be made to each
product or where multiple items of differing shape and texture must be assembled,
for example in sandwich making. The advantages to the food industry lie in higher
levels of adaptation to demand, improved consistency, longer shelf life and higher
levels of hygiene. For example robots can be operated in an inert atmosphere to
stop oxidation, or can be consistently cleaned to avoid cross contamination. Future Opportunity
Future opportunities in the Food industry are likely to focus on the lowering of
production costs and meeting hygiene and regulatory standards. and the speeding
up of processing that is currently limited by human factors. At the retail end of the
market there may be niche applications for on demand food preparation, for example
in the production of ready-meals (e.g. Pizza, or microwave meals) to adaptable
specifications. These systems would allow a customer to specify the inclusion or
exclusion of specific ingredients, for example to account for allergies or taste, this
would also allow the system to individually price meals.
Much of the development in this sector comes in the form of specialised
manipulation and ingredient handling technology as well as dealing with the high
flexibility demands arising from short product life and the very short product runs
typical of a SME food manufacturer. Key Market Data
The European food industry can be characterised by the following:
Largest European Manufacturing sector (14.9% of turnover and 12.9% of
added value for EU manufacturing industries)
Leading employer in EU manufacturing sector (4.25 million)
Turnover €1,017 billion
14.5% of household expenditure
Exports €76,.2 billion
Trade balance € 13.2 billion
287,000 companies
99.1% SME
0.53% of turnover spent on R&D
{Source: Data and Trends of the European Food and Drink Industry 2012 –
FoodDrink Europe.} Relationship to other domains
There is linkage to the Agriculture sector specifically in the balance between the
preparation of ingredients at harvest vs preparation prior to food preparation. There
are also links to marketing robotics and to Domestic Appliances where the food
preparation process might be split between in factory and at home.
2.2.12. SME Manufacturing Sub-Domain Overview
It is widely understood that SME manufacturing is an important manufacturing sector
within Europe. SMEs are the engines of innovation within Europe and represent the
seed corn of industrial growth. The EC recognises this:
“What usually gets lost is that more than 99% of all European businesses are, in fact,
SMEs (see definition of SMEs). They provide two out of three of the private sector
jobs and contribute to more than half of the total value-added created by businesses
in the EU. Moreover, SMEs are the true back-bone of the European economy, being
primarily responsible for wealth and economic growth, next to their key role in
innovation and R&D.”
Addressing the manufacturing needs of SMEs is therefore an important step change
in capability for robot technology suppliers. These needs centre around the following
The need to design systems that are cost effective at lower lot sizes.
The need to design systems that are intuitive to use and are easily adapted to
changes in task without the need to use skilled systems configuration
The ability to work safely in close physical collaboration with human
In addition to these important design challenges there is also a need to address the
dissemination of good practice and knowledge about automation to SMEs. This is
made more difficult by the geographic spread of SMEs and the diversity of their
requirements. Current Opportunity
There are relatively few robotic systems designed specifically for the SME market.
The current opportunity relies on the acceptance of robotics as a means of
production within an SME environment. SME’s are typically unwilling to invest unless
there is a very clear benefit in terms of cost saving or revenue generation. The
specialised nature of most SME manufacture means that solutions must be highly
adaptable and deployment must be low cost.
There is also an opportunity for using robotics technology in the automated testing of
products, emulating physical user interactions to provide life cycle data. Future Opportunity
Future opportunity will depend on modularity and adaptability. Both adaptation to
individual tasks by unskilled users and adaptation between different tasks as the
manufacturing output shifts between product types. Barriers to Market
SME uptake of new manufacturing technology will depend strongly on perceived
economic benefit or competitive advantage.
2.2.13. Soft Products Sub-Domain Overview
The manufacture of clothing, shoes, and goods made from flexible materials
presents novel and complex problems relating to localisation and adaptation to parts.
Combined with the need for precision fixing required to manufacture a product where
look and feel are as important as function this area presents significant challenges.
In the wider context of bringing manufacturing back to Europe the garment and shoe
industries while still strong within Europe no longer have a mass production base in
Europe. The presence of leading global brands in Europe should provide an
incentive to investigate how robotics technology can impact on this type of
production. Current Opportunity
There is limited deployment of robotics technology in the manufacture of products
that involve soft materials. Most notably the food and garment industries are
currently labour intensive. While there is limited deployment of robots within the food
industry the garment industry is still dominated by hand assembly. Future Opportunity
Particular opportunities exist for specialised soft materials handling processing both
in terms of mass production and bespoke production. There are also opportunities in
the mixed processing of soft and hard materials where one is used as a coating,
fixed by gluing or defined pressure. Barriers to Market
The ability to predict the behaviour of flexible materials while being handled and
grasping technology are the main technical limitations. In mass market applications
the loss of capacity to the far east has reduced the manufacturing base within
Europe from which adoption of robotics technology might seed.
2.2.14. Craft and Bespoke Sub-Domain Overview
There is an increasing market trend to use the internet to allow customers to
customise and adapt products prior to purchase. Robotics technology may be able to
increase the levels of customisation while retaining low costs, and may also be able
to reduce time to delivery by allowing cost effective manufacture to take place closer
to the customer.
Similarly there are many areas of high value production which rely on craft skills. If
robotics technology is able to lower the cost of manufacture the high value margins
may present an opportunity.
2.2.15. Key System Ability Targets Configurability
The main requirement is being able to reconfigure industrial robots and their
applications with regard to both software and hardware. The hardware may include
peripheral devices, such as sensors, but may also include the kinematic chain of the
manipulator itself. Software configuration may take place during or prior to
installation or as a result of the end user selection of operating parameters. An
important step change in usability will come with the adoption of Intuitive
Within certain environments systems are at TRL9 for Level 3 (Run-time self
configuration) for limited mechatronic reconfiguration such as tool changing.
Mechatronic Kit (modular set up for
Configurability Level 2 for a wider range of
mechatronic options that are user
Introduction of Intuitive programming
Configurability Level 2
Standardised interfaces for modular
controller software:
Configurability Level 3 for software
configuration in plug and play architectures.
Autonomous configuration of
safeguarding strategies:
Configurability Level 4 coupled to Safety
Interaction ability at Level 3/4. Adaptability
The requirement is for the robot to respond to changes in the operating environment
include the ability to self-learn and apply auto-configuration strategies.
Adaptive control systems are deployed in some large scale manufacturing systems
(Level 1/2).
Self-learning robot with prepared
strategies provided in Knowledge
Adaptability Level 3 - Process chain
Self-learning robot utilising reasoning
Adaptability Level 4 - Task Adaptation
coupled to Cognitive reasoning ability Level
3 - Basic Environmental Reasoning. Interaction Capability
In manufacturing applications robots need to be able to interact with operators, other
robots and other systems within a production environment. The main requirement is
for these interactions to be safe, intuitive and appropriate. A step change in ability will
occur with the adoption of intuitive tasking interfaces.
Systems are deployed at TRL9 for Human Robot Interactions at Level 2, some
limited deployment exists in particular applications at Level 3 - Direct Physical
Interaction. Most current systems are at Level 2 - Basic Operator Safety for Safety
Interaction ability.
Intuitive Human Robot Interfaces for
use and configuration, teach or
specify task using domain specific
Human Interaction Level 4 - Domain
Specific Interaction.
Safe physical interaction.
Safety Interaction Levels 3-6 depending on
the level of operator risk.
Autonomous interaction with other
Robot to Robot Interaction Level 4 - Team
Learning through human-robot and
robot-robot interaction.
Cognitive Human Interaction at Level 4 Constrained Command interaction,
Knowledge Acquisition at Level 6 Knowledge scaffolding, and Robot Robot
Interaction at Level 5 - Team Co-ordination
for Communicated Adaptation between
systems. Dependability
Today, the state of the art is a Mean Time To Failure (MTTF) of approximately 10
years for the robot only. The limiting factor for current applications is very often the
periphery and integration environment. The relevant interpretation of “dependability”
in this case is both maintaining uninterrupted productivity, minimising necessary
downtime, and intelligent recovery procedures.
The majority of deployed systems have dependability at Level 2 - Fails Safe.
Capability of detecting upcoming
failures enabling preventive
Dependability Level 4 - Task dependability
Self-maintenance between robots.
Dependability Level 4/5 - Task/Mission
dependability coupled to Robot Robot
Interaction Level 5 - Team Coordination.
Maintenance performed robots in
hazardous places:
Cognitive Action Ability Level 7 - Success
Driven Action coupled to Robot Robot
Interaction Level 5 - Team Coordination. Motion Capability
The primary requirements for motion ability relate to the kinematics and dynamics of
manipulators as well as the positioning and navigation of autonomous platforms in a
manufacturing context as well as mobile manipulation for logistics tasks and for
advanced reconfigurable work cells.
Current deployed systems are TRL 9 for Level 3 - Open path motion.
Accurate positioning of mobile systems,
fast calibration, self-calibration;
consistency of coordinate systems in
sensors, platform, end-effector,
Location Perception at Level 2 provides
external reference points for position,
level 4/5 provides for mobile platform
localisation where there is an integrated
fixturing, etc:
Mode Switching, from flexible motion
(Human Interaction) to fixed motion
(Autonomous), e.g. variable stiffness,
controllable stiffness:
Motion Capability Levels 7-9 provide for
dynamic motion control.
Motion planning for HRI vs. motion
planning for autonomous operation,
plus orderly transitions between the
Motion Capability Levels 7-9 provide for
dynamic motion control. Manipulation Ability
The requirement concerns the ability to handle material objects and tools in a
manufacturing context. Adaptability and robustness are primary goals along with the
need for accuracy and repeatability.
Currently deployed systems are at TRL 9 are typically at Level 2 - Tolerant pick and
place. Some systems exist at Level 3 but without wide deployment.
Manipulation of flexible objects:
Cognitive Object Interaction Level Property Identification coupled to Level
8/9 of object perception, and Level 5
envisioning ability.
Free-form, shape-adaptable
manipulators and grippers:
Level 5-8 of manipulation ability
Human-robot collaborative
manipulation, load-sharing:
Level 3-5 of Human Robot interaction.
Robustness in the face of uncertainties.
Cognitive Reasoning Level 5 Reasoning with conflicts. Perception Ability
In this domain perception ability requirements vary significantly with application
domain. Of primary concern are a suitable choice of sensing modality, efficient
signal and data analysis, as well as generating the maximum information output from
the data at hand. Guaranteed safe perception is also a key requirement.
Most deployed systems are at Level 2 - Marker sensing, a limited number are at
Level 3 - Multi-Parameter Sensing.
Standardised data model for robot,
application, environment, etc:
Cognitive Knowledge Acquisition Level
6 - Knowledge scaffolding.
Integration of multiple sensors:
Perception Ability Level 3 - MultiParameter Perception.
Classification of status of perceived
information, e.g. quality information,
Dependability Level 4 - Task
error conditions, etc:
Context-aware perception to reduce
Perception: Object Recognition Level 6:
Context Based Recognition
Verification of contextual expectations
against current data, leading to
modifications of motion strategy
(supervisory control):
Cognitive interpretation Ability Level 5 Structural Interpretation coupled with
Decisional Autonomy Level 5. Decisional Autonomy
The primary goal is to increase the level of responsibility in the control processes of
the production system. The resulting autonomy is focused on reducing energy
consumption, increasing throughput, and providing context aware task control in the
interaction with operators.
Current deployed systems are TRL 9 for Level 4 - Simple Autonomy
Reacting to perceived status of
application (error condition, production
conditions, etc.)
Decisional Autonomy Level 6 Constrained task autonomy.
Online rescheduling of tasks in HRI
scenarios based on task, ergonomic
and safety information.
Decisional Autonomy Level 6 Constrained task autonomy coupled to
Safety Interaction Level 4 - Work Space
Energy efficiency criteria for path
Decisional Autonomy Level 6 Constrained task autonomy
Decentralised production knowledge
and decision-making instances to
augment robustness of manufacturing
Decisional Autonomy Level 10 Distributed Autonomy and Robot
Interaction Levels 3-6 depending on
system complexity.
Self-evolving systems capable of
autonomous manufacturing decision
Decisional Autonomy Level 10 Distributed Autonomy coupled to
Cognitive Reasoning Ability Level 8, and
Knowledge Acquisition levels 9-11. Cognitive Abilities
In the context of manufacturing, the greatest potential is for functions which
contribute to a reduction of programming and configuration requirements in deployed
systems. There are clear benefits for small lot size systems in reducing the time and
skill needed to reconfigure an adapt systems to new processes.
Current Deployed systems are at TRL 9 for Level 1 of Acquired Knowledge, Level 2
for Human Interaction, Level 1/2 for Interpretive ability, Level 3 for Action ability,
Level 1/2 for envisioning, Level 2 for Reasoning.
On the fly exchange of hardware (robot)
(enabled by abstracted task
Configuration ability Level 3/4 coupled to
Decisional Autonomy levels 5-7
representation with context-aware selfconfiguration).
Autonomous interpretation of situation,
constraints and relevant part of
production plan:
Cognitive Knowledge Acquisition Level 4
- Deliberate Acquisition coupled to
Interpretation ability level 5/6.
Situation interpretation through
heterogeneous sensors to enforce a
correct safety behaviour in HRI:
Safety Interaction ability Levels 4-7
depending on work context.
Human-robot interaction with open-end
learning process; robot apprentice
learning from experience, from various
workers, abstraction, etc:
Human Interaction Level 4 - Domain
Specific Interaction coupled to
Knowledge Acquisition Level 4-6.
Cloud-based cognition with access to
remote robot experience and ability:
Cognitive Knowledge Acquisition Level 7
- Distributed Knowledge.
Information perception, management
and interaction of individual robots
within the overall manufacturing
environment (sort of along the cloud
manufacturing idea in dealing with
digital resource management):
Cognitive Knowledge Acquisition Level 7
- Distributed Knowledge
2.2.16. Key Technology Targets
The key technology targets for the application of robotics technology in
manufacturing need to concentrate on systems with the following properties:
intuitive handling,
easy to use,
easy to (re-)configure,
provide safe perception and safe actuation with certified components and
provide an ergonomic design for human interaction
are energy efficiency, provide energy autonomy and short charging cycles
provide privacy for personal data gathered during human interaction. Systems Development
There is a strong relationship between the manufacturing sector and the Systems
Development technologies. Large productions facilities involving multiple robots and
multiple types of robot, for example part delivery AGV systems and robot arms used
in assembly, are highly complex. There is a strong imperative to manage this
complexity as efficiently as possible and the Systems Development technologies that
impact systems integration and deployment are a key part of the delivery of robot
manufacturing. The challenges of additional flexibility, and the increase in
collaborative working present a challenge to systems development technologies that
must be addressed if the expected deployment of smart manufacturing systems is to
be cost effective.
Systems Integration
Multi human – multi robot stations with seamless integration of humans and
robots in the same production line.
Modelling and Knowledge Engineering
Modelling technologies are a central aspect for modern application development.
They avoid premature investment and unnecessary changes to hardware aspects of
the application. Advances in the degree of realism will further contribute to this gain
of application development efficiency.
Mid term
Standard software for modelling environment / robot cell / robot line, including
sensors and actuated components.
Physics engine for real-time information on physical quantities in robot
Long term
Multi-physics enabled model of robot application, including all relevant effects
(e.g. solid, fluid, electrical, magnetic, thermal, etc.)
Real-time availability of all relevant physical information on environment and
application, to be used as a basis for real-time adaptive motion planning,
prediction and control.
Domain-specific ontologies for application description
22 Mechatronics
Mechanical Systems
The goals for mechanical systems design can be summed up as “smaller, lighter,
faster stronger”.
Appropriate design for physical interaction, design principles for safe
Zero cable robot
High performance robot based on low-cost / low-accuracy components
Appropriate design of drive components and kinematic structures for physical
interaction, design principles for safe interaction
Low-cost, modular drive systems with integrated sensing (e.g., position,
Low-power consuming drives and control methodologies
Multi-fingered industrially proven robust grippers
Safe components (SIL / performance level D)
Light-weight, intelligent structures (with sensors integrated)
Lightweight actuation principles, high power density, low-friction gears with
high transmission ratio
Direct drives for high loads
New safety-rated sensors for Physical Human Robot Interaction (e.g. Capable
of returning positions of objects / operators in scene)
Sensor redundancy for safety-rated applications, e.g. Information fusion from
diverse sensing types
General 3D Work/Object scan and monitoring for real-time path correction
Use of information available in the area from distributed sensors, e.g. to treat
occlusions and lift perspective redundancy of 3D perception
New control paradigms with constraint-based optimisation and use of task
redundancy for best trade-off among different objectives (e.g. productivity,
manipulability, safety, ergonomics...)
Sensor-based control with adaptation to unforeseen situations (e.g. obstacles,
Online control-based dynamic path replanning (e.g. from sensor information)
Increasing sophistication of control approaches can serve to increase the level of
robustness of applications, particularly in the event of uncertainties.
Sensors and Sensing
One essential ingredient of any approach to add more and new functionality to
control, motion planning, application adaptivity, etc. is always the availability of an
increased level of information on the environment and on the application. Therefore,
advances in sensors and sensing are a basic enabler for such progress.
Power Supply and Management
Wireless power transmission Human Computer Interaction
To avoid additional hardware such as fences and fixed guards, future applications
will rely more heavily on sensor-based support for safety functions as well as safe
behaviour of industrial robots. While this can make possible various degrees of
direct human-robot interaction, it also can serve to make application layout more
compact and cost-efficient.
Methods and tools to adapt robot motion to injury risk knowledge (see TG
Intelligence and decision-making capability for autonomously generating
dynamic safety zones based on live robot movements (as opposed to preprogrammed motions). Perception
Use sensor information redundancy to detect faulty situations (e.g. sensor
failures, control failures, etc.)
Combination of various sensing technologies to achieve safety-rating of the
Bringing new sensing capabilities into routine industrial use as safety-rated
Long Term
Self-calibrating safety sensors
Combination of various sensing technologies to achieve safety-rating of the
Integrate new sensing capabilities into existing systems as safety-rated
systems Navigation
Task appropriate indoor positioning in industrial environment, e.g. combination
of platform + manipulator
Motion Planning
Capability to autonomously generate alternate motions to avoid collisions
(safety rated algorithms)
Autonomous path planning with obstacle avoidance in cluttered environments
Reactive motion planning, i.e. online planning revision, based on current
sensor information
24 Cognition
Learning Development and Adaptation
Learning Affordances for Robot Object Interaction.
Task learning by demonstration, human-robot and robot-robot interaction
Natural Interaction
Passive and Active Safety of Mobile Manipulation in Human Workspace
Ergonomic Evaluation, Analysis of Workspace Sharing Systems
Instruction and Assistance in Semi-Automated Assembly Processes
Intelligence and decision-making capability for autonomously generating
dynamic safety zones based on live robot movements (as opposed to preprogrammed motions)
2.2.17. Technology Combinations
Flexible Grasping
Systems which are able to grasp arbitrary objects of varying geometry and weight
while requiring only few to no user input. The grasping system will be able to
generalise knowledge from previously learned grasping tasks to novel grasping
situations. It will be able to handle objects, unknown objects similar to previously
known objects, and also flexible parts. This capability is brought forth by a
combination of grasp planning + cognitive abilities + sophisticated sensing means.
Model Driven Engineering of Complex Systems:
Providing an engineering environment for a robot designer that dramatically
improves the time and effort required to program and design a robotic system to
tackle a new task. The robot designer will be empowered to efficiently reuse
components in new and creative ways, while at the same time most engineering
tasks like robot program generation will be performed automatically by the underlying
framework. The robot designer can therefore concentrate on the creative tasks while
many engineering tasks are automatically handled by the software framework. Here,
we see a combination of systems engineering and integration + modelling +
knowledge representation.
Mobile Manipulation:
The goal is to develop systems which can support a human worker with manipulation
tasks. For seamless and flexible operation, the system has to be able to execute
complex manipulation tasks in unstructured and dynamic environments. This brings
together technology targets in motion planning + safety + collaboration and
interaction + learning and adaptation.
Passive and Active Safety of Mobile Manipulation in Human
Seamless and safe human-robot interaction on the work floor. The development of
new safety concepts for human-robot interaction is based on existing industrial
standards and regulations. The realised safety will conclude as well avoiding static
obstacles (e.g. tables, etc.) as well as reacting actively to dynamic obstacles (e.g.
humans and other robotic systems) that are moving around in the environment. With
active safety the robot will avoid the human operator, thereby trying to continue to
fulfil its assignment. New planning and control paradigms, where different objectives
are concurrently optimised, need to be addressed. Here, we draw on the technology
targets of motion planning + safety + collaboration and interaction + learning and
Ergonomic Evaluation, Analysis of Workspace Sharing Systems:
Define ergonomics requirements for a safe human-robot interaction. These
requirements will function as guidance for the development of the mobile manipulator
and the workspace for collaborative manufacturing. The design process will be
validated against the defined requirements and updated to uphold the ergonomics
principles. Ergonomics requirements will also serve as one of the objectives to be
optimised with motion planning, through e.g. use of intrinsic kinematic redundancy or
task redundancy of the manipulator arm.
Relevant technology targets are
collaboration and interaction + cognitive abilities + sophisticated sensing.
Instruction and Assistance in Semi-Automated Assembly
Holistically improvement and facilitate a flexible development of the robot systems in
co-existence with the human. These cooperative processes have to address safety
issues, and the robot system has to be highly flexible to be able to fulfil new tasks.
Thus the main goal is the development of a system interface that provides an
intuitive way to teach a robot’s behaviour in assembly sequences without the need of
professional trained robot programmers. Important technology targets for this
objective are collaboration and interaction + safety + motion planning + sophisticated
sensing + cognitive abilities.
Rapid Deployment in Realistic Industrial Environments
A key capability is the ability to quickly deploy robotic systems in realistic industrial
environments. A large portion of the cost of automation solutions is spent on
deploying solutions to new customers and under slightly varying requirements.
Current deployment strategies rely on a long set-up process by experienced system
operators and are generally not automated. A key mid-term goal will thus be reducing
the time and effort spent by operators in configuring a perception system to operate
in a new application domain or a new operational environment. The major
technological advance in this respect is expected to come from better learning
capabilities and more robust solutions for interpretation, as well as synergies with
more robust mapping and localisation systems in semi-structured dynamic
environments. Important directions to investigate include limiting dependence on
costly infrastructure solutions, increased transferability of experience, life-long
learning as well as learning by demonstration.
2.2.18. Product Visions
There are a number of different product visions in the manufacturing sector, these
relate to the breadth of the sector and the different driving forces in the market. On
the one hand systems need to be developed that improve cost vs performance in the
traditional manufacturing sectors so that Europe can retain its current market
position. On the other hand new markets based on increasing human interaction and
more flexible adaptation and configuration suited to SME manufacturing processes
represent an important and growing new market.
The key product vision in manufacturing is of a robot able to safely operate in an
semi-structured environment in physical collaboration with human operators. To be
configured using intuitive interfaces by operators rather than by specialised
programmers. These new systems need to have flexibility not only with respect to the
user interface but also with respect to the task. Generic grippers, gripping strategies
and planning and control systems able to adapt to different optimisation parameters,
and to a dynamic environments without compromising safety.
This vision involves the integration of a much broader range of sensing and
interpretation technologies with advanced systems development and human robot
interaction technologies.
As with all technology related to manufacturing the R&D&I activity must result in
deployable systems that provide an economic advantage.
2.3 Agriculture Domain Domain Overview
Agriculture is a general term for production of plants and animals by use of
paramount natural resources (air, water, soil, minerals, organics, energy,
information). Products are used in a large variety – nutrition, renewable energy and
renewable materials. Agriculture can be divided into subcategories like shown in
Figure 1. As in every other industry and part of an entire system this domain has
relationships to suppliers, buyers, processing, services, administration and end“users”.
Figure 1: Simplified structure of agricultural production categories
Several interrelated drivers make agriculture a challenging business:
Figure 2: Interrelationships among key drivers of change in food systems, and their
connection to human well-being
{Source: FAO - IFPI's long-term outlook for food and agriculture}
With reference to the key European Societal Challenges. The following major
conflicting trends can be seen:
World population is growing from todays figure of 7 Bn. to an estimated 10 Bn.
in 2050.
The available agricultural area (4.9 Bn. ha) is limited and can only be increased
marginally. In effect it decreases by degradation. 1/3 of 1.4 Bn. ha arable land
is degraded.
Expected climate change has negative impacts on soil productivity
Consumer habits in emerging countries are rapidly changing. Producing 1 kg of
meat needs up to 16 kg of grain.
The growth in land based renewable energy and renewable materials
production conflict with food production.
Land and agricultural products become the subject of speculation reducing
In highly developed countries agricultural machines are reaching their capability
limits. Additional productivity has to be achieved in other ways.
FAO expects that by 2050 only 0.15 ha of arable land will be available per
Within Europe agricultural machines are typically equipped with a high level of
installed performance. Analysing tasks carried out with these machines it can be
observed that realised performance differs significant (Figure 3).
Figure 3: Gap between installed and realised performance
The gap can be reduced by enhancing single machine capabilities i.e. with
assistance systems or autonomous functions on one side and by improving the
entire process performance – the interrelation to other machines – on the other side.
In the context of agriculture robotics autonomous capabilities should not only should
focus on single machines but whole processes (self-organising machines and selforganising processes).
Introducing robot systems significant advantages:
Increase precision and quality in the process
Extend operation time
Provide a quantitative decision base
Improve interoperability and coordination
Reduce unit costs
These advantages are counterbalanced by several barriers:
safety: Large and heavy machines in accessible environment, Robots on public
privacy: data capture, processing
environments (contractor vs. farmer)
• organisation: control in mixed environments, how to get a robot to the field and
the driver back home? Current and Future Opportunity
The prime opportunity in the application of Robotics Technology to farming is to
increase farm efficiency while maintaining economic and ecological standards.
Robotics technology has the potential to transform all types of farming while
significantly increasing data gathering to enable better decision making. Key to these
improvements is the interoperability and communication both between machines
working on the farm and to organisations outside of the farm. Such connections allow
improvements in the processing of harvested crops, efficient transport and faster
time to market. The increasing use of technology will also impact on the life quality of
farm workers and may also attract a younger generation back into farming. Robotics
technology has the potential to make ecological targets for the use of pesticides,
fungicides etc achievable at lower levels, as well as contributing to good soil
Farming has an important place within the European community but is also a global
opportunity to export machinery and expertise. Europe has taken a strong lead in a
number of areas where robotics technology can be applied and it is important that
this progressive approach is promoted and supported through R&D&I actions.
The following are important impact points for the application of robotics technology:
Improving interoperability through standardisation (for example: ISOBUS)
Moving processes from a batch oriented 'factory' to a flow oriented, continuous
process (24h milking, 8 times feeding by robots)
Applying path strategies to reduce soil compaction
Interconnecting multiple autonomous systems to improve quality and efficiency
Interoperability of machines with different degrees of autonomy. Such a feature
will allow replacing the classic tools step by step with intelligent ones (e.g. an
intelligent tractor will still work with a simple plough).
Development of driver assistance systems suggesting corrective actions,
leaving the responsibility with the farmer. In the future provided there is legal
proof of dependability corrective actions can be taken directly by the system.
Possibility of increased communications enabling tele-operation.
Increased energy efficiency through optimised use of machines.
Develop systems and strategies that reduce the use of antibiotics and *icides.
Boosting ecologically sound farming.
As robotics technology develops the extent to which farming tasks and processes
can be automated will increase. While current systems act in an advisory capacity, or
provide limited automation it is anticipated that levels of autonomy will gradually
increase. This increase in autonomy will be driven by greater data collection and
built-in knowledge about each farm and the preferences of the farmer and staff, for
example improved soil analysis driving more effective crop distribution.
Specific areas of future opportunity are:
Utilisation of sensing and data processing on autonomous machines operating
on the farm to gain more fine grained data about the land and crops. This data
can then be utilised to improve decision making and in driving increasingly
accurate simulations of the farm. This presents a number of different
Using “Big Data” methods to provide the farmer with access to broader data
analysis, including comparative analysis.
Driving crop distribution patterns to maximise land use and yield year on year.
Provide data about livestock feed stuff vs yield.
Allow better land and equipment utilisation.
Maintain ecological standards.
Improved data will also allow the development of more realistic farm simulations of
the actual farm (with all its fields, barns, animals etc.). Giving a tool to develop and
test new tools, machines or work strategies and provide the basis for a fully
automated farm.
Investigate the possibility that multiple smaller machines allow more flexibility
than one large machine, the opportunity to reduce soil compression could be
cost-effective even on very small fields, and the combined reliability may be
Development of modular machines where a core autonomous machine is
usable on multiple tasks during the year with changeable tools.
The progressive development of greater levels of autonomy, particularly the
development of systems able to assess risk and impact. Highly autonomous
systems would allow the farmer leaving the farm for a longer period. Leaving
the farmer able to access data via the internet once or twice a day to provide
high level supervision. Such high levels of automation might allow “part time
farming” by supporting liveability and managing land in rural communities.
Develop energy systems able to utilise the natural energy sources (e.g. byproducts from the current process) and provide continuous operation. Barriers to Market
There are numerous barriers to market:
The legal framework for the operation of autonomous systems does not yet
exist. This means that systems must remain “human in the loop” for safety and
liability reasons.
There are potential issues with Cross-Contamination (grain, weeds, bacteria
etc) where autonomous machines operate across the whole farm.
Within farms there are issues about the storage and maintenance of
autonomous systems, storage facilities will often need to be upgraded and new
skills acquired. Support infrastructure, power, communications, supplies etc will
also present an additional cost.
Interface standards between machines and to the wider community will need to
be developed and the adoption of extensive autonomy will depend on this.
• The privacy of data about the farm also needs to be assured as this will be
extremely valuable to third parties. Key Market Data
In 2007, gross value added in the agriculture sector amounted to €16.0 Bn.
(production value €46.3 Bn; intermediate inputs €30.3 Bn) . Some 370.5 thousand
enterprises farmed around 17 million hectares of agricultural land and had 1.251
million employees (of whom 336.3 thousand were seasonal workers). Actual work
done corresponded to 529.7 thousand full-time equivalents. Approximately 95% of
agricultural businesses are family-run farms, which farm less than one-quarter of
agricultural land. At 55% of farms, activity in the agricultural sector is only a sideline.
The dominant legal form is the sole proprietorship, which accounts for 93.5% of all
farming businesses, followed by partnerships (5.1%) and legal persons (1.4%).
Based on area farmed, legal persons farmed an average of 561.6 hectares, with
partnerships farming an average of 125.7 hectares and sole proprietorships farming
an average of 33.1 hectares.
According to a VDMA market research in 2013 agricultural machinery valued at €96
Bn. will be produced (estimated). Europe's share is at about 30%. Considering the
introduction of mobile agricultural service robots as an evolving process an
increasing share of €30 Bn can be assigned to the robotic market.
World production agricultural machines (VDMA, 2013 estimated)
€96 Bn.
Investment €/ha Germany 2012 (VDMA)
324 €/ha
Investment €/ha Netherlands 2012 (VDMA)
599 €/ha
Investment €/ha Spain 2012 (VDMA)
36 €/ha
Investment €/ha EU 2012 (VDMA)
150 €/ha
Agricultural area ha EU
170 Mn. ha
Table 1: Some key market data
Table 2: Number produced and sold in leading countries
Compared to the number of agricultural machines produced (see table 2) the number
of mobile service robotics (5,000 see Figure 4 "field) is comparably small. This
number mainly results from milking robots.
Note that in these figures autonomous in-door service robots for farms (mainly barnfloor cleaners) are not included. This markets currently has a growth rate of
approximately 3000 units annually.
Figure 4: Source IFR statistical department
The term “robot” in an agricultural context requires proper definition if the extent of
the market is to be properly established. For example auto-guidance is often
available as a retrofit to manual equipment. It is expected that as the market
develops its exact nature will become clearer. Relationship to other domains and markets
There is a strong link between the agriculture domain and other markets. There are
possible synergies with the automotive sector. Autonomous driving and safety are for
example needed technologies in both application domains. Furthermore autonomous
feeding systems in livestock farming have a large technology overlap with automated
guided vehicles in logistics and logistic support systems in the manufacturing
The need for robustness of autonomous agriculture machines is similarly needed for
autonomous systems in the construction and mining industry.
There are possible synergies regarding technologies such as system design, power
management and safety.
There are strong links to the component supply industry because of the need to drive
the down the cost of robot parts, mechanisms, sensors and associated subassemblies through mass production. There are also links to more consumer driven
markets such as smart phones and tablet computers as these are most commonly
used as interface devices to farm systems.
As the food processing industry is a direct successor of the agriculture domain it has
the strongest link of all to the above named markets. The transfer of robotic
technology into the agriculture domain might lead to more efficient agriculture
processes with, for example, new timing constraints. The food processing industry
also has to adopt these processes by responding to changes in farm practice
brought about by increased use of robotics technology.
34 Europe’s Place in the Market
The value of agricultural machines in Europe is €28bn. which is 30% of worldwide
production. Whereas the worldwide export rate averages 50% it is significant higher
in Europe (estimated 70%).
Compared to other markets Europe’s agricultural machinery industry consists of
numerous well known and highly specialised manufacturers providing high-end
solutions to their customers. Unlike America European agriculture machine fleets
primarily consist of a large variety of brands. Interoperability has been a long held
Robotic developments can mainly be seen in Europe (dairy), the US (high value
crops) and Japan (high value crops).
Considering the engineering strength of European manufacturers and high market
shares worldwide, there is considerable potential for future development within
Europe feeding a global market for autonomous agricultural machines.
The global market is also changing. China is being forced to modernise its economy
and to correct the lag in its machinery base. China may therefore become in
important producer of agricultural machines. The construction machine domain can
be sen as a blueprint for this expansion. In 2010 China became the worlds largest
manufacturer of construction machines. In 2010/11 600,000 tractors were sold in
China spending 1.5 €Bn. (For comparison: Germany 28,000 tractors). It is highly
likely, as with other industries, that China will try to enter the European market. It is
important that Europe has a secure technological leadership in advance of this
happening. Key Stakeholders
Europe has an extensive agricultural machinery business. It contains significant
global players and has a strong innovation mentality. Agriculture has a very broad
range of stakeholders that reflect both the diversity of products and the diversity of
farm management strategies. Europe’s wide geographic spread means that almost
every aspect of farming can be found within Europe from large arable farms, where
many farms are managed as a conglomerate, to individual family run farms
specialising in unique products.
In addition to the end users and manufacturers there are suppliers of consumables,
fertiliser, feedstuff, livestock, seeds etc. As well as energy suppliers, legal services,
analytical services and veterinary practices. Government departments and standards
bodies are also significant stakeholders in agriculture.
CEMA is the European association representing the agricultural machinery industry.
In the agricultural machinery sector, there are some 4,500 manufacturers, that
generated a turnover of around € 28 Bn. in 2008. 135,000 people work in this sector
and a further 125,000 people work in distribution and maintenance. A significant
number of the larger manufacturers are already engaged in the design, development
and production of robotics technology based products. EurAgEng is the European
Society of Agricultural Engineers (EurAgEng) exists to promote the profession of
Agricultural and Bio-systems Engineering. It lists Research Institutes all over Europe.
In addition to these industrial organisations there are a number of academic and
research organisations that have significant facilities for the development of
autonomous agricultural machines. These organisations, some of which are
dedicated to agricultural research, are spread across the main farming nations within
Europe and have an extensive base of research on which to draw.
35 Current Key Projects
HUBRINA (HUman-roBot co-woRking IN Agricultural master-slave systems) - Tyker
Technology (NL), Wageningen University (NL) - ECHORD. Master-slave robot
control for agricultural activities. Advance the research to master-slave systems in
agriculture beyond just the level of simulation and prove the feasibility of a fully
automated master-slave system.
FutureFarm (WP6: Influences of robotics and biofules on economic and energetic
efficiencies of farm production) - University of Wageningen (NL) and 14 more - ERANET ICT Agri. Typify current and new robot technology and their potential tasks in
farming. Single and multiple machine fleet management in view of energy usage and
costs will be optimized. Demonstrate current research robotic platforms for
CROPS (Clever Robots for Crops) - University of Wageningen (NL) and 13 more ERA-NET ICT Agri. Intelligent sensing and manipulation for sustainable production
and harvesting of high value crops
RHEA (Robot Fleets for Highly Effective Agriculture and Forestry management) CSIC (ES) and 18 more - ERA-NET ICT Agri. Design, development, and testing of a
new generation of automatic and robotic systems for both chemical and physical –
mechanical and thermal– effective weed management focused on both agriculture
and forestry.
GEOPAL (GNSS-based Planning system for Agricultural Logistics) - Aarhus
University (DK), LACOS (D), CLAAS-Agrosystems (D), LEE Engineering &
Construction Company (UK?). Research on Technologies for improving cooperation:
Fleet Radar, Infield Route Planning with background of renewable energies.
QUAD-AV (Ambient Awareness for Autonomous Agricultural Vehicles) - Fraunhofer
IAIS (D), Cemagref (F), University of Salento (I), Claas (D/DK) - ERA-NET ICT Agri.
Enhancing Saftety-Level of autonomous agricultural vehicles during process in terms
of threads to humans, animals and tangible goods. 4 different types of sensors are
combinde. these are Stereo Vision, Radar LADAR and Thermal Imaging.
SmartBot - (here: Subproject AgroBot ) - INTERREG. Develop basic technologies
needed for constructing multiple, agriculture, robotic demonstration models with
different application European Products
European products in terms of agricultural autonomous machines can be divided in
mainly three categories: Arable Farming, Livestock Farming and Special Crops.
Milking robots, with an installed base of approximately 30,000 world wide, is
dominated by European companies. In some countries up to 50% of newly built
barns nowadays have milking robots instead of milking parlours. Milking robots can
milk cows more than 2 or 3 times per day, hence increasing the udder health of the
cow, and creating less stress for the animals. The farmers are released from having
to do a heavy task at set times each day, 7 days per week.
Mobile autonomous barn cleaning robots, typically used for barn floor cleaning &
feed pushing have an instaled base of between 10,000 and 15,000 units. Cleaning
barn floors frequently has a positive effect on NH3 emissions, and on the hoof health
of livestock.
Mobile autonomous feeding robots. This is a new development, replacing the
manually operated mixer feeding wagons. For example cows can now be fed 6 to 8
times per day, instead of twice per day, giving the animals a more natural eating
pattern. The feeding robot can also provide more balanced rations to specific groups
of animals within the barn.
Arable Farming and High Value Crops:
The Robotics & Automation Society lists manufacturers of agricultural mobile service
robots. All companies are engaged in the domain of livestock farming
( Opposite from livestock farming where most robotic
systems are developed from scratch, arable farming uses an incrementally enhances
existing machines with autonomous capabilities. This results in a threefold situation:
Prototypes: BoniRob (Amazone), Demeter (University of Illinois;), HortiBot, Kinze
Grain Auger,
Traditional Machines with autonomous capabilities like Autoguidance and Master/Slave-Procedures: John Deere, CNH, AGCO, CLAAS
Robots for high value crops (planting, pruning, harvesting): Harvest Automation,
Robotic Harvesting, AGROBOT,
Agriculture Sub-Domains:
2.3.2. Agriculture Domain Overview
There has been considerable progress in recent years in the deployment of
agricultural robotic systems. Dairy heard milking can now be fully automated with
improved milk yields and lower infection rates. Automated ploughing is becoming
increasingly used and systems are being trialled for selective weeding and
Primary concerns in this market are the optimal use of resources, improvement in
yields, and minimisation of environmental impact for example soil erosion and
compaction, pesticide and fertiliser use. The automation of source tagging for
livestock and arable crops, the inspection of fields and livestock and the monitoring
of crop condition all drive the development of fully integrated systems.
The industry traditionally has long service life from its equipment and there will be an
expectation that robotic products will give similarly long life cycles. Cost effectiveness
is a driving concern. Current and Future Opportunity
Current opportunities in agriculture and forestry are extensive. There is a strong
collaborative relationship between research organisations and the agriculture
equipment companies. Opportunities are well understood in the industry and this is
supported by growing level of deployed systems.
The domain is ripe for technology transfer in terms of sensing and manipulation.
Future developments are likely to concentrate on systems abel to selectively harvest
ripe produce and those able to recognise the early signs of pest infestation or
disease and selectively respond. Systems abel work with high value delicate crops
and those that currently require hand harvesting will provide exploitation
opportunities provided that the cost benefit analysis can be proved. Relationship to other markets
There is an interface relationship with the Food domain. In terms of crop assessment
there will be linkages to other service robot domains such as Civil Infrastructure and
Service and Utilities. Europe’s Place in the Market
Europe has a leading position in this market with a large number of agriculture
equipment companies located in Europe.
2.3.3. Forestry Domain Overview
Europe has considerable forestry resources that cover very large areas of land.
Managing this resource both in terms of monitoring it and felling trees can be
automated to a certain degree. Historically there has been significant interest in
forestry robotics within Europe. The automation of felling per tree is now at a high
level where a felling machine can fell and process a tree without much human
intervention. The use of robotics technology to monitor forests is still in its infancy
because of the large areas of terrain that need to be covered and the limitations of
current UAV usage regulation.
2.3.4. Fisheries Domain Overview
There are numerous applications for robotics technology in fisheries. Primarily
applications focus on the monitoring of fish stocks and water condition. The use of
autonomous systems to monitor both shoal size and type, as well as longer term
trends have the potential to alter the application of fisheries policy. Robotics
technology may also contribute to the implementation of fisheries policy through the
monitoring of fishing practices.
2.3.5. Key System Ability Targets Configurability
All farms are different both in terms of their product mix and physical characteristics.
Farmers will want to be able to use the best machine for each task and these will
come from different OEMs. The mechatronic configuration of different agricultural
machines to enable inter-operation is a critical part of “plug and Play” technologies in
agriculture. Similarly the dynamic configuration of software don physical interfaces
during missions as machinery enters and leaves each task is an important part of
raising autonomy levels.
Configuring systems to each farm and to the specific needs of each farmer is critical
to the wider market adoption of robotics technology.
Optimal configurations for different
farm sizes.
Mechatronic Configuration: Level 2 User
Run-time Configuration
Configuration to specific crops,
harvesting parameters, crop size
Mechatronic Configuration: Level 3 Run-time
Self Configuration
Self configuration of groups of
Mechatronic Configuration: Level 4
Autonomous Configuration Adaptability
As the growing season progresses, as crops are grown, as boundaries are altered
and feed and pesticides are used the autonomous systems on the farm will need to
adapt to provide optimal output from each task.
Adaptation to farm layout and crop
and field patterns.
Adaptation Level 3-5
Adapt to the long term dynamics in
the farm cycle.
Adaptation Level 3-5
Adaptation to crops, new crops or
Adaptation Level 3-5 Interaction Capability
Interaction between different agricultural machines and their inter-operation is critical
to many of the task visions for agriculture. The need for certifiable decision making
also requires close collaboration between human and autonomous machine.
Machines aware of each other’s
Robot Robot Interaction: Level 2
Communication of Task Status
Interaction between machines to
establish capability.
Robot Robot Interaction: Level 2
Communication of Task Status
Human machine interfaces
appropriate to farm environment.
Human Robot Interaction:
Safe human interaction with large
Human Robot Interaction Safety:
Machine to machine knowledge
Robot Robot Interaction: Level 4-5 Dependability
Farming requires high levels of dependability for fully autonomous operation.
Livestock and crops have high value and in many cases decision making needs to
be certifiable and traceable.
Safety guarantee under all
operating conditions.
Certification and Classification of Safety Levels
{Note: This requires environmental safety
Proof of dependability
Dependability: Certifiability Target Motion Capability
Agricultural machines are designed to endure the harsh conditions on the farm. The
ability to maintain location and control on sloping surfaces and in poor ground
conditions without impacting on the environment or compromising safety guarantees
presents a significant challenge.
Safe motion on difficult and
dynamic terrain
Cognitive Interpretive Ability: Level 9
Environmental Affordance
Track and path planning to optimise
energy and ecological parameters
such as ground compaction.
Motion Capability: Level 6 Parameterised
Motion Manipulation Ability
Harvesting and crop handling, particularly of soft items, will require robust and
dependable manipulation solutions.
Ability to control force and manipulate
Motion Capability: Level 5 Force
constrained motion
Livestock manipulation
Handling of soft and delicate items e.g.
Fruit harvesting. Perception Ability
While farms contain a subset of objects that will need to be recognised the ability to
correctly interpret novelty and more importantly condition, particularly of livestock
and crops will presents a challenge to perception abilities. Detecting field and crop
boundaries throughout the growing season, minimising wastage and reducing
environmental impact will all require advances in perception ability.
Ability to identify boundaries, crop
condition, objects including animate
objects in fields, distinguish plant types
and pests.
Perception Ability: Level 5-7 combined
with Object Recognition Level 7:
Novelty Recognition and Scene
Perception: Level 4-5.
Maintain perception ability in extreme
weather conditions, rain, fog, snow, ice.
Object Recognition Parameter:
Perception Ability Target: Immunity to
natural variations.
Perceive other machines and farm
products and objects (e.g. hay bales,
feed bags etc).
Scene Perception: Level 2-6
Object Recognition Levels 4-6
Perception of people and other animates
obscured or partly obscured by crops.
Object Recognition: Level 10 Animate
Objects Decisional Autonomy
Higher level decisional autonomy will be critical to increased levels of autonomy in
the application of robotics technology to agriculture.
Self or co-repair of machines.
Decisional Autonomy: Level 7 Multiple
Task Autonomy. Cognitive Abilities
The variability both in the short and long term coupled to the long term data
gathering on the farm combined with external knowledge sources means that
agriculture will require advances in cognitive ability to enable autonomous operation
and optimally manage tasks.
Ability to Semantically map the farm.
Action Ability: Level 6 Plan Driven Actions
combined with Acquired Knowledge: Level
Farmers assistant based on deep
knowledge of historic farm operation.
Acquired Knowledge: Level 5 Place
Acquired Knowledge: Level 6 Knowledge
Ability to learn successful strategies
from machines, other farms or
external experts.
Acquired Knowledge: Level 9 Interaction
Goal based task planning, including
weather and ground conditions.
Action Ability: Level 5 Knowledge driven
Overall and long term strategic
Action Ability: Level 6 Plan driven actions
planning based on Farmers Assistant.
Autonomous experimentation with
new strategies.
Acton Ability: Level 7 Dynamic planning
2.3.6. Key Technology Targets Systems Development
High level requirements for dependability and safety will drive system development
methods. The need for standardised interfaces and certified systems will also impact
on the design of both mechanical and software systems. Many of solutions for
robotics in agriculture are only achievable if they are completely manufacturer
System Architecture
Cross platform architectures, common interface standards and
Systems Integration
Interoperability of systems from different original equipment manufacturers
Modelling and Knowledge Engineering
Key to the development of better farm management and optimisation is the
development of better models based on captured data. The use of simulation
in the loop decision making is a key technology in strategic decision making.
Development of farm models based on data gathered from the farm by
autonomous systems.
Development of semantic representations of agricultural information, for
communication and within decision systems.
System of Systems
Farms will become a integrated system of diverse devices cooperating to
carry out the work and assess results. Farms will become an important
exemplar for System of Systems research and development. Mechatronics
The agriculture industry has a long history in the design of large mechatronic
structures and mechanisms and control systems are well understood. However
autonomous systems will create new opportunities for design and in particular in the
interactions between machines where material and crops need to be transferred
during operation. There are also opportunities in developing mechatronic systems
that operate where high dexterity and fine control are needed, for example in fruit
Mechanical Systems
Systems for easy maintenance.
Retrofit autonomy for existing machines.
Miniaturised actuation
Power Supply and Management
Power systems for continuous operation.
Machine to machine communications in-field, with no infrastructure.
Control strategies that require minimal calibration. Human Computer Interaction
It is important that existing farm workers are able to continue to operate autonomous
equipment as well as manual machines. This requires intuitive and interactive user
interfaces. The value resting on specific decisions in the agricultural environment can
be high so it is important to ensure that interfaces present information in a clear and
intelligible way for all types of operator.
Human Machine Interface
Operation of complex machines by multiple users with different skill levels.
Certification of systems, including certification of sensing and decision
Analysis of safety in large autonomous machines in unstructured
environments. Perception
Perception is critically important to the operation of many applications of robotics
technology in agriculture. Perception ranges from the application of multiple sensing
modalities for the assessment of crops to the detection of crop and field boundaries
and the identification of novel events in the agricultural environment. The wide range
of operating conditions and the need for certifiable performance present significant
challenges to the development of workable solutions.
Develop all weather processing of sense data.
Detection of novelty in agricultural context. From detection of crop condition,
e.g. Early detection of infections and infestations, to the identification of
unexpected animates in a scene. Navigation
The agricultural environment presents numerous difficulties in terms of 3D
navigation. The weather creates a dynamic environment as do the changing seasons
and crop growth and rotation.
Semantic mapping within a agricultural context.
Fine grained localisation in 3D for harvesting.
Motion Planning
Motion planning accounting for vehicle condition and ground conditions to
minimise soil/crop impact.
Motion planning against dynamically changing weather conditions. Cognition
Aspects of decision making and perception in agriculture involve a cognitive element
where long term strategic planning, learning or task optimisation are critical. The
optimisation of yield, and the identification of trends an patterns in data gathered
from the environment all require cognitive processing.
Learning Development and Adaptation
Learning with low rates of repetition
Learning from simulation
Knowledge Representation and Reasoning
Knowledge acquisition during normal operation
Action Planning
Planning the optimal use of multiple machines during a process.
2.3.7. Technology Combinations
Cooperating (autonomous) agricultural machines
Many processes in the agricultural domain depend on the cooperation of machines.
If autonomous machines are to impact on farm processes then each process must
be automated and this requires a set of machines to cooperate. To co-ordinate such
collaboration will require a distributed planning system because individual machines
often enter and leave a process dynamically. Such a planning system has to
consider the abilities, for example the motion ability, of all participating machines
(autonomous or not) and needs to find an overall process optimum for all machines.
Subsequently the missions/tasks have to be distributed to the machines. The
development of such distributed planning processes will increase the impact of
autonomous systems within farming.
Safety design and certification
Safe interaction of humans and agriculture machines is one of the central
preconditions for the market introduction of autonomous agriculture. As with
industrial robots for some applications ti may be sufficient to protect the work area
and ensure safety by excluding people form the working environment. However
many applications will require greater levels of safety because of the direct physical
interaction either with people or livestock. Safety will need to be embedded in the
physical human machine interface. This interface needs to be able too operate in
diverse conditions.
Similarly the safety inherent in the machine’s ability to detect its environment will
need to be enhanced by developing optimal combinations of sensors, sensing,
obstacle detection and obstacle avoidance strategies. Of equal importance is the
certification of safe operation. Lack of certification will present a barrier to many farm
applications of robotics.
Hardware in the Loop
Testing agriculture scenarios in simulation is important. Especially in arable farming
realistic tests are often only possible once a year during harvest. This can be
mitigated by high quality simulation. A planning system that coordinates multiple
cooperating machines can use the simulation in the development phase but can also
use the simulation in the loop during the real scenario. The building of farm models
both in terms of developmental models that allow systems to be designed and in
terms of providing models of individual farms based on gathered data are critical to
the long term deployment of robotics technology.
Semantic Environment Awareness
A significant step ahead is possible in many robotic applications if the robot has a
semantic representation of its environment and is able to keep this representation up
to date in a dynamic unstructured environment. This enables the system to reason
about the tasks it has to do to achieve a high level goal or work within task level
constraints. In the agriculture domain this means for example that a robot that today
follows a wire in the ground and pushes feed to livestock would progress to a system
that is able to fulfil a range of tasks driven by the farmer for example: “Please feed
my cows in all my barns and keep my farm clean!”
“Plug and Play” Systems and Architectures
Since the agriculture domain has various constraints (field size, weather, ground
conditions,…) it is impossible for a single agriculture machine to fit all these
constraints. Machines therefore need to be easily configured or adapt to the current
application conditions. This implies the need for a “Plug and Play” type architecture
for agriculture robotic systems where sensors, power supplies, communications can
be adjusted (by an non-expert user) to the current needs.
48 Product Visions
There are numerous product visions that are already elaborated for farming. From
harvesting to livestock handling. The issue in agriculture is not about vision but about
technology, and in particular dependable safe technology where performance can be
certified and guaranteed. This market his therefore driven by the availability of
technical solutions at high TRL levels and R&D&I activity will need to focus on its
delivery to market.
Near Market Activities
A primary enabling activity will be manufacturer independent evaluation of new
machines. Stimulating common testing standards and enabling national
organisations to conduct standardised tests and providing reports and certificates will
significantly help to reduce market barriers.
These tests will have different scopes. R&D&I activity needs to focus on developing
new, or extend existing, methods towards evaluating autonomous systems. Subjects
of evaluation could be:
Functionality and performance metrics
Safety certification and operating procedures.
Confirmation of adherence to standards
Suitability and usability parameters to allow assessment of compatibility with
particular farm characteristics
integration/interoperability of different machines and systems.
Assessment of cooperation between machines, and between humans and
Assessment of the expected economy benefits from utilising the technology.
2.4 Civil Domain Domain Overview
The Civil domain covers applications managed by civil authorities, national and local
government and robots operated by regional and national agencies or by contractors
engaged in public works. Many applications for robotics technology exist within the
services provided by national and local government. These range from support for
the civil infrastructure, roads, sewers, public buildings, rivers, rubbish collection etc to
support for law enforcement and the emergency services. These public services are
most often managed by civil authorities, city councils, local governments and national
government departments and agencies. The protection of the public and the efficient
maintenance of services mean that the basis for the purchase of services must show
either cost saving or an enhancement of service delivery in order to justify public
These robots will be operated/supervised by trained personnel and may be operating
in hazardous, contaminated or extreme environments where people may be at risk.
Certification and validation of operation will be important elements in the deployment
of this type of robot system.
The legal and ethical operation framework will be that of the civil authorities. This
market is broadly characterised by Business to Government (B2G) business
Applications in the Civil domain cover the provision of services related to national
civilian infrastructure these services are typically non-commercial services provided
to, or on behalf of, the public, these services are not specified or purchased directly
by the public and are often provided on a non-competitive basis.
Typical applications include civil infrastructure services, such as:
Urban maintenance and cleaning;
Civil security services (police services or boarder security agencies; including
tasks such as border and site surveillance, law enfacement, and crowd
Emergency services (fire service, ambulance and coast guard) involved in
disaster management including Search and Rescue for both rural and marine
Environmental services such as surveillance of rivers, air quality, and pollution.
The Civil Domain also covers the broader area of Scientific Support covering robotics
technology used in scientific investigations such as oceanic survey, volcanology and
geological survey.
These tasks may be carried out by a wide variety of different types of robot and
operating modality ranging from single robots or small fleets of homogeneous or
heterogeneous robots. Often robot teams will need to cooperate to span a large
workspace, for example in urban rubbish collection, and range over all environments;
in air, ground, sea surface, underwater or space. These systems are also likely have
extensive interaction with people and their environments.
Civil robots are typically purchased and operated by organisations with high levels of
technical and operational skill. In some of the proposed Civil applications robotic
systems would be deployed in hostile and complex conditions where they may need
to integrate into mixed teams of manned, unmanned and tele-operated vehicles.
In terms of the primary domain needs these can be summarised as follows:
Improvements in academic and industrial research in the fields of algorithms,
sensors and platforms creating greater levels of autonomy for Civil domain
Demonstrations and trials of higher technological maturity of domain
applications and functions. In particular; all terrain motion and sensing; all
weather performance; and the autonomous navigation and coordination of
teams of heterogeneous robots;
Progressively lower system, subsystem and component costs through
increased use of commercial-off-the-shelf components rather than custom
development of systems, while providing increased robustness and
Formulation of laws and regulations concerning unmanned systems for
example the ability to over-fly populated areas, or allow terrestrial vehicles to
share the same roads with manned vehicles and pedestrians;
Development of standards for interfaces / protocols / function, to improve:
system interoperability, payload/mission reuse and diversification. Particularly
within the different areas of the Civil domain and related domains in order to
reduce costs and establish a vibrant component market.
Raising of user awareness of the availability of robotics technologies to drive
the market;
Availability of high performance ad-hoc communications networks which are
critical for the effective integration of multiple robots. Current and Future Opportunity
Compared with the other fields of application, Civil Robotics is characterised by
having a unique combination of environments and end users. It is also characterised
by the heterogeneity of the involved agents (i.e., heterogeneous types of robot, with
heterogeneous capabilities and equipped with different sensors or effectors,
operating in mixed teams with humans).
Typical purchasers/operators of civil robots are likely to include:
Civil authorities running or contracting services that can be augmented by
robotics technology.
National governments or agencies contracting services for national projects
Public institutions, at regional, national or transnational level;
Private companies operating under contract within the Civil domain.
Organisations entrusted to public functions, such as airport or harbour
authorities, environmental monitoring agencies, airborne and space agencies;
Organisations with high levels of technical and operational skill deploying
robotics technology in hostile and complex conditions;
Organisations providing civil services where it is difficult to deploy people
(either due to safety risks, budgetary constraints, or unpopularity of the tasks to
be performed).
Some examples of Civil applications for Robotics Technology are:
Provision of civil authority services in urban areas. (e.g. Road maintenance,
rubbish collection, etc).
Monitoring and maintenance of the civil infrastructure (roads, dams, bridges,
tunnels ...);
To provide assistance in decommissioning tasks.
Environmental quality monitoring of industrial sites, harbours, rivers, lakes and
sea (monitoring air, water and ground quality);
Security monitoring of strategically important sites. (e.g., airports, energy
plants, nuclear plants, pipelines, railways, industrial sites)
Monitoring of urban environments - residential & commercial zones, civil
buildings, streets, pedestrian areas, parks, entertainment/recreation areas,
tourist sites;
Monitoring of crops and forests against fires and other natural events;
Surveillance and intervention in areas characterised by flows of people and
goods (national borders, maritime areas);
Support to human officers in operations to uphold civil law;
Surveillance and inspection of areas of historical and artistic importance;
Monitoring and intervention in disaster areas (large destroyed areas, e.g. due
to earthquake, partly collapsed buildings or sites that are dangerous to enter by
Assistance in training of personnel working in the civil sector. (e.g. Law
enforcement, emergency services, hazardous environment operations etc).
Space operations (Earth orbit and planet surface).
Ocean science exploration.
There exist compelling opportunities in the current market in environment monitoring,
surveillance and emergency services where there are already commercial products
able to satisfy some applications.
In many inspection and maintenance applications robots will need to become
intuitively integrated with human operators. Systems will be designed to complement
and act as an aid to a human mission expert. The envisioned paradigm is an easily
deployable system, able to provide the relevant information (e.g. the map of a
workspace) to the mission expert, while seamlessly and autonomously performing
the tasks that do not require operator input in the background.
This domain is also driven by changes in legislation resting to services, such
legislation is often itself influenced by advances in technical capability. For example
the European Marine Strategy, that commits each Member State to provide a
detailed assessment of the state of the environment, a definition of "good
environmental status" at regional level and the establishment of clear environmental
targets and monitoring programmes. It is possible that robotics technology offers the
opportunity of developing and marketing unmanned robots for the environmental
monitoring of coastal zones allowing nation states to implement the strategy.
Examples of future opportunity markets are for instance the long term large scale
ocean monitoring, both for security and search and rescue operations, the use of
integrated teams of multiple small autonomous aerial robots for terrestrial
surveillance (fire detection, site and border surveillance, etc.) or human activity
support (such as real time observation, communication link establishment and
It is widely recognised that robotics technology has a key role to play in the
decommissioning of a wide variety of civil infrastructures, most notably in the nuclear
and oil and gas industries. Out of the 437 worldwide nuclear plants catalogued by the
IAEA, 162 have been in operation for more than 30 years and although their life
expectancy has been extended through maintenance the problem of
decommissioning still remains. Robotics technology has a key role to play both in
extended life maintenance programmes, in the decommissioning of legacy facilities
and in the eventual decommissioning of currently active reactors.
The Fukushima disaster has also shown that it is extremely complex, risky and costly
to have human workers performing tasks in such environments. It is expected that
robotics technology can be more cost effective and safer than current methods in this
type of emergency decommissioning and containment activity.
As technology progresses in its capability systems dedicated to surveillance and
inspection will expand their function to intervene in the environment and start to carry
out maintenance and manipulation tasks. For example cleaning or decontaminating
surfaces, or effecting repairs. Barriers to Market
The major barriers to market are divided between technical and non-technical
barriers. In certain areas of application within the Civil domain the non-technical
barriers are the most significant. For example until recently it has not been possible
to fly autonomous vehicles in public air space and the use of autonomous ground
vehicles on public roads is still not permitted. These areas of regulation are currently
under review however it is highly likely that restrictions imposed by civil authorities
will be limiting on the applications that are proposed. The establishment of clear
compliance goals and testing regimes and the early demonstration of compliance
and adherence will help to progress the deployment of applications on a wide scale.
Critical to opening up public spaces to autonomous vehicles will be the execution of
large scale demonstrators able to show real world deployment of robotics
technology, firstly to prove compliance and secondly to show capability. It will also be
important to ensure that common legislation is enacted widely across Europe to
maximise the potential market, and to ensure that the cost of certification is not
prohibitive as may early operators are SMEs.
In particular regulations will need to address the significant differences between
conventional aircraft or road vehicles and autonomous ones. Autonomous systems
often enable different modes of operation, for example collective and cooperative
operation, and that decision making may not involve a human in the loop. It may also
be important for any regulation to define smaller zones of permitted operation where
the regulatory requirements are reduced in proportion to the risk.
Closely tied to the issues of regulation are issues of liability. The risk levels will be
determined by the application and by the type of vehicle being deployed. Insurance
solutions will need to be developed that match the application and market sectors
within the Civil domain.
With respect to the operation of autonomous ground vehicles there is an additional
barrier in terms of public acceptance. While robots operating in public spaces can be
seen as novel and interesting simply because of their rarity there has been no real
assessment of public attitudes towards wide spread deployment. The lack of
deployable systems makes the assessment of current public attitude problematic.
Public acceptability and the development of regulation will have to be addressed
within the deployment process if there is to be a wide scale use of robotics
technology. In the interim systems will need to be deployed in limited and controlled
circumstances where the risk can be more easily managed.
In nearly all Civil areas of application safe operation will need to be certified to
predefined levels prior to deployment. Both public and operator safety will need to be
at a high level in order to maintain a positive public perception of robotic deployment.
The use of robotics technology in the marine environment is more well established
and the barriers to market are significantly lower than other Civil areas of application.
However the treatment of autonomous surface vehicles close to the coast or in rivers
may require review but there is some acceptance of them as either floating wrecks or
piloted craft within current regulations.
In addition to these significant non-technical barriers there are also a number of
significant technical barriers to the deployment of robotics technology in the Civil
domain. These barriers range across the technology spectrum from limitations on
operation time caused by insufficient on-board power storage, to the need to
correctly interpret scenes and human actions in order to make the correct
autonomous decisions. It is expected that these limitations will shape the early
market for Civil applications but that despite these technical limitations there are a
number of application areas that can be impact on by 2020.
The lack of regulations for small aircraft has restricted the development of the aerial
robotics market. New regulations for Light Unmanned Aerial Systems (LUAS) or Very
Light Aerial Robotic Systems (VLUAS) already developed or being developed in
many countries are starting to remove this barrier. In Europe, the United Kingdom
Civil Aviation Authority published in 2002 the CAP722, the UK policy for the
certification and operation of UAV Systems, both military and civil. Since publishing
CAP722, the CAA has further reviewed and developed its UAV policy, both in the
light of recent experiences and as a result of changes in regulatory responsibilities
since the formation of the European Aviation Safety Agency (EASA). CAP722 last
issue was published in 2012 (5th edition), taking into account legal, certification,
spectrum and security issues. Several other countries have developed similar
regulations. Key Market Data
The application of robotics technology to the Civil domain is still at an early stage and
it is therefore difficult to estimate eventual market size. Its is likely that technology
limitations will restrict early deployment to well controlled areas of application where
robots are operated by skilled personnel for example in nuclear and environmental
inspection tasks, including marine inspection.
In the case of marine robots, the Remotely Operated Vehicles (ROV) market is
expected to grow at near 14% CAGR (compound annual grow rate) in the period
2011-2015 up to a value of about $1,546 million in 2015. ROVs sales for defence &
security and scientific research equalled 25% of the total market for each sector. In
the meantime, also the Autonomous Unmanned Vehicles (AUV) market is expected
to grow in the defence and scientific research sectors with a CAGR equal to 12%
and 8% respectively by 2016.
It is often the case that unmanned vehicles are cheaper and faster to produce than
manned vehicles. Global Unmanned Marine and Ground Vehicles market is foreseen
to reach $1.96 Billion by 2017. The potential market in Europe for Unmanned Aerial
Vehicles over the next 10 years could amount to about €11B. In the short to medium
term demand is likely to be driven by monitoring and surveillance applications.
The coming decade will probably witness the rapid expansion of decommissioning
activity, costing tens of billions of dollars. The decommissioning industry’s
performance will be critical to the future of nuclear power generation.
The decommissioning sector has been steadily forming over a few years but it is
expected to see some major progress over the next five to ten years. Hundreds of
offshore oil and gas platforms will be recovered from the North Sea over the coming
years. Analysis by industry body Oil and Gas UK and decommissioning agency
Decom North Sea put the value of this work at £30 Bn. over the next 25 years.
Key Market drivers are:
Growing interest in UAS not only by US and European countries but also by
emerging countries.
Potential for improved coverage of large areas for environmental monitoring.
Increase in quality of monitoring data and regularity of monitoring due to lower
cost per task.
Reduction of total operational costs with respect to existing manned systems.
Increasing acceptance of robotics technology. Relationship to other Domains and Markets
The Civil Robotics domain, has many relationships with the following domains listed
in the SRA:
Commercial Robots
Logistics and Transportation
Military Robots (note: it is not intended that programmes developed under the
PPP or Horizon 2020 will specifically address this area).
In a number of application areas these domains share key abilities targets and
technology requirements. There are also common requirements in terms of systems
design technologies and safety certification in particular.
In addition, Civil domain applications may provide added value to robotic systems
serving other domains. For instance, an agriculture robot (Commercial domain) may
use terrain data built by a network of UAVs operated by a national mapping agency
(Civil domain).
In terms of relationships to robotics markets there are strong links to the marine
robotics market, to teleoperated robotics, aerial and space robotics markets. Europe's Place in the Market
In maritime applications Europe has an established position in the market, in
particular for underwater systems. The global market is currently dominated by US
companies, although European companies have leadership and good market
positions both in specific robot development or in the supply of subsystems (as an
example one can cite the UK's SMD leading in deep ocean trenching robots or
Norway's based Kongsberg with AUVs and subsystems ).
If Europe is to gain a greater share of this market then the “dual use” position of the
US government agencies must be echoed within Europe to ensure that technology
transfer into the Civil domain is fully enabled.
In ground based robotic vehicles Europe represents a growing market despite a
strong US market lead. As in the marine domain investment is needed to establish
and grow the European industry.
In aerial vehicles there are a large number of SME’s operating within Europe
providing a wide range of small to medium scale systems. For example The Hearing
on Light Unmanned Aircraft Systems (LUAS) (Brussels, October 2009) listed 252
unmanned aerial systems with a Maximum Take-off Mass lower than 150 Kg.
There is also good progress towards the opening of airspace to remote operation
and it is expected that this sector will expand significantly to 2020. R&D&I investment
is needed to enhance the existing technology produced within Europe and secure a
slice of the global market. Key Stakeholders
Europe has a number of well established companies contributing to the growth of the
Civil domain across all areas of application. In many cases the larger organisations
have a strong background in either the military energy sectors, which is where the
majority of historical expenditure has been. There are a growing number of SMEs
operating in this sector specifically in the small scale aerial surveillance sector
addressing environmental monitoring and surveying applications from agriculture to
building inspection and growth in these sectors is expected to be strong.
Within Europe there is strong expertise in the nuclear industry and in civil
infrastructure applications of robotics with a strong bias towards the marine industry.
This commercial market is also served by various research centres and university
laboratories dedicated to Civil application areas, with a historic bias towards marine
With the development of certification programmes, particularly in the Aerial sector,
regulatory authorities are becoming key stakeholders and gate keepers for the
growing industry. Current Key Projects
The following is a non-exhaustive alphabetical summary of important research
projects and initiatives related to the Civil Robotics market. Due to space limitations,
details of each project are not included; they can be readily found on the Internet.
Cognitive Autonomous Diving Buddy
Cooperative Autonomous Robotic Towing system
CFD OctoProp
Computational Fluid Dynamics Aided Design of the
Propulsion and Locomotion Systems of a Bioinspired Robot
CoLlAborative eMbedded networks for submarine
Cooperative Cognitive Control for Autonomous Underwater
COnservazione programmata, in situ, dei Manufatti
Archeologici Sommersi (Planned conservation, “in situ”, of
underwater archaeological artefacts)
Control for Coordination of Distributed Systems
Robotic FIsh LOcomotion and SEnsing
Floating Sensorised Networked Robots for Water Monitoring
Integrated Components For Assisted Rescue and Unmanned
Search Operations
Marine Autonomous Robotics for InterventionS
Marine INspection rObotic Assistant System
Marine robotic system of self organizing, logically linked
physical nodes
autoNomous, self Learning, OPTImal and compLete
Underwater Systems
Persistent Autonomy through Learning, Adaptation,
Observation and Replanning
Intelligent Platform for Multimodal Characterization of the
Seafloor and Submerged Structures
Search and monitoring of Harmful contaminants, other
pollutants and leaks in vessels in port using a swarm of
robotic fish
Smart Unmanned aerial vehicle sensor Network for detection
of border crossing and illegal entrY
Marine Robots and Dexterous Manipulation for Enabling
Autonomous Underwater Multipurpose Intervention Missions
Underwater acoustic networks
Veicolo Filoguidabile per l’Ispezione, la Detezione e
l’Esplorazione Subacquea) Underwater vehicle, optionally
wireguided, for inspection, detection and exploration)
57 European Products
There are a wide range of both marine, ground and air systems produced within
Europe. However many of these are low volume or bespoke products developed to
meet specific needs. The global market is not yet established for mid to small scale
aerial systems and the variation in legislation makes compliance approvals on a
global scale difficult to achieve. In the marine sector there are some significant
products with good commercial track records, particularly in the oil and gas sector. In
nuclear decommissioning there is considerable potential within Europe and a number
of SME’s are engaged with specific projects, however the global market is not yet
Civil Sub-Domains:
2.4.2. Civil Infrastructure Sub-Domain Overview
This sub-domain represents a large and growing area of application for robotics
technology. There are currently two primary areas of application that have attracted
interest over the past decade; Decommissioning and environmental monitoring. The
decommissioning and inspection of hazardous infrastructure notably in the energy
supply sectors is a primary area of application for robotics technology. Current Opportunity
In the near term the use of autonomous inspection systems has the potential to both
reduce costs and increase the thoroughness of inspections particularly of tall
industrial structures such as chimneys by reducing the need to scaffold them.
Historic building inspection should also benefit from reduced closures and quicker
Bridges and tunnels require continuous monitoring much of which is carried out using
built-in infrastructure however older structures need regular inspection and this can
result in closure and subsequent transport disruption. The use of autonomous
systems may reduce closure times, or provide better early warning of issues within
structures, allowing better long term planning.
The same technology can also be used for environmental monitoring ensuring
pollution targets are met, monitoring the source of pollutants and inspecting water
and air quality. There is an opportunity to provide services to survey installations
both in terms of assessing physical infrastructure but also in terms of resource
usage, for example the effectiveness of heat insulation, or the assessment of raw
material quantities, for example by accurately assessing roof area, or the area of a
surface to be coated.
The civil infrastructure now extends into space and communication satellites and
earth observation satellites are part of a vital communication and monitoring
infrastructure. Robotics has the potential to provide maintenance and
decommissioning services in this domain.
Robots have been deployed in the nuclear industry for internal reactor inspection in
hazardous environments reducing human risk levels. Future Opportunity
Future opportunities lie in the development of control and user interface systems that
allow rapid data gathering and assessment. The full autonomous inspection of
external infrastructure may become a possibility in the near future. It is expected that
future systems will be able to enter hazardous environments and carry out
maintenance and repair tasks that maintain the operation of existing infrastructure
and reduce unknown risks. Ultimately the goal is to use robot systems as a
significant element in the safe decommissioning of hazardous infrastructure.
Across Europe there are plans to close up to 80 civilian nuclear power reactors in the
next ten years. While many of these reactors are likely to have their operating
licenses extended, they will eventually be decommissioned. Under a recent EU
Directive establishing a Community framework for the responsible and safe
management of spent fuel and radioactive waste, all Member States are to ensure
that funding resources are available for decommissioning. At a global level the need
to have adequate resources available for decommissioning is being addressed by
the Joint Convention on the Safety of Spent Fuel Management and on the Safety of
Radioactive Waste Management. Key Market Data
Annual exploitation costs for the 58 French reactors were estimated to 8.9 billion
euros in 2010. At the same date, dismantling costs were estimated to 18.4billion
euros. In 2013, Europe had 185 nuclear reactors taken both operating reactors and
those under construction. Cost estimates for decommissioning in the UK are
currently estimated at £50 Bn over an extended time period. Germany also has
similar nuclear infrastructure that required decommissioning. While these costs cover
the whole process of decommissioning the potential utilisation of robotics technology
may represent some 10-15% of these costs.
The Brent North Sea oil field was one of the UK's earliest and largest oil & gas
development projects, with all four platforms (Alpha, Bravo, Charlie and Delta)
coming on-stream in 1975-76. Brent Delta ceased production at the end of 2011
Decommissioning of all four platforms could take as long as ten years.
Decommissioning is also taking place in the Ekofisk field, off Norway. Relationship to other domains
Close relationship to the commercial sector and in particular the Service and Utilities
domain. There is also strong linkage to energy supply companies and to national
regulatory bodies.
The decommissioning of infrastructure provides a close linkage to the Construction
and Demolition domain.
With regard to the transport infrastructure there is linkage to the large civil
engineering companies and to the dedicated inspection services, to the national
transport authorities, both road and rail, and to companies maintaining the transport
The application of robotics technology in the maintenance of space based systems
and satellites provides linkage to trans-national space agencies and to the
organisations engaged in the commercial and scientific use of space technology.
2.4.3. Search and Rescue Sub-Domain Overview
The use of robots in search and rescue, both over wide areas such as at sea, or in
closed spaces such as buildings is widely seen as providing a significant increase in
the likelihood of success in locating trapped or missing people. The effectiveness of
a single helicopter carrying out a wide area sea search can be scaled through the
use of multiple coordinated guided search systems both at sea and in the air. A
single operator will be able to monitor a more extensive search area in a shorter time
at lower risk with autonomous assistance to search for assets and personnel,
particularly in hazardous conditions. Current and Future Opportunity
Experimental systems have been deployed to carry out search operations after
natural disasters. The deployment of undersea systems has had notable success.
The uptake of search systems has, to date, been low and the potential to deploy
robot systems requires further investigation and exploitation.
The scaling up to high TRL levels of collaborative systems able to autonomously
scan large areas during search and rescue operations, and the ability to create maps
of spaces and identify voids in collapsed buildings will significantly improve search
and rescue outcomes.
The use of multiple robots providing coordinated search in unknown and dynamic
environments that are typical of disaster zones could provide enhanced safety to
rescue workers and increase the likelihood of discovering victims and identifying
threats and hazards. However investment in the development of these systems may
depend on other market domains developing the initial technology, notably the civil
infrastructure and military domains.
There are potentially significant safety gains that may occur with a one to one
collaboration between a searcher and a tele-operated semi-autonomous robot used
to enter buildings and carry out search and possibly rescue tasks. The search
function alone can have significant impact in that the robot will be able to reach
spaces and regions of a building that a human operator may not, and it may be able
to move faster and with significantly lower risk. On finding a person its internal map
of the space can be used to plot the optimal route to effect a recovery. In more
advanced systems the robot may be able to provide basic medical assessments and
even basic medication (for example pain relief) increasing survivability. Even simple
tasks such as delivering water to earthquake victims trapped in inaccessible spaces
could significantly increase survivability chances. Relationship to other markets
There are strong links to similar functions in the military domain and to other
emergency services functions. There will be links to equipment suppliers and
considerable shared technology with civil sector companies engaged in
environmental monitoring. Europe’s Place in the Market
Europe has a number of experimental systems and the rescue equipment industry
needed to support their eventual exploitation.
2.4.4. Environment Sub-Domain Overview
Monitoring the environment and providing up to date information about changes can
provide early warnings that allow faster and more effective responses to hazards,
and to long term changes in the environment. The ability of robotics technology to
provide multi-modal data accurately mapped to terrain data makes it a valuable data
collection tool. This data often has value in its own right, to farmers, civil authorities
and utility supply companies. The potentially low cost of performing this type of
environmental monitoring will accelerate the development and deployment of such
systems and enhance those services that rely on this data. Current and Future Opportunity
Potential applications range form crop monitoring to building inspection, pollution
control to water quality monitoring. Almost all aspects of the environment can be
monitored by using robotics technology as the main means of mapping an area. Barriers to Market
The main barriers to market are regulatory based on the restrictions on the use of
autonomous vehicles in public spaces and in the air.
2.4.5. Law Enforcement Sub-Domain Overview
Using robots for law enforcement is an area of application that will require extensive
ethical and legal debate. For this reason it is not seen as a short term area of
application with the exception of tracking and monitoring. There are clear benefits in
the use of autonomous systems for the pursuit and tracking of people using multi
modal systems just as there are in search and rescue and in environmental
monitoring. Some civil security forces are experimenting with remotely guided air
vehicles for information gathering. Current and Future Opportunity
The use of multiple aerial vehicles as surveillance platforms that can monitor ground
movements often without attracting attention, and the use of water based boarder
patrols augmented by fleets of monitors and ground patrol robots on boarders are
obvious application areas.
These applications carry ethical and legal issues related to privacy, legal rights and
the admittance of automatically collected data in law.
At some point in the future a critical boundary will be crossed where a robot is
designed to physically engage a subject in much the same way that dogs are
currently trained to do. The impact on civil rights and the establishing of legal control
over the robot will require wide spread and careful analysis in order to avoid the
controversy that military drones are currently undergoing. These robots have the
potential to negatively alter the perception of robotics technology in the public eye. Relationship to other domains
There will be shared technology between this domain and the Search and Rescue
domain and Civil infrastructure domain. There are strong links to the emergency
services domain. There is likely to be significant technology transfer from the military
2.4.6. Emergency Services Sub-Domain Overview
While some aspects of the emergency service use of robots is covered in other subdomains, notably law enforcement and search and rescue there are a number of
application areas that do not fall into these other categories. In particular firefighting,
hazard reduction, pollution control, and the provision of emergency aid and
assistance. Current and Future Opportunity
Tele-operated fire hoses may be able to reach closer to a fire than a human and may
be able to sustain operation for longer at higher temperatures, or in situations where
there are other hazards such as the risk of building collapse, or toxic fumes.
In pollution control autonomous systems may be able to deploy barriers, dispersants
and absorbers more quickly by exploiting multi modal collaboration to both monitor
and deploy dynamically as a disaster unfolds. Early intervention has the potential to
reduce disaster impact and cleanup costs.
In the medium term emergency service workers may also benefit from “buddy”
systems, either exo-skeletons that increase lift capability, reach, or companions that
jointly perform collaborative tasks.
In the future the use of remote surgical robots may provide immediate assistance for
traumatic injury. Ultimately Robots may also be deployed to lift and support accident
victims during extraction from accident sites. Relationship to other markets
Clear and strong links to search and rescue domain and to law enforcement.
Eventual possible links to the Healthcare domain.
2.4.7. Science Support Sub-Domain Overview
The domain of science support covers a wide range of different activities that relate
to scientific enquiry. Robots in this domain are often highly specialised research tools
designed for a specific purpose. These robots are often made singly and may be
regularly upgraded during their life time. This is the domain where the public are
most likely to have encountered robots. The Mars rovers, and deep sea explorers
such as Alvin have high public profiles and act as technical ambassadors for
While these robots may have a degree of semi-autonomy they are typically teleoperated. The requirements for extremely high levels of dependability and the high
cost of replacement means that control strategies are extremely cautious. Current Opportunity
This is a mature market with well established suppliers and its roadmap is driven by
large scale research funding, for example trans-national space or marine research
The use of robots for long term environmental monitoring is still in its infancy. The
use of robots for the monitoring of pollution and resources carried out for research
purposes is still at an early stage of development.
A secondary area of scientific support robotics is in the supply of research robots to
the robotics community itself. These can take the form of kits, modules or whole
systems including software and development infrastructure. This is a growing
62 Future Opportunity
There are numerous areas of application for science support notably in:
Wildlife monitoring
Deep space exploration
Planetary rovers
Deep sea exploration Relationship to other markets
The developments in this domain often feed other more commercial domains. The
uniqueness of the the robots means that almost every aspect of them has to be
specially developed.
2.4.8. Key System Ability Targets
The system abilities for Civil Robotics are those that enable a robot or a team of
robots to endure loosely supervised missions in large unstructured scenarios:
interaction, dependability, perception, autonomy, navigation, motion capabilities,
cooperation with other robots or humans and, to an increasing extent, cognition. Configurability Adaptability
Real time real-world learning
Adaptability: Level 4 - Task adaptation Interaction Capability
Cooperative behaviour limited to specific
Cognitive Human Interaction: Level 2
Task context interaction.
Robot and human-robot teams, full
cooperative behaviour
Cognitive Human Interaction: Level 3
Object and location interaction
combined with Human robot Interaction
Level 4-6 and Robot robot interaction
Level 4-5. Dependability
All-weather missions
Dependability Levels 4-5
Long Term (Permanent) Deployment
Dependability Levels 4-5
Long Range Deployment
Dependability Levels 4-6 Motion Capability
High speed and agile autonomous driving
on uneven and sloping terrains
All terrain high speed and dexterous
autonomous driving
Motion capability: Level 10 Dynamic
motion Manipulation Ability
Collaborative robot-robot and human-robot
manipulation (e.g., load sharing)
Human Robot Interaction: Level 2
Direct physical interaction
Robot robot interaction: Level 5 Team
Mobile manipulation on uneven sloping
terrain and with floating robots
Motion capability: Level 10 Dynamic
motion combined with Location
perception: Level 5 Object coupled
location combined with Decisional
autonomy: Level 8 Dynamic autonomy. Perception Ability
Coarse scene classification; update of the
model based on observations
Scene perception: Level 4 Multiple
object detection combined with Object
recognition: Level 4 Object recognition
- one of many.
Operation possible in most weather and
environmental conditions;
Object Recognition Parameter:
Perception Ability Target: Immunity to
natural variations.
Detailed scene classification &
understanding; build high-level abstract
Scene perception: Level 5-6 combined
with Object recognition: Level 6-9.
Cooperative Mapping (breakthrough in
communications required)
Cognitive Action Ability: Level 5
Knowledge driven action combined
with Robot robot interaction: Level 5
Team coordination. Decisional Autonomy
Pre-planned missions; medium complexity
tasks; limited human supervision;
integrated planning among heterogeneous
fleets of manned and unmanned vehicles
Decisional Autonomy: Level 4 - Simple
Re-configurability of robot task depending
on the changed environmental conditions
Decisional Autonomy : Level 6 Constrained task autonomy.
Adaptability to 3D Structured or
unstructured underwater environments
Intelligent “Motivation Dynamics” with
temporarily changing priorities (situationspecific priorities)
Decisional Autonomy: Level 9 Mission oriented autonomy
High complexity tasks performed
autonomously and in cooperation;
collective behaviour; no human in the loop
(human monitoring only); opportunistic
mission planning capability, goal-based
Decisional Autonomy: Level 9 Mission oriented autonomy combined
with Cognitive Action Ability: Level 8-9
Re-configurability of more robots working
cooperatively, reassignment of task
domain/goals between robot
Robot Robot interaction Level 5-6,
combined with Decisional Autonomy:
Level 9 - Mission oriented autonomy
combined with Cognitive Action Ability:
Level 8-9 Cognitive Abilities
Interpretation of scenarios of limited
complexity taking into account different
Interactive prediction of dynamics
Adaptability: Level 2 Multiple
parameter adaptation.
Wide comprehension of scenarios taking
into account different/conflicting inputs;
Reasoning Ability: Level 4 Reasoning
with conflicts.
2.4.9. Key Technology Targets
Key technology targets for the Civil domain can be categorised in to three different
Safety by design: The need to provide safety certification for systems operating
in the Civil domain will be driven by advances in systems design methods that
integrate safety into the design cycle such that safety performance can be
guaranteed by design rather than proof of operation post design. Establishing
acceptable levels of safety for each type of application and application area will
also be a key part of design capture
Operation in unstructured and dynamic environments: The unstructured nature
of the main operating environments for the Civil domain require significant
improvements in perception both in unstructured 3D terrains, under water and
on the ground, and in terms of handling the effects of extreme weather
conditions on perception and localisation. The unstructured and often dynamic
nature of the operating terrain also provides significant challenges in the design
of mechanical and motion control systems able to operate on unstable and
rough terrain while maintaining sufficient localisation.
Cooperative missions: In a number of potential application areas the Civil
domain will require teams of robots to carry out tasks and will therefore rely in
the underpinning technologies for the command and control of teams of both
heterogeneous and homogeneous robots. Including the mixture of robot teams
with human teams working in close collaboration. In more advanced
applications scene interpretation and cognitive interpretation of both object and
environment in play an increasingly important role.
The following lists specific technology targets that are relevant to the Civil domain: Systems Development
System Design
Development of standard architectures.
Design tools for the integration of robots in wider pre-existing systems.
Design of common platforms.
Long Term
Standardised and certified platforms and development tool chains for mission
critical and safety critical operations.
Standard interfaces and systems for the deployment and retrieval of remote
Systems Integration
Development of interoperability standards for robotics components
Development of system interoperability standards for robot and human teams
Self-configurability of off-the shelf robotics components
Modelling and Knowledge Engineering
Mechanisms for scene analysis and knowledge acquisition based on the
perception of simple scenarios
Mechanisms for scene analysis and knowledge acquisition based on
distributed perception in teams
Development of methods tools and techniques, for knowledge representation
in domains of low and medium complexity
Development of methods tools and techniques, for knowledge representation
in in complex domains and scenarios Mechatronics
Mechanical Systems
Automatic buoyancy control systems
Appropriate miniaturisation of current capable systems leading to reduced
equipment and deployment costs,
Commoditisation of common parts and systems,
Simple and reliable systems for the deployment and retrieval of marine robots.
Design of energy efficient autonomous robots for specific environments
Multi-functional/multi-task and flexible end-effectors.
Development of sensors for navigation
Development of sensors for the reliable detection of people
Acoustic sensors for marine remote sensing and map building applications.
Specific new low maintenance and low cost chemical sensing mechanisms to
enable the use of robot systems for environmental protection tasks.
Energy efficient propulsion systems in multiple environments.
High efficiency miniaturised underwater propulsion systems
Deep ocean propulsion systems
Power Supply and Management
High-density energy systems
Energy management systems
Fuel cells for underwater applications
Ad hoc robust (broad-band and out of line of sight) communication through
different media, technologies and capabilities (i.e. by means of fleets of
Robust communication & localisation systems for underwater applications.
Advanced composites for vehicle hull and container fabrication
New materials for deep water and water column exploration (such as variable
forms for hydrodynamic task adaptation, reduced weight and lower
deployment costs)
Integrated vehicle-arm control and vehicle stabilisation for mobile
manipulation on uneven sloping terrains and floating robots
High speed autonomous off-road path following and obstacle avoidance
Cooperative control of multiple heterogeneous platforms, including air,
surface, and marine robots.
Low cost, medium performance, integrated sensors for accurate guidance &
control. Human Computer Interaction
Human Machine Interface
Augmented reality tool for the remote operation and interaction with
unmanned vehicles
Systems for seamless mission specification and mission programming.
Systems for mission follow-up and post-mission analysis
Tools an ergonomics for reducing remote-operator workload and stress
Natural human machine interface methods Perception
Scene interpretation fusing different sensor modalities; multi robot perception
in heterogeneous teams (both in terms of platform and sensors).
Video and acoustic imaging data fusion for underwater applications.
Distributed multi sensor fusion; Sensing/Control/Planning integration (i.e., plan
to sense to control) Navigation
Large scale mapping in dynamic environments, capable of handling vast
areas of operation and supporting navigation for extended periods of time.
Automatic underwater pollution mapping.
Sea bottom/sub bottom mapping.
All weather map management and update
Precise localisation in difficult environments such as indoor GPS denied
scenarios or underwater will allow the use of systems in a wider variety of
Cooperative localisation/geo localisation with robot teams.
Motion Planning
Planning with kinematics and environmental constraints
Real-time planning with kino-dynamics and environmental constraints
Systems for cooperative, multiple vehicle motion planning in the presence of
environmental disturbance and obstacles. Cognition
Cognitive Architectures
Development of architectures and models for representing and implementing
cognitive tasks of low to medium complexity
Development of Architectures and models for representing and implementing
cognitive tasks of high complexity
Development of methods, tools and techniques for modelling cognitive tasks
of low to medium complexity
Development of methods, tools and techniques for modelling cognitive tasks
of high complexity
Learning Development and Adaptation
Adaptation with respect to changing environmental conditions
Adaptation to large vehicle parameter variations
Knowledge Representation and Reasoning
Development of methods, tools and techniques for knowledge based
reasoning in domains of low to medium complexity
Development of methods, tools and techniques for knowledge based
reasoning in complex domains and scenarios
Action Planning
Advanced systems for multiple vehicle cooperative task and mission planning
Highly abstracted mission definition and mission/task planning algorithms for
interaction and operation with untrained users;
Systems for operator-assisted manipulation
Systems for autonomous intervention in underwater structures
Systems for cooperative grasping and transportation of heavy objects.
Natural Interaction
Cooperation and interaction among air, ground, surface, and underwater
Systems for human-robot interaction and mixed team operations
2.5 Commercial Domain Domain Overview
This domain covers the use of robots working as part of a commercial process. As
with the other market domains these are well established markets where there is an
opportunity to employ robotics technology.
The Commercial domain covers a wide range of different markets, two of these
major sectors have already been highlighted elsewhere in the MAR (Manufacturing
and Agriculture) so the purpose of this section is to detail other sub-domains where
robots can have an impact on commercially led processes.
These robots are most likely to be operating within a work environment. They will be
operated by skilled workers but may also come into contact with the general public
through the work environment.
These robots have a single overriding goal, they must be cost effective. This can
come about in many different ways for example by carrying out a process more
quickly, or with less wastage than a human worker, or by working with a person to
extend their skills, perhaps by reach or load capacity.
These robots work as part of a commercial process; manufacturing goods, providing
service functions within a commercial organisation, or operated by a commercial
organisation. Most often they will be operated by trained personnel, operating with or
in cooperation with other people in a work environment.
The legal framework of operation is that of the work environment, be that a farm or a
factory. This high level domain is characterised by Business to Business transactions
(B2B). Current and Future Opportunity
The sections on Manufacturing and Agriculture detail many of the key opportunities
in this domain and these are currently seen as being the main growth areas in terms
of goods production assisted by robots.
In a number of areas within the Commercial sector robotics technology provides and
enhanced set of tools to allow existing and well established industries to operate
more efficiently. The gaol of R&D&I funding must be to demonstrate the opportunity
in order to enable to large take-up and adoption of robotics within these large scale
Mining and Minerals
There is a long standing use of robots and remote guided vehicles in the oil and gas
sectors and more recently in mining. Many of the Mining and Mineral industries
operate within hazardous environments and the extraction of earth resources is often
limited by the level of risk associated with human working conditions. There is a
significant opportunity to utilise robots for extraction in order to reach more
inaccessible mineral resources. In particular there are considerable mineral
resources on the deep ocean bed where robots could provide the solution to long
term and viable extraction. Working in flooded mines may also be possible.
The significant cost of failures and the potential environmental impacts make this an
area of application that demands high levels of reliability and dependability, but the
high value of the infrastructure enables appropriate funding.
Utilities and Service
The utilities industries, power generation, water, gas and electrical supply all have
high value assets that must remain operational 24/7 in order to minimise costs.
Down time is costly in terms of lost capacity. In all of these industries the assets are
widely spread geographically and although monitoring systems are in place regular
inspection is a key part of maintaining operational integrity. Robotics technology has
a key role to play in providing continuous inspection capabilities and in the future
both inspection and repair (possibly in advance of failure) could be carried out by
In particular the robot monitoring of power lines and robot assisted repair has been
trialled over a number of decades, it is possible that new approaches may eventually
make this a viable field.
The installation of new services using underground robots has the potential to reduce
installation cost and time and decrease disruption to supply. Using multi-modal
information sources and context aware sensing to detect other unknown services or
errors in existing utility maps can help to reduce delays that can often cause the replanning of new installations.
Construction and Demolition
The construction industry is highly cost competitive, faster time to completion is a
key driver in the adoption of new working methods. Robotics co-working in
construction has the potential to speed up construction and at the same time enable
new ways of building that may be more cost effective. For certain competitive
construction tasks such as road or rail construction a higher level of automation may
be effective. In the demolition industry there are similar drivers, decreasing time to
completion and correctly handling the deconstruction of complex buildings at speed
may drive the adoption of robotics technology.
Inspection and Monitoring
There are many different types of inspection an monitoring industry, ranging from the
inspection of manufactured parts to the inspection of buildings and land usage.
These commercial organisations range from equipment suppliers to individual
inspectors and service providers; for example the inspection of domestic drains.
Robotics technology has the potential to significantly impact all aspects of this
market from the automated inspection of large manufactured parts, to remote
camera systems used by builders to assess a roof repair more quickly and
In an entirely different commercial sector, marketing, robots have been used as
visual icons and as the focus of attention for decades, in fact some of the earliest
robots were constructed for advertising purposes. Modern robotics technology is also
used extensively as a way of demonstrating technical sophistication for high tech
companies. However the use of robotics in marketing has the potential to take on a
much more direct role where roving vending machines service clusters of people,
and robots are used to provide assistance and information within shopping
environments. Such systems may range from animated advertising to interactive
systems demonstrating product use.
72 Key Market Data
The commercial domain is very broad based and this makes it difficult to assess total
market value. Limited information is available for certain parts of this domain and this
is detailed below.
For the inspection of industrial plant significant costs are involved. The most
expensive aspect of an inspection is taking the asset off-line and the related lost
production. The required off-line time may range between one day and several
weeks. For a refinery typical costs associated with this down-time may reach over
$10 million per day in lost production. In a practical real-world exercise it was
estimated that 80% of the time spent in the context of an internal pressure vessel
inspection was related solely to human entry of the vessel. Also in terms of human
safety, from time to time accidents occur that are related to human entry of confined
spaces and the erection of scaffolding.
As similar opportunity arises in the inspection of power generation plants where for
example, the inspection of many plants can only be performed when the plant or
critical subsets are stopped. In a 300 MW power plant the deduction of the outage
gains is about €0.75 M per day.
It is clear that robotics technology could have a major impact on these figures
warranting significant investment in R&D&I. Relationship to other markets
There are strong links to the system deployed in the Civil domain and it is
reasonable to expect common platforms and modules to be developed for both
markets. Europe’s Place in the Market
Europe has significant investment in the oil and gas sectors and a history of using
robotics technology in inspection tasks.
Europe has invested in R&D&I, particularly in marine robotics and is well placed to
exploit the global market. Key Stakeholders
All of the Commercial sub-domains are related to strong existing industries with
significant presence across Europe. Critical to the development of this sector is the
engagement of these industries in addressing R&D&I to enable the benefits of
Robotics Technology within their respective industries.
Within each of these industries are multiple tiers of producers and service providers
and each layer in this structure will need to engage with the opportunity.
In many cases regulatory bodies are also significant stakeholders in that safety
legislation and quality standards are often limited by technical capability.
In some sectors there are embedded methods with a low incentive for change, the
step change impact of robotics technology will be highly disruptive and is likely to
create significant change.
3. System Abilities
Robot systems operate through the integration of a wide range of different
technologies. In addition to the characterisation of each of these technologies it is
also important to characterise the overall performance of the system. This set of
system abilities capture the important system level performance of robots. Abilities
allow the sate of the art to be identified future targets to be set for robot systems.
The different system abilities are defined in a way that is independent of any
particular robot configuration or market domain. Abilities provide a basis for setting
performance metrics and for application providers to specify desired levels of system
Each ability captures one specific aspect of the operation and behaviour of a robot
system. For each different type of robot and application there will be critical abilities
that can be identified. By establishing the state of the art it is possible to benchmark
progress and target R&D&I activity towards next step targets. The list of abilities is
intended to cover all the different types of ability that robots possess.
Within the Roadmap each ability is described together with the current state of the
art, and the expected targets that might be reached by 2020. The Roadmap provides
extended detail and explanation of these targets. Cross referencing these targets
with the ability needs identified in each domain provides an insight into where R&D&I
activity may have domain impact. Similarly different capabilities in each technology
will impact on the abilities and step changes in technology can be expected to impact
on key ability targets.
3.1 Configurability Description
The ability of the robot to be configured to perform a task or reconfigured to perform
different tasks. This may range from the ability to re-program the system to being
able to alter the physical structure of the system. (e.g. by changing a tool).
Configurability applies to different aspects of a robot system:
To the configuration of software modules and components
To the configuration of sensing and other electronic systems
To the configuration of mechanical structures of the system.
The ability to configure a robot system must be designed into the system. In most
cases software systems inherently contain a degree of configurability, electronic and
mechanical systems require configurability to be designed in.
Configurability must be carefully distinguished from Adaptability and Decisional
Autonomy which relate to how a robot system alters its responses (Adaptability) and
how it changes its behaviour as it performs an operating cycle. Current Technology Drivers
The following technology areas currently impact on the configurability of a system
System Design
System Architecture
Mechanical Systems
Human Machine Interface Ability Levels
Mechatronic Configuration
There are some robot systems that contain modular components allowing
mechatronic configuration prior to the operation of the robot. Typically these modules
are assembled into a form specific to the task.
Level 0 - Static Configuration
The configuration files or mechatronic configuration are set prior to installation and
cannot be altered by the user.
Level 1 - Start-up Configuration
The configuration files, or the mechatronic configuration can be altered by the
user prior to each task in order to customise the robot system in advance of each
cycle of operation.
Level 2 - User Run-time Configuration
The configuration, both in terms of software operating parameters and
mechatronic configuration can be altered by the user during the cycle of operation.
For example a robot may have an end effector changed part way through an
operating cycle.
Level 3 - Run-time Self Configuration
The system can alter its own configuration within a pre-determined set of
alternative configurations designed into the system. For example it can change its
own end-effectors, or alter configurations based on the set of plug and play
modules in use.
Level 4 - Autonomous Configuration
The system can alter its own configuration in response to eternal factors, for
example altering its morphology in response to the failure of a sensor or actuator.
Note that altering configuration must be carefully distinguished from actions taken
as a part of the normal autonomous operation of the robot system.
Configuration mechanisms
Configuration of a system can take place via a number of different mechanisms. For
the higher ability levels more than one of these mechanisms will be used.
Design time configuration. Configuration settings are fixed as the system is
Configuration files are used to set configuration parameters in software. These
may also impact on mechatronics via the controllers in the system.
Skilled operator interaction. A skilled operator is able to alter the software
configuration or mechatronic configuration.
Unskilled user interaction. An unskilled user is able to alter the configuration.
Configuration is automatically set by the arrangement of plug and play modules
in the system (software and mechatronic).
Remote Communication of configuration (Single system). The system
configuration is communicated remotely to a single robot system.
Remote Communication of configuration (Multiple Systems). The system
configuration is communicated remotely to a set of robots. Ability Targets
The following ability targets for configurability have been identified:
Design time configuration: The development of system design processes and
methods that enable and promote system configuration as a part of the design
process. The development of standard means by which design time
configurations can be captured and altered.
Verifying configuration validity: The development of system design processes
that enable and promote system configuration by ensuring that consistent,
safe, and dependable configurations can be validated at run time.
Plug and play standards: The development of standardised interfaces, both
software and mechatronic that can support configuration at each of the
identified levels but particularly in the development of plug and play systems.
User configuration interfaces: The development of configuration interfaces,
both HMI, software and mechatronic, that allow unskilled users to alter the
configuration of a robot system while maintaining consistent, safe and
dependable operation.
Mechatronic configuration: The development of configuration systems able to
handle a diverse range of impacts on system operation caused by significant
changes in mechatronic and mechanical configuration. For example surgical
instrument tool changing where tool function and morphology may necessitate
alternative control and sensing strategies, or in agriculture where general
purpose robot systems can be made crop specific, or where a platform
changes its locomotion from wheels to legs.
Distributed configuration: The development of configuration mechanisms where
common configurations can be distributed to robot systems with different
mechatronic configurations and mechanical morphologies. Key Barriers
Plug and play architectures require standardised interfaces and the identification of
system interconnection points that do not compromise system integrity or function.
Barriers can be created by the long time scales on the adoption of standards. State of the Art Exemplars
Level 0
Many robot systems in industrial automation employ static configuration. (TRL 9)
Level 1
Many systems exist that utilise start-up configuration based on configuration files
or user input. (TRL 9)
Level 2
Many systems can be software configured by the user through a variety of
different mechanisms. (TRL 9)
Level 3
Some industrial robot arms are able to change the end effector tool as they
progress through different stages of a process. (TRL 9)
3.2 Adaptability
The ability of the system to adapt itself to different work scenarios, different
environments and conditions. Adaptation may take place over long or short time
scales. It may relate to local control systems or actions, or to the whole system or to
Adaptability implies that the system performs optimisation against some performance
criteria. This requires defined performance metrics that can be used to drive the
adaptation process.
Adaptability can be applied to a wide range of elements in a robot system:
Adaptation of sensor processing to account for different environmental conditions, for
example a vision sensor adapting to a failed sensing element.
The control parameters of a controller adapting to account for changes in the specific
properties of an actuator (increased friction, reduced power etc.).
The adaptation to a new environment with respect to the strategy used to achieve a
function. For example adapting a cleaning scan pattern to optimise time in a room
after examining performance metrics.
Adaptability must be clearly distinguished from Configurability and Decisional
Autonomy. Adaptability involves the altering of parameters over time based on
experience with the goal of optimising performance.
It is particularly important to distinguish between Adjustment and Adaptation.
Adjustment is the result of Decisional Autonomy based on sensing or perception. A
platform will “adjust” to a changing environment (e.g. Walking onto a slippery
surface) as a result of perception and the decisional autonomy mechanisms in the
platform. Over time it may “adapt” its response to slippery surfaces based on the
number of times it falls over.
Adaptation takes place over time based on an accumulation of experience. The time
scale for adaptation will depend on the process being adapted. For example a PID
controller operating at 1kHz may be adapted over a period of a second or more,
where as task adaptation may take place over a period of days or weeks. The time
scale of adaptation therefore depends on the cycle time of the process being
It is also important to distinguish between adaptation and knowledge acquisition
where a robot changes its behaviour because it has acquired knowledge about the
environment by using perception and cognitive abilities. Technology Drivers
The following technology areas impact on the adaptability of a system;
Learning and Adaptation
78 Ability Levels
The following are identified as different levels of adaptation within a robot system.
Many current robots operate with little or no adaptation.
Level 0 - No Adaptation
The system does not alter its operating behaviour in response to experience
gained over time.
Level 1 - Parameter adaptation
The system alters individual control parameters based on assessments of
performance local to the module on which the parameter operates. For example
the differential term constant in a PID controller is altered to maintain stability,
where stability is measured in relation to the oscillation in the control term.
Level 2 - Multiple parameter adaptation
The system alters several control parameters based on the aggregate
performance of a set of interconnected or closely coupled modules. For example
the tuning of feature extraction filters over time to optimise performance in the
Level 3 - Process chain adaptation
The processes applied to achieve a function are adapted over time, or over
position, to optimise the outputs from a process chain. The adaptation may alter,
over time, the weight applied to different processing outputs in a decision making
process, or change which alternative or additional processing stages are switched
into a process chain as a result of the long term assessment of performance.
These adaptations have a degree of permanence with respect to multiple
operating cycles and are controlled by long term observations of the behaviour,
state or effectiveness of the process outputs.
Level 4 - Task adaptation
The task performed during the process cycle is adapted over time to optimise a
particular parameter. This adaptation is achieved by strategic overview of the
performance of the system while carrying out the task. Adaptation is the result of
accumulated experience.
Level 5 - Communicated adaptation
The process of adaptation is carried out between multiple independent agents.
The adaptation is communicated between agents and applied individually within in
each agent. Agents can be both real or simulated and of different types including
non-robotic agents. Ability Targets
Level 0 and 1 are well understood in the context of controllers. Higher levels of
adaptive ability are less well understood and targets for improving the adaptability of
robot systems should concentrate on advancing ability levels within diverse
Resource allocation adaptation: Adaptation in complex dynamic environments
to seek out optimal solutions to resource allocation problems and scan pattern
problems. For example in warehouse picking and packing tasks.
Adaptation API: Development of generic interface mechanisms for the
adaptation of multi-stage processes particularly in sensor data processing and
in planning.
Distributed Adaptation: Development of mechanisms for the communication of
adaptation within heterogeneous multi-agent systems, including the use of
cloud computing.
Adaptation drivers: Development of systems with internal modelling able to
provide outputs to drive the adaptation of systems to improve performance over
3.3 Interaction Capability Description
The ability of the system to interact both cognitively and physically either with users,
operators or other systems around it, including other robots. The ability to interact
may be as simple as the use of a communication protocol, or as advanced as
holding an interactive conversation.
The ability to interact is critical to many areas of application. Interaction depends on
both the medium of interaction and on the context and flow of the interaction. The
ability to interact covers three specific areas of interaction:
Human robot interaction
Robot robot Interaction
Interaction safety
Each of these three types of interaction requires a set of ability levels. In a number of
application scenarios two or more of these types of interaction ability will be mixed. Technology Drivers
The following technology areas impact on the interaction capability of a system;
Human Machine Interface
Human Robot Collaboration
Cognition Current Ability Levels
Human Robot Interaction Levels
The following set of levels relate to the interaction between users and the robot
system. This set of ability levels are distinct from the cognitive human robot
interaction levels as they define the method of interaction independently of the
cognitive context:
Level 0 - No interaction
It is possible that some robots will effectively have no operational interaction with
a user.
Level 1 - Direct control
The user provides control of the robot moment to moment. The system can
translate, alter, or block these controls within parameters set by the user or
system. The user controls are in the form of parameters that alter the control of
the robot. These parameters may be continuous quantities, for example a steering
direction, or binary controls.
Level 2 - Direct physical interaction
The user controls the robot by physically interacting with it. The robot reacts to the
user interaction by feeding back physical information to the user via the contact
point. For example the user teaches a motion sequence to the robot, or feels the
surface of an object the robot is in contact with.
Level 3 - Position selection
The system is able to execute pre-defined actions autonomously. The user selects
the subsequent action at the completion of each action. For example a robot is
able to move between defined waypoints in its environment, or carry out a fixed
action such as releasing an object, as commanded by the user.
Level 4 - Traded autonomy
The system is able to operate autonomously during some parts of a task or in
some tasks. Once this task or sub-task is complete the user will either select the
subsequent task or intervene to control the system by direct interaction to carry
out a task. This results in alternating sequences of autonomous and direct control
of the system by the user.
Level 5 - Task sequence control
The system is able to execute sub-tasks autonomously, these sub-tasks will
involve a higher level of decisional autonomy than the pre-defined tasks in Level
3. On completion of the sub-task user interaction is required to select the next
sub-task resulting in a sequence of actions that make up a completed task.
Level 6 - Supervised autonomy
The robot system is able to execute a task autonomously in most operating
conditions. The system is able to recognise when it is unable to proceed or when
it requires user input to select alternative strategies or courses of action. These
alternatives may involve periods of direct control.
Level 7 - Task selection
The system is able to autonomously execute tasks but requires the user to select
between strategic task alternatives in order to execute a mission.
Level 8 - Mission Goal setting
The system is able to execute tasks to achieve a mission. The user is able to
interact with the system to direct the overall objectives of the mission. Parameters
These interaction levels are also modulated by parameters of the interaction. These
factors can increase or decrease the difficulty of achieving levels of interaction ability:
Interaction time: The length of time over which the interaction takes place.
Longer sequences of interaction will in general be harder to achieve than
shorter interaction times.
Interaction Environment: The environment where the interaction occurs will also
affect the difficulty. Interactions in controlled environments will be easer than
interactions taking place in work or every day environments where the robot
needs to focus attention on the user. Highly dynamic or hazardous
environments will also significantly affect the interaction.
User expectation: The level of expectation of the user, the level of user
experience and training will impact on difficulty. Trained users able to
understand how to command the robot and users that have realistic bounded
expectation, or experience, will reduce the difficulty in achieving a particular
level of ability.
Robot to Robot Interaction Levels
The following set of levels relate to the interaction between robots in carrying out a
task or mission. No distinction needs to be made between separate robots that
communicate and systems of dependent robots that carry out a task. However there
is a distinction between systems that rely on a central controller and those that use
distributed decision making.
Level 0 - No interaction
The robot operates on its own without communication with another robot.
Level 1 - Communication of own status
Two or more robots communicate basic status information and task specific
status. Status information is pre-defined for the task. The information
communicated only relates to the state of the robot within the task.
Level 2 - Communication of task status
Two or more robots are able to communicate information about the task they are
performing in terms of task completion, time to completion, and information about
task barriers, resources etc. This information is at a high level and will impact on
the planning of a common task, or tasks in a common space.
Level 3 - Communication of environment information
Two or more robots share information about their local environments, or share
wider scale information that they have acquired or been given. The robots are able
to assimilate the information and extract task relevant knowledge from it.
Level 4 - Team communication
Two or more robots are able to communicate task level information during
execution of the task such that it is possible to implement dynamic planning
between the robots in the team. Each robot carries out its own tasks with
awareness of the other robots in the team.
Level 5 - Team coordination
Two or more robots are able to collaborate to achieve a task outcome that could
not be achieved by either robot alone, or by each robot operating independently.
Level 6 - Capability Communication
Robots are able to communicate their own task capabilities and utilise cooperative
working between teams of heterogeneous robots where there is no prior
knowledge of the composition of the team.
Robot Robot Interaction Parameters
Robot to robot interaction is governed by the parameters of the interaction channel.
At a basic level this is governed by the standard communication channel parameters
Communication bandwidth
Communication latency
Noise levels.
The values of these parameters are fundamentally governed by the communication
medium which in turn will be determined by the environment of operation for each
The level of achievement in robot to robot interaction is also modulated by the level
of generalisation in the task being undertaken. For tasks that are specific and well
defined it is easier to achieve the higher levels of ability. Similarly in systems with a
central control node task specific communication mechanisms are likely to have
been designed in.
Human Robot Interaction Safety Levels
A third aspect of interaction ability is the level of safety within the interaction. While
safety technology will focus on the delivery of specific safety mechanisms it is the
system as a whole that expresses the level of safety for the task.
The following levels only apply to robots that have an inherent level of un-safety. For
example if a robot is safe at Level 0 then there is no need for it to reach Level 1
safety. For this reason each successive level relates to systems that exhibit
increased levels of potential harm.
It is assumed that all robots meet safety criteria appropriate to their operating
environment with respect to electrical and battery safety requirements, typically
specified by European CE marking criteria. It is also expected that appropriate safety
criteria have been applied with respect to consumables used by the robot. For
example heated liquids, liquids under pressure, or chemical agents.
Level 0 - Intrinsic safety
The mechanism of the robot is safe because by design it cannot exert a force that
is damaging to a person at any time during its operating cycle. The maintenance
of this level of safety does not depend on software.
Level 1 - Basic safety
The robot operates with a basic level of safety appropriate to the task. Maintaining
safe operation may depend on the operator being able to stop operation or
continuously enable the operating cycle. The maintenance of this level of safety
does not depend on software.
Level 2 - Basic operator safety
The robot is made safe for the operator by physically bounding the operating
space of the robot. Access gates trigger stop commands to the robot. The robot
will not operate unless the bounding space is closed.
Level 3 - User detection
The robot is informed when a user enters the work zone. The robot operates in an
safe way while the user is present in the operating zone.
Level 4 - Work space detection
The robot operates within a well defined space where a zone of safe operation is
identified to the operator and programmed into the robot. While the robot is
occupying the safe zone it will control its motion such that it is safe. The system
may also use sensing to detect that the user does not enter the unsafe zone.
Level 5 - Dynamic User detection
The robot or its support systems detect users within its operating zone and
dynamically defines a safe zone that envelopes the user where the robot controls
its motion to be safe.
Level 6 - Reactive safety
The robot is designed to be safe under all reasonable circumstances such that if it
impacts a person the impact forces are minimised below the level that may cause
injury during the impact.
Level 7 - Dynamic safety
The robot is able to exert strong forces as a part of an interaction task with a user,
but recognises when the use of these forces may endanger the user. In this case
the robot alters its motion to ensure safe operation.
Level 8 - Context dependent safety
The robot is able to recognise circumstances where it needs to behave in a safe
way because it is uncertain about the nature of the environment. Ability Targets
The primary targets for interaction ability are concerned with providing increasing
levels of interaction ability in the three sets of levels. In particular the improvement of
ability against the interaction parameters outlined above will be key in some areas of
application. In addition to this general progression of ability the following R&D&I
activities have been identified:
Certification and Classification of Safety Levels: Methods need to be developed
that are able to classify, and provide certification for, the safety levels needed
within different domain contexts.
Interaction Design: Interaction design has been used extensively in the design
of everyday technical objects. The added physical element in robotics will
require new interaction design methods to be developed particularly in
applications requiring physical interaction. This has particular relevance in
smart manufacturing and assistive robotics.
Robot Communications: Robot to robot interaction will rely on communication
protocols able to handle dynamic peer to peer networking. The higher levels of
robot to robot communication may need additional protocols or layers over and
above that currently being developed. Early engagement in the establishment
of these protocols will be important particularly in the communication of robot
centric information and information with a local context.
Establishment of Cloud based services: In some applications robots may be
able to rely on Cloud based services for the processing of information. Methods
will need to be developed to allow the communication of information that
preserves semantic and cognitive content. Key Barriers
The user acceptability of interaction, and the establishment of interaction norms will
represent key barriers to establishing and raising the level of user interaction. In
safety critical tasks the certification and validation of systems will present a
significant barrier to deployment.
3.4 Dependability Description
The ability of the system to perform its given tasks without systematic errors.
Dependability specifies the level of trust that can be placed on the system to perform.
This may be in terms of a MTBF or that we trust it to look after a person for a day. Technology Drivers
The following technology areas impact on the dependability of a system;
Systems Development
There are four fundamental elements to dependability:
Failure Dependability: The system remains dependable when it fails.
Functional dependability: The system is dependable in the tasks it performs.
Performance is consistent and of a standard appropriate to the task.
Environmental dependability: The robot is dependable in its interaction with the
operating environment.
Interaction dependability: The robot is dependable in its interaction with users
and other independent robots.
Critically the dependability in a robot system must be at an appropriate level for the
task being carried out. Not all tasks require very high levels of dependability. Part of
the design of any robot system must be to assess the level of sufficient dependability
for each task. Current Ability Levels
Level 0 - No dependability
All useful robots are dependable to some degree, even laboratory prototypes. This
level exists for completeness.
Level 1 - Mean failure dependability
The dependability of the robot is based on the mean time to failure of its
components. The dependability is based on the design of the robot. The robot is
not itself able to increase its dependability. For Failure Dependability this relates
to the failure of all component parts of the robot including software components.
For Functional dependability this relates to the frequency of failure of the system
functions with respect to the task being undertaken, and for environmental
dependability it relates to the failure of the robot to correctly interpret the
environment, for example falling down a step, or failing to detect a hazard. For
Interaction dependability it relates to the failure of the the robot to interact with a
human in a functional or intuitive manner.
Level 2 - Fails Safe
The robot design is such that there are fail safe mechanisms built into the system
that will halt the operation of the robot and place it into a safe mode when failures
are detected. This includes any failures caused by in-field updates. Dependability
is reduced to the ability to fail safely in a proportion of failure modes. Fail safe
dependability relies on being able to detect failure.
Level 3 - Failure Recovery
The robot is able to recover from a proportion of failures by restarting or resuming
its operation.
Level 3 - Graceful Degradation
The robot is able to recognise the impact of a proportion of failures on its function
and operation and is able to compensate for the effect of the failure to maintain
dependable operation. Function effectiveness or the ability to achieve optimal
working may be impacted.
Level 4 - Task dependability
The robot system is able to recognise the impact of a failure on the overall task it
is undertaking and re-task activities in order to minimise the impact of the failure
on the task. This may also include self repair as an alternative task.
Level 5 - Mission dependability
The robot is able to recognise the impact of a failure on the overall objectives of a
mission and communicate the nature of the failure to other systems and robots to
minimise the impact on the mission objectives. In turn the robot is able to receive
and interpret mission failures from other robots and systems and re-task its
actions to compensate.
Level 6 - Predictive dependability
The robot system is able to predict that a planned future action may result in a
loss of dependability, or that the effect of the partial failure of a component can be
mitigated by altering future actions. Thus the robot is able to extend its
dependability by taking action in advance of failure in order to reduce the effect on
Dependability components
In addition to the above generic levels there are specific aspects of dependability that
have a varying significance depending on the task being carried out. In essence
each of the other abilities also inform a specific level of dependability in addition to
the four outlined above.
Motion dependability:
The level of motion dependability defines the
dependability that can be placed on the motion of the robot. Certain
applications require very high levels of motion dependability in order to provide
high levels of operational safety guarantee. Motion dependability is a key
element in both functional and environmental dependability. It may also be an
important part of physical interaction dependability.
Manipulation dependability: A robot that relies on the manipulation of objects
as a part of its function will require a level of manipulation dependability that
relates to the success of the manipulation of objects. This is a key element of
environmental, functional and interaction dependability
Cognitive dependability: A robot that requires some level of cognitive
understanding to achieve its task will require a level of cognitive dependability.
This will range from the success with which the context of objects in the
environment is correctly handled to the correct interpretation of user states from
Dependability Parameters
Achieving high levels of dependability will be affected by a number of parameters
that are task dependent:
Failure criticality: The relationship between mean time to failure and mission or
task duration has a critical impact on the dependability of a system. This can be
characterised in three levels; “non critical” where the time to failure does not
impact on the task or mission; “task critical” where meantime to failure must be
much greater than the typical task time; and “mission critical” where meantime
to failure must be much greater than typical mission time. Ensuring a system
achieves the desired criticality rating is strongly related to the System Design
Time scale: The length of time a robot needs to perform a task will affect its
dependability. A robot carrying out a short repetitive task is more likely to have
a higher level of dependability than a robot with equivalent technology that
carries out a longer task. For a number of proposed applications the ability to
achieve long term dependability is important.
Task or mission risk: The risk level of a particular task or mission will impact on
the dependability of a system. High risk activities on the limit of ability will
necessarily result in lower levels of dependability. For example trying to
balance plates one on top of the other while moving across a room.
Environmental variability: Robots that operate in environments with high levels
of variability, both in terms of objects and in terms of dynamics, will inevitably
display lower levels of dependability for a given technology level than robots
working in structured and near static environments. Extremes of operating
environment, for example working in poor visibility, will also impact on
dependability. Ability targets
In addition to the goal of developing systems and methods that are able to
implement higher levels of dependability, and extend dependability over a wider
range of above parameters the following aspects of dependability require R&D&I
Assessment: To develop the means by which different dependability
components, outlined above, can be assessed.
Certifiability: To develop a system of certification that can provide guarantees
about dependability sufficient for deployment in high risk or safety critical tasks.
Notably this is most critical in physical human interaction tasks in Healthcare,
Manufacturing, Agriculture and Civil tasks.
Design for Dependability: Much of the dependability of a system is intrinsic to
the design process. Methods need to be developed that identify the key parts of
the design process that support design for dependability and can analyse a
system during its design for dependability risks. In particular task assessment
methods able to identify sufficient levels of dependability for a given task that
take into account the different components of dependability and the
dependability parameters identified above.
Intrinsic dependability: While a significant part of dependability is embedded
within system design, there is also a runtime component to the maintenance of
ongoing dependable operation. The identification of methods and mechanisms
for increasing this intrinsic dependability, particularly over long time periods is
of interest for many different application areas and particularly in those
performing long term high risk tasks and missions, particularity in the Civil and
Healthcare sectors.
Post failure analysis: The development of mechanisms and methods that allow
analysis of the operation of a complex robotic system leading up to a
dependability failure, and the development of tools to meaningfully deconstruct
the behaviour of the system will be critical to improving system dependability. Key Barriers
The key barrier to establishing higher dependability levels is the testability of system
for dependability. Dependability can be designed into a system through appropriate
design processes, but the testing of dependability on tasks and missions requires
real world exposure to the working environment.
3.5 Motion Capability Description
The ability of the system to move. Motion may be highly constrained where ability is
measured by the precision of the motion, or its repeatability. Alternatively motion may
be unconstrained and is measured by the ability to move effectively in different
media or between media. For example in unstable environments such as on ice or
sand this might specify the ability to maintain balance or achieve effective motion.
Motion ability includes the ability to maintain a position. Maintaining a position
typically involves motion unless the position is statically held when the system is
It is important to note that all aspects of the motion of a robot are constrained by the
mechanical operating envelope of its design. Technology Drivers
The following technology areas impact on the motion capability of a system;
Mechanical Systems
Localisation and mapping Current Ability Levels
The following are a set of ability levels for motion:
Level 0 - No motion
All robots move in their environments, movement defines a robot. This level exists
only for completeness.
Level 1 - Pre-defined open loop motion
The robot carries out predefined moves in sequence. The motion is independent
of the environment and events in the environment. The robot may not be able to
maintain a position if subject to external forces, may be able to statically rest at a
given position.
Level 2 - Pre defined closed loop motion
The robot carries out predefined moves in sequence where each motion is
controlled to ensure position and/or speed goals are satisfied within some error
bound. So for example a robot can move to and maintain a position (within some
error margin) against forces less than the resultant motive force at the point of
contact. A platform will similarly be able to execute fixed motions where the
accuracy of these motions in the environment will depend on other abilities such
as its perception ability.
Level 3 - Open path motion
The robot can execute a motion that follows a path with a given path accuracy.
This path is described by a specific point on the robot. The robot is able to return
to any given point on the path with an accuracy that is appropriate to the task.
Level 4 - Position constrained path motion
The robot can execute a path motion where the path is constrained by physical
objects or by defined zones that must be avoided. For example a robot arm that
can operate through a physically constrained region such as a hole in a wall, or a
platform that can move to avoid a known area of the environment such as a step
down. The robot is able to execute a path to an unvisited location obeying
Level 5 - Force constrained path motion
The robot can execute a path motion while applying a specified force in a given
direction related to the motion. For example moving over the surface of an object
while applying a force perpendicular to the surface as might be required when
polishing a surface.
Level 6 - Parameterised motion
The robot can execute a path move that optimises for a parameter. For example a
path that reduces energy consumption, covers an area, or constrains the angle
range of a joint, or the torque or force in a joint or linkage.
Level 7 - Position constrained parameterised motion
The robot can operate through a physically constrained region while at the same
time optimising a parameter or set of parameters that constrain the motions of the
robot. For example a robot arm may be able to reach a high shelf while
maintaining a centre of gravity, or a platform robot operate in a room away from a
charging station while optimising power usage.
Level 8 - Compliant motion
The robot can execute motions that alter in response to external forces applied to
the robot such that the force exerted on the external body is controlled. The robot
is able to maintain position and path in the absence of any external force.
Level 9 - Reactive motion
The robot is able to react to externally applied forces contacting any part of the
robot. The reaction may result in stiffening to resist the force or in lowering
stiffness to reduce impact effect. The system is able to apply a force in a given
direction and maintain that force against a rigid or semi-rigid body.
Level 10 - Dynamic motion
The robot is able to alter its motion strategy in response to multiple external
dynamic forces in order to optimise motion parameters. The robot is able to exert
a force in a given direction relative to a dynamic body, or within an environmental
medium while maintaining position or path.
Level 11 - Soft medium motion
The robot is able to move into and within a soft medium. It is able to maintain a
position and path within this medium while optimising motion and force
parameters as demanded by the task.
Motion Parameters
The above levels of ability define a framework for assessing motion ability. The
motion ability of a whole robot is also assessed according to a set of motion
Accuracy, repeatability, path error
Speed, acceleration.
Degrees of freedom (physical form)
Load carrying capability
Applied force/torque range
Robot size and scale
The exact figures for these parameters will vary with each task. Achieving the higher
levels of ability with extreme values of the motion parameters will be harder. Ability Targets
The primary objectives are in raising the ability level of systems in order to expand
the market. In addition the development of systems that usefully push the boundaries
of the motion parameters significantly beyond current values for a given task may
result in step changes in applicability. In particular the smaller and larger ends of the
size scales, the range of environments and maintenance of stability are all key goals.
In addition the following R&D&I objectives can be identified:
Operation in air and water or other dynamic environments: The ability to
maintain a position both absolute and relative to other robots or objects in the
air or under water against the natural forces of the environment is fundamental
to a number of key application areas. Similarly the ability to apply a given force
in a particular direction in the air or underwater to another object will also be
fundamental to may applications.
Transition Environments: The development of motion systems able to operate
in transitionary environments for example in waterlogged ground, mud, sand,
gravel etc and in transition between environments, for example between air and
Terrain following in 3D: The development of control systems able to maintain a
controlled distance during the motion of a robot (of any physical form) and to do
so with smooth motion in a dynamic environment. For example following an
unknown terrain in air and under water, including the case where the terrain
involves high relief structures.
Develop motion control capable of stable modal switching. For example
between autonomous direct motion and compliant motion compatible with
physical human interaction.
Exploitation of kinematic redundancy: To develop systems able to use
kinematic redundancy as a part of achieving parameter constrained motion.
Load transfer: To develop mechanisms and control systems abel to transfer
load to fixed surfaces in the environment and make use of this to control
constrained motion.
Interaction motion: Development of systems able to execute motion against
objects that have constrained motion. For example the development of systems
required to open any door, or be compatible with human interaction.
Constrained space operation: Development of systems able to operate in
confined spaces, such as inside pipes or flexible tubes, or where the
environment severely constrains the motion path.
Motion in flexible materials: The development of systems able to operate within
flexible objects and soft objects and react appropriately to textural changes and
object density changes. These systems have special relevance in healthcare
and in in-vivo surgical robotics in particular, and in manufacturing where robots
operate within materials of variable density and flexibility.
Human compatible motion: The development of systems able to deliver human
compatible motion and interaction, both in terms of impact, the ability to
realistically mimic human motion in terms of range reach and capacity. This is
of particular importantance in assistive robotics and where human compatible
interactions are required, and in environments where human scale motion is
required, for example climbing stairs or working in a kitchen.
Cognitively aware motion: The development of cognitively aware motion where
a system is able to react to changes in the cognitive context of an interaction
using motions that can be interpreted by a user. For example the reaching out
of a robot arm to steady an elderly person must not frighten them with a sudden
unexpected movement. Key Barriers
The primary barriers to achieving high levels of motion ability are technology
limitations, Actuation technology, high resolution sensors, power densities,
mechanical constraints etc.
3.6 Manipulation Ability Description
The ability of the system to handle objects. Where end effectors are fixed or specific
to the task this will specify the accuracy and repeatability of the manipulation, for
example the ability to absorb tolerances in parts. For dexterous manipulation it might
specify the ability to discover how to hold and move unknown objects, or the ability to
match two objects together in specific ways. (e.g for joining or stacking) Technology Drivers
The following technology areas impact on the manipulation ability of a system;
Systems Engineering
Modelling and Knowledge Engineering
Manipulation ability is the result of a combination of the other abilities. As a result
each of the levels of manipulation ability relies on particular pre-requisite abilities.
The following defined ability levels concentrate on describing the ability to grasp,
manipulate and move a single object. It is a natural consequence of being able to
manipulate one object that the system is able to manipulate multiple objects by
executing sequential manipulation actions. This is captured by the phrase “within the
context of a task” in the following level descriptions. For example the context may be
the movement of a single known object or a sequence of object moves. The process
of deciding how to execute the sequence is not a part of manipulation ability but a
part of Decisional Autonomy and Cognition. Similarly human collaborative
manipulation results from abilities in Interaction, Manipulation and Motion working
together. Current Ability Levels
Level 0 - No Manipulation Ability
Many robots will not require the ability to manipulate objects.
Level 1 - Simple pick and place
The robot is able to grasp an object at a known pre-defined location using a predefined grasp action. The robot is then able to move or orient the object and finally
un-grasping it. The robot may also use its movement ability to move the object in a
particular pattern or to a particular location. Grasping uses open loop control.
Level 2 - Tolerant pick and place.
The robot is able to grasp an object that is not located at an exact location, may
have some orientation variation and is in the general location within the span of
the gripper from some known location. Tolerance in the grasp action is able to
absorb the difference in location or orientation. The operation is able to
compensate for the differences in the picking location without affecting the
required placement accuracy.
Level 3 - Variable placement
The robot is able to alter its placement action to accommodate small changes in
location of the destination for a picked object. For example it is able to join two
parts where the positional tolerance of the mating part is greater than the
accuracy needed to place the part correctly.
Level 4 - Compliant placement
The robot is able to use compliance in the placement process to fit a picked part
into a statically held part. For example the insertion of one part into another where
the insertion forces vary during insertion as a result of friction.
Level 5 - Location unknown pick and place.
The robot is able to pick up a known object where the location and orientation of
the object are not pre-defined. It is able to orient and align the object (this may be
achieved without placement) and then place it within the context of a task.
Level 6 - Generic pick and place
The robot is able to pick up an object belonging to a certain parameterised type
where the dimensions, location and orientation are unknown. It is able to orient
and align the part and place it appropriately in the context of the task.
Level 7 - Object manipulation
The robot is able to pick up and manipulate an object belonging to a certain
parameterised type where the object can be articulated, or consists of multiple
separate parts. The robot is able to exercise the articulations of the object or
disassemble it within the context of a task.
Level 8 - Unknown Object Manipulation
The robot is able to pick up an unknown object and determine the generic
grasping properties of that object. It is able to use those properties to determine
how to manipulate and place the object. The robot may be able to categorise the
object based on a set of known object types from its manipulation of the object.
Manipulation parameters
The following parameters significantly affect the assessment of manipulation ability:
Object scale
Object shape complexity
Object properties; surface texture and material (smooth, rough, transparent,
soft, hard etc.)
Object dynamics; Object motion under load, compliance, stretch, flexibility and
animate properties.
In assessing manipulation ability it is important to ensure that the identified levels of
ability are suitably technology independent. The mechanical nature of manipulation
ability will be significantly affected by the specific gripper and sensing technologies
employed. The development of these elements of manipulation is covered by
technology capability targets and therefore do not form part of the classification of
manipulation ability. Ability Targets
In addition to the on-going development of systems that raise ability levels the
following have been identified as R&D&I objectives:
In-hand manipulation and re-grasping: The ability to orient a part while it is
being held “in-hand” using a process of un-gripping and re-gripping will be
fundamental to the ability to orient and align everyday objects particularly when
interacting with users and handing over objects for collection.
Contextual assistance: Significant information can be gained about an object
while manipulating it. Information can be gained from direct tactile sensing of
the pressures but also from the dynamics sensed as the object is moved (for
example the slosh of fluid in a bottle, or the sound emitted by an object when it
is grasped). In some cases object may simply be picked up and replaced in
order to gain contextual information.
Tool manipulation: The use and manipulation of tools allows generic
manipulators to carry out a wider range of tasks. The ability to handle variable
material densities, apply forces and orient tools are key to their use.
Human level dexterity: Everyday objects have the parameters of human
dexterity built into their design. Instilling human compatible dexterity and
manipulation ability is a key goal for the effective manipulation of everyday
objects within any human compatible task particularly where interaction is
Collaborative manipulation: There are many manipulation tasks that require
collaboration either between users and robots, between robots, or simply
between multiple manipulators on the same robot. Each presents a challenge
to manipulation ability in terms of the grasping and exchange of objects.
Adaptive plans for dynamic objects: Robots operating in manufacturing, in
healthcare and in everyday environments will need to be able to handle and
manipulate dynamic objects, objects that deform, or move independently. The
ability to handle dynamic objects will mark a significant step change in ability.
Manipulation sequences: While the ability to carry out sequences of
manipulation actions depends on decisional autonomy and cognitive abilities
the focusing of these abilities on tasks related to manipulation is an important
goal. Of particular importance to numerous applications is the planning and
execution of sequences involving multiple objects, occluded objects and
objects that require multiple grasp strategies. The ability to pick out objects with
minimal disturbance to other objects and the ability to select and grasp specific
objects from a collection containing repeats will be a significant step change.
Haptic SLAM: The ability to discover the shape of an object and establish grasp
properties purely or mainly from haptic information provided by the process of
touch and manipulation. Key Barriers
The main barriers to manipulation ability vary with application and size scale.
Fundamentally they relate to the focusing of other abilities to support manipulation
and to mechanical design, actuation and tactile sensing. Manipulation ability
specifically requires the integration of perception and motion abilities.
The provision of realistic generic benchmarks for manipulation ability, so that results
can be transferred to real world tasks, is also a potential barrier to the assessment of
3.7 Perception Ability Description
The ability of the robot to perceive its environment. At the simplest level this is about
specifying the probability of accurately detecting objects, spaces, locations or items
of interest in the vicinity of the system. It includes the ability to detect the ego motion
of a robot arm and the ability to interpret information and to make informed and
accurate deductions about the environment based on sensory data. Technology Drivers
The following technology areas impact on the perception ability of a system;
The word “object” in the following level descriptions does not imply a physical or well
defined object but does imply a distinct object with respect to sense data, so for
example the “object” may refer to the thermal image of a fire, or the sound made by a
saw cutting wood as well as a segmented image. These level descriptions use the
phrase “sense data” to cover all types of sense data from chemical, visual, acoustic,
thermal etc. Ability Levels
Perception is a key part of the ability of any robot system. It is composed of several
different types of ability each of which require the integration of different
Perception ability
Tracking ability
Recognition ability
Scene perception
Location perception
Levels of perception ability
The following levels refer to the generic ability of a system to perceive:
Level 0 - No external perception
Some robots do not sense their environment but simply carry out sets of preprogrammed moves triggered by a starting event. Although there may be safety
systems that cause the robot to fail safe these do not alter the operating cycle
Level 1 - Single parameter sensing
A robot uses sensors that provide a single parameter output. For example a
distance sensor, or a contact sensor. The robot utilises these outputs to directly
alter behaviour within an operating cycle.
Level 2 - Marker sensing
A robot system may use fixed and known markers in the environment to indicate
objects or waypoints (e.g. Barcodes, reflective strips etc). The detection of these
markers provide triggers to alter or switch between behaviours or sequences of
Level 3 - Multi-Parameter Perception
A robot uses multiple single parameter sensors to create a unified model of the
environment. Sense data can be collected from multiple types of sensor as well as
multiple sensors of the same type. Each sensor contributes information to the
model. The model us used to alter the behaviour of the system.
Level 4 - Feature based perception
Sense data is gathered from a region of the environment such that the sense data
has a spatial mapping. The richness of the sense data information content is such
that it is possible to apply feature extraction to the sense data and thereby
interpret the content of the sense data as a set or sets of features. The presence
of features is used to alter behaviour.
Level 5 - Grouped feature detection
The sense data gathered from the environment can be processed such that
features can be aggregated to capture linkages between features. A group of
features may relate to the same real object in the environment, but where the
object has not been identified. The characteristics of the feature group can be
used to alter the behaviour of the system. For example a set of features of the
same colour that move in the same way may relate to a pink ball.
Level 6 - Object identification
The system can identify objects that it has detected in the environment through
sets of grouped features and can use this identification to alter the system
Level 7 - Property identification
The system is able to deduce the properties of objects in the environment and
utilise those properties within system behaviour.
Level 8 - Hidden state identification
The system is able to infer properties of an object, person or scene that are not
directly observable.
Levels of Tracking Ability
Because robots move within the environment the sense data for a distinct object will
alter as the robot moves. It is important that the robot is able to track and maintain its
sense of a distinct object during motion.
Level 0 - No tracking
Some robots will be able to carry out their tasks without any tracking ability.
Level 1 - Tracked Feature Perception
Features detected in the sense data are tracked over time. The tracking of
features is used to build internal models of the environment. The tracking of
markers in the environment is equivalent to tracking derived features.
Level 2 - Static Object tracking
It is possible to track a detected object. The detected location of the object can be
maintained with a reliability and accuracy that is compatible with the task.
Level 3 - Dynamic object tracking
It is possible to identify an object and track it using sense data. As the object
moves the system is able to disambiguate the motion of the robot from the motion
of the object.
Level 4 - Flexible object tracking
It is possible to identify a flexible or deformable object and track it.
Level 5 - Animate objects
It is possible to identify and track an animate object and extract the pose of the
Object Recognition Levels
Many robot applications require the robot to recognise objects in the environment.
This ability may range from being able to recognise instances of a single object, to
being able to distinguish between many different objects or even identify objects that
fit a generic pattern.
Level 0 - No Recognition
The robot system does not need to detect or recognise objects in the environment
in order to carry out its task.
Level 1 - Feature detection
Sense data is gathered from a region of the environment such that the data has a
spatial component and can be mapped to a model of that region. The richness of
the sense data is such that it is possible to apply a feature detection process to
create a set or sets of features that persist.
Level 2 - Object detection
Multiple persistent features can be grouped to build models of distinct objects
allowing objects to be differentiated from each other and from the environment.
Level 3 - Object recognition - single instance
Object models created from sense data can be matched to specific known
instances of an object with a reliability that is appropriate to the task.
Level 4 - Object Recognition - one of many.
Object models created from sense data can be matched to one of a number of
specific known objects with a reliability that is appropriate to the task.
Level 5 - Parameterised object recognition.
Object models created from sense data can be matched to a number of known,
parameterised object types. The settings for the parameters (e.g. size ratio,
curvature, joint position etc) can be deduced from the sensed object model. Note
that in conjunction with single instance recognition ability this implies the ability to
recognise a known (possibly learned) instance of a generic object, for example a
particular brand of canned drink based on the generic recognition of a drinks can
Level 6 - Context based recognition
The system is able to use its knowledge of context or location to improve its ability
to recognise objects by reducing ambiguities through expectations based on
location or context.
Level 7 - Novelty Recognition
The system is able to recognise novelty in a known object, or parameterised
object type. For example a known mug where the handle is missing or broken.
Level 8 - Unknown object categorisation (Rigid)
The system is able to assess an unknown rigid object based on sense data and
deduce properties that are relevant to the task.
Level 9 - Object property detection
It is possible to use sense data and the derived object model to deduce the
properties of an object. For example analysis of the sense data may provide
surface texture information, knowledge about deformability, or the content of an
Level 10 - Flexible object detection
The system is able to detect the shape and form of objects that are deformable
and generate parameterised models of flexible objects. This includes articulated
objects and objects with flexible and rigid components.
Level 11 - Flexible object classification
The system is able to classify flexible objects by their properties and parameters.
It is able to recognise specific known objects relevant to the task with an
appropriate level of reliability.
Level 12 - Animate Objects
The system is able to detect animate objects and provide a classification
appropriate to the task.
Level 13 - Pose estimation
The system is able to estimate the pose of an animate object within the
Recognition parameters
The recognition of features and objects is parameterised by different parameters of
the objects. These parameters alter the difficulty of achieving the above levels in any
particular task. The settings for these parameters are task dependent. In describing
the level needed for a particular task it is important to state the requirements for
these parameters:
Object orientation: Some tasks may only present objects within a limited range
of orientations, in other tasks there may be considerable variability in
orientation. The difficulty of recognising objects increases with the number of
presented orientations.
Object composition: Object surface variation, texture, reflectivity, transparency,
patterning etc. All affect the difficult of performing object recognition. Reflections
and patterns are an integral part of everyday objects and can present
significant difficulty.
Scale and range: Some objects may be visible within a single field of view
larger objects may require a sequence of views to enable recognition. The
identification of scale and range in relation to sense data is also a key
component in recognition performance.
Object types: Recognition ability will depend on the number of different object
types that must be disambiguated and the sensitivity of the recognition process
to similarities between the objects.
Environment: Nearly all sensors are affected by environmental factors, either
directly (e.g. the use of vision systems in bright sunlight) or indirectly (e.g. the
sound of rain hitting a window in acoustic recognition).
Levels of Scene perception
In many applications robots will need to be able to interpret the context of a wider
scene, identifying static elements in the scene such as walls doors ceilings floor etc.
as well as the delineation of objects. This scene interpretation is not related to the
recognition of specific objects but to the wider identification of spaces and objects
within a working environment.
Level 0 - No scene perception
The robot does not need to be able to interpret the environment in order to carry
out its task.
Level 1 - Basic feature detection
The robot is able to detect features in the environment that relate to static
structures in that environment.
Level 2 - Static Structures
The robot is able to identify static structures in the environment in a way that is
appropriate to the task.
Level 3 - Combined Structures
The system is able to provide a consistent interpretation of the static structures in
the environment over time. For example it is able to identify the floor, walls and
ceiling of a room and apply these as physical constraints to a model.
Level 4 - Multiple object detection
The system is able to delineate multiple objects from the static environment where
there may be partially occluded with respect to the sense data gathered. For
example it is able to delineate objects on the floor of a room.
Level 5 - Object arrangement detection
The system is able to detect arrangements of objects, for example objects in a
stack or mixed in a receptacle and identify the relationships between objects with
a success appropriate to the task. For example a chair with books on it and a wine
glass on top of the books.
Level 6 - Dynamic object detection
The system is able to detect an object that is moving within a static environment.
Levels of location perception
In addition to the ability to locate and recognise objects and spaces and perform
tracking robots also need to be able to identify their own location within their
environment. This may be an absolute location, or a relative location.
Level 0 - No perception of location
The robot has no perception of its own location either in terms of its position
relative to its environment or with respect to the relative position of its own
Level 1 - Actuator position
The robot knows where its own mechanical structures are because of an
assessment of the position of each of its actuators. For example a platform can
assess its own position based on the amount its wheels have turned.
Level 2 - External beacons
The robot knows its own location as a result of information derived from the
inspection of external beacons. Beacons may be active or passive and include
global beacons such as GPS and GLONASS.
Level 3 - Feature based Location
The system calculates its position within an environment based on the motion of
fixed features in the environment. For example by using SLAM to build and
maintain a local map.
Level 4 - Mapped location
The robot is able to relate its own position to a map that it has been given or that it
has acquired. This may be a location within a task relevant space.
Level 5 - Self location
The system calculates the position of its own mechanical structures based on
indirectly gathered sense data (i.e. Sense data gathered other than from the
motion control system). This provides a spatial notion of occupancy.
Level 6 - Object coupled location
The system is able to calculate the position of its own mechanical structures in
conjunction with objects it is connected to. For example an object that is being
gripped by the robot, or the position of the user in an assistive task. Ability Targets
The wide range of different types of a perception ability provides significant
opportunity for R&D&I activity to raise ability levels and extend the range of ability
parameters both in terms of generic systems development and in terms of specific
application areas. In addition to this general goal the following tasks have also been
Object property perception: The use of perception to create or confirm generic
property information about known and unknown objects (semantic property
grounding). For example the ability to perceive qualities of an object such as
will it break if dropped, or is it too slippery to hold. Such information can be
used to inform planning and control tasks.
Scene attention strategies: The development of strategies for analysing large
scenes and identifying the elements of the scene that are task relevant within
the current stage of an on-going task. This involves the development of
improved methods for object segmentation and classification.
Immunity to natural variations: The development of perception strategies that
can overcome the impact on perception ability of natural environmental
variations. For example different times of year, different weather conditions,
different lighting conditions etc.
102 Key Barriers
The most important barrier is the limitation of the sensor technology for accurate
measurement of specific materials (reflective, absorbing or transparent) using off the
shelf, affordable and eye-safe sensors. Fusing these different modalities together
into a common representation is also not generally solved. Currently, common sense
knowledge is integrated only at higher level systems, but methods are missing to
select which information to use at the sensor fusion level. Furthermore, the use of
shape similarities in order to cover large variety of object types remains an open
question. Biological systems are still not fully understood in particular on higher
perception levels.
3.8 Decisional Autonomy Description
The ability of the robot to act autonomously. Nearly all systems have a degree of
autonomy. It ranges from the simple motion of an assembly stopped by a sensor
reading, to the ability to be self sufficient in a complex environment. Technology Drivers
The following technology areas impact on the decisional autonomy of a system;
It is important to distinguish between the actions of a system that are the result of
Decisional Autonomy and those that are caused by long term adaptation. Current Ability Levels
The following are a set of ability levels for decisional autonomy:
Level 0 - No autonomy
All robots exhibit a degree of autonomy. This level remains for consistency with other
Level 1 - Basic action
A robot that executes a sequence of actions that are unaffected by the
environment and makes decisions based on the locations of actuators to proceed
to the next action step.
Level 2 - Basic decisional autonomy
The robot makes decisions based on basic perceptions and user input and
chooses its behaviour from predefined alternatives.
Level 3 - Continuous basic decisional autonomy
The system alters the parameters of a behaviour in response to continuous input
from perceptions, or based on input control from a user interacting continuously
with the system. The system may be able to override or ignore user input when
certain criteria are encountered.
Level 4 - Simple autonomy
The system uses perception to make moment to moment decisions about the
environment and so controls interaction with the environment in order to achieve a
predefined task.
Level 5 - Task autonomy
The system utilises its perception of the environment to sequence different subtasks to achieve a higher level task. For example cleaning a room based on a selfconstructed room map where it returns to areas that have been missed and to a
recharging station when the battery runs low. The events that cause behavioural
changes are external and often unpredictable.
Level 6 - Constrained task autonomy.
The system adapts its behaviour to accommodate task constraints. These might
be negative impacts in terms of failed sensors, or the need to optimise power
utilisation or other physical resources the process depends on, (water, chemical
agents, etc). Alternatively these might be constraints imposed by sensing ability,
the environment or the user.
Level 7 - Multiple task autonomy
The system chooses between multiple high level tasks and can alter its strategy
as it gathers new knowledge about the environment. Will also take into account
resource limitations and attempt to overcome them.
Level 8 Dynamic autonomy
The system is able to alter its decisions about actions within the time frame of
dynamic events that occur in the environment so that the execution of the task
remains optimal to some degree.
Level 9 - Mission oriented autonomy
The system is able to dynamically alter its tasking both within and between
several high level tasks in response to dynamic real time events in the
Level 10 - Distributed autonomy
The source for task and mission decisions can originate from outside of the
system. The system is able to balance requests for action with its own tasking and
mission priorities and can similarly communicate requests for action.
Autonomy parameters
There are a number of task based parameters that will affect the achieveability of
individual levels of autonomy on a task by task basis.
Environmental factors: The operating environment will significantly affect the
ability to achieve any particular level of decisional autonomy. In particular
cluttered, dynamic environments are more likely to affect perception and thus
decision making. Extreme environments will similarly cause a reduction in the
ability to make decisions.
Decision cost: Higher levels of decisional risk and reduced recovery options will
decrease the confidence required to raise autonomy levels. In healthcare or in
space where decisions have high cost implications the confidence levels
required in the interpretation of sense data are significantly higher.
Time scale: The longer a system must maintain autonomous decision making
the harder it will become to rise through the ability levels.
Decision range: A system that is only required to make a small range of
decisions will be more likely to have a high level of decisional autonomy. Ability Targets
In addition to R&D&I activity that focuses on the raising of decisional autonomy
levels the following priorities have also been identified:
Decision validation mechanisms: The development of systems and methods
able to provide validation of decisions made within a task context. Where high
risk decisions must be made such systems should be able to rate the risk level
of the decision in the task context and provide corroborating evidence for the
bounding of the decision.
User driven decisions: In many complex tasks it may be necessary to confirm
decisions with a human operator during the task. These confirmation decisions
or requests for direction need to be framed so that the user understands the
context of the decision and can provide added value in the decision making
Certification of decisions: In safety critical systems it will be important for the
decision mechanism to be certified. Design and implementation methods that
allow certification and validation processes need to be developed to create high
levels of confidence in decision mechanisms. In turn decision audit trails will be
required to post-analyse failures of decision making.
Decisions based on uncertain data: In many real applications there will be a
task balance between the capture of new knowledge about an environment and
the execution of the task at hand with incomplete or uncertain data. The
development of decision support mechanisms that are able to manage this
balance will be critical to a number of application domains.
Decision layering: In complex missions there will be multiple layers of decision
making from moment to moment decisions to high level mission decisions.
These layers will need a decision support environment so that the relative
priorities of decisions are handled in a task appropriate way. Key Barriers
Complex decision making in uncertain environments will push the boundaries of
current technical capability. Progress to application and deployment is likely to be
delayed until the decision making technology is developed. This progress will also
depend on advances in perception and cognitive ability.
3.9 Cognitive Ability Description
The ability to interpret the task and environment such that tasks can be effectively
and efficiently executed even where there exists environmental and/or task
uncertainty. The ability to interpret human commands delivered in natural language
or gestures. The ability to interpret the function and interrelationships between
different objects in the environment and understand how to use or manipulate them.
The ability to plan and execute tasks in response to high level commands. The ability
to work interactively with people as if like a person.
Currently, different aspects and faculties of the Cognitive Ability as a whole have
different degrees of maturity and pose different challenges. Attempting to combine
these differences into a single rating or overarching targets are likely to lead to
invalid or misleading conclusions.
The assessment of cognitive ability is therefore divided into several components, or
faculties. The assumption being that the cognitive ability of a system can be
assembled and described more accurately by referring to a mixture of component
abilities. Technology Drivers
The following technology areas impact on cognitive ability.
Systems Design
Human Robot Interaction Ability Levels
Cognitive ability grows out of the framework built by the other abilities, particularly
perception, interaction and decisional autonomy and is composed from a number of
underlying components:
Action ability
Interpretive ability
Envisioning ability
Learning Ability
Reasoning Ability
Individual sets of ability levels can be described for each of these component
There is also a close relationship between Interaction Ability and Cognitive Ability.
This closeness derives from the essentially interactive nature of robotics and its
physical embodiment in the real world. There is a distinction between levels of
interaction that do not involve a cognitive element and those that do. The following
Cognitive ability components characterise the levels of cognitive interaction with
objects and people and are closely related to the Interaction Ability levels.
Object interaction ability
Human interaction ability
Ultimately it is the integration of these abilities which will create robots able to
interact meaningfully in their environment. This integrated cognitive ability of a robot
can be summarised as its ability to acquire knowledge about its environment, adapt
its plans to fit the dynamics of that environment, including the user and their actions,
and to be able to envision its own actions on the environment and reason about
goals and interactions such that it can effectively carry out its tasks.
Action ability levels
Action ability concerns the ability of the robot system to act purposefully within its
environment and the degree to which it is able to carry out actions and plan those
actions. These abilities build on perception and decisional autonomy abilities. Action
ability also co-depends on the other cognitive abilities.
Level 0 - No Action Ability
Robots are defined by having some level of action on the environment. This level
remains for compatibility.
Level 1 - Defined action
The robot executes fully pre-defined actions as a sequence of sub-actions. This
sequence can repeat until stopped by an operator or other system event.
Level 2 - Decision based action
The robot is able to alter its course of action based on perceptions or system
events. It is able to select between a set of pre-defined actions based on its
decisional autonomy ability.
Level 3 - Sense driven action
The robot is able to modulate its action in proportion to parameters derived from
its perceptions. The perceptions are used to drive the selection of pre-defined
actions or the parameters of pre-defined actions.
Level 4 - Optimised action
The robot is able to alter the sub-task sequence it applies to the execution of a
task in response to perceptions or a need to optimise a defined task parameter.
Level 5 - Knowledge driven action
The system is able to utilise knowledge gained, from perceptions of the
environment including objects within it, to inform actions or sequences of action.
Knowledge is gained either by accumulation over time or through the embedding
of knowledge from external sources, including user input that associate properties
with perceptions.
Level 6 - Plan driven actions
The system is able to use accumulated information about tasks to inform its plans
for action.
Level 7 - Dynamic planning
The system is able to monitor its actions and alter its plans in response to its
assessment of success.
Level 8 - Task action suggestions
The system is able to suggest tasks that contribute to the goals of a specific
Level 9 - Mission proposals
The system is able to propose missions that align with high level objectives.
Interpretive ability levels
The interpretation of sense data is key to the ability to identify, recognise, classify
and parameterise objects in the environment. It particularly refers to the ability to
amalgamate multi-modal data into unified high level object descriptions that create
knowledge for tasks to draw on. The ability to interpret also engages knowledge
sources to build increasingly complex interpretations of the environment and human
interaction, in particular building frameworks of relationships between the
environment and objects and between objects.
Level 0 - No interpretive ability
The robot does not need to interpret the environment or user interface actions.
Level 1 - Fixed sensory interpretation
The robot has a fixed interpretation of the perceptions that occur because they are
pre-categorised. For example all sensed objects are applied to an occupancy grid
and assumed to represent actual objects in the environment.
Level 2 - Basic environment interpretation
The robot uses sense data to interpret the environment into fixed notions of
environmental space that are pre-categorised. For example it will search for floor
and wall segments in the sense data as these are relevant to its task even if the
environment it is sensing has neither.
Level 3 - Object delineation
The robot is able to disambiguate objects from an interpretation of its static
environment. The disambiguation of objects is based on built in notions of object
and environment. These notions may only be valid within a narrow operating
Level 4 - Object category interpretation
The robot is able to interpret the shapes and forms of objects based on categories
of objects that are task relevant. It is able to interpret sense data to identify
coherent instances of an object over a time scale appropriate to the task. Note
that this ability level is particularly affected by the Cognition Ability Parameters.
Level 5 - Structural interpretation
The robot is able to interpret perceptions so as to extract structural information
from the environment. It is able to identify the structural relationships between
objects in the environment.
Level 6 - Basic semantic interpretation
The robot is able to apply semantic tags to locations and objects allowing it to plan
actions based on functional objectives that depend on the semantics of objects
and locations.
Level 7 - Property interpretation
The robot is able to interpret perceptions to determine the properties of objects or
locations in the environment.
Level 8 - Novelty interpretation
The robot is able to interpret perceptions to identify novelty in objects or locations.
Level 9 - Environmental affordance
The robot is able to interpret the environment in terms of what it affords. For
example it is able to interpret the ground conditions in a muddy field as being too
unstable for the load it is carrying.
Envisioning ability levels.
Envisioning refers to the ability of the robot system to assess the impact of actions in
the future. This may reduce to prediction but in the higher levels involves an
assessment of the impact of observed external events.
Level 0 - No envisioning ability
The robot is not able to predict subsequent states.
Level 1 - Motion prediction
The robot is able to project the effect of its motion to predict short term local
interactions with detected objects in the environment. The robot only has the
ability to predict its motion with respect to static objects.
Level 2 - Dynamic motion prediction
The robot is able to project the effect of its motion to predict short term interactions
with both static and dynamic objects in the environment that the system can
Level 3 - Function projection
The system is able to project the effect of its function onto the local environment in
order to be able to assess its effectiveness. For example a robot may assess the
coverage of a room it has cleaned in order to identify areas it has missed.
Level 4 - Rigid interaction prediction
The system is able to envision the effect of its planned actions on rigid objects and
structures that it has identified. For example it is able to predict how an object will
behave when grasped in a particular way.
Level 5 - Flexible object interaction.
The system is able to envision the effect its planned actions will have on flexible
objects that it has parameterised.
Level 6 - Basic environment envisioning
The system is able to observe events in the environment that relate to the task
and envision their impact on the actions of the robot.
Level 7 - Envisioning safety
The system is able to assess the safety implications on users of observed events
occurring in the working environment.
Level 8 - Envisioning user responses
The system is able to envision the actions of a user responding to events in the
Acquired Knowledge Levels
Operating environments will always contain a number of unknowns. In many
proposed application areas robots will encounter unknown objects and environments
as a normal part of task execution. The acquisition of knowledge about both
environments and objects is fundamental to the success of these new application
Level 0 - No Acquired Knowledge
The robot does not acquire knowledge during its operation. Required knowledge is
embedded in the system.
Level 1 - Sense data knowledge
The system is able to acquire knowledge about its environment based on sense
data gathered moment to moment.
Level 2 - Persistent sense data knowledge
The system is able to accumulate knowledge about its environment based on
sense data that persists during the execution of the current task.
Level 3 - Property knowledge
The system is able to acquire knowledge about the properties of objects in the
environment by observation.
Level 4 - Deliberate acquisition
The system is able to acquire knowledge about the composition of its operating
environment by executing actions that are deliberately designed to increase
knowledge through exploration. For example to determine if a cup is full of liquid.
Level 5 - Place knowledge
The system is able to accumulate knowledge about the location and types of
objects and environmental features in terms of matching objects to pre-defined
and known types.
Level 6 - Knowledge scaffolding
The system has the ability to integrate embedded knowledge of objects and
places with related knowledge gained from the environment.
Level 7 - Requested knowledge
The system is able to recognise that it has insufficient knowledge about an object
or place relevant to the task and can formulate a question to gain that knowledge
either from a person, or an external data source such as the internet or another
Level 8 - Distributed knowledge
The system is able to communicate its gained knowledge to other robots or
systems and can receive and integrate knowledge from other robots or systems.
Level 9 - Interaction acquisition
The system is able to acquire knowledge about its environment and objects within
it through planned interactions with the environment and objects. For example the
robot deliberately selects an object of interest and picks it up to examine it more
closely, putting it back where it picked it from.
Level 10 - Object function
The system is able to acquire knowledge about the function of objects in the
environment. This knowledge may be acquired directly or indirectly through
Level 11 - User knowledge
The system is able to acquire knowledge about the user by observation.
Level 12 - Critical feedback
The system is able to acquire knowledge about its actions by analysis of critical
feedback that follows completion of the action.
Level 13 - Long term observation
The system is able to distinguish between long term and short term changes in
the environment and the objects within it.
Level 14 - Patterns of behaviour
The system is able to acquire knowledge about the patterns of behaviour of the
user that relate to the task. For example learning how to carry out an assembly
process by observation.
Level 15 - Observation learning
The system is able to acquiring knowledge indirectly from observing other robots
or people carrying out tasks.
Reasoning levels
Reasoning ability is the glue that holds the cognitive structures together. Perception,
knowledge acquisition, interpretation and envisioning all rely to a certain extent on
the ability to reason from uncertain data. As application tasks become more complex
the need to provide task and mission level reasoning increases.
Level 0 - No Reasoning
There are numerous simple robots that do not carry out any form of reasoning but
simply execute a pre-determined pattern of activity.
Level 1 - Reasoning from sense data
The robot is able to make basic judgements of sense data sufficient to allow
actions to be controlled.
Level 2 - Pre-defined reasoning
The robot is able to use basic predefined knowledge about structures and objects
in the environment to guide action and interaction.
Level 3 - Basic environment reasoning
The robot is able to use knowledge of the environment gained from perception in
conjunction with stored knowledge to reason about the environment. For example
it can build a map of the environment and plot a path to a goal.
Level 4 - Reasoning with conflicts
The system is able to reason about the environment and objects when there is
conflicting or incomplete information. For example missing sections of a map, or
competing classifications for an object.
Level 5 - Dynamic reasoning
The system is able to reason about the perceived dynamics in the environment.
Level 6 - Safety reasoning
The system is able to reason about safety in the environment.
Level 7 - Task reasoning
The system is able to reason about the appropriate courses of action to achieve a
task where there are alternative actions that can be undertaken. Typically the
system will be able to identify the course of action which matches the desired task
parameters, typically these involve time to completion, resource usage, or a
desired performance level.
Level 8 - Task hypothesis
The system is able to reason about the priorities of different tasks within a mission
and propose priorities based on its knowledge of the mission and the tasks. The
system will be able to fix on a task that must be achieved but make decisions
about how tasks will sequence to achieve mission objectives.
Object interaction levels
Cognitive ability plays a vital role in the interaction with objects in the environment.
The application of cognitive knowledge to the manipulation and interpretation of
objects through interaction provides a significant step change in the ability of a
system to interact.
These ability levels are modulated by the complexity and number of objects within a
given task as detailed by the cognitive ability parameters.
Level 0 - No cognition based interaction with objects
Many applications will not need to use any kind of cognitive interpretation or
knowledge in their interaction with objects.
Level 1- Environmental context utilisation
The system is able to use context information about the environment to guide
interaction with a specific object. This relates to the transfer of knowledge from the
environment to the manipulation of a specific object. For example knowledge
about a surface onto which an object is to be placed altering the placement
strategy, or knowledge about the relationship between objects.
Level 2 - Property Identification
The robot is able to pick up an object that belongs to one of a number of known
object types and determine properties of the object from its holding and
manipulation of it. It is able to use these determined properties to control how the
object is manipulated and placed. For example a robot may pick up a cup and
determine that it is full of liquid.
Level 3 - Object placement
The system is able to manipulate and place an object in a way that is compatible
with its state and context. For example property knowledge is used when orienting
and object.
Level 4 - Composite object manipulation
The system is able to identify that an object is composed of multiple different
objects that are connected but which may be separable. Within the context of the
task the system may be able to separate the parts, or exploit the union between
Level 5 - Generalised object manipulation
The system is able to interact with an unknown object and as a result of the
interaction categorise the object in terms of its categorical relationship to other
known or discovered objects. This includes generic categorisations such as “it is a
container for liquid”.
Level 6 - Novel object manipulation
Based on contextual and historical knowledge the system is able to establish that
an identified object is novel as district from being unknown. Novelty may result
from the object being broken or incomplete. For example a known mug is missing
a handle, or a bottle its cap. The system is then able to manipulate the object
taking into account its altered state.
Level 7 - Use of affordances
The system is able to deduce that an object affords an action. The robot is able to
grasp an object that has desired affordances within the context of the task or
mission and manipulate the object in order to gain use of the afforded action.
Human interaction ability levels
The following set of levels relate to different levels of human interaction with a robot
that have a cognitive element. They specifically relate to the interaction between a
human and a single robot. Where multiple robots are involved a corresponding set of
levels applies.
Level 0 - No Cognitive Human Interaction
Many robot systems will be able to operate successfully without cognitive
interaction with the user.
Level 1 - Fixed interaction
Interaction between the user and the robot follows a fixed pattern. Typically this
takes place via a user interface with well defined inputs and outputs. Typical of
this type of interaction are domestic vacuum cleaning robots which offer simple
button interfaces and display a minimum amount of status information. Fixed
interaction also includes interaction via a computer based user interface where
interactions directly control the robot according to pre-defined sets of commands
with specific meaning. The connection between the user and the robot may
involve a wireless link. Any interpretation of commands is fixed and embedded.
Level 2 - Task context interaction
The system is able to interpret commands from the user that utilise task context
semantics within a domain specific communication framework appropriate to the
range of the task. The system is able to relay task status to the user using task
context semantics suitable for the task.
Level 3 - Object and location interaction
The system is able to interpret user interactions that refer to objects, locations or
actions in as is appropriate to the task. This includes the ability to interpret user
interactions that identify objects locations and actions as well as processing
commands that reference locations, objects and actions relevant to the task.
Dialogues are initiated by the user.
Level 4 - Robot triggered interaction
The system is able to start a dialogue with the user in a socially appropriate
manner relevant to its task or mission. The robot has a basic understanding of the
social interaction appropriate to the task/mission domain. Interaction may continue
throughout the operating cycle for each task as is appropriate to the task/mission.
Level 5 - Social interaction
The system is able to maintain dialogues that cover more than one type of social
interaction, or domain task. The robot is able to manage the interaction provided it
remains within the defined context of the task or mission.
Level 6 - Complex social interaction
Dialogues cover multiple social interactions and tasks, where the robot is able to
instruct the user to carry out tasks, or enter into a negotiation about how a task is
specified. The interaction is typified by a bi-directional exchange of commands.
Level 7 - Intuitive Interaction
The robot is able to intuit the needs of a user with or without explicit command or
dialogue. The user may communicate to the robot without issuing explicit
commands. The robot will intuit from the current context and historical information
the implied command.
Cognitive parameters
The difficulty in achieving the above levels in each component ability depends on a
number of characteristics of the task and environment:
Environment: If the environment is unstructured and contains a wide variety of
objects this will increase the difficult in achieving cognitive ability levels. If the
environment contains dynamic elements or complex relationships between
objects then this will also increase the difficult in achieving higher levels of
cognitive ability.
Object Density: The object density of an environment refers to the number of
different objects that a system will encounter simultaneously. Where there are
many objects within the perception range of the system their number will make
cognition harder, the more objects there are the harder it will be to envision,
learn, interact and interpret. This parameter is orthogonal to the complexity of
each object and the variety.
Prior Knowledge: The ability to achieve cognitive abilities with respect to
environments, objects and interactions is strongly influenced by the level of
prior knowledge about each element. Prior knowledge may range from
knowledge about specific instances of an object or room, to no prior
knowledge. It will always be harder to achieve a cognitive ability level where
there is no prior knowledge of the elements that will be encountered.
User expectation: The level of user expectation and experience will impact on
the perceived attainment of cognitive ability levels. Users with realistic
bounded expectation, or experience, will reduce the difficulty of achieving a
particular level of ability.
Time scale: If the time span of operation is longer then the difficulty of
achieving higher levels of cognitive ability increases. Similarly if the time scales
for observation and knowledge acquisition are longer there will be an increase
in difficulty levels.
Task risk: The difficulty in guaranteeing outcomes and the potential need for
the certification of decision making mechanisms in tasks with high levels of risk
will make the attainment of high levels of ability more difficult.
115 Ability Targets
The broad impact of cognitive ability on a wide range of robotics applications puts
the main emphasis on the improvement of the different components of cognitive
ability in a wider range of tasks. Each application will have different requirements for
cognitive ability and this has an impact on the development of generic cognitive
The following R&D&I activities can be identified in addition to the above:
Cognitive integration: Integration of the different types of cognitive ability in
terms of formalisms and control. Focus should be placed on developing the
interoperability of multiple, hybrid representation and reasoning frameworks
and on the principles of designing flexible and scalable control architectures
that can change with the robot morphology and with the multiplicity and
distribution of processing units.
• Robot centric AI tools: The development of tools for knowledge representation
and reasoning that account for the specifics of the the robot environment that
can be applied across all the cognitive abilities will have a significant enabling
impact. Handling aspects of user, object and environment interaction within a
common framework able to handle uncertainty and failure modes will enable a
more integrated approach to the development of cognitive systems. It is
important that such systems are demonstrated within real robot contexts to
encourage early experimentation and real world deployment. Key Barriers
The main barrier is the problem of integrating into a complete robot control system
state-of-the-art modules that locally realise abilities in isolation. Formalisms and
algorithms over different modules (perception, planning, learning, envisioning etc.)
are typically incompatible. The top-performing state-of-the-art modules are often the
hardest to integrate, because they use sophisticated and incompatible
representations and algorithms that must first be adapted to the needs of robot
Designing a suitable robot control architecture that includes a number of abilities as
part of the integration process remains a key goal. Currently there is no dominant
solution. There will be considerable benefit in making it possible to transfer a
cognitive architecture from one robot system to another. This will enable cross
architecture comparisons to be made which in turn will contribute to developing
deployable cross platform systems.
These issues currently represent significant barriers to the wide scale deployment of
cognitive abilities in robot tasks.
3.10 MAR Technology Readiness Levels
The titles for each TRL level are taken from the definitions in agreed Horizon 2020
documentation that can be found here:
( )
However these titles are not obviously applicable to the robotics domain and so the
following represent an expansion of terminology and a followed by a series of basic
examples to clarify the intent at each level.
Level 1 - Basic Principles Observed
Idea: Basic technology research.
Document elaborated which describes a product / feature idea and/or
potential market requirement:
Functional description, customer benefit, ideas for realisation.
Level 2 - Technology Concept Formulated
Concept Formation: Basic technology research.
Proof of principle developments including algorithm development and
Concept formulated with details on potential development risks, including
coarse resource planning.
Level 3 - Experimental Proof of Concept
Experimental Development: Technology development.
Realisation of parts of the Concept to visualise the product / feature idea;
proof of concepts, first components and interfaces developed;
lab experiments carried out; future technical scope of work identified.
Level 4 - Technology Validated in Laboratory
Experiment: Technology development.
Testing of system or major sub-systems; validation against established
Testing of internal and external inter-connectivity.
Initial normative testing with trained users possible.
Level 5 - Technology Validated in Relevant Environment
Lab prototype: Internal technology demonstration.
Main functionality of product / feature idea can be demonstrated.
Major risks for the realisation of a future product / feature have been
documented as part of the description of the Demonstrator / realisation.
Level 6 - Technology Demonstrated in Relevant Environment
Functional model/First Field Trials: External technology demonstration.
Main functionality of a of product / feature idea is realised at a degree that
selected customers can carry out tests, when accompanied by developers.
Level 7 - System Prototype Demonstration in Operational
Engineering Prototype
Development of prototypes with final technology sub-systems or close
analogues in a close to complete form factor.
All identified functionality is capable of being demonstrated.
Customer verification trials (independent of developer support) possible.
Level 8 - System Complete and Qualified
Production Prototype
Development of prototypes with final functionality and form factor.
Sufficient for end user testing in limited launch markets.
Initial batch production of the products.
Level 9 - Actual System Proven in Operational Environment
Series production and sales.
118 TRL Examples
TRL Level
Example Flying
Inspection Robot
Example: Stroke
Rehabilitation Robot
Example: Shared Assisted
Living Robot
High Energy density
propulsion system
Mass individualised stroke
Economically feasible assisted
living support robot through
shared use
Long duration flight
capability using small
flying robots
Method of safely adapting
support and / or resistance
for leg exercises
Concept for determining
maximum utility tasks that can
be achieved with minimum time
and minimal robot functionality.
Realisation of long
Development of exoskeleton Development of an assessment
duration jet propulsion unit for adaptively supporting
system for determining
maximum utility tasks that can
be achieved within a given
timescale and robot resources.
Development of long flight System for additive exercise,
duration flying robot under including motivational video
tele-operation control
feedback, tested with healthy
Development of a robot for
efficiently undertaking maximum
utility tasks in a lab from a
limited range of tasks
Development of flying
robot unit with ability to
identify and follow wires in
a relevant environment
System for assessing and
applying clinically valid
exercise regimes developed
with clinician support and
tested in a relevant
Development of a robot that
assesses potential tasks to be
undertaken in a relevant
environment and performing
those which achieve maximum
utility for the user within a fixed
Outdoor tests of flying
robot capable of following
high tension cables with
manual supervision
Full system tested in limited
clinical setting with close
engineering and clinician
Full system tested in an
individual home setting with
engineering support.
Testing of high tension
cable inspection robot
capable of following
cables and collecting
detailed data on
anomalous areas
Full (improved) system
tested in a clinical setting
with limited engineering
support and thorough clinical
analysis compared with
standard techniques.
Full system tested by a service
provider in sequential multihome settings with multiple use
in each home. In depth user
satisfaction studies performed.
Testing by utility company
and developer of
autonomous high tension
inspection robot capable
of being deployed and
recovered by support
Full system linked to patient
database available to trained
clinicians for long term
rehabilitation trials.
Full system deployed by service
provider in long term tests with
user driven robot request and
allocation system.
Autonomous long duration System available to clinicians Full system available to lease by
high tension cable
and home users on a
local care authorities.
inspection robot
commercial basis.
commercially available
4. Technologies
One of the primary outputs from investment in research and innovation are new
technologies and improvements in existing technology. It is the purpose of the
innovation pipeline to push these technologies to market while generating expertise
and creating the skill base is needed to deploy and develop technologies and
A fundamental part of this strategy is the desire to identify technologies that are
mature enough to be pushed to market coupled to the identification of technologies
that are critical to multiple application domains allowing a maximisation of impact.
Critical to this process is the identification of “step changes” in the capability of each
technology. In particular the identification of generic step changes that will impact
across multiple types of application and domain.
This strategy requires the ability to measure the current “state of the art” of each
technology and to characterise its potential impact on applications and domains by
establishing a progression for that technology. Both of these tasks carry significant
difficulties. The measurement of maturity is compounded by the differing needs of
market domains, and establishing a progression of capability is often dependant on
the synergetic function of multiple technologies.
Technologies and Definitions
The roadmap uses a set of high level technology clusters to organise the underlying
technologies. Within each technology methods and techniques to achieve capability
are only outlined. The progression in each technology is identified by capability
targets rather than through descriptions of method and technique. Technology Clusters
The following major technology clusters are used within the roadmap:
Systems Development: Better systems and tools.
Human Robot Interaction: Better interaction
Mechatronics: Making better machines
Perception, Navigation and Cognition: Better action and awareness
4.2 Systems Development
“Better Systems and Tools”
As the processes required to develop robots become established the development
and analysis of those processes becomes more important. This cluster of
technologies relate to the design process of robots and robot systems. It is well
understood that saving time and cost during the development of a new product is
most easily done during the early parts of the development phase. Tools, processes,
and design systems can all help to streamline development. Investment in these
technologies is critical to the timely development of products and services and a key
enabling factor in the stimulation of a viable robot industry. Technology Description
The technologies underlying Systems Development focus on the methodological and
software aspects used in system development and integration process. This
combination of technologies aims to deliver the underlying engineering methods and
software tools for robot developers and manufacturers to use during the design,
development and integration phases.
Robots are the result of integrating a wide range of technologies. With sophisticated
software components and complete subsystems of ever-increasing functionality
becoming more and more available as commodity products from other sectors
(mobile devices, communications, automobile driver assistance systems), the
success of robotics will crucially depend on the ability to manage the integration of
these complex systems into powerful and cheap robots.
As shown in the EFFIROB study,2 “the development of software leads to significant
costs for both manufacturers and integrators of service robots”. This situation arises
from the current lack of sufficient technologies and methods regarding system
integration necessary to deal with the broad scope of heterogeneous robotic
systems. Software-framework and robotic middleware development projects focus on
sub-domains (e.g. real-time sensor-actuator networks, sensor data fusion, subsystem integration, cognition). This has led to greatly refined capabilities regarding
sub-system design as well as horizontal integration. Smooth vertical integration has
yet to be properly addressed.
There is a demand for systematic and scientifically backed approaches for a handover of robot building blocks for system integration in form of modules with explicated
properties from one stakeholder to the next. Such a system of modular integration
will stimulate component supply chains and significantly alter the robotics market
place. For these reasons Systems Design is a key cluster of technologies that
require R&D&I focus.
Current state of the functional landscape:
Usually no support for system development processes (indicative of the lack of
overall architectural models and methods).
Lack of decision support for package/module choice with a semantic description
(machine interpretable) including additional information such as performance
parameters, component level certification or standards met.
Available integration frameworks and middleware are diverse and ultimately narrow
in scope. ROS, YARP, OROCOS, Open-RTM, SmartSoft, … are good for particular
use, especially when specific functions are known to a developer.
Control theoretical approaches usually focus solely on mathematical models and
methods while neglecting capabilities and limitations of both current computer
systems and dependent higher-level software.
Safety and complexity of systems (autonomous/cognitive) are non-natively
compatible, due to the presence of inherently unsafe components, the rigidity of
safety requirements/validation and the complexity of functional safety relationships to
be checked within a complex system.
Explicated resource-awareness and Quality-of-Service attributes are missing. In
consequence, one cannot configure or provide robotic systems with resources just
adequate and sufficient for an application. Key Techniques and Methods
Generic approach
Systematic approach to the core areas of the development processes, including
advanced requirements engineering. Specification of the overall system architecture
and its automated design includes issues such as communication topology,
reference architecture, open architecture and modular service architecture. A major
goal is the high degree of re-usability of components.
Modelling in terms of levels of abstractions separating technical issues of
implementation from conceptual issues of functionality is a first step. Modelling
includes usability and reliability, scalability and evolvability, usage (human-machine
interaction, user acceptance, ergonomics).
Separation of roles is a key property to be supported in the overall life-cycle of
robotics software (development, system integration, deployment, run-time) in order to
allow for innovation along structures as typical and established in other high-tech
industries. Main roles are component developers, system integrators, application
domain experts, framework developers, tool developers, professional user or
consumer. This enables the collaboration at the level of meta-models, models,
standards etc. Especially, free and open standards, maintained in a dialogue
between academia and industry, can act as a structuring element to facilitate a
robotics business ecosystem
Key techniques/methods:
Model-driven engineering: core technology which has reached mature level in other
domains, but not in robotics. Model-driven software development (MDSD) and DSL
(domain specific languages) are core technologies in order to achieve a separation
of roles in the robotics domain while also improving compose-ability, system
integration and also addressing non-functional properties.
Use of semantic technologies, e.g. by semantic description of functionality and the
interfaces, which allows to search for "off-the-shelf components" with explicated
properties and rely on conformance for compose-ability during design space
exploration. This increases the re-usability and will enable to offer black-box (protect
intellectual property) or white box (open source) components.
Use of patterns on all levels (architectural, for model-driven design, …)
Architectures and design methods which allow integration of cross-sectional
requirements, e.g. safety, fault-tolerance, uncertainty etc. in different parts of the
system, but ensure overall system properties regarding these aspects. The
architectures should allow for hard, firm, and soft real-time as a system level
property supported by Hardware/Software platforms both in processing and
communication (mixed-real-time).
Hardware/Software co-design
Formal methods in software/system engineering are techniques and tools based on
mathematics and formal logic used to describe the high-level behaviour and
properties of a system without constraining the implementation. The tools can check
automatically if the system behaviour and properties satisfy system requirements,
such as real-time guarantees, dependability, reliability, robustness. Even if most
techniques are domain-independent, research is needed to effectively use them in
robotics, which is more challenging than other domains, such as avionics and
Middleware and Virtualisation are technologies used to improve computer system
performance, customisation, resource control, and reliability by decoupling the
behaviour of hardware and software resources from their physical implementation.
Simulation and Hardware/Software-in-the-loop Expected Step Changes
Technical step changes:
New techniques for adding software-intensive components, for plug-and-play
of smart devices and smart physical subsystems
Standardisation (plug and produce as candidate for standardisation,
programmability, etc)
Semantic modelling and re-use of components (link to cognition,
Functional integration of complex systems and simulation of overall system
behaviour including HW/SW-in-the-loop techniques.
Mixed real-time system capability, explicated resource-awareness and
Quality-of-Service attributes
Advanced configuration (from programming to task specification) by user
New processes for integration instead of adding components as long-term
goal (“Science of integration“)
Development of design patterns and the creation of reference architectures.
The import of best practice in Systems Design technologies from other market
domains (e.g. Automotive, aerospace etc).
Non-technical step changes:
Provide added-value to the European robotics community by proactively
disseminating the scientific outcomes to research and innovation projects in
order to avoid duplication of efforts and to accelerate industrial innovation and
exploitation cycles (e.g., by improving their software development activities).
Establish Robotics Software Systems Engineering as a science in itself to
enable innovation and to help build a robotics business ecosystem relying on
a balance of effective collaboration and competition.
Development of business ecosystem based on emerging standardisation and
modularisation of components
Development of certification and qualification systems for standard modules
and components.
Systems Design
Robot design methodologies that consider system integration
Unified tools allowing for: graphical design, design verification, code
generation, model-based verification, safety-by-design.
Standardised metrics for robotic component quality
Requirements engineering
Methods for automated design/ configuration of robotic applications based on
neutral models of production resources, products and processes
Automated production system design and verification down to the process
and control level
Systems Engineering
To develop Systems Engineering tools specific to the design of autonomous
and semi-autonomous robots, in particular addressing the integration and
deployment of whole systems composed of multiple robots, and the
interaction between system and environment.
To ensure best practice in the wider systems engineering community is
rapidly absorbed into the robotics community through collaboration and to act
as a driver for system engineering tool development.
To establish widely used standard interfaces that enable the modular
construction of systems.
Develop strong open source support structures within Europe to enable the
wider use of modular systems.
Definition of developer roles (component developers, system integrators,
application experts, framework/ tool developers etc.)
Software Product Lines: automatic selection of components - Automated
testing and verification for components, systems and applications
Generic process model for cross-market robot software development,
providing dependable, safe, configurable, adaptable and reusable applications
Tools and methods for engineering complexity in multi-robot systems.
Systems Architecture
Development of modular system architectures with well defined interfaces
Development of modular architectures able to allow system modification and
the provision of additional capability.
Cross domain architectural solutions and standardised interfaces
Plug and play component systems
Architectures based on multifunctional sub system blocks, for example
allowing integrated sensing and actuation or multi modal sensing.
System architectures that inherently support distributed systems components.
Architectures that work across multiple operating environments and robot
Reference architectures for several domains, considering mixed-real time
aspects, resource awareness, dynamic deployment, communication
topology, open architecture, and service modularity
Instantiation and tool support for reference architectures (including
verification, versioning etc.), support for distributed computing and control
Formulation of fault tolerance requirements - Mechanisms for ensuring fault
tolerance, feedback for System architecture
Architectures considering challenges in areas of security, usability and
reliability, scalability, evolvability, human- machine interaction, user
acceptance, ergonomics
General reference architecture, able to be instantiated for different domains
addressing the points mentioned under medium term
Systems Integration
New processes for adding software-intensive components,
Model-based verification (automatic transition from system design to deployed
run-time system)
Standardisation and integration of physical parts (hardware and mechanics)
Off-the-shelf components with open interfaces verified on the field or with a
significant TRL to base robotics application upon
Integration of mixed-real-time systems in form of applications
Reusable software components and model-based design components that
meet requirements for industrial applications (safety, security, reliability,
timing, throughput, response time, fault tolerance, verified, deployment,
scalability, maintainability)
System integration simulations.
Methods for the creation of self identifying modules and self configuring
Methods and tools for implementing multiple instances of a system and
adapting it to slight changes in specification
Validation of such systems in production environment
Tool support for component compatibility verification (“integration simulator”)
Complete description of mechatronic behaviour by merging mechanical and
electrical modelling methods with computer science modelling methods 100% virtual validation of the system
Development of domain specific languages
Models for the entire life-cycle of a robot
Models for dealing with information and knowledge processing that are fused
with models of physical world, involving concepts of time and space
Multi physics modelling of robot components (fluid, current etc.) - integration in
real time control
Knowledge Representation
Semantic web technology, as already used in some projects
Reasoning to connect information from different databases (data reflection)
Standardised semantic description of multi-modal representation of
environmental information, robotic resources (ontologies) and production
Resources capable of automatically translating process data into resourcespecific operation programs/parameters Benchmarks and Metrics
Technical metrics:
Standardised tests, time-to-failure, and other known metrics
Time needed to design and commission a new production system
Non-technical metrics (or assessment criteria):
Assessment by industry, success indicated by investment in methodology or
tool set.
Measured re-use of project results, define degree of reusability
the same set of meta-models (they are independent from a specific
programming language, operating systems, middleware, robot platforms,
toolchains) is being used in 5 to 10 robotics projects (academic, industrial)
Budget of project dedicated to system integration (currently management,
research and demo costs are delineated, can system integration costs/efforts
be identified during planning process?)
Pilot installations (with solid documentation, accessibility) to compare different
approaches, e.g. mobile manipulation for individual commissioning in food
processing. These installations could help to analyse the current approaches
for integration and to extract detailed requirements for integration processes
and the tools.
Project implementing the same system with different approaches to assess
the pros and cons in a vertical way (from the model to the physical
installation) instead of an horizontal way. Impact on Domains and Products
The management of complexity in design is critical to every domain.
The envisioned processes and tools are applicable to all application domains.
However it is likely that no single process model or tool chain will apply to all
domains. Generic process covering several related application domains will be
developed, tailored to the individual requirements of those domains.
Systems Design technologies will provide the following key benefits:
To reduce risks and effort during system development
To reduce costs, development time and time-to-market
To increase robustness and the quality of products and services.
More specifically, the expected impact includes:
Systems with well defined properties (safety, robustness, modularity,
resource-awareness, quality-of-service)
Make development simpler and more manageable increasing the take up of
robotics technology
Lower barriers to entry for small companies
Commercial use of new safety and cognitive functionalities, e.g. dynamic, selfdefined safety zones around a robot, decision-making functionalities that are
“safe” for use in defined settings
Current Key Projects
EU projects
Former project RoSta (still mention?)
National projects:
iserveU / Service Robotics / BMBF ICT 2020 - Research for Innovations
Model-Driven Software Development is being used to implement intelligent
modular service-robotics functionalities in everyday environments by the
example of a hospital robot
ROAMFREE (Robust Odometry by Applying Multi-sensor Fusion to Reduce
Estimation Errors, funded by Italian Ministry for University an Research, Italy)
developed a general framework for robust pose estimation by sensor fusion in
mobile robots and the corresponding component (in the form of open source
C++ and Python library + ROS node)
R2P (Rapid Robot Prototyping, funded by Regional and Private funds, Italy)
developed an embedded framework for the rapid prototyping of robotics
applications through off the shelf, low-cost, robotics components (e.g., motor
control, inertial measurement unit, uROSnode gateway, proximity sensors, etc.)
and a real-time communication middleware (based on a real-time protocol for
CANbus). Impact on System Abilities
System abilities in the context of systems engineering and systems integration are
twofold: On the one hand, the development tools and methods will have specific
abilities but the application of these tools will allow the resulting systems to have
specific properties and abilities resulting from a systematic development processes.
The following highlight the impact that Systems Development technologies can have
on the achievement of System ability levels.
Open, standardised and simplified/ automated integration of individual
components (e.g. sensors) – zero setup time.
Fast, software based reconfiguration for a given product range
Abstract (black-box) modelling of resource requirements, capabilities and
variation points. Enhanced system synthesis methods for ensuring smooth
system level composition,
Self-configuring, modular systems capable of autonomously re-arranging the
shop floor structure in compliance with safety or other non-functional
Real time reconfiguration intelligence using design-time models to guarantee
an appropriate quality-of-service given only sparse resources in open-ended
Advanced interfaces for enhanced and simplified programming by user
Multi role capable equipment – Auto-selection of functional equivalent
components according to system internal and external environment.
Interaction Capability
Standardised communication protocols (machine – machine)
Programming assistants
Easy monitoring and diagnosis of system - runtime inspection
Robot to robot contextual transactions for local cooperation in decision making
and troubleshooting
Methods for error tolerance when dealing with mixed-real-time systems,
application based
Dependency aware update and maintenance of components and applications
(updateability/ maintainability) - Self healing
Preventive maintenance, in-time detection of imminent failures
Overarching design patterns / architecture with best practices
Fully automatic recovery of the system – self repair through changes in
production operations or physical system structure
Perception Ability
Sensorial networks / sensor fusion - real time diagnostics, fault analysis and
Decisional Autonomy
Initial experiments with methods for error tolerance when dealing with mixedreal-time systems, application based
Resource level autonomy – Selecting and implementing among pre-defined
reaction scenarios
Decentralised, service based control at line level - multi actor and multi-level
decision making
Resource level autonomy – Generating and implementing new reaction
scenarios based on previous knowledge
Cognitive Ability
Generalised, neutral task and resource modelling- Resource programming
automatically achieved based on resource and task characteristics
System architecture allows for smooth integration of cognitive functions with
real-time control system and with safety functionalities
Resources capable of autonomously assigning themselves with the tasks to
Programmability is a system ability that uniquely relates to Systems Development
Task-oriented programming, i.e. specification instead of functional text-based
programming, (e.g. graphical programming environments, programming by
demonstration techniques)
Seamless migration between simulation and real robot - Hardware in the loop,
software in the loop techniques available.
Seamless migration between robot / computing platforms during runtime (e.g.
virtual machines on clouds)
Impact on other Technologies
Because of the wide impact of Systems Development technologies across all
aspects of Robotics Technology there are strong cross links between Systems
Development and the other technology clusters. The following provides an overview
of these common linkages which point to the co-development of solutions:
128 Mechatronics
Standardised component meta model to allow for better interoperability
between different middleware
Open and standardised communication architectures for integration of new
resources with minimal time and effort
Unified communication architecture/ protocols for all production related
equipment from the tool to the production line levels
Robots integrated into internet of things and making use of big data methods
and semantic web technology
Neutral format control models. Resource agents/controllers capable of
generating/adjusting control setting based on task description
Human Computer Interaction
Modelling and integrated development of interfaces
Standard interfaces for human robot interaction, including physical interfaces.
Safety by design. Cognition
Cognitive Architectures
First linkage of software models with decision mechanisms: software models
used at design-time explicate purposefully left-open variability for run-time
decisions and run-time configurations of the robot in order to better match
non-functional requirements.
Seamless interaction of software models and cognitive architectures: robots
make use of their software models for resource awareness and quality-ofservice decisions and configurations in order to achieve appropriate
robustness with scarce resources in open-ended environments.
Action Planning
Intelligent algorithms for the collaborative planning of tasks through real time
system status awareness
Unstructured generation and implementation of reaction plans based on full
system reaction capabilities (tool, resource and system levels)
System architectures that allow for distributed planning
Natural Interaction
Intelligence in safety mechanisms
Unification/ simplification of communication (modelling and real) channels
between hardware and software
Process models supporting distributed development and collaboration
Service Oriented Open Architectures (selection of reconfiguration scenarios)
Autonomous interaction and negotiation between resources for decision
4.3 Human Robot Interaction
“Better Interaction”
The new robot applications currently being explored are in many cases characterised
by greater levels of human robot interaction. The development of intuitive and natural
interfaces allowing the operation of complex robotic systems with less training and
lower fatigue levels is a key driver of this cluster of technologies. Interaction will take
many different forms from immersive virtual worlds to direct and precise physical
interaction. In all application domains improved human robot interaction will enable
new applications and new markets. Technology Description
Any of the proposed application areas for robotics technology involve human robot
interaction. This interaction can take place at a number of different levels ranging
from direct tele-operation to high level mission direction. Within these applications
many of the proposed interactions are physical and it is this development of robust,
safe and dependable physical human robot interaction that represents a technical
step change in robotics technology.
This step change in interaction has been triggered by the increasing use of
intrinsically elastic mechanisms equipped with embodied intelligence. The use of
Series Elastic Actuation (SEA) or its generalisation Variable Impedance Actuation
(VIA)) are key to this step change. Increased levels of physical interaction demand
changes in systems design methods to ensure that physical contact is fully
considered in the design phase. This is necessary to mitigate the possibility
unintentional contacts through better system design.
Early systems of this type are now entering industrial markets. In the development of
the next generation of interactive robots, Europe is clearly at the forefront and
leading the innovation in embodying intelligence in the mechanical design of robots.
{European examples needed to validate statement}
Potential applications range across all market domains with specific application in
Healthcare, Manufacturing, Commercial and Civil domains.
Future developments expect to be able to take into account operator intention and
create systems capable of longer term interaction sequences by integrating cognitive
abilities. Key to this integrated approach are better bio-mechanical models and
improved analysis of human injury.
Human robot interaction also covers the development of intuitive interfaces for the
command and control both of individual robots and teams of robots. Many
applications in the civil and commercial domains rely on single operators being able
to monitor and control teams of autonomous systems carrying out tasks and
missions. In these applications ensuring operators are fully aware of mission
progress and able to issue clear commands will require novel forms of interaction
design and multi-modal interfaces.
In contrast there are a number of application areas where there is close coupling
between the user and the robot, either physical coupling or one to one via an
interface. Typical examples are in Healthcare, surgical robots and rehabilitation
systems, in Manufacturing, co-workers and tele-operated systems, and in logistics,
warehouse systems and transport loading and unloading.
Future systems may be able to accurately and dynamically assess the safety of a
workspace controlling interaction and providing warnings to the user. If such systems
are to become a commercial reality certification and validation of safety will become
a critical part of the development cycle.
Bio-compatible mechanisms are also a critical part of physical human robot
interaction. While the design of these mechanisms is covered by the Mechatronics
technology cluster the nature of the interaction rests in this cluster. Barriers to market
The identified barriers to market can be clustered into the following ones:
Lack of safety standards.
Lack of certification and validation processes for human robot interaction
System Design tools for the integration of human robot interactions into
systems. Key Techniques and Methods
Historically robots have adopted the interaction methods of computer systems;
keyboards, mice, and screens. More recently virtual reality and immersive systems
have been used to interact with robot systems. The physically embodied nature of
robots provides the opportunity to extend these interfaces to provide physical
interaction. This can take place through haptic interfaces or by direct interaction with
the mechanical systems of a robot. Recent advances in soft robotics have provided
compliant interaction and this are now reaching commercialisation. However current
technologies for interaction provide significant scope for future development. Expected Step Changes
As with the other technology clusters there are considerable developmental links
between the clusters, the following lists highlight areas of cross development where
Human Robot Interaction technologies are impacted by technology developments in
other clusters. Systems Development
System Design
Design for high dependability of system components and interrelations
Concepts for systematic dependability analysis in the physical human robot
interaction domain
Dependability analysis frameworks for complex distributed soft-robots in the
physical human robot interaction domain
Systems Engineering
Formal methods application to produce safety proof
Numerical precision assessment of algorithms implementation
Programming models for interaction
System Architecture
System architectures for interactive robots
Basic physical human robot interaction architecture for complex robotic
Systems Integration
Integration methodologies for physical human robot interaction schemes
Modelling and Knowledge Engineering
Model-based testing and validation
Programming models for interaction Mechatronics
Mechanical Systems
Paradigms for human compatible robot design (soft-robotics design)
Novel human-friendly designs through synergy analysis and understanding
Intrinsic safety achieved through mechanical design
Analysis of human musculo-skeletal system, sensorimotor control and
biomechanics by embodiment analysis
Proprioceptive sensors for soft-robots (torque sensing)
Tactile skins
External sensors for workspace surveillance and interaction
Novel actuators and model based control principles for soft-robots,
sophisticated interaction control
New VIA actuators with human-like performances in weight, torque, stiffness,
power and efficiency
Dynamic modelling of human-friendly robots
Interaction control/soft-robotics control
Reflex control for collision safety
Embodied intelligence control methods for soft actuators Human Computer Interaction
Human Machine Interface
Multimodal interaction for interpretation of robot's state by the user including
acoustic and tactile indicators.
High resolution and multi-dimensional haptic feedback
More systematic understanding of human injury in robotics with particular
focus on representing the results in a robot centric view
Bio-mechanical safety metrics and standards
Industrial and domestic standards for human safety in physical human robot
Research targeted to meet international safety requirements Perception
Human sensing for safety
Human tracking and activity interpretation Navigation
Motion Planning
Human aware motion planning accounting for safety and ergonomics aspects
Basic motion planners for physical human robot interaction tested in complex
lab environments
Bio-mechanically Safe Motion Planning
Dynamic motion planning for online collision avoidance
Risk-metric based motion planning
Trading-off productivity with safety through multi-objective motion/trajectory
Real-Time motion planning for safe physical human robot interaction Cognition
Cognitive Architectures
Systematic frameworks improving the unification of perception, planning, and
control for physical human robot interaction
Advanced cognitive architectures able to estimate intention and social
intelligent behaviour based on physical human robot interaction frameworks
Learning Development and Adaptation
Definition/taxonomy of interaction skill library and according role distribution
Modelling adaptive interaction processes
Learning and adaptation of interaction control
Unified models for coordinated control & learning
Modelling of complex contact and force interaction skills enabled by pHRI and
imitation learning
Execution and control of complex contact and force interaction skills by pHRI
and imitation learning
Knowledge Representation and Reasoning
Real-time dynamic modelling of external environment and planning/modifying
actions accordingly.
Action Planning
Reflex planning and behaviours for physical contacts
Monitoring of interaction plans and behaviours
Fault tolerant planning of interaction schemes
Interaction planning for human-robot joint actions/tasks and according role
Planning/modifying actions in dynamic environments
Natural Interaction
Robot feedback for intuitive and safe interaction
Interaction planning and learning solutions for rather complex and dynamic humanrobot interaction scenarios
133 Benchmarks and Metrics
The effectiveness of Human robot interfaces has to be assessed by adapting HCI
criteria to robotics. It is expected that a knowledge base of interation techniques will
be built up over time as different types of interaction are tested.
For the assessment of physical interactions testing can be divided into physical
evaluation and psychophysical evaluation. Physical evaluation is related to
kinematics, elastostatics, dynamics, actuation and sensing capabilities, impedance
range and control bandwidth. Psychophysical performance metrics is more related to
human perception of quantities (forces, velocities, vibrations) generated by
In addition to this type of interaction assessment it is critical to establish certification
for physical human robot interactions and to formalise processes and methods that
are able to provide confidence in the safety of robotic systems that physically interact
with humans. This also requires fundamental assessments of the type and severity
of injuries that could result from interaction with given types and designs of robot
system. From data gathered about injury risk it will be possible to create levels of
certification that can be applied to deployed systems.
The initial injury assessment process might consist of the following elements:
The generation of a set of standard controllers and performance problems (e.g.
maximum peak velocity achieved by energy storage and release) should be
defined, tested, and validated
Standard protocols for crash-testing (e.g. inspired from automobile crashstesting, however, tailored to the needs of robotics)
Set of injury indices that describe potential injuries (soft-tissue injury, fractures,
Robot centric representation of safety data in order to make it applicable to
robotics (very different from automobile, biomechanics, forensics, and accident
International injury database for collecting injury data from experiments all over
the world and making this available to the community
Crash-test dummy experiments can be used to assess the level of injury that
the robot might cause to a human during different types of collisions. A crash
test dummy needs to be specifically tailored for such experiments, since
dummies used in car crash tests do not have all necessary sensors (for
example collision of a robot with a human arm with or without clamping).
Classification of injury in robotics based on medically accepted descriptions,
large scale soft-tissue injury experiments that capture the inherent collision
characteristics and making use of existing literature/knowledge in
biomechanics, accident research,...
Quantify the safety performance of collision detection and reaction by suitable
experimental setups that are able to predict human injury/prevention during
accidental collisions
From these results a set of concrete certification assessments can be developed in
conjunction with a safety database. Data from benchmarks carried out against these
certification assessments can potentially be used in international industrial and
domestic robot standards for safety.
From this knowledge it should be possible to create safe systems. This will require
the integration of many different technologies in all areas of robotics. As a
consequence it will be important to define benchmarks to assess interaction safety
which it is hoped will eventually lead to a process of certification. In some robot
applications close to 100% dependability is required even under failure conditions
and developing Systems Design and Mechatronic techniques to effect this will be a
significant but vitally important challenge. Without being abel to sufficiently guarantee
performance many potential application areas will be blocked. This is a critical area
for R&D&I investment.
Independently of the above safety related tests it is necessary to test human robot
interaction systems for their performance. The field of physical human robot
interaction is still developing and so testing performance is a critical part of making
progress simply because human subjective experience of different scenarios needs
to be captured. The following represent possible benchmarks that might be used:
User studies rating the intuitiveness, complexity, … of the robot’s interaction
Robotic devices that work in physical contact with a human also convey
information through the physical connection. A number of parameters will be
critical for efficient human-robot interaction. These parameters can be
quantified either based on physical measurements, or through subjective
assessments of user interactions.
Human perception and state estimation is still at an early stage of development
and needs further research to be suitable for benchmarking. It is important to
begin to define metrics so that the required information needed can be used to
prioritise research.
Intention detection is a relatively new field of robotics research, thus no
standardised benchmarks exist. Guidelines and benchmarks for evaluation of
robot based intention detection will have to be proposed and evaluated.
Future robots will increasingly cooperate with humans and will need to predict
human actions based on the context of the cooperation and real time
observation of human activities. In certain cases the robot will need to have
close to 100% reliability, for example in the case of exoskeleton based power
augmentation. Other applications may be less critical. Metrics are therefore
required for the assessment of a robot’s decision making competence. Dependent Domains
The physical interaction with partially unknown environments in conjunction with
humans, is a key ability for many advanced robotics applications; particularly in the
Manufacturing, Healthcare and Civil domains. The potential impact of human robot
interaction technology is expected to be significant. For example, rehabilitation and
prosthetics are entirely reliant on safe highly dependable interaction. Impact on Domains and Products
The application of advanced human robot interaction technologies to manufacturing
will be highly disruptive. It has the potential to impact industrial competitiveness and
enable new forms of manufacturing in high value economies such as Europe. In
particular it has the potential to remove complexity from the application of robotics
technology in small scale manufacturing allowing SME’s to benefit from higher levels
of productivity.
In healthcare dependable human robot interaction is one of the main core
technologies that are essential for applying robotics to helping the ageing society
and thus impacting one of Europe’s main societal challenges.
Safety certification will become an important aspect of the development and
deployment of close coupled human robot interaction systems especially when these
systems are deployed in everyday environments and in systems which are
Achieving safe and dependable human robot interaction is likely to become one of
the most significant technical step changes in robotics.
4.4 Mechatronics
“Better Machines”
Mechanical systems moving under control form the basis of all robotic devices. The
development of “Better Machines” has always been at the heart of R&D&I activity in
robotics. The success of future applications depends on making step changes in
capability within this technology cluster and in particular the development of
certifiably safe mechanical systems is paramount to the closer interaction between
humans and machines. Description
Mechatronics, defined as the close coupling of mechanism, sensors, actuation and
control, has always been a cornerstone of robotics. This group of technologies is
critical and fundamental to the functioning of every robot. A high proportion of the
core innovation in robotics lies within developments in mechatronics.
Mechatronics includes some of the most mature technologies associated with robots.
The importance of these technologies means that any significant developments or
improvements in capability will have a wide impact across all domains. Step changes
in capability are likely to result in observable product steps, impact across markets
and enhance competitiveness. Progress in mechatronics technologies will also
impact on improved system abilities notably:
Motion capability
Perception ability
R&D&I progress is also expected to yield safer systems for close working with
humans. Such systems will need to be highly adaptive to interaction constraints both
in pre-programmed or in unplanned and accidental interactions while at the same
time demonstrate efficiency and physical robustness.
It is expected that new principles of sensing, actuation and control will enable
performance levels to be closer to human abilities, in dynamic range, scalability,
adaptability, intrinsic compliance, damping, flexibility and dexterity. Developments
will lead to robots which are robust against mechanical stress and able to store
mechanical energy for highly dynamic motions, close to human athletic performance.
Key to these developments will be a deeper integration of sensors in a fault-tolerant
and synergistic way with control and actuation mechanisms.
The current interest in mechanically elastic actuation clearly illustrates the
advantages of executing mechanical design at a system level rather than at the
component level. Functionalities originally achieved by dedicated system
components (including control and software) can be embodied at system level with
superior performance and with a higher degree of integration. This holistic approach
requires new design methods, new multidisciplinary simulation techniques, and a
more general understanding of materials and manufacturing technologies. It is
expected that the resulting “Smart Mechanics” will have a significant impact on the
possible morphologies and functionalities of future robots.
In the future advanced manufacturing processes, particularly for micro-fabrication
and where sensing, actuation and mechanism are manufactured as an integrated
whole will provide higher levels of performance at the cost of complex production
pathways including chemistry and physics. This type of integrated multi-process
manufacturing will also require new design tools.
137 Key Techniques and Methods
The mechatronics cluster of technologies subsumes a range of individual
technologies each with its own methods and techniques. Much of the expected
progress in mechatronics comes form the integration of these technologies and in
the exploitation of synergies between them.
Details for each individual component technology are given in the technology
sections below detailing each underlying technology. Expected Step Changes
The step changes for each underlying technology are detailed in the individual
technology sections below, with respect to mechatronics as a whole and the goal of
making “Better Machines” in particular the following represent major challenges and
step changes:
Smarter mechatronic design: Tighter integration during design utilising highdensity sensor-actuator-mechanism integration, multiple articulation and
adaptable compliant actuators, mechanically embodied functionality and macro
to nano-scale robot manufacturing technologies.
Robust control methods: Mechanical and control design concepts enabling full
and continuous operation of the system even in case of individual joint failure.
Requiring self-diagnoses and safe fallback states for joint mechanics and
sensors as well as control modules for reconfigurable kinematic structures.
Smarter Mechanical System Design: Design and simulation approaches that
allow for a holistic system design that embodies functionalities using multiple
components of the system including software and control.
Interface standardisation: Standardisation of interfaces to sensors and
actuators as well as definition of standard communication architectures,
protocols and code generators for hard real time integration enabling high TRL
levels for plug and play systems.
Modular mechatronic components: Development of generic parts that can be
used for different kinds of robots and in different domains that allow for high
levels of integration. Mechatronic components such as sensors, actors,
structures and communication electronics that can be produced in quantity at
low cost.
Soft robotic systems: Development of soft robotics approaches utilising nonrigid structures and actuators and by breaking up the boundaries between
actuators and structures enabling mechatronic systems that intrinsically
integrate compliance, damping and sensors.
Bio-compatible robotic components: Development of mechatronic systems that
allow the seamless integration of robotics technology with the human body both
in healthcare and in terms of close coupled interaction systems. Ranging from
interfaces between the nervous system and computers, bio-compatible
materials and structures, implantable micro-sensors, bio-compatible actuators
and light-weight materials including micro-robots for navigation and action
inside the human body.
Reducing mechatronic component cost by a factor of ten: This can be
achieved by careful cost control and analysis better volume manufacturing
yielding more cost effective sensors (torque, position, tactile). Through
standardisation to widen the market for each component and plug and play
design to allow system flexibility.
138 Benchmarks and Metrics
Typical metrics for mechatronic systems are:
Cost per unit property
Energy efficiency
The primary benchmark for a number of mechatronic systems is comparison to
human performance in an equivalent task. For more industrial tasks benchmarks are
set by performance comparison to existing systems. Mechatronic systems will also
need to be benchmarked for safety and bio-compatibility both in terms of human
impact and in healthcare applications. Impact on Domains and Products
Mechatronics represents a baseline technology for robotics, all other domains and
nearly all embodied robotics development depends on mechatronic systems for its
success. Advances in mechatronics will impact on all domains and markets in
particular lower costs for a given mechatronic performance level will enable new
markets and expand existing ones. The ubiquitous nature of mechatronics within
robotics means that well focused and targeted R&D&I investment has the potential to
yield significant gains if major step changes can be achieved.
Research led advances in mechatronics has the potential to enable new robot
generations that have richer and more robust motion capabilities, are safer for
human-robot interaction, and more energy-efficient than current robots. Unknowns
One primary unknown is the impact of new materials technology on mechatronic
systems. While new materials scuba as electro-active polymers or nano materials
are often expected to yield major advances in performance that can take a
considerable time to permeate to the market. It is therefore important to also
incrementally improve systems by maximising the utility of known materials. While
this conventional development approach is less likely to yield substantial
improvements in performance the shorter time to market will allow the consequential
impact to be more easily realised.
4.4.2. Mechanical Systems Description
Many different types of robot depend on complex mechanical structures to perform
their tasks. Walking machines able to traverse rough or icy ground, micromanipulators used in surgical robots, or robots able to respond to an elderly person
falling all require specially designed mechanical systems. The wide variety of
different types of mechanical system mean that there are different perspectives on
key development themes which range from design methods to manufacturing
techniques and standardisation.
Key areas of interest for R&D&I funding are the design of miniaturised systems for
healthcare and manufacturing, the design of large scale systems for civil domain
applications and the development of systems and mechanisms inspired by the
impressive efficiency of the natural world. Key Techniques and Methods
Key techniques and methods centre around the development of improved
mechanical design tools and the use of novel structures and materials. The
integration inherent within mechatronics means that the mechanical system of a
robot is rarely considered in isolation from the other technologies within
mechatronics and the focus of R&D&I activity is on “mechatronic design” rather than
“mechanical design”.
Bio-inspired design also has a part to play in replicating many of the advantages
exhibited by natural systems by inspiring novel approaches to problems. Expected Step Changes
Step changes in mechanical systems relate to either cost vs performance criteria,
typically on the basis of multiples of previous performance, or on achieving higher
metric performance levels than previous designs. It is expected that even a doubling
of performance can have a significant impact in some market areas.
Step changes in miniaturisation relate to a scaling of magnitudes of size and
performance, for example moving from centimetre to millimetre scale mechanisms
will open new market opportunities.
The following have been identified as generic domain needs for mechanical
Low cost multiple DOF systems
Light weight mechanical systems
Large scale structures
Standardised mechanical interfaces enabling plug and play for mechanical
High mobility structures
Mechanical structures for grasping and manipulation, especially of soft
Smart materials, including self healing.
Miniaturisation of mechanisms particularly for grasping and manipulation (e.g.
Medical and manufacturing)
Bio-compatible mechanisms, both for exoskeleton use and use in-vivo.
Mechanical systems design compatible with extreme environments
Energy efficient and energy conserving structures.
Deformable structures able to absorb impacts and reform.
Intrinsically safe structures.
Bio-inspired mechanisms and properties such as adhesion. Benchmarks and Metrics
Mechanical system metrics can be characterised as follows:
Number of Degrees of Freedom
Limiting forces both exerted by and applied to the structure.
Torque and force related to mechanism size and cost.
Overall physical properties such as density and mass of structures with given
Energy efficiency.
Strouhal number (a dimensionless measure of thrust optimality).
Froude number (a measure of manoeuvrability), used as metrics for
benchmarking the bio-inspired robots against their natural counterparts.
Mechanical system benchmarks can be established based on combinations of
metrics relevant to specific tasks and to scalability. For bio-compatible structures
comparison to human equivalent performance is a useful benchmark. Impact on Domains and Visions
All embodied systems depend on mechanical structures for their operation, therefore
each domain application can be impacted on by advances in the design of
mechanical systems, either in terms of design process and modelling or in terms of
mechanism design and manufacturing processes.
The development of standardised systems, large and small scale systems, energy
efficient mechanisms and smart mechanical structures have the potential to enable
markets in all domains.
The development of miniaturised robotics assembly systems for manufacturing will in
itself impact on the assembly and cost reduction of miniaturised robotic assemblies.
The development of robust mechanical systems for use in hazardous environments,
particularly in marine, space and industrial environments will help to expand these
The development of bio-compatible systems both in terms of human interaction and
in terms of human embedding will be a significant enabler for numerous healthcare
The standardisation of mechanical interfaces will lead to cost reductions and system
flexibility that will enable many different applications and allow a modular approach to
system construction.
4.4.3. Sensors Description
What sets robots apart from other types of machine is the ability to sense their
environment. Sensing in 3D, sensing fine movements in a mechanical joint, or
providing a sense of taste and smell all require sensors.
Sensor technology is one of the most mature technologies within robotics. Sensors
are used in many products outside of robotics. The majority of recent advances in
sensor technology have come from the automotive, mobile phone and games
industries. Many novel types of sensor are also being developed for bio-medical
applications. Robotics stands to benefit from all of these developments. Most of the
sensors currently used in robotics are commodity parts readily available from
Sensors can be used to sense almost any measurable physical property. In robotics
sensors are typically used to measure motion, position, objects both in contact and
at a distance, light, both infra-red and visible, and sound, both audible and ultrasonic.
Sensors for heat, magnetism, specific chemical elements, pressure, colour, texture
and other physical properties are also used in specific applications.
All robots rely on the signals produced by sensors, for moment to moment decision
making, for interpretation of the environment and for control functions. Sensors are
used to detect the motion and position of mechanical links and joints, to detect the
physical environment around a robot, typically using sensors that operate at a
distance, and to detect direct contact between the mechanical elements of the robot
and objects in the environment. Sensors also play an important role in the user
interface allowing users to interact with the robot, either through touch screens,
buttons or by physically moving the robot.
Many different types of sensor are now produced with a wide range of parameters
(size, cost, power consumption etc.). This is a direct result of the increased use of
sensors in mass produced products in non-robotic applications.
The most recent advances critical to robotics have been in the development of direct
3D sensors able to deliver moment to moment depth information within a field of
view. These 3D sensors now beginning to be manufactured in small sizes at at mass
market costs. Key Techniques and Methods
There are numerous different techniques and methods in use across the different
types of sensor.
These can be categorised as follows:
Direct sensing of a physical quantity with an individual sensor element, typically
delivering an electronic signal.
Sensing of a physical quantity with an array of sensor elements.
The emission of energy, light, sound etc and the detection of a reflection to
measure distance by time of flight, phase difference or some other distance
related parameter.
The use of physical effects to influence an electrical circuit, for example
resonance, proximity or motion used to alter a circuit’s electrical character
(capacitance, resistance etc.).
The use of chemical interactions to create an electrical signal or pattern from
specific elements for example in gas detectors. Expected Step Changes
The following are seen as important changes that will take place in sensor
The integration of robot specific sensor processing within commonly used
sensors as commodity parts.
Establishment of standard sensor interfaces.
An increase in the resolution, range and signal to noise ratios of 3D sensors.
The development of improved mechanical force and torque sensing and their
Development of improved bio material sensing.
Development of improved chemical sensing.
The exploitation novel sensing mechanisms and materials, and the
development of multi-modal sensors.
The development of broad spectrum sensing technologies that increase the
dynamic range of sensors.
The extension of working ranges to different natural environments (e.g. allweather, temperature, light, etc.).
The development of low power sensors for use in distributed sensor networks.
The development of self diagnosing sensors and embedded signal quality
Multiple DOF tactile sensing
The maturity of sensor technology means that the step changes that will result in the
greatest impact are those which impact on extended sensing ranges, miniaturisation
and reduced cost per part. Benchmarks and Metrics
The primary metrics for sensing are;
dynamic range,
speed of response,
signal to noise ratio,
environmental operating limits (temperature range, supply voltage etc),
With optical sensors the following metrics are also relevant;
depth of field,
field of view.
All sensors are also governed by physical parameters that affect their utilisation in
robotics applications;
power consumption. Dependent Domains
All robotics domains depend on sensing. Some domains have a safety critical
dependence on sensing include medical and healthcare applications where sensing
failure may result in system failure and injury to users.
In many consumer applications optimising cost vs function is critical and designs
often require miniaturisation or low power consumption in order to be product viable. Impact on Domains and Products
Sensors are expected to undergo incremental improvements in in terms of cost vs
function, as defined by the above parameters, over the coming decade. There will be
critical cost points where a factor of ten improvement in cost vs function has been
achieved. These points are likely to enable new markets by providing significantly
better sensing for the same cost. These steps will result in an increment in system
dependability/functionality creating a significant improvement in end user experience
and therefore market.
In specialist applications the enhancement of response times, sensitivities and signal
to noise ratios may allow faster control loop times to be achieved in mechanical
systems, or faster sensing of the environment.
The miniaturisation and integration of force and displacement sensing into
mechanical joints will enhance control resulting in the development of more complex
multiple degree of freedom mechanical systems. These developments will have to be
matched by developments in actuation technology in order to achieve a broad
4.4.4. Actuators Description
A principle function of robots is to interact with the physical world. Actuators provide
this motive force. The rage of robot operation means that actuators must be
individually designed to meet a wide range of electro-mechanical requirements. From
large manipulators able to lift a car, to the high precision actuation used in surgical
robots, actuators must be able to meet the demands of continuous dynamic use.
Robotics needs new actuator concepts that directly provide the specific
characteristics required by robot mechanisms such as high forces/torques at low
speed, overload protection capabilities, back-driveability or ideal torque source
behaviour. These properties are needed at the micro- as well as in macro scale.
Such characteristics can be achieved by energy storage and release concepts,
functionally adapted designs, new materials and clever usage of micro-macro scaling
properties. Key Techniques and Methods
Actuator development is relatively mature as a core element in every robot. Electrical
motors provide the motive force in the majority of actuators. Hydraulic and pneumatic
and piezoelectric actuation are well understood. Novel actuators based on electroactive polymers and on bio materials are at an experimental stage.
The design of bio-compatible actuators for mechatronic devices that will cooperate
with humans and that have similar size and mass as humans; also require
comparable efficiency, power, strength, velocity and interaction compliance. Expected Step Changes
Incremental improvement in power to weight ratios and energy efficiency are to
be expected. Novel actuation technology and the employment of new materials
are expected to yield new opportunities.
Fine scale high resolution actuation, particularly in surgical applications is likely
to be a major growth area.
Next generation “ideal torque source”: Actuator prototypes that provide high
torques/forces at low speed without transmissions, are overload proof and can
provide energy storage or release capabilities available at the macroscopic
I-PIA (industrial passive impedance actuation): Variable impedance and serial
elastic actuation will provide enhanced robustness and dynamics to robots in
industrial applications and environments and enable a new era of automation.
Bio-compatible actuation and the development of actuation systems that are
intrinsically safe will be critical to applications where there is close coupled
human interaction.
High power to volume ratio actuation.
Actuators exploiting novel materials and advances in power generation and
energy storage. Benchmarks and Metrics
Metrics for assessing performances are:
Tensile stress
Power consumption. Impact on Domains and Visions
New developments in actuators will impact on all robotics domains. Improved biocompatible actuation will specifically impact in Healthcare and Manufacturing, where
robots operate in close and even physical cooperation with humans.
Rehabilitation robotics: compliant and controllable stiffness actuators for
improved safety and more flexible exercise programming
Exoskeletons: safe actuation due to compliance of the actuators
Bio-mimetic robots
4.4.5. Power Supply and Management Description
Robots will need to be able to operate for long periods without access to a source of
power. Managing their stored energy, designing systems that have low energy
requirements and managing the use of energy are key to extending the working time
of each robot.
This is an area of technology development where incremental improvement and step
changes come from outside of the robot R&D&I activity. Key Techniques and Methods
Systems design increasingly employs multiple power domains and allows for their
management. Energy storage typically relies on conventional battery systems, or in
rare cases on on-board electrical generation. Experimentation with electrical
generation from bio-mass is being carried out. Expected Step Changes
To increase system level efficiencies to reduce power requirements through
improved design and systems engineering.
To utilise alternative energy sources for long term operation. (Solar, thermal,
To improve the storage and recovery of power from mechanical systems
including human operators.
To investigate alternative power sources and track these trends in other
industrial sectors. Benchmarks and Metrics
The following metrics are typically used to characterise power systems:
Power consumption
Leakage under no load
Standby time
Energy mass density (Watts per kilogram). Dependent Domains
All robots require an energy source to operate. In many applications the operation of
mobile robots are constrained by the availability of energy supply and behaviours are
included to either recharge or stop before the available energy becomes too low to
operate. Impact on Domains and Visions
The lack of long term energy solutions will significantly impact on many long time
scale applications and alternative sources of energy for on-board electrical
generation will need to be considered, (bio-mass combustion, fuel cells and chemical
4.4.6. Communications Description
Robots need to communicate with each other, with internet based services in the
“cloud” and to the “internet of things” around them. Internally different modules need
to communicate and the development or adoption of existing of internal
communication protocols are a vital part of standardised module construction.
Wireless communication protocols, particularly ad-hoc protocols and mobile to
mobile systems will be of importance to robot communication. Key Techniques and Methods
Robots use a wide variety of existing communication protocols and methods, both
wired and wireless. Some specific industrial protocols are used in factory automation.
To date robots have been able to make use of generic protocols established for
mobile wireless communication and local area network communication these
protocols. In the future it may be necessary to define higher layer protocols
specifically for communication with robots, in particular the latency requirements of
teleoperation, the need for remote haptic feedback and the processing of high level
cognitive functions in the cloud may all require alternative network protocols.
Communication security is also a of critical importance in many personal applications
of robotics particularly where the users are elderly or vulnerable. The future ubiquity
of robots will mean that the data they gather as a natural part of their daily operation
will be of great value. Secondly the threat of robot “hacking” will be heightened by
poor security in control systems both in industrial and domestic settings.
In transport the current proposals for car to car and car to road systems will need to
be integrated into autonomous transport systems and in the limi these new
communications protocols may need to be augmented with autonomy specific layers
and security mechanisms.
The use of robots in the setting up of local communication infrastructures has a
number of applications:
In disaster scenarios where existing communication systems may be overloaded or
out of operation mobile robots can establish a communication network by positioning
themselves around the disaster site, placing themselves on existing structures or by
providing an airborne communication umbrella. Such a system can provide whole
site visibility and can be deployed rapidly using small vehicles. In the limit it can
dynamically reconfigure to adapt to changing demands and circumstances. Such a
network also allows other autonomous systems to communicate efficiently without
absorbing bandwidth on existing networks.
In agriculture multiple autonomous machines need to communicate to complete
tasks and missions, however the remoteness of farms and fields means that it is
likely that there is no existing communication network available. Systems are then
needed to establish a communication network across the fields and possibly into the
internet. Since autonomous machines may enter and leave the process it is
necessary to set up these networks independently of these machines and the use of
small autonomous systems to create a network is one solution.
Similar issues arise in other application scenarios involving teams of robots that will
need to establish ad-hoc networks at the operating site when this may be remote
from existing systems or where high bandwidth low latency communication is
essential to the team. Expected Step Changes
Ad-hoc networks built by and into autonomous teams.
Sub-sea network technology and protocols.
Robot specific protocols for low latency networks.
Low energy communication systems.
In-vivo communication.
Communication systems for micro and nano robots. Benchmarks and Metrics
Communication metrics are well known and understood and fully apply to robotics
Bandwidth/Data rate
Channel capacity
Error rate.
Signal power Dependent Domains
Communication capability will be particularly important in team operation, and
therefore in many Civil domain applications. High reliability communication will be
required in some Healthcare applications.
Various operating environments present specific communication issues. Notably
communication between aerial robots without access to a ground based network.
Underwater marine robots where conventional wireless communications cannot be
used can use either optical communication or low bandwidth acoustic
communication. Impact on Domains and Visions
In a number of application domains communication either between robots or to the
internet is a critical part of the application. In most cases existing networks will be
able to provide appropriate capacity. In some particular applications ground based
capacity will either not exist or have insufficient capability to allow tasks to progress.
In these cases the autonomous system must establish its own communication
4.4.7. Materials Description
Mechanical structure, sensors, and energy sources form the core of every robot.
Materials provide the engineered mechanical and physical properties of these core
elements. Advances in materials have the potential to have a disruptive effect on the
impacted technologies.
As a result advances in Materials technology often underpin new developments in
robotics, from the creation of novel sensors, and more advanced mechanical
structures to more efficient drive mechanisms and active structures. In time the use
of active materials and nanotechnology will enable mechanical structures that can
mimic natural systems and thereby create a new generation of robotic systems.
Advances in materials technology used in robotics are expected to have the greatest
impact in the following areas:
Materials for bodies and structures
Energy and power sources.
The development of novel sensing devices.
A primary challenge is to replace the current metallic mechanical structures with
polymeric or nano-composite materials having engineered physical and chemical
properties. The possibility to modify at nanoscale the physical properties of fibres
and to control the polymer properties enables the synthesis of nano-composite
materials with controlled stiffness and elasticity, light weight, electrically-active and at
a lower cost than the standard metal-based production.
Materials with such characteristics will allow in the future the design of “soft robots”,
with passive and active compliance and lower weight in comparison to today’s
machines. In addition, nano-composites such as carbon fibre materials, offer the
possibility to introduce conductive fibres or optical fibres as well as sensing arrays
that serve as an embedded nervous system in the scaffold of the robot.
Highly integrated sensing networks in the robot structure are crucial to accomplish
distributed sensing capabilities (such as touch and force sensing) which in turn are
crucial to develop the cognitive capabilities of the robots. Noteworthy embedded
sensing networks require new mechanical energy harvesting materials, exploiting
piezoelectric, tribo-electric, or electromagnetic effect to convert motion and vibration
into electrical energy. To this aim flexible, new soft functional materials, such as
hybrid piezoelectric (e.g. III-V Nitrides grown on flexible Kapton substrates),
polymeric piezoelectric, soft magnetic and piezoelectric nano-composites, and triboelectric composites, represent promising and emerging approaches.
This opens up the development of self-powered sensors and sensing networks and,
possibly, their wireless operation leading to a substantial simplification of the robot
design. The optimisation of materials in terms of weight and elastic compliance,
together with the architectural simplification given by the distributed
sensing/harvesting network will considerably reduce power consumption, which is
conversely another main challenge of future robotics. Robots will be required to
operate for many hours (>10), autonomously, in human and natural environments.
As such hybrid battery/super-capacitor as well as fuel cell technologies with power
density approaching the value of 400 Wh/kg have to be developed in conjunction
with flexible harvesting devices distributed throughout the body of the robot,
providing a few tens of watts during normal movements.
150 Key Techniques and Methods
Typically robots use conventional materials and processes for forming and shaping
components. Additive manufacturing is used extensively in the development of
robots. Research is being carried out to exploit the properties of novel materials in a
number of robotic domains.
Materials for bodies and structures:
The structural materials for future robots must be robust enough to sustain the
weight of motors, batteries, and other parts without undue deformation due to
bending or flexing. At the same time they have to be lightweight in order to avoid high
energy consumption during the robot’s activities; they also need to efficiently absorb
energy of vibrations and impacts without damage. In this perspective, for the short to
mid-term (e.g. next 5 years) the materials of choice to be developed for robots will be
polymeric-based for the entire endoskeleton (or exoskeleton) structure. Polymer
nano-composites, such as Carbon fibre materials, will substantially reduce weight
compared to metallic structures, also enabling the possibility to embed sensing
networks in the fibre matrix and to design the elasticity and mechanical
characteristics of the materials by a proper choice of the fibre chemistry and texture.
In the long-term structural materials will embed more life-like features, which in turn
will be similar to a wide range of natural structural materials like wood or bones. They
will be easily employed both as structural elements and for energy storage.
Examples of material classes to be investigated are metal foams and soft polymer
Actuation represents the bottleneck in many robotic applications, especially in
biomimetic ones. The available actuators are mainly electromagnetic and their
performance is far from natural muscles. The main limitations are in the relatively
large inertia and lack of back-drivability, stiffness control and power consumption.
Nevertheless, new and promising technologies are emerging, thus offering new
possibilities to fill the gap between natural muscles and their artificial counterparts.
Ideally, these novel technologies will be able to match, or even outperform, natural
muscle in terms of key robotic actuation metrics such as active stress, active strain,
power density, inherent compliance and stiffness control.
Energy and power sources for robots:
A future goal is to develop energetically autonomous machines possibly equipped
with an on-board supply system, which would be able to power the robot for the
duration of a mission or task. The power requirement depends on the
embodiment/size of the robot. For example, for human-size, service robots, power
consumption is estimated in the 800W-1kW range. It is expected that materials
technology will play a significant role in energy harvesting, energy conversion and
Novel materials for sensing devices:
Improving the capability and properties of sensors is an on-going challenge; smaller
size, lower power consumption, better performance. In addition closer human robot
interaction requires highly distributed sensing. Tactile sensors are one typical
example. Copying nature in this case would require as much as 1700 sensing point
in one finger with four categories of sensors detecting stimulus frequency and the
direction of forces and adapting over time to the external stimulation. Measurement
of the contact location, the direction of force, its distribution, incipient slip, texture,
temperature and moisture may all be required for dexterous object manipulation.
New materials through nano and micro-fabrication technologies (NEMS, MEMS) are
likely to be the only way to provide these types of step change in sensing. Expected Step Changes
To exploit new materials that can enhance the design of robots, though
improved sensing, mechanical systems or manufacturing processes.
To engage the materials science community in seeing robotics, particularly
medical robotics, as a new growth area in need of new materials to meet
complex sensing and mechanical design requirements.
The development of robot body structures using soft polymeric materials.
The use of polymeric nano-composites in the development of integrated
sensing and actuation.
The use of soft polymers for actuation; particularly Liquid Crystal Elastomers,
Dielectric Elastomers, Conducting & Ionic Polymers and Carbon Nanotube
• The development of novel and integrated muti-process fabrication techniques
based on new materials.
The use of metal foams as structural elements in robot mechanisms.
Development of multi-technology power systems based on new materials;
Including super capacitors, fuel cells, and energy harvesting.
The application of novel materials to the construction of Smart sensors
including the use of electro-mechanical and electro-magnetic properties Benchmarks and Metrics
Benchmarks and metrics in materials is an established discipline owing to the
consolidated physics and formalisation of the measurements of materials. Dependent Domains
The availability of new robot-specific materials has theoretically a wide impact on our
capacity to develop more efficient machines. Since material science research is a
long-term endeavour, we do not expect here to see progress in all possible areas
and especially with the more wild research ideas. It is likely that progress will be
made in some areas within a 3 to 5 years time frame. Particularly in providing new
composite materials for body structures – to e.g. provide weight saving and/or
controlled stiffness as well as embed some simple wiring in the robot structure.
Combined tactile sensing built on flexible materials (e.g. capacitive and piezoelectric)
is almost ready to be used at high TRL levels for applications and able to impacting
the ability of a robot to interact with people in real time.
It is expected that the impact of new materials will be most effective in advancing the
following robot abilities:
Motion Capability: the possibility of designing bodies with intrinsic controlled
stiffness improves adaptability to different environments, as e.g. adaptation to
rough terrains, to unexpected impacts with the environment, etc. Energy
autonomy is also important for adapting the robot to different operating
Interaction Capability: a full body skin system is crucial for human-robot
Manipulation Ability: new materials in the design of the hand (e.g. with intrinsic
compliance) can improve manipulation simplifying control considerably.
Perception Ability: a skin system can give robots a new level of perception
about the environment, objects, and their contact with the world (e.g. measure
forces at various points of contact while moving).
Integration and innovation of existing and new smart soft materials and composites
into micro- and macro-scale robot designs will adequately address sustainability of
future robots in terms of their maintenance, longevity and recycling. Particular impact
is expected in: sensing (e.g. soft skin to touch; deformable tissue; pressure, heat and
humidity sensing); movement (e.g., elastic and compliant materials for muscles,
tendons; variable compliant actuators; less energy needed for movement due to
lighter-weight materials); interaction (e.g., soft movements, social and cognitive
skills). Impact on Domains and Visions
Materials being a determinant of the shape of the robot have a comprehensive
impact into the final products as well as into their time to market. Materials will be a
key enabling technology of future robotics. It is expected that existing new materials
can be utilised to build better robots, sensors and actuators within a relatively short
time frame provided appropriate R&D&I investment is made. The following list some
the initial areas where this impact can be expected:
Apply newly engineered composites to the construction of the robot body;
Design robust sensor networks for advanced measurement of mechanical
interaction of the robot in various environments (being these marine, industrial,
harsh, air);
Design appropriate “energy” packs to make the robot autonomous and
rechargeable with reasonable down time;
Develop flexible and stretchable circuits to conform to the robot shape and
Use existing known fabrication techniques.
In the longer term the following areas will be impacted by materials R&D&I:
High-performance artificial muscles;
Functionalised exo- or endoskeletons (body structures);
Mixed mode energy packs (combining various forms of generators);
Self-healing materials;
Impact is to be expected in all domains albeit possibly to arrive first in micro or nanorobotics and partially in larger robots. For example, we can expect to see skin
systems and energy packs available in the short term on service or personal robots
and new body/actuation structures to reach first the micro or nano world of robotics.
Clearly, self-healing materials, fully stretchable electronics are for the longer term.
Although examples and prototypes are already available, turning them into
production is still a future goal.
4.4.8. Control Description
Control is one of the core technologies required in all robotic systems. It is relevant
for all domains.
Position, velocity, and force control for serial manipulator arms are well known and
available in industrial products. Vision based control is applicable in well controlled
environments. Control algorithms for floating base robot systems, like space robots,
aerial robots, and legged robots including humanoids are at a lower TRL. Additional
resources in these fields are needed to bring these algorithms closer to the market.
The new trends in mechatronic developments towards compliant, kinematically
complex robots interacting with humans and unstructured environments gives rise to
new and very challenging control problems. To address them, dynamic simulation
and control design needs to be pursued simultaneously with the mechatronic design
and to influence this in an early stage. Robust and adaptive control methods,
optimally exploiting the intrinsic mechanical properties of robots are attractive
alternatives to already advanced nonlinear, model based design approaches.
Safety issues in controlled robot systems interacting with humans have seen a major
progress in the last decade and are progressing towards application in commercial
Active compliant control is mature in the research environment with first products
emerging on the market.
Novel and efficient actuators connected to complex mechanical structures require a
coevolution of sensory feedback control methods to ensure that motion is fast,
accurate, stable, robust to environmental changes and repeatable. Controllers that
limit the impact of contact with the environment and cause the robot to react with a
compliant behaviour are critical for safe operation in collaboration with people. To
optimise the final robot performance hardware design and controller design need to
be closely interconnected from the very early design stage in concurrent engineering
and simulation processes.
Advanced nonlinear control approaches require precise models and sensory data of
high quality. In current industrial scenarios this leads to cost intensive solutions and
robust methods which can work at a high performance level under noisy, uncertain,
and incomplete sensory data are desired. Key Techniques and Methods
In the automatic control community exist a large variety of advanced control methods
which were adapted or specially developed for robotic systems. An exhaustive
overview of these techniques is beyond the scope of this document. The following
overview clarifies the terminology in the state-of-the art and highlights the most
common methods which were successfully applied to robotic applications.
Depending on the type of plant model it is useful to distinguish between:
Kinematic control that is based on a plant model which refers to the kinematics
rather than the dynamics. Kinematic control is often used in the context of redundant
robots and in mobile robotics.
Dynamic control refers to control approaches, where the plant model contains
dynamics effects like inertia. Depending on the type of actuator used and the level of
abstraction in the model, the control input can be a generalised position, velocity, or
force. Control problems for multi-physics system models with dynamic actuator
models (electric, hydraulic, pneumatic, etc…) also belong to this class.
Position, Velocity and Force control of serial manipulators on a fixed base are well
known and available in industrial products. Active compliance and impedance control
are mature in a research environment with first products emerging on the market.
Control problems for time delayed systems arise from several perspectives. In
telepresence applications, the combined master and slave dynamics is considered.
In visual servoing applications, the time delay is mainly related to the processing
time of the vision sensors and algorithms.
Among the industrial and research developments, several key techniques can be
PID control is widely seen in industrial applications. Tuning procedures are
based on a multitude of approaches including heuristic guidelines, formal
robustness analysis, and H_inf analysis.
A major breakthrough in position control of serial manipulators has been
achieved by the concept of inverse dynamics, which is also known under the
term “computed torque control” in robotics. The key idea of inverse dynamics
lies in an active compensation of the nonlinear dynamics by control.
Operational space control can be seen as a variant of inverse dynamics, which
utilises a model description based on task coordinates. Extensions to prioritised
control of multiple tasks have been proposed and are currently under
experimental evaluation on real systems.
Energy based approaches based on passivity generalise more simple concepts
like PD control. These methods often show stronger robustness than control
approaches, which aim at a compensation of nonlinear dynamics.
Interaction control based on force or joint torque sensing is well understood.
Control of passively compliant mechanical systems is at a high interest in
research laboratories around the world. Despite its lower TRL, first products
based on compliant actuation are appearing on the market.
Nonlinear model based control approaches like Back-stepping, Lyapunov
based control, etc. are established in the research community.
Adaptive control in robotics benefits from the linearity of the dynamics
parameters in the model.
Model predictive control is popular in mobile robotics and in walking control
where simplified template models are used for computationally intensive
Optimal control is used in both industrial and research context for offline
trajectory planning. The use of novel actuators with compliant elements has
renewed the interest in optimal control as a method for exploiting the full
capabilities of the intrinsic system dynamics.
Iterative learning control (ILC) is dedicated to precise control of repetitive tasks,
where the control action along a periodic task is iteratively refined based on the
control error during the previous iteration.
Flatness properties can be utilised in systems which allow for static or dynamic
feedback linearisation. Typical examples are mobile robots and under-actuated
Neural and Fuzzy control are model-free approaches used mainly in lab
environments. Expected Step Changes
Development of robust and adaptable control methods for the increasing diversity
and complexity of robot morphologies and tasks. Controllers will optimally exploit
intrinsic mechanical properties and provide safe interaction with humans, stable
locomotion and dexterous mobile manipulation in unknown environments.
Comparison with biological body control will give new impulses to robotics control
and provide a new perspective to biomechanics and neuroscience.
The steady progress in hardware developments including novel actuators and
materials as well as the realisation of more and more advanced robot systems for
demanding applications requires a co-evolution of appropriate control methods.
Expected step changes include the following challenges:
To devise adaptive, self-calibrating controllers: Controllers which do not apply
pre-set controller gains and control topologies but allow for self-organisation
and adaptive systems.
To achieve a closer integration of the reactive (feedback) and deliberative
(planning) parts in the control architecture. Therefore, the capabilities of
modern reactive control approaches have to be represented in an appropriate
form at the planning levels.
To develop fault-tolerant and resilient control methods: In complex robotic
systems, the operability of the whole system should not critically depend on the
operation of a single component (e.g. joint),
To control highly redundant and under-actuated robots.
To operate robots under hybrid (i.e. switching) contact situations with strong
dynamic constraints, e.g. in legged locomotion or humanoid multi-contact
To develop robust control methods able to work in noisy and uncertain
environments with incomplete sensory information.
To allow teleoperation of robots everywhere on Earth: Teleoperation is at a
mature development level from a research point of view, ready to be
transferred to industrial scenarios.
To develop controllers which are energy-efficient and optimally exploit the
dynamic properties of the system.
To develop bio-inspired control approaches.
To devise safe control strategies allowing for physical interaction between
human and robot, including wearable exoskeletons. Benchmarks and Metrics
Stability properties (of different types) for regulation and tracking tasks are seen as a
basic requirement in closed loop control. In the context of physical interaction with
humans or the environment, passivity plays a prominent role. Stability definitions
related to legged locomotion are less well established.
Quantitative measures on absolute accuracy and repeatability are common use in
industrial robotics.
Qualitative metrics and evaluation criteria of control systems are:
the complexity of the control law (for implementation),
the robustness with respect to parametric model uncertainties, as well as
the robustness with respect to structural model uncertainties (unknown
Specific benchmarks on generic control questions are not intended. Instead, domain
and application specific benchmarking plays a stronger role than benchmarking of
different control strategies. Dependent Domains
All robotic domains depend on control. Human robot interaction, healthcare, and
space applications have increased safety requirements and therefore critically
depend on reliable and safe control approaches. Impact on Domains and Visions
While control, in a mechatronic context, is necessary for nearly all applications of
robots in most domains the types of control are well understood and successfully
In many proposed applications for robotics technology it is the implementation of
novel control paradigms that are critical to safe and effective mechatronic systems.
In rehabilitation, in physical human robot interaction and in bio-compatible
mechatronic control methods play a critical functional role in providing safe and
appropriate motion.
Perception, Navigation and Cognition
“Better action and awareness”
The cluster of technologies that make up perception, navigation and cognition are
the ones that will have the greatest impact on the future performance of robot
systems. Enhanced awareness, characterised by step changes in Perception,
Navigation and Cognition, are at the core of all advanced robotics applications. Many
of the applications with the greatest societal impact require step change
improvements in this cluster of technologies.
4.5 Perception
Perception technology provides a robot with the means to measure and interpret its
environment. In order to enable intelligent behaviour, perception technologies
process raw sensor measurements to infer additional information and represent
sensor data in a useful way, for example, compacting sensor data, filtering irrelevant
or erroneous data, inferring relations between sets of data, or learning patterns from
data collected over time or from different settings.
Perception techniques can be categorised roughly into Sensing and Interpretation.
The goal of Sensing is to distill useful information from the stream of data produced
by a robot’s sensors. Interpretation aims to produce a higher level understanding of
the environment based on the stream of data produced by the sensing processes.
Perception technologies are fundamental to all robot domains.
Perception technologies are also influenced by fields beyond robotics, especially by
the consumer and gaming industry. Many current sensors used in robotics have
been developed for consumer and games industry applications, such as many of the
current 3D sensors.
This has also spawned a number of non-robotics applications that impact on human
robot interaction and robot sensor interpretation. Much of this work has found its way
into the robotics domain through open source adaptation.
Perception technologies also integrate multi-modal sensor data to build a coherent
picture of the environment, allowing the recognition of both objects and their
properties and arrangements. Combining non-visual sensing, such as chemical
signatures, heat, or tactile sensing helps to provide a rich perceptual dataset for
higher level robot control systems. Key Techniques and Methods
Key techniques within Sensing include
2-D visual sensing and processing (using, for example, colour or thermal
3-D sensing and processing (using, for example, laser range finders or stereo
imaging systems)
Tactile and force sensing and processing
Fusion of measurements from several sources
Imaging of glossy/reflective objects such as: Mirrors, reflectors, bathroom
accessories, etc.
Imaging of high curvature and reflective objects: Bearings, artificial joints,
aeroplane parts, engine and automotive parts
Imaging of micro parts and components.
Key techniques within Interpretation include:
Detection, recognition, localisation, tracking and modelling of objects and active
agents based on 2D or 3D data
Recognition, localisation and modelling (e.g. shape, stiffness, surface material)
of objects based on tactile and/or force data
Interpreting scenes semantically (e.g., determining which parts of a scene
correspond to vegetation or buildings, or determining which areas are drivable
and estimating the cost of driving there)
Learning patterns over time (e.g., learning traffic rules by observing how
vehicles drive, learning common motion patterns in order to avoid disturbing or
interrupting humans in the environment) Expected Step Changes
The perception technologies have a high level of maturity and have already achieved
wide spread application in mobile communication devices, digital cameras and in the
games market. Technologies such as gesture recognition, object recognition, and
location awareness all have wide application outside of robotics and many of the
recent developments are “in-flowing” to robotics.
To these wider applications Robotics adds specific requirements, most specifically in
tactile sensing, and the self localisation of multiple degree of freedom and articulating
structures. Robotics also utilises of multi-modal sensing and in some applications
requires the fusion of multiple sources of data some of which may be mobile.
Step changes are likely to arise in the following areas:
Improvements in real world object recognition and segmentation under variable
environmental conditions; sunlight, different weather conditions, seasonal
changes etc.
To develop the ability to recognise characteristics and properties of objects. Benchmarks and Metrics
The following are generic metrics for sensing:
Sample rate
Dynamic range
Metrics for interpretation are difficult to define ,although specific aspects of
interpretation can be characterised. Metrics for the recognition of objects are typically
based on statistical rates of success and failure.
There are numerous benchmarks for visual recognition performance although these
may not always be relevant to robotics, further robotics benchmarks can be included
in competitions, such as RoboCup, ELROB, or RoCKIn. A similar effort for perception
systems in isolation has been the Visual Object Classes Challenge and the ICRA
2012 Perception Challenge.
There are also some benchmarks for SLAM and localisation defined within the
Rawseeds Project ( Dependent Domains
Many advanced applications depend on perception to provide information about the
operating environment.
160 Impact on Domains and Products
Improved sensing and perception impacts on nearly all domains in
on interpretation of the environment. The ability to better extract
multiple sensor streams can impact on the placement and type
improvements in the speed of processing and recognition rates
affect the viability of applications.
robotics that rely
information from
of sensors, and
can significantly
There is a strong link between perception technologies and cognitive technologies.
Many cognitive technologies rely on high quality perception output and any
enhancement in the perception technologies is likely to have an impact on cognitive
4.5.2. Sensing Description
If a robot is to correctly interpret its environment it must be able to distill useful
information from the stream of data produced by its sensors. Transforming and
merging this data so that salient information is extracted is a critical step in the
process of interpretation. Key Techniques and Methods
Raw sensor data processing involves scaling, noise filtering and integrity and
consistency checking. In more advanced applications data is fused from multiple
sensors to provide a broader range of information over time. The time taken to carry
out sensor processing operations can limit the minimum loop time in dynamic control
Key techniques within Sensing include
2-D visual sensing and processing including segmentation and texture
Feature extraction and object identification and recognition.
3-D sensing and processing including the generation and filtering of depth
Processing and interpretation of tactile and force sensing
Fusion of sense data from several sources with different modalities.
Conversion of sense data to measurements.
Sensor calibration. Expected Step Changes
To increase the distribution of basic sensor processing closer to the sensors,
through increased integration of sensing and processing.
To develop techniques to enhance sensor fusion in both non-distributed and
distributed systems,
To develop methods for sharing sense knowledge between cooperating robots.
To standardise sensor interfaces to enable a component market place to
To perform all-weather sensing in the natural environment.
To improve the visual recognition of of glossy/reflective objects such as:
Mirrors, reflectors, plated object etc.
To improve the imaging of high curvature reflective objects that apparently
distort the visual image: Bearings, artificial joints, aeroplane parts, engine and
automotive parts
To develop techniques applicable to the imaging of micro parts and
162 Benchmarks and Metrics
Metrics for sensing vary with different types of sense data stream (Samples, frames,
sequences etc). In general metrics concentrate on the speed of processing, the
volumes of data being processed, the efficiency of data abstraction and the error
rates associated with the processes being applied. Dependent Domains
All domains depend on sensing and most applications require some processing of
raw sense data prior to it being used in decision making. Impact on Domains and Products
Improvements in sensing will impact on dependability since better sensing leads to
better decision making. Advances in fusing sense data from multiple sensors has the
potential to reduce the cost of sensor systems and this may have an impact in cost
sensitive applications.
In certain domains, notably Healthcare, Manufacturing and Agriculture the
development of novel sensing techniques may have a significant impact on
4.5.3. Interpretation Description
Robots need to interpret and perceive the objects and features in their environment
based on the information extracted from their sensors. Recognising objects, knowing
where to grasp an object, or where to place it. Noticing that something important has
happened against a background of other events. These skills are essential for a
robot to carry out tasks where it must operate in an everyday environment in
conjunction with people. A critical stage in achieving these abilities is the
interpretation of sense data streams. Key Techniques and Methods
Recognising a handful of objects in constrained circumstances is well known and
applied in commercial systems. The recognition of faces, hands and body pose is
well understood and widely used in commercial products. The recognition of
gestures and to a lesser extent facial expressions is beginning to be exploited both in
user interfaces but also in video analysis for example in the advertising industry. The
interpretation of 3D and visual sensory data is well known for the identification and
recognition of salient features in an environment. The extraction of shape and 3D
reconstruction from vision sensing is well understood particularly in relation to body
pose estimation, and is exploited in a limited number of commercial products.
Key techniques within Interpretation include:
Detection, recognition, localisation, tracking and modelling of objects and active
agents based on 2D or 3D data
Recognition, localisation and modelling (e.g. shape, stiffness, surface material)
of objects based on tactile and/or force data
Interpreting scenes (e.g. determining which parts of a scene correspond to
vegetation or buildings, or determining which areas are drivable and estimating
the cost of driving there)
Detecting patterns over time (e.g., learning traffic rules by observing how
vehicles drive, learning common motion patterns in order to avoid disturbing or
interrupting humans in the environment)
Interpreting the arrangement of objects in a scene.
The tracking of objects and features in the environment. Expected Step Changes
To improve recognition rates for robot task related object detection.
Development of standard perception engines for sense data types and sense
data fusion.
To be able to reliably recognise a wide range of known objects in everyday
To be able to reconstruct 3D object shapes from sensor data to allow fast and
efficient grasp planning and visual servoing.
To be able to recognise object properties form both tactile and visual data.
To be able to recognise novel objects or novel properties and configurations of
known objects
To be able to recognise dynamic relationships between objects and features.
To exploit the potential for facial expression recognition.
To be able to recognise and interpret complex gestures.
To provide reliable salient point and situation recognition over wide scale
To be able to track dynamic objects in real world environments.
To be able to track and estimate the pose of animates. Benchmarks and Metrics
The field of perception for robotics is quite diverse, and so it is difficult to enumerate
commonly used benchmarks. Competitions provide snapshots of perception ability
though the specification of specific tasks. Dependent Domains
Nearly all advanced applications involve sense data interpretation in order to
properly interpret the workspace. Many applications have limited environments
where specific techniques can be honed to deliver high levels of dependability. In
more open environments where systems may regularly encounter unknown objects
or where objects may be encountered in widely different conditions improvements in
interpretation will significantly affect the viability of applications. Impact on Domains and Visions
The interpretation of data is needed in all advanced applications of robotics where
sensors are used to detect real world environments.
The provision of standard perception engines will help to create greater performance
reliability and in turn will increase dependability levels. As techniques and methods
are developed to interpret both physical and visual data with greater reliability new
applications will become viable in almost all domains. This is a key area where
R&D&I investment can have a broad impact on domains and the realisation of
product visions.
4.6 Navigation Description
Purposeful motion in robots requires the ability to navigate. Navigation technologies
are required to successfully move through the operating environment. In advanced
applications robots need to perform in unstructured and dynamic environments
without continuous human guidance. In the future applications may require extended
periods of continuous operation in which case the provision of task appropriate levels
of autonomous navigation performance may become a critical barrier to market
Navigation can be defined as the combination of the three fundamental technologies:
Map-building and map interpretation
Path and motion planning and execution
Typically the two first competences are enablers of the third. Not being able to
localise in a known or unknown environment in a reliable way inhibits the generation
of successful path/motion plans. For this reason, it is a priority is to increase
localisation and mapping reliability and dependability at high TRL levels.
The autonomy of a robot is realised by its ability to navigate independently in the
environment and in cooperation with other elements involved in its function such as
humans or other robots. Key Techniques and Methods
Robots mainly rely on on-board sensors to localise and create reliable maps of the
environment so as to establish where to move autonomously. Robots can also make
use of intrinsic environmental information to establish position, or rely on beacon
based systems in order to increase robustness.
There are different self location mechanisms that can be used according to the scale
and nature of the operating environment. These can be categorised as:
Satellite and network-based,
Sensor-based (including vision-based),
Satellite and network-based technologies are mainly used for global navigation and
absolute positioning outdoors, namely global satellite positioning systems and
cellular networks, whereas sensor (inertial systems, odometry, range based
systems, visual sensing and 3D depth sensors) and beacon-based (WiFi, RFID, NFC
etc) approaches are mostly used for an indoor navigation and local/relative
positioning. In practical cases, in outdoor scenarios, a mixed approach is used in
order to fully exploit both global and local information coming from different sources.
For instance, light aerial vehicles perform self-localisation by fusing global
positioning data, inertial sensor and on-board camera data.
Mapping generates an absolute or relative reference frame for navigation. Mapbuilding concerns world representation and closely relies on the localisation process.
The most common technique involves Simultaneous Localisation and Mapping
(SLAM) where mapping and localisation occur moment to moment as a robot
explores its environment.
Global positioning
Most of global positioning technologies and techniques are well understood and
commercially mature. Satellite-based technologies use the following techniques:
Multi-lateration is the basic technique used for global positioning receivers. Here the
distances from the receiver to the satellites are calculated and used to compute the
receiver’s position. The number of satellites, its geometry and the obstacles
surrounding the receiver significantly affects the accuracy achieved. Accuracy of
about 10 to 20m can be easily expected.
The other most common techniques used for satellite-based positioning are assisted,
differential and augmented global positioning. These techniques use additional
information such as orbital data, correction from a reference station or source errors
(clock drifts, ionospheric delays...) to improve the algorithms used in the position
computation. These techniques easily achieve accuracies between 1 and 5 meters,
some centimetre accuracies and the more sophisticated methods such Real-Time
Kinematics millimetre accuracies.
Alternative techniques used by network-based technologies for localisation are
based on the estimation of signal parameters such as the received strength and time
or angle of arrival, and are mainly the following: cell identification, radio-mapping or
multi-lateration. Since in most of these cases the networks are not designed for
positioning, the nodes and the signal are not optimised for localisation and that
directly impacts on the accuracies that can be achieved.
Localisation and mapping
Sensor and visual-based technologies are the most commonly used for on-board
localisation. In most of cases they are combined using SLAM techniques. Most of
this techniques are well understood at research level and tested in real
The most common techniques used for these technologies are the following:
Dead reckoning (from motion sensors such as wheel encoders or internal sensors
such as accelerometers and gyros)
Data association (These methods aim at correlating data segments which represent
the same part of the environment. These allow local data registration and loop
closure detections)
Bayesian Filtering, also known as online SLAM e.g., Kalman Filter, Information Filter,
Particle Filter): on-line state estimation where the state of the system represents the
current position and the map. The state estimate is incrementally refined by
incorporating new sensor measurements as they become available.
Smoothing (Graph-SLAM), also known as off-line SLAM: the full robot trajectory and
the final map are estimated from the full history of sensor measurements typically
relying on the solution of a large error minimisation problem. This error represents
the consistency of the built model with the collected measurements and
For vision-based methods the most common techniques are:
Fixed arrays of external cameras monitoring the work space, typically from the outer
Landmark-Based Positioning (Artificial Landmarks/Markers)
Model-Based Approaches (Three-Dimensional Geometric Model-Based Positioning
and Digital Elevation Map-Based Positioning)
Feature-Based Visual Map Building (Natural Landmarks)
Appearance-based Localisation (These class of methods aim at identifying revisited
places on the basis of sensor similarity and are mainly used as visual SLAM relocalisers.
Each observed scene is represented by an appearance model
constituted by a set of visual attributes (e.g. Tree-based classifiers and Bag-ofWords-based methods)
Localisation by using geo-referenced information. For example aerial vehicle
localisation can be performed by registering the currently acquired ground image
with an available geo-referenced map.
Motion planning
Trajectory planning and simple motion planning round obstacles is well understood.
Complex motion planning for multiple degrees of freedom arms is well understood as
an optimisation problem. Expected Step Changes
Seamless indoor to outdoor navigation: Development of methods to achieve
seamless Indoor-Outdoor navigation and robust navigation through different
positioning systems totally transparent at the application level. Resulting
systems that fuse different localisation sources should be able to provide more
accurate and robust navigation.
High accuracy low cost systems: The development of new mapping techniques
through the combination of existing algorithms with cheaper mass market
sensors (such as 3D depth sensors, miniature cameras or similar) to generate
improved navigation performance at a lower cost.
Deployment of map hierarchies: hierarchical mapping, conditional independent
mapping and tectonic mapping type techniques combined with distributed
computing are expected to reduce the computational cost and increase the
scalability of autonomous navigation ability.
Dynamic map building: Development of map-building methods for dynamic and
changing environments (e.g. crowded environments, human body mapping,
mapping of flexible structures).
Cloud based localisation: The integration of third party external cloud based
services in the robots system design for localisation and mapping, both indoor
and outdoor. This has the potential to reduce computational load and speed up
map generation and localisation.
Localisation in dynamic environments: Prediction methods for estimating
localisation in crowded and dynamic environments. Including localisation in
moving media such as under the sea or on unstable ground.
Efficient motion planning for multiple degree of freedom structures: Including
energy minimised planning and motion planning in flowing water.
Motion planning in dynamic environments: Including high density environments
and against changing environmental conditions.
Motion planning for teams: Distributed cooperative motion planing for teams in
air, water and on land
Collaborative navigation: Development of collaborative map-building and
interpretation strategies, collaborative positioning. Heterogeneous robots
working as a fleet could help each member of the group by offering positioning
or mapping information.
Improved localisation systems: Development of advanced signal processing
algorithms to achieve high accuracy positioning (RTK-like) with cheaper
receivers such as single-frequency receivers.
Novel localisation techniques: Ability to create localisations from non-visual
clues, such as ambient sounds, tactile sensing, air flow etc. Including the
integration of maps generated using different sense modalities.
Virtualisation of base stations for absolute localisation in outdoors
Long term map correction: The relevance of most maps decays over time and
need repeat references, beacons or place detection strategies to enable map
maintenance over long time spans.
Long term localisation: For some applications it will be important to accurately
return to known locations over long periods of time (several years). This is a
challenge because the environment around the location can change
Semantic mapping: Including semantic information in the world models is
important in order to move towards navigation strategies where the robot can
interpret, in human terms, where it is and where it has to move.
Social navigation: A social navigation is needed so that robots can easily be
accepted in environments and collaborate with people. Robots will need to
follow commonly accepted human behaviours when navigating such as
following the people flow when moving in the same direction. This type of
navigation will be strongly linked with cognitive ability.
Human like motion: The ability to interact and work in close collaboration with
users will require motion planning that is compatible with human expectations.
Motion planning with flexible links and tools: Motion planning while holding a
flexible tool or object, or where links react flexibly to external forces in the
environment. Benchmarks and Metrics
The primary metrics for localisation are:
Accuracy: Degree of conformance of the estimation or measurement against
its truth value at a given time.
Integrity: Ability to provide timely warning to the system when the solution
should not be used for navigation.
Availability (in terms of update frequency): Ability to remain usable within the
intended area, defined as the portion of time during which the estimation is
reliable for navigation.
Continuity: Ability to perform without interruption in the intended application
The primary metrics for mapping are:
Map entropy (in case we are dealing with a probabilistic map)
Map convergence
Correspondence to ground-truth. The aforementioned metrics depend on the
availability of a ground truth that is often provided by other measurement
system, that in turn has its own accuracy.
Reliability: confidence interval to cope with uncertainty measurements. For
instance, if the system does not find a landmark for a long time, it may
consider its position is not reliable any more.
Repeatability: ability to provide the same result from a given same observation
or realisation.
Although many robotic datasets are available on the web which constitute a reliable
mean for benchmarking algorithms at research level, the definition of standard
metrics and tests, such as in automotive or aeronautics sectors, is needed. This will
allow system integrators to compare among different systems and platforms. Dependent Domains
In some applications the failure of navigation capability can direct or indirectly put
humans at risk. In some other applications or domains a failure in navigation can
cause material losses. The following describe some applications where navigation is
a critical ability:
Transportation of goods: Applications in logistics or supply chains such as in
harbours or interchange of goods or in decommissioning and construction services
Transportation of people Autonomous navigation becomes a safety critical ability
when transporting people; For example in driverless cars and autonomous aerial
vehicles. In these cases systems will need strong certification.
Aerial robotics (excluding transportation of people): Aerial robots will be able to fly in
non-segregated airspace in the medium to long term. As such, they will have to
operate in harmony with the other occupants of the airspace and inhabited overflown
Agricultural robots, both indoor and outdoor: Farm environments are dynamic and
alter with ground conditions, the weather and the seasons, maintaining localisation,
long term maps and motion paths is critical to many farm applications.
Search and rescue: Unstructured non-anthropic outdoor environments, possibly
severe weather conditions, speed constraints, interaction with dynamic environments
and other human and robotic teams all contribute to a challenging application
Manipulation and assembly in aerial and marine robotics: Precise pose are required
for manipulation and assembly in aerial and marine robotic applications against
environmental flows and turbulence. Impact on Domains and Products
Many of the proposed application for robotics technology require high quality
navigation solutions in both indoor and outdoor environments. Particular challenges
exist for aerial and marine vehicles and in environments where satellite based
positioning systems cannot be used.
High quality navigation, at task appropriate scales, will significantly affect the quality
of may tasks and impact on long term deployability. Low cost modules and
components for indoor navigation which provide high levels of dependability and
accuracy will enable faster system integration and development.
4.7 Cognition Description
Cognitive robots are able to express adaptive anticipatory behaviour in real time on
the basis of the contingent situation, past experience, and inferred future conditions.
Cognition is the system-wide process that provides an agent with the ability to
understand, given only partial knowledge, how things might possibly be, not just now
but at some point in the future, and to use this understanding to influence action.
Predicting the future requires the robot to remember past, so learning is critical to all
cognitive systems. Cognition breaks free of the present in a way that allows the
system to act proactively, reliably, to adapt, and to improve.
Many of the other technologies exploit cognitive processes and techniques, and vice
versa, including sensing, planning, navigation and human-robot interaction (HRI). All
of these will benefit from further improvements in cognition and in turn cognition
depends on them. Key Techniques and Methods
Knowledge representation and reasoning (KR&R)
KR&R is a wide and mature field of AI, which is pivotal to high impact application
domains like data mining, search engines and recommendation systems. The main
assumption behind KR&R is that knowledge should be represented in the system in
an explicit, machine-readable form, to provide the ability to reason from this
knowledge and about this knowledge, and to prove formal properties about the
knowledge and about the reasoning mechanisms. This concerns both descriptive —
declarative— knowledge (“know-what”) and prescriptive — procedural— knowledge
(“know-how”), particularly important in robotics. More generally, knowledge can be of
many different types, including ontological, causal, deontic, procedural, temporal,
spatial, and still others. Accordingly, different KR&R formalisms have been
developed, each one being typically geared toward one type of knowledge.
Formalisms have also been developed to deal with the uncertainty that may affect
knowledge, including those based on probability theory, Dempster-Shafer theory,
and fuzzy logics. Reasoning in the above formalisms is often done by logic-based
methods, and a great attention is put in the formal and computational properties of
these methods, like soundness, completeness, decidability and complexity. A
plethora of tools for KR&R are available, both commercially and in the open source
community. Some languages produced in this community have become standard,
like the OWL family.
KR&R in Robotics.
The issues addressed in the field of KR&R have recently started to be also
addressed by the robotics community, as a means to endow robots with the ability to
represent and use higher level knowledge. Notable examples are the representation
of higher level concept in semantic maps, and the use of ontologies to enable robots
to elicit information from the web. Ensuring a high level of communication and
synergy between robotics and the field of KR&R is essential in order to leverage the
large body of knowledge, experience and tools acquired in the former, and in order to
make this knowledge and tools suitable to the needs of robotics systems.
Knowledge-based planning
Knowledge-based planning is an active and one of the oldest areas of research in
the field of AI. The type of planning addressed in AI is task planning in its wider
sense, which includes scheduling, dealing with resources, and observation planning.
Many results are also available with respect to planning under uncertainty, planning
for distributed systems, and distributed planning. Geometric or motion planning are
usually not addressed in this area. Planning techniques in AI have a high degree of
maturity, and the field has developed its own de-facto standard language (the PDDL
family), benchmarks and competitions. Commercial and research planners are
Knowledge-based planning in robotics.
Planning in AI was originally developed for use on robots, namely on the SRI robot
Shakey from the late 60’s. Since that time, there has been an extensive
development, and AI planning is now a mature field with applications to, e.g.,
logistics, production, and space exploration. Advanced AI planning capabilities in
space have been notably demonstrated by NASA in the ”remote agent experiment”
on-board DeepSpace-1 back in 1999. However, only the most basic AI techniques
for planning have been typically used until now in deployed robotic systems, due to
the fact that tasks and domains were simple and predictable. In order to move to
changing, more complex and less predictable application domains, especially those
where robots and human co-exist, the more sophisticated AI planning techniques
available will need to be incorporated into robotic systems.
Machine learning
Machine learning techniques most often used in robotics can be roughly categorised
to 1) supervised classification or regression; 2) unsupervised clustering; and 3)
reinforcement or policy learning. Supervised recognition is widely used in robotics for
skills such as speech recognition or visual recognition, in most cases with off-line
training. Similarly, supervised regression methods are used for learning the robot’s
dynamics or other physical properties based on sensory data. On-line supervised
learning using programming by demonstration (PbD) and learning from (direct and
indirect) demonstration (LfD) is set to become increasingly important. Unsupervised
clustering is mainly used for pre-processing of sensor data or more recently as part
of semi-supervised methods. Policy learning is mainly performed by reinforcement
learning approaches where advances have been made recently especially in the
learning of individual motion skills (e.g. the PI2 method). The machine learning
community is actively developing advanced methods but many of the general
advances require being customized to robotics because they either rely on vast
amounts of labeled training data, on off-line processing or often on unrealistic
computational budgets. The long-term autonomy of robotic systems will be
dependent on machine learning because the agent’s environment will be dynamic
and environment models then need to be updated during the life-time of the agent.
Machine learning in robotics.
Machine learning methods are increasingly being used in robotic applications owing
to the recent progress in the formalization of many of them and consequently the
mathematical guarantees that they now provide. The advances of the past couple of
decades enabled the mathematical characterization of the problem of generalization
in high-dimensional data spaces: the use of the mathematics of reproducible kernel
Hilbert spaces and associated kernel methods enabled the design of efficient
classifiers and similarly of regressors since then successfully applied to the problem
of image (and more in general sensory data) recognition. Regression methods
(conceptually identical to classifiers) were employed for system identification – to
acquire a model of the plant – to later use in model-based control methods. These
led to the widespread use (in research but also in industrial application) of computed
torque and the related force control methods. Methods for temporal data
classification and modeling, as e.g. speech, have been also applied consistently
driving the advances in speech recognition. The formalism of graphical models was
employed together with probabilistic algorithms for both learning and later inferring
solutions to specific problems of classification as well as regression. In many cases
(e.g. Gaussian Processes), the same algorithm had different possible interpretations
depending on the mathematical tools being employed. For problems when labeled
data are not available, reinforcement learning techniques have been improved
leading to learning from demonstration approaches for movement acquisition in
robotics. The research of learning in robotics has been fruitful in characterizing and
fine-tuning machine learning methods to specific robotic problems and tackling the
issues of on-line learning as well as the computational efficiency of methods to be
e.g. used in tight robot control loops. This last topic has long range connections to
the problem of life-long learning in AI and robotics.
Natural interaction
Humans communicate via a large number of rich means, many if not all of which are
now exploited by contemporary AI systems. This communication includes but is not
limited to natural languages such as French or Mandarin, which have both spoken
and written forms. But natural human interaction includes deliberate gestures such
as pointing, and also incidental “backchannel” feedback which can take the form of
grunts, shifting posture, or facial expression. Such feedback may not be explicitly
noticed by either the sender or the receiver, yet serves to allow both participants to
monitor each other’s interest and level of understanding. In addition, humans have
evolved social emotions that help shape and motivate the extent and nature of their
interactions, including turn taking and conflict resolution.
Natural interaction in robotics.
Natural interaction is fairly advanced in many cognate technologies such as virtual
reality, wearable computing, and smart phones. Key techniques include natural
language processing, speech recognition and generation, gesture recognition and
gestural communication, emotion recognition and expression, and indirect
communication by observation. Natural human-robot interaction is clearly needed for
robots aimed at the service and domestic sectors, and it is highly desirable in the
industrial sector as a way to instruct the robot and cooperate with the robot. As much
as possible interaction should be seamless, and certainly human users will expect
interaction to improve rapidly with familiarity of the task. Expected Step Changes
To be sure of success in step development, candidate techniques and methods must
satisfy the following three requirements: (1) they have a good degree of maturity, (2)
they require focused investment in research to make them applicable in a robotic
system, and (3) their application to robotics would enable critical new capabilities
which are essential for some important domains.
These include the following:
Knowledge Representation and Reasoning
Increased use of predictive models and hybrid knowledge representations:
planning has stressed and progressed extensively in search issues with shallow
(abstract) descriptive models. Complex challenges remain for deliberate action
(including planning, monitoring, etc.) with possibly much less search but deeper
predictive models, combining declarative and procedural knowledge.
Hybrid KR&R: one key missing ingredient in current KR&R is the ability to
integrate multiple types of knowledge (declarative – semantic and episodic – and
procedural), and to reason across multiple types of knowledge. This is important
in robotic systems, in which spatial, causal, temporal, resource and still other
types of knowledge must be considered.
Planning and reasoning under uncertainty: existing KR&R and planning
techniques need to be extended to allow a realistic and tractable treatment of the
uncertainty and incomplete knowledge inherent to the robotic domain.
Knowledge-based Planning
Closed-loop planning: task planning and plan execution are usually loosely
coupled; tight coupling of planning, plan execution, and learning will enable plans
that are updated on-line based on observations but also planning how to learn
about the attributes and dynamics of the environment, which will be necessary for
long-term autonomy.
Hybrid planning: task planning and motion planning are usually done
independently from each other: a more integrated approach is needed in robotic
systems, in which task-level actions and geometric motions may have mutual
dependencies. Planning for robotic systems should also take into account other
aspects related to physical execution, like time, observability and resources.
Formal specification and synthesis of control systems: the ability to prove
formal properties on the behaviour of a cognitive robotic system, especially related
to safety, is pivotal to their introduction in the market and society. Formal methods
are needed to specify and synthesize robot control systems starting from rich
models of the robot, of the environment and of the task. These methods should
enable the formal verification of the system, the run-time monitoring of its
performance, and the diagnosis in case of failures. Tools like temporal logics and
description logics have reached a good degree of maturity, and are good
candidate to create such methods.
Internal simulation: symbolic and non-symbolic associative knowledge
representations for predicting the many possible outcomes of a robot’s actions
and their consequences for subsequent interaction are needed to deal with
uncertainty. This will be realized in the form of internal simulation – effectively
cognition without engaging the motor commands – through multiple causal chains
of perception-action couplings. Action selection will be effected by internal
attention winner-take-all processes, adjudicated according to a context-sensitive
(possibly application dependent) value system.
Meta-cognition: in a similar vein, there is a need for cognitive robots to be able to
reason about their own performance: their current state, their state of
preparedness for upcoming actions, and the behaviours that satisfy ethical
standards and provide for safe operation. Such autonomic processing is
necessary to support greater autonomy and promote user acceptance.
Machine learning
Symbol grounding and perceptual anchoring: connecting the discrete symbols
used in a knowledge-based methods to the continuous signals linked to the
sensors and actuators is a fundamental problem that is often solved in a ad-hoc
way on a case-by-case basis; a principled, well-founded approach is still lacking.
In particular, a principled approach that combines top-down and bottom-up
representations would enable knowledge based sensor interpretation, reasoning
and planning with limited knowledge.
Automatic categorization and generalization across categories: by
generalizing learned knowledge and exploring innovative strategies for using that
knowledge, cognitive systems can uncover innovative ways of addressing
problems and dealing with unforeseen situations. This will involve the ability to
improve existing world models (pre-programmed or learned), including plans,
solution strategies, action repertoires, and affordance-based interaction.
Learning in unconventional data domains: learning for vector valued functions
is a requirement in many domains and it is often treated as learning a collection of
scalar functions, effectively missing the opportunity of learning the deeper
structure of certain vector valued problems (e.g. the robot body dynamics).
Similarly, there are situations where the data set is incomplete and techniques for
semi-supervised learning have to be employed. For a robot it is often easier to
acquire and store lots of data rather than waiting for a teacher to provide the
labels (which are typically scarcer). These basic improvements are needed to
move from the large databases required today to the robotic applications of ML
techniques. In all cases, care must be taken, to guarantee bounded memory and
time so that machine learning is applicable to reasonably sized robotic
computational infrastructures.
Deep Learning methods: automatic extraction of semantics from experiential
learning is required to allow robots to adapt to new situations, either on-the-fly or
when being reconfigured for new applications. Deep Learning has recently surged
to the state-of-the-art by demonstrating high quality performance both in training
and testing time and accuracy. Restricted Boltzmann machines for example have
been shown useful in the classical domains of image classification and more
recently in continuous speech recognition. These methods are less understood
mathematically than kernel methods. Curiously, multi-layer (deep) representations
and methods are more brain-like and thus they are proving important both as a
computational and as a modelling tool for certain brain architectures. Dictionary
and feature learning are two other methods that fall to a certain extent in the
category of deep learning as they act as data pre-processing before shallow
learning methods are applied. These include sparsification principles which have
shown exceptional performance increments.
Learning from Demonstration (LfD): building on established results in
programming by demonstration, new techniques are required to allow robots to
resolve ambiguities in non-expert supervised learning and exploit accelerated
forms of reinforcement learning. Furthermore, there is a need for learned skills to
be fine-tuned by implicit or explicit coaching by the end-user.
Integration of learning in knowledge-based technologies: cognitive robots
need to learn to plan and plan to learn, especially when using knowledge-based
techniques such as semantic maps and production systems.
Natural Interaction
Human affective and social behaviour detection. New algorithms and design
principles for machines that can detect, interpret and recall complex human
behaviours in everyday contexts such as the home, the office or in public places.
The investigation of “natural”. Assess the essence and various means of
natural interaction with robots, developed from a research-based understanding of
human behaviours and needs, dignity and well-being.
Robot soft communication skills. New mechanisms, control and scheduling
techniques to realise socially aware robot responses that are physical as well as
affective and social.
Autonomous navigation and localisation in crowded environments. Precise
localisation and mapping is needed for crowded scenarios where large numbers
of features can correspond with mobile objects. Advanced models of peoples’ and
groups’ motion patterns are needed as well as applying social constraints to
navigation planning. Additionally, a guide robot may adapt its path planning to its
user’s road preferences.
Physical interaction. This includes handover, joint construction, tele-operated
and collaborative tasks.
Adaptivity and learning triggered by social interaction. Learning in this sense
aims to achieve better performance in a social interaction, in contrast to other
learning applied to robotics activities devoid of HRI. For instance, learning may
aim to promote human-robot collaborative achievement of a task or to improve
human-robot communication (such as the robot changing its behaviour to help the
human better understand the robot’s intent).
User personalisation. New machine learning approaches to learn and adapt to
user preferences in a natural way, over multiple interactions.
The ethical and social implications of social robots. Autonomy and decision
power of the robots oblige the researchers to think also about ethics and the
respect of intimacy of
subjects. Creating a robot that is indistinguishable in
appearance from a human is subject to controversy for ethical point of view. Coevolution between humans and robots has also social implications. We need a
new interdisciplinary mix of computer, social/psychological sciences and
engineering to understand the substantial impact of Robots in the society.
Finally, techniques which have a strong potential but are currently at the basic
research stage, so their exploitation in robotic systems is on the long term scale.
Neuromorphic engineering & learning: neuromorphic engineering comprises a
set of techniques that imitate brain architectures at the hardware level. Recently
progress in hybrid neuromorphic and digital architectures has enabled the
implementation of learning methods based on neural principles (as for example
LTP and LTD or spike-dependent synaptic plasticity techniques). Neuromorphic
engineering leads naturally to efficient encoding of either sensory or motor signals
in terms of spikes and to a completely new class of data processing algorithms
which rely on “sparse” event-based representations. Besides, hardware
neuromorphic neural network are now under investigation. One of the major
advantages of the neuromorphic hardware is the extremely low power
consumption owing to the use of transistors in the sub-threshold regime. Albeit all
these techniques are certainly experimental, they are worth exploring as a longer
term endeavour for autonomous robots to operate on batteries or other portable
energy sources.
Dynamical systems theory: dynamical systems theory has been successfully
used to model the behaviour and learning of animals and humans in various
sensorimotor tasks including rhythmic behaviours, locomotion, etc. In short,
concepts of trajectory generation become attractors in the language of dynamical
systems, learning becomes the modification of certain potential functions, etc. The
same mathematics has been used in robot control and it found its application also
in reinforcement learning (PI2). Motion primitives based on dynamical systems
have been very popular recently to encode the robot’s tasks in a small and
manageable number of parameters. This area of cognitive systems theory needs
further development in order to transition from a mostly analysis tool into a proper
engineering method, i.e. to develop methods that allow the design of generic
dynamical systems given a set of behavioural goals of the robot. Benchmarks and Metrics
Evaluation and benchmarking of cognitive robotic systems is a challenging area
because cognition is a system-wide process that effectively integrates the
sensorimotor capabilities in a manner that achieves the meta-functional
requirements necessary for deploying in uncertain and poorly-specified application
scenarios: dependability, reliability, usability & reusability, versatility, robustness,
fault-tolerance, safety, security, and maintainability, among others.
We distinguish between evaluation of the individual components, and evaluation of
the whole cognitive system. For the former, there are many standard methods
already available. For the latter, we adopt the same approach that is used in
software engineering for assessing the quality of software products. Since quality
cannot be measured directly, we need to identify a set of indicators that correlate will
with the degree to which the meta-functional (often referred to as non-functional)
attributes are exhibited by the robot. Performance indicators will target environmental
variability, task diversity, and range of interactions with human and other machines.
Regarding benchmarks, we make the same distinction between benchmarks aimed
at assessing the performance of a specific cognitive component, and benchmarks
aimed at assessing the performance of the full robotic system. For the former,
several benchmarks exist for specific types of components, e.g., artificial vision
(Pascal), planning (the International Planning Competition), knowledge contents (the
QuerySet from the University of Bremen), etc. For the latter, the best established
benchmarks that stress the cognitive abilities are RoboCup and the RockIn EU C.A.
Current work (e.g., in the RoCKIn EU FP7 CA, in the RAWSEEDS EU FP7 CSA) has
set several directions that should be extended in HORIZON2020 upcoming calls,
especially system benchmarks in unconstrained or partially-constrained test
scenarios. It is important that benchmarks assess both functional and metafunctional performance.
both individual components/subsystems/abilities/subtasks and the integrated
system/task should be separately evaluated and benchmarked - there are different
ways in which abilities can be integrated, but abilities themselves should also be
applicable to a wide variety of domains and perform well across them;
software tools should be developed to implement the measurement of performance
based on (also to be developed and established) metrics of common use for different
abilities (e.g., error with respect to a reference path during robot navigation, but also
comfort index for a human being followed by a robot while traversing a given path;
number of correctly classified gestures or recognized words, but also gesture
naturalness index for the humans) and be used in benchmarking test beds specified
to allow reproducibility and repeatability of tests. Dependent Domains
Cognition is critical in all domains that cannot be modelled completely and
accurately, e.g., because they exhibit variability over time, diversity of tasks and
diversity of interactions. These domains include marine robotics, flexible
manufacturing, health robotics, elderly assistive robotics, most service robotics, and
in general all domains in which humans and robot share the same environment and
possibly cooperate. Impact on Domains and Products
Already today, commercial service robots meant to operate in previously unknown
environments or in environments where humans are present already leverage some
simple cognitive technology. These include domestic cleaning robots (e.g., Roomba
floor cleaners), industrial service robots (e.g., Atlas Copco’s automatic mining
vehicles), and marine robots (e.g., Seetrack by
This document has been prepared for euRobotics aisbl under the RockEU
Coordination Action by iTechnic Ltd and RURobots Ltd
The content of this document has been assembled from submissions made to
euRobotics via Workshops and Topic Groups.
A second revision of this document will be prepared for Call 2 (ICT24).
Was this manual useful for you? yes no
Thank you for your participation!

* Your assessment is very important for improving the work of artificial intelligence, which forms the content of this project

Download PDF