D-3-2-CORBYS-Physical-Physiological-sensing-devices

D-3-2-CORBYS-Physical-Physiological-sensing-devices
CORBYS
Cognitive Control Framework for Robotic Systems
(FP7 – 270219)
Deliverable D3.2
Physical/Physiological sensing devices
Contractual delivery date: Month 20
Actual submission date: 30st September 2012
Start date of project: 01.02.2011
Duration: 48 months
Lead beneficiary: BBT
Responsible person: Marco Creatura
Revision: 1.0
Project co-funded by the European Commission within the seventh Framework Program
Dissemination Level
PU
Public
PP
Restricted to other program participants (including the Commission Services)
RE
Restricted to a group specified by the consortium (including the Commission Services)
CO
Confidential, only for members of the consortium (including the Commission Services)
X
D3.2 Physical/Physiological sensing devices
Rev. 1.0
Document History
Author(s)
Anders Liverud
Marco Creatura
Anders Liverud
Anders Liverud,
Steffen Dalgard
Anders Liverud
Marco Creatura
Anders Liverud
Marco Creatura
Anders Liverud
Marco Creatura
Revision
0.1
0.2
0.3
0.4
Date
02-07-2012
23-07-2012
10-09-2012
14-09-2012
Contributions
Draft document structure
Revision of the BBT contribution
Added IMU Sensor module documentation
Reset of "track changes" after much editing
0.6
0.8
0.9
0.10
0.11
0.12
21-09-2012
24-09-2012
26-09-2012
26-09-2012
27-09-2012
28-09-2012
Marco Creatura
1.0
30-09-2012
Draft document
BCI section
Input from review on HSS
BCI section – internal review changes
HSS sections updated
Executive Summary, Introduction and BCI section
(including Safety , Requirements, and Conclusion) updated
Release version
II
D3.2 Physical/Physiological sensing devices
Rev. 1.0
CORBYS Definition of Terms
Term
Definition
CORBYS Demonstrators
The 1st CORBYS Demonstrator
(Demonstrator I)
The 2nd CORBYS Demonstrator
(Demonstrator II)
Mobile Robot-assisted Gait Rehabilitation System
Reconnaissance Robot for Investigation of Hazardous Environments –
(RecoRob)
CORBYS Roles
User
CORBYS End-user
Mobile Robotic
Gait
Rehabilitation
System Roles
Patient
Therapist
Engineer
Reconnaissance Operator
robot for
Hazardous Area
Investigation of Examination Officer
Hazardous
Engineer
Environments
Roles
CORBYS Domain Knowledge
Sensor Fusion
Situation Assessment
Cognitive Control
Human-Robot Interaction
Neural Plasticity
Cognitive Processes
CORBYS Technology Components
SAWBB
SOIAA
User Interface
Brain Computer Interface (BCI)
Human
Sensory
System (HSS)
HSS
Chest Unit (CU)
IMU Sensor Unit
Any user interacting with the CORBYS systems, for example,
in case of gait rehabilitation system, users with the following roles: a
patient, therapist or an engineer;
In the case of reconnaissance robot, users with the following role: (tele)
operator or a hazardous area examination officer.
The companies/entities that use/exploit (aspects of) CORBYS
technology in their commercial products or services.
The person receiving gait rehabilitation therapy aided by the CORBYS
system
The medical professional configuring and assessing rehabilitation
therapy aided by the CORBYS system.
A professional dealing with the CORBYS system based on a need to do
technical maintenance, repairs or system configurations.
The person steering the robot by remote control
The person that robot follows in a team work on investigation of
hazardous areas
A professional dealing with the CORBYS system based on a need to do
technical maintenance, repairs or system configurations.
Method used to combine multiple independent sensors to extract and
refine information not available through single sensors alone.
Estimation and prediction of relation among objects in the context of
their environment.
Capability to process variety of stimuli in parallel, to “filter” those that
are the most important for a given task to be executed, to create an
adequate response in time and to learn new motor actions with
minimum assistance (Kawamura et al., 2008).
Ability of a robotic system to mutually communicate with humans.
Ability of neural circuits, both in the brain and the spinal cord, to
reorganise or change function.
Processes responsible for knowledge and awareness, they include the
processing of experience, perception and memory.
Situation Awareness Blackboard
Self-Organising Informational Anticipatory Architecture
User interface designed to meet the needs of the various users in
exchanging information between the robot and human user
The sensor system measuring the brain waves using EEG and detecting
patterns identifying movement actions
The sensors measuring aspects of the human physiology and movement
patterns.
Sensor unit located at chest of patient.
Sensor Unit located at back of patient.
III
D3.2 Physical/Physiological sensing devices
Rev. 1.0
Chest Belt
Belt around patient chest with Chest Unit at front and IMU Sensor Unit
at back.
HSS Controller
Computer and software system that receives data from HSS sensors and
forward these to the General Purpose Network.
Low Level Control
Localised control of actuators, usually torque, current or position
control. Sensory data is passed to the real-time control, actuation
commands are calculated and sent to the actuators.
Smart Actuators
Highly integrated mechatronic units incorporating a motor and the
complete motion control electronics in one single unit.
Generic CORBYS Robot Control Components
Cognitive System
Incorporates situation awareness and intention detection to enable
optimal man-machine interaction towards achievement of set goals in
the specific usage context.
Executive Layer
The executive layer is responsible for translating the high-level plans
(cognitive inputs) into low-level actions, invoking actions at the
appropriate times, monitoring the action execution, and handling
exceptions. The executive layer can also allocate and monitor resource
usage.
Communication Server
Manage subscriptions of sensor data between different control modules.
The sensor data to the cognitive modules are not flowing through the
Communication Server, but are forwarded directly.
Task manager
The task manager manages operation modes to be executed by the
system. Performs specific tasks when the operation mode is changed.
FPGA Reflexive Module
Field Programmable Gate Array (FPGA) based hardware subsystem of
Situation Awareness architecture (SAWBB) for acceleration of robot
reflexive behaviour.
Safety Module
Verification that actuator output is in line with the commanded output
and that it satisfies safety-related position, velocity, current and/or
torque constraints.
Real-Time Data Server
Real-time data server is a software module responsible for
communicating sensor data from real-time (RT) bus to other software
modules.. This excludes communication of RT modules with sensors
and actuators which communicate with sensors and actuators directly,
in order to preserve RT control behaviour.
Real-Time Network (RTN)
Sensor network for real-time, safety critical data transmission
General Purpose Network (GPN)
Network for robot control and interface to the cognitive modules
Demonstrator Specific Technology Components
Mobile Robotic Gait
Pelvis Link
Mechanical interface between the mobile platform and the powered
Rehabilitation
orthosis equipped with an appropriate actuation and sensing system.
System
Powered
Exoskeleton system to help the patient in moving his/her legs and
Orthosis
receiving an appropriate rehabilitation therapy.
Mobile
The platform for the entire system, including Pelvis Link, Powered
Orthosis, necessary computational, storage, and power supply modules,
Platform
as well as motored wheels for movement
Reconnaissance robot Vision System
Cameras of the 2nd demonstrator used for environment perception
for Investigation of
including human tracking
Hazardous
Robot Arm
7DOF lightweight robot arm mounted on the Mobile Platform used for
Environments
the object manipulation (for contaminated area sample drawing)
Mobile
Mobile platform of the 2nd demonstrator which consists of a variable
drive system that is equipped with chains. It is used for mounting of the
Platform
robot arm and sensors for environment perception as well as sensors for
platform navigation and robot arm control. Containers for samples are
also placed on mobile platform
IV
D3.2 Physical/Physiological sensing devices
Rev. 1.0
Table of Contents
CORBYS DEFINITION OF TERMS .................................................................................................................... III
EXECUTIVE SUMMARY ..................................................................................................................................... 1
1
INTRODUCTION .......................................................................................................................................... 4
1.1
1.2
1.3
2
HUMAN SENSORY SYSTEM REALIZATION ........................................................................................... 6
2.1
2.2
2.3
2.4
2.5
2.6
3
HSS ARCHITECTURE ........................................................................................................................................ 6
CHEST UNIT ................................................................................................................................................... 11
IMU SENSOR UNIT ........................................................................................................................................ 17
EMG SENSORS ............................................................................................................................................... 22
HUMAN SENSORY SYSTEM CONTROLLER....................................................................................................... 23
OFFLINE CHARGING STATION FOR WIRELESS SENSORS ................................................................................... 39
BRAIN COMPUTER INTERFACE (BCI) .................................................................................................. 41
3.1.1
3.2
3.3
4
DOCUMENT SCOPE ........................................................................................................................................... 4
DOCUMENT STRUCTURE .................................................................................................................................. 4
ASSOCIATED DOCUMENTS ............................................................................................................................... 5
EVALUATION OF EEG ACQUISITION SYSTEMS THAT COULD REDUCE THE NOISE LEVEL ................................. 43
ANALYSIS AND REMOVAL OF MOVEMENTS’ ARTIFACTS DURING LOCOMOTION.............................................. 52
DETECTION OF ATTENTION DURING ASSISTED PASSIVE LEG MOTION.............................................................. 65
SAFETY ........................................................................................................................................................ 72
4.1
4.2
4.3
4.4
4.5
4.6
4.7
4.8
4.9
CHEST UNIT ................................................................................................................................................... 72
IMU UNIT ...................................................................................................................................................... 72
EMG UNITS ................................................................................................................................................... 72
HSS CONTROLLER COMPUTER ....................................................................................................................... 72
OFFLINE CHARGER ......................................................................................................................................... 72
SAFETY ANALYSIS: ........................................................................................................................................ 73
EEG UNIT ...................................................................................................................................................... 76
BCI SOFTWARE ............................................................................................................................................. 76
BCI SAFETY ANALYSIS: ................................................................................................................................. 76
5
REQUIREMENTS FROM D2.1................................................................................................................... 79
6
CONCLUSIONS AND FUTURE WORK .................................................................................................... 91
7
REFERENCES ............................................................................................................................................. 93
V
D3.2 Physical/Physiological sensing devices
Rev. 1.0
Executive Summary
This deliverable document (D3.2) reports on work related to the development of Human Sensory System
(HSS) and Brain Computer Interface (BCI) as performed in Work Package 3 "Sensing systems for assessing
dynamic system environments including humans" of the CORBYS project.
Based on deliverable documents, D2.1 Requirements and Specification, and D2.2 Detailed Specifications of
the System, this document describes the realisation of the Human Sensory System and the analysis of Brain
Computer Interface processes related to development and integration of the BCI module in the CORBYS 1st
demonstrator.
To meet challenges in physiological monitoring in robotics systems a human sensory system has been
realized. Compared to available commercial sensor systems the CORBYS human sensory system has features
that make it suitable for robotics environments. It is compact, easy to attach and remove and does not disturb
or cause any discomfort for the user. The sensors provide relevant physiological measures and can coexist
with robotic systems like gait rehabilitation robots. Data is provided on a standardized robotics software
interface.
The sensors have been realized through development of a chest belt with sensors situated on the chest and
back of the patient. Physiological parameters like heart rate with ECG, humidity, skin temperature are
measured together with velocity and orientation. These parameters will be used by the cognitive modules of
the CORBYS Demonstrator I, robot-assisted gait rehabilitation system, for assessing physical effort,
identifying psyco-physiological states and for identifying intentions of the patient. Similarly the Human
Sensory System may be used for providing physiological data for CORBYS Demonstrator II, the
Reconnaissance Robot. The infrastructure for transmitting sensor data to the cognitive framework has been
developed. Sensors are transmitting sensor data wirelessly with a predictable low latency to a computer. The
computer is running controller software that is synchronizing and time-stamping sensor data before data is
provided to the cognitive modules, therapist and engineering user interfaces through the general purpose
network. The sensor modules are based on state of the art components that are highly integrated and optimized
for long term wireless physiological monitoring. Effort has been put into compact integrated design and low
power consumption enabling long term physiological monitoring.
1
D3.2 Physical/Physiological sensing devices
Rev. 1.0
Figure 1: Human sensors interaction with other modules in CORBYS Demonstrator I
Figure 1 illustrates how the Human Sensory System (HSS) and Brain Computer Interface (BCI) interact with
the cognitive framework and executive and cognitive control part of CORBYS Demonstrator I. The cognitive
framework, composed of the Situation Awareness Blackboard (SAWBB) and the Self Organizing Information
Anticipatory Architecture (SOIAA), endows robotic systems with cognitive capabilities and are responsible
for identifying the current state of the system and maintaining a cognitive image of the environment
(including representation of the humans interacting with the system) and generating high-level commands.
The physiological data will be used as input via SAWBB to the reasoning sub-systems within the Situation
Assessment architecture for observation and learning of patterns, identification of deviation from established
reference points, and rectifications suggested to the human-in-the-loop (therapist in case of Demonstrator I).
EEG data is largely utilized by SOIAA for detection of intention and attention of motion. Details of cognitive
modules will be described in documents deliverables under WP4 and WP5.
This deliverable, D3.2, gives detailed information about development of Human Sensory System sensors and
controller software. The main document contains an overall description of the system; however some
implementation details for the sensor systems and test specifications are given in the project internal
appendices.
To meet challenges in the integration of BCI – related algorithms for detection of cognitive processes in real
time robotic applications (i.e. CORBYS first demonstrator), studies on the impact of different artifacts have
been performed and documented in the current deliverable. Ocular, mechanical and electromagnetical
contaminations on the EEG signal have been addressed through the execution of several experiments. Due to
the result obtained, the TMSi system has been chosen as EEG system for the CORBYS demonstrator I. In
addition, the Independent Component Analysis (ICA) has been showed to be a feasible technique to use in
artifact
removal
within
a
simulation
of
the
CORBYS
rehabilitation
scenario.
An initial analysis of passive lower limb movements with attention or non-attention to the motor task have
been performed, in order to improve rehabilitation programs, where the movement repetitions could lead to a
2
D3.2 Physical/Physiological sensing devices
Rev. 1.0
lack of patient engagement compromising the adherence to the therapy. Results reported showed the
feasibility to distinguish between the two conditions.
Dataset from human sensors (including BCI data) have been recorded, analysed and provided to the cognitive
partners as an early integration activity. This have served the basis for which all data is handled, establishing
protocols for sampling, handling data with different time scales, and interfaces for testing various informationtheoretic tools.
The safety of the Human Sensory System and of the Brain Computer Interface is discussed and FMEA risk
analysis is provided. There are no safety concerns for the Human Sensory System for the clinical testing in
WP9.
The relevant requirements from D2.1 are analysed and compared to the current implementation. Most
requirements of the Human Sensory System are fulfilled in this deliverable; the remaining requirements will
be fulfilled in D3.4. Regarding BCI most requirements will be fulfilled in D3.3 and D3.4
The Human Sensory System will be extended in Deliverable D3.4 in month 26. The main addition will be
EMG sensors to measure muscle activity, but improvements in the sensors and the controller will also be
added. The Demonstrator I development will at that time be ready for early integration and hence the Human
Sensory System will be enhanced to fit with the cognitive modules, and the engineering and therapist user
interfaces. Options to simulate the Human Sensory System from recorded data will be added so to ease system
integration.
The analysis Brain Computer Interface submodule will be advanced in D3.3 (month 38) and D3.4 (month 26).
The main addition will include the design of BCI software architecture, addressing also network integration
and synchronization issues, and the implementation of the decoding CORBYS – related decoding algorithms.
3
D3.2 Physical/Physiological sensing devices
Rev. 1.0
1 Introduction
The focus of CORBYS is on robotic systems that have a symbiotic relationship with humans. Such robotic
systems have to cope with highly dynamic environments, as humans are demanding, curious and often act
unpredictably. CORBYS will design and implement a cognitive robot control architecture that allows the
integration of 1) high-level cognitive control modules, 2) a semantically-driven self-awareness module, and 3)
a cognitive framework for anticipation of, and synergy with, human behaviour, based on biologically-inspired
information-theoretic principles.
The CORBYS control architecture will be validated within the two CORBYS demonstrators. The first
demonstrator is the novel mobile robot-assisted gait rehabilitation system CORBYS.
Further information about the design challenges of CORBYS can be found on the CORBYS web page
[www.corbys.eu]. Additionally, general information about the field of Cognitive Robotics can be found on
the EU Framework Program 7 web pages on Cognitive Systems and Robotics
[http://cordis.europa.eu/fp7/ict/cognition/].
One of the main CORBYS objectives is development of advanced sensing module for assessing the physical
and psychological state of human in robots environment. The physiological sensing devices in the CORBYS
project consist of the Brain Computer Interface (BCI) and the Human Sensory System (HSS).
Non invasive Brain Computer Interface (using EEG) performs online detection of human cognitive
information such as intention of leg motion, feedback error-related potential and attention states.
The Human Sensory System consists of a set of physiological sensors to measure patient effort and
movement. The sensors are grouped into four categories, i.e., physiological sensors, movement sensors,
environmental sensors and mechanical sensing technologies.
BCI and HSS are connected to controllers that convert and transmit data to the cognitive framework in the
CORBYS system.
1.1 Document Scope
The present document corresponds to Deliverable 3.2 in the CORBYS project, and is the outcome of work in
CORBYS Task 3-1 on Sensoring, data acquisition, fusion and interpretation as well as Task 3-5 Human-robot
sharing of cognitive information.
1.2 Document Structure
This document is structured as follows: After a brief introduction, the Human Sensory System and BCI
implementation are described. The next chapter focuses on safety followed by a chapter in which relevant
requirements (as identified in WP2) are listed and discussed. The last section concludes the document and also
presents a list of future work.
4
D3.2 Physical/Physiological sensing devices
Rev. 1.0
1.3 Associated Documents
The following documents give additional perspectives for the present work:
-
D2.1 Requirements and Specification: State-of-the-Art, Prioritised End-User Requirements, Ethical
Aspects
D2.2 Detailed Specification of the System: System Architecture Specification with control and data
flow, module interdependencies, user scenarios etc.
D3.1 Sensor Network
5
D3.2 Physical/Physiological sensing devices
Rev. 1.0
2 Human Sensory System realization
EMG interface
EMG
EMG– –sensor
sensorunit
unit
IMU – sensor unit
Environment
USB powered
Cabinet
HSS controller PC
LINUX
ROS
CORBYS
ROS
FRAMEWORK
eth0
Eth
ttyftdi
USB
IMU – sensor unit
Back
Battery powered
HSS controller SW
USB
BLE
BlueGiga if
BLE protocol
ttyacm
USB
USB
Charger
USB HUB
GPN switch unit
Legend
HSS HW
Bluetooth
COM port if
BT protocol
HSS SW
HSS SENSORS
IMU – sensor unit
Back
Charging
Chest unit
Charging
Chest unit
Battery powered
Other HW / SW
Figure 2: Human Sensory System overview
Figure 2 gives an overview of all HSS modules that will be presented in this chapter.
2.1 HSS architecture
The Human Sensory System (HSS) consists of a set of HSS Sensor Modules that interface with the HSS
Controller, which in turn interfaces with the CORBYS network as illustrated in Figure 3. The HSS Sensor
Modules are one or more sensors connected to a sensor controller handling communication. The HSS
Controller handles all communication with the CORBYS Demonstrators via the CORBYS General Purpose
Network. The module-based architecture facilitates the use of HSS modules in multiple Cognitive Robotics
applications.
6
D3.2 Physical/Physiological sensing devices
Rev. 1.0
Figure 3: Human Sensory system architecture
2.1.1
Sensor Module architecture
Sensor modules differ in type and functionality. The simplest module can consist of a single sensor and a
communication module transmitting raw data to the HSS controller. HSS Sensor Modules contain multiple
sensors and a sensor controller handling multi-sensor interpretation as well as transmitting data.
The sensors are grouped into 4 main groups: physiological sensors, movement sensors, environmental sensors
and mechanical sensing technologies. Mechanical sensors are not in the scope of HSS. Figure 4 shows the
gait rehabilitation HSS Sensor Module measurement positions along with their use in the gait rehabilitation
application.
Figure 4: Measurement positioning of the gait rehabilitation HSS Sensor Module
Table 1 lists all the sensors in CORBYS Human Sensory System.
7
D3.2 Physical/Physiological sensing devices
Rev. 1.0
Sensor
Physiological sensors
Electromyography – EMG
Purpose
Gait rehabilitation measurement position
Measuring the electrical activity
produced by the skeletal muscles.
Heart rate and ECG signal
Used in the assessment of physical
effort.
Skin temperature
Monitor skin temperature.
Humidity (sweat)
Used in the assessment of physical
effort
Measured on thigh and calf, both front
and rear muscle, on both legs. A total of 8
sensors.
The heart rate is derived from the ECG
signal measured on the patient chest. The
full ECG signal is also provided.
Skin temperature is measured on the
patient chest.
The humidity is measured on the back of
the patient.
Movement sensors
Inertial Measurement Units
– IMU
Environmental sensors
Environment temperature
The IMU consists of a 3 axis
accelerometer, a 3 axis gyroscope
and optionally a magnetometer.
The IMU is measuring velocity,
orientation and gravitational forces.
In CORBYS it is used for
measuring patient balance and
movement.
One IMU is located at the chest and one at
the back of the patient. The IMU at the
back also includes a magnetometer.
Exterior temperature, used in
humidity/sweat calculations.
Located externally on the HSS Controller
Table 1. CORBYS HSS Sensors
These sensors are grouped into the following sensor modules:
•
•
•
•
Chest unit – a unit connected to a belt around the patient chest measuring heart rate, ECG, skin
temperature and IMU data. See Figure 5.
IMU sensor unit connected to the same belt as chest unit, but at the back of the patient. The IMU
sensor unit measures skin temperature, humidity and activity via the IMU. This IMU also includes a
magnetometer. See Figure 5.
Environment temperature, an IMU sensor unit is attached to the mobile platform and is used for
environment humidity measurements.
EMG sensors, 8 sensors all together. Two sensors are connected to each thigh and similarly two
sensors to each calf for each leg of the patient.
8
D3.2 Physical/Physiological sensing devices
Rev. 1.0
Figure 5: Belt with Chest unit sensors and IMU sensors at back.
2.1.2
Communication links and timing issues
The Human Sensory System is linked to Demonstrator I via the General Purpose Network, a TCP/IP network.
This network is defined in WP 3.
The various HSS sensors are connected to the HSS controller via Bluetooth wireless technology. The Chest
unit uses standard Bluetooth communication via SPP protocol (Serial Port Profile).
The IMU sensor unit uses Bluetooth Smart 1 a part of the Bluetooth 4.0 specification (formerly named
Bluetooth Low Energy) for communication; this is an emerging low-power, low latency communication
protocol providing defined profiles for various sensor devices.
All computers on the General Purpose Network (GPN) will have synchronized clocks, the accuracy is
expected to be in the range 1-5msec.
Testing made using Bluetooth communication between the Chest unit and a computer showed that the link
introduced a delay variation in the range 0-250msec. To compensate for this a timestamp is added to each
measurement. The chest unit has a local timer and a timestamp is added before sending to the computer. This
solution has good accuracy for inter sample timing. In order to obtain an absolute accuracy, the local timer
needs to be synchronized with the computer clock. The current synchronization algorithm does a single
handshake during connection, this algorithm does not compensate for the delay in the messages. The result is
an uncertainty in the absolute accuracy in the range 0-250ms. There are currently no specific requirements for
the absolute accuracy, the minimum required accuracy will be investigated and added to D3.4. These
requirements will be implemented in the final version of the HSS Controller.
1
http://www.bluetooth.org/Technical/Specifications/adopted.htm
9
D3.2 Physical/Physiological sensing devices
Rev. 1.0
The delay variation for the Bluetooth Smart communication between IMU sensor unit and a computer has not
been yet tested, however it is expected to be better than the delay variation measured for the Chest unit.
The current implementation adds timestamp when sensor data arrives at the computer. The relative delay
between samples will be low, however the absolute accuracy will not be better than the delay characteristics
of the Bluetooth Smart link. This will be tested during setup of the system and test results will be presented in
D3.4.
10
D3.2 Physical/Physiological sensing devices
Rev. 1.0
2.2 Chest Unit
EMG interface
EMG
EMG– –sensor
sensorunit
unit
IMU – sensor unit
Environment
USB powered
Cabinet
HSS controller PC
LINUX
ROS
CORBYS
ROS
FRAMEWORK
eth0
Eth
ttyftdi
USB
IMU – sensor unit
Back
Battery powered
HSS controller SW
USB
BLE
BlueGiga if
BLE protocol
ttyacm
USB
USB
Charger
USB HUB
GPN switch unit
Legend
Bluetooth
COM port if
BT protocol
HSS SENSORS
IMU – sensor unit
Back
Charging
Chest unit
Charging
Chest unit
Battery powered
Other HW / SW
Figure 6: Chest unit in Human Sensory system
The Chest Unit shown in green in Figure 6 is used for measuring ECG, heart rate, temperature at the chest of
the patient. In addition it can measure acceleration and rotation of the patient.
The Chest Unit and the belt are connected by a mechanical and electrical connector device. The majority of
electronics, as well as radio and battery are to be hosted in the Chest Unit. The belt contains contact electrodes
for heart rate measurements and ECG.
The Chest Unit is manufactured in a rugged plastic material, and is designed without any sharp edges. It is
protected against splash water and entry of foreign objects, but it is not designed to be immersed in water, see
Figure 7.
The Chest unit integrates both activity and physiological sensors in the same device. A 3-axis accelerometer is
used to detect activity; and a 3-axis gyroscope used for measuring rotation. Both the accelerometer and
gyroscope raw data are provided as part of the HSS dataset to the cognitive modules. Skin temperature is
measured by an infrared (IR) sensor. Heart rate measurement is based on detection of electrical signals from
the heart measured on the skin. Extraction of heart rate from the complex analogue signals is performed in the
digital domain by the chest unit processor system. Power is provided by a rechargeable battery, and
11
D3.2 Physical/Physiological sensing devices
Rev. 1.0
components are selected for minimum power consumption. Wireless communication is using the Bluetooth
SPP protocol stack.
Figure 7: Belt with Chest unit sensor.
Microcontroller software is based on the real-time kernel µC/OS-II, and optimized for low power
consumption by putting microcontroller and radio into sleep mode when idle. Sensors are read at regular
intervals, and data is transmitted using Bluetooth, avoiding requests from the client application. To ease
sensor data analysis, a Java based monitoring application for recording and visualization of sensor data is
developed and shown in Figure 8.
The main development of the Chest Belt and Chest Unit has been done by SINTEF in a project called
ESUMS 2. However, the embedded microcode has been enhanced to fit the usage in the CORBYS project
providing more sensor data at a better timing accuracy to the cognitive system than required for use in the
ESUMS project.
2
•
Gyroscope functionality has been added.
•
The sample rate has been increased from 300ms to 52ms better follow the movement of the patient.
•
The wireless data rate has been doubled compared to previous versions.
•
The resolution of the sample timestamps has been increased from 1seconds up to 4 milliseconds.
ESUMS, http://www.sintefannualreport.com/2011/en/with-your-heart-in-his-hands/
12
D3.2 Physical/Physiological sensing devices
Rev. 1.0
Figure 8 A screen dump of the Java application receiving sensor data from Chest unit.
2.2.1
Chest Unit Test
The Chest Unit has been thoroughly tested in lab by SINTEF and externally towards reference systems before
use in the CORBYS project. Hence only the modified configuration is tested in the CORBYS project. This
covers:
- Gyroscope
- Raw accelerometer data
- Time stamping
- Higher Bluetooth data-rates
2.2.1.1 Gyroscope
The gyroscope has been tested with respect to static rotation using a turntable. This has been used to verify
scaling of the outputs and rotation axis. The combined message with gyroscope and accelerometer data
introduced in CORBYS has been verified towards the Java application at the PC. Visual inspection of the
graphs has been done as a check of data usefulness with respect to body movements.
2.2.1.2 Raw accelerometer data
13
D3.2 Physical/Physiological sensing devices
Rev. 1.0
Raw accelerometer data has been provided and tested before the use in CORBYS, but then at a lower rate of
300 milliseconds. The rate is increased to 52 milliseconds for CORBYS as requested by the cognitive partners
to be able to detect gait movements. A regression test of the accelerometer functionality with activity and
posture algorithms has been performed. The new combined message with gyroscope and accelerometer data
has been tested towards the Java application at the PC. Visual inspection of the graphs has been done as a
check of data usefulness with respect to body movements.
2.2.1.3 Timestamping
The enhanced timestamp resolution has been tested by analysing log files created at the computer side. The
information has been compared to the known sample rate for the different sensors. No deviations have been
found.
2.2.1.4 Bluetooth delay variation
Test made using Bluetooth communication between Chest unit and a computer showed that the link
introduced a delay variation in the range 0-250msec. To compensate for this a timestamp is attached to each
measurement. The chest unit has a local timer used to add timestamp before sending to the computer. This
solution gives an accuracy of relative timing between samples. In order to have an absolute accuracy, the local
timer needs to be synchronized with the computer clock. The current synchronization algorithm does a single
handshake during connect. This algorithm does not compensate for the delay in the messages used. The result
is an uncertainty in the absolute accuracy in the range 0-250ms.
Test datasets with sensor data from the Chest Unit have been recorded using the Java monitoring application.
Based on these datasets the delay variation has been analysed.
The figure below shows delay variation for a person standing. This gives a static distance for the
communication path. This is also expected to be the case for the demonstrator I where the patient is attached
to the powered orthosis and the Bluetooth interface to the HSS controller is placed on the mobile platform
cabinet. The delay variation is given in milliseconds (one line per 10 milliseconds). The graph is based on a
dataset containing 1500 message packets send over a period of one minute.
Figure 9 Test 1- Person standing
The graph shows the distribution of delay for 1500 packets. X-axis: Delay in milliseconds. Y-axis: Number of packets
14
D3.2 Physical/Physiological sensing devices
Rev. 1.0
The next figure shows a person walking in a corridor. The person is starting and stopping. The computer is not
moved with the patient. Here some longer delays show up. This is probably due to some retransmissions. The
graph is based on a dataset containing 2200 message-packets send over a period of two minutes.
Figure 10 Test 4 - Person walking
The graph shows the distribution of delay for 2200 packets. X-axis: Delay in milliseconds. Y-axis: Number of packets
2.2.1.5 Bluetooth transfer rate
When enabling raw data at higher data rates there were a concern whether the data were above the practical
Bluetooth transfer rate. From previous projects SINTEF had experienced large variations related to CPU load
on the computer. Different brands of Bluetooth dongles had also shown different performance. We chose to
use a dongle from TARGUS and a dongle from Roving. The TARGUS dongle has the Bluetooth protocol
stack running on the computer while Roving has an external protocol stack running on a processor inside the
dongle. Running an external protocol stack offloads the computer and makes the communication more robust
with respect to CPU load on the computer.
A special test program was made generating a test sequence that could detect lost and duplicated data.
The tests showed that it was possible to run at data rates of 7500 bytes/sec, but data loss was frequent. It also
showed that the Roving dongle handled much higher data rates than the TARGUS dongle.
The goal to transfer 2000 bytes/sec was achieved by both dongles.
Test duration
(sec)
Rate (bytes/sec)
Sequence
length
(bytes)
Device
Loss (sequence/sec)
Handshake
1580
3750
15
Targus
0
No
46983
3750
15
Targus
0,001
RTS
198
5000
20
Targus
0,510
RTS
15
Comment
D3.2 Physical/Physiological sensing devices
Rev. 1.0
310
5000
20
Targus
0,135
RTS
256
5000
20
Targus
0,871
No
521
5000
20
Targus
0,576
No
271
7500
30
Targus
5,070
No
18388
3750
15
Roving
0
RTS
Much other
activity on
computer
68529
5000
20
Roving
0
RTS
Much other
activity on
computer
492
5000
20
Roving
0
RTS
253
7500
30
Roving
1,273
RTS
Table 2 Bluetooth bandwidth measurement results
16
D3.2 Physical/Physiological sensing devices
Rev. 1.0
2.3 IMU Sensor Unit
The IMU sensor is used in two settings in the CORBYS Demonstrator I system, connected to the Chest Belt
on the back of the patient, and on the mobile platform for measuring environment humidity and temperature.
2.3.1
IMU Sensor Back unit
EMG interface
EMG
EMG– –sensor
sensorunit
unit
IMU – sensor unit
Environment
USB powered
Cabinet
HSS controller PC
LINUX
ROS
CORBYS
ROS
FRAMEWORK
eth0
Eth
ttyftdi
USB
IMU – sensor unit
Back
Battery powered
HSS controller SW
USB
BLE
BlueGiga if
BLE protocol
ttyacm
USB
USB
Charger
USB HUB
GPN switch unit
Legend
Bluetooth
COM port if
BT protocol
HSS SENSORS
IMU – sensor unit
Back
Charging
Chest unit
Charging
Chest unit
Battery powered
Other HW / SW
Figure 11: IMU sensor unit in Human Sensory system
The IMU sensor is used for humidity and IMU measurements on the patient back and for environment
temperature measurements. The development for the IMU sensor was started in an internal SINTEF project
called TRÅLE [Liverud et all, 2012]., however most of the firmware development and device manufacturing
were done in the CORBYS project. The following sections give a technical description of the implementation
of the sensor unit, including basic application and user information.
The IMU sensor unit shown in Figure 12 integrates both activity and physiological sensors in the same
device. A combined 3-axis accelerometer and gyroscope, in addition to a magnetometer, form an inertial
measurement unit (IMU). Skin temperature is measured by an infrared (IR) sensor. Additionally, two I2C
ports are available for external sensors. In CORBYS usage, a combined humidity/temperature sensor is fitted
measuring the humidity at the patient back. Power is provided by a rechargeable battery, and components are
selected for minimum power consumption. Wireless communication is by the new Bluetooth Smart
17
D3.2 Physical/Physiological sensing devices
Rev. 1.0
technology, previously called Bluetooth Low Energy. This is a feature of the Bluetooth 4.0 standard,
providing low latency, low power short range communication. Profiles are defined for sensor values like
temperature and heart rate; allowing Smartphones or other devices to receive data without proprietary drivers.
The Bluetooth Smart technology is included in the Continua Health Alliance design guidelines 3.
Figure 12: The IMU sensor unit without and with enclosure, measuring 54x34x15 mm.
Microcontroller software is based on the real-time kernel µC/OS-II, and optimized for low power
consumption by putting microcontroller and radio into sleep mode when idle. Sensors are read at regular
intervals, and data transmitted using the Bluetooth Smart attribute indicate operation, avoiding requests from
the client application. To ease sensor data analysis, a Java based monitoring application for recording and
visualization of sensor data is developed and shown in Figure 13.
3
http://www.continuaalliance.org/products/design-guidelines.html
18
D3.2 Physical/Physiological sensing devices
Rev. 1.0
Figure 13: A screen dump of the Java application receiving sensor data from IMU sensor unit.
19
D3.2 Physical/Physiological sensing devices
2.3.2
Rev. 1.0
IMU Environment humidity and temperature sensor unit
A special version of the IMU is used for measuring environment temperature and humidity. These
measurements are needed as a reference value for calculating the humidity at the patient.
The unit is mounted on the mobile unit and connected to the HSS controller using a USB cable. The external
sensor, voltage regulator and the IMU unit is embedded into a small box.
•
The external sensor is the same type that is used for IMU located on the patient’s back.
•
Voltage regulator is used for providing power from the USB plug to the external sensor and the IMU
environment unit. The IMU unit will not have any battery to avoid procedures for charging the unit.
The unit will be operating as long as the HSS controller computer is running.
•
All parts are integrated into a single box to avoid cabling between multiple boxes.
USB
Cabinet
Reg
HSS controller PC
Eth
USB
USB
USB
IMU – sensor unit
Environment
Sensor
USB
BLE
BlueGiga if
BLE protocol
Legend
HSS SENSORS
Other HW / SW
Figure 14: IMU environment sensor unit internals
The wireless interface is the same as for IMU back sensor unit. The IMU unit may be enabled providing
movement information for the mobile platform.
2.3.3
IMU Sensor Unit Test
The IMU sensor is currently in an internal design verification test where all features are tested against
specifications. The planned tests are listed in Table 3
ID
Module
B1
Battery
B2
Battery
B3
Battery
Test description
From completely empty battery, charge battery. Measure charging time. Record charging
curve from console printout
From fully charged battery, measure usage time for measuring IR temperature and dual
humidity sensor before battery empty.
Record data using IMU Sensor Unit Java app and console
From fully charged battery, measure usage time for measuring all sensors before battery
empty. Record data using IMU Sensor Unit Java app and console
20
D3.2 Physical/Physiological sensing devices
Rev. 1.0
L1
LED
Test led at power on, off, charging, low battery – may change
BU1
Button
Push button for 3 seconds to switch on, similarly 3 seconds to switch off
BU2
Button
P1
Power usage
Short push to make unit start advertising
Measure standby current before and after running unit with all sensors (multimeter, average
of 200 samples)
P2
Power usage
Measure current usage not transmitting any data
P3
Power usage
Measure current usage transmitting IR temperature and dual humidity sensors
P4
Power usage
P5
Power usage
Measure current transmitting data from all sensors
Test that the unit is automatically switched off after 5 minutes when not connected on
Bluetooth.
H1
Humidity sensor
IR1
Skin temperature
Compare humidity sensor readouts for humidity and temperature at low, medium and high
humidity and compare results with Sensirion development kit 4 results. 1 second update rate.
Compare IR temperature readout at 1 seconds readout interval with values from ESUMS
Chest Belt, high, low and medium temperature.
M1
Magnetometer
Compare readout with other device - reference device to be specified
A1
Accelerometer
Test against g in x,y,z directions
G1
Gyroscope
Mount unit at stepper motor and compare readout with stepper-motor speed at various speeds
C1
Communication
Test max distance before communication lost
C2
Communication
Communication performance tested and presented at pHealth 2012 [Liverud et all, 2012].
Table 3. IMU sensor Unit design verification test
Tests have so far uncovered that battery charging and discharging does not work fully as intended and needs
to be corrected. This will probably be corrected by an embedded firmware update for the IMU Sensor Unit.
Full tests results will be provided in D3.4
4
http://www.sensirion.com/en/products/humidity-temperature/humidity-sensor-sht21/
21
D3.2 Physical/Physiological sensing devices
Rev. 1.0
2.4 EMG sensors
EMG interface
EMG
EMG– –sensor
sensorunit
unit
IMU – sensor unit
Environment
USB powered
Cabinet
HSS controller PC
LINUX
ROS
CORBYS
ROS
FRAMEWORK
eth0
Eth
ttyftdi
USB
IMU – sensor unit
Back
Battery powered
HSS controller SW
USB
BLE
BlueGiga if
BLE protocol
ttyacm
USB
USB
Charger
USB HUB
GPN switch unit
Legend
Bluetooth
COM port if
BT protocol
HSS SENSORS
IMU – sensor unit
Back
Charging
Chest unit
Charging
Chest unit
Battery powered
Other HW / SW
Figure 15: EMG sensors in Human Sensory System
The purpose of the EMG sensors is to measure the muscle activity of the patient. EMG will be measured on
the thigh and calf of each leg. The CORBYS EMG system will be presented in D3.4 in month 26.
22
D3.2 Physical/Physiological sensing devices
Rev. 1.0
2.5 Human Sensory System controller
EMG interface
EMG
EMG– –sensor
sensorunit
unit
IMU – sensor unit
Environment
USB powered
Cabinet
HSS controller PC
LINUX
ROS
CORBYS
ROS
FRAMEWORK
eth0
Eth
ttyftdi
USB
IMU – sensor unit
Back
Battery powered
HSS controller SW
USB
BLE
BlueGiga if
BLE protocol
ttyacm
USB
USB
Charger
USB HUB
GPN switch unit
Legend
HSS HW
Bluetooth
COM port if
BT protocol
HSS SW
IMU – sensor unit
Back
Charging
Chest unit
Charging
Chest unit
Battery powered
Other HW / SW
Figure 16: HSS controller in Human Sensory System
2.5.1
HSS Controller Hardware
The HSS controller software will be running on an industrial ruggedized Intel Atom based computer mounted
in the Carrier Frame of Demonstrator I. At this point the HSS Controller Software is developed and tested in a
VirtualBox virtual machine with a hard drive of 16GB and 2GB of allocated RAM.
23
D3.2 Physical/Physiological sensing devices
2.5.2
Rev. 1.0
CORBYS GPN and Robot Operating System (ROS)
The HSS Controller is one module of the overall CORBYS System (as described in deliverable D2.2). The
integration between the different CORBYS modules is done through the CORBYS General Purpose Network
(GPN). The CORBYS GPN is a standard TCP/IP network over Ethernet. To ease the integration of the
different module the CORBYS system uses the Robot Operating System (ROS) 5 framework. The ROS
framework runs on top of the operating system and TCP/IP network and provides standard ways to describe
and publish both services and data in a distributed network.
ROS (Robot Operating System) provides libraries and tools to help software developers create robot
applications. It provides hardware abstraction, device drivers, libraries, visualizers, message-passing, package
management, and more. ROS is licensed under an open source, BSD license.
ROS facilitates communication between processes running on the same or on different machines. The
communication between processes can take place either by a blackboard-style paradigm with publishers and
subscribers, called Topics or in a server-client mode, called Services.
A ROS node is a process that performs computation and runs in a separate process, using a socket (IP address
and port). Nodes are combined together into a graph and communicate with one another using streaming
topics, remote procedure call (RPC) services, and the Parameter Server. These nodes are meant to operate at a
fine-grained scale; a robot control system will usually comprise many nodes. For example, one node controls
a laser range-finder, one node controls the robot's wheel motors, one node performs localization, one node
performs path planning, one node provide a graphical view of the system, and so on.
The use of nodes in ROS provides several benefits to the overall system. There is additional fault tolerance as
crashes are isolated to individual nodes. Code complexity is reduced in comparison to monolithic systems.
Implementation details are also well hidden as the nodes expose a minimal API to the rest of the graph and
alternate implementations, even in other programming languages, can easily be substituted.
The ROS nodes of the CORBYS system are all running Ubuntu Linux version 10.04LTS 32bits and the
Electric version of ROS. These choices are expected to provide a stable platform for the development and use
of the CORBYS system. A set of common CORBYS ROS messages and topics have been defined and allow
for the integration between the HSS Controller and the cognitive modules.
2.5.3
HSS Controller setup and operating system
Figure 17 details the installation steps for setting up the HSS controller. The first step is a standard installation
of Ubuntu Linux. A single user account is setup on the HSS Controller:
Login: hsscontroller - Password: corbys
The next installation step installs some required packages and security updates from Ubuntu repositories.
5
http://www.ros.org/wiki/
24
D3.2 Physical/Physiological sensing devices
Rev. 1.0
###############################################################################
# This script describes the steps to setup the HSS Controller.
###############################################################################
# Install Ubuntu 10.04 LTS from the iso image
# Install all updates
# Install packets for git, svn, ssh, Java and Maven2 (for hss controller JAVA debug UI)
sudo apt-get install subversion default-jdk eclipse openssh-server maven2 git-core
###############################################################################
# Install ROS
###############################################################################
sudo sh -c 'echo "deb http://packages.ros.org/ros/ubuntu lucid main" > \
/etc/apt/sources.list.d/ros-latest.list'
wget http://packages.ros.org/ros.key -O - | sudo apt-key add sudo apt-get update
sudo apt-get install ros-electric-desktop-full
echo "source /opt/ros/electric/setup.bash" >> ~/.bashrc
. ~/.bashrc
mkdir ~/ros_workspace
echo "export ROS_PACKAGE_PATH=~/ros_workspace:\$ROS_PACKAGE_PATH" >> ~/.bashrc
echo "export ROS_WORKSPACE=~/ros_workspace" >> ~/.bashrc
. ~/.bashrc
echo $ROS_PACKAGE_PATH
###############################################################################
# Install QT 4.8.1
###############################################################################
# Download "QtSdk-online-linux-x86-v1.2.1.run" from QT web page
mv /home/hsscontroller/Downloads/QtSdk-online-linux-x86-v1.2.1.run /opt/
cd /opt
chmod u+x QtSdk-online-linux-x86-v1.2.1.run
./QtSdk-online-linux-x86-v1.2.1.run
###############################################################################
# Install QWT
###############################################################################
# Download "qwt-6.0.1.zip" from qwt web page
cd /opt/
cp /home/hsscontroller/Downloads/qwt-6.0.1.zip .
unzip qwt-6.0.1.zip
cd qwt-6.0.1/
qmake
make
sudo make install
sudo su
# The following is required for the CORBYS_GUI to run
echo "/usr/local/qwt-6.0.1/lib" > /etc/ld.so.conf.d/qwt.conf
ldconfig
###############################################################################
# Checkout CORBYS SVN CODE
###############################################################################
svn checkout https://corbys.eu/svn/CORBYS
Figure 17 HSS Controller Installation Script
2.5.4
HSS Controller Software
This section describes the drivers and software parts of the HSS controller which makes the link between the
sensor units and the CORBYS general purpose network. Figure 18 presents a high level view of the
architecture of the HSS Controller software. The HSS software is composed of a set of driver components (on
the left hand side of Figure 18) which manage the connection with individual sensors and of a front-end
component which provides the interface of the HSS controller on the CORBYS GPN. The interface of the
HSS controller is made of 3 ROS topics which have been defined in collaboration with other CORBYS
modules. The Config_Data topic is a common topic for all modules and allows distributing the configuration
of the CORBYS system to all its modules. The Heart_Beat topic is also used by all modules to publish status
information at regular intervals. Finally the Sensor_Data_HSS topic is specific to the HSS Controller and used
to publish the sensor data.
25
D3.2 Physical/Physiological sensing devices
Rev. 1.0
HSS Controller
Chest Unit
Sensors
IMU
Sensors
Bluetooth
BLE
EMG
Sensors*
Chest Unit
Driver
IMU Sensor
Driver
EMG
Driver*
Engineer GUI*
Parameter
Server
HSS_Chest_Phi
HSS_Chest_IMU
HSS_Chest_ECG
Config_Data
HSS_Back_TH
HSS_Back_IMU
CORBYS
GPN
HSS
Front-end
HSS_Env_TH
HSS_Env_IMU
Sensor_Data_HSS
Heart_Beat
HSS_EMG_Data*
(*) Not implemented in D3.2 Prototype
Figure 18 Architecture of the HSS Controller Software
Each component described in Figure 18 runs in its own process (ROS node) in the HSS controller in order to
avoid having the failure of one component bring down the whole HSS controller. The communication between
the drivers and the front-end component uses a set of ROS topics which publish the data received by the
drivers. These ROS topics are internal to the HSS controller and should not be used by other CORBYS nodes.
The HSS front-end receives the data from all drivers, performs some sanity checking on the data and forwards
sensor data to the CORBYS Sensor_Data_HSS topic.
The following sub-sections describe the components of the HSS controller software.
2.5.5
Chest Unit Driver
Figure 19 presents the interfaces of the Chest Unit driver. In the initialization phase the driver needs to
discover the chest units within range and read their battery status. Once a session is initialized the driver is
connected to only one chest unit, collects the sensor data and publishes it on three separate ROS topics. These
three topics were designed to bundle together the data with common update rates:
•
3 seconds for skin temperature, heart rate, posture and activity data. The battery status has been added
to this message even if its actual update rate is around 30 seconds.
•
52ms for the IMU data (accelerometers and gyroscopes).
26
D3.2 Physical/Physiological sensing devices
•
Rev. 1.0
32ms for 8 samples chunks of ECG data which has an update rate of 4ms. At this stage chunk of 8
samples seems to be a good trade-off between bandwidth and latency. It might be adjusted if required
by the nodes using the data.
On the sensor side the chest units are connected to the HSS controller via Bluetooth using the Serial Port
Profile (SPP). The SPP is supported by recent Linux kernel modules but turned out to be unstable in Ubuntu
10.04LTS. To avoid any instability the HSS controller is setup to use an external Bluetooth adapter which
implements its own Bluetooth stack as well as the SPP. The chosen adapter is a BlueSMiRF Bluetooth modem
which is based on a Roving Networks Bluetooth module 6. The HSS controller communicates with the
Bluetooth using a standard serial interface.
ChestBelt Engineer / Test Interface
Publish data on 3 ROS topics
3s
ChestBelt
protocol
over UDP
Chest Unit
Sensors
USB/Serial
HSS_Chest_Phi
timestamp : Long
sequence : Int16
heart_rate : Int16
skin_temp : Int16
activity : Int8
posture : Int8
battery : Int8
Chest Unit
Driver
32ms
Roving Networks
Bluetooth Module
52ms
HSS_Chest_IMU
timestamp : Long
sequence : Int16
accel : Int16[3]
gyro : Int16[3]
HSS_Chest_ECG
timestamp : Long
sequence : Int16
raw_ecg : Int16[8]
Discover several sensors but
connect to one belt at the time
Figure 19 Interface of the Chest Unit Driver
For testing and debugging purposes, a graphical user interface can be connected to the Chest Unit driver. This
graphical interface communicates with the driver over UDP which allows executing it either locally on the
HSS Controller or remotely on a separate computer. This interface mainly displays and logs the data coming
from the sensor. It might also be used to send commands to the sensor for debugging purposes. This
functionality will be disabled in the deployed HSS Controller in order to avoid conflicting commands reaching
the sensors.
6
https://www.sparkfun.com/products/158
27
D3.2 Physical/Physiological sensing devices
Rev. 1.0
Figure 20 presents the architecture of the Chest Unit driver and shows the three main part of the driver: the
operating system part, the Chest Unit driver and the testing GUI respectively.
On the operating system side (bottom of Figure 20), the Bluetooth module is connected to the HSS Controller
via a FTDI USB to TTL serial chip. The Linux kernel includes a driver for the FTDI chip and recognizes it is
a standard serial port. By default the device is automatically mounted and provided with a device handle
named /dev/ttyUSBX where X varies. In order for the Chest Unit driver to always be able to connect to the
right device we have used the Linux UDEV service in order to automatically create a symbolic link with a
fixed name whenever the specific FTDI chip is connected. Figure 20 presents the UDEV rule added to create a
link from /dev/bluesmirf to the appropriate device. This rule includes the serial number of the FTDI chip used
with the Bluetooth module. This serial number should be changed to match the actual chip being used on the
HSS Controller. The /dev/bluesmirf can be used as a standard serial port with an 115200 baud rate, 8 data bits,
1 stop bit and no parity (8N1). Hardware flow control is supported all the way from the computer to the FTDI
chip, Bluetooth module and Bluetooth SPP.
The Chest Unit driver itself consists of four main components (blue in Figure 20):
•
The Roving Network AT / Serial driver. This component manages all the communications with the
serial device. It uses AT commands to interact with the Bluetooth stack of the module when no sensor
is connected, and it decodes serial packets coming from the sensor once a connection has been
established. It provides a simple API for discovering and connecting to Bluetooth devices, and allows
sending and receiving packets to the Chest Unit.
•
The Chest Unit Driver encodes and decodes serial packets according the Chest Unit protocol. It
provides an API to configure the Chest Unit and receive the data from its different sensors.
•
The ROS Publisher uses the Chest Unit driver to subscribe to the sensor data and forward the data on
to the HSS Controller internal ROS topics.
•
The UDP GUI Server is implemented for debugging purposes. It forwards the data exchanged on the
serial port to a UDP socket. A test application can be connected to display the sensor data.
All these components are modelled using the ThingML languages 7 which allow automatically generating
C/C++ implementations for Linux and ROS packages.
7
http://www.thingml.org
28
Chest Unit
(Generated from ThingML)
D3.2 Physical/Physiological sensing devices
To HSS Front-end
GUI / Data log
ChestUnit-Java
HSS_Chest_Phi
HSS_Chest_IMU
HSS_Chest_ECG
Test / Engineer
Interface
UDP Driver
ROS Publisher
UDP
GUI
Server
Chest Unit Driver
Roving Networks AT / Serial
Serial Port (115200 bps)
Roving Networks AT commands /
Transparent serial over Bluetooth
Linux
(Ubuntu 10.04)
Rev. 1.0
/dev/bluesmirf
10-ftdi.rules
# Get a fixed device for the Roving Networks Bluetooth Adapter
BUS=="usb", SYSFS{idProduct}=="6001", SYSFS{idVendor}=="0403",
SYSFS{serial}=="AE01AAEF", SYMLINK+="bluesmirf"
Linux FTDI Driver
Linux kernel USB
Roving Networks
Bluetooth Module
Figure 20 Architecture of the Chest Unit Driver
The test and engineer interface on the top right corner of Figure 20 is a Java application which implements its
own driver and GUI for the Chest Unit sensors. It can be connected locally or remotely to the HSS Controller
Chest Unit module in order to visualize the low level communications with the sensors. Since the debug
application implements its own driver for the Chest Unit, this allows comparison of the data provided by the
debug application and the data forwarded on the ROS topic, which in turn enables validation of the
implementation of the HSS Controller driver.
2.5.6
IMU Sensors Driver
The structure and design of the IMU sensor driver is similar to the Chest Unit driver. Figure 21presents an
overview its architecture.
29
D3.2 Physical/Physiological sensing devices
Rev. 1.0
IMU Sensors Engineer /
Test Interface
HSS_Back_TH
HSS_Env_TH
IMU Back Units
100ms
BGAPI
protocol
over UDP
Bluegiga
BLED112
USB/Serial
Bluetooth
Smart (4.0)
IMU Sensor
Driver
Discover units and connect to
2 units (back and environment)
IMU Environment Unit
(mounted on the demonstrator)
3s
HSS_Back_TH /
HSS_Env_TH
timestamp : Long
sequence : Int16
skin_temp : Int16
temp1 : Int16
humidity1 : Int16
temp2 : Int16
humidity2 : Int16
battery : Int8
HSS_Back_IMU
HSS_Env_IMU
HSS_Back_IMU /
HSS_Env_IMU
timestamp : Long
sequence : Int16
magneto : Int16[3]
accel : Int16[3]
gyro : Int16[3]
quaternion : Int16[4]
Publish data on 4 ROS topics:
2 from the back unit and 2 from
the environmental unit.
Figure 21 Interface of the IMU Sensor Driver
The IMU Unit sensors use the Bluetooth Smart 4.0 protocol for which no support is available in Linux. To
communicate with the sensors, the HSS Controller uses an external Bluetooth Smart 4.0 module which
implements the protocol stack and provides and API over a USB virtual serial port. The chosen module is a
Bluegiga BLED112 USB dongle 8.
The IMU Sensor driver connects to two different IMU sensors. One is mounted on the back of the chest belt
of the patient, and the other is mounted on the CORBYS demonstrator mobile platform and used as an
environmental sensor. The environment device is a single unit which has an address that can be hard coded in
the HSS Controller. The back unit is one of several units available for training sessions and needs to be
discovered and configured in the initialization phase of each training session. At least two units need to be
available in order to be able charge one while the other is being used on a patient.
The IMU Sensor driver outputs data on 4 different topics: 2 topics for the environment unit and 2 for the back
unit. The topic messages have been designed to group together data which have common update rates:
•
8
3 seconds for the temperature and humidity measurements and the battery status. The temp2 and
humidity2 value are currently not populated in the CORBYS demonstrator but provided for future
http://www.bluegiga.com/BLED112_Bluetooth_low_energy_dongle
30
D3.2 Physical/Physiological sensing devices
Rev. 1.0
applications. The skin temperature for the environment unit is not meaningful in the context of the
demonstrator since the sensor is mounted on the chassis of the demonstrator.
•
100 ms for the IMU data. This value might be adjusted in order to cope with the Bluetooth Smart
bandwidth limitations. The current implementation provides all IMU data (raw accelerometer, gyros
and magnetometer data as well and fused attitude data from the accelerometers and gyros as a
quaternion).
IMU Sensor
(Generated from ThingML)
To HSS Front-end
IMU Sensor-Java
HSS_Back_TH
HSS_Back_IMU
HSS_Env_TH
HSS_Env_IMU
BGLIB-Java
Test / Engineer
Interface
UDP Driver
ROS Publisher
IMU Sensor Driver
BGLIB: BLE Bluegiga API
UDP
GUI
Server
Bluegiga transport protocol
Serial Port (115200 bps)
Bluegiga proprietary binary
protocol for Bluetooth 4.0
Linux
(Ubuntu 10.04)
GUI / Data log
udev rule: /dev/bled112
46-bluegiga.rules
# Get a fixed device for the bluegiga BLE dongle at /dev/bled112
ATTRS{idVendor}=="2458" ATTRS{idProduct}=="0001" MODE="0660" \
GROUP="dialout" SYMLINK+="bled112"
Linux CDC ACM Driver
Bluegiga
BLED112
Linux kernel USB
Figure 22 Architecture of the IMU Sensors Driver
Figure 22 presents the structure of the IMU sensor driver. The bottom part of the figure represents the
operating system drivers which link to the BLED112 dongle. The BLED112 dongle implements an USB CDC
ACM driver which is recognized by the Linux kernel as a virtual serial port and automatically mounted as a
/dev/ttyACMX device (X varies depending on the number of devices plugged to the computer). In order to get
a fixed device inode, the HSS controller uses a custom UDEV rule which creates a /dev/bled112 symbolic link
for the BLED112 dongle. The UDEV rule is generic for all BLED112 dongles and assumes that only one is
plugged to the HSS Controller. In case several dongles need to be used specific rules which include the serial
numbers of the dongles should be created. The /dev/bled112 devices is a standard virtual serial port with a
115200 baud rate, 8 data bits, 1 stop bit and no parity (8N1). The BLED112 dongle uses a proprietary packet
based protocol over this serial link.
The implementation of the IMU Sensor driver is composed of four layers and a UDP GUI server which is
used for diagnosis and debugging purposes. These five components are modelled using the ThingML
languages and the C/C++ and ROS implementation code is automatically generated:
•
The Bluegiga transport protocol manages the connection with the /dev/bled112 device and sends and
receives packets conforming to the Bluegiga proprietary protocol.
31
D3.2 Physical/Physiological sensing devices
Rev. 1.0
•
The BGLIB layer provides an API to interact with the Bluetooth Smart stack of the BLED112 dongle.
API commands are transformed to packets to be transmitted to the dongle and incoming packets
coming from the dongle are decoded and forwarded as messages (or callbacks) the layer above.
•
The IMU Sensor driver uses the Bluetooth Smart API provided by the BGLIB layer and implements
the specific services provided that the IMU Sensor Unit. It provides a specific API to the layer above
which allow connecting to IMU Sensor units, subscribing to the different sensors it contains and
collecting sensor data.
•
The ROS publisher is the top layer of the stack. It subscribes to data from the two IMU Units to use
for a training session and forwards the sensor data on the HSS Controller as ROS topics.
The test and engineer interface on the top right corner of Figure 22 is a Java application which implements its
own driver and GUI for the IMU Unit sensors. It can be connected locally or remotely to the HSS Controller
IMU Sensor driver in order to visualize the low level communications with the BLED112 dongle. Since the
debug application implements its own driver for the Bluegiga protocol and IMU Sensor, comparing the data
provided by the debug application and the data forwarded on the ROS topic allow validating the
implementation of the HSS Controller driver.
2.5.7
EMG Sensors Driver
Not implemented as part of D3.2. Once the EMG sensor will be selected a specific driver will be implemented
to provide the sensor data internal HSS ROS topics (in the same way as other sensors).
2.5.8
HSS Controller Front-End and ROS Interface
The role of the HSS front-end is to collect all the data coming from the sensors and provide this data on the
CORBYS general purpose network using the SensorData_HSS ROS topic. The sensor data on this topic is
represented using a common CORBYS format which is used by all CORBYS modules.
#define CORBYS_NODES_HSS 42
CORBYS_Heart_Beat
flags : uint32
HSS_Chest_Phi
HSS_Chest_IMU
HSS_Chest_ECG
HSS_Back_TH
HSS_Back_IMU
HSS_Env_TH
HSS_Env_IMU
HSS_EMG_Data*
HSS_Chest_Phi
timestamp : Long
sequence : Int16
heart_rate : Int16
skin_temp : Int16
activity : Int8
posture : Int8
battery : Int8
HSS_Chest_IMU
timestamp : Long
sequence : Int16
accel : Int16[3]
gyro : Int16[3]
HSS_Chest_ECG
timestamp : Long
sequence : Int16
raw_ecg : Int16[8]
HSS_Back_TH /
HSS_Env_TH
timestamp : Long
sequence : Int16
skin_temp : Int16
temp1 : Int16
humidity1 : Int16
temp2 : Int16
humidity2 : Int16
battery : Int8
header
CORBYS_Message_Header
hostID : uint8
nodeID : uint8
Heart_Beat
CORBYS
GPN
header
CORBYS_Sensors_Data
message : String
HSS_Back_IMU /
HSS_Env_IMU
timestamp : Long
sequence : Int16
accel : Int16[3]
gyro : Int16[3]
magneto : Int16[3]
quaternion : Int16[4]
HSS internal topics and messages
sensorData[]
Sensor_Data_HSS
CORBYS_Sensor_Samples
sensorID : uint8
samples[]
CORBYS_Sensor_Reading
timestamp : time
reading : float32[]
CORBYS topics and messages
Figure 23 From the Internal topics to the CORBYS topics
32
D3.2 Physical/Physiological sensing devices
Rev. 1.0
Figure 23 presents the ROS messages used by the HSS Controller front-end. The left hand side of the figure
presents the ROS messages provided by the sensor drivers. These messages are specific to each individual
sensor and use the data types provided by the sensor (no scaling or any other form of processing is done in the
sensor driver nodes). The right hand side of the figure presents the structure of the ROS messages exchanged
on the CORBYS GPN.
The CORBYS_Sensor_Data message is a generic structure which is designed to represent data coming from
all the different sensors. Each message is composed of a header which provides the origin of the data, and a
set of sensor samples. Each sensor sample corresponds to a specific sensor ID and can contain a set of sensor
readings. Each reading has a timestamp and a set of sensor data represented as 32bit floating point numbers.
This structure allows representing in a single message data coming from different sensors as well as several
sets of readings for a single sensor.
2.5.8.1 HSS Controller data interface
The HSS Controller main functionality is to collect the data from the different sensors and publish it on the
Sensor_Data_HSS topic. The front-end receives the raw data from the sensors, the data needs to be processed,
scaled and converted to a set of 32bit floats which can be provided to the CORBYS topic.
Other CORBYS nodes should not subscribe to the HSS internal topics but only to the HSS_Sensor_Data
topic.
Several alternatives can be used to define the sensor IDs on the Sensor_Data_HSS topic:
•
One Sensor ID per individual sensor value. This option leads to a large number of sensors (about 24
without the EEG sensors). Since many sensors are sampled at the same rate this requires to duplicate
many timestamps in the output messages.
•
One Sensor ID per physical sensor unit. This option allows reducing the number of sensors to the
minimum, but is impractical because the sensors of a single unit have different sampling rates, which
means that not all data can be populated for each update.
•
One Sensor ID per HSS internal topic. This is the solution which was chosen because it minimizes the
required bandwidth on the CORBYS network (no duplication of timestamps and no missing data).
Message
SensorID
HSS Sensor Data Reading Format (float32[])
Size
Rate(ms)
HSS_Chest_Phi
0xA1 (161)
[heart_rate, skin_temp, activity*, posture*, battery]
5
3000
HSS_Chest_IMU
0xA2 (162)
[accel_x, accel_y, accel_z, gyro_x, gyro_y, gyro_z]
6
52
HSS_Chest_ECG
0xA3 (163)
[ecg1, ecg2, ecg3, …, ecg8]
8
32
HSS_Back_TH
0xA4 (164)
[skin_temp, temp1, humidity1, temp2*, humidity2*, battery]
6
3000
HSS_Back_IMU
0xA5 (165)
[ax, ay, az, gx, gy, gz, mx, my, mz, qw, qx, qy, qz]
13
100
HSS_Env_TH
0xA6 (166)
[skin_temp*, temp1, humidity1, temp2*, humidity2*, battery]
6
3000
HSS_Env_IMU*
0xA7 (167)
[ax, ay, az, gx, gy, gz, mx, my, mz, qw, qx, qy, qz]
13
100
HSS_EMG_*
0xAx*
?
?
?
Figure 24 Data format on the CORBYS Sensor_Data_HSS topic
33
D3.2 Physical/Physiological sensing devices
Rev. 1.0
Figure 24 presents how the data from the internal sensor topics is provided in CORBYS_Sensor_Samples as
well as the sampling rate of each sensor. Variables marked with a * are currently not populated or populated
with a non specified value, and hence should not be used at this point.
The amount of data generated by the HSS controller sensors will use some bandwidth on the CORBYS GPN.
As an early estimation of the required bandwidth: given the size if the messages and their sampling rate the
required bandwidths are 2.53kByes/s to transmit the sensor data and 3.32kBytes/s when taking headers and
timestamps into account. The overhead of the ROS framework might require some additional bandwidth;
however these numbers seem reasonable and should not pose any problem to the CORBYS GPN.
Subscribers to the Sensor_Data_HSS topic should use the IDs provided in Figure 24 (and the corresponding
macros in the CORBYS_Common source folder) to decode the data according to the specified sensor reading
format.
Name
heart_rate
skin_temp
activity*
posture*
battery
accel_x
accel_y
accel_z
gyro_x
gyro_y
gyro_z
ecg[1..8]
skin_temp
temp1
humidity1
temp2*
humidity2*
battery
skin_temp*
temp1
humidity1
temp2*
humidity2*
battery
ax
ay
az
gx
gy
gz
mx
my
mz
qw
qx
qy
qz
Internal Topic
HSS_Chest_Phi
HSS_Chest_Phi
HSS_Chest_Phi
HSS_Chest_Phi
HSS_Chest_Phi
HSS_Chest_IMU
HSS_Chest_IMU
HSS_Chest_IMU
HSS_Chest_IMU
HSS_Chest_IMU
HSS_Chest_IMU
HSS_Chest_ECG
HSS_Back_TH
HSS_Back_TH
HSS_Back_TH
HSS_Back_TH
HSS_Back_TH
HSS_Back_TH
HSS_Env_TH
HSS_Env_TH
HSS_Env_TH
HSS_Env_TH
HSS_Env_TH
HSS_Env_TH
HSS_[Back|Env]_IMU
HSS_[Back|Env]_IMU
HSS_[Back|Env]_IMU
HSS_[Back|Env]_IMU
HSS_[Back|Env]_IMU
HSS_[Back|Env]_IMU
HSS_[Back|Env]_IMU
HSS_[Back|Env]_IMU
HSS_[Back|Env]_IMU
HSS_[Back|Env]_IMU
HSS_[Back|Env]_IMU
HSS_[Back|Env]_IMU
HSS_[Back|Env]_IMU
Internal Type
Int16
Int16
UInt8
UInt8
UInt8
Int16
Int16
Int16
Int16
Int16
Int16
Int16
Int16
Int16
Int16
Int16
Int16
UInt8
Int16
Int16
Int16
Int16
Int16
UInt8
Int16
Int16
Int16
Int16
Int16
Int16
Int16
Int16
Int16
Int16
Int16
Int16
Int16
Scaling
0,1
0,01
?
?
1
?
?
?
?
?
?
?
0,01
0,01
0,01
?
?
1
0,01
0,01
0,01
?
?
1
?
?
?
?
?
?
?
?
?
?
?
?
?
Unit
BPM
°C
?
?
%
m/s²
m/s²
m/s²
deg/s
deg/s
deg/s
mV
°C
°C
%
?
?
%
°C
°C
%
?
?
%
m/s²
m/s²
m/s²
deg/s
deg/s
deg/s
nT/m
nT/m
nT/m
?
?
?
?
Mininum
25
20
?
?
0
?
?
?
?
?
?
?
20
10
0
?
?
0
20
-10
0
?
?
0
?
?
?
?
?
?
?
?
?
?
?
?
?
Maximum
230
40
?
?
100
?
?
?
?
?
?
?
40
50
100
?
?
100
40
50
100
?
?
100
?
?
?
?
?
?
?
?
?
?
?
?
?
Resolution
0,1
?
?
?
1
?
?
?
?
?
?
?
?
?
0,1
?
?
1
?
?
0,1
?
?
1
?
?
?
?
?
?
?
?
?
?
?
?
?
Accuracy
1
0,1
?
?
1
?
?
?
?
?
?
?
0,1
0,1
5
?
?
1
0,1
0,1
5
?
?
1
?
?
?
?
?
?
?
?
?
?
?
?
?
Notes
Enumerated values
Enumerated values
No Sensor
No Sensor
No Meaning
No Sensor
No Sensor
Figure 25 HSS Controller sensor data specification
Figure 25 details the scaling applied to the raw data, the minimum and maximum value and the unit for all
the individual sensor data published by the HSS Controller. As an indication the table also provides the
resolution and accuracy of each sensor. The missing information in this table as well as the specification of the
EEG data will be added in deliverable 3.4.
In the event of missing sensor data or invalid sensor data the HSS front-end will not output the corresponding
data on the Sensor_Data_HSS topic. A common strategy for handling these cases should be defined for all
CORBYS Sensor data topics. Solutions such as outputting zero data, or NaN (Not-a-Number) values in
34
D3.2 Physical/Physiological sensing devices
Rev. 1.0
combination with some flags could be used to keep populating the topics at the expected rate while clearly
marking the data as abnormal.
2.5.8.2 HSS Controller status and flags
In addition to the Sensor_Data_HSS topic, the HSS Controller has to publish heartbeat messages on the
general CORBYS Heart_Beat topic. These heartbeats are processed by the CORBYS functionality supervisor
node in order to check the health of all CORBYS nodes. Heartbeat messages include a 32bit flag which can be
use to indicate the status of the node. At this point no common format for these flags has been defined yet, but
the HSS Controller will use the flag value to indicate the status of each individual driver node, and the
presence of valid data at the expected rate for the different sensors.
2.5.9
HSS Controller Engineer GUI(s)
The CORBYS GUI developed as part of the CORBYS ROS framework provides a highly customizable
interface which covers the visualization of all the sensor data produced by the HSS Controller. No additional
GUI is expected to be developed to display the HSS Controller sensor data. However a specific GUI module
will be developed in order to display the set of available sensor units and initialize training sessions by
choosing the sensors to be used by the patient. This HSS Controller specific GUI will also display information
about the state and status of the HSS Controller (based on the heart beat messages sent by the HSS
Controller). According to the CORBYS ROS framework, the interactions between the GUI and the HSS
Controller will be implemented via the central ROS parameter server. The exact parameters to be used will be
defined and documented in D3.4.
2.5.10 HSS Controller Initialization
The HSS Controller has two main modes: A standby mode in which sensors are discovered and the state of
their battery is monitored. In this mode no data is produced on the CORBYS ROS topic, the list of available
sensors, their serial number and battery status are just populated in the ROS parameter server. The second
mode is a "training session" mode in which the HSS Controller is connected to the sensors and continuously
transfers data on the CORBY ROS topics. On startup the HSS Controller automatically goes to standby mode.
The following sub-section briefly describe the initialization steps of the HSS Controller at startup and when
transitioning to the "training session" mode.
2.5.10.1HSS Controller startup
On startup the HSS Controller automatically starts the sensor drivers and the HSS front-end as system
daemons. The drivers start in standby mode and scan for available sensors to collect their serial numbers,
status and battery levels. The front-end also starts in standby mode and monitors the executions status of the
sensor drivers. All nodes wait for the CORBYS ROS core and parameter server to be available. Once the
parameter server is available, the HSS front-end populates it with the list of available sensors and their
characteristics. The HSS Controller stays in this mode until a session is initialized.
2.5.10.2Training session initialization
In order to start a training session and publish sensor data on the CORBYS ROS topic the HSS controller
needs to go through a number of steps:
•
Discovery of the sensor within range (and battery status). This is continuously done in the standby
mode and the set of available sensor is kept updated in the ROS parameter server.
35
D3.2 Physical/Physiological sensing devices
Rev. 1.0
•
Selection of the desired sensors and connection. The user of the demonstrator has to choose among
the available sensors the ones to use for the training session. At this point the sensor should be set up
on the patient and the selection should be populated in the ROS parameter server using the HSS
Controller GUI. Once the selection of the sensors has been made, the drivers attempt to connect to
these sensors.
•
Log all versions of software, hardware and firmware. The first step after connecting to the sensor is to
query the sensors for their versions and status in order to populate the parameter server with complete
information about the sensors being used and confirm that the connection is properly established.
•
Synchronization of the clocks of the sensors with the HSS Controller. In order to get accurate time
information the wireless sensors need to be synchronized with the HSS Controller clock (which is
itself synchronized with other CORBYS nodes). Experiments will be made in order to evaluate if a
time synchronization "over the air" allows for a sufficient accuracy. In case wireless transmission
delays are too unpredictable, the clock synchronization could be done using serial communication
with the sensor on the charging station. In that case the synchronization would have to be made before
the sensors are installed on the patient.
•
Configuration of the sensors and subscription to the data. At this stage the sensor drivers will
configure the sensors according the required CORBYS update rates and start collecting data from the
sensors.
•
Start forwarding data on the CORBYS GPN. After all sensors have been configured and the driver has
stated forwarding data on the HSS internal topics, the front-end will start forwarding the data on the
CORBYS topic. At this stage the HSS Controller is still in initialization mode and the subscriber to
the sensor data should use the incoming data only for initialization checks.
•
Basic checks of the selection and positioning of the sensors. The CORBYS GUI should be used to
visualize the data coming from the HSS Controller and confirm that all sensors are providing sensible
data and that all sensors are properly mounted. In particular:
•
o
ECG Signal should be visualized to confirm that the electrodes are properly placed. A drop of
water can be used to moisten the electrodes and the belt size should be adjusted for a good
contact between the electrodes and the skin.
o
Chest IMU and Back IMU data should be checked to confirm the correct orientation of the
Sensors.
o
Skin temperature should be checked for both the Chest and Back sensors to make sure that no
obstacles block the IR sensors.
o
Specific checks will be required for the EMG sensors.
Switch to training mode. After the sensor mounting and connection has been confirmed by the user,
the HSS controller switches to the training session mode in which the data can be exploited by
subscribers.
36
D3.2 Physical/Physiological sensing devices
Rev. 1.0
At the end of a training session the HSS controller stops forwarding sensor data, disconnects from the
sensor and goes back the standby mode.
2.5.11 HSS Controller Test
The HSS controller is designed to allow testing its components in isolations and to ease the testing of its
integration in the CORBYS demonstrator. This section provides a short description of the different testing
phases for the HSS Controller. The specification of the tests and results will be provided in D3.4.
2.5.11.1Unit tests of the driver modules
All the HSS Controller sensor drivers are independent processes connected to the sensors on one side and
outputting sensor data on the HSS Controller internal ROS topics. For each sensor a testing application can be
connected at the lowest level of the driver. The debugging GUI implements it own version of the sensor
driver. Inconsistencies and faults can be detected by logging the sensor data produced by the HSS Controller
driver and comparing it to the data produced by the debugging GUIs.
2.5.11.2Unit tests of HSS Front-End
The front-end of the HSS controller front-end can be tested in isolation of the sensor drivers by using ROS
bags in order to capture data sequences on the internal HSS ROS topic and playing these data to the HSS
Controller front-end. The output data of the front-end can be compared to the data played on the internal
topics in order to detect potential malfunctions.
2.5.11.3Robustness tests
An important aspect of the HSS controller is its robustness with respect to potential failures. Special attention
will be made at testing the ability of the HSS Controller to detect failures and recover from them. Testing will
include scenarios with a high occurrence probability such as:
•
Loss of connection with a sensor.
•
Crash of a driver and/or of the radio adapter.
•
Poor wireless connection leading to packet loss and/or high latency.
•
Failure of a sensor and/or abnormal sensor values.
2.5.11.4QoS and extra-functional properties tests
Specific tests will be performed to evaluate typical data latency and time stamping offset in order to ensure
that the CORBYS requirements are met. The time stamping offset mostly depends on the time
synchronization between the HSS Controller and the sensor devices. Latency can be introduces at every stage
of the sensor data processing and distribution. It is expected that the main source of latency is the wireless
communications between the sensor units and the HSS controller but latency introduced by the HSS internal
communication though ROS topics, the processing time of the drivers and front-end as well as distribution of
the data on the CORBYS ROS topic will be evaluated.
2.5.11.5Integration tests
37
D3.2 Physical/Physiological sensing devices
Rev. 1.0
To test the integration of data from the HSS controller with the reset of the CORBYS demonstrator, a set of
front-end datasets will be provided as ROS bags which can be replayed to simulate the presence of the HSS
Controller on the ROS network. Multiple data sets will be provided for the different phases of the execution of
the HSS controller.
Also, to test the HSS controller front-end ROS module integration with the CORBYS demonstrator, a set of
datasets will be provided as ROS bags which can be played to simulate the presence of the HSS sensor
modules for the HSS controller front-end. This enables usage of the HSS controller front-end without having
the HSS sensor modules present.
An aggregated dataset containing data from HSS, BCI and hardware sensors should be made. This will give
synchronized data from all modules for a training session.
38
D3.2 Physical/Physiological sensing devices
Rev. 1.0
2.6 Offline charging station for wireless sensors
EMG interface
EMG
EMG– –sensor
sensorunit
unit
IMU – sensor unit
Environment
USB powered
Cabinet
HSS controller PC
LINUX
ROS
CORBYS
ROS
FRAMEWORK
eth0
Eth
ttyftdi
USB
IMU – sensor unit
Back
Battery powered
HSS controller SW
USB
BLE
BlueGiga if
BLE protocol
ttyacm
USB
USB
Charger
USB HUB
GPN switch unit
Legend
HSS HW
Bluetooth
COM port if
BT protocol
IMU – sensor unit
Back
Charging
Chest unit
Charging
Chest unit
Battery powered
Other HW / SW
Figure 26: HSS controller in Human Sensory system
The chest unit and the IMU sensor need to be charged regularly. There will be two sets of sensors available,
one set in use while the other set is charging. Charging is done in the offline charging station. The charging
time is less than 2 hours and the following operational time is more than 2 hours (exact numbers will be
provided in D3.4). The charge level can be monitored over Bluetooth and shown at the therapist GUI. During
the training session only the charge level for the sensors in use are shown. Between training sessions the
charge level for all sensors can be shown.
Location of the charger will be decided when the EMG system is specified in D3.4, the goal is to have a
common location for charging of all sensors.
There are three alternatives for powering the charger:
•
The HUB is powered from the HSS controller computer USB interface. Location has to be on the
mobile platform.
•
The HUB is powered from an external 24VDC power supply. Location has to be on the mobile
platform.
39
D3.2 Physical/Physiological sensing devices
•
Rev. 1.0
The HUB is powered from an external 220VAC power supply. Location has to be on a table
somewhere near CORBYS Demonstrator I.
Figure 27 HSS - offline charging station
40
D3.2 Physical/Physiological sensing devices
Rev. 1.0
3 Brain Computer Interface (BCI)
CORBYS project focuses on a robotic system that have a symbiotic relationship with humans. One of Its
objective concerns the development of a perception system for assessing the physical and mental state of the
environment including humans. The perception system includes multimodal physiological sensing devices
such as the Brain Computer Interface (BCI) and the Human Sensory System (HSS) that are important for
human motor control and learning.
The Non invasive Brain Computer Interface (using EEG) will detect human cognitive information (in realtime) used by the robot cognitive control architecture for perception of the human mental state. In particular
BCI will decode cognitive processes such as intention of leg motion, feedback error-related potential and
attention states.
In the CORBYS gait rehabilitation system scenario, the subject will be walking assisted by a robotic device.
In this context the EEG data will be affected by typical artifacts such as EOG and EMG, but also
electromagnetic artefacts, due to the mobile platform (i.e. motors used for the powered orthosis and for the
mobile platform’s wings) and mechanical artefacts associated with head movements and locomotion. In
section 3.2, Analysis and removal of movements’ artifacts during locomotion, ocular and mechanical artefacts
and their removal process based on Independent Component Analysis (ICA) technique are addressed. Section
3.1, Evaluation of EEG acquisition systems that could reduce the noise level, focuses on the study of the
electromagnetic noise (i.e. noise due to the DC motors) and their impact in the CORBYS demonstrator. The
EEG system that showed to be less affected by electrical noise contamination will be chosen for the CORBYS
gait
rehabilitation
system.
In robotic related rehabilitation programs it has been suggested that human cognitive processes, such as motor
intention, attention, and higher level motivational states play an important role in the success of the therapy
[Tee et al, 2008]. In this context the BCI module will online detect cognitive processes of interest for the
CORBYS gait rehabilitation system such as intention of leg motion, feedback error-related potential and
attention states. Section 3.3, Detection of attention during assisted passive leg motion, reports the progress in
designing a BCI system to decode passive lower limb movements with attention and non-attention to the
motor task.
41
D3.2 Physical/Physiological sensing devices
Rev. 1.0
Figure 28 Brain Computer Interface submodule overview
Figure 28 presents an overview of the BCI submodule. It will be integrated in the architecture of the 1st
CORBYS demonstrator as a ROS Node (task 6.1 - Architecture decomposition and definition).The inputs
required from the BCI (i.e. configuration parameters and system state, D2.2) will be read from the ROS
Parameter Server; its outputs will be sent from the BCI ROS Node using a ROS Topic (Sensors_Data_EEG).
The ROS Message sent will include EEG data and BCI decoding outputs from those decoders enables in the
current demonstrator state. Further information will be reported in D3.4.
42
D3.2 Physical/Physiological sensing devices
Rev. 1.0
3.1.1 Evaluation of EEG acquisition systems that could reduce the noise level
3.1.2
Introduction
EEG systems take an important place in BCI applications as they provide an excellent way to describe brain
signals with good time resolution and lower cost than other brain activity acquisition systems. A key
parameter of studies of EEG systems is the signal-to-noise ratio, (SNR), as it provides a measurement of the
signal quality. In the EEG context, this parameter is affected by a significant number of internal and external
sources which influence the recorded signal, independent of the system used for recording. Sensor-scalp
contact (i.e. to guarantee low contact impedance between the electrode and scalp) and the capability to
automatically remove or reduce noise sources are also important aspect of the acquisition system that
determines the quality of the recording. There are several studies in literature about EEG acquisition
technologies, such as those with focus on classical gel-based electrodes and novel water-based electrodes
applied in specific brain-computer-interface [Searle and Kirkup, 2010; Matteucci et al, 2007]. Research teams
have also developed working prototypes of dry EEG sensors and demonstrated that the signal obtained can be
largely comparable to wet electrodes [Popescu, 2007].
Among those existing sources of noise that can affect the EEG signals, the electromagnetic noise caused by
external devices such as motors, and the power supply are those that could have a significant impact on the
use of BCI in CORBYS scenario. This deliverable focuses on a technical evaluation of the quality of EEG
recordings of two commercial systems to find adequate equipment for the CORBYS scenario. This is because
in CORBYS, the EEG system will be forced to work close to a robot in movement, which is a source of
electromagnetic activity and noise and might affect the EEG readings. This document reports the evaluation of
two commercial EEG systems in scenarios similar to the CORBYS 1st demonstrator. The first one is the Porti
amplifier, a multichannel device of TMSi (Twente Medical System International BV, Netherlands) with
passive electrodes of water solution. The second one is g.USBAmp, a biosignal amplifier produced by g.Tec
(Graz, Austria), with active electrodes of gel solution.
The first section describes the technical specifications of these two commercial EEG systems. The second
section presents an analysis of both EEG systems in two different experiments. The objective of the
experiments was to study the possible effect of electromagnetic noise in the EEG signals caused by electric
motors in two different scenarios: the first experiment was developed under a controlled situation, where a DC
brush motor was used as proposed by related partner developing mobile platform (i.e. there will be the motors
used for the powered orthosis and for the wings in the mobile platform); the second experiment involved a
less-controlled scenario, where the source of electromagnetic noise was caused by a generic AC motor.
43
D3.2 Physical/Physiological sensing devices
Rev. 1.0
Figure 29 Commercial EEG systems. On the left: passive electrode (water solution) and Porti-16 of TMSi. On the right: active
electrode (gel solution) and g.USBamp of g.Tec.
3.1.3
EEG Systems: Evaluation of technical specifications
The electroencephalogram (EEG) is an acquisition system that measures electrical brain activity through
electrodes placed on the surface of scalp. These sensors measure neural activity by electrical potentials
(voltage) over time between a signal electrode and a reference one. Every EEG system includes three
acquisition blocks: the electrodes, the wires (interface sensor-amplifier) and the amplifier (filtering,
amplifying and signal conversion).
- Electrodes
The brain activity is acquired by electrodes which build an interface between the scalp skin and the metal of
the wire. The electrode is a metallic sensor that converts the local differences of the concentration of charged
ions into an electrical potential signal. However, the measurement of the electrical activity of target
physiological processes can contain a number of undesired components, for example 50 Hz-mains
interference, electrode-offset-potential, drift in offset potential or fluctuations caused by mechanical
influences such as movements.
An EEG sensor is tested concerning its technical specifications, such as (active, passive), level of noise, DC
behaviour and variability, frequency response, impedance and its stability, weight, wearability and sensor
material. However the main challenge to EEG electrodes is to get a good low impedance contact to the skin
(for comparison about different electrode technologies refer to CORBYS Deliverable D2.1- Section 15 State
of the Art in Non Invasive Brain Computer Interface). The other point need to be concerned is whether the
contact of the electrode is affect by the locomotion of the human. From the technical specifications, there is
not a clear reason to state which of these electrodes is better suited for this purpose.
- Wires
Both devices have shielded wires which send the EEG signals that are measured at the electrode into the
amplifier. The main advantage of the shielded cable is that movements of the cable and environmental noise
do not influence the signals in the cable. Thus, this source of noise does not affect the recorded EEG signals in
the CORBYS scenario. In TMSi equipment, the cables are covered with active shielding, so that the signal is
immune to the cable movement artifacts and to mains interference (i.e. 50 or 60 Hz coupling). Thus, the inner
44
D3.2 Physical/Physiological sensing devices
Rev. 1.0
wire of the input cable is not affected by any capacitive coupling, which means that there will be no artifact
when the cables are moving. On the other hand, g.Tec provides no information about the specific cable
shielding used for their systems. In CORBYS, it is expected to find artifacts from wire movements due to the
user locomotion and thus, the TMSi wires seem to be more appropriate as the movement of the cables will be
difficult to avoid (notice that this statement is mainly due to the lack of information about the g.Tec wires).
- Amplifier
The amplifier includes the following functions: filtering, amplifying and signal conversion. The next table
resumes the technical specifications of the commercial amplifiers related to the signal quality such as input
impedance, the common-mode rejection ratio (CMRR).
Table 4 Technical specifications of the Porti-16 and g.USBamp amplifiers.
Input referred noise
Input impedance
CMRR
Connector
Special features
Porti-16
<2 μVpp (@ Fs = 128 Hz)
>1012 Ω
>100 dB
g.USBamp
< 0.3 μV RMS (0.1-10Hz)
>1010 Ω
>100 dB
Micro coax, active shielding
1 mm 2-pin touch-proof
and electrodes
filter
Active guarded shielded leads
50-60Hz selectable hardware notch
One of the most relevant amplifier parameters is the input impedance as it alleviates the need of a low
impedance contact to the skin. High input impedance in an amplifier can decrease the dependency of the
electrode contact impedance, on the contrary low input impedance causes load of bio-signal source and it
results in damaging of the signal. The input impedance in the Porti-16 is of two orders of magnitude higher
than in the g.USBamp. The need of low contact impedance can be mitigated partly by the high input
impedance of the amplifier and the shielding of the electrode wires or connectors as with the TMSi equipment
as well as by using local impedance adapters as the active electrodes of g.Tec electrodes. CMRR is a measure
of the ability of a test instrument to reject interference that is common to both of its measurement inputs,
which is equal in both amplifiers. Mains interference (i.e. 50-60 Hz) can be prevented with active guarded
shielded leads and electrodes, also it is possible to reject this interference using a notch filter of main
frequency (i.e. 50 Hz) by hardware (as g.USBamp) or by software as well.
From a theoretical point of view, the high input impedance of the amplifiers could be a technical parameter
that makes the TMSi amplifier more appropriate for CORBYS (as both amplifiers have very similar
characteristics). However, this decision needs to be confirmed by the experimental evaluation as there are
many other aspects that mediate in the EEG recordings.
3.1.4
Evaluation scenarios
This section addresses the influence of electrical motor noise in the EEG recordings of both EEG acquisition
systems in two different experiments. The first experiment was developed under a controlled situation, where
a DC brush motor was used as proposed by related partner developing mobile platform; the second
experiment involved a less-controlled scenario, where the source of electromagnetic noise was caused by a
45
D3.2 Physical/Physiological sensing devices
Rev. 1.0
generic AC motor.
Data recording
Both experiments used the same data recording setup. Three subjects participated in both experiments. EEG
signals were recorded simultaneously from g.Tec and TMSi systems. Data acquired from g.Tec system was
recorded from 10 active electrodes placed F3, Fz, F4, C3, Cz, C4, P3, Pz, P4 and Oz (according to the 10/10
system). Ground and reference electrodes were placed on FPz and on the right earlobe respectively. Data
acquired from TMSi equipment was recorded from 10 water electrodes placed next to g.Tec electrodes at the
following locations: AFF3, AFFz, AFF4, FOC3, FOCz, FOC4, OCP3, OCPz, OCP4 and POOz (according to
the 10/5 system), see Figure 30.
Figure 30 On the Left: EEG montage of gel electrodes in blue (g.Tec) and water electrodes in green (TMSi). On the
right: picture of real montage gel electrodes (red) and water electrodes (white).
Electrodes were placed over a g.Tec cap according to the extended 10/5 international system. For both EEG
systems, data were digitized at a sampling rate of 256 Hz, band-pass filtered at 0.5-60 Hz and recorded
simultaneously with the same computer using BBT proprietary software.
Experiment 1: Noise influence of a DC brush motor
In this section, both EEG systems are exposed to electromagnetic (capacitive or inductive) noise
from a DC brush motor, as these types of motors were as specified in D2.2.
Experimental protocol
Experiments were carried out in a real scenario environment with ambient light and without any particular
restriction on the background noise and luminance. Subjects were seated in a comfortable chair approximately
100 cm from a LCD monitor, which displayed the tasks instructions. A DC brush motor (i.e. PITTMAN
46
D3.2 Physical/Physiological sensing devices
Rev. 1.0
GM9236E 349-R1 12 VCD 500 CPR) was used to analyze its influence in EEG recording. Motor was
working with constant velocity at 17 motor revolutions per second (RPS), and it was located at two different
positions from the subject’s head respectively: distance_1 = 0.5 m. and distance_2 = 1 m.
Figure 31 Subject during experimental procedure, EEG was recorded during relaxing task with noise influence
produced of DC brush motor.
The experimental design is shown in Figure 32 with three conditions (distance_1, distance_2 and motor off).
Each condition was recorded for 60 trials, where after each condition the subject has 2 minutes of rest. Each
trial was composed of a 3 seconds resting time and 5 seconds baseline executing task. The subject was asked
to avoid blinking during the baseline condition.
Figure 32 Time diagram of protocol in the experimental procedure.
Data processing and results
For each trial, the EEG data from the baseline were visually inspected, and one-second epochs were discarded
if a physiological artifact was identified. EEG power spectrum was calculated by a sliding window
periodogram of one second with 30 ms of overlapping and then averaged. To compute the periodogram a 1s
hamming window was used with a resolution of 0.25 Hz (1024 points using zero-padding) and power-line is
notch-filtered at 50 Hz.
For each condition the power spectrum density (PSD) was averaged across channels and then averaged across
subjects. Figure 33 shows the PSDs for each condition and amplifier. The only visible difference is around 56
Hz where it seems to be an artifact as, up to our knowledge, this abnormal activity cannot be attributed to any
known neural activity. This artifact appears in all three conditions of the g.Tec system (even when the motor is
OFF), and it appears only in one condition of the TMSi system (i.e. motor ON). This suggests that the nature
47
D3.2 Physical/Physiological sensing devices
Rev. 1.0
of the artifact is not due to the DC motor. The rest of the PSD is very similar in three conditions and thus there
is no other apparent effect of artifacts on the EEG. This result is confirmed by the Figure 34, which shows
PSDs resulted from both EEG systems in the 3 conditions recorded.
Figure 35 shows the ratio between the PSDs of motor ON and motor OFF conditions for each subject (this
ratio eliminates the effect of the different gains of the amplifiers and allows the comparison). The results show
that there is not apparent difference below 20 Hz (that is the range where the difference should be due to the
RPS of the motor – about 17 Hz). Greater difference is present in the range 20-50 Hz; it can’t be attributed to
the DC motors since in the analysis performed in Figure 33 and Figure 346 this is not visible. Artifacts above
50 Hz are not due to the motor activity.
Figure 33 Average power spectrum density for every conditions: motor OFF, large distance, short distance using TMSi system
(left) and g.Tec one (right).
Figure 34 Average power spectrum densities for three conditions recording: motor on located 0.5 m far subject’s head, motor
on located 1 m far subject’s head and motor.
48
D3.2 Physical/Physiological sensing devices
Rev. 1.0
Figure 35 Ratio between power spectrum of noise conditions and motor OFF for TMSi (blue) and g.USBamp (red) systems of
each subject. Noise condition is considered motor located 1 m (left) and 0.5 m (right) from subject’s head.
Experiment 2: Noise influence of an AC electrical engine
Experiments were carried in a real scenario environment with ambient light, without any particular restriction
on the background noise and luminance. Subjects were seated in a comfortable chair approximately 60 cm
from a LCD monitor, which displayed the tasks instructions. Close to the subject there is an electric mini-bike
activated by remote control by the subject. From an electrical point of view, this device behaves as a solenoid
which works in different frequencies depending on the speed bike and for the experimentation the slowest
speed motor was used.
Remote
control
Electric
mini-bike
Figure 36 Subject during experimental procedure, EEG was recorded during relaxing task with noise influence
produced by an electric mini-bike.
49
D3.2 Physical/Physiological sensing devices
Rev. 1.0
The experimental design is shown in Figure 37 with two conditions (motor ON and motor OFF). A
trial was composed of a rest time of 3 seconds and baseline task of 5 seconds. The subject was asked
to avoid blinking during baseline period. A total of 60 trials were recorded (30 trial of each
condition).
Figure 37 Time diagram of protocol in the experimental procedure.
Data processing and results
Artifact filtering was performed by visual inspection of the baseline recordings in both conditions. Epochs of
one second were discarded if a physiological artifact was identified. EEG power spectrum was calculated by a
sliding window periodogram of one second with 30 ms of overlapping and then averaging. To compute the
periodogram a hamming window was used with a resolution of 0.25 Hz (1024 points using zero-padding) and
power-line notch-filtered at 50 Hz and bandpass filtered between 0.5 and 60 Hz.
Figure 38 shows EEG signal affected by noise (i.e. 20 Hz sinusoidal contamination).
Figure 38 Example of four EEG channels contaminated by electric mini-bike recorded using TMSi (left - with scale of 100
uV) and g.Tec (right- with scale of 1000 uV) system, over a time window of 4 seconds.
Figure 39 shows power spectrum averaged across channels of EEG recorded for both systems for Subject 2. A
peak at 20 Hz and at its fundamental harmonic frequencies (i.e. 40 Hz, 60 Hz, etc.) reflects the electric noise
contamination.
50
D3.2 Physical/Physiological sensing devices
Rev. 1.0
EEG system 1
EEG system 2
Figure 39 EEG spectrum recording during motor ON (blue) and motor OFF (red) conditios, averaged across 10 channels
registered with TMSi (left) and g.Tec (right) system for subject 2.
Figure 40 Ratio between power spectrum of noisy and free noise one for Porti (blue) and g.USBamp (red) sytem of each
subject.
Since both amplifiers have different gain across frequency it is not possible to compare their EEG or
corresponding PSD. To avoid this phenomenon, the ratio between the power spectrums of both conditions is
used, this is presented in Figure 40 for all subjects. Notice that the level in Noise condition in g.USBamp is
more than two orders of magnitude greater than in the TMSi system for frequencies between 0.5-60Hz and for
all the subjects. For both systems the ratio frequency spectra show a peak at 20 Hz and at its fundamental
harmonic frequencies (i.e. 40 Hz, 60 Hz, etc.), although in case of g.Tec noise data is spread over all
frequencies while it is not the case for the TMSi system.
3.1.5
Discussion
Two commercial EEG systems are presented in this document. Results show that TMSi equipment was less
affected by electrical noise than g.Tec system. There could be two reasons result in this effect: differences in
technical characteristics in the electrodes or the differences in the technical features of the amplifiers. While
the electrodes could affect the data quality due to the impedance between sensor and scalp, the amplifier could
alleviate deficiency regards signal quality. In summary the TMSi system has a better performance in the
presence of motor activity and unknown sources of electrical noise, which is the best replication that has been
devised to emulate the working conditions of the CORBYS experimentation.
51
D3.2 Physical/Physiological sensing devices
Rev. 1.0
3.2 Analysis and removal of movements’ artifacts during locomotion
3.2.1
Introduction
Wide range of artifacts can occur in EEG recordings. One possible categorization of artifacts is based on their
origin: technical (originated from outside the human body, such as the 50/60 Hz power-line noise, changes in
electrode impedances, etc) and physiological (originated from a variety of bodily activities, such as potentials
introduced by eye or body movements, muscular activity, cardiac activity, etc.). While some of these artifacts
are easily identified, others may have similar characteristics to the neural activity and therefore difficult to
recognize and to eliminate. The presence of artifacts in EEG signal has an impact on the analysis and could
lead to unreliable results if they interfere with the neural process under study. This is the reason why one
important part of biomedical signal processing is to understand the noise and the artifacts in order to minimize
their impact either in the analysis or in the development of the technology.
The two physiological artifacts being mostly studied are the eye movements (EOG) and in the body
movement (EMG). EOG artifacts are due to electrical eye activity propagated throughout the body and
recorded at the scalp surface [Schlögl et al, 2007]. They are generally high-amplitude patterns in the brain
signal caused by the blinking of the eyes, or low-amplitude patterns caused by movements such as rolling the
eyes [Anderer and Roberts, 1999]. The EOG amplitude is attenuated approximately with the square of the
distance [Croft and Barry, 2000], and thus contaminates mostly the frontal EEG channels. The EOG activity
spans a wide frequency range, being maximal at frequencies below 4Hz. The EMG artefacts are generated in
the muscles of the face, neck and on the scalp, and are caused by movement, chewing, swallowing, muscle
twitches, anxiety, tremor or general muscle tension [Van de Velde et al, 1998]. The EMG activity also spans a
broad frequency range distribution, and even weak muscle contractions produce an EMG activity that can
mimic or obscure the EEG activity over the entire scalp [Goncharova et al, 2003].
In the CORBYS gait rehabilitation system scenario, the subject will walk with assistance of a robotic device.
In this context, the EEG data will be affected by typical artifacts such as EOG and EMG, but also
electromagnetic artifacts due to the mobile platform and mechanical artifacts associated with head movements
and locomotion. The study of the impact of the electromagnetic noise (i.e. noise due to the DC motors) is
addressed in the section 3.1, Evaluation of EEG acquisition system that could reduce the noise, of this
deliverable.
Several research groups have studied the EEG artifacts during human locomotion. For instance [Gwin et al,
2010] studied the brain activity during walking and running on a treadmill in a controlled scenario. The results
showed that during the walking condition the artifacts slightly contaminate the EEG signals in an eventrelated analysis, while during the running conditions the EEG signals are strongly affected by movement
artifacts.
This deliverable addresses the mechanical artifacts and their removal process based on Independent
Component Analysis (ICA) technique. In the present study two experiments were conducted, the first one
addresses lateral head movement, and the second one studied the walking movement with a ‘walker’ device.
The relevance for the CORBYS project is due to the presence of linear and angular head movement during
locomotion [Hirasaki et al, 1999]; the second one emulates the walking condition of the 1st CORBYS
demonstrator. Although in the clinical EEG the artifacts are addressed with generalist filters, in the brain
52
D3.2 Physical/Physiological sensing devices
Rev. 1.0
computer interface technology community they are addressed with filter design for each user / application.
This is the reason why this study analyzes both average and individual artifacts.
3.2.2
Methods
A. Subjects
Data were collected from 4 healthy subjects, 3 women and one man (23.25 ± 3.59 years). None of them had a
history of a neurological, psychiatric disorder or was under chronic medication. The group included 3 rights
and one left hand-dominant people. The participants were duly informed about the experiment before they
signed the consent form.
B. Data Collection
EEG was recorded using two g.Tec amplifiers with 32 active electrodes. The electrodes were placed at FP1,
FP2, AF3, AF4, F7, F3, Fz, F4, F8, FC3, FC1, FCz, FC2, FC4, T7, C3, C1, Cz, C2, C4, T8, CP3, CPz, CP4,
P7, P3, Pz, P4, P8, O1, Oz and O2 according to the international 10/10 system (Figure 41a). The ground
electrode was placed on FPz and the reference on the left earlobe. EEG signals were sampled at 256 Hz,
bandpass filtered (0.5 – 60 Hz) with a Butterworth filter of order 4 and power notch filtered at 50 Hz.
Vertical and horizontal EOG were acquired with the ground electrode on the right mastoid and the reference
electrode on the left mastoid (Fig.1b). EOG signals were recorded with the gUSBamp amplifier from g.Tec at
a sampling frequency of 256 Hz, bandpass filtered (0.5 – 60 Hz) with a Butterworth filter of order 4 and
power notch filtered at 50 Hz.
Two unipolar EMG electrodes were placed on both left and right sternocleidomastoid muscle as they are
activated with the head rotation movements [Costa et all, 1990] (see Figure 41c). The ground electrode was
placed on the left forearm and the reference electrode was placed on the right forearm. EMG signals were
registered with the gUSBamp amplifier from g.Tec at a sampling frequency of 256 Hz, bandpass filtered (0.5
– 100 Hz) with a Butterworth filter of order 4 and power notch filtered at 50 Hz.
a)
b)
c)
Figure 41 a) Scalp electrode position according to the American EEG Society, 1994, b) subject wearing an EEG cap with 32
active electrodes, c) superficial muscular electrodes (EMG) on the right-sternocleidomastoid muscle.
The data recording was carried out with three g.Tec amplifiers (two of them for the EEG and the other one for
the EMG and EOG) synchronized. The experiment was executed using the proprietary Bit&Brain
Technologies (BBT) software.
53
D3.2 Physical/Physiological sensing devices
Rev. 1.0
C. Experimental Procedure
The experimental procedure comprised a baseline recording and two experiments. It lasted about 30 minutes.
Baseline
Subjects were seated in a comfortable position with eyes open and were asked to try to minimize movements
and blinking for an interval of 5 minutes.
Experiment I
The subjects were in a standing position in front of a fixation cross placed in the center of a wall, 2 meters
away from them (0° - initial position, straight ahead) and four targets located at an angle of ±25° and ±45°
with the 0° (Figure 42a). The experiment consisted of successive trials of four different conditions that
correspond to movements to align the head with the targets (i.e. target -25°, target +25°, target -45° and target
+45°) and return to the initial position. Each trial started with a 3 seconds interval of time where the subjects
relaxed in the initial position and minimized movements and blinking, followed by a variable (depending on
the velocity of the subject head movement) execution interval where the subjects moved the head from the
initial position to the target specified and move it back, and then finished with a 2 seconds rest time where
subjects were allowed to minimally move and blink. The experiment had 10 trials for each condition with a
rest period of 30 seconds between conditions. This experiment lasted approximately 12 minutes.
Experiment II
The subjects were in a standing position with a table on wheels emulating the walking device with the
instrumentation located above (Figure 42b). The reaching point was set to 5 meters away from the start
position. The experiment consisted in trials of one condition where the subjects walked from the starting point
to the reference point. Each trial started with a 3 seconds interval of time where the subjects relaxed, followed
by a variable duration between 10-15 seconds (depending on the velocity of the subject in reaching the
reference point) execution interval where the subjects were walking until the reaching point, and then it
finished with a 5 seconds rest period. The experiment comprised 22 trials and lasted approximately 9 minutes.
54
D3.2 Physical/Physiological sensing devices
a)
Rev. 1.0
b)
Figure 42 a) Experimental setup and protocol of Experiment I: to look at the targets subjects need to turn the head ±25° and
±45°. b) Experimental setup and protocol of Experiment II: subjects were asked to walk pushing ahead a table on wheels
emulating a walking device.
D. Data Processing
Power spectral density (PSD) was obtained using Welch’s method. For each trial the PSD was estimated on
every 1s in the band 1 - 40 Hz with 0,25Hz resolution and then averaged across trials.
For each subject, the EEG data was filtered by Independent Component Analysis (ICA) algorithm (FastICA
based on [Hyvärinen et Oja, 2000]) to eliminate the components from blinking and EMG artifacts. FastICA
algorithm is based on a fixed-point iterative method that maximizes the non-Gaussianity as a measure of
statistical independence [Hyvärinen et Oja, 2000]. The assumption is that the number of sources is the same as
the number of electrodes (i.e 32). Each component was reprojected back to the sensor space, time-frequency
representations of these reprojections with the associated spatial maps were visually inspected to assess
whether the component was artifacted. The components free from artifacts were reprojected back to the sensor
space to obtain an artifact free EEG.
For source localization, Standardised Low Resolution Brain Electromagnetic Tomography (sLORETA)
[Pascual-Marqui, 2002] was employed. sLORETA is a linear method to compute from EEG data a statistical
map that gives the location of neural generators within the brain. The source localization was used to assess
whether the filtered EEG is free of artifacts as follows: the source activity of the pre filtered and post filtered
EEG was computed and checked whatever it corresponds to the motor cortex. Notice that if the artifacts do
not contaminate the EEG, then the motor cortex will be one of the neural generators. This strategy was used
only in the first experiment, as the walking behaviour does not involve the motor cortex in a large extent since
the majority of the behaviour is solved in the spinal cord.
55
D3.2 Physical/Physiological sensing devices
3.2.3
Rev. 1.0
Results
The channels located in central areas were selected for further analysis as they are the location where the legs
movements are better observed [Pfurtscheller and Lopes da Silva, 1999]. The following nine channels were
chosen: FC1, FCz, FC4, C1, Cz, C2, CP3, CPz and CP4.
Experiment I
Figure 43 This figure displays for subject 3 two epochs of EEG signal recorded during relaxation (top), target -25° (middle
left), target +25° (middle right), target -45° (bottom left) and target +45° (bottom right).
Figure 43 presents the EEG signal recorded during relaxation and target ±25° and ±45°, where the EEG is
contaminated by the EMG artifact generated from the head movement. The influence of the artifacts in the
frequency domain is that the power is higher when the angular movement is higher, and this effect is more
prominent at high frequencies (10 – 40 Hz). Figure 44 displays the result for one subject and Figure 45 the
56
D3.2 Physical/Physiological sensing devices
Rev. 1.0
results of the mean of all the subjects. Notice that the spectra shows a peak in the alpha band (8 -13 Hz) that is
largest in the centro-parietal location (i.e. CP3, CPz and CP4) in the not motion state, and this activity
desynchronizes in the motion condition as it is the typical response in a motor behaviour [Pfurtscheller and
Lopes da Silva, 1999].
Figure 44 This figure displays for subject 3 the logarithmic power spectral density of the EEG signal recorded during relaxation
(black), target -25° (green), target +25° (red), target -45° (blue) and target +45° (cyan).
Figure 45 Average logarithmic power spectral density of the EEG signal recorded during relaxation (black), target -25°
(green), target +25° (red), target -45° (blue) and target +45° (cyan).
57
D3.2 Physical/Physiological sensing devices
Rev. 1.0
For each subject, the ICA decomposition was computed on the EEG concatenated of the five conditions (i.e.
0º, targets ±25° and ±45°), where the size of the EEG data satisfies the condition of the minimum amount of
data needed to obtain ICA good performance [Groppe et al, 2009].The ICA algorithm decomposed the EEG
into spatially-fixed and temporally independent components (ICs). The components that contained artifacts
were eliminated and then the remaining components were reprojected back to the sensor space [Jung et al,
1998].
Figure 46 This figure shows for subject 3 scalp maps (right), and the component reprojected back to the sensor space (left) of
the four ICs that contain artifacts: muscular activity, electrode without gel, eye blinking and eye lateral movement.
58
D3.2 Physical/Physiological sensing devices
Rev. 1.0
For one subject the Figure 46 shows several components eliminated and some EEG trials without artifacts
reconstructed through reprojection (Figure 47). Notice how the presence of the main artifacts due to the head
movement is mitigated, although in the last two conditions some EMG is still present.
Figure 47 This figure shows for subject 3 two seconds portion of EEG signal (left), recorded during relaxation, target +25°,
target -25°, target +45° and target -45°. The same EEG free of artifacts (right). The components eliminated are displayed in
Figure 46.
59
D3.2 Physical/Physiological sensing devices
Rev. 1.0
The source localization of each trial was computed by sLORETA. EEG data were converted to cross spectrum
using default frequencies, delta (1.5–6 Hz), theta (6.5–8 Hz), lower alpha (8.5–10 Hz), upper alpha (10.5–12
Hz), lower beta (12.5–18 Hz), beta (18.5–21 Hz), upper beta (21.5–30 Hz) and all bands (1.5–30 Hz). Figure
48 (a)–(d) presents localization results for the raw EEG (left) and EEG filtered (right) recorded during target
+25° (a), target -25° (b), target +45° (c) and target -45° (d) for subject 3. The source localization from the raw
EEG filtered in the upper alpha is in BA 37, 19, 39, 21 for all conditions respectively (see Table 5), where this
neural generators are not apparently related to motor behaviour .Notice that in this case the artifacts act as
noise for the sLORETA which is not able to locate activity on the motor cortex. The source localization from
the EEG free of artifacts and filtered in the upper alpha/mu shows a prominent activity in the Brodmann area
BA4 (primary motor cortex on the precentral gyrus). These results are consistent with previous description of
motor cortex activation associated with motor execution and imagery [Dyson et al, 2010; Hanakawa et al,
2003; Pfurtscheller and Lopes da Silva, 1999]. This result shows that after the filtering, the primary motor
cortex is one of the most relevant sources of the EEG (Table 5).
a
b
c
d
Figure 48 Source localisation results for raw EEG (left) and EEG data filtered (right) recorded during target +25° (a), target 25° (b), target +45° (c) and target -45° (d) for subject 3. EEG filtered shows activation in Brodmann area 4 in the upper
alpha/mu band (10.5–12 Hz).
60
D3.2 Physical/Physiological sensing devices
Brodmann Area (BA) - Best Match
Rev. 1.0
Raw EEG
EEG filtered
Target +25º (a)
BA : 37, 19, 39, 21
BA : 7, 5, 4, 3
Target -25º (b)
BA : 37, 19, 39, 21
BA : 7, 5 , 4, 3
Target +45º (c)
BA : 37, 19 , 39, 21
BA : 7, 5 , 4, 3
Target -45º (d)
BA : 37, 19 , 39, 21
BA : 4, 3 , 5, 6
Table 5 The 4 Brodmann area localized trough sLORETA that show prominent activity in the raw EEG and in the EEG
filtered.
Experiment II
Fig 49 displays the EEG signal recorded during relaxation and walking for subject 4. Notice that the
locomotion creates a movement artifact that is more visible at low frequencies.
Figure 49 2 seconds portion of EEG signal recorded during relaxation (left) and walking (right) for subject 4.
Figure 50 presents the power spectral densities for subject 4 of the EEG signal recorded during walking and
relaxation. The walking condition has a greater PSD at low frequency range (i.e. up to 4 Hz) reflecting the
effect of the low frequency movement as it is observed in Figure 49. This artifact is also present in high
frequency (30 – 40 Hz) and more prominent over central and parietal areas. Notice also that the walking
creates a desynchronization in the alpha/mu rhythms in the central areas and synchronization around beta
bands.
61
D3.2 Physical/Physiological sensing devices
Rev. 1.0
Figure 50 Logarithmic power spectral density of the EEG signal recorded during relaxation (black) and walking (blue) for
subject 4
Figure 51 Average logarithmic power spectral density of the EEG signal recorded during relaxation (black) and walking
(blue).
Figure 51 presents the average (i.e. all 4 subjects of the experiment) logarithmic power spectral density for
EEG signal recorded during walking and relaxation. The spectral landscape is similar for all the channel
locations, peaking at low frequency and decreasing smoothly as the frequency is higher. The spectra show a
62
D3.2 Physical/Physiological sensing devices
Rev. 1.0
peak in the alpha band (8-13 Hz) that is largest in the centro-parietal locations (i.e. CP3, CPz and CP4).
Individual PSD displays sharper spectral peaks than the average one.
As in the previous experiment ICA decomposition was computed on the walking dataset separately for each
subject. The ICA components were selected by visual inspection and depending on the subject, the
components chosen varied between 7 and 8. Figure 52 presents some EEG trials without artifacts
reconstructed through reprojection.
Figure 52 2 seconds portion of raw EEG (left) and EEG filtered (right), recorded during walking for all subjects. The filtered
signal is obtained by removing the selected ICA components identified as artifacts.
63
D3.2 Physical/Physiological sensing devices
3.2.4
Rev. 1.0
Discussion
This deliverable reports on the analysis which describe temporal and spectrual characteristics of the main
contamination introduced in the EEG by locomotion and head movements. These artifacts are likely to be
present in the CORBYS 1st demonstrator. The results indicate that artifactual components can conceal or
mimic EEG alpha and beta rhythms over the entire scalp as previously described for EMG contamination
[Goncharova et al, 2003] and are also relevant at low frequencies.
Independent Component Analysis technique was applied to remove the artifact contamination generated as it
appears to be an effective method for removing artifacts from EEG data [Jung et al, 1998]. The results suggest
the feasibility of the use of this technique for artifact removal in CORBYS. The key issue for a BCI system is
to what extent the artifacts components interferes with its goal. Notice that decoder-specific analysis is
extremely important in order to study the feasibility of removing gait-related artifacts allowing correct
interpretation of cognitive related process. In addition to this work, in the section 3.3, Detection of attention
during assisted passive leg motion, of this deliverable, the ICA method was used to filter eye blinking and
muscular artifacts. Results obtained indicate that ICA does not interfere with the attention decoding process,
on the contrary, it improves its performances.
64
D3.2 Physical/Physiological sensing devices
Rev. 1.0
3.3 Detection of attention during assisted passive leg motion
3.3.1
Introduction
Neurological disorders or brain injuries such as brain stroke or spinal cord injury can cause problems in
patient’s movement, as the Central Nervous System (CNS) or the efferent channels from the CNS are
compromised. Brain Computer Interfaces (BCI) is a technology that addresses this problem by creating a new
channel to bypass the injury and recover the communication between the non affected CNS and the limb
muscles. The principle is to measure the brain activity to extract some meaningful information to move the
limbs by, for instance, functional electrical stimulation (FES) [Pfurtscheller et al. 2005] or by robotic
exoskeletons [Pons 2008].
The electrophysiology of the motor skills is usually characterized by the event related
synchronization/desynchronization (ERS/ERD), which refers to the increase or decrease in synchrony of the
neural population of a determined area of the brain cortex [Pfurtscheller et Lopes da Silva 1999]. This
technique has been used to quantify changes in EEG signal by calculating the increase or decrease of spectral
power during a mental process compared to the brain activity during a reference time or baseline. There are
associated techniques to visualize [Graimann et al. 2002] and compute [Gómez et al. 2012] neural changes in
motor areas of the brain. For instance, the movement of a limb has been characterized by this method and thus
it is known that neural population of the central brain cortex area (motor cortex) desynchronize in alpha and
beta bands (7-13 Hz and 15-30 Hz. respectively).
Rehabilitation programs for patients with these injuries are usually based on the execution of repetitive
movements to regain muscle control or to delay the loss of mobility due to the disease. One limitation in these
rehabilitation therapies is that excessive repetitive movements could lead to a lack of patient engagement,
compromising the adherence to the therapy. In this direction, it is well known that cognitive processes such as
attention mediate in the rehabilitation and play an important role in the success of the therapy [Tee et al.
2008]. The possibility to monitor the patient attention could be a key issue in rehabilitation because it directly
measures the cognitive process and indirectly the adherence to the therapy. To date, very few works have
addressed the characterization of attention during the execution of motor tasks. An fMRI study has shown that
the degree of attention to the motor task modulates brain activity in sensory-motor areas in such a way that
focused attention produces higher activations of the motor brain rhythms [Johansen-Berg et Matthews 2002].
This result has been confirmed in the EEG domain for passive upper limb mobilization [Antelis et al. 2012].
In addition, the last paper shows that it is possible to build an offline classifier to distinguish between two
conditions: passive movement with/without attention to the motor task. The present work builds in this
direction by extending the previous research to the lower limbs.
The CORBYS project proposes a robot-assisted gait rehabilitation system. The objective of this deliverable is
to report the progress developed by Bit&Brain Technologies (BBT) to design a BCI system to decode passive
lower limb movements with attention or non-attention to the motor task. To study the viability of this decoder
an experimental setup was built, where passive leg movements were performed by a mini stationary bike
device while the subject was paying attention to the movement or executing a distractive task from the leg
motion. The study spanned the electrophysiology and the development of an off-line classifier to differentiate
between both conditions.
65
D3.2 Physical/Physiological sensing devices
3.3.2
Rev. 1.0
Methods
A ) Experiment
4 right-footed and one left-footed healthy subjects participated in the experiment aged 20 to 28 years (mean
24.4 and standard deviation 3.28). They all signed an informed consent. Volunteers were seated in a
comfortable chair in front of a computer screen. Their feet were held to the pedals of a mini stationary bike
that is motor assisted to actively move the user’s legs.
a)
b)
Figure 53 a) Mini-bike and b) Experimental setup
The subjects were seated in a comfortable position, far enough from the mini stationary bike to allow a
smooth passive movement of the legs and to avoid possible contact between the feet and the floor or furniture.
The feet where held to the pedals by an elastic strip. Subjects were instructed to completely relax the limbs
and to avoid voluntary leg movements during the experiment. The mini-bike and the experimental setup are
displayed in Figure 53.
The experiment consisted of trials of two conditions (with attention or distraction to the motor task). In the
first condition, subjects focused their attention to the legs during the execution time, while in the second one
the subjects were instructed to perform mental algebraic computations as a distractor to the passive leg
movement [Johansen-Berg et Matthews 2002, Antelis et al. 2012]. The algebraic computations were
subtractions starting from a 3-digits voluntarily selected number and subtracting a one-digit figure voluntarily
selected. Each trial started with a 5 seconds interval of time where the subjects relaxed minimizing
movements and blinking while they were informed of the condition (attention or distraction) they had to
66
D3.2 Physical/Physiological sensing devices
Rev. 1.0
perform (the bike is stopped); followed by a 5 seconds execution time where the subject was mentally
performing the task (bike is on); and it finished with a 5 seconds rest time where subjects were allowed to
perform minimal movements and blinking (bike is stopped). Figure 54 shows the structure of one trial. The
experiment consisted in 3 series of 30 trials, with 15 trials of each condition, presented in random order. There
was a rest period of 1 minute between series. The experiment lasted 25 minutes.
Figure 54. Example of a trial (darker grey: data processed)
B ) Data recording and mini-bike
1) EEG system: EEG data was recorded by a TMSi amplifier with 16 electrodes according to the 10/10 system
(FC3, FC1, FCz, FC2, FC4, C3, C1, Cz, C2, C4, CP3, CP1, CPz, CP2, CP4, Pz). Ground was located in FPz
and the reference on the right earlobe. Notice that the large majority of electrodes were situated close to Cz as
it is usually the sensor where the leg motor cortical activation is observed [Graimann et Pfurtscheller 2006].
The EEG signal was acquired with a sampling rate of 256 Hz, power-line notch-filtered (50 or 60 Hz) and
bandpass-filtered from 0.5 Hz to 60 Hz. The acquisition and experimental software was property of Bit&Brain
Technologies.
2) Mini-bike: A mini stationary bike (YF612 Tecnovita by BH) was used to move the user’s legs. The angular
velocity of the pedals was ω ~ 2π rad/s. The activation of the mini-bike was manually controlled by the
supervisor of the experiment.
C ) Data processing
For each subject, the EEG data was filtered by an Independent Component Analysis (ICA) algorithm (FastICA
based on [Hyvärinen et Oja 2000]) to eliminate the components from blinking, EMG artifacts and electrical
noise from the mini-bike. Each component was reprojected back to the sensor space, and time and timefrequency representations of these reprojections with the associated spatial filter were visually inspected to
assess whether the component was artifacted. The components free from artifacts were reprojected back to the
sensor space to obtain an artifact free EEG. This EEG was filtered and then, for each trial, the [-3s,3s] interval
with respect to the mini-bike onset was extracted (see Figure 54). These new trials were again visually
67
D3.2 Physical/Physiological sensing devices
Rev. 1.0
inspected to assure they are artifact-free. Then, for each trial, the across-trials average was subtracted for each
condition in order to remove the evoked potential caused by either the beep sound from the mini-bike device
when it was activated and/or the sudden start of the motion.
The ERD/ERS during the motion process was obtained by (Pj-Ppas)/Pbas*100, where Pj is the power of each
time-frequency bin of the j-trial from the onset of the mini-bike (0s) to +3s and Pbas means power of the
baseline (-3s to 0s). Power spectra were computed by the Welch method with a sliding window of 128
samples with 120 samples of overlapping between consecutive windows. ERD/ERS is represented as a timefrequency plot where time varies from -3s to +3s and frequency from 0 Hz to 50 Hz. Significant ERD/ERS
was computed by a bootstrap algorithm with a significance level of 0.05 [Graimann et Pfurtscheller 2006]
(Figure 55).
Figure 55 Example of bootstrap algorithm in a ERD/ERS time frequency map
D) Feature extraction and classification
A classifier was built to detect each condition (i.e. if the subject is paying attention or not to the leg movement
task). The features extracted were the spectral power of channels (with significant desynchronization) located
in the motor cortex area in alpha and beta bands (individually selected by visual inspection of the ERD/ERS
from the most discriminant electrode). Welch method was used to compute the spectral power with a sliding
window of 128 samples and a 120 samples overlapping. Features were concatenated, z-score normalized and
used to train a linear discriminant analysis (LDA) classifier. A 10-fold cross validation procedure was
performed to assess the generalization properties of the classifier.
3.3.3
Results
Figure 56 displays the ERD/ERS maps for the 5 subjects that participated in the experiment. Subjects 3, 4 and
5 showed a higher desynchronization during attention to the motion than in the other condition. This
68
D3.2 Physical/Physiological sensing devices
Rev. 1.0
desynchronization was more prominent in both bands (alpha and beta) for subjects 3 and 5, and in the beta
band for subject 4. For Subject 2 the results are the contrary (i.e. higher activity in the distraction task) which
could be due to the way that this subject performed the mental task. For subject 1 is difficult to visually
identify the differences as it seems that the desynchronization in alpha is lower but the beta is higher in the
attention condition than in the other, so more elaborated tools to quantify this desynchronization might need
to be applied [Gómez et al. 2012]. In summary, in all the cases individually is possible to visually detect
differences between the two mental states.
Figure 56 ERD/ERS maps for the five subjects from top row (Subject 1) to bottom row (Subject 5) for both conditions. First
column: Passive Motion + Attention. Second Column: Passive Motion + Distraction. Black dashed line: individualized alpha
band. Red dashed line: individualized beta band.
A Linear Discriminant Analysis (LDA) classifier was used to differentiate between both conditions. Three
69
D3.2 Physical/Physiological sensing devices
Rev. 1.0
different sets of features were used to train our classifier. In the first one only power in alpha band was used,
in the second one only beta band power, and in the third one both previous features were used together. Bands
and channels were visually and manually selected for each subject to calculate the features. The final results
are shown in Table 6.
Subject
Accuracy alpha
Accuracy beta
Accuracy alpha+beta
1
75.00
63.33
81.67
2
62.86
82.86
80.00
3
57.14
70.00
64.29
4
61.67
70.00
66.67
76.67
75.00
81.67
5
Table 6 Results for each band of each subject
Figure 57 Results for each band of each subject (dotted line = chance level according to [Müller-Putz et al. 2008] (p<0.05))
As shown in Table 6, the classifier performance for all subjects is over the chance level which is situated in
62.5 % according to [Müller-Putz et al. 2008]. Beta band features achieved higher accuracy for subjects 2, 3
and 4 while for subjects 1 and 5 the combination of both bands features was the better election. The mean
accuracy is 77.24% (selecting the best result of each subject), showing that it is possible to build a classifier
over the chance level and thus showing the feasibility of building an off-line system to differentiate the two
conditions.
3.3.4
Discussion
70
D3.2 Physical/Physiological sensing devices
Rev. 1.0
Cognitive processes such as attention affect rehabilitation exercises and they play a key role in the result of
the therapy. Presented work shows that the attention modulates brain activity during passive leg movements as
observed in the EEG signal. An off-line classifier was developed to differentiate between two states (attention
or distraction to the motor task). The results show the feasibility to distinguish between both conditions above
the chance level for all the subjects. This is the first step in the development of the attention decoding unit of
CORBYS. This research could be improved in several directions. For instance, source localization techniques
will be needed to confirm that the EEG changes observed between conditions have the neural origin in the
motor cortex (as demonstrated by fMRI studies [Johansen-Berg et P. Matthews 2002]). In addition to this, the
subjects had to focus in the screen but their legs were not concealed from his or her gaze, which could have an
influence on the motor activity. Although this aspect influences both conditions, further investigation is
required. Future work will consider the movement artifacts (analysed in the section 3.2, Analysis and removal
of movements’ artifacts during locomotion, of this deliverable) and the automation of feature selection
process. Eventually, notice that the final next step is to build an online classifier to assess the feasibility of
decoding attention during motion in real time which is the user scenario in the CORBYS project.
71
D3.2 Physical/Physiological sensing devices
Rev. 1.0
4 Safety
Work with safety aspects of the CORBYS demonstrator I and of all its components is started and will be
documented in documents that will accompany the deliverable on the final demonstrator D7.4 (M36),
covering safety standards and procedures. In the following safety aspects of HSS and BCI module are given
together with FMEA risk analysis
4.1 Chest unit
The chest unit is a battery powered device using wireless communication. This ensures patient safety with
respect to electrical shock.
The CORBYS Chest Unit uses conductive rubber electrodes for ECG measurement. These gives signal quality
comparable to the medical electrodes, also when the patient is very active. A disadvantage is possible skin
irritation when sweat collects under the rubber and this may cause some skin irritation during prolonged use.
However CORBYS training sessions will be of limited duration. The sensors are integrated in the chest unit,
and automatically placed correctly when the patient puts on the device.
The sensor belt will attract sweat and dirt after use and should be rinsed in lukewarm water (no soap) once a
week, and then left to air-dry. However one must remember to detach both the chest unit and the IMU-unit
from the belt before washing as the units are not water-resistant. The units can be wiped with a soft damp
cloth and towel-dried.
4.2 IMU unit
The IMU unit is a battery powered device using wireless communication. This ensures patient safety with
respect to electrical shock. The unit has no electrical contact with the patient.
4.3 EMG units
The EMG sensors will be covered in D3.4
4.4 HSS controller computer
The HSS Controller will be located on the mobile platform for CORBYS demonstrator I. The controller is
powered from the mobile platform batteries (24VDC). It will not be physically connected to the patient.
Electrical safety insulation towards the grid power supply (220VAC) is handled by the Central Power System
on the mobile platform.
4.5 Offline charger
The sensor unit shall not be charged while attached to the patient. This means that the offline charger will not
be physically connected to the patient. The charger will be powered from the HSS controller or by use of a
separate power supply.
72
D3.2 Physical/Physiological sensing devices
Rev. 1.0
4.6 Safety analysis:
As part of the safety work for demonstrator I, FMEA safety analysis has been performed for all components of
the system as reported in Deliverable 7.1. The analysis for the HSS is shown in the figure below. It shows that
there are no high risks connected to the HSS. The medium risks will be addressed at the system level risk
analysis and handled by other modules. Incomplete datasets due to failing or missing sensors must be handled
by receivers of data from HSS, all sensor data shall be defined with a valid range of values. To avoid
incorrect data due to incorrect sensor location there will be a therapist procedure to validate sensor data before
each training session.
73
D3.2 Physical/Physiological sensing devices
Pos.
1
2
FUNCTION
Human
Sensory
System
Controller
Human
Sensory
System
Wireless
Body
Sensors
Rev. 1.0
POTENTIAL
failure MODE
connection loss to
GPN
POTENTIAL
CAUSES
Network error
POTENTIAL
EFFECTS
no sensor data
to cognitive
modules
DETECTION
METHOD
ROS heartbeat
failure / time lag
runtime error for
HSS controller
hardware
Timestamped
sensor data.
Timestamps are
synchronized by
ROS framework.
7
5
failure / time lag
Configuration
error
delay on sensor
data to cognitive
modules, may
lead to incorrect
cognitive
recognition
missing or delay
on sensor data
to cognitive
modules, may
lead to incorrect
cognitive
recognition
Timestamp on
sensor data/
cognitive
modules detects
missing data.
Timestamps are
synchronized by
ROS framework.
7
measurement
failure
Sensors pick up
external noise
incorrect sensor
data to cognitive
modules
HSS controller
detects out of
range sensor
data
7
SEV
7
OCC DET
5
1
RPN
35
Recommended Action(s)
FS must check HSS heart
beat and handle any
detected errors according to
FS specification.
3
105
Cognitive modules must
check timestamps.
7
5
2
70
5
3
105
Cognitive modules must
check timestamps. Cognitive
modules should handle
incomplete datasets. Data
from HSS to be verified in
pre-session procedures.
7
5
2
70
4
6
168
measurement
failure
Defect sensor,
incorrect sensor
connection
no sensor data
to cognitive
modules
HSS controller
detects
inoperable
missing sensor
data
7
3
1
21
measurement
failure
Defect sensor or
incorrect sensor
location during
setup
incorrect sensor
data to cognitive
modules
Therapist detect
in pre-training
session
procedure
7
3
2
42
74
SEV OCC
DET RPN
0
0
Cognitive modules should
validate sensor data - to be
discussed with cognitive
partners
HSS controller reports errors
to FS, which handle detected
errors according to FS
specification. Cognitive
modules should handle
incomplete datasets.
0
0
Therapist procedure to be
developed, also included in
safety document
D3.2 Physical/Physiological sensing devices
measurement
failure
Rev. 1.0
Defect sensor or
incorrect sensor
location training
session
incorrect sensor
data to cognitive
modules
measurement
failure
Sensor module
run out of power
no sensor data
to cognitive
modules
biocompatibility
irritations/
inflamatic
responses or pain
patient
Patient
uncomfortable or response and
unable to wear
inspection
sensor
7
HSS controller
monitor
remaining
battery capacity
7
6
1
42
6
5
2
60
Table 7 Safety analysis for HSS
75
3
6
126
HSS controller
detects out of
range sensor
data
0
Cognitive modules should
validate sensor data - to be
discussed with cognitive
partners
Status on sensor modules is
sent via ROS heartbeat to
FS. FS must handle any
detected errors according to
FS specification. Cognitive
modules should handle
incomplete datasets.
Patient inspection should be
added to post training
session procedure.
0
0
D3.2 Physical/Physiological sensing devices
Rev. 1.0
The safety design of the Brain Computer Interface (BCI) submodule has two different levels of analysis:
hardware and software:
4.7 EEG unit
The amplifier Porti System – TMSi of the BCI complies with the following safety requirements:
•
•
CE0044: meets all the requirements of the MDD (93/42/EEC) MDD Classification IIa (rule 10).
Applied
standards
-
•
:
IEC 60601-1:1988 + A1:1991 + A2:1995 : Medical electrical equipment - Part 1: General
requirements for safety
IEC 60601-1-2:2001 : Medical electrical equipment - Part 1-2: Electromagnetic compatibility
Safety class (IEC 60601-1): Internally or externally powered, type CF
4.8 BCI Software
The Software of the BCI submodule provides the following output regarding the state of the BCI sensors.
1. EEG sensor failure : information about the actual state of the EEG sensor indicating if it is working
properly or not (e.g. it can happen that an electrode gets disconnected or broken during the
rehabilitation session causing wrong recordings).
The EEG sensor failure will allow then to have a complete overview on the BCI submodule status.
4.9 BCI Safety analysis:
The table below presents the risk analysis for the BCI submodule. Medium and high risks are related to EEG
signal contamination (i.e. patient movement noise and physiological noise); section 3.1, Analysis, evaluation
and removal of movement artifacts from EEG measurement during locomotion, of this deliverable addresses
the mechanical artefacts and their removal process. Removal algorithms mitigate the effect of the artifacts on
the EEG signal, where the detection value in the table indicates how much the filter can reduce this
contamination. This value will change since it is strictly dependent on the decoding algorithm implementation
(actually under study).
76
D3.2 Physical/Physiological sensing devices
Pos.
1
2
POTENTIAL
FUNCTION
failure MODE
BCI software GPN connection
loss
BCI sensors
POTENTIAL CAUSES
network error
Rev. 1.0
POTENTIAL
EFFECTS
no sensor data
to other
module/s
DETECTION
METHOD
ROS heartbeat
SEV
8
OCC DET
4
1
RPN
32
data package
loss
GPN network error
delay on
sensor data
Data timestamp
7
4
1
28
measurement
failure
incorrect sensor setup
no sensor data
to other
module/s
BCI software
detects
incorrect
sensor setup
7
5
2
70
measurement
failure
incorrect sensor
connection
no sensor data
to other
module/s
BCI software
detects
incorrect
sensor
connection
7
2
1
14
measurement
failure
faulty sensor
no sensor data
to other
module/s
BCI software
detects faulty
sensor
7
3
3
63
measurement
failure
sensor module out of
power
no sensor data
BCI software
detects when
the sensor
module is out
of power
8
3
1
24
77
Recommended
Action(s)
Check Timestamp
SEV OCC
7
4
DET RPN
1
28
D3.2 Physical/Physiological sensing devices
Rev. 1.0
measurement
failure
instrumental/environmental incorrect
noise
sensor data to
other module/s
Signal
processing
algorithm of the
BCI software
mitigate the
potential effects
7
3
3
63
measurement
failure
patient movement noise
incorrect
sensor data to
other module/s
Signal
processing
algorithm of the
BCI software
mitigate the
potential effects
7
8
5
280
measurement
failure
physiological noise
incorrect
sensor data to
other module/s
Signal
processing
algorithm of the
BCI software
mitigate the
potential effects
7
9
3
189
Table 8 Safety analysis for BCI
78
D3.2 Physical/Physiological sensing devices
Rev. 1.0
5 Requirements from D2.1
This section lists the relevant requirements for Human Sensory System and Brain Computer Interface from
D2.1 and how these have been fulfilled or implemented.
Req. #
Name
Description
Realization
HSS1
Sensors implemented in the
CORBYS system
The actual sensors selected
will be defined in the detailed
specification
HSS2
Sensor locations
The actual sensors selected
will be defined in the detailed
specification.
HSS3
Patient user size
HSS4
Sensor output of primary
parameter values
Adult users. Height, weight
circumferences will be
discussed with clinical
partners
Definition on how human
sensor parameters are shared
with the rest of the CORBYS
system, as well as in export to
Implemented:
Heart rate and ECG
Skin temperature at chest and back
Humidity sensors at patient back
Inertial measurement units (3 axis
accelerometer, gyroscope and
magnetometer) at patient chest and
back
To be implemented in D3.4
EMG (muscular activity)
Mechanical sensing (force, torque,
angular joint movements,
force/pressure distribution) will be
implemented by sensors in
Demonstrator I in WP7.
Implemented sensor and
functionality:
Homing positions of inertial
sensors is handled at session startup by therapist.
A sensor for measuring
environment temperature and
humidity is developed and will be
attached on mobile platform.
Requirements that will be handled
by mechanical sensors in
Demonstrator I in WP7:
Positions of robot mechanical
support to the patient (such as limb
fixation) and patient movement
actuators.
HSS will fit all patients relevant
for CORBYS usage.
79
Implemented:
Sharing is done through ROS
topics, details are described in
HSS Controller section.
D3.2 Physical/Physiological sensing devices
Rev. 1.0
log files: Details are to be
defined.
HSS5
Safety-related sensor output
– information derived from
sensor fusion of multiple
sensors
HSS6
Sensor output related to
physical effort assessments
HSS7
Sensor output related to
gait parameter assessments
Sensor output related to
identification of psycophysiological states
Sensor output related to
identification of intentional
states
HSS8
HSS9
Status information or flags
should be raised if sensor
readings indicate a potentially
hazardous situation. To be
discussed with partners with
stakes in the design of
CORBYS control system
HSS will provide human
sensor data related to physical
effort, which is heart rate,
ECG, humidity, temperature,
EMG and movement data.
Same data provided as listed
in HSS6
Same data provided as listed
in HSS6
Same data provided as listed
in HSS6
80
Logging to file is used in early
integration phases, will also be
available as ROS bags. See HSS
controller sections.
Project has not identified need for
sensor fusion from HSS for Safety.
Handling of potentially hazardous
situations will be discussed in
D7.4.
Data will be processed by
SAWBB (to be described in D4.3)
Initial discussions have taken
place between SINTEF and UR on
input requirements for the
Situation Assessment architecture,
with respect to physiological
sensor output. Based on these
requirements, sampling rate,
timestamp resolution etc. have
been revised for heart rate, ECG.
The revised ESUMS HSS dataset
from SINTEF was received by
UR, and is currently being utilised
as part of effort undertaken in
WP4. Similarly, activity sensing
output (IMU data) provided as
input to the Situation Assessment
architecture; UR is making use of
revised HSS ESUMS dataset, in
which gyroscope was included.
Also SOIAA will make use of this
data in the adaptive walking mode
that will be presented in D5.2 and
D5.3
Same as HSS6
Same as HSS6
This will be handled in D5.3 for
adaptive walking mode. This is
where SOIAA provides gait
trajectories from BCI intention
data, HSS sensor data, which
facilitate the patient and orthosis
being able to walk freely in a
structured environment.
D3.2 Physical/Physiological sensing devices
HSS10
Data processing and signal
analysis requirements
HSS11
Integrating physiological
sensor measurements
(SINTEF) with BCI (BBT)
HSS12
Interfacing physiological
sensor measurement system
with the main CORBYS
cognitive robot control
system
Mains requirements
HSS13
HSS14
Sensor network architecture
requirements
HSS15
Online access to past
rehabilitation sessions
HSS16
CORBYS system
intermittence, delay and
synchronisation
requirements for sensors
and actuators
HSS17
Number of physiological
sensor probes on the
patients
Rev. 1.0
The project needs to sum up
all the data processing
requirements, both capacity
and platform wise.
Discussions between BBT and
SINTEF regarding finding a
shared platform for integrating
sensor signals.
HSS controller will be running on
separate computer with sufficient
processing power for HSS tasks.
All sensor measurements are
presented to cognitive modules as
individual ROS topics. BCI and
HSS sensor data will be provided
for cognitive modules with
consistent timings allowing for
various latencies, to provide a full
picture of the patient over time.
Wireless sensors are connected to
General Purpose Network (GPN)
for interfacing cognitive modules
and control.
It must be anticipated that
some of the measurement
equipment will require
220V/50Hz
The project needs to compile a
summary of all sensors and
actuators (with detailed
operation characteristics and
worst-case values) in order to
specify the total sensor
network architecture.
Online access to past (and
possibly ongoing) therapy
sessions implies a software
architecture solution, as well
as probably a WiFi node on
the CORBYS system
A shared understanding of
signal propagation will have
to be reached between the
partners.
The detailed number is TBD.
No limitations are stated at
this stage.
81
HSS Controller will be running at
24VDC
Worst case values will be
specified as part of the GPN
specifications in D3.4.
Human sensory system values can
be recorded at the GPN and will
be described in D3.4
All sensor data will be time
stamped. D3.4 will discuss
synchronization of computers on
GPN.
HSS sensor data delays are
discussed in section 2.1.2
Implemented:
There will be one Chest unit at
front and one IMU sensor unit at
the back of the patient.
One IMU sensor unit will be
located at the mobile platform to
measure environment humidity
D3.2 Physical/Physiological sensing devices
HSS18
Number of physiological
sensor probes on the
patients
HSS19
Signal connection of
physiological sensors to the
CORBYS system
HSS20
Signal connection of
physiological sensors to the
CORBYS system
HSS21
Intended duration of
continuous usage of
physiological sensors
HSS22
Intended duration of
continuous usage of
physiological sensors
Intended duration of
continuous usage of
physiological sensors
HSS23
HSS24
Electrical measurement
system safety
HSS25
Electrical measurement
system safety
Rev. 1.0
For ease-of-use purposes, it
will be desirable to combine
several sensors into single
devices, thereby reducing the
experienced system
complexity
Sensor data signals will be
sent through electrical
wires/cables
For ease of use purposes,
possibilities to make some
sensor units transmit data
using wireless communication
protocols will be considered
Anticipation: A therapy
session will last up to 2 hours
Anticipation: 8 hour sessions
If CORBYS becomes a
“community walker” gait
assistance system, usage
sessions could last from
morning to evening.
Within Consortium electronic
systems are acceptable as long
as they are tested and deemed
safe for the CORBYS users
(e.g. complete user shielding
from 220V/50Hz).
CE Medical device standard
electrical safety
82
and temperature
Postponed:
Four EMG sensors located at each
patient leg.
See HSS17
Implemented
Physiological sensors are
transmitted using Bluetooth to
HSS controller.
HSS controller use wired Ethernet
for communication over GPN,
See HSS19
The chest unit and IMU sensor
unit will have battery capacity for
more than 2 hours. Final battery
capacity measurements will be
provided in D3.4.
Two sets of units will follow the
demonstrator. One set for use and
another charging. Charging time is
less than 2 hours.
There will be one charging station
following the demonstrator.
See HSS21
See HSS21
This requirement will be handled
in the safety discussion in D7.4
See HSS24
D3.2 Physical/Physiological sensing devices
HSS26
Electrical measurement
system safety
HSS27
Sensor systems should not
be invasive or excessively
obtrusive
HSS28
Limitations in the range of
acceptable users
HSS29
Mounting and removal
sensors on the patient
HSS30
Mounting of individual
sensor components directly
on the user’s skin
HSS31
Mounting of individual
sensor components directly
on the user’s skin
Placement of sensor
components on the patient
HSS32
Rev. 1.0
Medical device CE approvals
on all sensor components (For
a commercial product after
CORBYS)
In vivo (implanted) sensor
systems are not a part of the
CORBYS physiological
measurement system
Sensor concepts probing
human fluidic samples (blood,
urine, saliva etc.) are excluded
Sensor concepts probing
human body openings (such
as rectal core temperature
measurements and breath air
gas analysis) are excluded
Based on user safety concerns,
the physiological
measurements system might
not be used on patient groups
such as:
•
Patients with
electronic implants
•
Patients with certain
dermatologic conditions
•
Patients with
limitations in cognitive
capabilities
•
Others - to be
decided
The physiological sensors will
be mounted and removed by
trained clinical rehabilitation
professionals
Certain physiological sensors
(for example electrode based)
can be placed at optimum
measurement locations,
directly on the skin of the
patient.
Less optimal, but more user
friendly locations can be used.
See HSS24
All sensor components should
be clearly marked in order to
reduce the risk of placing
sensors in incorrect
measurement positions (e.g.
See HSS29
83
Fulfilled in HSS designed.
HSS should not be used by patient
with electronic implants.
Usage scenario and sensor
mounting will be described as part
D7.4
See HSS29
See HSS29
D3.2 Physical/Physiological sensing devices
HSS33
Placement of sensor
components on the patient
HSS34
Physiological sensor
biocompatibility issues
HSS35
Physiological sensor
biocompatibility issues
HSS36
Physiological sensor
hygienic issues
HSS37
Time required to mount or
dismount all physiological
sensors
CCM10
Connection to sensing
network sub-system.
SIREF2
Sub-systems to be
integrated must be
accompanied by sufficient
mix left and right)
Preferably automated
detection mechanisms to
avoid the risk of incorrect
placement
Sensors should not cause
irritations/inflamatic
responses or pain during the
designated duration of
CORBYS rehabilitation
sessions.
Sensors can be temporarily
attached to the patient using
e.g. medical grade adhesive
tape
EC Medical device standard
biocompatibility testing of all
materials interfacing the
patient
Physiological sensor
interfacing the patient’s skin
directly should be possible to
clean or replace from patient
to patient:
Single use probes
Multiple use probes that have
smooth surfaces and that can
be cleaned in appropriate
detergents
For a trained user, it should be
possible to mount all sensors
within the maximum time
required for the entire start-up
and shut-down times targeted
for the entire CORBYS
system. Time allocated for the
physiological sensor system
alone is TBD.
Sensing network sub-system
should provide pre-processed
sensor data for cognitive
modules in appropriate time
rate.
The documentation required
in order to integrate subsystem components into a
84
Rev. 1.0
Therapist GUI shall be used for
sanity check of sensor data at
session startup and during training
sessions. This will be described as
part of D7.4.
The Chest belt has been tested
with patients and no
biocompatibility issues has been
observed.
EMG sensors will be covered in
D3.4
This is an optional requirement
and no testing towards EC
Medical device standard
biocompatibility has been done.
However informal testing has been
dine, see HSS34.
Cleaning and replacement of Chest
Belt will be described in D7.4.
Chest unit and IMU sensor unit
should be possible to mount within
one minute.
EMG mounting time will be given
in D3.4
ROS is used for interfacing HSS
and cognitive modules. ROS
topics are described in the HSS
Controller sections.
This document together with D3.4,
CORBYS Users manual and
source code will be sufficient
D3.2 Physical/Physiological sensing devices
documentation
CTREF
7
Conformance testing test
protocol design
Rev. 1.0
complete system, such as:
Functional specification
Mechanical design drawings
User manual
Source code
Interface definitions
Installation guidelines
Test plans shall have unique
definition of test objects
(physical components shall be
uniquely marked and software
shall have correct version
numbering). It shall further
contain information about test
site, test date and test
personnel
The test protocol when
feasible will be designed with
the following information for
each test item:
Unique test item number
Description of test activity
Description of expected test
result (which should be in
accordance with target
specifications)
Check box field for entering
test result with the following
alternatives: Passed/Failed
Field for entering test
observation (in particular
observations when the
“Failed” box was checked.
documentation for sub-system
integration.
HW replaceable components will
be marked with serial number.
SW replaceable components will
be marked with ID and revision.
Req. #
Name
Description
Realization
BCISW1
BCI communication
interface
External interface that communicates with
other subsystems using a TCP/IP messages
protocol.
To be addressed in WP6 (i.e.
ROS)
BCISW2
Therapist GUI
The graphical user interface (GUI) allows the
therapist to interact with the BCI software.
BCISW3
Subject GUI / User GUI
In the training and decoding process subjects
are asked to perform some tasks.
To be addressed in WP6
and WP7 (i.e. User interface
design and implementation,
Demonstrator development)
It has been already
addressed in D2.2.
Further analysis will be
85
D3.2 Physical/Physiological sensing devices
BCISW4
BCI software portability
BCISW5
EEG sensor cap size
BCISW6
EEG Electrodes location
and number
Rev. 1.0
The BCI software can run over different
operating systems (Windows, Linux, etc.)
The cap is available in 3 sizes (small, medium
and large); the most appropriate one needs to
be chosen depending on the subject head
circumference. Anyway the medium-sized cap
is suitable for over 95% of all adult subjects.
The EEG electrodes are inserted via small
holes in the cap. Their position on the scalp,
indicated on the cap according to the extended
international 10/20 system, and number
depends on what brain areas are activated
during a specific cognitive task. Ongoing
CORBYS research will identify those (Tasks
3.3 and 3.5).
The graphical user interface (GUI) displays
commands to the subject (e.g. a visual cue
indicating that the subject has to start
walking).
BCISW7
Subject screen / User
screen
BCISW8
Therapist screen
The graphical user interface (GUI) allows the
therapist to interact with the BCI software.
BCISW9
BCI processing unit
The minimum computing power needed
depend on the results of the ongoing
CORBYS research (e.g. laptop, netbook,
personal digital assistant, etc.).
BCISW10
EEG system montage
A fast and easy EEG system montage, cap and
electrodes placement is required. Associated
with these requirements the most appropriate
system will be used.
BCISW11
EEG system portability
A reduced size and weight EEG system is
required. Associated with these requirements
the most appropriate system will be used.
86
addressed in WP6 and WP7
(i.e. User interface design
and implementation,
Demonstrator development)
To be addressed in Task
3.4/Deliverable D3.4
Requirement that needs
always to be accomplished
since the quality of the
signal depend on it.
In the present deliverable it
was accomplished.
In this deliverable different
EEG setup has been used
depending on the task to
focus on. This issue will be
addressed in detail during
the development of Task 3.3
and 3.5/ Deliverable D3.4
To be addressed in WP7
(i.e. Demonstrator
development )
To be addressed in WP6
and WP7 (i.e. User interface
design and implementation,
Demonstrator development)
To be addressed in Task
3.4/Deliverable D3.4
The choice of the CORBYS
EEG hardware has been
addressed in the current
deliverable (section 3.1).
The EEG system chose is
water-based; log time and
uncomfortable preparation
issues have been solved.
The choice of the CORBYS
EEG hardware has been
addressed in the current
deliverable (section 3.1).
Even if the priority in the
EEG hardware selection has
been given to the
performance of the EEG
D3.2 Physical/Physiological sensing devices
BCI1
Execution Mode
Rev. 1.0
Indicates which operation between training
and decoding is going to be used.
BCI requires a machine free training stage
before users can work the technology. The
training process modifies some internal
parameters that successively the decoding
process uses. This procedure must be observed
for each cognitive-related task is planned to be
used.
A training phase is also needed for the
artefacts removal processing (Training
artefacts).
The possible values of the execution mode
input are
- Training motion *
- Training feedback
- Training attention
- Training artefacts
- Decoding motion
- Decoding feedback
- Decoding attention
- Decoding motion & feedback
- Decoding motion & attention
- Decoding feedback & attention
- Decoding motion & feedback & attention
- Stop
The independence of the cognitive-related
tasks allows their simultaneous execution in
the decoding process (i.e. Decoding motion &
feedback, Decoding attention & feedback,
etc.).
A stop input value has been also added to
87
system in a noisy scenario,
this requirement has been
accomplished.
Signal processing/Decoding
part of the requirement will
be addressed in Task 3.3 and
3.5/Deliverable D3.3;
Integration in the BCI
architecture will be
addressed in Task
3.4/Deliverable D3.4
D3.2 Physical/Physiological sensing devices
Rev. 1.0
allow the interruption of the running
processes.
*Motion, feedback and attention are the
abbreviations for intention of legs motion,
feedback error-related potential and attention
states respectively.
BCI2
Configuration File
Values of the optional parameters, a default
setting is provided
To be addressed in Task
3.4/Deliverable D3.4
A list of possible optional parameters is
available below:
- Number of electrodes to be used.
- Sampling rate of the EEG signal.
- Decoders parameters.
BCI3
BCI4
Raw EEG
Filtered EEG
A complete list will be provided depending on
the results of the ongoing CORBYS research.
Electroencephalographic signal acquired by
the BCI hardware
Electroencephalographic signal filtered from
occurring artefacts
Integration in the BCI
architecture will be
addressed in Task
3.4/Deliverable D3.4
Integration in the CORBYS
system will be addressed in
WP6.
Signal processing/Decoding
part of the requirement has
been partially addressed in
the current deliverable (i.e.
section 3.1 and 3.2) where
an analysis of EEG hardware
and EEG artefacts has been
performed.
Further analysis will be
addressed in Task 3.3 and
3.5 / Deliverable D3.3
Integration in the BCI
architecture will be
addressed in Task 3.4 /
Deliverable D3.4
Integration in the CORBYS
system will be addressed in
WP6.
BCI5
Intention of legs motion
decoding flag
Information about which leg the subject is
going to move. It also provides a “no
88
Signal processing/Decoding
part of the requirement will
D3.2 Physical/Physiological sensing devices
Rev. 1.0
movement” output value. The intention of legs
motion decoding flag output can take the
following values: right leg, left leg and no
movement.
be addressed in Task
3.3/Deliverable D3.3;
Integration in the BCI
architecture will be
addressed in Task
3.4/Deliverable D3.4
BCI6
Decoding accuracy
It provides a numerical value (e.g. a
percentage) related to the ability of the BCI
subsystem in detecting the intention of legs
motion
Refer to BCI5
BCI7
Error marker
Feedback stimulus that informs the subject
about the correctness of their response to a
specific task.
Signal processing/Decoding
Refer to BCI8
The feedback stimulus presented after the
accomplishment of a task, informs the subject
about the correctness of his response and
therefore provide the critical information that
would enable the error detection. Feedback
stimulus, provided by other subsystems, can
be auditory, visual, somatosensory, etc.
The error marker input is a time marker that
informs the BCI subsystem when the feedback
is presented to the subject.
BCI8
Feedback error-related
potential decoding flag
Information about the presence of the
feedback error-related potential in the brain
signal
The feedback error-related potential decoding
flag output can take the following values:
present and absent depending on the presence
of the feedback error-related potential in the
brain signal.
Signal processing/Decoding
part of the requirement will
be addressed in Task
3.5/Deliverable D3.3;
Integration in the BCI
architecture will be
addressed in Task
3.4/Deliverable D3.4
BCI9
Decoding accuracy
It provides a numerical value (e.g. a
percentage) related to the ability of the BCI
subsystem in detecting the feedback errorrelated potential.
Refer to BCI8
BCI10
Attention states decoding
flag
Information about the subject’s attention level
during a specific task
Progress report on the Signal
processing/Decoding related
part of the requirement has
been addressed in the current
deliverable.
Final analysis will be
addressed in D3.3.
The attention states decoding flag output
provides a numerical value indicating the
user’s level of attention.
89
D3.2 Physical/Physiological sensing devices
Rev. 1.0
Integration in the BCI
architecture will be
addressed in Task
3.4/Deliverable D3.4
BCI11
Decoding accuracy
It provides a numerical value (e.g. a
percentage) related to the ability of the BCI
subsystem in detecting the attention states.
90
Refer to BCI10
D3.2 Physical/Physiological sensing devices
Rev. 1.0
6 Conclusions and future work
This report describes the work related to development of Human Sensory System (HSS) and Brain Computer
Interface (BCI) performed in Work Package 3 "Sensing systems for assessing dynamic system environments
including humans".
The human sensory system has been realized through development of a lightweight chest belt with sensors at
chest and back of patient measuring physiological parameters during gait rehabilitation training sessions.
Infrastructure for transmitting sensor data wirelessly with a predictable low latency to a computer,
synchronizing and time-stamping data and interfacing the user interface and cognitive modules has been
provided. The sensor modules are based on state of the art low power components that are highly integrated
and optimized for long term wireless physiological monitoring. The Chest Belt can easily be fit to and
removed from the patient at start and end of a training session and will not cause any discomfort, but still
provide relevant physiological sensor data to the cognitive modules of the CORBYS Demonstrator I. Patient
safety is handled by using a wireless battery powered sensor modules.
The remaining work for the Human Sensory System will be presented in D3.4 in month 26. This will include:
-
EMG system
- Description of the sensor modules
- Time to mount sensors on patient
- Driver component for the EMG sensors.
-
Complete testing description
- Chest unit
- IMU unit
- HSS controller SW
-
Complete specification of charging of wireless devices
- Operational and charging time for all units
- Localization of charging station
-
ROS network integration
- Complete set of test data exploitable for simulating the execution of the HSS Controller without
actual sensors.
- Front-end implementation fully compliant with the final CORBY ROS Guideline.
- Implementation of the training initialization process and supporting GUI.
- Integration with Task Manager and Executive Supervisor
-
Synchronization issues
- Implementation of an appropriate mechanism for time synchronization with the wireless sensor
(either "over the air" or through serial communication on the charging station).
- Evaluation of latency and time-stamping offsets at the different layers of the HSS Controller.
91
D3.2 Physical/Physiological sensing devices
Rev. 1.0
-
Evaluate effect of noise for human sensory system
Electromagnetic noise from actuators, motors, radios etc. at the mobile platform and in the
environment may interfere with senor measurements. The possible effect of this will be investigated
and reported in D3.4.
-
Physical integration
The Human Sensory System will be integrated with Demonstrator I from WP7 as a part of WP8,
System Integration. When Demonstrator I is ready for integration this work will start by integrating
the HSS computer in the carrier frame together with external antennas and cabling to the general
purpose network. HSS will then be tested in Demonstrator I environment and results will be compared
to results from similar tests prior to integration. This activity is dependent on WP7 and WP8 work and
may not be completed for D3.4
In this present deliverable three topics related to the design of a BCI submodule within the CORBYS gait
rehabilitation system have been addressed:
In section 3.1, Evaluation of EEG acquisition systems that could reduce the noise level, two EEG systems, i.e.
TMSi and g.Tec, have been analysed in order to choose the most appropriate for the CORYS gait
rehabilitation system. Due to the results obtained where the g.Tec system showed to be more affected by
electrical noise contamination, TMSi has be chosen as EEG system for the CORBYS 1st demonstrator.
In section 3.2, Analysis and removal of movement artifacts from during locomotion, the description of
temporal and spectral characteristics of the main contamination introduced in the EEG by locomotion and
head movements have been described. ICA technique was applied to remove the artifacts contamination
generated showing its feasibility within a simulation of the CORBYS rehabilitation user scenario. Due to the
importance of the decoder-specific analysis in the removing process of gait-related artifacts, further
evaluations are needed once the CORBYS decoders have been implemented.
Section 3.3, Detection of attention during assisted passive leg motion, shows how the attention modulates
brain activity during passive leg movements. An off-line classifier has been developed to differentiate between
attention and non-attention to the motor task. Results showed the feasibility to distinguish between both
conditions above the chance level for all the subjects. This is the first step in the design a BCI system to
decode passive lower limb movements with attention and non-attention to the motor task. Future work will
consider the automation of the features selection process and build an online classifier to assess the feasibility
of decoding attention during real time motion.
The remaining work for the Brain Computer Interface will be presented mainly in D3.3 (month 38) and D3.4
(month 26). This will include:
- Design of a Brain Computer Interface software architecture:
-Analysis
-Design
92
D3.2 Physical/Physiological sensing devices
Rev. 1.0
-Implementation
-Documentation
- Development of the CORBYS-related decoding algorithms:
- Attention states
- Feedback error-related potential
- Integration of the decoding algorithms in the BCI software architecture
- BCI module testing:
- Bluetooth acquisition
- Decoders activation/deactivation
- Measurement failure
- Training procedures
- Decoding Procedure
- ROS network integration
- Generation and integration of simulated EEG data (node version for simulation)
-Design of BCI output messages (including EEG data and decoding outputs)
- Design of BCI configuration parameters (from Parameter Server)
- Integration of BCI architecture in ROS node template
- Development of GUI ROS node
- Synchronization
- Evaluation of BCI processing delays
- Evaluation of submodule sampling frequency and cycle time
- Evaluation of synchronization between HSS and BCI data
- Integration and Testing
The Brain Computer Interface submodule will be integrated and tested within the CORBYS mobile platform
in WP7 and WP8. Testing of functional and other requirements will be analysed in WP8.
7 References
Anderer P, Roberts S, Schlogl A, Gruber G, Klosch G, Herrmann W, Rappelsberger P, Filz O, Barbanoj MJ,
Dorffner G, Saletu B .Artifact processing in computerized analysis of sleep EEG – a review
.Neuropsychobiology 1999;40:150-157.
Antelis J.M., Montesano L., Giralt X., Casals A. and Minguez J., “Detection of movements with attention or
distraction to the motor task during robot-assisted passive movements of the upper limb”, International
Conference of the IEEE Engineering in Medicine and Biology Society, San Diego (USA) , 2012.
Costa D, Vitti M, De Oliveira Tosello D., Electromyographic study of the sternocleidomastoid muscle in head
movements, Electromyogr Clin Neurophysiol,1990.
93
D3.2 Physical/Physiological sensing devices
Rev. 1.0
Croft RJ, Barry RJ. Removal of ocular artifact from the EEG: a review. Neurophysiol Clin 2000;30(1):5–19.
Dyson M, F. Sepulveda, J.Q. Gan, Localisation of cognitive tasks used in EEG-based BCIs, Clinical
Neurophysiology, Volume 121, Issue 9, September 2010
Gargiulo G, Calvo RA, Bifulco P, Cesarelli M, Jin C, Mohamed A, van Schaik A. (2010)A new EEG
recording system for passive dry electrodes. Clin Neurophysiol, 121(5):686-93.
Gomez J., Aguilar M., Horna E., and Minguez J., “Quantification of Event-Related Desynchronization /
Synchronization at Low Frequencies in a Semantic Memory Task”, International Conference of the IEEE
Engineering in Medicine and Biology Society, San Diego (USA), 2012 .
Goncharova II, McFarland DJ, Vaughan TM, Wolpaw JR., EMG contamination of EEG: spectral and
topographical characteristics.,Clin Neurophysiol. 2003
Graimann B, Huggins J.E, Levine S.P and Pfurtscheller G, “Visualization of significant ERD/ERS patterns in
multichannel EEG and ECoG data”, Clinical neurophysiology : official journal of the International
Federation of Clinical Neurophysiology, 1 January 2002, volume 113, issue 1, Pages 43-47.
Graimann B. and Pfurtscheller G., “Quantification and visualization of event-related changes in oscillatory
brain activity in the time frequency domain”, Progress in Brain Research, no. 159, pp. 79–97, 2006.
Groppe DM, Makeig S, Kutas M., Identifying reliable independent components via split-half comparison,.
Neuroimage, 2000
g.Tec Technologies, n.d. g.Tec EEG system [online] Available at < http://www.gtec.at/> [Accessed 20
September 2012].
Gwin, J.T., Gramann, K., Makeig, S. and Fer, D. P. (2010). Removal of movement artifact from
highdensity eeg recorded during walking and running. J Neurophysiol., 103:3526–3534.
Hanakawa T, Immisch I, Toma K, Dimyan MA, Van Gelderen P, Hallett M., Functional properties of brain
areas associated with motor execution and imagery, J Neurophysiol. 2003 Feb;89(2):989-1002.
Hirasaki E., Moore S., Raphan T. Cohen B., Theodore, Effects of walking velocity on vertical head and body
movements during locomotion, Exp Brain Res,1999
Hyvärinen, A; Oja, E . Independent Component Analysis: Algorithms and Applications, Neural Networks
2000.
Johansen-Berg H. and Matthews P., “Attention to movement modulates activity in sensori-motor areas,
including primary motor cortex”, Experimental Brain Research, vol. 142, pp. 13–24, 2002.
Liverud A.E, Vedum J., Fleurey F. and Seeberg T.M. (2012). "Wearable Wireless Multi-parameter Sensor
Module for Physiological Monitoring." pHealth 2012 Proceedings of the 9th International Conference on
Wearable Micro and Nano technologies for Personalized Health, pages 210-215, 2012
94
D3.2 Physical/Physiological sensing devices
Rev. 1.0
Matteucci, M., Carabalona, R., Casella, M., Di Fabrizio, E., Gramatica, F., Di Rienzo, M., Snidero, E.,
Gavioli, L. and Sancrotti, M. (2007). Micropatterned dry electrodes for braincomputer interface. Journal
Microelectronic Engineering, 84(2).
Müller-Putz G. R., Scherer R., Brunner C., Leeb R., and Pfurtscheller G., “Better than random? A closer look
on BCI results”, International Journal of Bioelectromagnetism, vol. 10, pp. 52–55, 2008.
Pascual-Marqui R. Standardized low resolution brain electromagnetic tomography (sLORETA): technical
details. Methods Find Exp Clin Pharmacol 2002;24D: 5–12.
Pfurtscheller G., Lopes da Silva F.H., “Event-related EEG/MEG synchronization and desynchronization:
basic principles", Clinical neurophysiology : official journal of the International Federation of Clinical
Neurophysiology, 1 November 1999 (volume 110 issue 11 Pages 1842-1857)
Pfurtscheller G., Müller-Putz G. R., Pfurtscheller J. and Rupp R., "EEG-Based Asynchronous BCI Controls
Functional Electrical Stimulation in a Tetraplegic Patient", EURASIP J. Adv. Sig. Proc., 2005(19):3152-3155,
2005.
Pons J.L., “Wearable Robots: Biomechatronic Exoskeletons”, Wiley Books, 2008.
Searle, A. and Kirkup, L. (2010). A direct comparison of wet, dry and insulating bioelectric recording
electrodes. Physiological Measurement, 21(2):271–283.
Popescu, F., Fazli, S., Badower, Y., Blankertz and Mller, KR. (2007). Single trial classification of motor
imagination using 6 dry eeg electrodes. PLoS ONE, 2(2).
Schlögl A, Keinrath C, Zimmermann D, Scherer R, Leeb R, Pfurtscheller G. , A fully automated correction
method of EOG artifacts in EEG recordings. Clin Neurophysiol. 2007
Tee K. P., Guan C., Ang K. K., Phua K. S., Wang C., and Zhang H., “Augmenting Cognitive Processes in
Robot-Assisted Motor Rehabilitation,” Proceedings of the 2nd Biennial IEEE/RAS-EMBS International
Conference on Biomedical Robotics and Biomechatronics, pp. 698–703, 2008
Teplan, M., Fundamentals of EEG measurement., Measurement Science Review, 2002
TMSi: Twente Medical Systems International, n.d. EEG TMSi system [online] Available at < http://
http://www.tmsi.com/> [Accessed 20 September 2012].
Tzyy-Ping Jung, Colin Humphries, Te-Won Lee, Scott Makeig, Martin J. Mckeown, Vicente Iragui, and
Terrence J. Sejnowski. Extended ICA removes artifacts from electroencephalographic recordings, In NIPS
'97, 1998. MIT Press.
Van de Velde M, van Erp G, Cluitmans PJ., Detection of muscle artefact in the normal human awake EEG,
Electroencephalogr Clin Neurophysiol. 1998 Aug;107(2):149-58.
95
Was this manual useful for you? yes no
Thank you for your participation!

* Your assessment is very important for improving the work of artificial intelligence, which forms the content of this project

Download PDF

advertisement