Human Machine Interaction Using Hybrid Biological Signals: Next step of powered wheelchair controller

Table of Contents

1 Introduction
1.1 History
1.2 Types of wheelchairs
1.2.1 Manual wheelchairs
1.2.2 Electric-powered wheelchairs
1.2.3 Limitation of the electric chairs
1.2.4 Smart or Intelligent wheelchair
1.3 Current research

2 Goals and objective
2.1 Project description
2.2 Hardware
2.2.1 CyberlinkTM headband
2.2.2 Data acquisition box
2.2.3 Wheelchair
2.3 Software
2.3.1 Software description
2.3.2 Data collection
2.3.3 Pre-processing and Data Segmentation
2.3.4 Feature Extraction
2.3.5 Classification
2.3.6 Algorithm
2.3.7 Training and Testing
2.3.8 Pattern Mapping
2.3.9 Decision process
2.3.10 User interface

3 Evaluation
3.1 Component testing
3.2 Subsystem testing
3.3 System testing

4 Future development
4.1 ANN training:
4.2 Extra features in GUI:
4.3 Headband:
4.4 Additional improvements:
4.5 Challenges encountered

5 Conclusion

6 References

Table of Figures

Figure 1: Child’s bed on rollers. From a hydria, lonian made 530B.C [9]

Figure 2: An Intelligent wheelchair [27]

Figure 3: A powered wheelchair with JACO Robotic arm [26]

Figure 4: The Smart Powered Assistance Module (SPAM) for manual Wheelchair [28]

Figure 5: The Home Lift, Position and Rehabilitation (HLPR) chair [29]

Figure 6: Toyota BMI wheelchair [30]

Figure 7: Intelligent Wheelchair control using Computer Vision and Bio-signals [13]

Figure 8: System Block Diagram

Figure 9: The Cyberlink TM headband and data acquisition box

Figure 10: Action potential travels along the axon of a neuron [31]

Figure 11: Picture of the smart wheelchair (RoboChair) for the project [18]

Figure 12: Eleven frequency bands representing EOG, EEG and EMG signals [32]

Figure 13: EEG, EOG and EMG signal displayed using Brainfinger software

Figure 14: Flow chart of software stages to control the wheelchair

Figure 15: Different states in a training sample data

Figure 16: Feature fusion for better movement discrimination

Figure 17: ANN with three input neurons, a hidden layer and four output neurons [20]

Figure 18: A weighted neuron structure with bias [19]

Figure 19: Mean square error plot of intermediate iterations

Figure 20: Mean square error of the finalized ANN after training

Figure 21: Output of ANN for 20 test samples

Figure 22: User interface dialog box

Figure 23: A simple path for testing the wheelchair [32]

Figure 24: Simulation of wheelchair movement using directional arrows

Nomenclature

illustration not visible in this excerpt

1 Introduction

The current surge and advancement of technology has increased the social demands for the quality of life. This has given rise to the development of consumer conscious gadgets for everyday use, such as the latest mobile phones. These devices have made our life easier, faster, safer and more entertaining with improved user experience. As part of the efforts to improve the quality of life for the disabled and the elderly, robotic researchers have been trying to merge the robotic techniques into systems that can assist them in their daily life. The latest developments in research areas such as computer science, robotics and Artificial Intelligence have broadened the possibility to support disabled and elderly people with new assistance systems [1].

The assistance robotic system has to be safe and reliable to use but also user friendly. For this the user should have certain degree of control over the system to overwrite undesired actions of the machine. This is achieved by providing a prototype model of the system and giving it to the actual user for testing for a period of time and improvements in the design are made from their feedback. These types of systems are called the human-in-the-loop control system. These systems need to be tested thoroughly before any commercial production to meet standard requirements [2]. Human Machine interaction (HMI) is fast becoming one of the prominent technologies used for improvising the available resources.

The keyboard and mouse are often used as the Human Computer Interface (HCI) devices. However, it requires more training for the disabled and the elderly to get familiar with a computer. With the improvement of the processing power of computers, many researchers have tried to use computer vision, voice recognition and similar techniques. However the techniques have some flaws. Voice recognition systems are slow in interpreting the results and are easily affected by noise e.g. during a party. The vision based HCI still has to overcome the detection of the individual in real world environments with changing light conditions. Other researchers have proposed to use Bio-signals such as Electromyogram (EMG) [3], Electroencephalogram (EEG) [4], and Electrooculograph (EOG) [5] for HCI. Each of the Bio-signal has its own uniqueness which is used for extracting eminent information. Current researchers are trying to tap these by improving their detection and classification methods with the aid of new technology.

In the Bio-signals used for this project, EEG refers to the recording of the brain’s electrical activity measured using multiple electrodes placed on the scalp region. The brain is always active and generates signals of different intensities. Researchers’ are now able to identify different regions of the brain which are activated when we use different sensory organs or think of using them. EEG signal is produced by the triggering of neurons within the brain. They are widely used for diagnostic of epilepsy patients’. For better usability of the signals, we have to take into account the quality of the EEG recorded, user involvement and more accurate ways of signal analysis. EOG signals are mostly obtained by using two electrodes which are placed on the forehead region. When the eyes move towards one of the sides, it gets closer to one of electrode and away from the other electrode thus creating a potential difference. The signals are then compared with the resting potential of the retina and the movement of the eye in a particular direction is detected.

EMG refers to the recording of the electrical activity of the skeletal muscles, when they are electrically or neurologically activated. These signals are used for applications involving understanding of proper motion of muscle group, controlling bionic parts for the amputees, etc. These signals are strong compared to EEG and easy to notice. EMG signal are often applied to the rehabilitation system, e.g. electric prosthetic hand, because it can be generated by voluntary muscle contraction and it has better properties such as, high amplitude and signal to noise ratio (SNR) than other Bio-signals . EMG signals can be efficiency used as control command with higher accuracy has been concluded in previous research papers [3], [6] and [8].

In this project EMG, EOG and EEG signals generated by eyes and facial muscle movements are recorded using the head band with embedded electrodes. For a paralyzed individual the forehead is the most crucial area to capture useful signals. Also the headband appears less evident when used in public places compared to the electrode cap used in Brain Computer Interface (BCI). The signals recorded are further processed to extract unique features that can be used to control the wheelchair and are easier to be replicated by quadriplegic individuals. These features are given to the ANN to obtain the relevant decision logic. The logical table is used to maps particular movements with the control commands of the wheelchair. The following sub-parts explain more about different wheelchairs, their limitations and current research. The later chapter will give more detail about the controller algorithm software, hardware, GUI, testing of the entire system and future developments.

1.1 History

The first record of combining wheels to furniture was a Greek vase image of wheeled child’s bed around 530 B.C. A picture of the painting is shown in figure 1.The use of wheelchair by people with activity limitation mainly started in the early 1900s. Since then manual wheelchairs have undergone many changes to fit the need of today’s user. Despite disability, wheelchair helps persons to maintain mobility and have a social life. From the manual wheelchair, we have now moved to electric powered wheelchair. The needs of many disabled individuals are satisfied by use of manual or powered wheelchair, but there is a segment of disable individuals who cannot use them for independent mobility. To help these individuals, researchers are using technologies applied in the fields of mobile robots to build intelligent wheelchair with embedded devices and sensors. Nowadays even the traditional wheelchairs are available with improvements such as light weight structure, postural stability, efficiency in propulsion and portability in cars [9] [23].

illustration not visible in this excerpt

Figure 1: Child’s bed on rollers. From a hydria, lonian made 530B.C [9].

1.2 Types of wheelchairs

1.2.1 Manual wheelchairs

Manual wheelchairs are propelled by the occupant by turning the hand-rims or by an attendant using handles. These are mostly used by individuals with upper-body mobility. They are of two main subtypes, rigid and foldable. Rigid chairs are preferred by active users as they have less moving parts and are light in weight. Foldable chairs are easy for storage or placement into a vehicle during travel and are mostly used at airports and hospitals. These wheelchairs can also be fitted with shock absorbers, to cushion bumps on the path [10]. The light weight wheelchairs reduce shoulder and wrist injuries due to strain, decrease the total energy expenditure and are easier for transportation.

1.2.2 Electric-powered wheelchairs

An electric-powered wheelchair uses an electric motor, a handle bar or joystick for navigation and is powered by batteries. Electric wheelchairs are used to travel a longer distance without physical exhaustion. These wheelchairs are useful for individuals who are too weak or otherwise unable to move around in a manual wheelchair. These wheelchairs are also provided to persons with cardiovascular conditions.

Furthermore Electric wheelchairs can be customized to cater individual needs by adding suspension to the front and back wheels, cushioning, light weight frame, pneumatic tires for softer rolling resistance, etc. There are also sports varieties built for wheelchair athletics, playing tennis, basketball, etc [8].

1.2.3 Limitation of the electric chairs

- Steered in an upright posture, hand strength and upper-body mobility are required.
- Mobility scooters have longer length, which limits their turning radius in smaller lanes.
- It has a low ground clearance that can make it difficult to navigate around poor structured paths.
- They have fewer options for body support, such as head or leg rests.
- They are quite heavy and not portable.
- Need to be charged regularly.

1.2.4 Smart or Intelligent wheelchair

A Smart wheelchair is a motorized chair with an superficial control system designed to assist the user. This control is generated with the help of software running on special computer accompanied with sensors and applying technology from the fields of robotics. The user can interact with the system using a joystick, touch sensitive display, a sip-and-puff device, etc. For obstacle detection and avoidance sonar, infrared sensors or Lasers are implemented. Some wheelchairs are attached with robotic manipulators, usually a robotic arm to grab household things.

Smart wheelchairs are designed with specific user requirements in mind. For a user with cerebral palsy the role of the smart wheelchair is to interrupt small muscular signals as high-level commands and execute them. Different techniques can also be implemented on a smart wheelchair such as face detection, path finder, artificial reasoning or behaviour based control techniques [8]

illustration not visible in this excerpt

Figure 2: An Intelligent wheelchair [27].

Earlier intelligent wheelchairs were developed by adding a seating arrangement to mobile robots. The VAHM wheelchair belongs in this group. Its earlier models consisted of a wheelchair with a mobile robot base. It had three control modes, manual mode, semi-automatic mode and full automatic mode. In full automatic mode, autonomous navigation is based on internal maps. Semi-automatic mode involves wall following and obstacle avoidance [24].

The later models of smart wheelchairs were modified commercial powered wheelchair, with added functionality. The figure below shows a powered wheelchair with robotic arm which can grasp a bottle and pickup books and other similar objects.

illustration not visible in this excerpt

Figure 3: A powered wheelchair with JACO Robotic arm [26].

The other variety available is called the “add-on” unit. These units can be easily assembled or detached to any commercial wheelchair. These types of units are valuable for children. As they continue to grow, their wheelchair needs to be altered according to their needs. The figure below shows the Smart Powered Assistance Module (SPAM) for manual Wheelchair [23].

illustration not visible in this excerpt

Figure 4: The Smart Powered Assistance Module (SPAM) for manual Wheelchair [28].

1.3 Current research

An intelligent software algorithm to control a wheelchair is indeed a challenging and significant problem, which has triggered attention of the research community. The current researchers are working on two arenas, first the control system for better utilization of the recorded Bio-signals such as feature extraction, classification algorithm, etc. The other area of research interest is the user-interface, adding different ways a user interacts with the machine such as hand gesture detection, voice recognition, etc. Solving safety concerns and usability issues without major changes in the user’s environment are also important areas under development.

The simplest method used earlier was to activate a switch when certain electrical activity goes above a threshold. For better control more sophisticated methods are applied. Researchers are still tackling problems of real world implementation. Some of the problems faced are cross talk between the nearby electrodes, random variation in the muscle activation, noise due to muscle movements, change in the skin resistance due to sweat and many more. Some of the methods used to overcome these are, the use of differential electrodes, finding a suitable location for electrode placement, etc. Many of the other researchers have implemented different methods such as the NavChair project which uses Vector Field Histogram (VFH) for obstacle avoidance and navigation assistance for powered wheelchair. It uses obstacle avoidance algorithm developed for autonomous robots [25]. This method finds a trade-off between the best path and the user’s goal with obstacle avoidance, making it more acceptable [11]. The HLPR chair (Home Lift, Position and Rehabilitation) is specialized for indoor purposes and provides functionalities such as placing the individual on a bed, providing means to access tall kitchen shelf, etc. Its main focus is to reduce back injury due to prolong use of the wheelchair. The figure below shows the chair in action. It is also used in rehabilitation centers after surgery. The chair can also support trajectory planning for independent mobility by creating a map from the sensor reading [12].

illustration not visible in this excerpt

Figure 5: The Home Lift, Position and Rehabilitation (HLPR) chair [29].

Other researchers are using BCI methods to control the wheelchair. The specialized hardware and software processes the EEG signals when the user thinks of moving in a particular direction. These brainwaves are then translated and the wheelchair moves in that direction. This technology is still in its initial stages and more research is in progress. The figure below show the wheelchair Toyota is currently working on to be used for navigation using brainwaves.

illustration not visible in this excerpt

Figure 6: Toyota BMI wheelchair [30]. 0

Other researchers are combining two methods such as combining computer vision with Bio-signals to get better results. The computer vision is used for pattern detection when a particular movement is performed and at the same time Bio-signals are captured. Combining the information from two separate systems using different technique helps in increasing the confidence level of the control system and makes it more reliable [13].

illustration not visible in this excerpt

Figure 7: Intelligent Wheelchair control using Computer Vision and Bio-signals [13].

[...]