Mirror-therapy as a way to start BCI robot-assisted rehabilitation:a single case longitudinal study of a patient with hemiparesisRoman Rosipal(1,3), Natália Porubcová(1), Barbora Cimrová(2,3), Igor Farkaš(2)
(1) Institute of Measurement Science, Slovak Academy of Sciences, Bratislava (2) Faculty of Mathematics, Physics and Informatics, Comenius University in Bratislava, Slovakia(3) Pacific Development and Technology, LLC, CA, USAhttp://www.um.sav.sk/projects/BCI-RAS/
The Seventh International BCI Meeting, May 21 – 25, 2018, Pacific Grove, CA, USA
SummaryTo improve upper-limb neuro-rehabilitation in chronic stroke patients we apply new methods and tools of clinical training and machine learning for the design and development of an intelligent systemallowing the users to go through the process of self-controlled training of impaired motor pathways. We combine the brain–computer interface (BCI) technology with a robotic arm system into a compactsystem that can be used as a robot-assisted neuro-rehabilitation tool: (1) We use mirror therapy (MT) not only to improve motor functions but also to identify subject’s “atoms,” i.e. spectral-spatial EEG patternsassociated with imagined or real-hand movements, using parallel factor analysis. (2) We designed and tested a BCI-based robotic system using motor imagery in a patient with an impaired right upper limb.The novelty of this approach lies in the control protocol which uses spatial and spectral weights of the estimated sensorimotor atoms during the MT sessions.
Study DesignMirror therapy (9-months), is an innovative treatment approach
where an individual rehearses a specific limb movement byreflecting the movements of the non-paretic side in the mir-ror as if it were the affected side [1]. A link between motorimagery and passive action observation was found and as-sociated with the concept of mirror neurons [2].
PARAFAC - EEG “Atoms”, logarithmically transformed powerspectra densities (PSD) of EEG segments are analyzed bythree-way parallel factor analysis (PARAFAC) [3, 4]. Definea 3-dim. data matrix X (I×J×K) of PSD estimates at I timesegments, J electrodes and K frequencies. Then, three load-ing matrices, A, B, and C with elements a(f)i (time scores),b(f)j (spectral wights) and c
(f)k (spatial weights) define the
PARAFAC model which decomposes X as
xijk =
F∑f=1
a(f)i b
(f)j c
(f)k + εijk
where xijk are elements of X, εijk are the residual errors andF stands for a number of components (atoms). The loadingelements are found by minimizing the sum of squares of εijk
mina(f)i b
(f)j c
(f)k
‖xijk −F∑
f=1
a(f)i b
(f)j c
(f)k ‖
BCI-Controlled Robotic Splint (18 months), following the rec-ommendations of a clinical expert, we designed and con-structed a robotic splint (with one degree of freedom). Thesplint is controlled using the time scores of the selectedatoms extracted from EEG recorded during the MT sessions.Flexible score thresholds can be set.
Oscillatory Rhythms
5 10 15 20
0.40.8
Fre
quency
weig
hts Mu
FC3 C1 C3 C5 CP3 FC4 C2 C4 C6 CP4
0.250.5
Spatial
weig
hts Mu
5 10 15 20
0.40.8
Fre
quency
weig
hts SMR1
FC3 C1 C3 C5 CP3 FC4 C2 C4 C6 CP4
0.250.5
Spatial
weig
hts SMR1
5 10 15 20
0.40.8
Fre
quency
weig
hts SMR2
FC3 C1 C3 C5 CP3 FC4 C2 C4 C6 CP4
0.250.5
Spatial
weig
hts SMR2
5 10 15 20
0.40.8
Fre
quency
weig
hts Beta1
FC3 C1 C3 C5 CP3 FC4 C2 C4 C6 CP4
0.250.5
Spatial
weig
hts Beta1
5 10 15 20
0.40.8
Fre
quency
weig
hts Beta2
FC3 C1 C3 C5 CP3 FC4 C2 C4 C6 CP4
0.250.5
Spatial
weig
hts Beta2
5 10 15 20
0.40.8
Fre
quency
weig
hts Alpha
FC3 C1 C3 C5 CP3 FC4 C2 C4 C6 CP4
Electrode
0.250.5
Spatial
weig
hts Alpha
5 10 15 20
Frequency [Hz]
0.40.8
Fre
quency
weig
hts Theta
FC3 C1 C3 C5 CP3 FC4 C2 C4 C6 CP4
Electrode
0.250.5
Spatial
weig
hts Theta
Mean values of the PARAFAC spectral (left column) and spatial (right column) weight vec-tors obtained during the Mirror Therapy (MT, black) and training with the Robotic Splint(red). Each row represents the means for one of the seven extracted oscillatory rhythms(atoms). Shaded area represents the standard deviation of MT averages.
Mu SMR1 SMR2 Beta1 Beta2 Alpha Theta40
60
80
100
120
Ave
rag
ed
ato
m s
co
res
Mirror Therapy EC - VLH - L
*** * ** *
pre-training
post-training
Mu SMR1 SMR2 Beta1 Beta2 Alpha Theta40
60
80
100
120Mirror Therapy EO - VLH - L
*** *** *** *** ***
Mu SMR1 SMR2 Beta1 Beta2 Alpha40
60
80
100
120
Ave
rag
ed
ato
m s
co
res
Robotic Splint EC - VLH - L
** *** *** **
Mu SMR1 SMR2 Beta1 Beta2 Alpha40
60
80
100
120Robotic Splint EO - VLH - L
*** ** ** *** ***
Mean values of the PARAFAC atom scores computed during the resting period with eyesclosed. The blue bars represent pre-training session, yellow bars post-training. Significantdifferences between the two sessions are highlighted (∗∗∗: p < 0.001, ∗∗: p < 0.01, ∗:p < 0.05)
To study and test changes of oscillatory rhythms at each EEG electrode separately, the irregular-resampling auto-spectral analysis (IRASA)method was applied to separate fractal (representing background EEG) and harmonic (representing oscillatory EEG) components in thepower spectrum of EEG segments [5].
0
1
2
3
4
Mean
PSD
[µV
2/Hz]
FC3 **
0
1
2
3
4
C1 ***
0
1
2
3
4
C3 *
0
1
2
3
4
C5
0
1
2
3
4
CP3
0
1
2
3
4
O1
0
1
2
3
4
FC4 ***
0
1
2
3
4
C2 ***
0
1
2
3
4
C4 ***
0
1
2
3
4
C6 **
0
1
2
3
4
CP4 **
0
1
2
3
4
Mean
PSD
[µV
2/Hz]
FC3
0
1
2
3
4
C1
0
1
2
3
4
C3
0
1
2
3
4
C5
0
1
2
3
4
CP3
0
1
2
3
4
O1
0
1
2
3
4
FC4
0
1
2
3
4
C2
0
1
2
3
4
C4
0
1
2
3
4
C6
0
1
2
3
4
CP4
08/14 04/15
Date
-2
-1
0
1
2
Mean
PSD
Diff.[µV
2/Hz]
FC3 *
08/14 04/15
Date
-2
-1
0
1
2C1 **
08/14 04/15
Date
-2
-1
0
1
2C3 **
08/14 04/15
Date
-2
-1
0
1
2C5 **
08/14 04/15
Date
-2
-1
0
1
2CP3 *
08/14 04/15
Date
-2
-1
0
1
2O1
08/14 04/15
Date
-2
-1
0
1
2FC4 *
08/14 04/15
Date
-2
-1
0
1
2C2 *
08/14 04/15
Date
-2
-1
0
1
2C4 *
08/14 04/15
Date
-2
-1
0
1
2C6
08/14 04/15
Date
-2
-1
0
1
2CP4
Mirror Therapy: Averaged Mu rhythm harmonic part of the EEG power spectrum for theeyes closed condition. Each value is an average computed for a session (day). The first andsecond rows represent averages of pre-training and post-training periods. The third row rep-resents the post- and pre-training difference. Solid lines represents linear fit to data, 95%confidence interval for each point is represented by dotted lines. Linear fits with significantnon-zero slopes are denoted by stars (∗∗∗: p < 0.001, ∗∗: p < 0.01, ∗: p < 0.05).
0
1
2
3
4
Mean
PSD
[µV
2/Hz]
FC3
0
1
2
3
4
C1 *
0
1
2
3
4
C3 *
0
1
2
3
4
C5 *
0
1
2
3
4
CP3
0
1
2
3
4
O1 **
0
1
2
3
4
FC4 ***
0
1
2
3
4
C2 *
0
1
2
3
4
C4 **
0
1
2
3
4
C6 **
0
1
2
3
4
CP4 ***
0
1
2
3
4
Mean
PSD
[µV
2/Hz]
FC3
0
1
2
3
4
C1
0
1
2
3
4
C3
0
1
2
3
4
C5
0
1
2
3
4
CP3
0
1
2
3
4
O1 **
0
1
2
3
4
FC4
0
1
2
3
4
C2
0
1
2
3
4
C4
0
1
2
3
4
C6
0
1
2
3
4
CP4 *
06/16 10/17
Date
-2
-1
0
1
2
Mean
PSD
Diff.[µV
2/Hz]
FC3
06/16 10/17
Date
-2
-1
0
1
2C1
06/16 10/17
Date
-2
-1
0
1
2C3
06/16 10/17
Date
-2
-1
0
1
2C5 *
06/16 10/17
Date
-2
-1
0
1
2CP3
06/16 10/17
Date
-2
-1
0
1
2O1
06/16 10/17
Date
-2
-1
0
1
2FC4
06/16 10/17
Date
-2
-1
0
1
2C2 *
06/16 10/17
Date
-2
-1
0
1
2C4 *
06/16 10/17
Date
-2
-1
0
1
2C6 *
06/16 10/17
Date
-2
-1
0
1
2CP4 *
Robotic Splint: Averaged Mu rhythm harmonic part of the EEG power spectrum for theeyes closed condition. Each value is an average computed for a session (day). The first andsecond rows represent averages of pre-training and post-training periods. The third row rep-resents the post- and pre-training difference. Solid lines represents linear fit to data, 95%confidence interval for each point is represented by dotted lines. Linear fits with significantnon-zero slopes are denoted by stars (∗∗∗: p < 0.001, ∗∗: p < 0.01, ∗: p < 0.05).
5 10 15 20
Frequency [Hz]
0
5
µV
2/H
z
FC3
5 10 15 20
Frequency [Hz]
0
5
µV
2/H
z
C1
5 10 15 20
Frequency [Hz]
0
5
µV
2/H
z
C3
5 10 15 20
Frequency [Hz]
0
5
µV
2/H
z
C5
5 10 15 20
Frequency [Hz]
0
5
µV
2/H
z
CP3
5 10 15 20
Frequency [Hz]
0
5
µV
2/H
z
O1
5 10 15 20
Frequency [Hz]
0
5
µV
2/H
z
FC4
5 10 15 20
Frequency [Hz]
0
5
µV
2/H
z
C2
5 10 15 20
Frequency [Hz]
0
5
µV
2/H
z
C4
5 10 15 20
Frequency [Hz]
0
5
µV
2/H
z
C6
5 10 15 20
Frequency [Hz]
0
5
µV
2/H
z
CP4
Mirror Therapy: Pre-training (black) vs. post-training (red) averaged harmonic part ofthe EEG power spectrum for the eyes-closed condition, pre-trainng (black). Each plot is anaverage computed over 50 sessions (days).
5 10 15 20
Frequency [Hz]
0
2
µV
2/H
z
FC3
5 10 15 20
Frequency [Hz]
0
2
µV
2/H
z
C1
5 10 15 20
Frequency [Hz]
0
2
µV
2/H
z
C3
5 10 15 20
Frequency [Hz]
0
2
µV
2/H
z
C5
5 10 15 20
Frequency [Hz]
0
2
µV
2/H
z
CP3
5 10 15 20
Frequency [Hz]
0
2
µV
2/H
z
O1
5 10 15 20
Frequency [Hz]
0
2
µV
2/H
z
FC4
5 10 15 20
Frequency [Hz]
0
2
µV
2/H
z
C2
5 10 15 20
Frequency [Hz]
0
2
µV
2/H
zC4
5 10 15 20
Frequency [Hz]
0
2
µV
2/H
z
C6
5 10 15 20
Frequency [Hz]
0
2
µV
2/H
z
CP4
Robotic Splint: Pre-training (black) vs. post-training (red) averaged Mu rhythm harmonicpart of the EEG power spectrum for the eyes-closed condition. Each plot is an average com-puted over 132 sessions (days).
Clinical & Behavioural ResultsA clinical evaluation of the subject’s upper limb movement abili-ties was carried out. The undertaken clinical tests indicate a slightimprovement in movement and spasticity of the arm but without adetectable progress for wrist and fingers. It is worth noting that thesubject entered the study as late as two years after stroke with severeplegic hand. Subjectively, we observed an improvement in subject’sspeech and social communication, but this was not clinically tested.The subject showed strong enthusiasm to participate.
For the rehabilitation trainingwith the robotic splint we com-pared 3 different protocols. Weobserved that it was the most dif-ficult when the subjects was in-structed to keep eyes open dur-ing the robotic splint control.
ConclusionsTo our knowledge, this is the first longitudinal study (over 9 months long) of the mirror-box therapy showing effects on the modulation of sensorimotor EEG oscillatory rhythms. We observed significantshort-term (a single session pre- versus post-training) and longer-term EEG effects lasting from day-to-day as well as spanning the whole period of the experiment. Analysis of the EEG data recorded duringthe mirror therapy sessions reveals stable day-to-day space-spectral atomic EEG representation of dominant sensorimotor oscillatory rhythms. The atomic representation of EEG allowed us to develop andtest an efficient and flexible BCI protocol for the control of the constructed robotic splint for neuro-rehabilitation. Longitudinal robotic splint training of a 58-years-old man who had a right-hand hemiplegiadue to an ischemic stroke is promising and although it is a single case study, it is used as a proof of concept, not as a population based statistical proof. Clinical efficiency of this procedure requires furtherevaluation by considering a wider, clinically heterogeneous population of patients with motor impairment.
References[1] Ramachandran V. S., Rogers-Ramachandran D. C., Cobb S. Touching the phantom. Nature, 377:489–490, 1995.[2] Mulder T. Motor imagery and action observation: cognitive tools for rehabilitation. Journal of Neural Transmission. , 114:1265–1278, 2007.[3] Bro R. PARAFAC. Tutorial and applications. Chemometrics and Intelligent Laboratory Systems, 28:149–171, 1997.[4] Rosipal R., Trejo L.J., Nuñez P.L. Application of multi-way EEG decomposition for cognitive workload monitoring. In Proc. of the 6th ICPRM, Beijing, pp. 145-149, 2009.[5] Wen H, Liu Z. Separating Fractal and Oscillatory Components in the Power Spectrum of Neurophysiological Signal. Brain Topography. , 29:13–26, 2016.
AcknowledgementThe work was supported by the Slovak Research and DevelopmentAgency (projects APVV-16-0202 and APVV-0668-12) and by the Slo-vak Grant Agency for Science (projects VEGA-2/0011/16 and VEGA2/0151/18).
1