Presentation Mapping for Biofeedback Display

What is Presentation Mapping?

The term of “presentation mapping” is from an interaction model for visualization, which is proposed by Yvonne Jansen and Pierre Dragicevic in [1].  The essence of biofeedback is information that is measured from the user’s body and fed back to the user self. Their interaction model enlightened us to look the biofeedback loop from a view of information pipeline. So we propose a framework of biofeedback interaction as shown in Fig 1.

Fig 1_2

In the first stage of Biofeedback computing, the raw bio-signals are processed and prepared for feedback. In the second stage, there are two key transformations between the bio-data and the final presentations that can be perceived by the users. The first transformation is ‘Audio-visual Mapping’. The bio-data are directly mapped to the basic audio-visual attributes, such as the height of a bar, or the volume of a sound. The outcome of the audio-visual mapping transformation makes up an abstract audio-visual form. We see it ‘abstract’ because, at this point, the data are only coupled with the basic audio or visual attributes (e.g., volume, pitch, color, border, width). For instance, the data are coupled with the height of bars, but at this point, the color of the bars, the location of the bars, the scale of the bars, or the direction of bars are not yet specified. And these operations will be addressed in the second transformation called Presentation Mapping.

The presentation mapping turns the abstract audio-visual form into a fully-specified presentation that can be displayed, perceived and understood by the users. In presentation mapping, the abstract forms can be rearranged into a more representational or figurative form, such as transforming the bar chart into the Heart Bloom flower pattern. In the presentation mapping, we focus on designing an appropriate interface expression to represent the biofeedback data in a more understandable and meaningful way. Specifically, here we think more about the mapping between the interface’s dynamic action and the changing pattern of the bio-data, or the mapping between the overall appearance of visualization and the meaning of this piece of data.

Expressive parameters for presentation mapping

Beyond the basic audio-visual attributes used in audio-visual mapping, for presentation mapping, we extract a set of expressive parameters from the intended interface expression. One expressive parameter often involves multiple basic attributes of visual or audio elements or the working of actuators. For instance, the frequency (basic attribute) of bird sounds and the volume (basic attribute) of wind sound are manipulated together to control the quietness (expressive parameter) of the nature soundscape. If we consider the working of the interface as a model, the expressive parameters can be considered as the model’s external parameters to control the interface expression. And the audio-visual attributes are internal to the model.

Why presentation mapping should be addressed for biofeedback display?

Most of the traditional biofeedback displays address the audio-visual mapping transformation, so they often take the form of graphics or a simple tone. In medical biofeedback devices, this type of biofeedback displays is very common. But in everyday use, for self-use without the assistance of the therapist, these abstract audio-visual display might be difficult for average users to understand its meaning. For example, what does a small heart rate variability mean? From the physiological perspective, it means a good physical resistance to stress; it means a better balance of autonomic nervous system to cope with stress. And this information should be presented to the users in the use of HRV biofeedback system. But unfortunately, few systems present this information well.

During the interaction with a biofeedback system, the interface’s expression is closely connected with users, affecting their perception, understanding, and use of the biofeedback information. So we suggest addressing presentation mapping, which is the next step after the ‘audio-visual mapping’. In my understanding, in the presentation mapping, the abstract forms are further transformed into a more specified and meaningful presentation that can be more intuitively understood and easily utilized by the users.

For instance, in the presentation mapping of the Heart Bloom visualization, we just rearranged the conventional bar chart in four concentric circles. So it looks like a flower. We try to let the visualization speak for itself. The pattern with a flexible shape looks more like a blooming flower to indicate a healthier state and a withered flower indicates a sub-healthy, high-stress state. In the design of BioMirror, the surface #2 curves and bulges outwards and flatten out to represent the IBI data, which is used for guiding the user to practice deep breathing.  The shape-changes of the BioMirror look like ‘imitating’ human breathing, which makes the IBI feedback naturally link with user’s breathing regulation.

A good biofeedback display which addresses presentation mapping well can leverage the associations with the existing knowledge of the users to facilitate the decoding of information and finally facilitate the understanding of the feedback meaning and the use of feedback in self-regulation.

Embodying the idea of “Natural Coupling” in Presentation Mapping 

The term of Natural Coupling is from a design framework proposed by Stephan Wensveen in his paper ‘Interaction Frogger’ [2]. In most mechanical products, the appearance, the action possibilities, the action and the function are all naturally coupled, which allows for intuitive interaction. As stated by Stephan, to achieve intuitive interaction in electronic products the user needs information to guide his actions towards the intended function. He suggests focusing on “the creation of information through feedback and feedforward”. Although Stephan’s interaction framework is proposed for tangible interaction, it still inspired me a lot. Different from tangible interaction emphasizing ‘natural coupling’ between the appearance, the user’s action, and the device’s reaction, here we suggest that a natural coupling between biofeedback data and interface expressions in the presentation mapping. To be honest, the idea of ‘Natural Coupling’ came very late in my Ph.D. study. After I explored a lot of on biofeedback interface design and looked back on my explorations, I thought of most of our designs are consistent with his principles the idea of ‘Natural Coupling’.  I found that it could also apply to the design of biofeedback display.

Fig 12_1

In the presentation mapping, we suggest addressing natural coupling between the biofeedback data and interface expressions by creating a metaphor. For instance, we use the visual tree or flower as a metaphor to represent a healthy or unhealthy state of users. The semantic relevance between the physiological data and its representation helps to interpret the physiological implications [3]. We used the inflation and deflation of an airbag as a metaphor of lung movement during breathing to provide breathing guidance. The natural coupling between the shape changes of the interface and the user’s chest facilitates breathing regulation. The same thinking goes for the design of RESonance display.  We used the brightness transition between the lights and the changes of wind volume as a metaphor of ‘airflow of respiration’ to present IBI feedback, which helps users in breathing regulation. We used the quietness of nature soundscape as a metaphor of ‘peace of mind’ to represent HRV data, which indicate the results of relaxation training.

 

[1]. Jansen, Y., & Dragicevic, P. (2013). An interaction model for visualizations beyond the desktop. IEEE Transactions on Visualization and Computer Graphics, 19(12), 2396-2405.
[1]. S. A. G. Wensveen, J. P. Djajadiningrat, and C. J. Overbeeke, “Interaction frogger: a design framework to couple action and function through feedback and feedforward,” in Proc. DIS’04, Cambridge, MA, USA, 2004, p. 177.
[2].Yu, B., Feijs, L., Funk, M., & Hu, J.  Breathe with Touch: A Tactile Interface for Breathing Assistance System In The 15th IFIP TC.13 International Conference on Human-Computer Interaction (INTERACT 2015), Bamberg, Germany, September, 2015

 

Doctoral Dissertation​

Designing Biofeedback for Managing Stress

Summary

Mainly provoked by increasing stress-related health problems and driven by recent technological advances in human-computer interaction (HCI), the ubiquitous physiologically-relevant information will potentially transform the role of biofeedback from clinical treatment to a readily available tool for personal stress management. The primary motivation for this thesis is to bring biofeedback techniques closer to everyday use so that the average people can harness it more intuitively, effortlessly and comfortably.

The first part of the thesis aims to understand the current status of biofeedback technology in the context of stress management. We first decompose a biofeedback-assisted self-regulation process from the perspective of an information pipeline. Then we present an overview towards state of the art relevant to the biofeedback techniques for stress management and relaxation training. A systematic literature review is carried out to classify biofeedback systems for stress management, regarding bio-sensing technique, bio-data processing approach, biofeedback protocol, feedback modality, and evaluation approach. This systematic review helps us to identify the challenges and opportunities for biofeedback design.

In the second part of this thesis, we present four novel biofeedback interfaces that are developed with various HCI technologies, such as sonification, metaphorical visualization, shape-changing displays, and haptic interfaces. The ‘presentation mapping’ in these biofeedback displays embodies a similar design rationale around the idea of ‘natural coupling’. The user studies provide sufficient confirmation of the design rationales presented. The natural coupling in presentation mapping may help users associate the interface expressions with their physiological activities and specific meanings relevant to life and health. These associations make the biofeedback representations meaningful and further facilitate the users to understand and leverage the biofeedback information intuitively in their self-regulation and self-reflection.

In chapter 3, we apply the idea of nature coupling to the design of an audio interface for heart rate variability (HRV) biofeedback. The heart rate variability is presented by the changing rhythm of short melodies. In this design, the timing variations of heart rate are directly mapped to the rhythmic variations in MIDI notes. In chapter 4, we present StressTree and HeartBloom, two metaphorical visualizations of HRV data. We introduce the images of tree and flower as visual metaphors in the visualization of HRV data. The traditional IBI tachogram and HRV Poincaré plots are transformed into a common flower or tree image that can be understood intuitively. Besides visualizing the IBI dataset, the appearance of the flower-shape or tree-shape visualization also represents a health-related physiological meaning semantically. In chapter 5, we present LivingSurface, an interactive wall-like surface as a shape-changing display of biofeedback. LivingSurface aims at using the qualities of a physical change to enhance the interaction with digital biofeedback information. The expressivity of LivingSurface is explored to embody the interface with a sense of life. In chapter 6, we present Breathe with Touch, a tactile interface that simulates human breathing movements through the inflation and deflation of an airbag. The natural coupling between the user’s breathing behaviour and the interface’s action is designed to facilitate an automatic breathing regulation.

In the third part, we explore using ambient media for biofeedback display. The initial intention is simple: to turn the biofeedback system into ‘invisible background’, where the users can perceive their internal physiological states from an environment without computer screens. We utilize nature sounds and ambient lights for an engaging and comforting biofeedback interaction. In chapter 7 and 8, we design an auditory display with nature sounds. We conduct an experimental study to test the modulation of various nature sounds for calm information display. The study concludes with a simplified three-layer structure and a nature soundscape model, which is used in the design of BioSoundscape in chapter 8. BioSoundscape harnesses nature sounds to create a ‘calm’ nature soundscape that responds to the user’s physiological activities. It can not only serve as an ambient biofeedback display but also be integrated into an indoor acoustic environment as a natural augment. In chapter 9, we present DeLight, an ambient lighting biofeedback system. The intent of DeLight is not only to present biofeedback data but also offer a comforting environmental stimulus for relaxation training. In chapter 10, we integrate the BioSoundscape and DeLight together into a room-scale audio-visual biofeedback system: RESonance. It offers a lightweight solution for immersive biofeedback training. In chapter 11, the developed RESonance biofeedback system was applied in a multi-session biofeedback training with five PhD students and five young soccer players. We evaluate the effectiveness of biofeedback in skill-learning for stress coping, and also investigate the users’ learning curve with biofeedback.

In the last part of the thesis, we formulate our answers to the research questions and conclude our contributions to the research and design biofeedback for stress management. We indicate four future research directions in everyday biofeedback: Inherent Biofeedback, Adaptive Biofeedback, Casual Biofeedback and Peripheral Biofeedback. In conclusion, this thesis presents a design-driven exploration of biofeedback applications for managing stress. The design explorations cover a broad design space including data sonification, metaphorical visualization, shape-changing displays, tangible interactions, and ambient displays. In these explorations, the designs themselves become a resource for new knowledge through empirical research surrounding the evaluation of these works. We hope this work could be a starting point for initiating a new field of ‘Everyday Biofeedback’.

vision of PhD thesis



BioMirror

Featured

A mirror that lets you see inside yourself

Biofeedback, like a mirror, enables individuals to see inside their body and improve self-regulation on their physiology to a healthy direction. However, for a long time, Biofeedback is considered only within the context of clinical settings. In this project, we aim to bridge the gap between biofeedback technique and its application in daily life.

BioMirror is a set of interactive surfaces that respond to users’ heartbeats, respiration and autonomic nervous system activities. As a traditional mirror reflects one’s outside appearance, BioMirror reflects human internal bodily processes. The surface is complex paper-based structure with repetitive incisions created by laser cutting. the rear serves as a medium to transform force from servomotors, vibration motors or fans into an action, stimulating the patterns on the surface to vibrate, swing, bulge or rotate to display physiological information in dynamic physical form.

Collaboration with ALISSA+NIENKE Design Studio

Yu, B., Bongers, N., Van Asseldonk, A., Hu, J., Funk, M., & Feijs, L., LivingSurface: Biofeedback through the Shape-changing Display In Proc. International Conference on Tangible, Embedded and Embodied Interaction (TEI 2016), Eindhoven, the Netherlands


Biomirror_2_small

BioMirror #1 reflects human heartbeat activities


Biomirror_1_small

Biomirror_1_2_small

BioMirror #2 reflects human breathing movements


Biomirror_3_small

BioMirror #3 reflects human heart rate variability level


close-up of Biomirror

Close-up of Biomirror


2016, BioMirror was exhibited at Milan Design Week, Dubai Design Week and Dutch Design Week. It was featured by CNN as one of “the most innovative projects from the world’s leading design schools”.CNN

Interacting with BioMirror in Milan Design Week ( Photograph by Nienke)

2

6

BioMirror in Global Grad Show, Dutbai Design Week, November 2016

BioMirror_DDW_2

BioMirror in Mind the Step, Dutch Design Week, October 2016

BioMirror_DDW_7

BioMirror_DDW_8

Heart Bloom

Featured

Flowers for Life and Love

Heart Bloom is a biofeedback device that collects heart rate data and visualizes it with the assistance of a pen plotter. The path and rhythm of the pen’s movements are dynamically coupled to physiological information, such as heart rate variability, creating a new means for visualizing and monitoring personal medical information. Heart Bloom Illustrates the visitor’s heartbeat into a floral graphics, which reflects the natural biorhythm inside the human body. By interacting with Heart Bloom, participants can see, hear and feel their physiological signals transformed into a unique flower drawing on the paper.

This work is created in collaboration with Studio Rogier Arents

Yu, B., Arents, R., Funk, M., Hu, J., & Feijs, L., HeartPlotter: Visualizing Bio-data by Drawing on Paper  In Proceedings of Extended Abstracts on Human Factors in Computing Systems  (CHI 2016), San Jose, CA, USA, 2016.


3 flowers

Different flower pattern of Heart Bloom


HEART BLOOM for Love at Dubai Design Week, 2016

In November 2015, Heart Bloom was exhibited in Global Grad Show, Dubai Design Week.  HEART BLOOM transforms the participant’s heartbeat data into a blooming flower on a postcard, which is then mailed to the beloved ones with special and ‘heartful’ greetings, shortening the distance of the heart and heart.

A participant was trying Heart Bloom

A participant was trying HEART BLOOM

The postcards of HEART BLOOM on the wall

The postcards of HEART BLOOM on the wall

Set up of Heart Bloom In Global Graduation Show, Dubai Design Week 2016

child with heart bloom

A kid with his heart bloom

people with heart bloom 2

200+ Heart Bloom postcards have been created in Dubai and send around the world


HEART BLOOM for Charity at Dutch Design Week, 2017

“Let’s make every heartbeat count!”

In 2017, Heart Bloom served as a charitable initiative that aims to help children with congenital heart disease (CHD). Heart Bloom invited visitors to donate their heart rate data by creating a personal heart bloom card. Then the participants could receive their heart bloom card by making a donation to the Hartstichting, the Dutch Heart Foundation. The donation went towards the foundation’s efforts to improve the chances of children with heart disease.

With Heart Bloom, we attended a series of charitable public activities organized by the Hartstichting. At these events, we brought Heart Bloom to children with heart disease. The interaction with Heart Bloom empowered them to see, hear and feel the ‘seed’ inside the body blooming into a ‘flower’ with every beat of their heart.

Heart Bloom aimed to bring a positive message that everyone is special and that the heartbeats of children with CHD might be abnormal but create inimitably beautiful flowers. These activities tied the relationship with of children with their own heart, their families and the friendships among other children with heart disease.

Fig 14_13_3

The major Heart Bloom fundraising event was held at one of the leading exhibition halls in Eindhoven during DDW from 22nd to 30th October 2017. Th goal was to invite more than 500 participants to experience the installation and make a large collection of 500 HeartBloom drawings during the 9-days event.  During the DDW fundraising event, a total of 823 participants interacted with the Heart Bloom installation. Heart Bloom greatly inspired enthusiasm and improve the engagement of the participants with the charitable activity. Heart Bloom installation triggered the bystanders to participate in the event.

14715044_1784240558498303_7682036705101819768_o14712771_1783783605210665_2805707187289770331_o14691367_1784240511831641_1732087084161032970_o14883431_1785638325025193_5550957086834826331_o14711034_1783782341877458_3360918395997315116_o

Fig 14_13_2


HEART BLOOM Mobile Fundraiser, 2018

In 2018, for street fundraising, we developed a Heart Bloom mobile application which can be installed on a service cart, serving as a mobile ‘fundraiser’ for Dutch Heart Foundation in long-term use. The Heart Bloom application captures heartbeat data with a phone’s camera and visualizes the flower pattern interactively on the screen. The mobile Heart Bloom ‘fundraiser’ worked more ‘casually’ at a shopping center.

Heart Bloom mobile version_2

DSC_0261

RESonance

Featured

To be a physiological resonance with the body

RESonance is an immersive audio-visual biofeedback system that supports relaxation training should take an ‘ambient’ form beyond a GUI interaction paradigm. The system informs the users about their internal states (i.e., breath and heart rate viability) through ambient light and nature soundscape which can be deployed in a living space or home environment seamlessly. In this way, RESonance turns a room into a biofeedback ‘Mind Room’ which adapts to its inhabitants, breathes as they do, and changes ambient light and soundscape to facilitate deep breathing, calmness and stress management.

Fig 10_3


RESonance is interfaced with an HRV biofeedback system, where two types of physiological data are presented: instantaneous heartbeat interval (IBI) and short-term HRV. Both IBI and HRV are calculated from the Blood Volume Pulse (BVP) signal, which is measured by a noninvasive photoplethysmograph (PPG) sensor on the finger. Firstly, the BVP signal is processed with peak detection algorithm into the time series of IBI. The IBI data are presented back to the users immediately to guide them in breathing regulation. The standard deviation of IBI data (SDNN) is calculated with a moving window of 16 heartbeats as the index of short-term HRV.

Fig 10_1

The mapping between IBI wave to the brightness changes of lights

Fig 10_2

The mapping between HRV level to the color of lights


RESonance mind room space was constructed by a white jersey fabric stretched over a cube-shaped wooden frame (2.5m × 2.5m). Within the mind-room space, there are an armchair, a bio-sensing device, and the center light. The ambient lights are installed outside of the space and shine the light on the fabrics. The center light is a wireless and portable lamp which is in the sight of the user and can also be held during the relaxation. The ambient lights aim to create an immersive lighting environment. Four speakers from a surrounding sound system are installed around the space to create a virtual surround-sound effect of a nature soundscape.

Fig 11_2_3

Set-up of RESonance Biofeedback Mind Room

Picture3

Heart Rhyme

Featured

Listen to the Inner Self

Heart rate is regulated by the autonomic nervous system (ANS), producing a natural heart rate rhythm. When we are under chronic stress, the ANS activates the ‘fight-or-flight’ response constantly, which reduces the variability in our heart rate. On the contrary, greater flexibility in heart rhythm can be observed when we are relaxed. The more flexible the heart rhythm is, the more capable we are of dealing with everyday stressors. Heart-Rhyme is an exploration on a real-time sonification of heart rate variability. It transforms the heartbeat rhythm into a musical rhythm for biofeedback purpose.

Following the idea of ‘natural coupling’, we try to present the timing variations of heartbeat data with the timing variations in sounds instead of timbre, pitch or other acoustic properties. The rhythm of music describes a specific temporal pattern of sounds. It is about the variations in the arrangement of sounds through time. Therefore, we focus on presenting the heart rate data with the rhythmical changes in a short chord. We assume the natural coupling between the heart rate variations and the rhythmical changes can be perceived by naturally and intuitively.

The heart rate variability can be represented by the variations of rhythm in MIDI notes. Here, we proposed four promising audio-forms:


1. arpeggio chords with speed variation

Th audio-form 1 presents the heartbeat intervals by changing the intervals between three notes of a chord. It is the most direct mapping between timing variations of heartbeats and the sounds.

Fig3_2


2. arpeggio chords with emphasis variation

In the audio-form 2, the emphasis of a G chord arpeggio is being modulated among four notes according to the latest heartbeat interval data.

Fig3_3


3. two distinct notes with inter-beat interval delay

4. two stereophonic notes with inter-beat interval delay

Th audio-forms 3 and 4 use the interval between two successive notes to present the most recent heartbeat interval data. In these two forms, the differential of the heartbeat interval data is presented as sound.

Fig3_4


Yu, B., Feijs, L., Funk, M., & Hu, J.Designing Auditory Display of Heart Rate Variability in Biofeedback Context  In 21th International Conference on Auditory Displays (ICAD 2015), Graz, Austria, July 2015