Adaptive Musical Instruments (AMIs): Past, Present, and Future Research Directions

Florian Grond, School of Physical and Occupational Therapy, McGill, Montreal, Canada

florian [dot] grond [at] mcgill [dot] ca

Keiko Shikako-Thomas, School of Physical and Occupational Therapy, McGill, Montreal, Canada

keiko [dot] thomas [at] mcgill [dot] ca

Eric Lewis, School of Physical and Occupational Therapy, McGill, Montreal, Canada

eric [dot] lweis [at] mcgill [dot] ca

Abstract

We review and discuss the literature related to adaptive musical instruments since 2000, focusing on the use of such instruments with children with disabilities. The aim of this review is to provide a synthesis of perspectives and answer the following questions: How have music technologies, including both software and hardware, been used for children with disabilities and how have they been tested and evaluated? What have been the research questions asked and outcomes evaluated concerning these instruments? The studies reviewed include intervention, narrative and descriptive studies. One observation is that adaptive instrument design and research cuts across many different disciplines including music therapy, education and engineering. We considered articles taking functional and rehabilitation informed perspectives as well as critical disability studies, for which music making is often discussed as a human right independently of potential benefits. We discuss methodological approaches used in these studies, and reports of user’s opinions concerning the use of AMIs. It is worth noting that most uses of AMIs by the population under consideration are highly improvisatory, and so a methodological challenge frequently reported is how can the effectiveness of AMIs be assessed without focusing only on easily measurable outputs? We reveal divisions existing between research focusing on the use of AMIs with precise therapeutic and pedagogic goals in mind, and that interested in more general positive effects of improvised collective creative activity and its role in community building. With this two-fold perspective, we analyse the limitations of current research and derive questions for future directions.

Keywords: Adaptive musical instruments, adaptive use, inclusion, accessibility, design, participatory design, disability studies

Introduction

We report the results of the scoping review we conducted on the literature broadly related to AMIs focusing on the use of these instruments with children with disabilities. Scoping reviews, common in scientific research, have two related goals: to identify a particular research question or topic and to systematically survey and describe the research literature addressing this question or topic, identifying the scope of the question, and the potential existing gaps that can guide future studies. Adaptive instruments and disabilities is a subject for which such a scoping exercise is particularly important due to the highly heterogeneous nature of the researchers working in this area and the disciplines they draw upon. Scoping reviews follow a precise methodology (Arksey et al. 2005 & Anderson et al. 2008) and are typically based on clearly defined inclusion and exclusion criteria. As a point of departure for generating these criteria, we adopted a definition of AMIs or adaptive music technology (Graham-Knight et al. 2015, 416) stating that “Adaptive music technology can be defined as the use of digital technologies to allow a person who cannot otherwise play a traditional musical instrument, to play music unaided.” This straight-forward statement shows the challenges of clear-cut definitions in the field: Is a mainstream music application on a tablet computer an adaptive music instrument if played by a person with disabilities who could not use a traditional instrument? And how wide is the scope of “unaided”—need the user be able to control all parameters of the adaptive instrument on their own? What, for example, if they cannot plug it in, or directly engage with all possible modifications to the instrument? This immediately leads to other questions like, is triggering a button that plays back a sound file a form of musical expression? With these “non-traditional instruments” does the result need to be music as traditionally conceived? If not, what other forms of musical expression can be produced or may be beneficial? Who makes adaptive instruments and why? What are the origins of the barriers for music making other than physical or mental limitations?

Scoping this literature, indicated that inclusion and exclusion criteria must not be set too strictly in order to capture the richness of the existing research, and that at least three main fields address our questions:

These three fields on their own have been reviewed and commented upon in the past. Research on the role of music for children with disabilities mostly evolve around the “Sounds of Intent” project and subsequent work Welch et al. (2009), Ockelford (2015), Faux (2015). For technology in music therapy, a very early literature review can be found with Crowe et al. (2004). More recent reviews, mostly in the context of Electronic Music Technology (EMTs) and music therapy, were motivated by the desire to better understand the difference between EMTs and traditional acoustic instruments as well as the inclusion and exclusion criteria (formal or informal) music therapists use in clinical decision-making, as discussed in an exploratory study by Magee et al. (2008a). A review of musical apps on tablet computers, specifically the iPad, has been conducted by Knight (2013). The emerging yet relatively unexplored field of adaptive music technology has also been reviewed by Graham-Knight, et al. (2015) in a case study with two selected devices. A complementary CDS perspective on the topic is Williams (2013) which examines the social model of disability in music employed in occupational therapy directed at people labelled as having learning disabilities.

Despite all this work, fundamental questions remain concerning the conception, development and testing of AMIs. To list only two: How can a control surface engineer designing new adaptive instruments know of the work of an occupational therapist, and so design in a fashion relevant to their needs? How can phenomenological observations of children with disabilities and the known role and benefits of music in their lives translate into meaningful AMIs, which are not only adapted but also appreciated and therefore adopted?

We therefore aim to give a synoptic survey of a very broad scope of the existing literature to, on the one hand, avoid future researchers reinventing the wheel, and on the other facilitate cross disciplinary partnerships between researchers working in otherwise widely disparate fields. Although scoped systematically, the information we report here is at times eclectic, highlighting what we believe are questions and findings that can help bridge the various boundaries between distinct disciplines, practices and abilities.

A Compilation of the Reviewed Literature

We examined research articles in English published from 2000-2017. There were no geographical restrictions on where the research was done. The initial selection of the literature included 14 articles on empirical studies, 15 articles on descriptive and narrative studies, and 14 conference papers. The type of studies included were intervention studies describing how AMIs have been tested and for what purposes, and studies describing more generally a particular technology which might help inform future research and practice on the types of AMIs used for this population. We were specifically interested in studies testing and applying music technology with children aged 0-21 with all types of disabilities (physical, intellectual, sensory, and mental health disorders of any origin). We included both qualitative studies describing interventions and ethnographic studies concerning how children engage and profit from music making mediated by technology. Beyond this review’s generic scope of Adaptive Musical Instruments (AMI) it also specifically includes the Adaptive Use Musical Instrument (AUMI) and its application in community music therapy settings.

The primary question we aimed to answer is: How have music technologies been used for children with disabilities and how has their effectiveness been tested? The term “music technologies” was defined very broadly to include: software, hardware and adapted surfaces, adapted controllers, triggers, or generally adapted instruments. With this broad definition came a broad range of questions we attempted to answer and synthesize.

We extracted data from studies related to methods used in AMI-related research and did not limit study design including randomized control trials, cross-sectional, qualitative, mixed-methods, with pre/post assessment as well as qualitative studies describing interventions and ethnographic studies concerning how children engage and profit from music making mediated by technology. After searching several scientific databases (Medline, PubMed, CINHAL, PsychInfo) two authors selected articles based on titles, abstracts, and finally full text, against the inclusion and exclusion criteria, and extracted information with respect to the type of population under consideration, concerning children’s age and types of disabilities. Finally, we considered the type of evaluation methods used, i.e. the success criteria employed, how the outcomes were measured, and whether the measures were standardized as with questionnaires or were only observations used. Overall, we aimed to obtain a better understanding of the current state, limitations and gaps of current research.

Literature Concerning the Function and Benefit of Music and the Resultant Motivations for Using it with Children with Disabilities

A variety of beliefs and assumptions concerning music in general underlie the way music is used with children with disabilities, and so also inform the design principles behind AMIs. The literature focuses on the following points:

Objectifiable Benefits of Music Producing Measurable Outcomes

The first category of research papers use AMIs for their known benefits with respect to non-musical rehabilitation goals. Researchers engaging with this approach argue for standardization of practices in music therapy using new technologies. These studies consider music less for its traditional properties, and more for its ability to enact assorted closed-loop feedback mechanisms in interactions with an Application (software or hardware). For instance, working with the MusicGlove Zondervan et al. (2016) report that “music has been found to be highly motivating,” (458). “It has been shown to encourage movement via tightly coupled interaction between the auditory and motor cortices”, that as a consequence it may “improve motor recovery” and “enhance neural reorganization”, thus “increasing functional outcomes from the therapy” (466). Also working with the Music Glove Friedman et al. (2014) highlight that music is “motivating, challenging, sensory-rich, and repetitive”, and that it can “induce plastic changes in the motor cortex” as well as “increase attention span, neuropsychological scores, cognitive functioning and well-being” (2). They add that “motor and auditory temporal processing are coupled during the act of listening, meaning the motor system is responsive to the auditory system” (2). Graham-Knight, et al. (2015, 416) highlight the many benefits music-making has for persons with disabilities, including but not limited to increased self-awareness, increased agency, and increased control. Blain-Moraes, S. et al. (2013) refer to the “long-established history of creating emotional relatedness through music” and that it has been “successfully used to create interaction[s] with non-communicative persons” (3).

In this literature the effects of music are sometimes couched somewhat neutrally, such as “mood calming”, or more normatively as in, “reducing inappropriate behaviour in the classroom”, as well as drawing upon stereotypical distinctions between musical genres (rock music versus classical music).

Music in its own Right, Music as a Human Right

There are important contributions to this field that focus primarily on music in its own right and emphasize less objectifiable extra-musical benefits such as the opportunities afforded by music with respect to development, self-expression and recreation. However, progress with respect to musical skills is often still objectively measured in order to chart educational progress and improve pedagogy. Such research does not necessarily focus on the augmented or adapted nature of AMI’s, rather making first and foremost a general case for music as an important resource for children with disabilities.

A foundational work in this vein is ‘Sounds of Intent’: mapping musical behaviour, by Welch et al. (2009) who argue that “music in the education of … pupils [with disabilities] should be recognized as having two distinct strands: music ‘in its own right’ (… musical skills, knowledge and understanding)” and “music ‘to promote wider learning and development’. (… social and communication skills and improving motor control) (348). Related to this second strand Ockelford (2015) note that “significant non-musical benefits had been noted by teachers and parents as one of the outcomes of their children’s participation in musical activity” (349). In a commentary on The Sounds of Intent project. Faux (2015, 195) highlight the educational importance of “seeking ways in which to provide evidence of progress against pre-determined scales and prioritising the importance of measuring small steps of achievement.” Swingler et al. (2009, 49) argue that “the opportunity to learn a musical instrument is a basic human right … however profound an individual’s degree of cognitive, sensory or physical impairment may be”.

In this type of literature, the extra-musical benefits noted often focus on awareness and social skills. For example, based on the work of Ellis et al. (1995) Swingler et al. (2009, 51) report “some children are now more self-aware and are interacting ... other children show more tolerance and a growing awareness of other people”. Such noted extra-musical effects align well with the model of collective improvisation which has arisen out of Improvisational Studies which emphasises collective improvisation’s potential to both be a site of self-knowledge and understanding of the other. Similarly, Clements-Cortes (2014) working with an EMT called Tenori-on, highlight that EMTs facilitate “self-expression and identity formation.” They also point out that EMTs can promote “social development, play and bonding” (60) beyond physical and cognitive improvements. Nabb et al. (2010) note that music “allowed for expression of creativity within populations limited in communication skills” and “provided opportunities for positive social experiences, self-growth, self-awareness, and improved well-being” (311). Politi et al. (2012) also highlight that people with disabilities (in this case autism spectrum disorder) may simply “have a particular affinity with music” and “may display intense emotional arousal responses to music” (552). In a literature review of AMIs applied to children by Ganeshamoorthy et al. (2015, 3), additional rationales include that “musical games, songs, chants and lullabies have been used for centuries to teach children [...] to integrate […] into their social environments.” This review also refers to the work of Brown et al. (2012) stating that “the research on the role of AMIs in the paediatric population is still scarce” (Ganeshamoorthy et al. page 3). Ganeshamoorthy’s reasoning about the function of music for integration in social environments is in line with Jellison (2012), who argues that music can be an opportunity for cooperative learning and inclusive classrooms and that this is beneficial for everyone involved and “particularly effective for vulnerable students” (9). By discussing the concepts of universal design applied to the pedagogy of inclusive music classrooms, Jellison’s work also connects the issue of music education as a human right with the question of design beyond the design of musical instruments as mere physical artefacts.

Music, Disability and Technology in a Social Context

A third perspective, found less frequently in the literature, is grounded in Critical Disability Studies (CDS). Here the focus is not only on musical instruments and their adaptation but equally on social circumstances and resulting barriers as the main cause of disability. Not all research articles are necessarily explicitly informed by CDS perspectives, and some have a pronounced design component which views adaptive instruments as an enabling rather than assistive technology. Overall this category is somewhat eclectic since the focus on social concerns needs to also attend to the bodily limitations of the musicians AMI are designed for.

Samuels (2015) for instance, talks about how “the social model of disability perceives the exclusionary designs of musical interfaces, as well as non-inclusive social attitudes to music-making, as the disabling factor” (25). Similarly, Bell (2014) highlights in his arguments that the instrument itself defines disability reviewing for instance the physiological conditions that may be barriers for playing the guitar. In arguing for inclusive access to music education Nabb et al. (2010) note that “students with disabilities have similar music interests as their peers without disabilities and that the use of adapted instrument can contribute to their physical and psychological well-being.

In a design-focused article, Crowe et al. (2012) refer to the AMTA Standards of Clinical Practice (2009) which state, “that the highest quality musical instruments must be used in sessions” and that “in the early literature, … adaptations tend to look improvised or ‘‘cobbled together,’’ and further that “visual and tactile aesthetics are important considerations for client’s self-esteem and motivation to participate” (101). This conclusion may be brought to bear on explaining shortcomings/failures of research programs employing AMIs that are prototypes and points towards the importance of establishing partnerships with industrial instrument makers or true artisan instrument builders. A prominent example of such a partnership is the AUMI conceived by Oliveros et al. (2011), which brought together in its design process composers, instrument makers and artists. The AUMI is also a testimony to the force of art and activism in research and development as it literally aims to stretch the boundaries of what it means to participate in music making (Tucker et al. (2016)). In the same spirit of participation, Finch et al. (2016) argue for the importance of including users as active agents within the team of researchers and music therapists developing and adapting the AUMI.

Types and Genre of Music

The descriptors used concerning the musical output of AMIs is diverse; genres terms, e.g. jazz, classic music, rock; sonic qualities such as timbre, rhythm or melody; technical terms such as MIDI or haptic controller; or only the name of the interface or instrument e.g. The Sound Beam, etc.

These different levels of descriptors reflect to some degree the level of engagement or interaction that is available to the users of AMIs, i.e. the extent to which the user’s interaction combined with the technology allow for shaping properties of sounds or musical phrases. This variation also reflects the diverse backgrounds of those involved in AMI design, from engineers to musicians deeply committed to a certain genre of music. Concerning music therapy, aesthetic issues indexed to musical genres and sounds in AMIs are discussed by Ellis et. al. (2000), who point out that music therapy may be embedded in a traditional model of music leading to value judgements of right or wrong ways of playing. The AUMI project by Oliveros et al. (2011) is a prominent corrective to this tendency since it celebrates musical improvisation over notated music as a model for social change and participation and challenges traditional notions of who can make music and how.

Study Design

Research projects focusing on extra-musical effects tend to implement a thorough and quantitative study design. This comes at times at the cost of participant numbers as it can be challenging to recruit large numbers of participants with homogeneous characteristics to satisfy the quantitative design parameters. Blain-Moraes et al. (2013) for instance evaluated biomusic in a complex continuing care unit with 3 clusters, each consisting of one or several caregivers and one noncommunicative patient (age ranging from 12 to 27). With Riddoch et al. (2003), we find in an educational setting a 2x3 study design, varying 2 groups of students with and without disabilities with 3 different teaching styles.

Many qualitative studies, such as the work on Electronic Music Technologies by Magee et al. (2008a), collect interview data and adopt open coding procedures for data analysis, using a grounded theory approach. Data collection is sometimes based on telephone interviews as with Clements-Cortes (2014) who collected the opinion of music therapy professionals. Similar methods are used by e.g. Nabb et al. (2010), who collected the opinions of music band directors using online surveys. Very few studies attempt to capture the first-person perspectives of children with disabilities themselves.

Research that focuses on particular instruments is often based on case studies of the design process, as for example with Graham-Knight et al. (2015). As much as this perspective on AMIs attempts to implement good scientific practice in terms of reproducibility and standardization, design decisions are often unique to particular AMIs and are difficult to generalize, which does not come as a surprise given both the number of design decisions that go into the production of any musical instrument, and the multiplicity of medical conditions and the big variation within those which AMIs also design for.

Approaches focusing on music and disability often adopt study designs and methods typical for ethnography e.g. participatory observation governed by project advisory groups as in Welch et al. (2009). Ockelford (2015) employs a two-step process with an exploratory, qualitative component used in the development of their model, followed by a quantitative evaluation of it. Magee et al. (2008b) compiles clinical recommendations based on illustrative data extracts from thick descriptions of the clinical practices of music therapists. The different study designs employed by distinct disciplines highlight the interdisciplinary nature of AMI research. It is noteworthy that for observational studies with no controls advisory groups may help to avoid researcher bias and are a great opportunity for involving stakeholders in the study. Projects that focus on and involve design practices, such as Samuels (2015), often employ eclectic methods. Crow (2012) for instance downplays formal study design criteria and insists that their project is conceived as a demonstration rather than a research project. One of their outcomes is the final presentation of their results in an art gallery. Such projects often view the presentation of concerts using AMIs, and involving performers with disabilities, as itself a powerful success indicator, in keeping with the non-clinical goals of such projects.

Age, Population

Often authors name the population who experience the benefits of AMIs in very general terms, such as “children”, “adolescents”, or “adults”. Information concerning the end-user population is also often obtained indirectly, as in the study by Magee et al. (2008b), in which music therapists are interviewed about their experience with EMT’s with their clients. In ethnographic studies the population is at times generally described as “children with complex needs”, or “children with profound and multiple learning difficulties”.

The variability on disability definitions in the literature and application in research design and protocols makes it very difficult to draw unified conclusions across studies. There is a great need for more precision in population descriptions and greater focus on what are the research limitations and affordances for working with a given population – limited by age group, disability type and other specific characteristics of individuals. One trend observed is that the larger the population sample the more information about it is mediated through care professionals, and not directly obtained by the researchers. With respect to age, unless an educational setting or paediatric institution provides access to a defined age group the age range varies greatly, from children through adolescents and even adults. This makes it difficult to draw conclusions concerning the goals and benefits of AMIs with respect to age.

Evaluation Methods

The ethnographic methods of qualitative studies are most often based on open-ended semi-structured interviews employing comparative coding methods, e.g. Blain-Moraes et al. (2013), Magee et al. (2008a, 2008b), Clements-Cortes (2014), Bell (2014). The Sounds of Intent framework started with phenomenological observations of the musical behaviours of individual children and young people Welch et al. (2009). Samuels (2015) conducted a year-long ethnographic study based on participant observation and training of music tutors. Crowe et. al. (2012), who sees their work as a demonstration and not research, chose an approach that comes close to what is known as “research through design” by pairing industrial design and music therapy students to create together the adaptive equipment and musical instruments for their assigned client(s). Ganeshamoorthy et. al. (2015) applied a battery of instruments to evaluate the Adaptive Use Musical Instrument (AUMI), amongst them Dimensions of Mastery Questionnaire and the Goal Attainment Scale (both pre and post interventions) and participated in focus groups pre and post interventions.

Comparing these evaluation methods in the light of the information compiled above, some questions arise: if extra-musical effects are evaluated, what is the role of the type of music played or generated through the AMI beyond generating stereotypical genre distinctions? Has the problem of evaluating the quality of interactive music making been reflected upon in the evaluation methods? How can more recent approaches such as research through design, research creation or artistic research be integrated and appreciated as systematic contributions to this field?

Outcomes

For EMTs in music therapy the outcomes of Magee’s (2008a) research propose a five-step treatment model when working with people with complex needs. This includes an analysis of available resources, understanding the movement options, enabling the client as musician, studying cause and effect, and finally the development of musical play. Magee et al. also provides indicators for use of EMTs over acoustic instruments including the limits of physical abilities, sensory impairments, self-expression and expression of identity that may be enhanced through EMTs, and improved motivation through EMTs.

Graham-Knight et al. (2015) derives similar guidelines for the design and application of AMIs, while Bell (2014) offers important insights from a CDS perspective on the importance of the visual appearance of AMIs due to their social functions and what that means with regards to acceptance.

Magee et al. (2008b) systematically covers three aspects: 1) limitations of acoustic instruments versus benefits of technology, 2) opportunities for the therapist, 3) the limits of using technology. Complementary to Bell (2014), we refer here to Magee’s (2008b) finding that there is often an aesthetic drawback with EMTs/AMIs: “Acoustic instruments are generally perceived as more aesthetically appealing than technology, particularly in terms of visual appearance. Acoustic instruments have greater evocative capabilities due to their multisensory capacities, encompassing sight, touch and smell in addition to sound.”

A remarkably strong entanglement between the physical structure of the instrument and high-level outcomes is reported by Crowe et al. (2012) using a two-person cabasa that improved socialization and co-operation significantly. On the other hand, there is also benefit in less physical interactions as reported by Knight (2013) as with touch display-based instruments, stating that for “those who may be overly sensitive to feel a steel guitar string or the resonance of an autoharp, [they] may still strum a guitar app … “ (191).

The last two examples show that conceiving AMIs such that their users benefit from music is a truly creative challenge. In this light, the primary factor affecting accommodation might not only be the awareness of existing options, as reported by Nabb et al. (2010), but also the creation of new options.

The recommendation of Graham-Knight et al. (2015, 418), which is “to make sound quickly” also corroborates Ockelford’s (2008, 26) finding that “responses about attainment and progress suggested that these two phenomena were perceived as interchangeable.” “Making sound quickly” would hence ensure a progressive and motivating learning trajectory. This finding has been reiterated in the work by McHale (2015, 19) emphasizing the need for being attentive to the balance between ability and challenge.

With respect to the discussion of attainment and progress and AMI design we want to draw attention to Magee et al. (2008b, 11) who note that:

technology can enable a client to play highly complex musical structures which could never be played without the assistance of technology. Whilst this might be tempting from a musical viewpoint, therapists felt that they were at risk of losing sight of the client's needs and becoming caught up in producing music of unnecessary sophistication.

The importance of the reciprocal social ties between the musician and assistant/therapist as co-creators has also been elaborated with Finch et al. (2016, 14) and particularly Swingler et al. (2009, 50) and their observation on the relationship between the music therapist and the client:

“the less the client is able to say something with sound because of a physical or cognitive disability, the heavier becomes the therapist’s responsibility for empathy and interpretation. The main focus and engine for the mood and meaning of the music which is happening is on the therapist. The creative and interpretative role is increasingly shifted away from the client with more profound levels of disability. [… as a simple] result from what is physically possible. […] The experience of initiation is central to the success of this approach, […] the vital ‘that was me!’ experience, can function as the foundation stone for further learning and interaction.”

If carefully crafted in all their aspects, this is where AMIs can make a true difference, such as giving control to clients with disabilities, so that they can experience themselves as actors and creators that is to say as individuals with agency embedded within a wider community.

The above synthesis of the existing literature points towards some current issues with AMI design, evaluation and research as it presently stands:

The observational studies suggest additional questions such as:

We believe that paying attention to these questions will help to: ground future studies that look more closely into the potential of AMIs to actually give clients with disabilities independence and control, establish how sustainable certain technological solutions are for repeated use, and demonstrate to what extent AMIs effect social inclusion in the long term. To answer these questions additional perspectives beyond considering the AMI merely as a device but as a device designed, used, analysed and modified within a culture with often conflicting attitudes and concerns for the disabled will help to systematically shape this area of research in ways that will allow it to be a more effective tool for creating truly inclusive communities.

References