- Open Access
Mobile collocated interactions with wearables: past, present, and future
mUX: The Journal of Mobile User Experience volume 5, Article number: 6 (2016)
This Special Issue focuses on the emerging use of wearable technologies for mobile collocated interactions. Rather than merely listing off the articles that follow, we wish to introduce this Special Issue by discussing the past, present, and future of this field in HCI through its theories, ongoing designs and research efforts, and emerging frameworks. We conclude by providing an overview of a series of workshops on the topic, and introduce the two main articles that comprise this Special Issue.
Mobile collocated interactions: origins of the field
Mobile devices such as smartphones and tablets were originally conceived and have traditionally been utilized for individual use. Over the last decade, research on mobile collocated interactions [1–4] has explored situations in which collocated users engage in collaborative activities using their mobile devices, thus going from personal/individual toward shared/multiuser experiences and interactions.
Early research in this field frequently encouraged people to share their devices to create a collective experience or reach a common goal. Various physical and social contexts of use were investigated including teamwork at the office , sharing media content at home  and outdoors , public expression in a theme park  and in a pub [3, 6], location-based mobile disaster response games , and sharing educational stories in rural, developing-world contexts .
The investigation of mobile collocated interactions began almost a decade ago when work in engineering platforms that allow designers to easily create applications for mobile groups began to emerge around 2007/2008 [2, 9, 10]. Hardware and sensing innovations coupled with the smartphone revolution and researchers’ newfound ability to design, build, and deploy mobile applications provided an opportunity for HCI researchers to begin to prototype mobile collaborative group experiences. Additionally, some researchers had earlier focused on how general groupware issues such as ad-hoc group formation and floor control policies are impacted by the switch from standard group-use hardware, such as tabletop displays or interactive whiteboards, to a collection of mobile devices [11, 12]. More recently researchers have looked into simple ways to bind devices together in the context of mobile collocated interactions [13, 14]. The gaming community also showed interest in understanding and designing for the mobile collocated research domain with the production of various network connected handheld gaming consoles. Work in this area varied from attempts at understanding existing practices  to determining the impact of player-to-player and player-to-device orientation . There was even research into the design of mobile collocated gaming experiences (though these did not explicitly examine the mobile group experience but instead used the games to uncover interesting technological findings) [17, 18].
Most of this first-wave research initially looked at the use of smartphones (and tablets) to study mobile collocated interactions. As a result, early mobile collocated interactions research tended to be device-centric.
Mobile collocated interactions with wearables: where the field is currently
As computers become smaller, more powerful, and closer to our bodies, mobile collocated interactions research is inevitably starting to include ever-smaller computers in different form factors. Computers have transitioned from being in a large room (e.g., ENIAC), to our desks (e.g., PCs), to a bag (e.g., laptops), and to our pockets (e.g., mobile phones). Wearable computers (e.g., the WIMM watch) have continued the trend towards ever-smaller computers, ones that can be worn on our wrists (e.g., Apple Watch) or other parts of the body (e.g., Google Glass, Microsoft HoloLens).
Researchers are designing novel input techniques for wearable devices that could enable new forms of mobile collocated interactions. For example, Nenya  consists of a magnetic finger ring that provides an always-available input mechanism. It allows for simple input actions such as twist to select and slide along the finger to click. A wrist-worn sensor tracks this small and socially acceptable ring. Facet  is a multi-display bracelet consisting of multiple independent touch-sensitive segments. It supports multi-segment touch, yielding a rich set of touch input techniques. BitWear  is a prototyping platform for small, wireless, interactive devices. BitWear incorporates hardware, wireless connectivity, and a cloud component to enable collections of connected devices.
While the first-wave of mobile collocated interactions seemed device-centric, this current second wave is experience-centric. An example of this is Blast theory’s Can You See Me Now? (CYSMN?) , a multi-player pervasive game that combines online players with a group of collocated street-based runners. CYSMN? engages critically with the ubiquity of mobile devices, and engages people in the act of gaming and connection in unique ways, extending user and audience affect through visceral gameplay. As a mobile interaction framework, CYSMN? places devices at the service of enriched experiences. At the other extreme, hipDisk  leverages the powerful social possibilities of collocation and embodied engagement, yet overlooks the technological possibilities. Collocated users of hipDisk may play music together, and thereby enjoy a heightened experience of its use. Yet no change in the functioning of the technology occurs to support or extend this shift. Similarly, Hug ShirtFootnote 1 captures temperature, heart rate, pressure and location of touch when capturing a hug, to recreate this ‘hug’ in the paired shirt. When used in demos, it collocates a pair of shirts to demonstrate the valence of ‘sharing’ hugs from a distance. What might be possible if Hug Shirt leveraged what can be afforded by technological collocation?
As wearables gain popularity, contexts in which groups of people are wearing and interacting with multiple wearable devices on their body are becoming more commonplace. In those situations, people can use a rich ecosystem of wearables that support collaborative tasks and experiences through multi-user applications. Such novel mobile collocated interactions may include clothing, accessories, prosthetics, and jewelry. One such example is It’s About Time , which explores extending smartwatch interactions to turn personal wearables into public displays. Three technology probes were created that publicly show the wearer’s content (i.e., when their next meeting starts), a glancer’s content (i.e., missed messages), and public content (i.e., timely information such as news or weather).
Proxemic mobile collocated interactions: future research
Research in mobile collocated interactions has shifted from being device-centric to experience-centric. However, there remains a pressing need to understand the importance of spatial relationships between people and the digital devices in space. Adopting ideas of proxemics could allow for designers to better shape each individual’s personal motivations and perceptions of their interactions with both devices and others, to better support their experiences.
Proxemics, as defined by anthropologist Edward Hall, is a research area focused on the culturally-dependent use of space and physical measures (e.g., distance, orientation, and posture) to mediate and comprehend interpersonal interactions . The knowledge of proxemics has long been employed in other disciplines such as architecture, although its use in HCI is a relatively recent addition (e.g., [26–28]). One particularly pertinent aspect of the theory is that of proxemic ‘zones’, which are essentially boundaries of people’s interpretations of interpersonal distance defined as intimate (less than 1.5 feet), personal (1.5– 4 feet), social (4–12 feet), and public (12–25 feet).
Greenberg et al. , in highlighting the importance of adopting proxemics to help realize the UbiComp vision of technologies that are indistinguishable from everyday life, state that “[people] naturally expect increasing connectivity and interaction possibilities as they bring their devices in close proximity to one another.” This vision drives the idea that as we move through space, the ways in which we understand and interact with our devices should change also, essentially adopting Hall’s idea of proxemic zones.
As HCI moves towards embracing and actualizing the ideas of proxemics in design , for example, to create proxemic-aware digital devices in living rooms , we are also motivated by the idea of proxemics being used to support mobile collocated interactions, to allow our devices to not only react to presence and interaction, but also other indicators, such as the interpersonal distance people naturally use in their everyday interactions.
This special issue on mobile collocated interactions with wearables
This Special Issue is the result of a series of workshops that began with the MobileHCI 2011 workshop “Designing and Evaluating Mobile Systems for Collocated Group Use”  organized by Nirmal Patel and James Clawson. At this first workshop, several design and evaluation challenges were identified as being at the core of this research area: group size, physical distance, device-binding, operating systems, privacy, extending to public displays and tabletops, and conducting in-the-wild evaluations. The second workshop on “Mobile Collocated Interactions: From Smartphones to Wearables”  at CHI 2015Footnote 2 focused on technology and prototyping. After individual presentations and group discussions, relevant topics were identified for a ‘hands on’ session where we used LightBlue BeansFootnote 3 to create working prototypes. The third workshop on “Mobile Collocated Interactions With Wearables”  at MobileHCI 2015Footnote 4 concentrated on bodily exploration with wearable devices. Following presentations of accepted position papers, Lundgren et al.’s Mobile Experiences for Collocated Interactions framework  was introduced to explore ideas in small groups around embodied interactions using different materials, conductive fabrics, and strappingsFootnote 5 to provoke thought. In a fourth workshop on “Proxemic Mobile Collocated Interactions”  at CHI 2016,Footnote 6 the ideas of proxemics were adopted for mobile collocated interactions, leading to in-depth discussion in two large groups on proxemics from an evaluation and technological standpoint. These workshops indicated the field was mature enough to warrant a Special Issue.
The two papers in this special issue offer distinct perspectives on mobile collocated interactions, contributing both a survey of the diverse research that is now encompassed within the field; and, specific interaction examples that go beyond standard mobile approaches.
In the first article, Grubert, Kranz and Quigley uncover impediments to the creation and use of mobile multi-device ecosystems. They report on the results of a comprehensive review of the mobile collocated interaction literature, as well as a survey of expert designers and researchers who focus in this field. Their article categorizes the design, technical, social, and perceptual challenges that designers and researchers need to understand in order to be able to successfully design, develop, deploy, and evaluate mobile multi-device environments that support the exploration of mobile collocated interactions.
Tomico and Wilde approach collocated interaction with wearables from the perspective of clothing. Their article discusses the opportunities and challenges that exist around designing truly wearable wearables, exploring notions of situatedness and personal meaning-making, and applying these through three separate design prototypes. Sound Embracers, for example, uses stretch sensors and speakers to support performance through movement, while Trailblazer is intended to support balance and provide tactile cues while running, and Open Up uses proximity sensors embedded within clothing to impart a sensation when others move into the wearer's personal space. Using these examples to stimulate discussions around situated, embodied and connected wearables, the authors identify a series of opportunities and challenges, and three broad recommendations for future soft wearables.
Clawson J, Voida A, Patel N, Lyons K (2008) Mobiphos: a collocated-synchronous mobile photo sharing application. In: Proceedings of the 10th international conference on Human computer interaction with mobile devices and services (MobileHCI '08). ACM, 187–195. doi:10.1145/1409240.1409261
Lucero A, Keränen J, Jokela T (2010) Social and spatial interactions: shared co-located mobile phone use. In CHI '10 Extended Abstracts on Human Factors in Computing Systems (CHI EA '10). ACM, 3223–3228. doi:10.1145/1753846.1753962
Lucero A, Jones M, Jokela T, Robinson S (2013) Mobile collocated interactions: taking an offline break together. interactions 20(2):26–32. doi:10.1145/2427076.2427083
Patel NJ, Clawson J (2011) Designing and evaluating mobile systems for collocated group use. In: Proceedings of the 13th International Conference on Human Computer Interaction with Mobile Devices and Services (MobileHCI '11). ACM, 765–768. doi:10.1145/2037373.2037508
Durrant A, Rowland D, Kirk DS, Benford S, Fischer JE, McAuley D (2011) Automics: souvenir generating photoware for theme parks. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '11). ACM, 1767–1776. doi:10.1145/1978942.1979199
Porcheron M, Fischer JE, Sharples S (2016) Using Mobile Phones in Pub Talk. In: Proceedings of the 19th ACM Conference on Computer Supported Cooperative Work & Social Computing (CSCW ’16). ACM, 1649–1661. doi:10.1145/2818048.2820014
Fischer JE, Jiang W, Kerne A, Greenhalgh C, Ramchurn SD, Reece S, Pantidim N, Rodden T (2014) Supporting team coordination on the ground: requirements from a mixed reality game. In: Proceedings of the 11th International Conference on the Design of Cooperative Systems (COOP ’14). Springer, Switzerland, pp 46–67. doi:10.1007/978-3-319-06498-7_4
Robinson S, Jones M, Vartiainen E, Marsden G (2012) PicoTales: collaborative authoring of animated stories using handheld projectors. In: Proceedings of the ACM 2012 conference on Computer Supported Cooperative Work (CSCW '12). ACM, 671–680. doi:10.1145/2145204.2145306
Merrill D, Kalanithi J, Maes P (2007) Siftables: towards sensor network user interfaces. In: Proceedings of the 1st international conference on Tangible and embedded interaction (TEI '07). ACM, 75–78. doi:10.1145/1226969.1226984
Pering T, Want R, Rosario B, Sud S, Lyons K (2009) Enabling Pervasive Collaboration with Platform Composition. In: Tokuda H, Beigl M, Friday A, Brush AJ, Tobe Y (eds) Proceedings of the 7th international conference on pervasive computing (Pervasive '09). Springer-Verlag, Berlin, pp 184–201, doi:10.1007/978-3-642-01516-8_14.
Ah Kun LM, Marsden G (2007) Co-present photo sharing on mobile devices. In: Proceedings of the 9th international conference on Human computer interaction with mobile devices and services (MobileHCI '07). ACM, 277–284. doi:10.1145/1377999.1378019
Kohno M, Rekimoto J (2005) Searching common experience: a social communication tool based on mobile ad-hoc networking. In: Proceedings of the 7th international conference on Human computer interaction with mobile devices & services (MobileHCI '05). ACM, 15–22. doi:10.1145/1085777.1085781
Jokela T, Chong MK, Lucero A, Gellersen H (2015) Connecting devices for collaborative interactions. interactions 22:(4)39–43. doi:10.1145/2776887
Lucero A, Jokela T, Palin A, Aaltonen V, Nikara J (2012) EasyGroups: binding mobile devices for collaborative interactions. In: CHI '12 extended abstracts on human factors in computing systems (CHI EA '12). ACM, 2189–2194. doi:10.1145/2212776.2223774
Szentgyorgyi C, Terry M, Lank E (2008) Renegade gaming: practices surrounding social use of the Nintendo DS handheld gaming system. In: Proceedings of the SIGCHI conference on human factors in computing systems (CHI '08). ACM, 1463–1472. doi:10.1145/1357054.1357283
Kauko J, Häkkilä J (2010) Shared-screen social gaming with portable devices. In: Proceedings of the 12th international conference on Human computer interaction with mobile devices and services (MobileHCI '10). ACM, 317–326. doi:10.1145/1851600.1851657
Benford S, Rowland D, Flintham M, Drozd A, Hull R, Reid J, Morrison J, Facer K (2005) Life on the edge: supporting collaboration in location-based experiences. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '05). ACM, 721–730. doi:10.1145/1054972.1055072
Soute I, Markopoulos P, Magielse R (2010) Head Up Games: combining the best of both worlds by merging traditional and digital play. Personal ubiquitous comput. 14(5):435–444. doi:10.1007/s00779-009-0265-0
Ashbrook D, Baudisch P, White S (2011) Nenya: subtle and eyes-free mobile input with a magnetically-tracked finger ring. In Proceedings of the SIGCHI conference on human factors in computing systems (CHI '11). ACM, 2043–2046. doi:10.1145/1978942.1979238
Lyons K, Nguyen D, Ashbrook D, White S (2012) Facet: a multi-segment wrist worn system. In: Proceedings of the 25th annual ACM symposium on User interface software and technology (UIST '12). ACM, 123–130. doi:10.1145/2380116.2380134
Lyons K, Nguyen D, Seki S, White S, Ashbrook S, Profita H (2013) BitWear: a platform for small, connected, interactive devices. In: Proceedings of the adjunct publication of the 26th annual ACM symposium on User interface software and technology (UIST '13 Adjunct). ACM, 73–74. doi:10.1145/2508468.2514716
Benford S, Crabtree A, Flintham M, Drozd A, Anastasi R, Paxton M, Tandavanitj N, Adams M, Row-Farr J (2006) Can you see me now?. ACM Trans. Comput.-Hum. Interact. 13( 1):100–133. doi:10.1145/1143518.1143522
Wilde D (2012) hipDisk: understanding the value of ungainly, embodied, performative, fun. In: CHI '12 extended abstracts on human factors in computing systems (CHI EA '12). ACM, 111–120. doi:10.1145/2212776.2212789
Pearson J, Robinson S, Jones M (2015) It's About Time: Smartwatches as public displays. In: Proceedings of the 33rd annual ACM conference on human factors in computing systems (CHI '15). ACM, 1257–1266. doi: 10.1145/2702123.2702247
Hall E (1963) A system for the notation of proxemic behavior. American anthropologist 65(5):1003–1026. doi:10.1525/aa.1963.65.5.02a00020
Greenberg S, Hornbaek K, Quigley A, Reiterer H, Rädle R (2014) Proxemics in human-computer interaction. Dagstuhl Reports 3(11):29–57. doi:10.4230/DagRep.3.11.29
Kortuem G, Kray C, Gellersen H (2005) Sensing and visualizing spatial relations of mobile devices. In: Proceedings of the 18th annual ACM symposium on user interface software and technology (UIST '05). ACM, 93–102. doi:10.1145/1095034.1095049
Mueller F, Stellmach S, Greenberg S, Dippon A, Boll S, Garner J, Khot R, Naseem A, Altimira D (2014) Proxemics play: understanding proxemics for designing digital play experiences. In: Proceedings of the 2014 conference on designing interactive systems (DIS '14). ACM, 533–542. doi:10.1145/2598510.2598532
Greenberg S, Marquardt N, Ballendat T, Diaz-Marino R, Wang M (2011) Proxemic Interactions: The New Ubicomp?. interactions 18(1):42–50. doi:10.1145/1897239.1897250
Marquardt N, Diaz-Marino R, Boring S, Greenberg S (2011) The proximity toolkit: prototyping proxemic interactions in ubiquitous computing ecologies. In: Proceedings of the 24th annual ACM symposium on User interface software and technology (UIST '11). ACM, 315–326. doi:10.1145/2047196.2047238
Ledo D, Greenberg S, Marquardt N, Boring S (2015) Proxemic-Aware Controls: Designing Remote Controls for Ubiquitous Computing Ecologies. In: Proceedings of the 17th international conference on human-computer interaction with mobile devices and services (MobileHCI '15). ACM, 187–198. doi: 10.1145/2785830.2785871
Lucero A, Clawson J, Lyons K, Fischer JE, Ashbrook D, Robinson S (2015) Mobile Collocated Interactions: From Smartphones to Wearables. In: Proceedings of the 33rd annual ACM conference extended abstracts on human factors in computing systems (CHI EA '15). ACM, 2437–2440. doi:10.1145/2702613.2702649
Lucero A, Wilde D, Robinson S, Fischer JE, Clawson J, Tomico O (2015) Mobile Collocated Interactions With Wearables. In: Proceedings of the 17th international conference on human-computer interaction with mobile devices and services adjunct (MobileHCI '15). ACM, 1138–1141. doi:10.1145/2786567.2795401
Lundgren S, Fischer JE, Reeves S, Torgersson O (2015) Designing Mobile Experiences for Collocated Interaction. In: Proceedings of the 18th ACM conference on computer supported cooperative work & social computing (CSCW '15). ACM, 496–507. doi:10.1145/2675133.2675171
Porcheron M, Lucero A, Quigley A, Marquardt N, Clawson J, O'Hara K (2016) Proxemic Mobile Collocated Interactions. In: Proceedings of the 2016 CHI conference extended abstracts on human factors in computing systems (CHI EA '16). ACM, 3309–3316. doi:10.1145/2851581.2856471
The field of mobile collocated interactions has greatly expanded since the beginnings of this workshop series, and we have been excited to follow and participate in new developments in such a stimulating area. We would like to thank all of the participants and position paper authors for their contributions to the different workshops, as well as our fellow workshop co-organizers Nirmal Patel, Kent Lyons, Daniel Ashbrook, Danielle Wilde, Oscar Tomico, Martin Porcheron, Aaron Quigley, Nicolai Marquardt, and Kenton O’Hara. We look forward to many fruitful discussions and collaborations in the future.
About this article
- Co-located Interaction
- Handheld Devices
- Device Binding