Publikationen von Prof. Dr. Michael Rohs

Social events such as cocktail parties are valuable opportunities to build and sustain interpersonal relationships. However, meeting new people, the self-mixing of cocktails, and tracking one's beverage consumption can be difficult and intimidating. To support users during social events, we introduce CheersMate, a smart drinking glass that enables the simultaneous use of a private and a public wearable display. The private display provides cocktail mixing guidance to the user and shows personal consumption statistics during conversations, while the public display encourages social interaction by revealing the user's name and current drink. By clinking two CheersMates, the user can transfer additional information from the private to the public display, for example, showing the number of people the user has toasted with and the number of already consumed drinks. A pilot study with eight participants provides first insights on the acceptance of our concept and on the perceived usefulness as a social catalyst.

On the skin, a small distance between two actuators can result in the perception of a single, centralized stimulus rather than two distinct stimuli – this phenomenon is known as the funneling illusion. In this work, we explore the electrotactile funneling illusion on the forearm in a user study with 16 participants. We placed an electrode strip with 9 electrode pairs along their forearm – from wrist to elbow. The calibration of the same perceived intensity of each electrode pair for each participant shows that the calibrated intensities near the wrist are significantly higher compared to the intensities calibrated near the elbow. A linear regression corresponds to this behavior as well as the qualitative feedback of our participants. Based on this, we created an equation that helps to reduce the calibration time considerably. The results of our study show that the funneling illusion can be reliably evoked at distances of up to 7.2cm. Further, we provide detailed information on occurrence frequency and precision and explored whether an approach adapted for electrotactile feedback can create an apparent tactile motion.

Iterative design cycles for tangible user interfaces and wearable devices require efficient prototyping techniques to optimize development and to elevate the overall design efficacy. A key challenge for rapid prototyping techniques such as cardboard prototyping, 3D printing, or laser cutting is the integration of conductive surfaces. Additional wiring, conductive paint, or special materials like conductive filament often lack the necessary high conductivity and sufficient durability for designing on-skin wearables to measure muscle activity or to electrically stimulate the skin and muscles. To solve this problem we propose to combine spraying and electroplating to create surfaces that exhibit high conductivity, are solderable, corrosion-resistant and skin-friendly, and embody both practical functionality and aesthetic value. In this paper, we describe an effective spraying and electroplating process for rapid prototyping and demonstrate its applicability using several examples of tangible user interfaces. Further, we discuss advantages and disadvantages and describe limitations of the approach.

Operating small touchscreens with the finger occludes a large part of the screen. We propose using the watch case as the input space, without enlarging the smartwatch. Therefore, we created two prototypes, one with a touch surface on the watch case (CASE) and one with touch surfaces on the watch case and the wristband (CASE+BAND). In a comparative study, we analyze their suitability in a 1D list scrolling task and 2D map navigation task. The results show that occlusion is less of a problem for the list scrolling task, as visibility is sufficient. In the map navigation task, participants reached task completion times with CASE+BAND that are comparable to touch input. CASE was significantly slower, but only requires minimal additional hardware. However, the results of a subsequent longitudinal study demonstrates the learnability of CASE, which led to task completion times comparable to touch input, and provides insights in the gradual development of expert performance.

In the domain of health and well-being, proactive voice interventions have demonstrated their efficacy. However, users often encounter privacy concerns and social embarrassment due to the lack of control over these proactive interventions, especially in formal and social settings. This study introduces a novel approach called “dual-mode intervention.” It begins with primary interventions using different modalities (like graphical, tactile, or auditory). If users do not respond to these primary interventions, the system delivers voice interventions after a short interval. We conducted a study employing a within-subjects design, which involved 15 participants. The study compared dual-mode interventions with direct voice interventions in office settings, focusing on addressing health and well-being issues. Our findings indicate that knowledge workers preferred dual-mode interventions over direct voice interventions. Moreover, direct voice interventions received significantly lower ratings compared to dual-mode interventions. Also, we identify user preferences for different dual-intervention modalities. Our findings reveal that the user preferences depend on the type of health intervention. Vibration emerged as the preferred modality, followed by graphical output, auditory icons, and ringing interventions.

Cycling navigation is a complex and stressful task as the cyclist needs to focus simultaneously on the navigation, the road, and other road users. We propose directional electrotactile feedback at the wrist to reduce the auditory and visual load during navigation-aided cycling. We designed a custom electrotactile grid with 9 electrodes that is clipped under a smartwatch. In a preliminary study we identified suitable calibration settings and gained first insights about a suitable electrode layout. In a subsequent laboratory study we showed that a direction can be encoded with a mean error of 19.28° (σ = 42.77°) by combining 2 adjacent electrodes. Additionally, by interpolating with 3 electrodes a direction can be conveyed with a similar mean error of 22.54° (σ = 43.57°). We evaluated our concept of directional electrotactile feedback for cyclists in an outdoor study, in which 98.8% of all junctions were taken correctly by eight study participants. Only one participant deviated substantially from the optimal path, but was successfully navigated back to the original route by our system.

Prior research has designed and evaluated Voice Assistance (VA) for different settings such as the home, school, and public spaces. Office environments have been relatively understudied, leaving a gap in understanding the essential factors for designing a VA specifically for work settings. In this study, we developed the WorkFit VA specific for the office environment, focusing on the health and well-being of knowledge workers. WorkFit was designed to monitor knowledge workers for sedentary behavior, inconsistent hydration, and stress, and to deliver proactive voice interventions followed by a health recommendation to mitigate those issues. We evaluated WorkFit in a field study with 15 knowledge workers for 5 working days. In the study, we determined challenges and opportunities for voice interactions in work settings. We identified contextual factors for identifying inopportune moments for voice interactions in an office setting. We found that 92\% of knowledge workers accepted WorkFit’s hydration interventions while 79\% of them engaged in walking breaks. Moreover, breathing exercises recommended by WorkFit significantly stabilized the heart rate of knowledge workers during stress. Based on our findings, we propose five design recommendations for the development of VA customized to office settings.

The earlobe is a well-known location for wearing jewelry, but might also be promising for electronic output, such as presenting notifications. This work elaborates the pros and cons of different notification channels for the earlobe. Notifications on the earlobe can be private (only noticeable by the wearer) as well as public (noticeable in the immediate vicinity in a given social situation). A user study with 18 participants showed that the reaction times for the private channels (Poke, Vibration, Private Sound, Electrotactile) were on average less than 1 s with an error rate (missed notifications) of less than 1 %. Thermal Warm and Cold took significantly longer and Cold was least reliable (26 % error rate). The participants preferred Electrotactile and Vibration. Among the public channels the recognition time did not differ significantly between Sound (738 ms) and LED (828 ms), but Display took much longer (3175 ms). At 22 % the error rate of Display was highest. The participants generally felt comfortable wearing notification devices on their earlobe. The results show that the earlobe indeed is a suitable location for wearable technology, if properly miniaturized, which is possible for Electrotactile and LED. We present application scenarios and discuss design considerations. A small field study in a fitness center demonstrates the suitability of the earlobe notification concept in a sports context.

Providing rich feedback on small devices, like smartwatches, can be difficult. We propose colorful electrotactile feedback on the back of a smartwatch. Colorful electrotactile feedback provides private notifications, is energy efficient, and can express various sensations in different qualities. In a first study, 13 participants explored 49 different combinations of frequency and pulse width regarding the perceived “colorfulness” of electrotactile feedback. We investigated what sensations can be expressed with electrotactile feedback and which qualities of these sensations are conveyed. To describe the sensations, participants chose the best fitting terms from a list of 21 terms. The three most frequently selected terms were prickling (177), vibrating (163), and irritating (112). The three least frequently selected ones were twitching (31), tickling (29), and itching (28). In a second study with 17 participants we evaluated a reduced set of 9 sensations that we selected and refined based on the results of study 1. We evaluated these sensations regarding recognition rates and achieved recognition rates of up to 84% without prior learning. Furthermore, we investigated the acceptance of colorful electrotactilefeedbackandpresentamethodforaneasierandfaster calibration of electrotactile feedback.

Urban environments are often characterized by loud and annoying sounds. Noise-cancelling headphones can suppress negative influences and superimpose the acoustic environment with audio-augmented realities (AAR). So far, AAR exhibited limited interactivity, e. g., being influenced by the location of the listener. In this paper we explore the superimposition of synchronized, augmented footstep sounds in urban AAR environments with noise-cancelling headphones. In an online survey, participants rated different soundscapes and sound augmentations. This served as a basis for selecting and designing soundscapes and augmentations for a subsequent in-situ field study in an urban environment with 16 participants. We found that the synchronous footstep feedback of our application EnvironZen contributes to creating a relaxing and immersive soundscape. Furthermore, we found that slightly delaying footstep feedback can be used to slow down walking and that particular footstep sounds can serve as intuitive navigation cues.

The need for online meetings increased drastically during the COVID-19 pandemic. Wearing headphones for this purpose makes it difficult to know when a headphone wearing person is available or in a meeting. In this work, we explore the design possibilities of headphones as wearable public displays to show the current status or additional information of the wearer to people nearby. After two brainstorming sessions and specifying the design considerations, we conducted an online survey with 63 participants to collect opinions of potential users. Besides the preference of the colors red and green as well as using text to indicate availability, we found that only 54 % of our participants would actually wear headphones with public displays attached. The benefit of seeing the current availability status of a headphone-wearing person in an online meeting or phone call scenario were nonetheless mentioned even by participants that would not use such headphones.

Although in many cases contracts can be made or ended digitally, laws require handwritten signatures in certain cases. Forgeries are a major challenge with digital contracts, as their validity is not always immediately apparent without forensic methods. Illiteracy or disabilities may result in a person being unable to write their full name. In this case x-mark signatures are used, which require a witness for validity. In cases of suspected fraud, the relationship of the witnesses must be questioned, which involves a great amount of effort. In this paper we use audio and motion data from a digital pen to identify users via handwritten symbols. We evaluated the performance our approach for 19 symbols in a study with 30 participants. We found that x-marks offer fewer individual features than other symbols like arrows or circles. By training on three samples and averaging three predictions we reach a mean F1-score of F1 = 0.87, using statistical and spectral features fed into SVMs.

A common problem of touch-based smartwatch interaction is the occlusion of the display. Although some models provide solutions like the Apple "digital crown" or the Samsung rotatable bezel, these are limited to only one degree of freedom (DOF). Performing complex tasks like navigating on a map is still problematic as the additional input option helps to zoom, but touching the screen to pan the map is still required. In this work, we propose using a trackball as an additional input device that adds two DOFs to prevent the occlusion of the screen. We created several prototypes to find a suitable placement and evaluated them in a typical map navigation scenario. Our results show that the participants were significantly faster (15.7%) with one of the trackball setups compared to touch input. The results also show that the idle times are significantly higher with touch input than with all trackball prototypes, presumably because users have to reorient themselves after panning with finger occlusion.

Recent research in Human-Computer Interaction for work has shown that conversational agents (CA) are beneficial for supporting focused work and well-being while at work. Knowledge workers struggle in maintaining focus, work schedule, and well-being. Typically, they rely on multiple tools and services for work productivity, scheduling tasks, and reminding breaks. With the goal of tackling these problems, we propose the concept of a ubiquitous work assistant (UWA), which consists of two components: a stationary CA (S-CA) and a wearable CA (W-CA). S-CA is meant to be placed on user’s work desk while W-CA is fixed on the user’s wrist. The UWA interface is distributed between S-CA and W-CA. We initiated our study by conducting semi-structured interviews with knowledge workers (N = 14). We identified their expectations from conversational agents (CAs) that would assist them in their daily work life. From the interview findings, we developed an UWA prototype that could assist users by briefing their daily schedule, monitoring their schedule, and reminding breaks. We conducted a lab study simulating a home-office environment. The findings of the study show that the knowledge workers see potential in the UWA system. Further, we discuss implications of distributed user interface (DUI) for UWA design.

Tactile patterns are a means to convey navigation instructions to pedestrians and are especially helpful for people with visual impairments. This article presents a concept to provide precise micro-navigation instructions through a tactile around-the-head display. Our system presents four tactile patterns for fundamental navigation instructions in conjunction with continuous directional guidance. We followed an iterative, user-centric approach to design the patterns for the fundamental navigation instructions, combined them with a continuous directional guidance stimulus, and tested our system with 13 sighted (blindfolded) and 2 blind participants in an obstacle course, including stairs. We optimized the patterns and validated the final prototype with another five blind participants in a follow-up study. The system steered our participants successfully with a 5.7 cm average absolute deviation from the optimal path. Our guidance is only a little less precise than the usual shoulder wobbling during normal walking and an order of magnitude more precise than previous tactile navigation systems. Our system allows various new use cases of micro-navigation for people with visual impairments, e.g., preventing collisions on a sidewalk or as an anti-veering tool. It also has applications in other areas, such as personnel working in low-vision environments (e.g., firefighters).

Previous literature has reported that users consider hands-free and eyes-free interaction as one of the prime features of IPAs (Intelligent Personal Assistants). Hands-free and eyes-free interaction enables dual tasking. Although users prefer dual tasking with IPAs, it is unknown to what degree current IPAs are compatible with dual tasking. To determine IPA efficiency while dual tasking, we investigate cognitive load in dual-task scenarios with IPAs. In our experiment, we selected a rhythm game as the primary task and everyday IPA requests as secondary tasks. The secondary tasks belonged to four common categories: information search, multimedia control, smart home control, and turn-taking conversations. The findings show that IPAs need significant improvement to support dual tasking. Out of the four categories, only tasks in the smart home and multimedia categories were appropriate for dual tasking, whereas turn-taking conversation and information search had a high cognitive load. Task completion time was significantly different between tasks, but the penalty on the accuracy of the primary task was small. In interviews we found that, due to information abundance in IPA responses and high time pressure during task completion, users tended to make several mistakes. Based on our findings and observations we derive four design recommendations that facilitate dual-tasking while using IPAs.

People with visual impairments face challenges in scene and object recognition, especially in unknown environments. We combined the mobile scene detection framework Apple ARKit with MobileNet-v2 and 3D spatial audio to provide an auditory scene description to people with visual impairments. The combination of ARKit and MobileNet allows keeping recognized objects in the scene even if the user turns away from the object. An object can thus serve as an auditory landmark. With a search function, the system can even guide the user to a particular item. The system also provides spatial audio warnings for nearby objects and walls to avoid collisions. We evaluated the implemented app in a preliminary user study. The results show that users can find items without visual feedback using the proposed application. The study also reveals that the range of local object detection through MobileNet-v2 was insufficient, which we aim to overcome using more accurate object detection frameworks in future work (YOLOv5x).

We present SpectroPhone, a surface material sensing approach based on the rear camera of a smartphone and external white LED light sources. Warm and cool white LEDs, as used for dual or quad flashlights in smartphones, differ in their spectral distribution in the red and blue range. Warm and cool white LEDs in combination can produce a characteristic spectral response curve, when their light is reflected from a surface. We show that with warm and cool white LEDs and the rear-camera of a smartphone 30 different materials can be distinguished with an accuracy of 99 \%. Based on a dataset consisting of 13500 images of material surfaces taken at different LED light intensities, we report recognition rates of support vector machines with different parameters.

Creating tactile patterns on the body via a spatial arrangement of many tactile actuators offers many opportunities and presents a challenge, as the design space is enormous. This paper presents a VR interface that enables designers to rapidly prototype complex tactile interfaces. It allows for painting strokes on a modeled body part and translates these strokes into continuous tactile patterns using an interpolation algorithm. The presented VR approach avoids several problems of traditional 2D editors. It realizes spatial 3D input using VR controllers with natural mapping and intuitive spatial movements. To evaluate this approach in detail, we conducted a user study and iteratively improved the system. The study participants gave predominantly positive feedback on the presented VR interface (SUS score 79.7, AttrakDiff “desirable”). The final system is released alongside this paper as an open-source Unity project for various tactile hardware.

Public bookcases offer the opportunity to serendipitously discover books and to anonymously share books with others. The set of available books as well as the sharing patterns are highly dynamic, as anybody can freely take or donate books. This makes it difficult for users to see what is available or of interest to them. To support book sharing via public bookcases we developed a mobile AR application that highlights relevant books in the camera viewfinder and that facilitates searching for specific books. The application recognizes books via text and color features on the spine. In a lab study with 15 participants we evaluated our book recognition algorithm and found that it outperforms unaided visual search. We interviewed users of public bookcases and analyzed the bookcases’ setup and rate of change. A subsequent field evaluation of the AR application on nine public bookcases found a recognition accuracy of 80 % for 450 books under different conditions. The proposed approach provides the basis for effectively sharing books via public bookcases.

We propose around-the-head spatial vibrotactile patterns for representing different kinds of notifications. The patterns are defined in terms of stimulus location, intensity profile, rhythm, and roughness modulation. A first study evaluates recall and distinguishability of 30 patterns, as well as agreement on meaning without a predetermined context: Agreement is low, yet the recognition rate is surprisingly high. We identify which kinds of patterns users recognize well and which ones they prefer. Static stimulus location patterns have a higher recognition rate than dynamic patterns, which move across the head as they play. Participants preferred dynamic patterns for comfort. A second study shows that participants are able to distinguish substantially more around-the-head spatial patterns than smartphone-based patterns. Spatial location has the highest positive impact on accuracy among the examined features, so this parameter allows for a large number of levels.

Winter sports like skiing are becoming increasingly popular for both competitive and recreational activities. To minimize the risk of injury, new innovations in skiing equipment have been developed in recent years. However, unexpected slope conditions can still increase risks during skiing. The static categorisation of ski slopes in winter sports resorts does not take into account dynamic changes of difficulty due to high traffic volumes or sudden weather changes. Up to now, efforts have been made to measure the current conditions via satellite imaging or installations on the slope. However, this requires intervention in nature and causes high maintenance costs. To solve these issues we present our preliminary design of a wearable system to let skiers implicitly measure current slope conditions during their skiing experience. Audio and motion data are recorded from a prototype mounted on a ski boot. We show that the data generated by the prototype can be successfully classified with a neural network. We collected data from a skiing activity to demonstrate our concept and discuss the identified challenges in fitting the proposed approach to winter sports equipment.

Wearables are getting more and more powerful. Tasks like notifications can be delegated to smartwatches. But the output capabilities of wearables seem to be stuck at displays and vibration. Electrotactile feedback may serve as an energy-efficient alternative to standard vibration feedback. We developed prototypes of wristbands and rings and conducted two studies to compare electrotactile and vibrotactile feedback. The prototypes have either four electrodes for electrotactile feedback or four actuators for vibration feedback. In a first study we analyzed the localization characteristics of the created stimuli. The results suggest more strongly localized sensations for electrotactile feedback, compared to vibrotactile feedback, which was more diffuse. In a second study we created notification patterns for both modalities and evaluated recognition rates, verbal associations, and satisfaction. Although the recognition rates were higher with electrotactile feedback, vibrotactile feedback was judged as more comfortable and less stressful. Overall, the results show that electrotactile feedback can be a viable alternative to vibrotactile feedback for wearables, especially for notification rings.

In this paper, we aim to draw attention towards wake words. Wake words are an integral part of every request addressed to Intelligent Personal Assistants (IPAs). Currently, a request made to an IPA is led by wake words, making a conversation with an IPA more tiresome than a conversation with a human being. The main question we pose in this paper is, whether we can eliminate the use of wake words at least in specific contexts. Based on our experience with IPAs we propose three less burdensome alternatives that avoid the need for speaking wake words in some cases. Based on these approaches we discuss how to design seamless conversations with IPAs.

The vibrotactile funneling illusion is the sensation of a single (non-existing) stimulus somewhere in-between the actual stimulus locations. Its occurrence depends upon body location, distance between the actuators, signal synchronization, and intensity. Related work has shown that the funneling illusion may occur on the forehead. We were able to reproduce these findings and explored five further regions to get a more complete picture of the occurrence of the funneling illusion on the head. The results of our study (24 participants) show that the actuator distance, for which the funneling illusion occurs, strongly depends upon the head region. Moreover, we evaluated the centralizing bias (smaller perceived than actual actuator distances) for different head regions, which also showed widely varying characteristics. We computed a detailed heat map of vibrotactile localization accuracies on the head. The results inform the design of future tactile head-mounted displays that aim to support the funneling illusion.

Smartwatches can be used independently from smartphones, but input tasks like messaging are cumbersome due to the small display size. Parts of the display are hidden during interaction, which can lead to incorrect input. For simplicity, instead of general text input a small set of answer options are often provided, but these are limited and impersonal. In contrast, free-form drawings can answer messages in a very personal way, but are difficult to produce on small displays. To enable precise drawing input on smartwatches we present a magnetic stylus that is tracked on the back of the hand. In an evaluation of several algorithms we show that 3D position estimation with a 7.5x20mm magnet reaches a worst-case 6% relative position error on the back of the hand. Furthermore, the results of a user study are presented, which show that in the case of drawing applications the presented technique is faster and more precise than direct finger input.

Creating tactile patterns for a grid or a 3D arrangement of a large number of actuators presents a challenge as the design space is huge. This paper explores two different possibilities of implementing an easy-to-use interface for tactile pattern design on a large number of actuators around the head. Two user studies were conducted in order to iteratively improve the prototype to fit user needs.

A visualization recommender supports the user through automatic visualization generation. While previous contributions primarily concentrated on integrating visualization design knowledge either explicitly or implicitly, they mostly do not consider the user's individual preferences. In order to close this gap we explore online learning of visualization preferences through dueling bandits. Additionally, we consider this challenge from a usability perspective. Through a user study (N = 15), we empirically evaluate not only the bandit's performance in terms of both effectively learning preferences and properly predicting visualizations (satisfaction regarding the last prediction: μ = 85%), but also the participants' effort with respect to the learning procedure (e.g., NASA-TLX = 24:26). While our findings affirm the applicability of dueling bandits, they further provide insights on both the needed training time in order to achieve a usability-aligned procedure and the generalizability of the learned preferences. Finally, we point out a potential integration into a recommender system.

Complex software-based systems involve several stakeholders, their activities and interactions with the system. Vision videos are used during the early phases of a project to complement textual representations. They visualize previously abstract visions of the product and its use. By creating, elaborating, and discussing vision videos, stakeholders and developers gain an improved shared understanding of how those abstract visions could translate into concrete scenarios and requirements to which individuals can relate. [Question/problem] In this paper, we investigate two aspects of refining vision videos: (1) Refining the vision by providing alternative answers to previously open issues about the system to be built. (2) A refined understanding of the camera perspective in vision videos. The impact of using a subjective (or “ego”) perspective is compared to the usual third-person perspective. [Methodology] We use shopping in rural areas as a real-world application domain for refining vision videos. Both aspects of refining vision videos were investigated in an experiment with 20 participants. [Contribution] Subjects made a significant number of additional contributions when they had received not only video or text but also both – even with very short text and short video clips. Subjective video elements were rated as positive. However, there was no significant preference for either subjective or non-subjective videos in general.

Conversational interfaces (CIs) have the potential to empower a broader spectrum of users to independently conduct visual analysis. Yet, recent approaches do not fully consider the user's characteristics. In particular, the objective of matching the user's language has been understudied in visual analysis. In order to close this gap, we introduce an answer space motivated by Grice's cooperative principle for framing personalized communication in complex data situations. We conducted both an online survey (N=76) to analyze communication preferences and a qualitative experiment (N=10) to investigate personalized conversations with an existing CI. In order to match the user's language properly, our results suggest to consider additional user characteristics along with their knowledge level. While mismatching communication preferences triggers negative reactions, a preference-aligned communication evokes positive reactions. As our analysis confirms the importance of matching the user's language in visual analysis, we provide design implications for future CIs.

Nowadays, wearables can easily monitor and display physical activities throughout the day. Health recommendations are often used to set daily goals, but these barely take individual requirements into account. In addition, due to limited individual adaptability, there are various life situations in which these goals are not achieved due to missing motivation or time. In this position paper we discuss in particular how health recommendations can be integrated into everyday life and what challenges arise. We also address spatial requirements that are necessary for an active lifestyle.

Everyday mobility encompasses different forms of public and private transportation and different forms of physical activity. However, in general everyday mobility does not involve substantial levels of physical activity. There are sometimes structural reasons or a lack of motivation and time to realize an active lifestyle in the context of mobility. The goal of this workshop is to investigate ways to integrate physical activity into everyday mobility in accordance with widely accepted health recommendations. We aim to explore wearable and ambient systems that sense and support active navigation as well as conceptual aspects from a variety of perspectives, such as persuasive technologies, and thus invite researchers from different disciplines to contribute their point of view by means of position papers, posters, and demonstrations. One planned outcome of this workshop is a set of design guidelines for navigation systems that explicitly consider health aspects. For the full-day workshop we aim to explore requirements and design challenges in a creative setting.

Receiving and reacting to notifications on mobile devices can be cumbersome. We propose MuscleIO, the use of electrical muscle stimulation (EMS) for notification output and electromyography (EMG) for reacting to notifications. Our approach provides a one-handed, eyes-free, and low-effort way of dealing with notifications. We built a prototype that interleaves muscle input and muscle output signals using the same electrodes. EMS and EMG alternate such that the EMG input signal is measured in the gaps of the EMS output signal, so voluntary muscle contraction is measured during muscle stimulation. Notifications are represented as EMS signals and are accepted or refused either by a directional or a time-based EMG response. A lab user study with 12 participants shows that the directional EMG response is superior to the time-based response in terms of reaction time, error rate, and user preference. Furthermore, the directional approach is the fastest and the most intuitive for users compared to a button-based smartwatch interface as a baseline.

Digital pens emit ink on paper and digitize handwriting. The range of the pen is typically limited to a special writing surface on which the pen's tip is tracked. We present Pentelligence, a pen for handwritten digit recognition that operates on regular paper and does not require a separate tracking device. It senses the pen tip's motions and sound emissions when stroking. Pen motions and writing sounds exhibit complementary properties. Combining both types of sensor data substantially improves the recognition rate. Hilbert envelopes of the writing sounds and mean-filtered motion data are fed to neural networks for majority voting. The results on a dataset of 9408 handwritten digits taken from 26 individuals show that motion+sound outperforms single-sensor approaches at an accuracy of 78.4% for 10 test users. Retraining the networks for a single writer on a dataset of 2120 samples increased the precision to 100% for single handwritten digits at an overall accuracy of 98.3%.

Tactile patterns are a means to convey general direction information to pedestrians (for example when turning right) and specific navigation instructions (for example when approaching the stairs). Tactile patterns are especially helpful for people with visual impairments in navigation scenarios and can also be used to deliver general notifications. This workshop paper is supposed to spark a discussion within the workshop about correctly identifying requirements and other needs of the visually impaired population in order to create a useful guidance tool to eventually replace the white cane as a primary navigation tool for the visually impaired.

Modern technologies enable data analysis in scenarios where keyboard and mouse are not available. Research on multimodality in visual analytics is facing this challenge. But existing approaches consider exclusively static environments with large screens. Therefore, we envision Valletto, a prototypical tablet app which allows the user to generate and specify visualizations through a speech-based conversational interface, through multitouch gestures, and through a conventional GUI interface. We conducted an initial expert evaluation to gain information on the modality function mapping and for the integration of different modalities. Our aim is to discuss design and interaction considerations in a mobile context which fits the user's daily life.

Emoji, a set of pictographic Unicode characters, have seen strong uptake over the last couple of years. All common mobile platforms and many desktop systems now support emoji entry and users have embraced their use. Yet, we currently know very little about what makes for good emoji entry. While soft keyboards for text entry are well optimized, based on language and touch models, no such information exists to guide the design of emoji keyboards. In this article, we investigate of the problem of emoji entry, starting with a study of the current state of the emoji keyboard implementation in Android. To enable moving forward to novel emoji keyboard designs, we then explore a model for emoji similarity that is able to inform such designs. This semantic model is based on data from 21 million collected tweets containing emoji. We compare this model against a solely description-based model of emoji in a crowdsourced study. Our model shows good performance in capturing detailed relationships between emoji.

The human body reveals emotional and bodily states through measurable signals, such as body language and electroencephalography. However, such manifestations are difficult to communicate to others remotely. We propose EmotionActuator, a proof-of-concept system to investigate the transmission of emotional states in which the recipient performs emotional gestures to understand and interpret the state of the sender.We call this kind of communication embodied emotional feedback, and present a prototype implementation. To realize our concept we chose four emotional states: amused, sad, angry, and neutral. We designed EmotionActuator through a series of studies to assess emotional classification via EEG, and create an EMS gesture set by comparing composed gestures from the literature to sign-language gestures. Through a final study with the end-to-end prototype interviews revealed that participants like implicit sharing of emotions and find the embodied output to be immersive, but want to have control over shared emotions and with whom. This work contributes a proof of concept system and set of design recommendations for designing embodied emotional feedback systems.

Current virtual and augmented reality head-mounted displays usually include no or only a single vibration motor for haptic feedback and do not use it for guidance. We present HapticHead, a system utilizing multiple vibrotactile actuators distributed in three concentric ellipses around the head for intuitive haptic guidance through moving tactile cues. We conducted three experiments, which indicate that HapticHead vibrotactile feedback is both faster (2.6 s vs. 6.9 s) and more precise (96.4% vs. 54.2% success rate) than spatial audio (generic head-related transfer function) for finding visible virtual objects in 3D space around the user. The baseline of visual feedback is as expected more precise (99.7% success rate) and faster (1.3 s) in comparison, but there are many applications in which visual feedback is not desirable or available due to lighting conditions, visual overload, or visual impairments. Mean final precision with HapticHead feedback on invisible targets is 2.3° compared to 0.8° with visual feedback. We successfully navigated blindfolded users to real household items at different heights using HapticHead vibrotactile feedback independently of a head-mounted display.

Exploratory data analysis is an essential step in discovering patterns and relationships in data. However, the exploration may start without a clear conception about what attributes to pick or what visualizations to choose in order to develop an understanding of the data. In this work we aim to support the exploration process by automatically choosing attributes according to an information-theoretic measure and by providing a simple means of navigation through the space of visualizations. The system suggests data attributes to be visualized and the visualization's type and appearance. The user intuitively modifies these suggestions by performing swiping gestures on a tablet device. Attribute suggestions are based on the mutual information between multiple random variables (MMI). The results of a preliminary user study (N = 12 participants) show the applicability of MMI for guided exploratory data analysis and confirm the system's general usability (SUS score: 74).

A high level of presence is an important aspect of immersive virtual reality applications. However, presence is difficult to achieve as it depends on the individual user, immersion capabilities of the system (visual, auditory, and tactile) and the concrete application. We use a vibrotactile grid around the head in order to further increase the level of presence users feel in virtual reality scenes. In a between-groups comparison study the vibrotactile group scored significantly higher in a standardized presence questionnaire compared to the baseline of no tactile feedback. This suggests the proposed prototype as an additional tool to increase the level of presence users feel in virtual reality scenes.

Compression feedback uses inflatable straps to create uniform pressure sensations around limbs. Lower-pressure stimuli are well suited as a feedback channel for, e.g., notifications. However, operating compression feedback systems at higher pressure levels allows to physically inhibit movement. Here, we describe this modality and present a pervasive jogging game that employs physical inhibition to push runners to reach checkpoints in time.

Current mobile devices commonly use vibration feedback to signal incoming notifications. However, vibration feedback exhibits strong attention capture, limiting its use to short periods and prominent notifications. Instead, we investigate the use of compression feedback for notifications, which scales from subtle stimuli to strong ones and can provide sustained stimuli over longer periods. Compression feedback utilizes inflatable straps around a user's limbs, a form factor allowing for easy integration into many common wearables. We explore technical aspects of compression feedback and investigate its psychophysical properties with several lab and in situ studies. Furthermore, we show how compression feedback enables reactive feedback. Here, deflation patterns are used to reveal further information on a user's query. We also compare compression and vibrotactile feedback and find that they have similar performance.

Electrical muscle stimulation (EMS) has been used successfully in HCI to generate force feedback and simple movements both in stationary and mobile settings. However, many natural limb movements require the coordinated actuation of multiple muscles. Off-the-shelf EMS devices are typically limited in their ability to generate fine-grained movements, because they only have a low number of channels and do not provide full control over the EMS parameters. More capable medical devices are not designed for mobile use or still have a lower number of channels and less control than is desirable for HCI research. In this paper we present the concept and a prototype of a 20-channel mobile EMS system that offers full control over the EMS parameters. We discuss the requirements of wearable multi-electrode EMS systems and present the design and technical evaluation of our prototype. We further outline several application scenarios and discuss safety and certification issues.

Electrical muscle stimulation (EMS) is a promising wearable haptic output technology as it can be miniaturized and delivers a wide range of tactile and force output. However, prototyping EMS applications is currently challenging and requires detailed knowledge about EMS. We present a toolkit that simplifies prototyping with EMS and serves as a starting point for experimentation and user studies. It consists of (1) a hardware control module that uses off-the-shelf EMS devices as safe signal generators, (2) a simple communication protocol, and (3) a set of control applications for prototyping. The interactivity allows hands-on experimentation with our sample control applications.

Current soft keyboards for emoji entry all present emoji in the same way: in long lists, spread over several categories. While categories limit the number of emoji in each individual list, the overall number is still so large, that emoji entry is a challenging task. The task takes particularly long if users pick the wrong category when searching for an emoji. Instead, we propose a new zooming keyboard for emoji entry. Here, users can see all emoji at once, aiding in building spatial memory where related emoji are to be found. We compare our zooming emoji keyboard against the Google keyboard and find that our keyboard allows for 18% faster emoji entry, reducing the required time for one emoji from 15.6s to 12.7s. A preliminary longitudinal evaluation with three participants showed that emoji entry time over the duration of the study improved at up to 60% to a final average of 7.5s.

In mobile contexts guidance towards objects is usually done through the visual channel. Sometimes this channel is overloaded or not appropriate. A practicable form of haptic feedback is challenging. Electrical muscle stimulation (EMS) can generate mobile force feedback but has a number of drawbacks. For complex movements several muscles need to be actuated in concert and a feedback loop is necessary to control movements. We present an approach that only requires the actuation of six muscles with four pairs of electrodes to guide the index finger to a 2D point and let the user perform mid-air disambiguation gestures. In our user study participants found invisible, static target positions on top of a physical box with a mean 2D deviation of 1.44 cm from the intended target.

In this course, participants create their own prototypes using electrical-muscle stimulation. We provide a ready-to-use device and toolkit consisting of electrodes, microcontroller, and an off-the-shelve muscle stimulator that allows for programmatically actuating the user's muscles directly from mobile devices.

Current generation virtual reality (VR) and augmented reality (AR) head-mounted displays (HMDs) usually include no or only a single vibration motor for haptic feedback and do not use it for guidance. We present HapticHead, a system utilizing 20 vibration motors distributed in three concentric ellipses around the head to give intuitive haptic guidance hints and to increase immersion for VR and AR applications. Our user study indicates that HapticHead is both faster (mean=3.7s, SD=2.3s vs. mean=7.8s, SD=5.0s) and more precise (92.7% vs. 44.9% hit rate) than auditory feedback for the purpose of finding virtual objects in 3D space around the user. The baseline of visual feedback is as expected more precise (99.9% hit rate) and faster (mean=1.5s, SD=0.6s) in comparison but there are many applications in which visual feedback is not desirable or available due to lighting conditions, visual overload, or visual impairments.

Electrical muscle stimulation (EMS) is a promising wearable haptic output technology as it can be miniaturized considerably and delivers a wide range of haptic output. However, prototyping EMS applications is challenging. It requires detailed knowledge and skills about hardware, software, and physiological characteristics. To simplify prototyping with EMS in mobile and wearable situations we present the Let Your Body Move toolkit. It consists of (1) a hardware control module with Bluetooth communication that uses off-the-shelf EMS devices as signal generators, (2) a simple communications protocol to connect mobile devices, and (3) a set of control applications as starting points for EMS prototyping. We describe EMS-specific parameters, electrode placements on the skin, and user calibration. The toolkit was evaluated in a workshop with 10 researchers in haptics. The results show that the toolkit allows to quickly generate non-trivial prototypes. The hardware schematics and software components are available as open source software.

Interaction with mobile devices currently requires close engagement with them. For example, users need to pick them up and unlock them, just to check whether the last notification was for an urgent message. But such close engagement is not always desirable, e.g., when working on a project with the phone just laying around on the table. Instead, we explore around-device interactions to bring up and control notifications. As users get closer to the device, more information is revealed and additional input options become available. This allows users to control how much they want to engage with the device. For feedback, we use a custom LED-matrix display prototype on the edge of the device. This allows for coarse, but bright, notifications in the periphery of attention, but scales up to allow for slightly higher resolution feedback as well.

Electromyography (EMG) and electrical muscle stimulation (EMS) are promising technologies for muscle sensing and actuation in wearable interfaces. The required electrodes can be manufactured to form a thin layer on the skin. We discuss requirements and approaches for EMG and EMS as on-skin technologies. In particular, we focus on fine-grained muscle sensing and actuation with an electrode grid on the lower arm. We discuss a prototype, scenarios, and open issues.

With the increasing popularity of smartwatches over the last years, there has been a substantial interest in novel input methods for such small devices. However, feedback modalities for smartwatches have not seen the same level of interest. This is surprising, as one of the primary function of smartwatches is their use for notifications. It is the interrupting nature of current notifications on smartwatches that has also drawn some of the more critical responses to them. Here, we present a subtle notification mechanism for smartwatches that uses light scattering in a wearer's skin as a feedback modality. This does not disrupt the wearer in the same way as vibration feedback and also connects more naturally with the user's body.

Current generation virtual reality (VR) and augmented reality (AR) head-mounted displays (HMDs) usually include no or only a single vibration motor for haptic feedback and do not use it for guidance. In a previous work, we presented HapticHead, a potentially mobile system utilizing vibration motors distributed in three concentric ellipses around the head to give intuitive haptic guidance hints and to increase immersion for VR and AR applications. The purpose of this paper is to explore potential application scenarios and aesthetic possibilities of the proposed concept in order to create an active discussion amongst workshop participants.

In relaxed living room settings, using a phone to control the room can be inappropriate or cumbersome. Instead of such explicit interactions, we enable implicit control via a posture-sensing couch. Users can then, e.g., automatically turn on the reading lights when sitting down.

Pedestrian navigation systems require users to perceive, interpret, and react to navigation information. This can tax cognition as navigation information competes with information from the real world. We propose actuated navigation, a new kind of pedestrian navigation in which the user does not need to attend to the navigation task at all. An actuation signal is directly sent to the human motor system to influence walking direction. To achieve this goal we stimulate the sartorius muscle using electrical muscle stimulation. The rotation occurs during the swing phase of the leg and can easily be counteracted. The user therefore stays in control. We discuss the properties of actuated navigation and present a lab study on identifying basic parameters of the technique as well as an outdoor study in a park. The results show that our approach changes a user's walking direction by about 16°/m on average and that the system can successfully steer users in a park with crowded areas, distractions, obstacles, and uneven ground.

Haptic feedback allows leveraging other faculties such as proprioception instead of using the visual sense, which is often overloaded with traditional UIs. However, most haptic technologies have been away from the current trend in Human-Computer Interaction (HCI) which is miniaturization (eg, mobile, wearable). Therefore haptic techniques, such as force feedback, tend to stay inside labs. In fact, most haptic devices resist miniaturization because they require physical motors and mechanics which do not scale down easily. Researchers have proposed miniaturizing and simplifying haptic devices by using electrical-muscle stimulation as to actuate the muscles directly, rather than actuating through mechanics. Electrical muscle stimulation (EMS) uses a small current to elicit action on the motor fibers/nerves, causing an involuntary contraction on the user’s body.

We present a novel way to recognize users by the way they press a button. Our approach allows low-effort and fast interaction without the need for augmenting the user or controlling the environment. It eschews privacy concerns of methods such as fingerprint scanning. Button pressing behavior is sufficiently discriminative to allow distinguishing users within small groups. This approach combines recognition and action in a single step, e.g., getting and tallying a coffee can be done with one button press. We deployed our system for 5 users over a period of 4 weeks and achieved recognition rates of 95% in the last week. We also ran a larger scale but short-term evaluation to investigate effects of group size and found that our method degrades gracefully for larger groups.

The amount of logistical data in the automotive industry drastically increases due to digitalization and data that is automatically generated due to Auto-ID-Technologies. However, new methods need to be devised to make sense of this data, in particular when users are mobile, and when users need to collaborate to solve complex logistical tasks, such as resource scheduling. We propose a visualization method for hierarchical event data that is designed for tablets. The main design goals have been to foster collaboration and enable mobility. Our think aloud user study shows that both the event recognition and understanding of the participants improved with the proposed solution.

Most common forms of haptic feedback use vibration, which immediately captures the user's attention, yet is limited in the range of strengths it can achieve. Vibration feedback over extended periods also tends to be annoying. We present compression feedback, a form of haptic feedback that scales from very subtle to very strong and is able to provide sustained stimuli and pressure patterns. The demonstration may serve as an inspiration for further work in this area, applying compression feedback to generate subtle, intimate, as well as intense feedback.

Free-hand interaction becomes a common technique for interacting with large displays. At the same time, providing haptic feedback for free-hand interaction is still a challenge, particularly feedback with different characteristics (i.e., strengths, patterns) to convey particular information. We see electrical muscle stimulation (EMS) as a well-suited technology for providing haptic feedback in this domain. The characteristics of EMS can be used to assist users in learning, manipulating, and perceiving virtual objects. One of the core challenges is to understand these characteristics and how they can be applied. As a step in this direction, this paper presents a design space that identifies different aspects of using EMS for haptic feedback. The design space is meant as a basis for future research investigating how particular characteristics can be exploited to provide specific haptic feedback.

For many people their phones have become their main everyday tool. While phones can fulfill many different roles they also require users to (1) make do with affordance not specialized for the specific task, and (2) closely engage with the device itself. We propose utilizing the space and objects around the phone to offer better task affordance and to create an opportunity for casual interactions. Such around-device devices are a class of interactors that do not require users to bring special tangibles, but repurpose items already found in the user's surroundings. In a survey study, we determine which places and objects are available to around-device devices. Furthermore, in an elicitation study, we observe what objects users would use for ten interactions.

When interacting casually, users relinquish some control over their interaction to gain the freedom to devote their engagement elsewhere. This allows them to still interact even when they are encumbered, distracted, or engaging with others. With their focus on something else, casual interaction will often take place in the periphery---either spatially by, e.g., interacting laterally or with respect to attention, by interacting in the background.

The thumb and the fingers have different flexibility, and thus, gestures performed on the back of a held tablet are suggested to be different from ones performed on the touchscreen with the thumb of grasping hands. APIs for back-of-device gesture detection should consider that difference. In a user study, we recorded vectors for the four most common touch gestures. We found that drag, swipe, and press gestures are significantly differently when executed on the back versus on the front side of a held tablet. Corresponding values are provided that may be used to define gesture detection thresholds for back-of-tablet interaction.

Free-hand interaction with large displays is getting more common, for example in public settings and exertion games. Adding haptic feedback offers the potential for more realis- tic and immersive experiences. While vibrotactile feedback is well known, electrical muscle stimulation (EMS) has not yet been explored in free-hand interaction with large displays. EMS offers a wide range of different strengths and qualities of haptic feedback. In this paper we first systematically inves- tigate the design space for haptic feedback. Second, we ex- perimentally explore differences between strengths of EMS and vibrotactile feedback. Third, based on the results, we evaluate EMS and vibrotactile feedback with regard to differ- ent virtual objects (soft, hard) and interaction with different gestures (touch, grasp, punch) in front of a large display. The results provide a basis for the design of haptic feedback that is appropriate for the given type of interaction and the material.

Motivated by the addition of gyroscopes to a large number of new smart phones, we study the effects of combining accelerometer and gyroscope data on the recognition rate of motion gesture recognizers with dimensionality constraints. Using a large data set of motion gestures we analyze results for the following algorithms: Protractor3D, Dynamic Time Warping (DTW) and Regularized Logistic Regression (LR). We chose to study these algorithms because they are relatively easy to implement, thus well suited for rapid prototyping or early deployment during prototyping stages. For use in our analysis, we contribute a method to extend Protractor3D to work with the 6D data obtained by combining accelerometer and gyroscope data. Our results show that combining accelerometer and gyroscope data is beneficial also for algorithms with dimensionality constraints and improves the gesture recognition rate on our data set by up to 4\%.

We present a wearable interface that consists of motion sensors. As the interface can be worn on the user's fingers (as a ring) or fixed to it (with nail polish), the device controlled by finger gestures can be any generic object, provided they have an interface for receiving the sensor's signal. We implemented four gestures: tap, release, swipe, and pitch, all of which can be executed with a finger of the hand holding the device. In a user study we tested gesture appropriateness for the index finger at the back of a handheld tablet that offered three different form factors on its rear: flat, convex, and concave (undercut). For all three shapes, the gesture performance was equally good, however pitch performed better on all surfaces than swipe. The proposed interface is an example towards the idea of ubiquitous computing and the vision of seamless interactions with grasped objects. As an initial application scenario we implemented a camera control that allows the brightness to be configured using our tested gestures on a common SLR device.

Mobile handheld projectors in small form factors, e.g., integrated into mobile phones, are getting more common. However, managing the projection puts a burden on the user as it requires holding the hand steady over an extended period of time and draws attention away from the actual task to solve. To address this problem, we propose a body worn projector that follows the user's locus of attention. The idea is to take the user's hand and dominant ngers as an indication of the current locus of attention and focus the projection on that area. Technically, a wearable and steerable camera-projector system positioned above the shoulder tracks the ngers and follows their movement. In this paper, we justify our approach and explore further ideas on how to apply steerable projection for wearable interfaces. Additionally, we describe a Kinect-based prototype of the wearable and steerable projector system we developed.

Rotating 3D objects is a diffcult task on mobile devices, because the task requires 3 degrees of freedom and (multi-)touch input only allows for an indirect mapping. We propose a novel style of mobile interaction based on mid-air gestures in proximity of the device to increase the number of DOFs and alleviate the limitations of touch interaction with mobile devices. While one hand holds the device, the other hand performs mid-air gestures in proximity of the device to control 3D objects on the mobile device's screen. A at hand pose de nes a virtual surface which we refer to as the PalmSpace for precise and intuitive 3D rotations. We constructed several hardware prototypes to test our interface and to simulate possible future mobile devices equipped with depth cameras. Pilot tests show that PalmSpace hand gestures are feasible. We conducted a user study to compare 3D rotation tasks using the most promising two designs for the hand location during interaction - behind and beside the device - with the virtual trackball, which is the current state-of-art technique for orientation manipulation on touchscreens. Our results show that both variants of PalmSpace have signi cantly lower task completion times in comparison to the virtual trackball.

When the user is engaged with a real-world task it can be inappropriate or difficult to use a smartphone. To address this concern, we developed ShoeSense, a wearable system consisting in part of a shoe-mounted depth sensor pointing upward at the wearer. ShoeSense recognizes relaxed and discreet as well as large and demonstrative hand gestures. In particular, we designed three gesture sets (Triangle, Radial, and Finger-Count) for this setup, which can be performed without visual attention. The advantages of ShoeSense are illustrated in five scenarios: (1) quickly performing frequent operations without reaching for the phone, (2) discreetly performing operations without disturbing others, (3) enhancing operations on mobile devices, (4) supporting accessibility, and (5) artistic performances. We present a proof-of-concept, wearable implementation based on a depth camera and report on a lab study comparing social acceptability, physical and mental demand, and user preference. A second study demonstrates a 94-99% recognition rate of our recognizers.

Graspable tangibles are now being explored on the current generation of capacitive touch surfaces, such as the iPad and the Android tablet. Because the size and form factor is relatively new, early and low fidelity prototyping of these TUIs is crucial in getting the right design. The problem is that it is difficult for the average interaction designer to develop such physical prototypes. They require a substantial amount time and effort to physically model the tangibles, and expertise in electronics to instrument them. Thus prototyping is sometimes handed off to specialists, or is limited to only a few design iterations and alternative designs. Our solution contributes a low fidelity prototyping approach that is time and cost effective, and that requires no electronics knowledge. First, we supply non-specialists with cardboard forms to create tangibles. Second, we have them draw lines on it via conductive ink, which makes their objects recognizable by the capacitive touch screen. They can then apply routine programming to recognize these tangibles and thus iterate over various designs.

This paper explores how microinteractions such as hand gestures allow executing a secondary task, e.g. controlling mobile applications and devices, without interrupting the manual primary tasks, for instance driving a car. We asked sports- and physiotherapists for using props while interviewing these experts in order to iteratively design microgestures. The required gestures should be easily performable without interrupting the primary task, without needing high cognitive effort, and without taking the risk of being mixed up with natural movements. Resulting from the expert interviews we developed a taxonomy for classifying these gestures according to their use cases and assess their ergonomic and cognitive attributes, focusing on their primary task compatibility. We defined 21 hand gestures, which allow microinteractions within manual dual task scenarios. In expert interviews we evaluated their level of required motor or cognitive resources under the constraint of stable primary task performance. Our taxonomy poses a basis for designing microinteraction techniques.

We present CapWidgets, passive tangible controls for capacitive touch screens. CapWidgets bring back physical controls to off-the-shelf multi-touch surfaces as found in mobile phones and tablet computers. While the user touches the widget, the surface detects the capacitive marker on the widget's underside. We study the relative performance of this tangible interaction with direct multi-touch interaction and our experimental results show that user performance and preferences are not automatically in favor of tangible widgets and careful design is necessary to validate their properties.

The information a small mobile device can show via its display has been always limited by its size. In large information spaces, relevant information, such as important locations on a map can get clipped when a user starts zooming and panning. Dynamic ambient lighting allows mobile devices to visualize off-screen objects by illuminating the background without compromising valuable display space. The lighted spots can be used to show the direction and distance of such objects by varying the spot's position and intensity. Dynamic ambient lighting also provides a new way of displaying the state of a mobile device. Illumination is provided by a prototype rear of device shell which contains LEDs and requires the device to be placed on a surface, such as a table or desk.

Rohs and Oulasvirta (2008) proposed a two-component Fitts' law model for target acquisition with magic lenses in mobile augmented reality (AR) with 1) a physical pointing phase, in which the target can be directly observed on the background surface, and 2) a virtual pointing phase, in which the target can only be observed through the device display. The model provides a good fit (R2=0.88) with laboratory data, but it is not known if it generalizes to real-world AR tasks. In the present outdoor study, subjects (N=12) did building-selection tasks in an urban area. The differences in task characteristics to the laboratory study are drastic: targets are three-dimensional and they vary in shape, size, z-distance, and visual context. Nevertheless, the model yielded an R2 of 0.80, and when using effective target width an R2 of 0.88 was achieved.

Protractor 3D is a gesture recognizer that extends the 2D touch screen gesture recognizer Protractor to 3D gestures. It inherits many of Protractor's desirable properties, such as high recognition rate, low computational and low memory requirements, ease of implementation, ease of customization, and low number of required training samples. Protractor 3D is based on a closed-form solution to finding the optimal rotation angle between two gesture traces involving quaternions. It uses a nearest neighbor approach to classify input gestures. It is thus well-suited for application in resource-constrained mobile devices. We present the design of the algorithm and a study that evaluated its performance.

Even with the rise of the World Wide Web, TV has remained the most pervasive entertainment medium and is nowadays often used together with other media, which allow for active participation. The idea of connecting non-collocated TV viewers via telecommunication technologies, referred to as Social TV, has recently received considerable attention. Such systems typically include set-top boxes for supporting collaboration. In this research we investigate if real-time opinion sharing about TV shows through a nonverbal (non-textual) iconic UI on mobile phones is reasonable. For this purpose we developed a mobile app, made it available to a large number of users through the Android Market, and conducted an uncontrolled user study in the wild during the soccer world cup 2010. The results of the study indicate that TV viewers who used the app had more fun and felt more connected to other viewers. We also show that by monitoring this channel it is possible to collect sentiments relevant to the broadcasted content in real-time. The collected data exemplify that the aggregated sentiments correspond to important moments, and hence can be used to generate a summary of the event.

Mobile devices are increasingly used in social networking applications and research. So far, there is little work on real-time emotion or opinion sharing in large loosely coupled user communities. One potential area of application is the assessment of widely broadcasted television TV shows. The idea of connecting non-collocated TV viewers via telecommunication technologies is referred to as Social TV. Such systems typically include set-top boxes for supporting the collaboration. In this work the authors investigated whether mobile phones can be used as an additional channel for sharing opinions, emotional responses, and TV-related experiences in real-time. To gain insight into this area, an Android app was developed for giving real-time feedback during soccer games and to create ad hoc fan groups. This paper presents results on rating activity during games and discusses experiences with deploying this app over four weeks during soccer World Cup. In doing so, challenges and opportunities faced are highlighted and an outlook on future work in this area is given.

We present the $3 Gesture Recognizer, a simple but robust gesture recognition system for input devices featuring 3D acceleration sensors. The algorithm is designed to be implemented quickly in prototyping environments, is intended to be device-independent and does not require any special toolkits or frameworks. It relies solely on simple trigonometric and geometric calculations. A user evaluation of our system resulted in a correct gesture recognition rate of 80%, when using a set of 10 unique gestures for classification. Our method requires significantly less training data than other gesture recognizers and is thus suited to be deployed and to deliver results rapidly.

We present a tabletop-based system that supports rapid paper-based prototyping for mobile applications. Our system combines the possibility of manually sketching interface screens on paper with the ability to define dynamic interface behavior through actions on the tabletop. This not only allows designers to digitize interface sketches for paper prototypes, but also enables the generation of prototype applications able to run on target devices. By making physical and virtual interface sketches interchangeable, our system greatly enhances and speeds up the development of mobile applications early in the interface design process.

We conducted a series of user studies to understand and clarify the fundamental characteristics of pressure in user interfaces for mobile devices. We seek to provide insight to clarify a longstanding discussion on mapping functions for pressure input. Previous literature is conflicted about the correct transfer function to optimize user performance. Our study results suggest that the discrepancy can be explained by different signal conditioning circuitry and with improved signal conditioning the user-performed precision relationship is linear. We also explore the effects of hand pose when applying pressure to a mobile device from the front, the back, or simultaneously from both sides in a pinching movement. Our results indicate that grasping type input outperforms single-sided input and is competitive with pressure input against solid surfaces. Finally we provide an initial exploration of non-visual multimodal feedback, motivated by the desire for eyes-free use of mobile devices. The findings suggest that non-visual pressure input can be executed without degradation in selection time but suffers from accuracy problems.

Interaction with 3D objects and scenes is becoming increasingly important on mobile devices. We explore 3D object rotation as a fundamental interaction task. We propose an extension of the virtual trackball metaphor, which is typically restricted to a half sphere and single-sided interaction, to actually use a full sphere. The extension is enabled by a hardware setup called the ¿iPhone Sandwich,¿ which allows for simultaneous front-and-back touch input. This setup makes the rear part of the virtual trackball accessible for direct interaction and thus achieves the realization of the virtual trackball metaphor to its full extent. We conducted a user study that shows that a back-of-device virtual trackball is as effective as a front-of-device virtual trackball and that both outperform an implementation of tilt-based input.

User interfaces for mobile devices move away from mainly button- and menu-based interaction styles and towards more direct techniques, involving rich sensory input and output. The recently proposed concept of Natural User Interfaces (NUIs) provides a way to structure the discussion about these developments. We examine how two-sided and around-device interaction, gestural input, and shape- and weight-based output can be used to create NUIs for mobile devices. We discuss the applicability of NUI properties in the context of mobile interaction.

In this paper we present a novel interface for mobile map navigation based on Semi-Automatic Zooming (SAZ). SAZ gives the user the ability to manually control the zoom level of an SDAZ interface, while retaining the automatic zooming characteristics of that interface at times when the user is not explicitly controlling the zoom level. In a user study conducted using a realistic mobile map with a wide scale space, we compare SAZ with existing map interface techniques, multi-touch and Speed-Dependent Automatic Zooming (SDAZ). We extend a dynamic state-space model for Speed-Dependent Automatic Zooming (SDAZ) to accept 2D tilt input for scroll rate and zoom level control and implement a dynamically zoomable map view with access to high-resolution map material for use in our study. The study reveals that SAZ performs significantly better than SDAZ and that SAZ is comparable in performance and usability to a standard multi-touch map interface. Furthermore, the study shows that SAZ could serve as an alternative to multi-touch as input technique for mobile map interfaces.

Impact force is an important dimension for percussive musical instruments such as the piano. We explore three possible mechanisms how to get impact forces on mobile multi-touch devices: using built-in accelerometers, the pressure sensing capability of Android phones, and external force sensing resistors. We find that accelerometers are difficult to control for this purpose. Android's pressure sensing shows some promise, especially when combined with augmented playing technique. Force sensing resistors can offer good dynamic resolution but this technology is not currently offered in commodity devices and proper coupling of the sensor with the applied impact is difficult.

Gestures can offer an intuitive way to interact with a computer. In this paper, we investigate the question whether gesturing with a mobile phone can help to perform complex tasks involving two devices. We present results from a user study, where we asked participants to spontaneously produce gestures with their phone to trigger a set of different activities. We investigated three conditions (device configurations): phone-to-phone, phone-to-tabletop, and phone to public display. We report on the kinds of gestures we observed as well as on feedback from the participants, and provide an initial assessment of which sensors might facilitate gesture recognition in a phone. The results suggest that phone gestures have the potential to be easily understood by end users and that certain device configurations and activities may be well suited for gesture control.

Mobile devices are increasingly used in social networking applications. So far, there is little work on real-time emotion and opinion sharing in large loosely-coupled user communities. We present an Android app for giving realtime feedback during soccer games and to create ad hoc fan groups. We discuss our experiences with deploying this app over four weeks during 2010 soccer world cup. We highlight challenges and opportunities we faced and give recommendations for future work in this area.

Passing around stacks of paper photographs while sitting around a table is one of the key social practices defining what is commonly referred to as the ‘Kodak Generation’. Due to the way digital photographs are stored and handled, this practice does not translate well to the ‘Flickr Generation’, where collocated photo sharing often involves the (wireless) transmission of a photo from one mobile device to another. In order to facilitate ‘cross-generation’ sharing without enforcing either practice, it is desirable to bridge this gap in a way that incorporates familiar aspects of both. In this paper, we discuss a novel interaction technique that addresses some of the constraints introduced by current communication technology, and that enables photo sharing in a way, which resembles the passing of stacks of paper photographs. This technique is based on dynamically generated spatial regions around mobile devices and has been evaluated through two user studies. The results we obtained indicate that our technique is easy to learn and as fast, or faster than, current technology such as transmitting photos between devices using Bluetooth. In addition, we found evidence of different sharing techniques influencing social practice around photo sharing. The use of our technique resulted in a more inclusive and group-oriented behavior in contrast to Bluetooth photo sharing, which resulted in a more fractured setting composed of sub-groups.

In this paper we explore the design space of around-device interaction (ADI). This approach seeks to expand the interaction possibilities of mobile and wearable devices beyond the confines of the physical device itself to include the space around it. This enables rich 3D input, comprising coarse movement-based gestures, as well as static position-based gestures. ADI can help to solve occlusion problems and scales down to very small devices. We present a novel around-device interaction interface that allows mobile devices to track coarse hand gestures performed above the device's screen. Our prototype uses infrared proximity sensors to track hand and finger positions in the device's proximity. We present an algorithm for detecting hand gestures and provide a rough overview of the design space of ADI-based interfaces.

We conducted a user study to investigate the effect of visual context in handheld augmented reality interfaces. A dynamic peephole interface (without visual context beyond the device display) was compared to a magic lens interface (with video see-through augmentation of external visual context). The task was to explore objects on a map and look for a specific attribute shown on the display. We tested different sizes of visual context as well as different numbers of items per area, i.e. different item densities. We found that visual context is most effective for sparse item distributions and the performance benefit decreases with increasing density. User performance in the magic lens case approaches the performance of the dynamic peephole case the more densely spaced the items are. In all conditions, subjective feedback indicates that participants generally prefer visual context over the lack thereof. The insights gained from this study are relevant for designers of mobile AR and dynamic peephole interfaces by suggesting when external visual context is most beneficial.

This article reports on two user studies investigating the effect of visual context in handheld augmented reality interfaces. A dynamic peephole interface (without visual context beyond the device display) was compared to a magic lens interface (with video see-through augmentation of external visual context). The task was to explore items on a map and look for a specific attribute. We tested different sizes of visual context as well as different numbers of items per area, i.e. different item densities. Hand motion patterns and eye movements were recorded. We found that visual context is most effective for sparsely distributed items and gets less helpful with increasing item density. User performance in the magic lens case is generally better than in the dynamic peephole case, but approaches the performance of the latter the more densely the items are spaced. In all conditions, subjective feedback indicates that participants generally prefer visual context over the lack thereof. The insights gained from this study are relevant for designers of mobile AR and dynamic peephole interfaces, involving spatially tracked personal displays or combined personal and public displays, by suggesting when to use visual context.

Efficient and effective communication between mobile units and the central emergency operation center is a key factor to respond successfully to the challenges of emergency management. Nowadays, the only ubiquitously available modality is a voice channel through mobile phones or radio transceivers. This makes it often very difficult to convey exact geographic locations and can lead to misconceptions with severe consequences, such as a fire brigade heading to the right street address in the wrong city. In this paper we describe a handheld augmented reality approach to support the communication of spatial information in a crisis response scenario. The approach combines mobile camera devices with paper maps to ensure a quick and reliable exchange of spatial information.

Mobile phones offer an attractive platform for interactive music performance. We provide a theoretical analysis of the sensor capabilities via a design space and show concrete examples of how different sensors can facilitate interactive performance on these devices. These sensors include cameras, microphones, accelerometers, magnetometers and multitouch screens. The interactivity through sensors in turn informs aspects of live performance as well as composition though persistence, scoring, and mapping to musical notes or abstract sounds.

With the miniaturization of projection technology the integration of tiny projection units, normally referred to as pico projectors, into mobile devices is not longer ction. Such integrated projectors in mobile devices could make mobile projection ubiquitous within the next few years. These phones soon will have the ability to project large-scale information onto any surfaces in the real world. By doing so the interaction space of the mobile device can be expanded to physical objects in the environment and this can support interaction concepts that are not even possible on modern desktop computers today. In this paper, we explore the possibilities of camera projector phones with a mobile adaption of the Playstation3 game LittleBigPlanet. The camera projector unit is used to augment the hand drawings of a user with an overlay displaying physical interaction of virtual objects with the real world. Players can sketch a 2D world on a sheet of paper or use an existing physical configuration of objects and let the physics engine simulate physical procedures in this world to achieve game goals.

The advantages of paper-based maps have been utilized in the field of mobile Augmented Reality (AR) in the last few years. Traditional paper-based maps provide high-resolution, large-scale information with zero power consumption. There are numerous implementations of magic lens interfaces that combine high-resolution paper maps with dynamic handheld displays. From an HCI perspective, the main challenge of magic lens interfaces is that users have to switch their attention between the magic lens and the information in the background. In this paper, we attempt to overcome this problem by using a lightweight mobile camera projector unit to augment the paper map directly with additional information. The "Map Torchlight" is tracked over a paper map and can precisely highlight points of interest, streets, and areas to give directions or other guidance for interacting with the map.

In many mid- to large-sized cities public maps are ubiquitous. One can also find a great number of maps in parks or near hiking trails. Public maps help to facilitate orientation and provide special information to not only tourists but also to locals who just want to look up an unfamiliar place while on the go. These maps offer many advantages compared to mobile maps from services like Google Maps Mobile or Nokia Maps. They often show local landmarks and sights that are not shown on standard digital maps. Often these 'You are here' (YAH) maps are adapted to a special use case, e.g. a zoo map or a hiking map of a certain area. Being designed for a fashioned purpose these maps are often aesthetically well designed and their usage is therefore more pleasant. In this paper we present a novel technique and application called PhotoMap that uses images of 'You are here' maps taken with a GPS-enhanced mobile camera phone as background maps for on-the-fly navigation tasks. We discuss different implementations of the main challenge, namely helping the user to properly georeference the taken image with sufficient accuracy to support pedestrian navigation tasks. We present a study that discusses the suitability of various public maps for this task and we evaluate if these georeferenced photos can be used for navigation on GPS-enabled devices.

Two-sided pressure input is common in everyday interactions such as grabbing, sliding, twisting, and turning an object held between thumb and index finger. We describe and demonstrate a research prototype which allows for twosided multitouch sensing with continuous pressure input at interactive rates and we explore early ideas of interaction techniques that become possible with this setup. The advantage of a two-sided pressure interaction is that it enables high degree-of-freedom input locally. Hence rather complex, yet natural interactions can be designed using little finger motion and device space.

In this paper we argue that for wide deployment, interactive surfaces should be embedded in real environments as unobtrusively as possible. Rather than deploying dedicated interactive furniture, in environments such as pubs, cafés, or homes it is often more acceptable to augment existing tables with interactive functionality. One example is the use of robust camera-projector systems in real-world settings in combination with spatially tracked touch-enabled personal devices. This retains the normal usage of tabletop surfaces, solves privacy issues, and allows for storage of media items on the personal devices. Moreover, user input can easily be tracked with high precision and low latency and can be attributed to individual users.

We demonstrate how multi-touch hand gestures in combination with foot gestures can be used to perform navigation tasks in interactive systems. The geospatial domain is an interesting example to show the advantages of the combination of both modalities because the complex user interfaces of common Geographic Information System (GIS) requires a high degree of expertise from its users. Recent developments in interactive surfaces that enable the construction of low cost multi-touch displays and relatively cheap sensor technology to detect foot gestures allow the deep exploration of these input modalities for GIS users with medium or low expertise. In this paper, we provide a categorization of multitouch hand and foot gestures for the interaction with spatial data on a large-scale interactive wall. In addition we show with an initial evaluation how these gestures can improve the overall interaction with spatial information.

Combining GPS tracks with semantic annotations is the basis for large data analysis tasks that give insight into the movement behavior of populations. In this paper, we present a first prototype implementation of a GPS tracking application that aims at subsuming GPS tracking and manual annotation on a standard mobile phone. The main purpose of this prototype is to investigate its usability, which is achieved by a tilt- and motion-sensing interface. We provide a GPS diary function that visualizes GPS trajectories on a map, allows annotating the trajectory, and navigating through the trajectory by moving and tilting the mobile phone. We present the design of our application and report on the very first user experiences.

In this paper we explore spatial audio as a new design space for applications like teleconferencing and audio stream management on mobile devices. Especially in conjunction with input techniques using motion-tracking, the interaction has to be thoroughly designed in order to allow low-dimensional input devices like gyroscopic sensors to be used for controlling the rather complex spatial setting of the virtual audio space. We propose a new interaction scheme that allows the mapping of low-dimensional input data to navigation of a listener within the spatial setting.

Some public display systems provide information that is vital for people in their vicinity (such as departure times at airports and train stations) whereas other screens are more ambient (such as displays providing background information on exhibits in a museum). The question we are discussing in this paper is how to design interaction mechanisms for the latter, in particular how mobile phones can be used to enable opportunistic and leisurely interaction. We present results from an investigation into the use and perception of a public display in a café, and we derive some requirements for phone-based interaction with (ambient) public displays. Based on these requirements, we briefly evaluate three different interaction techniques.

Negotiation and coordination of activities involving a number of people can be a difficult and time-consuming process, even when all participants are collocated. We propose the use of spatial proximity regions around mobile devices on a table to significantly reduce the effort of proposing and exploring content within a group of collocated people. In order to determine the location of devices on ordinary tables, we developed a tracking mechanism for a camera-projector system that uses dynamic visual markers displayed on the screen of a device. We evaluated our spatial proximity region based approach using a photo-sharing application for people sat around a table. The tabletop provides a frame of reference in which the spatial arrangement of devices signals the coordination state to the users. The results from the study indicate that the proposed approach facilitates coordination in several ways, for example, by allowing for simultaneous user activity and by reducing the effort required to achieve a common goal. Our approach reduced the task completion time by 43% and was rated as superior in comparison to other established techniques.

Virtual globes have progressed from little-known technology to broadly popular software in a mere few years. We investigated this phenomenon through a survey and discovered that, while virtual globes are en vogue, their use is restricted to a small set of tasks so simple that they do not involve any spatial thinking. Spatial thinking requires that users ask "what is where" and "why"; the most common virtual globe tasks only include the "what". Based on the results of this survey, we have developed a multi-touch virtual globe derived from an adapted virtual globe paradigm designed to widen the potential uses of the technology by helping its users to inquire about both the "what is where" and "why" of spatial distribution. We do not seek to provide users with full GIS (geographic information system) functionality, but rather we aim to facilitate the asking and answering of simple "why" questions about general topics that appeal to a wide virtual globe user base.

Many mobile devices, specifically mobile phones, come equipped with a microphone. Microphones are high-fidelity sensors that can pick up sounds relating to a range of physical phenomena. Using simple feature extraction methods, parameters can be found that sensibly map to synthesis algorithms to allow expressive and interactive performance. For example blowing noise can be used as a wind instrument excitation source. Also other types of interactions can be detected via microphones, such as striking. Hence the microphone, in addition to allowing literal recording, serves as an additional source of input to the developing field of mobile phone performance.

Real-world objects (and the world) are usually not at. It is unfortunate, then, that mobile augmented reality (AR) applications often concentrate on the interaction with 2D objects. Typically, 2D markers are required to track mobile devices relative to the real-world objects to be augmented, and the interaction with these objects is normally limited to the xed plane in which these markers are located. Using platonic solids, we show how to easily extend the interaction space to tangible 3D models. In particular, we present a proof-of-concept example in which users interact with a 3D paper globe using a mobile device that augments the globe with additional information. (In other words, mobile interaction with the "real world".) We believe that this particular 3D interaction with a paper globe can be very helpful in educational settings, as it allows pupils to explore our planet in an easy and intuitive way. An important aspect is that using the real shape of the world can help to correct many common geographic misconceptions that result from the projection of the earth's surface onto a 2D plane.

On mobile devices, navigating in high-resolution and high-density 2D information spaces, such as geographic maps, is a common and important task. In order to support this task, we expand on work done in the areas of tilt-based browsing on mobile devices and speed-dependent automatic zooming in the traditional desktop environment to create an efficient interface for browsing high-volume map data at a wide range of scales. We also discuss infrastructure aspects, such as streaming 2D content to the device and efficiently rendering it on the display, using standards such as Scalable Vector Graphics (SVG).

One compelling scenario for the use of GPS enabled phones is support for navigation, e.g. enabling a user to glance down at the screen of her mobile phone in order to be reassured that she is indeed located where she thinks she is. While service based approaches to support such navigation tasks are becoming increasingly available - whereby a user downloads (for a fee) a relevant map of her current area onto her GPS enabled phone, the approach is often far from ideal. Typically, the user is unsure as to the cost of downloading the map (especially when she is in a foreign country) and such maps are highly generalised and may not match the user's current activity and needs. For example, rather than requiring a standard map on a mobile device of the area, the user may simply require a map of a university campus with all departments or a map showing footpaths around the area in which she is currently trekking. Indeed, one will often see such specialised maps on public signs situated where they may be required (in a just-in-time sense) and it is interesting to consider how one might enable users to walk up to such situated signs and use their mobile phone to `take away' the map presented in order to use it to assist their ongoing navigation activity. In this paper, we are interested in a subset of this problem space in which the user `grabs' a map shown on a public display by taking a photograph of it and using it as a digital map on her mobile phone. We present two di erent scenarios for our new application called PhotoMaps: In the rst one we are having full control on the map design process (e.g. we are able to place markers etc., in the second scenario we use the map as it is and appropriate it for further navigation use.

Information navigation on handheld displays is characterized by the small display dimensions and limited input capabilities of today’s mobile devices. Special strategies are required to help users navigate to off-screen content and develop awareness of spatial layouts despite the small display. Yet, handheld devices offer interaction possibilities that desktop computers do not. Handheld devices can easily be moved in space and used as a movable window into a large virtual workspace. We investigate different information navigation methods for small-scale handheld displays using a range of sensor technologies for spatial tracking. We compare user performance in an abstract map navigation task and discuss the tradeoffs of the different sensor and visualization techniques.

The exploitation of finger and hand tracking technology based on infrared light, such as FTIR, Diffused Illumination (DI) or Diffused Surface Illumination (DSI) has enabled the construction of large-scale, low-cost, interactive multi-touch surfaces. In this context, access and security problems arise if larger teams operate theses surfaces with different access rights. The team members might have several levels of authority or specific roles, which determine what functions and objects they are allowed to access via the multi-touch surface. In this paper we present first concepts and strategies to authenticate and interact with subregions of a large-scale multi-touch wall.

When camera phones are used as magic lenses in handheld augmented reality applications involving wall maps or posters, pointing can be divided into two phases: (1) an initial coarse physical pointing phase, in which the target can be directly observed on the background surface, and (2) a fine-control virtual pointing phase, in which the target can only be observed through the device display. In two studies, we show that performance cannot be adequately modeled with standard Fitts' law, but can be adequately modeled with a two-component modification. We chart the performance space and analyze users' target acquisition strategies in varying conditions. Moreover, we show that the standard Fitts' law model does hold for dynamic peephole pointing where there is no guiding background surface and hence the physical pointing component of the extended model is not needed. Finally, implications for the design of magic lens interfaces are considered.

The mobile phone is the first truly pervasive computer. In addition to its core communications functionality, it is increasingly used for interaction with the physical world. This chapter examines the design space of input techniques using established desktop taxonomies and design spaces to provide an indepth discussion of existing interaction techniques. A new five-part spatial classification is proposed for ubiquitous mobile phone interaction tasks discussed in our survey. It includes supported subtasks (position, orient, and selection), dimensionality, relative vs. absolute movement, interaction style (direct vs. indirect), and feedback from the environment (continuous vs. discrete). Key design considerations are identified for deploying these interaction techniques in real-world applications. Our analysis aims to inspire and inform the design of future smart phone interaction techniques.

The development of FTIR (Frustrated Total Internal Reflection) technology has enabled the construction of large-scale, low-cost, multi-touch displays. These displays—capable of sensing fingers, hands, and whole arms—have great potential for exploring complex data in a natural manner and easily scale in size and the number of simultaneous users. In this context, access and security problems arise if a larger team operates the surface with different access rights. The team members might have different levels of authority or specific roles, which determines what functions they are allowed to access via the multi-touch surface. In this paper we present first concepts and strategies to use a mobile phone to spontaneously authenticate and interact with sub-regions of a large-scale multi-touch wall.