Modelling and correcting for the impact of the gait cycle on touch screen typing accuracy

The influence of posture is recognized as an important factor in designing more accurate text entry interfaces since different postures result in different typing patterns [3]. The ContextType system [4] improved text entry accuracy by inferring the interacting fingers. Posture is typically inferred from on-board sensors, which can also be used to extract other useful information, as in [5]. Here, gyroscope and swiping motions were used to infer which hand was holding the device and which hand was used for interaction. This technique yielded 84.3 % accuracy for hand posture detection and 99.7 % accuracy for table vs. hand detection. Movement also affects text entry accuracy. Goel et al. [2] described an accelerometer-based adaptive virtual keyboard named “WalkType” that employed an algorithm inspired by image stabilization for cameras. For two-thumb typing on a mobile phone, the system reduced uncorrected errors in the walking condition by 45.2 % and increased the typing speed by 12.9 %. Baldwin and Chai [6] implemented a dynamic, user-specific adaptation of virtual keyboard key sizes. This was achieved based on a language model and observed typing behavior. A possible limitation of such an approach is the inclusion of keyboard-related elements into the model. Mizobuchi et al. [7] found no significant interaction between target size and movement. Also, no effect of walking speed or input speed on accuracy was observed, while the decrease in walking speed while using a mobile phone compared to the usual walking speed was noted. In [8] subjects were placed on a treadmill, with the speed varied from 0–160 % of preferred walking speed (PWS). It was noted that no matter how slowly the users walked, accuracy was reduced compared to the static case. Additional observations was made: the mean decrease in speed when using a mobile phone was about 24 % from PWS compared when walking without the device. A similar decrease in walking speed (25 %) was observed in [9]. Crossan et al. [10] explored the relationship between gait phase and interaction accuracy. Analysis showed that there was part of the gait cycle that was about 3 times more likely to be used for tapping, and for which tapping accuracy was higher, with a lower error mean and lower variability. The correlation between the gait phase angle and typing accuracy was also studied in [11] for several typing postures. The authors also examined the influence of walking speed on different performance parameters, such as accuracy, response time, and throughput, as well as offset models.

The so-called Fat Finger Problem [12], referring to the issue of touching small targets with the ambiguous contact area of a soft finger, has long been studied as a source of error in touch interactions, including interaction on the go. This is compounded with perceptual issues, as studied in [13], leading to a systematic offset between the user’s intended and actual touch locations. Henze et al. [14] collected millions of touch examples in two Android games and used polynomials to model the offsets on a device-specific basis. Weir et al. [15] identified that offsets have a random component as well as a systematic one, and used Gaussian Process regression to probabilistically model offsets. In later work, the authors extended this model to typing data and combined it with a statistical language model to perform autocorrection [16]. Bi and Zhai [17] proposed a dual Gaussian explanation for touch accuracy, and used this to derive the Bayesian touch criterion. This was shown to improve selection accuracy for small targets. The vast majority of such models assume that either the use context of the device is not changing, or, it is changing but the change does not affect the user’s targeting behaviour. This motivates our investigation of whether targeting behaviour changes when in motion, and if so whether motion data can be used to improve targeting performance.

Acceleration data can be used to explore possible relationships between tapping accuracy and gait phase as well as relationships between input technique and gait phase. Figure 1 shows how gait phase angle can be physically interpreted: it illustrates the relationship between gait phase zero-crossings and actual movement for a walking user, and shows histograms of right knee angle values (kinematic parameter recorded using an optoelectronic motion capture system) sampled at time instances when zero-crossings occurred. The total duration of recording was 1.5 min. Note the existence of single peak in all subplots, demonstrating the consistency of our gait phase estimation algorithm. Further analysis is, however, needed to precisely define the relationship between gait phase as recorded by a mobile phone and kinematic parameters of the gait.

where ϕ(t) is the phase at time t, and A(t) is the amplitude of the signal at time t.

Although A(t) and ϕ(t) can be computed for an arbitrary s(t) they are only physically meaningful if s(t) is a narrow-band signal. Following previous work [10, 19], we band-pass filtered the acceleration with a band-pass frequency determined by subjects’ walking pace. However, since we noticed that several subjects had multiple distinct peaks in the raw acceleration frequency spectrum (for all sensitive axes) we slightly modified this approach to accommodate for this by using a bank of narrow band-pass filters. Output from all individual filters was then summed to obtain the filtered signal. This allows more intricate details of the gait to be captured without violating the narrow-band requirement for the Hilbert transform. The distributions of gait phase angle along each axis at tap events are presented in Fig. 2. The angle is defined on [ −π,+π], segmented into 10 bins.

We performed our analysis on a dataset from a previous study for which acceleration data were available [16]. The study evaluated a novel autocorrection technique combining touch and language models. 10 participants transcribed phrases from the Enron Mobile Email dataset [20] on a custom soft keyboard, implemented on a Galaxy S3 Mini smartphone running Android 4.0.

Participants entered text while sitting, standing and walking. Error rates in the walking condition were significantly higher than either of the static conditions. However, although accelerometer data were captured in this study, the correction technique did not make use of these. The goal of our gait phase analysis is to identify factors contributing to the observed differences in error rate when walking. Note that we discarded the data for one subject because accelerometer data were incomplete.

Typing accuracy

Figure 3 shows the variation of baseline error rates in the three conditions. One-way repeated measures ANOVA showed there is no statistically significant effect between conditions (p = 0.0914, F = 2.79, df = 2). Mean values were 9.24 % (±2.92 %), 6.78 % (±2.12 %) and 9.39 % (±4.59 %) for sitting, standing and walking conditions, respectively. Note that the inclusion of all error types in the original study did lead to statistically significant differences — walking lead to about 5 % more errors overall compared to sitting and standing. This is a potentially interesting result as it might suggest that it is primarily non-substitution errors that increase in frequency when walking, rather than the substitution errors studied here. This might be explained by the divide in attention between the phone and surroundings while walking or one of the other sources described in the previous subsection.

Figure 4 gives an impression of the accuracy over the full keyboard in the three experimental conditions, for Subject 7. The touch variability is aggregated to a standard character key in Fig. 5 across all test subjects. The walking condition appears the most diffuse, followed by sitting. Interestingly the standing touches appear to be the most accurate. This reflects the earlier findings in [16]. It remains unclear why this should be the case.

Gait phase analysis

Figure 2 shows the mean number of taps as a function of inferred x,y,z phase angles, averaged over all subjects. z corresponds to vertical motion, y is the direction of travel for the user, and x is lateral (left/right) movement. Looking at individual subjects tapping distribution against the y-axis inferred phase angle, all subjects apart from 4 and 9 have statistically significant deviations (at the α=0.001 level) from mean tapping distributions, based on a multinomial significance test, with correction for multiple testing performed using the False Discovery Rate (FDR) control approach [21].

The variation in key error rates by phase was not statistically significant, because of the relatively small number of errors. We made a further comparison by tightening the typing accuracy, looking at error rates for a virtual key of the same shape as the original keys but with side lengths reduces to 50 % of the original. At this level error counts were such that Subjects 1, 3, 6 and 7 showed statistically significant gait phase related error rate variation at α=0.05 after compensating for multiple comparisons using the FDR approach.

Inspection of the data suggested that the y axis acceleration was most reliable for inference of the gait phase angle. Figure 6 shows the raw acceleration values in all three axes over a three second window for a single subject, and the gait phase angle as extracted from the y acceleration. More detailed depiction of y axis acceleration and extracted gait phase angle is presented in Fig. 7. The peak in tapping density in bins 3–4 for y axis in Fig. 2 corresponds to −112 to −36 degrees. Additional insight into gait phase dependency of the interaction can be obtained if the number of taps and error rates are examined across the gait phase bins, as depicted in Fig. 8.

Following previous work [15, 16], we trained offset models using Gaussian Process (GP) regression [22]. A GP is determined by its mean and covariance functions. We choose a zero mean function — that is, we predict no offset in the absence of training data. As more calibration data is used, an offset function is learned.

For the covariance function, we use a standard squared exponential kernel with automatic relevance determination (ARD). The ARD and noise variance hyperparameters are set on a user specific basis by optimising the marginal likelihood on the training data. We train two GPs: one for predicting x offsets and one for y offsets.

The results above motivate the use of acceleration information as an additional input to the GP to improve the performance of our offset models for walking users. We investigated two methods of incorporating motion-sensitivity into the basic GP. First, we used the n filtered acceleration values leading up to a touch event as additional inputs. Since we found that the y signal gave the most reliable way of extracting gait phase, we used only that signal and ignored x and z. We experimented with a range of values of n, and found that models using more than 5 samples did not perform significantly differently from those with 5 samples, and so only report on 5 sample models here. The second approach was to use the y phase angle at the time of the touch event as an additional input.

As a baseline, we compared these augmented GPs to models using only the position as input. In each case, we trained our models on a user-specific basis using 300 training examples selected at random from all available touches. We tested the predictive performance of each model on the remaining points. The number of test points varied by user between 332 and 1866 (mean =1251), since participants typed for a fixed time rather than a fixed length of text. We averaged the performance over 10 restarts of this process, using a different random training set each time.

Figure 9 shows our results. The box plots show the distribution across users of the mean error rate over the 10 random training sets. As before, we measure error rate as the proportion of touches falling outwith the visual boundary of intended key.

The baseline is the error between the positions recorded by the device and the targets. All three GP models produce a significantly lower error rate than this baseline (t-test, p<0.05). In addition, both motion-augmented models offer a small but statistically significant further improvement of around 1 % on average compared to the position-only model (t-test, p<0.05). The two augmented models were not significantly different from each other.