A few years ago I was working for a project evaluating smart home technology, comparing user expectations at the point of installation with experiences 1 year later.
With this longitudinal research methodology, we followed the user experience of our participants via face-to-face, in-depth, semi-structured interviews and contextual enquiries. By doing so, we explored how experiences evolved over time, and discovered unique aspects of interactions that could be missed by the usual cross-sectional usability studies.
This project had 20 houses as volunteers, and we set out to analyse the expected benefits and anticipated challenges to the introduction of smart home technology. We then compared these expectations to post-use experiences after living with the technology for 1 year. Contextual interviews were undertaken with householders during the installation of a range of devices, and again approximately one year later, when householders had the option to keep the technology or have some or all of it removed. We were able to identify whether initial expectations were met, whether initial concerns persisted and whether new issues, concerns, and benefits (that had not been initially anticipated) arose after an extended period of use.
The technology
This research was part of the REFIT project, a multidisciplinary research initiative investigating the use of smart home technology in the UK. The project conducted a field study of two and a half years, deploying this technology in a sample of 20 dwellings. RWE SmarthomeTM devices were fitted within each home to provide heating system management, allow zonal thermal control, and provide home security features. The devices included one central controller and several battery-operated remote units. A typical installation comprised nine radiator thermostats with temperature and air humidity sensor, three room thermostats to control radiators in a particular room or zone, six indoor and one outdoor motion detectors with integrated brightness sensor, door and window contact sensors to record opening and closing events, one smoke detector and alarm, two wall-mounted transmitters working as physical switches to trigger actions, and one remote control providing eight buttons which could be configured to control devices in the RWE system. Additional devices include Z-Wave VeraTM controllers linked to smart plugs to give remote and automated control over selected appliances and Current Cost units to monitor electricity consumption. British Gas Hive Active Heating system was also offered to provide advanced control of boilers, and it was subsequently installed in 15 homes that agreed to have this technology.
Methods
Rather than traditional face-to-face interviews, the data collection in this project took the form of a contextual enquiry, including a tour of the house to define locations of equipment. When possible, all family members took part in the discussion. The second set of interviews were undertaken during home visits to determine whether householders wanted to keep any/all of the systems and to collect all remaining monitoring equipment. We obtained a total of 31 hours of recording. Following transcription, they were coded and analysed by theme, similarly to the method I describe here.
Results
Initial expectations from all households were high—related to comfort, convenience, improved control, energy demand reduction, and integration of technologies. There were also many initial concerns, including reliability, usability, and how these systems would be ‘domesticated’ to fit existing routines. After living with the systems for 1 year, many expectations were met, but the added value (and hence the match between expectations and reality) was higher for households with varied routines and large, partially occupied houses. Some of the anticipated challenges—such as concerns over aesthetics—largely disappeared, issues to do with the awkward location of network components persisted and new issues arose relating to the usability of some components and the time and effort required to configure them.
Unique findings
We obtained some expected results such as tech-savvy users being able to use advanced features, and older participants having a limited use, generally sticking to the basic system features. However, the technology was being used in unforeseen levels by one female participant of 71, living by herself with mobility limitations. The technology improved control and facilitated the access to difficult-to-reach radiators, as she explains:
“I did get used to it and realised just how useful it was for changing the radiator valve in the back bedroom, which is very hard to get at manually… Basically, I had to be on my stomach with a torch to see what the radiator valve said… Now I use the computer, I don’t have to do anything with the radiator valves.”
During the initial installation phase, aesthetic concerns such as the obtrusiveness of the devices or the addition of ‘clutter’ to the house were mentioned widely. In the exit interviews, householders explained that these issues were of less concern. Firstly, the devices faded into the background with time and occupants stopped noticing them:
“Actually, after it’s been in about two weeks, you forget it’s there”.
Secondly, the devices become part of existing belongings amongst other objects and are not noticed anymore. Finally, having these devices in homes is not unusual nowadays:
“People expect to see things like that when they go into people’s houses, ‘oh you’ve got one of those!’ [laughs]”.
If aesthetic concerns diminished with time, some annoyances were not foreseen and started to bother participants. These issues were probably overlooked during the product development phases in the lab and design studios. Examples include noisy motorised components and bright LED indicators which can be intrusive at night or in tranquil home environments. For longer-term satisfaction, we indicate that the design of smart home technology needs to tackle the issues that only arise when products are used in ‘real’ home environments.
We also noticed controls that were easily interfered with by children. One couple also mentioned that they had more problem with the cat, as it often changed the temperature settings on the radiator valve in the bathroom because it kept rubbing its head against it. A real home-based testing environment could be added to a product development cycle to identify issues that need to be tackled.
Conclusion
The majority of the challenges anticipated by participants at the beginning of the project did actually materialise—for example, incompatibility of pre-existing hardware and software, reliability of devices, and complexity of use. Although technology improvements are reducing the impact of these issues, there needs to be further focus on improving the installation of smart home technology into existing homes in order to obtain the functional, instrumental, and socio-technical benefits that it can bring.
The structures of domestic life clearly influenced the usefulness and level of adoption of smart home technologies. There turned out to be limited usefulness for (1) households with very regular routines, (2) families living in small houses with most rooms occupied, or (3) those with frugal lifestyles. However, households with varied occupancy of rooms, larger houses, and varying daily schedules generally obtained what they expected in relation to comfort, convenience, control, or management. Expectations at the purchase/installation phases need to be set accordingly; otherwise, householders will be disappointed.
This text is based on the journal article that I co-authored with my line managers. To see the full research please get in contact to obtain a copy or access the publication website via the DOI link: https://doi.org/10.1007/s00779-019-01302-4