P003
Part 1: Introduction Text
Hi, X. My name is Y, and I'm going to be walking you through this session.
You probably already know, but let me explain why we've asked you to come here today. We're testing an electricity display that we're working on so we can see what it's like for actual people to use it.
I want to make it clear right away that we're testing the display, not you. You can't do anything wrong here. In fact, this is probably the one place today where you don't have to worry about making mistakes.
We want to hear exactly what you think, so please don't worry that you're going to hurt our feelings. We want to improve it, so we need to know honestly what you think.
As we go along, I'm going to ask you to think out loud to tell me what's going through your mind. This will help us.
If you have questions, just ask. I may not be able to answer them right away, since we're interested in how people do when they don't have someone sitting next to them, but I will try to answer any questions you still have when we're done.
We have a lot to do, and I'm going to try to keep us moving, but we'll try to make sure that it's fun, too.
Preliminary Questions
Before we look at the device, I'd like to ask you just a few quick questions. First, what's your occupation?
- Interpreter at the hospital (Chinese / English)
Roughly what do you know about your electricity use?
- No
Do you know your bill, kWh?
- No
How do you feel about learning about your electricity use?
- Useful. Parents pay close attention to the bill.
Part 2: Looking at the Device and Unaided Exploration (i.e., see what the user can do by themselves)
First, I'm just going to ask you to look at this device and tell me what you think it is, what strikes you about it, and what you think you would do first.
For now, don't actually do anything. Just tell me what you would do.
And again, as much as possible, it will help me if you can try to think out loud so we know what you're thinking about.
Let them interact and think out loud
--When done: If you had to take a guess, what do you think it is about?
Ok. Now, if you were at home, what would you do first?
...continue
Notes (just let them explore, but take notes on these items):
- How much can the person do with no instruction; just “use the device”?
- Without any instruction, what do they think the purpose of the device is, the features are, the buttons do?
- Was able to state how he would use the remote to navigate the various settings of the device
- Can the person tell the state of the device just by looking at it?
- Can the person tell the action alternatives just by looking at the device?
- What is the mental model of the participant? Does this match the designer's model and the system image?
- Was able to get most of it, except he thought that the thermometers were referring to indoor temperatures rather than goal attainment feedback
- Thought the remote on the goal setting page would be used to adjust temperature, not the reduction percentage
- What are the errors people make? How do we fix them? Are they detectable, minimal consequences, and reversible?
- Was unable to tell us the "Now" information about electricity use
- Initially thought there were no "High usage" or red days on the calendar
- What are the mappings between controls and functions?
- What is the gulf of execution? What is the difference between intentions and allowable actions?
- What is the gulf of evaluation? What is the amount of effort required to interpret the physical state of the system and determine whether intentions have been met?
- Can the person determine the functions of the device unaided?
- Everything except the goal setting thermometer and remote
- Can the person tell what actions are possible unaided?
- Yes
- Can the person determine the mapping between intentions and physical movement unaided?
- Can the person perform intended actions unaided?
- Yes
- Can the person tell if the system is in the desired state unaided?
- Can the person determine mapping from system state to interpretation unaided?
- Can the person tell what state the system is in unaided?
- Can the person determine the relationships between actions and results?
- Yes
- Can the person determine the relationship between the controls and their effects?
- Yes
- Can the person determine the relationship between the system state and what is visible?
- Yes
- Can we adjust the arrangement so as to reduce the number of labels while improving c comprehension (i.e., can we find natural mappings)?
- The legend at the bottom of the calendar proved useful
- The bar graph was correctly interpreted
- Can people muddle through without reading the instructions or help from the experimenter?
- Almost, need to improve on presenting information about electricity usage "Now" in a way that can be easily interpreted but does not compromise information presented on the calendar
- Clearly state remote buttons on the screen to facilitate their navigation process
- Have a remote control legend on the screen for each display (i.e. On calendar: "Prev" - Less information, "Next" - More information)
Part 3: Goal-directed tests
- Can you think of something you'd like to learn from the device? If you wanted to learn that, what would you do?
- Learn about usage for different time periods (daily, weekly, annually)
- Would like to learn about usage of different appliances
- Would like to see how much was used at different hours on specific days (most and least usage hours)
- Could learn usage by turning on and off (or unplugging) appliances
- Can you access the calendar?
- Yes
- Can you show a bar graph of monthly consumption?
- Yes
- What is the monthly bill?
- $90.09 - correct
- How much electricity is being used right now?
- Tried to calculate the cumulative kWh through the daily usage bubbles (misinterpreted the question)
- What is the current price of electricity?
- $3.90 - incorrect, gave the daily cost instead of the price shown on the bottom
- Can you switch from calendar to bar graphs and back?
- Yes, but didn't know what the "previous" and "next" buttons were referring to (more information? less information?)
- How many kWh were used yesterday?
- 22.72 kWh - correct
- How many kWh are projected to be used tomorrow, next week, month...
- Can you switch from bar graphs to pictures?
- Yes
- Can you switch from pictures to bar graphs?
- Yes
- Can you switch from calendar to pictures?
- Yes
- Can you switch from pictures to calendar?
- yes
- Can you switch from bar graphs to calendar?
- Yes
- Can you switch from calendar to bar graphs
- Yes
Part 4: Trunk test
Have participant close eyes and take them to randomly selected pages (actually use a random number generator for this). Can they answer the following questions?
- What is this?
- What page am I on? (page name)
- What are the major sections of this site? (sections)
- What are my options at this level? (local navigation)
- Where am I in the scheme of things? (“you are here indicator”)
- How can I get back to pictures and the default page?
Part 5: In the dark test
Turn off the lights. Secretly give them the control upside-down. Can they still use the device?
Part 6: Dual task
Can participants dual task (e.g., n-back) and also use the IHD?
Part 7: Destructive Testing
Can they break it? Not physically.
Part 8: Open-ended suggestion period
- Add cumulative kWh information to show the relationship between total consumption and cost
- Add weekly summary (calendar breakdown by week instead of month)
- Add daily summary (usage breakdown by hour with emphasis on most and least usage)
- "up" longer time range (from week to month to year), "down" shorter time range (from month to week to day), "left" less information, and "right" more information
- When asked about CPD ($1.25/kWh), he could deduce that his bill might skyrocket given the fluctuations in usage as shown on the calendar
- When looking at the monthly usage line graph:
- Could not deduce why the fluctuations occur
- Would like to know what causes those fluctuations
- Tips are useful
- Would like the AC tip to tell you what costs are saved by changing the thermostat to 68F
- Help him remember what appliances needs to be turned off or plugs need to be pulled out
- Add indoor and outdoor temperature as a part of the electricity display
