User Interface / Wearable AR / UI Metaphor / Menu
Wearable AR
UI System
Design Research
ROLE
Researcher
Designer
Developer
TOOL
HoloLens
Unity3D
Keynote
SKILL
Storyboard / Information Architecture
Ergonomic / User Interfaces Design
User Research / Usability Testing
From Nov 2017 to April 2018, I worked with Professor Michael Nebeling to explore a more natural UI framework for AR workspace based on Microsoft HoloLens. We did multiple prototypes to conduct user testing and fully developed all design ideas via Unity 3D on Microsoft HoloLens.
Research Goal
Problems: One of the biggest values of AR is that it extends our digital workspace from some small metal boxes with screens to the entire room or even the world. Current UI system for Wearable AR application created by Microsoft HoloLens is just multiple windows floating in the mid-air, which requires us to gesture and head movement to select and manipulate these windows. Such a UI framework is not really functional or aesthetic for a long time working and the interaction techniques sometimes lead to frustration and fatigue due to the low accuracy of gesture recognition.
Our goal: A high-level UI framework to support future AR Operation system in the working scenarios
Our expectation for the Future AR workspace: Digital information from one file to applications to a big operation system merge with your physical workspace - Freely open, place and use your virtual applications or pages at any location in this workspace and in any size you want. 3D applications are 3D. Doing 3D works in the 3D world. Collaboration in a more seamless way without being restricted by your laptops’ shells or phone’s shell.
Design Philosophy
To support such AR workspace, we raise the following requirements for the interaction system:
-
Comprehensive Metaphor: Container + information - the solution is beyond Interfaces that can utilize the surface from the physical world and the human physical body as the container for displaying and storing information.
-
Natural Interaction: more natural interaction techniques to accomplish basic HCI tasks like Selection/Manipulation/Navigation/Typing/Drawing.
The following image is a prototype scene for the Metaphor of Container + Information.
Design Process
Scenarios
I synthesized the typical computer-related working scenarios into the following typical tasks:
-
Open multiple applications or files
-
Work on these applications or files
-
Adjust the position of them
-
Close some of them
Design Rationale
Because the concept of the workspace is so abstract, so after defining the scenarios, we decompose these tasks into the following components:
To realize the tasks we defined in the scenarios, we need a comprehensive control system that users can easily open, manipulate and close these information containers. We tested out several different UI combinations for this control system based on the location of the containers as you can see from the following diagram.
Outcome
After user testing, we finally settled down on the solution:
There is a control panel that attaches on the user's hand to act as a mode switcher to switch between adding, deleting and editing modes.
- Adding mode is for adding a new container
- Deleting mode is for deleting containers
- Editing mode is for adjusting the size and the position of the modes.
After creating a new container, you can decide which applications or files to put into the application.
AR drawing
AR reading and notes Taking
After this research experience, I then expanded this research topic to specific working scenarios: drawing, reading, and notes taking. I dig deeper in the meaning of UI metaphor and interaction we created in the research:
-
Metaphor: Container + information
-
Any surfaces in the physical world could become containers for digital information.
-
Give the physical attributes to the digital Surface.
-
-
Natural Interaction: natural hand movements to interact with these contents as we are in the physical world.
Following images are the storyboard.
Enjoy it!
AR drawing
TOOL
Keynote
Features
-
Prepare and place the virtual canvas, pen box, and palette as you want;
-
Draw the picture by selecting different crushes, colors, and shapes;
-
Display your work in the physical place.
Storyboard
Choose your working position - Sit down in front of a drawing board or Sit in front of a table
Preparation
Create your drawing board
Rearrange your virtual working zone
Horizontally move the virtual surface
Adjust the degree of the virtual surface
Attach your virtual toolbox to a physical surface
Attach your canvas to a physical wall
Draw
Select your drawing brush
Change Color
Move Shape
Resize Shape
Zoom in and zoom out (Canvas Level)
If you want to know the design rationale behind these storyboards
Click HERE
AR Reading and taking notes
Features
-
Place your reading materials and your notebooks wherever you want
-
Highlight the sentences or words while reading
-
Zoom in or zoom out
-
Multiple pages open
-
Take notes by typing
-
Take notes by handwriting
Storyboard
If you feel interested in this project, click HERE
Takeaway
Instead of getting rid of the physical world, we'd better embrace the existing world's resources, like these surfaces and explore the approach to merge it with the digital one. From 2D to 3D, increasing one dimension means more space for information display but it doesn't mean we should overload users with more information. As designers, we should help users navigate in this physical-virtural world.