Article: http://bit.do/anttipaper COMPUTATIONAL DESIGN OF user interfaces Antti Oulasvirta Associate Professor userinterfaces.aalto.fi COMPUTATIONAL DESIGN COMBINES DESIGN THINKING WITH COMPUTATIONAL SCIENCES (AND BEHAVIORAL/COGNITIVE SCIENCES) COMPUTATIONAL APPROACHES TO UID OVERVIEW Design generation Design refinement TODAY Choose a design δ1 Algorithm … Interaction UI (δ) δn User Design adaptation UI design 2. 1. Gallery of results Model (θ) Model-based combinatorial Observations 3. What next? Theories; Prior data; Opinion “ZOO” SOME STATE-OF-THE-ART RESULTS IN COMPUTATIONAL DESIGN [Proc DIS’14] PianoText: Optimizing Skill Transfer CD GAIN ADAPTATION USING INVERSE MODELING AND LOCAL OPTIMIZATION ATTENTION-OPTIMIZED SKIM READING WITH HEURISTIC ADAPTATION Title [Proc CHI’16] Keyboard design as the assignment problem Qwerty: 28 wpm KALQ 37 wpm < Frequent side changes > [Oulasvirta et al. CHI’13] Gesture set design using model-based black box optimization [Sridhar et al. CHI’15] 9 LAYOUT REFINEMENT USING AN ENERGY MINIMIZATION APPROACH DesignScape U Toronto ERROR REDUCTION USING STATE MACHINES GAME LEVEL DESIGN USING HUMAN PERFORMANCE MODELS EASY MEDIUM DIFFICULT Real gaming performance by a colleague [Proc CHI’16] SCATTERPLOT ENHANCEMENT USING PERCEPTUAL METRICS Matplotlib default design [Micallef et al. IEEE TVCG 2017] MENU SYSTEM GENERATION USING MODEL-BASED COMBINATORIAL OPTIMIZATION (ANT COLONIES) Body Level One [Proc. UIST’13] APPLICATION MENU OPTIMIZATION CASE: WINDOWS PHONE Baseline 3000 2500 2000 Baseline Evaluation Optimised Baseline Optimised 3500 3 Optimised Baseline Users (N=16) were significantly faster 3000 with the optimized 3 menu. While they rated it to be more 2500 cluttered, they liked the colors more. 2000 All All except first Mean selection time (ms) Optimised Mean selection time (ms) 3500 Mean selection time (ms) Baseline 2 2 All All except first [Todi et al. DIS’16] WEB PAGE DESIGN USING MULTI-OBJECTIVE COMBINATORIAL OPTIMIZATION Heading goes here Button Button Button www.firstlink.com www.secondlink.io www.thirdlink.org [Proc DIS’16] DATA-DRIVEN “DESIGN MINING” USING STATISTICAL FEATURE MODELS Ranjitha Kumar et al. (2013): a pattern miner that suggests user interfaces to queries. top nav topbar nav bar largelarge header header largelarge text text blockblock layout layout query query FigureFigure 13. Five 13.ofFive the 20 of the search 20 search resultsresults for the for three-part the three-part DQL layout DQL layout query visualized query visualized on the on left.theThe left.query, The query, which executed which executed in 2.1 minutes, in 2.1 minutes, returnsreturns pages pages that share that ashare common a common high-level high-level layout, layout, but exhibit but exhibit different different designs. designs. comparison, comparison, FigureFigure 15 shows 15 shows nearest-neighbor nearest-neighbor resultsresults for thefor the top query in Figure in Figure 14 using 14 using only the only vision-based the vision-based GIST GIST de- deWebzeitgeist Webzeitgeist also enables also enables a newa kind new of kind design-based of design-based ma- ma- top query scriptors. WhileWhile these these elements elements are visually are visually reminiscent reminiscent of of chine chine learning. learning. For the Forfirst thetime, first time, applications applications can stream can stream scriptors. the query, they bear theylittle bearstructural little structural or semantic or semantic relation relation to it. to it. structured structured visualvisual descriptors descriptors for page for elements page elements from afrom cen-a cen- the query, Machine Machine Learning Learning POSTER GENERATION USING PROBABILISTIC INFERENCE Image Parsing via Stochastic Scene Grammar ng y Hierarchical scene s (clusters) from al relations. ) Image Parsing via Stochastic Scene Grammar Yibiao Zhao, Song-Chun Zhu 1 Y Department of Statistics University of California, Los Angeles Los Angeles, CA 90095, Department of Statistics and Computer Science University of California, Los Angeles Los Angeles, CA 90095 Yibiao Zhao and Song-Chun Zhu University of California, Los Angeles Introduction Stochastic Scene Grammar This paper proposes a parsing algorithm for indoor scene understanding which includes four aspects: computing 3D scene layout, detecting 3D objects (e.g. furniture), detecting 2D faces (windows, doors etc.), and segmenting the background. The algorithm parse an image into a hierarchical structure, namely a parse tree. With the parse tree, we reconstruct the original image by the appearance of line segments, and we further recover the 3D scene by the geometry of 3D background and foreground objects. The grammar represents compositional structures of visual entities, which includes three types of production rules and two types of contextual relations: • Production rules: (i) AND rules represent the decomposition of an entity into sub-parts; (ii) SET rules represent an ensemble of visual entities; (iii) OR rules represent the switching among sub-types of an entity. • Contextual relations: (a) Cooperative + relations represent positive links between binding entities, such as hinged faces of a object or aligned boxes; (b) Competitive - relations represents negative links between competing entities, such as mutually exclusive boxes. P T (I(Av )) v2Cl T 9 Introduction This paper proposes a parsing algorithm for indoor scene understanding which includes four aspects: computing 3D scene layout, detecting 3D objects (e.g. furniture), detecting 2D faces (windows, doors etc.), and segmenting the background. The algorithm parse an image into a hierarchical structure, namely a parse tree. With the parse tree, we reconstruct the original image by the appearance of line segments, and we further recover the 3D scene by the geometry of 3D background and foreground objects. Stochastic Scene Grammar The grammar represents compositional structures of visual entities, which includes three types of production rules and two types of contextual relations: o Production rules: (i) AND rules represent the decomposition of an entity into sub-parts; (ii) SET rules represent an ensemble of visual entities; (iii) OR rules represent the switching among sub-types of an entity. o Contextual relations: between binding entities, such as hinged faces of a object or aligned boxes; (b) such as mutually exclusive boxes. (3) usters) in each guration )|AG (v)). (4) [1], Hedau et n the UIUC Bayesian Formulation We define a posterior distribution for a solution (a parse tree) pt conditioned on an image I. This distribution is specified in terms of the statistics defined over the derivation of production rules. Y Y P(pt|I) / P(pt)P(I|pt) = P(S) P(Chv |v) P(I|v) (1) v2V n v2V T Bayesian Formulation The probability is defined on the Gibbs distribution: and the energy term is decomposed as three potentials: E(pt|I) = 3D synthesis of novel views based on the parse tree X E OR (Ar(Chv )) + v2V OR X E AND (AG (Chv )) + v2V A ND X E T (I(⇤v )) We define a posterior distribution for a solution (a parse tree) pt conditioned on an image I. This distribution is specified in terms of the statistics defined over the derivation of production rules. (2) ⇤v 2⇤I ,v2V T The probability is defined on the Gibbs distribution: and the energy term is decomposed as three potentials: Inference by Hierarchical Cluster Sampling 3D synthesis of novel views based on the parse tree We design an efficient MCMC inference algorithm, namely Hierarchical cluster sampling, to search in the large solution space of scene configurations. The algorithm has two stages: Results Results • Clustering: It forms all possible higher-level structures (clusters) from lower-level entities by production rules and contextual relations. P+ (Cl|I) = Y v2Cl OR P OR (Ar(v)) Y AND P+ (AG (u), AG (v)) u,v2Cl AND Y P T (I(Av )) Inference by Hierarchical Cluster Sampling We design an efficient MCMC inference algorithm, namely Hierarchical cluster sampling, to search in the large solution space of scene configurations. The algorithm has two stages: o Clustering: It forms all possible higher-level structures (clusters) from lowerlevel entities by production rules and contextual relations. (3) v2Cl T • Sampling: It jumps between alternative structures (clusters) in each layer of the hierarchy to find the most probable configuration (represented by a parse tree). Q(pt ⇤ |pt, I) = P+ (Cl ⇤ |I) Y AND P(AG (u)|AG (v)). (4) u2Cl AND ,v2pt AND o Sampling: It jumps between alternative structures (clusters) in each layer of the hierarchy to find the most probable configuration (represented by a parse tree). Experiment and Conclusion Segmentation precision compared with Hoiem et al. 2007 [1], Hedau et al. 2009 [2], Wang et al. 2010 [3] and Lee et al. 2010 [4] in the UIUC dataset [2]. Experiment and Conclusion Segmentation precision compared with Hoiem et al. 2007 [1], Hedau et al. 2009 [2], Wang et al. 2010 [3] and Lee et al. 2010 [4] in the UIUC dataset [2]. e hierarchical m fast inference xtual relations. Compared with other algorithms, our contributions are o A Stochastic Scene Grammar (SSG) to represent the hierarchical structure of visual entities; o A Hierarchical Cluster Sampling algorithm to perform fast inference in the SSG model; o Richer structures obtained by exploring richer contextual relations. Compared with other algorithms, our contributions are • A Stochastic Scene Grammar (SSG) to represent the hierarchical structure of visual entities; • A Hierarchical Cluster Sampling algorithm to perform fast inference in the SSG model; • Richer structures obtained by exploring richer contextual relations. Website: http://www.stat.ucla.edu/ ybzhao/research/sceneparsing Website: http://www.stat.ucla.edu/~ybzhao/research/sceneparsing TEMPLATE DESIGN © 2008 www.PosterPresentations.com (b) Our result (c) Original poster[35] [Qiang et al. arXiv 2017] COMPUTER-ASSISTED UI SKETCHING USING MODELBASED INTERACTIVE COMBINATORIAL OPTIMIZATION [Proc. DIS’16] Choice architecture adaptation using MPI and ML Functionality selection using MIP Input modality and interface design using RL optimization COMPUTATIONAL INTERFACE DESIGN A CONFLUENCE OF METHODS • Combinatorial optimization - see http://bit.do/anttipaper • Bayesian inference • Reinforcement learning • Simulations • Control theory • Process algebra • State machines • Petri nets A WINDOW OF OPPORTUNITY • • Large potential gains: Internationally: Interaction design core to every major company. Finland: strong interaction and graphic design profession. USA: Increasing investments into “design intelligence” Maturing methodology: Significant improvements: • More and more examples and design problems tackled • First evidence for equal- or supra-human-level design • Enablers: Hardware and software is there • HELSINKI: Strong in computational sciences and design! BEWARE! • Misguided hype weils AI and design • First disappointments are in (e.g., grid.io) • Harder than it appears! • Realistic problems complex to formulate • Issues with data and models of users MODEL-BASED COMBINATORIAL OPTIMIZATION IN 5 MINUTES OUTLOOK COVER FEATURE OUTLOOK Contacts Settings Phone Messages Mail Browser App store Maps News (b) Alarm Skype Photoshop Stopwatch Recorder Skype Dropbox YouTube (a) User Interface Design with Combinatorial Optimization Facebook (c) (d) Antti Oulasvirta, Aalto University of engineering design, so why not user interface design? The author reviews progress and challenges in model-driven UI optimization, in which an optimizer utilizes predictive models of human perception, behavior, and experience to anticipate users’ responses to computer-generated designs. O 40 COM PUTE R PUBLISHED BY THE IEEE COMPUTER SOCIET Y (f) FIGURE 1. Examples of designs generated with recent model-based approaches to user interface (UI) optimization: (a) application menu design, where the positions, colors, and sizes of icons are optimized for visual search, motor performance, color harmony, and minimum clutter; (b) menu system design, where a hierarchy of items is organized into a menu optimized for visual search time and consistency with other menus; (c) gestural input design, where commands are mapped to hand gestures to optimize for performance, individuation of fingers, and learnability; (d) widget GUI design, where widget groups, types, and positions are decided based on users’ motor capabilities; (e) scatterplot design, where aspect ratio, marker sizes, and transparency are optimized for maximum performance in tasks like correlation estimation by using perceptual models of brightness, contrast, and unit perception; and (f) web layout design, where the elements of a page are positioned, colored, and sized to optimize visual search time, motor performance, color harmony, and minimum clutter. Optimization methods have revolutionized almost every field ptimization methods in user interface (UI) design is a long-standing topic in human– computer interaction (HCI) research. Currently, user-centered design is largely focused on human creativity, sensemaking, empathy, and creation of meaning,1 but optimization methods have been explored as supplemental ways to help speed up the design cycle and improve design quality (for more information about computational UI design, see the sidebar). Unlike any other design method in HCI, optimization methods offer a greater-than-zero chance of finding an optimal design, and some exact methods even offer mathematical guarantees for solutions.2 More importantly, if the design task and its (e) assumptions are appropriate, the outcome represents the best achievable design. Optimization methods do not stop at searching for a single best design; they can also be used to explore the design space for alternative ideas. In addition, optimization could promote a change in design culture by encouraging the explication, scrutiny, and accumulation of design knowledge that has tended to be tacit. However, progress has been slow. Scarcely 15 years ago, keyboard layout was the prime—and virtually only—application of optimization methods that had empirically demonstrated benefits to end users. Model-based UI optimization refers to the use of combinatorial optimization methods to solve a UI design 0 0 1 8 - 9 1 6 2 / 1 7/ $ 3 3 .0 0 © 2 0 1 7 I E E E problem formulated as a search problem by using predictive models of human behavior and experience. The significant benefit of modelbased UI optimization over previous approaches is that it can find and propose UI designs that are both visually appealing and verifiably usable. With model-based UI optimization, a webpage design can be automatically generated so that users regard its items and colors as aesthetically pleasing,3 and a programmer can task an optimizer to design a full menu system with hundreds of commands— complete with hierarchy, groups, and 42 shortcuts—simply by listing its command names.4 MODEL-BASED UI OPTIMIZATION Like a civil engineer whose objective is building a bridge with a certain capacity and tolerance, a designer can create a UI by defining how easily users should find their targets and how they should perceive it aesthetically. Technically, this is based on the formulation of a design task mathematically using predictive models of human behavior as objectives. Figure 1 shows examples of designs COMPUTER http://bit.do/anttipaper generated with recent model-based approaches to UI optimization, including application menu design and menu system design,4 gestural input design,5 widget GUI design,6 scatterplot design7, and web layout design.4 Figure 2 shows an overview of model-based UI optimization from a design team’s perspective. UI DESIGN WITH AN OPTIMIZER IN THE LOOP Tools might encourage or enforce user interfaces that were highly usable, rather than today’s stance that tools should be neutral… .8 W W W.CO M P U T E R .O R G /CO M P U T E R Combinatorial design problems are exceedingly hard for humans Multiple user groups Multiple design objectives Prohibitively large design spaces A menu with 50 commands can be organized in 10 “A scientific psychology should help us in arranging the interface so it is easy, efficient and error free – even enjoyable.’” APPROACH OVERVIEW Optimizer Generates Defines task Steers Design tool Designer Computer scientist Defines Evaluates Behavioral scientist Predictive model Defines GURE 2. Model-based UI optimization from a designer’s perspective. Predictive models mathematical or simulations) are used in an optimizer to evaluate generated designs ainst designer-supplied objectives. The designer can steer and redefine the tasks inter- COMPUTATIONAL DESIGN 1. DEFINE DECISION PROBLEM 2. DEFINE OBJECTIVES DESIGN TASK A COMBINATORIAL DEFINITION 0 1 x1 B x2 C B C Find x = B . C 2 X which minimizes f (x) @ .. A xn Multi-objective task f (x) = !1 f1 (x) + · · · + !q fq (x) EXAMPLE: LAYOUTS A B C D E F G H I J K L M N O P Q R S T U V W X Y Z A B C D E F G H I J A K L M N O P Q R S T U V W X Y BACKGROUND: THE LETTER ASSIGNMENT P Given n letters and n keyslots, the task of the le OBJECTIVE FUNCTION ment problem is to minimize the average cost ck` letter ` after k weighted by the bigram probability XX min p · c k` k` A B C D E F G H I J K L M N O P Q R S T U V W X Y Z Open# Save## Save#as...# Close# ...# About# k ` It is an instance of the Koopmans-Beckman Prob A B C D E F G H I J Bigrams Fitts’ law this prob which is NP-hard. As we will see later, A K L M N O P Q R S T U V W X Y modeled with a quadratic function on binary varia it belongs to a broad class of problems called qu signment problem (QAP) where the goal is to m total pair-wise interaction cost [22, 24]. n!## n!# “Minimize expected time used for typing in English language” Except for a few papers using heuristic cost fun ASSIGNMENT PROBLEM • Assign n elements to m slots on a rectangle-sized surface. Often n = m Elements … Slots … DEFINITION OF QAP Given P facilities and L locations, a weight function w, and a distance function d, find the bijection f : P ! L that minimizes • X a,b2P wa,b df (a),f (b) w weight d distance Intuition: We minimize average distance among facilities. We try to “pack” together the facilities that are commonly used with each other FORMULATION WITH BINARY DECISION VARIABLES • Identify a permulation matrix X of dimensions n x n such that n n X X f frequency min dih fjk xij xhk i,j=1 h,k=1 • d distance Observations: • Large problems (n > 30) may be impossible • Generalization of TSP and therefore NP-complete PRESENTLY ADDRESSED DESIGN PROBLEMS • Keyboard layouts • Key-target resizing • Web pages • Menus • Mobile apps • Dialogues • Forms • Gestural input • • Layouts Information architectures decreases more repetitions. As experienced Thedistribution relationship between time and the gaze distribution Serial search refers to with thetotal top-to-bottom inspection of items. users about thevisited. general location of users the target, Itemshave aftermemory the target are not With practice, not forEXAMPLE each component is: MODEL: MENU SELECTION they at thebut neighborhood of time the target and, evenonlycan skipglance more items also spend less on each visited i=l X item. For this 1 reason, the number gaze points eachsearch, item tually, directly move the eyes to of it [11]. As in on serial Tmenu (✓) = Gs (i,but ✓)users + Gdalso (i, ✓) G ✓)of rep(2) in aassume isthat notexperienced constant decreases as a+function p (i, items we skip more and R and d0 the from of the cursor to the when usingfor etitions. The gaze points for gaze i when looking i=0 spend lessdistance timenumber reading them than novices. t is: the target trail strategy. We assume that d0 = 0. where Gs ,model Gd , G the numbers of gaze points (seWe thus distribution of gaze with a normal distrip estimate ⇢this aspointing) ⇥+exp( bcaptured ⇥ p ) + c if i t rial,bution directed, using an eye-tracker s i s Tand athe btarget ⇥ log (1 + ↵ ⇥ t) (8) i in centered on item t. Probability for item p = 2 Gs (i, ✓) = (3) with sampling rate ofits R.probability In our studies, R otherwise = 50Hz. t, 2 (i) 0 by thea menu is given density function We can thus estimate Gp . However, to account for the factd modulated by the power law of learning: that users can see 2-5 items in the neighborhood of the gaze Serial search point, G wed (i, revisit the equation 8. We add a normal distribution 2 ✓) = (i) ⇥ (a ⇥ exp( b ⇥ p ) + c ) (4) d d i d t, Serial(i) search refers to d the top-to-bottom inspection of items. that covers the dispersion of gaze in the items around t,1.1 Items after the target are not visited. With practice, users not thewhere target: d is the standard deviation reflecting the spread of the only skip area. moreaitems but also spend less time on each visited search , b and b are the empirical parameters of the d d d Gp (i, ✓) reason, = ap + the bp ⇥number log(1 +of ↵ gaze ⇥ t) ⇥ (i) each (9)item t,1.1on item. For this points learning model. Note that the sum in Eq. 2 ranges from the in a menu is not constant but decreases as a function of repfirst item to the end of the menu (i 2 [1, l]), but not longer, [Bailly et al. CHI’14] where t,1.1 is a normal distribution centered around the tar- SOME OBJECTIVE FUNCTIONS… Target selection Fitts 1954 Internal clock Lee & Oulasvirta 2016 Saliency Clutter perception Itti & Koch 1998 Rosenholtz 2007 Unit perception Gestalt grouping Shipley-Kellman 1992 Wertheimer 1938 Reading Kieras-Hornof 2014 Salvucci 2001 Working memory Multitasking J.R. Anderson 1998 Wickens 2002 Color harmony Cohen-Or 2006 Grid quality Balinsky 2006 Visual search Strategy adaptation Chen et al. 2015; Guiard 2015 TOOLBOX FOR DESIGN PROBLEM-SOLVING WITH OPTIMIZATION Design tasks Objective 1.Best possible design? Des ig M. Gilli 2.How far away is the present design from the best? n va riab le A Optimization heuristics Variable B 6 3.Best changes to update a UI? 4.Best compromise for multiple user groups INTEGRATION WITH MACHINE LEARNING ON-GOING RESEARCH • • • • How to steer generated designs to resemble other designs in this domain? Design mining with statistical models of designs [CHI’13] What makes a design good in interaction? Learning predictive models (e.g., regression models) from data using Symbolic Programming [CHI’14] What are my users like? Learning user model parameters from data using Approximate Bayesian Computation [CHI’17] How to learn from designs in the wild? Bayesian optimization and multi-armed bandit approaches to iterative design improvement in the wild [CHI’16] CONCLUSION WHAT CAN I DO TO COMBINATORIAL UI DESIGN IS DESIGN FROM FIRST PRINCIPLES Models (principles of interaction) allow algorithms to identify best parameters of a user interface. All inputs are scrutinizable and empirically verifiable, thus contributing to accumulation of design knowledge. Integration with data-driven and human-inthe-loop approaches is possible. Design Superiority An organization emerges that can present guarantees for their designs Bulk Design Automated A design service emerges that use interactive optimization to significantly improve costefficiency in a popular category Computer Outperforms Human SelfDesigning Interfaces Computer is shown to reliably outperform human designers in controlled studies with end-users Log data is used to parametrize optimizers that can optimally adapt the interface without human intervention. LEARN MORE! Internationally • CHI’17 course (Denver, May) • Summer School (Zurich, June) • Book to appear with Oxford Uni Press (August) Helsinki • Self-optimizing Web Services project (email me) • HIIT Machine Learning seminar (weekly) • Computational UI Design course: students [email protected] http://bit.do/anttipaper
© Copyright 2026 Paperzz