13 · HCI & Graphics
This chapter is the top floor — the only surface in the tower a person actually touches, and the one that has to hide every floor beneath it from someone who never asked for a tower.
Where should the machine end and the person begin?
On 9 December 1968, in Brooks Hall in San Francisco's Civic Auditorium complex, Douglas Engelbart sat down in front of roughly a thousand computing professionals with a screen, a keyboard and a wooden box on wheels, and spent ninety minutes doing things none of them had seen (Doug Engelbart Institute). He moved a pointer with the box. He edited a document and it changed as he typed. He worked on the same file, live, with a colleague thirty miles away whose face was on the screen beside the text. Mouse, hypertext, real-time collaborative editing, video conferencing — in one sitting, in a decade when the normal way to use a computer was to hand over a deck of punched cards and come back tomorrow.
The demonstration is remembered for the inventions. The argument underneath it was more radical, and Engelbart had published it six years earlier: the point of a computer is not to automate human work but to augment human intellect, which requires machine and person to be inside the same continuous loop rather than taking turns (SRI, October 1962).
An interface is shared state between two minds. The machine holds a model of the task. The person holds a model of the machine. An interface is the narrow channel keeping those two models in agreement, and essentially every interface failure is a divergence between them: the file the user believes was saved, the setting they believe is off, the button they believe did nothing. This is why feedback is not politeness. A system that acts without visibly acknowledging it has let the two models drift, and the person will keep operating on the stale one.
Show the object; act on it directly. Ivan Sutherland's Sketchpad, an MIT doctoral thesis submitted in 1963, let a user draw with a light pen on a screen and have the drawing answer back — constraints maintained, shapes copied and scaled (Cambridge reissue). Xerox PARC's Alto put the idea into a research machine in 1973; the Xerox Star of April 1981 shipped it commercially, with files as icons on a desktop (Star). Ben Shneiderman named the principle in 1983 and gave it three properties still worth reciting: continuous visual representation of the object of interest; physical actions or labeled button presses in place of complex command syntax; and rapid, incremental, reversible operations whose effect is immediately visible (IEEE Computer 16(8)). The third is the quietly humane one. Reversibility converts a machine from an examination into a place to think, because it makes exploring cost nothing.
Affordance is a relationship; most of what you see is a signifier. J.J. Gibson introduced affordance in 1966 and developed it in his 1979 ecological theory: what an environment offers a particular creature — a chair-height surface affords sitting to an adult and not to an infant (affordance). Don Norman brought the word into design in 1988 in The Psychology of Everyday Things, later retitled The Design of Everyday Things, then spent twenty-five years watching designers misuse it to mean "visual hint" — so the 2013 revision introduced signifier for the perceptible mark advertising where an action is possible (Norman). The distinction is load-bearing on a touchscreen, because a sheet of glass affords almost nothing physically. Every apparent edge, shadow and handle on it is a signifier someone drew — which is why this discipline can be practiced badly with no immediate penalty: remove the signifiers and the affordances technically remain, invisible.
The human is a component with published specifications. Robert Miller measured the response times at which a conversation with a machine stops feeling like one, and Jakob Nielsen restated the same three thresholds decades later: about 0.1 second for an action to feel instantaneous, about one second to keep a person's flow of thought unbroken, about ten seconds before attention leaves altogether (Miller 1968 / Nielsen). Paul Fitts had already quantified pointing in 1954 — the time to reach a target grows with the logarithm of the ratio between distance and target width (Fitts's law), which is why the corners and edges of a screen are its most valuable real estate: they are targets of effectively infinite width, because the pointer stops there. These are the only constants in this book that describe the user rather than the machine, and unlike Moore's law they have not moved in seventy years.
Rendering is one integral you cannot solve, approached from two directions. Rasterization asks, for each triangle, which pixels does it cover — fast, local, and the reason real-time graphics existed at all for thirty years. Ray tracing asks the opposite question, for each pixel, which light arrived here, and follows rays backwards into the scene; Arthur Appel cast the first such rays in 1968 and Turner Whitted made them recursive in 1980, so surfaces could reflect and refract each other (CACM 23(6)). Jim Kajiya then wrote down what both were approximating: the rendering equation of 1986, stating that the light leaving any point is what it emits plus everything it reflects from everywhere else — recursive, with no closed-form solution (SIGGRAPH '86). All of photorealistic rendering is sampling that integral cheaply enough. It took until 2018 for the first GPUs with dedicated ray-tracing units to run the honest method at frame rate (NVIDIA Turing).
Accessibility is the floor, not a ramp bolted on afterwards. The World Health Organization estimates that 1.3 billion people — about one in six — live with significant disability (WHO, 7 March 2023). The web's standard, WCAG 2.2, became a W3C Recommendation on 5 October 2023 and rests on four principles: content must be perceivable, operable, understandable and robust (W3C); its successor remains a working draft (WCAG 3, March 2026). The argument for treating this as foundation rather than compliance chore has a name. Disabled activists in Berkeley took sledgehammers to curbs in the early 1970s, and the first official curb cut followed on Telegraph Avenue in 1972; the ramps turned out to serve parents with strollers, travelers with suitcases and delivery workers with carts — the curb-cut effect, named by Angela Glover Blackwell in 2017 (SSIR). Designing for the constrained case reliably produces a better general case: the rare argument in this book that is both ethical and cheap.
Sutherland received the Turing Award for 1988, Engelbart for 1997, and Ed Catmull and Pat Hanrahan for 2019 (ACM).
Connections
Below: chapter 04, because every principle here is rationed by hardware — the frame budget, the specialized processors that made both interfaces and rendering possible. Chapter 05 owns the multiplexing that keeps an interface responsive while the machine does everything else, and chapter 09 is where an interface's delays come from once the other mind is remote. Sideways: chapter 07, since usability must survive years of change by many hands, and chapter 11, whose 1975 list of security principles ended with psychological acceptability — the one this floor is responsible for and the one security most often sacrifices. Above: nothing. This is the top of the tower, and the reason all the hiding below it was worth doing.
Open questions
If natural language becomes the interface, what happens to direct manipulation — do Shneiderman's three properties survive when the object of interest is no longer continuously visible and actions are no longer obviously reversible? Interfaces that adapt to a person must first model that person; who holds that model, and what else it serves, is a question the field has largely left to other disciplines. And the oldest one, unanswered since 1968: Engelbart wanted augmentation, not automation. Sixty years on, most of what shipped was automation. Was that a technical inevitability or a commercial one?
Go deeper
- Don Norman, The Design of Everyday Things, revised and expanded edition, Basic Books, 2013 — the discipline's one indispensable general-reader book; the author's note on the revision is at jnd.org.
- Matt Pharr, Wenzel Jakob & Greg Humphreys, Physically Based Rendering: From Theory to Implementation — the full text of the 3rd and 4th editions, free, at pbr-book.org.
- Douglas Engelbart, "Augmenting Human Intellect: A Conceptual Framework," SRI Summary Report AFOSR-3223, 1962 — dougengelbart.org.