Designing for Risk: What Aviation and AR Reveal about Attention, Disaster, and Human Factors (with Chris Wickens)

In this episode, Thomas interviews Dr. Chris Wickens, a pioneer in cognitive engineering and human factors, and they discuss how designers can reduce errors and enhance decision-making when lives are on the line. They delve into the high-stakes world of design psychology for critical environments—think operating rooms, airplane cockpits, and military control systems. 

Together, they explore the real science of attention, what causes overload and confusion in high-pressure moments, and how augmented reality could revolutionize user interfaces in critical settings. Whether you're designing for surgeons, pilots, or autonomous vehicles, this episode is packed with essential takeaways from decades of research in applied cognitive science.

🔍 In This Episode, You’ll Learn:

• What every designer should know about how human attention actually works
  
  
  
• Why traditional design approaches often fail in high-pressure contexts
  
  
  
• How to reduce cognitive load and prevent life-threatening mistakes
  
  
  
• The surprising ways augmented reality is shaping the future of human-machine interaction
  
  
  
• Lessons from the deep history of human factors and applied psychology

Find The Design Psychologist on your favorite podcasting platforms (or share this link with a friend): https://designpsychologist.buzzsprout.com/2395044/follow

Transcript

welcome to the design psychologist,

00:01:40.045 --> 00:01:45.765
i have a wonderful guest joining me today that i'm deeply honored and uh just

00:01:45.765 --> 00:01:48.365
really glad to have Chris Wickens, PhD.

00:01:48.945 --> 00:01:55.325
He is a world-renowned expert, a towering figure in human factors and aviation psychology.

00:01:55.625 --> 00:02:01.085
His work and his interests exist very much at the overlap of theoretical and

00:02:01.085 --> 00:02:03.485
applied cognitive psychology research.

00:02:03.665 --> 00:02:08.685
And this research has been very relevant to designing for humans who work in

00:02:08.685 --> 00:02:11.285
high-stakes environments like aviation.

00:02:11.885 --> 00:02:16.845
Military operations, medical situations, and all sorts of areas where it's important

00:02:16.845 --> 00:02:22.645
to sustain and direct attention and not make errors. And we'll get into a lot of that stuff today.

00:02:23.145 --> 00:02:28.545
Because of him, the field of human factors knows a lot more about human cognition,

00:02:28.725 --> 00:02:33.805
decision making, and how people perceive, think, and respond under pressure.

00:02:34.205 --> 00:02:38.825
This includes situations like pilots navigating complex airspace or drivers

00:02:38.825 --> 00:02:41.105
making split-second decisions. Dr.

00:02:41.205 --> 00:02:43.145
Wickens has spent decades of

00:02:43.145 --> 00:02:47.185
his career uncovering the principles of attention, workload, human error.

00:02:47.585 --> 00:02:53.605
The effect has been that Wickens and his colleagues have contributed work that

00:02:53.605 --> 00:03:00.225
helped industries understand how to design interfaces in these kinds of high

00:03:00.225 --> 00:03:01.125
stakes work environments.

00:03:01.125 --> 00:03:06.865
He's published a massive number of articles and books. I've got two of them with me today.

00:03:07.205 --> 00:03:12.765
I've got Designing for People, An Introduction to Human Factors,

00:03:13.205 --> 00:03:16.025
Engineering, and Applied Attention Theory.

00:03:16.225 --> 00:03:20.125
So some of our questions today will be coming from the topics covered in those books.

00:03:20.125 --> 00:03:26.925
And nowadays, when Chris is not mountain climbing or watching basketball,

00:03:27.285 --> 00:03:32.765
women's basketball, his research today is expanding our understanding of augmented

00:03:32.765 --> 00:03:34.845
reality, AI interfaces.

00:03:35.645 --> 00:03:41.165
Human automated teaming, and how to develop decision aids for wildfire management.

00:03:41.165 --> 00:03:44.725
We'll get into some of those topics today, but we're thrilled to have cognitive

00:03:44.725 --> 00:03:49.165
psychology professor at Colorado State University, Chris Wickens.

00:03:49.385 --> 00:03:50.605
I'm so glad you could join us today.

00:03:51.485 --> 00:03:53.825
Thank you very much, Thomas. I'm glad to be here. Yep.

00:03:54.805 --> 00:03:59.885
Excellent. So with that introduction, helping folks understand folks who may

00:03:59.885 --> 00:04:06.325
not be so familiar with human factors and then therefore your work and other people from this field.

00:04:06.325 --> 00:04:11.185
Um, is there anything you'd first like to add to that background to kind of

00:04:11.185 --> 00:04:13.625
help orient people on who you are?

00:04:13.965 --> 00:04:18.725
Yeah. Well, again, the broad field of human factors is certainly designing,

00:04:19.025 --> 00:04:22.265
I would say, interfaces with which humans interact.

00:04:23.287 --> 00:04:29.687
Um, in a way that is conscious of the safety of the system in that interaction.

00:04:30.147 --> 00:04:35.287
And it's important to realize that we distinguish between human factors below

00:04:35.287 --> 00:04:37.247
the neck and human factors above the neck.

00:04:37.487 --> 00:04:42.167
Below the neck are all the issues of ergonomics, back place,

00:04:42.647 --> 00:04:46.407
uh, lower back injuries, workplace hazards, and so forth.

00:04:46.627 --> 00:04:52.967
So much dependent upon understanding the muscular skeletal system and human factors above the neck,

00:04:53.287 --> 00:04:58.327
which sometimes is called cognitive engineering, really focuses purely on the

00:04:58.327 --> 00:05:02.927
brain and the sensory system, eyes and ears, and particularly in, again,

00:05:03.127 --> 00:05:09.127
fostering that safe interaction to avoid accidents,

00:05:09.587 --> 00:05:12.607
injuries, and obviously fatalities.

00:05:13.647 --> 00:05:20.127
Yeah. Right. So below the neck might map on to what we normally think of as ergonomics.

00:05:20.247 --> 00:05:22.207
Would you say that's fair to say.

00:05:22.927 --> 00:05:28.367
That's a fair, colloquially ergonomics is usually applied to below the neck,

00:05:28.587 --> 00:05:34.087
you know, designing chairs and, you know, comfortable seating and shoes and things of that sort,

00:05:34.327 --> 00:05:38.907
as well as the whole issue of lifting and forcing and various kinds of,

00:05:38.907 --> 00:05:41.127
you know, heavy industry jobs.

00:05:43.067 --> 00:05:51.947
But the term cognitive ergonomics was designed to emphasize the cognitive aspects of that interaction.

00:05:53.347 --> 00:06:00.307
The term ergonomics in Europe is usually used to refer to all human factors,

00:06:00.647 --> 00:06:05.027
whereas certainly in this country, it's more assumed to be ergonomic.

00:06:06.228 --> 00:06:12.488
Issues of injuries and accidents resulting from muscular skeletal stresses.

00:06:13.568 --> 00:06:19.788
Right. Now, so you've obviously observed a lot of the history of human factors, um, over the decades.

00:06:19.828 --> 00:06:24.928
And I think, um, around the time that, uh, you know, kind of the generation

00:06:24.928 --> 00:06:29.668
that I was a part of getting into it, we heard like, okay, a long time ago, a lot of military stuff.

00:06:29.828 --> 00:06:33.848
It's, it's a little bit cloudy. Maybe if you get introduced to it later,

00:06:33.848 --> 00:06:36.768
um, what would you say in terms of,

00:06:37.308 --> 00:06:41.368
um, that would be nice for us to all to understand in terms of the nature of

00:06:41.368 --> 00:06:46.468
human factors work a little bit about, you know, kind of the early work and,

00:06:46.848 --> 00:06:51.548
um, you know, how that evolved over time from the early parts.

00:06:52.208 --> 00:06:59.668
Yeah. And, you know, we sometimes think of the history of human factors or cognitive human factors, um,

00:07:00.488 --> 00:07:08.068
as, you know, two key figures, um, and two key research thrusts coming out of World War II.

00:07:08.508 --> 00:07:11.568
One was the work that Paul Fitts did in

00:07:11.568 --> 00:07:18.128
this country looking at why pilots would crash perfectly flyable jet aircraft

00:07:18.128 --> 00:07:25.328
in combat and focused at Wright-Patterson Air Force base on identifying that

00:07:25.328 --> 00:07:29.328
these causes are not related to mechanical breakdowns of the plane,

00:07:29.528 --> 00:07:34.188
but rather the way the cockpit instrument panel is laid out,

00:07:34.188 --> 00:07:38.688
leading to confusions of altitude and so forth.

00:07:39.508 --> 00:07:44.028
And from that emanated a lot of work by Paul Fitz on really the fundamental

00:07:44.028 --> 00:07:51.248
designs of the cockpit in particular in aviation that,

00:07:52.050 --> 00:07:55.470
imposed on the limits of human information processing.

00:07:55.850 --> 00:08:02.050
The other strand came from the UK, and this was actually as a result of the

00:08:02.050 --> 00:08:09.250
defense of London during the Blitz in the years of World War II.

00:08:09.470 --> 00:08:15.190
And they were finding that the monitors of the radar displays didn't do a very

00:08:15.190 --> 00:08:19.850
good job of identifying many of the incoming German aircraft.

00:08:19.850 --> 00:08:27.670
Their vigilance failed And this led to a series of research studies by Mackworth

00:08:27.670 --> 00:08:33.070
originally But really the European founder of Human Factors,

00:08:33.130 --> 00:08:39.990
Donald Broadbent On initially what is causing perfectly visible signals to be

00:08:39.990 --> 00:08:46.310
missed on the radar scope And so you had Fitz on this side of the Atlantic Broadbent

00:08:46.310 --> 00:08:53.070
on that side of the Atlantic developing this understanding of how the limits of human attention,

00:08:53.790 --> 00:08:55.690
perception, cognition.

00:08:56.650 --> 00:09:03.770
Motor responding could contribute to these breakdowns in these initially particular

00:09:03.770 --> 00:09:08.830
task areas of aviation and of radar monitoring.

00:09:08.830 --> 00:09:15.450
And so, as a consequence, really the research on aviation human factors led

00:09:15.450 --> 00:09:18.810
the charge throughout the 50s and 60s.

00:09:19.010 --> 00:09:23.450
And really, most of the really good developments were in aviation displays.

00:09:23.850 --> 00:09:30.030
And then on through the 70s, well, what happened was the Three Mile Island nuclear

00:09:30.030 --> 00:09:38.450
power disaster in the late 70s said, well, we could be able to apply a lot of this stuff.

00:09:38.830 --> 00:09:44.510
To process control industry, and indeed, it was done.

00:09:46.010 --> 00:09:49.370
And then another big trigger was the American Institute of Medicine,

00:09:49.550 --> 00:09:58.510
or AIM, report in the late 90s that attributed something like 500,000 deaths

00:09:58.510 --> 00:10:05.130
per year as having human error in medicine as one of many causal factors.

00:10:05.130 --> 00:10:11.110
And so the medical community then began to jump on board and borrow a lot of

00:10:11.110 --> 00:10:17.330
the procedures and techniques from the aviation community, things like checklists, for example.

00:10:18.210 --> 00:10:24.110
To improve the safety and the medical climate.

00:10:24.110 --> 00:10:31.810
Um, the other parallel I think was the gradual realization of the high death

00:10:31.810 --> 00:10:35.650
rate of automobile accidents and the aggressive,

00:10:35.930 --> 00:10:40.310
uh, positions taken by some of the automobile industries to come up with new

00:10:40.310 --> 00:10:44.010
displays that could be both effective, but also ineffective.

00:10:44.630 --> 00:10:47.510
Um, and so in the mid 2000s.

00:10:48.125 --> 00:10:55.125
Mid-90s, I would say, the automotive industry got very much on board with human

00:10:55.125 --> 00:11:01.245
factors, cognitive human factors. And I'm emphasizing this as ergonomics above the neck.

00:11:01.845 --> 00:11:06.525
And the field has just spread ever since then.

00:11:06.825 --> 00:11:14.245
A lot of advances in computer technology have led to concerns with computer interface design.

00:11:14.245 --> 00:11:20.945
That also really came out of the probably the 80s and 90s through some of the

00:11:20.945 --> 00:11:24.125
formative work done at Apple Computer in particular.

00:11:25.925 --> 00:11:32.585
And so now we're talking about designing more usable computer interfaces,

00:11:32.585 --> 00:11:38.205
not necessarily for driving or flying, although that's part of it, or medicine,

00:11:38.445 --> 00:11:41.305
but for everyday applications.

00:11:41.885 --> 00:11:49.005
I think some folks listening to this might get the idea that the frontier is

00:11:49.005 --> 00:11:54.585
therefore on that sort of interface, you know, that appears on a phone or a watch.

00:11:54.725 --> 00:11:58.765
And certainly that's part of the frontier of interface design, right?

00:11:58.885 --> 00:12:03.085
It's all of the sort of consumer devices that we have now.

00:12:03.245 --> 00:12:06.965
And they're much more common of a use case. um do

00:12:06.965 --> 00:12:10.005
you think there's as big as or

00:12:10.005 --> 00:12:12.965
less of a demand or

00:12:12.965 --> 00:12:18.725
a need when we're dealing with some of those sort of hardcore uh survival sort

00:12:18.725 --> 00:12:24.045
of you know high stakes environments um when we're talking about cognitive human

00:12:24.045 --> 00:12:31.185
design well i'd say it depends a little bit on where,

00:12:32.265 --> 00:12:33.885
people's values lie.

00:12:35.345 --> 00:12:38.645
My value lies very much with safety.

00:12:38.925 --> 00:12:43.145
That's one of the, I guess I would say, the moral components of human factors

00:12:43.145 --> 00:12:51.725
that makes me so passionate about it is where it can save lives and reduce injuries.

00:12:52.705 --> 00:12:57.445
And now increasingly, as almost every complex system, safety-critical system

00:12:57.445 --> 00:13:03.665
does involve of a computer interface of some type, definitely,

00:13:03.765 --> 00:13:06.865
you know, these two areas overlap.

00:13:07.958 --> 00:13:11.938
Right so yeah yeah

00:13:11.938 --> 00:13:14.578
and one of the things you mentioned earlier on um that was

00:13:14.578 --> 00:13:17.958
interesting is designing for environments and

00:13:17.958 --> 00:13:20.998
situations where there's a human involved and

00:13:20.998 --> 00:13:24.198
you're supposed to design it with the high

00:13:24.198 --> 00:13:27.178
um sort of intentionality that a

00:13:27.178 --> 00:13:30.378
human is going to be using the system although you're designing

00:13:30.378 --> 00:13:34.218
this whole system that's got a lot of different pieces and

00:13:34.218 --> 00:13:37.218
factors and you know therefore a human factor

00:13:37.218 --> 00:13:40.098
is that does that map onto the original way of

00:13:40.098 --> 00:13:42.878
thinking about human factors and and the

00:13:42.878 --> 00:13:45.778
and even the term human factor that you're it's a human

00:13:45.778 --> 00:13:51.518
factor in a larger system yeah it does the sense of looking at the human factors

00:13:51.518 --> 00:13:56.378
in the larger system i think what's evolved so much is the sophistication of

00:13:56.378 --> 00:14:02.878
automation ai and so forth that humans are no longer interacting directly with

00:14:02.878 --> 00:14:04.558
mechanical systems that,

00:14:04.558 --> 00:14:07.518
you know, obey the natural laws of physics.

00:14:08.158 --> 00:14:10.698
But again, interacting with computers.

00:14:10.938 --> 00:14:14.358
It's certainly true of the modern commercial aircraft in which,

00:14:14.518 --> 00:14:17.918
you know, there are literally millions of lives of code.

00:14:18.358 --> 00:14:22.898
And that has a lot of implications, but, you know, one of the implications,

00:14:23.158 --> 00:14:26.198
and I guess this is one of the, we talk about this evolution kind

00:14:26.198 --> 00:14:29.618
of from the 60s to the 70s to the 80s and the

00:14:29.618 --> 00:14:33.258
increasing emphasis on the

00:14:33.258 --> 00:14:39.578
cognitive aspects not perceptual not motor or voice but the cognitive aspects

00:14:39.578 --> 00:14:46.318
of understanding what the heck the automation is doing you know whether it's

00:14:46.318 --> 00:14:51.638
the autopilot in the tesla or the automatic,

00:14:51.638 --> 00:14:52.218
you know,

00:14:52.378 --> 00:14:58.438
diagnostic system in medicine, and certainly whether it's issues on the flight deck.

00:14:59.333 --> 00:15:03.113
Yeah. Right. Um, and you have a, go ahead.

00:15:03.313 --> 00:15:07.273
Oh, I was going to say there's a whole nother domain that I haven't talked about,

00:15:07.453 --> 00:15:11.713
which is a huge consumer of good cognitive human factors.

00:15:11.733 --> 00:15:15.573
And that really is in the department of defense, military systems,

00:15:15.793 --> 00:15:21.673
defense systems, offensive systems, uh, protective systems, decision aids, things of that sort.

00:15:22.233 --> 00:15:25.313
Yeah. Now, in terms of a lot of the changes you, uh,

00:15:25.513 --> 00:15:28.173
have witnessed over time, you had a story that you

00:15:28.173 --> 00:15:30.973
had started to tell me last week i wanted you to share it here about

00:15:30.973 --> 00:15:34.133
when you were writing your dissertation and sort of the

00:15:34.133 --> 00:15:36.973
computer uh the analog computer and stuff

00:15:36.973 --> 00:15:39.713
yeah well it's a little it's kind of

00:15:39.713 --> 00:15:44.833
an amusing anecdote um when i first started in graduate school at michigan and

00:15:44.833 --> 00:15:49.713
down the basement of the laboratory there were two doors one door open to the

00:15:49.713 --> 00:15:55.673
pdp1 in which you could learn to You program a digital computer with thousands

00:15:55.673 --> 00:15:58.313
and thousands of computer cards,

00:15:58.813 --> 00:16:06.073
card readers, or the other door open to an analog computer where NASA had supported

00:16:06.073 --> 00:16:09.273
a lot of our aerospace human factors,

00:16:09.273 --> 00:16:19.713
basically using voltages to mimic the forces in space as you control an aircraft or a spacecraft.

00:16:20.373 --> 00:16:25.493
I don't know if I did an eeny, meeny, miny, moe. Somehow I chose the door that

00:16:25.493 --> 00:16:31.553
opened to the analog computer, and that's where my advisor was working as well.

00:16:31.953 --> 00:16:38.033
And so I developed this real passion for control of analog systems,

00:16:38.213 --> 00:16:41.873
in particular aircraft, because they obey the laws of physics.

00:16:41.873 --> 00:16:50.113
Um, and, uh, I, I kind of missed a lot of the digital revolution as a result of that.

00:16:50.393 --> 00:16:57.073
Um, the irony is that now that digitization had become so rapidly,

00:16:57.273 --> 00:17:03.073
they're now using digital computers, obviously to simulate a lot of analog systems, uh,

00:17:03.313 --> 00:17:08.013
you know, whether it's virtual control or joystick games and so forth.

00:17:08.013 --> 00:17:10.753
And interestingly now a lot

00:17:10.753 --> 00:17:14.093
of people who are doing that are sort of rediscovering all the

00:17:14.093 --> 00:17:16.973
stuff we discovered in the

00:17:16.973 --> 00:17:20.313
60s and 70s and 80s on you

00:17:20.313 --> 00:17:26.213
know using analog systems so right because there's a there's most folks switched

00:17:26.213 --> 00:17:30.293
away from the analog but you still had an analog crowd that understood some

00:17:30.293 --> 00:17:35.113
of the dynamics there yeah um and so the gaming industry for example or everything

00:17:35.113 --> 00:17:38.533
on joysticks and moving things around and flying through space.

00:17:38.773 --> 00:17:43.413
There's a lot of human factors in there. And sometimes I think if the people

00:17:43.413 --> 00:17:47.453
that were designing games could actually look back to the original research

00:17:47.453 --> 00:17:51.833
that we had done on control of analog systems, they'd be.

00:17:52.357 --> 00:17:56.617
They wouldn't, they would be reusing things rather than rediscovering things

00:17:56.617 --> 00:17:58.017
that we had discovered long ago.

00:17:58.477 --> 00:18:01.477
So, okay. So just to get into that a little bit, like, uh,

00:18:01.637 --> 00:18:06.217
you know, for the audience and also myself thinking about a joystick, if I'm pushing on it,

00:18:06.317 --> 00:18:13.557
it's a continuous sort of scale mapping the force of my hand onto the input

00:18:13.557 --> 00:18:17.357
that I desire to do versus digital where I just say up,

00:18:17.617 --> 00:18:19.557
left, upper left. Exactly. Right.

00:18:19.877 --> 00:18:23.237
That's correct. Yeah. so then when you digitize

00:18:23.237 --> 00:18:26.817
there's some nuances or details lost there

00:18:26.817 --> 00:18:29.957
and then in a digital world you have to figure out how to regain it

00:18:29.957 --> 00:18:35.037
yeah well you have to do now and now you're obviously when you push the stick

00:18:35.037 --> 00:18:39.137
you're no longer changing voltage you're just increasing the the number of bits

00:18:39.137 --> 00:18:45.097
or the amount of information on something on a digital scale but we can tell

00:18:45.097 --> 00:18:48.697
you a lot not you personally, but, uh,

00:18:49.097 --> 00:18:53.937
about how to optimize the gain, the responsiveness of a stick based upon what

00:18:53.937 --> 00:18:58.617
we learned back in the, uh, you know, in the sixties and seventies about optimal

00:18:58.617 --> 00:19:01.997
gains in flight control settings, for example.

00:19:02.477 --> 00:19:06.897
Yeah. So interesting. Um, and so I know you spent a lot of your careers, uh,

00:19:07.177 --> 00:19:10.497
specifically interested in attention and I'm

00:19:10.497 --> 00:19:13.557
interested in how you got into that i was watching a talk where

00:19:13.557 --> 00:19:16.637
you um you mentioned that you'd

00:19:16.637 --> 00:19:20.657
read um the book by danny daniel kahneman um

00:19:20.657 --> 00:19:26.237
attention and effort what was it that moment and what was it about that book

00:19:26.237 --> 00:19:30.997
that kind of pulled you in and what were you thinking that made you feel that

00:19:30.997 --> 00:19:36.097
this was an important area to research well by the way a precursor to that was

00:19:36.097 --> 00:19:38.617
when i was in the navy in the late 60s.

00:19:38.937 --> 00:19:42.137
And for my first tour of duty was on board a ship.

00:19:42.317 --> 00:19:47.757
I was a damage control assistant responsible for basically managing all the

00:19:47.757 --> 00:19:52.617
hull damage of a ship as in battle, you know, flooding, fires, and so forth.

00:19:52.937 --> 00:19:59.617
I had a central workstation, DC Central, and I was totally overloaded.

00:19:59.677 --> 00:20:05.237
And I could feel my attentional system being strained to the utmost in this

00:20:05.237 --> 00:20:10.317
multitasking environment of absorbing lots of information, talking on the phone,

00:20:10.457 --> 00:20:11.497
giving all sorts of commands.

00:20:11.677 --> 00:20:16.557
So I experienced attentional overload in spades during that time.

00:20:16.937 --> 00:20:23.017
Went to Michigan, got involved in doing research on flight control.

00:20:23.517 --> 00:20:25.297
Basically tracking tasks.

00:20:25.577 --> 00:20:31.097
And I had this thought, well, everybody has done a lot of research on the flight

00:20:31.097 --> 00:20:33.437
control in a single task environment.

00:20:33.937 --> 00:20:39.077
But what happens if you start loading a controller with concurrent tasks?

00:20:39.477 --> 00:20:42.957
And so that was the topic of my dissertation. Believe it or not,

00:20:43.117 --> 00:20:49.477
that had not been done before very much, if at all. So.

00:20:50.440 --> 00:20:55.000
I became very interested in how multitasking affects flight control while you're

00:20:55.000 --> 00:20:57.200
doing other perceptual motor tasks.

00:20:57.500 --> 00:21:02.720
I picked up Daniel Kahneman's book, and I think many people know Kahneman from

00:21:02.720 --> 00:21:08.760
his work on decision-making heuristics with Amos Tversky, a really genius work.

00:21:09.040 --> 00:21:16.100
But he is kind of like, as a side interest of him, he decided to write this

00:21:16.100 --> 00:21:17.420
book on attention and effort.

00:21:18.340 --> 00:21:24.940
And to me, effort and attention are closely linked. We only have so much effort we can give to a task.

00:21:25.460 --> 00:21:28.580
We only have so much attention we can give to a task.

00:21:29.680 --> 00:21:36.260
We are limited in our cognitive resources. And he just, he's a beautiful writer,

00:21:36.260 --> 00:21:41.540
and he expounded on that relationship in very nice writing.

00:21:41.540 --> 00:21:48.860
So that kind of inspired my train of work at this time. What are the limits

00:21:48.860 --> 00:21:50.580
of attention in multitasking?

00:21:51.520 --> 00:21:57.280
And I did a dissertation. And as often happens when you do a dissertation or

00:21:57.280 --> 00:22:01.420
advanced research project, you get some spinoff effects that you didn't expect.

00:22:02.458 --> 00:22:08.978
And one of the spinoff effects I got was the fact, well, it was interpreted by me and a few others,

00:22:09.458 --> 00:22:14.198
one of them is Kahneman's students, as the fact that our brain does not have

00:22:14.198 --> 00:22:18.518
a single pool of resources that we can allocate to all tasks.

00:22:18.518 --> 00:22:24.578
In fact, it has separate resources and that allows or avails better timesharing.

00:22:25.258 --> 00:22:30.238
If task A can use one type of resource, task B can use the other type of resource,

00:22:30.458 --> 00:22:35.458
you can do them at the same time better. You can timeshare them with a smaller decrement.

00:22:36.118 --> 00:22:39.678
Yeah. So, you know, making sure that, you know, everyone kind of like appreciates this.

00:22:39.858 --> 00:22:42.778
It's not none of this stuff is obvious before you research it.

00:22:42.918 --> 00:22:48.178
So even thinking about it as a limited resource. So you had early writings about

00:22:48.178 --> 00:22:52.678
attention, like William James would write about attention, like a flashlight,

00:22:52.718 --> 00:22:54.738
and you kind of direct it.

00:22:54.838 --> 00:22:57.438
And then we all intuitively know, like, oh, you're kind of paying attention

00:22:57.438 --> 00:22:58.518
to something or you're not.

00:22:58.598 --> 00:23:03.938
And you don't think too deeply about it. But there's high stakes scenarios like

00:23:03.938 --> 00:23:09.278
operating an aircraft or many other things where you have to direct your attention

00:23:09.278 --> 00:23:11.378
as the human being in the system.

00:23:11.838 --> 00:23:16.538
And it's an important to, you know, as a decision maker, that you have to be

00:23:16.538 --> 00:23:19.798
gathering the information from the right things at the right time.

00:23:19.798 --> 00:23:25.658
And then attention as a resource, it's not necessarily obvious that it's limited

00:23:25.658 --> 00:23:30.458
like a resource, like, you know, something that you have a limited amount of

00:23:30.458 --> 00:23:32.678
and you can use it in different ways.

00:23:33.598 --> 00:23:37.058
So um how should people think about attention

00:23:37.058 --> 00:23:40.118
you know so you know selectiveness the different

00:23:40.118 --> 00:23:43.338
types of attention whether it's divided selected all

00:23:43.338 --> 00:23:47.378
that kind of stuff well you know what's what's kind of help us wrap our minds

00:23:47.378 --> 00:23:52.898
around how to uh think about attention you know and attention is obviously a

00:23:52.898 --> 00:24:00.118
hugely complex field as is memory as is learning as is decision making the way

00:24:00.118 --> 00:24:01.938
i like to think about it, though,

00:24:02.218 --> 00:24:04.738
is attention is a limited resource.

00:24:04.738 --> 00:24:06.918
We always can understand that.

00:24:07.318 --> 00:24:11.898
What we don't understand as well is what those limits are.

00:24:12.238 --> 00:24:16.498
You know, how many bits per second can the human brain process?

00:24:16.678 --> 00:24:20.518
What's the upper limit as the human as a transmitter, a bandwidth,

00:24:20.858 --> 00:24:25.898
say in, you know, the old days reading and transcription, you're reading a message

00:24:25.898 --> 00:24:30.058
and you're typing it okay so that's one aspect what are the limits of attention

00:24:30.058 --> 00:24:33.538
but then given that that attention is limited it.

00:24:34.362 --> 00:24:38.562
If we go into a task environment, such as the aircraft cockpit,

00:24:38.822 --> 00:24:44.422
which is where I spent a lot of my early research career, and look at the number

00:24:44.422 --> 00:24:47.082
of tasks the pilot's asked to perform,

00:24:47.362 --> 00:24:54.982
and are there ways of exploiting multiple pools of attention to increase the

00:24:54.982 --> 00:24:59.282
bandwidth of processing in these overload situations?

00:24:59.282 --> 00:25:09.622
And two very common sense solutions that I think our early research helped evolve is,

00:25:09.902 --> 00:25:15.962
yeah, the pilot's world is heavily visual, not a whole lot is coming in the ears.

00:25:15.962 --> 00:25:21.122
And to the extent that the visual processes use different resources from the

00:25:21.122 --> 00:25:26.222
auditory process, think of the different cortices, cortex, then we ought to

00:25:26.222 --> 00:25:29.662
be able to offload stuff to the auditory task.

00:25:30.482 --> 00:25:34.402
Hence the development of, you know, synthetic voice displays, for example.

00:25:34.582 --> 00:25:38.242
Not everything that happens has to be, you know, presented visually.

00:25:38.402 --> 00:25:43.542
Now you've got computers that can recognize states and synthesize the voice.

00:25:43.882 --> 00:25:46.062
That increases the bandwidth.

00:25:46.902 --> 00:25:50.822
Same thing is true on the output side. Certainly in the traditional aircraft

00:25:50.822 --> 00:25:56.662
cockpit, so much of the, almost all of the inputs the human gives to the plane

00:25:56.662 --> 00:25:59.562
were either keyboards or joysticks.

00:25:59.662 --> 00:26:05.742
And so forth, but hey, you've got a voice channel that's rarely used other than, say, communications.

00:26:06.222 --> 00:26:15.622
Can we exploit voice control to increase the bandwidth of conveying information to the system?

00:26:16.002 --> 00:26:20.382
And yes, you can for certain kinds of tasks under certain kinds of situations.

00:26:22.042 --> 00:26:27.382
And so there we're looking at different resources associated with auditory and

00:26:27.382 --> 00:26:31.242
visual processing as well as vocal and manual processing.

00:26:32.802 --> 00:26:40.582
It gets more complex, but we've done some work on this if we look at the fact that to some extent,

00:26:40.762 --> 00:26:45.482
the human has different resources in the right brain for most of its spatial

00:26:45.482 --> 00:26:49.142
thinking and the left brain for most of its linguistic thinking.

00:26:49.862 --> 00:26:55.042
And are there environments in which we can better take advantage of processing

00:26:55.042 --> 00:27:01.182
by, should we say, combining spatial and verbal cognition.

00:27:02.442 --> 00:27:09.262
Combine the demands on working memory, on memory for both images and spatial

00:27:09.262 --> 00:27:11.902
information as well as linguistics.

00:27:12.874 --> 00:27:18.474
And so this was all a long line of research that led us to, you know,

00:27:18.554 --> 00:27:21.014
try to recommend design solutions,

00:27:21.394 --> 00:27:25.034
particularly in high-stake environments where there's a lot of information to

00:27:25.034 --> 00:27:29.994
be processing that can better exploit different resources.

00:27:30.614 --> 00:27:34.474
Right. So when we're thinking about attention and we're thinking about, um,

00:27:34.914 --> 00:27:39.714
the, you know, sort of the subtopics, of course, there's a giant field as,

00:27:39.794 --> 00:27:46.194
as you, um, as you mentioned, but, you know, things like you can direct your

00:27:46.194 --> 00:27:48.734
attention and your attention can be interrupted.

00:27:48.734 --> 00:27:52.814
That might be like one sort of area of thinking.

00:27:52.994 --> 00:27:58.134
And then another one is how do I, if I decide that I'm multitasking or my situation

00:27:58.134 --> 00:28:02.054
makes me multitask, how many errors will I make?

00:28:02.234 --> 00:28:07.094
You know, how, you know, do they conflict with each other if given the different

00:28:07.094 --> 00:28:12.234
types of tasks, some type of tasks, bad example, but walking and chewing gum.

00:28:12.234 --> 00:28:17.134
But a similar concept where they don't interfere with each other versus some

00:28:17.134 --> 00:28:20.034
things that you're doing at the same time really do interfere with each other

00:28:20.034 --> 00:28:23.974
and it has to be more serialized in terms of you're trying to execute it.

00:28:24.734 --> 00:28:28.134
Yeah, and Thomas, that's a real important dichotomy and this is something,

00:28:28.514 --> 00:28:35.954
again, fundamental to my approach of attention multitasking is two different modes.

00:28:36.654 --> 00:28:41.734
One is concurrent performance, you know, and I can certainly not only walk and

00:28:41.734 --> 00:28:46.114
chew gum at the same time, but I can certainly drive a car and converse with the passenger.

00:28:46.214 --> 00:28:49.734
I can drive a car and listen to the radio, listen to a podcast,

00:28:50.194 --> 00:28:53.294
listen to Thomas Watkins talking about whatever's on his mind,

00:28:53.494 --> 00:28:57.954
uh, with very little interference between them until things get really challenging.

00:28:58.254 --> 00:29:04.974
But many other situations that's impossible. And so one regresses to a sequential

00:29:04.974 --> 00:29:08.154
processing mode rather than the concurrent processing mode.

00:29:09.021 --> 00:29:13.601
And now I'm in sequential processing mode, and there's a whole different set

00:29:13.601 --> 00:29:20.121
of rules that govern the extent to which I do one task versus another, okay?

00:29:20.321 --> 00:29:28.521
Listen to my passenger versus listening to, say, a navigational command or a talk show.

00:29:28.721 --> 00:29:31.981
Now we're talking about switching. Multitasking is switching.

00:29:33.081 --> 00:29:41.461
And so what are the rules for effective multitasking and switching versus ineffective

00:29:41.461 --> 00:29:44.181
multitasking switching when it's switching?

00:29:44.621 --> 00:29:49.081
So I guess I would say if you're in a concurrent processing mode,

00:29:49.081 --> 00:29:55.701
we say, what are the rules for choosing resources that can both process information at the same time?

00:29:55.701 --> 00:30:01.021
But now if you're forced to be in a sequential processing mode,

00:30:01.521 --> 00:30:07.061
you know, again, think of reading a text while looking at the roadway,

00:30:07.541 --> 00:30:10.521
you know, clearly sequential processing.

00:30:10.881 --> 00:30:17.281
What are the rules that says how people do that when there are true violations

00:30:17.281 --> 00:30:21.041
of safety that are a result of that sequential processing mode? it.

00:30:21.321 --> 00:30:26.901
So then we've got a whole nother area of attention as a sequential processing

00:30:26.901 --> 00:30:29.201
and attention as a concurrent processing.

00:30:29.601 --> 00:30:33.541
And then vigilance of sustaining your attention right?

00:30:34.534 --> 00:30:38.374
Uh, slots into that. It, it sort of slots into that.

00:30:39.314 --> 00:30:44.194
Vigilance is a different kettle of fish. I, I don't actually always think of

00:30:44.194 --> 00:30:45.774
vigilance as being part of attention.

00:30:45.774 --> 00:30:50.894
It's not something I've done much research on because, you know,

00:30:51.014 --> 00:30:55.654
this is under arousal and I focus on the environments where they're typically

00:30:55.654 --> 00:31:00.634
over arousal or over overload versus under load.

00:31:00.774 --> 00:31:04.214
And the vigilance task usually kicks in when there's under load.

00:31:04.534 --> 00:31:08.734
And then what causes the lapses of attention under those underload situations.

00:31:09.354 --> 00:31:13.214
Gotcha. Okay. It's important in many environments, and particularly the way

00:31:13.214 --> 00:31:16.214
it interacts with sleep deprivation and mind wandering.

00:31:16.434 --> 00:31:22.414
But it's not an area that I have chosen to spend a lot of my research effort on.

00:31:22.414 --> 00:31:28.534
Now, in terms of when we think about AI and automation and increasing number

00:31:28.534 --> 00:31:31.314
of tasks being, you know,

00:31:31.874 --> 00:31:36.994
performed by machines that once were performed by humans and,

00:31:37.114 --> 00:31:39.614
you know, but humans are still part of the equation.

00:31:39.614 --> 00:31:42.274
So you know what are

00:31:42.274 --> 00:31:45.394
some of the as you see it um well maybe

00:31:45.394 --> 00:31:48.434
we could take a step back and talk about what you view ai.

00:31:48.434 --> 00:31:51.174
As period to begin

00:31:51.174 --> 00:31:54.374
with and then how that relates to sort

00:31:54.374 --> 00:31:58.214
of attention related tasks and decision making um

00:31:58.214 --> 00:32:01.034
uh um you know

00:32:01.034 --> 00:32:03.934
when when a human decides to use the ai

00:32:03.934 --> 00:32:06.774
or insert the ai into the process versus when they

00:32:06.774 --> 00:32:09.874
don't um and how this uh how

00:32:09.874 --> 00:32:13.734
that uh the issue of automation by machines um

00:32:13.734 --> 00:32:18.694
relates to all of this but um if we take a step back and you know just start

00:32:18.694 --> 00:32:25.014
by picking your brain about what is ai to you um and you know how that relates

00:32:25.014 --> 00:32:31.194
to some of these topics we're getting into interesting theme and thread yeah and uh so.

00:32:32.419 --> 00:32:36.299
Let me start with kind of the definition. There's automation.

00:32:36.819 --> 00:32:42.239
These are basically machines doing things, machines and computers that humans used to do.

00:32:42.419 --> 00:32:47.539
You know, that's kind of a thread that it's things that humans used to do,

00:32:47.719 --> 00:32:49.359
could do, but no longer done.

00:32:49.839 --> 00:32:56.379
Some of those things are very simple things. I always use the electric can opener as an example.

00:32:56.659 --> 00:33:00.859
You put your can in and press a button and it opens instead of doing this.

00:33:01.759 --> 00:33:08.659
Does not really involve intelligent action, but many of them are cognitive things.

00:33:09.559 --> 00:33:13.819
And so when we talk about cognitive things, we're talking about intelligent operations.

00:33:14.219 --> 00:33:18.219
And so that's where artificial intelligence lies as far as I'm concerned.

00:33:18.499 --> 00:33:27.099
It's the automation of more cognitive types of behavior rather than purely perceptual motor behavior.

00:33:27.099 --> 00:33:29.759
But you see it as just it's it's

00:33:29.759 --> 00:33:32.539
really just automation because a lot of times when

00:33:32.539 --> 00:33:35.659
it's talked about it's kind of talked about as you know

00:33:35.659 --> 00:33:39.039
humans are building an intelligence and perhaps that's true

00:33:39.039 --> 00:33:41.879
or or whatever but that i think we

00:33:41.879 --> 00:33:44.919
intuitively have an idea about intelligence that it's

00:33:44.919 --> 00:33:48.559
this sort of sentient capability of a

00:33:48.559 --> 00:33:51.819
thinker and it has ideas and

00:33:51.819 --> 00:33:55.439
it has its own internal world versus you

00:33:55.439 --> 00:34:00.519
know something that is kind of like a can opener but just way more sophisticated

00:34:00.519 --> 00:34:08.279
well yeah i and let's see how to like yeah i mean there's intelligent behavior

00:34:08.279 --> 00:34:15.399
it's pretty simple you know i mean clearly choosing which control to use or you know.

00:34:16.519 --> 00:34:21.319
Those simple choices simple decisions i consider them intelligent even if they're

00:34:21.319 --> 00:34:25.339
simple they're not, you know, higher level cognition involved in creativity

00:34:25.339 --> 00:34:29.059
or, you know, writing papers and, and, uh,

00:34:29.259 --> 00:34:33.099
and figuring out, you know, complex optimization problems.

00:34:33.219 --> 00:34:37.279
You know, what's the best way to, uh, drive around a city or to,

00:34:37.499 --> 00:34:40.819
for a pilot to dodge, uh, you know, storms.

00:34:41.299 --> 00:34:49.519
So yeah, there's huge ranges in the cognitive demands, but I would always still

00:34:49.519 --> 00:34:51.279
call this artificial intelligence.

00:34:52.557 --> 00:34:58.877
But there's artificial intelligence of very simple tasks and obviously more complex tasks.

00:35:00.697 --> 00:35:06.017
And the reason I'm emphasizing this is you certainly look at the media and there's

00:35:06.017 --> 00:35:09.677
this tendency to think, wow, AI is new.

00:35:10.057 --> 00:35:16.397
Look what this introduction of AI has done to society in the last five,

00:35:16.717 --> 00:35:19.677
four years, three years, whatever it is.

00:35:19.677 --> 00:35:25.517
And I said, you know, we were, you know, aviation designers were developing

00:35:25.517 --> 00:35:31.197
AI for aircraft algorithms to fly the aircraft as opposed to the pilots,

00:35:31.457 --> 00:35:34.117
you know, way back in the 50s and 60s, you know.

00:35:34.297 --> 00:35:41.177
And indeed, you could argue that a lot of the early development of AI or automated

00:35:41.177 --> 00:35:47.817
decision routines took place in the flight deck, the development of the so-called

00:35:47.817 --> 00:35:48.997
flight management system.

00:35:49.677 --> 00:35:54.917
By Boeing and Airbus and others back in the, you know, 50s and 60s.

00:35:57.117 --> 00:36:02.037
And so, you know, so to me, AI is not new,

00:36:02.337 --> 00:36:09.017
but what is new is the complexity of the things that AI are doing and the challenges

00:36:09.017 --> 00:36:16.717
to a human and AI or automation working together as a fluent team.

00:36:17.517 --> 00:36:25.257
So that's why i my current research interest is in the hat human automation team performance.

00:36:26.317 --> 00:36:31.697
Not looking at automation as replacing the human but what's the best synergy between the two,

00:36:32.397 --> 00:36:37.537
when do you start getting too much authority for the automation that the human

00:36:37.537 --> 00:36:43.217
remains out of the loop and doesn't understand what's happening when the ai algorithms fail,

00:36:44.248 --> 00:36:48.888
Or the converse, when does the imbalance start being such that the human's doing

00:36:48.888 --> 00:36:53.368
everything, the AI is doing very little, and therefore the purpose of AI is

00:36:53.368 --> 00:36:58.288
really defeated, which is to offload the human when necessary.

00:36:59.268 --> 00:37:04.428
What would be a design application of that? So, you know, so today you might

00:37:04.428 --> 00:37:10.388
have a person who works in an office and they say, hey, I don't feel like really writing this email.

00:37:10.628 --> 00:37:16.488
Let me pull up ChatGPT and then put in a few prompts and then get an email that

00:37:16.488 --> 00:37:19.548
I liked and then paste it into the email and send it.

00:37:19.548 --> 00:37:25.368
But where is our opportunity, or are we talking about right now we're kind of

00:37:25.368 --> 00:37:30.208
at the theory and just discovering maybe things about the relationship between

00:37:30.208 --> 00:37:34.928
the human intelligence and the machine intelligence and what is the right mix

00:37:34.928 --> 00:37:38.068
of it and what's the right approach to using both of them?

00:37:38.908 --> 00:37:43.028
Is it like in the sort of the theoretical construct, or do we have stuff that

00:37:43.028 --> 00:37:46.488
we can apply that you know if we're building a system? We have stuff that we

00:37:46.488 --> 00:37:50.348
can apply and are, or at least should be applying right now.

00:37:50.488 --> 00:37:55.268
And I would say the most visible application is the self-driving car.

00:37:55.588 --> 00:38:03.788
You know, how much does the autopilot take over of human driving and what are the consequences?

00:38:04.528 --> 00:38:08.888
And what we like to focus on is two generic consequences.

00:38:09.208 --> 00:38:13.468
One is what happens in the moment when AI fails.

00:38:13.468 --> 00:38:20.368
And all we need to do is look at the various particular Tesla accidents where, you know,

00:38:20.408 --> 00:38:29.008
the sensor has failed to recognize a hazard and therefore the driver runs into

00:38:29.008 --> 00:38:32.228
the hazard if he or she is out of the loop.

00:38:33.168 --> 00:38:37.608
And so that's one issue that we look at right now. How much is too much?

00:38:37.868 --> 00:38:42.568
Do we mandate that the human always have their hands on the steering wheel and

00:38:42.568 --> 00:38:45.668
the moment they're off, the AI disconnects?

00:38:47.908 --> 00:38:52.408
Do we need to have not only the hands on the steering wheel,

00:38:52.588 --> 00:38:55.828
which is easily sensed, but also the eyes open?

00:38:56.248 --> 00:39:00.288
And do the eyes not only have to be open, but they have to be focused on the

00:39:00.288 --> 00:39:02.848
road ahead and not on reading a book.

00:39:03.008 --> 00:39:08.768
So those kinds of issues are very real, having to do with how AI should be designed

00:39:08.768 --> 00:39:13.628
and, you know, what the appropriate safety constraints should be.

00:39:14.108 --> 00:39:20.488
So I mentioned this first issue is sort of the real-time issue of what happens

00:39:20.488 --> 00:39:28.748
when automation fails or fails to do the task that the human expects it to do. And that question...

00:39:29.944 --> 00:39:36.324
It pervades over driving, piloting, medical operations, medical procedures,

00:39:37.204 --> 00:39:41.564
combat, so forth, choosing an adversary.

00:39:42.104 --> 00:39:51.244
The other issue is a little bit less salient, but you kind of alluded to it

00:39:51.244 --> 00:39:55.524
in letting CHAP-GPT write X or Y or Z for you.

00:39:55.524 --> 00:40:00.624
And that is what eventually is going to happen to the human skill in writing.

00:40:00.984 --> 00:40:07.124
And we've got plenty of evidence from the flight deck that pilots who allow

00:40:07.124 --> 00:40:12.504
the autopilot to fly the plane too much lose their skill in flying,

00:40:12.524 --> 00:40:16.504
which is why there's this importance of bringing them back into the loop,

00:40:16.604 --> 00:40:18.444
having to fly the plane manually.

00:40:18.444 --> 00:40:21.664
Even if it's one of big humongous in

00:40:21.664 --> 00:40:24.864
a simulator just so they can reenter so they

00:40:24.864 --> 00:40:28.684
can retain the skill and we in human

00:40:28.684 --> 00:40:32.004
automation interaction research talk about this concept of

00:40:32.004 --> 00:40:35.424
de-skilling i use the example of um

00:40:35.424 --> 00:40:38.144
ai on a calculator how many of us

00:40:38.144 --> 00:40:42.104
can do long division as fluently when as

00:40:42.104 --> 00:40:45.184
we could back when we were doing it by by hand

00:40:45.184 --> 00:40:48.084
yeah um this is such an

00:40:48.084 --> 00:40:50.904
interesting topic because as you were talking about that i was thinking about when i

00:40:50.904 --> 00:40:53.784
drive a more advanced car that starts beeping

00:40:53.784 --> 00:40:57.624
when you're about to make a change lanes then

00:40:57.624 --> 00:41:01.404
i think about well i'm fairly good at changing lanes but if i started depending

00:41:01.404 --> 00:41:07.644
on that system too much well i'll start getting deskilled eventually and i'd

00:41:07.644 --> 00:41:11.764
have to if i wanted to maintain that skill i'd have to find some opportunity

00:41:11.764 --> 00:41:15.084
to train it and myself to keep it.

00:41:15.664 --> 00:41:19.204
Exactly. And two issues are directly relevant to this.

00:41:19.384 --> 00:41:24.544
First of all, is identifying what are the kinds of errors that automation makes?

00:41:24.704 --> 00:41:28.424
How frequently do they occur? What is the reliability of automation?

00:41:29.024 --> 00:41:32.584
And then secondly, and this is where I come back to my other love in life,

00:41:32.864 --> 00:41:38.444
what is the human's attentional capabilities of monitoring what automation is

00:41:38.444 --> 00:41:40.544
doing, even if I'm not doing it myself?

00:41:40.984 --> 00:41:44.724
And here's where attention as a visual spotlight comes in.

00:41:45.164 --> 00:41:51.304
What happens to the eye movements? How can we keep the eye brain loop engaged

00:41:51.304 --> 00:41:58.584
in that task at a sufficient level so that if automation fails, you can take over?

00:41:59.604 --> 00:42:06.804
So how about, tell us about AR and your work and interest in augmented reality.

00:42:07.704 --> 00:42:13.144
Yeah. Well, okay. I mean, basically, in part, we're back to the displays again.

00:42:14.584 --> 00:42:22.364
I came on this from really in the 90s by looking at how important the visual

00:42:22.364 --> 00:42:28.564
system is in monitoring multiple channels in flying and driving.

00:42:29.683 --> 00:42:35.843
That the proposed solution to this in aviation with Alaska Airlines initially

00:42:35.843 --> 00:42:38.203
was to take the pilot's instrument panel,

00:42:38.423 --> 00:42:43.703
move it up on the glass so the pilot could now see the world beyond through

00:42:43.703 --> 00:42:47.523
the display of instruments on the glass.

00:42:48.083 --> 00:42:49.663
Was it a good design?

00:42:50.383 --> 00:42:55.563
Yes, it was in general, except anytime you're looking through something to see

00:42:55.563 --> 00:42:59.943
something else, as you are looking through my fingers to see my face,

00:43:00.183 --> 00:43:01.383
there's going to be clutter.

00:43:01.663 --> 00:43:07.743
This is creating clutter. And it may not actually be obscuring what I'm seeing,

00:43:07.823 --> 00:43:11.803
but nevertheless, just like bugs on the windshield,

00:43:12.363 --> 00:43:18.703
literal bugs, squash bugs on the windshield, that clutter can disrupt the parallel

00:43:18.703 --> 00:43:19.983
processing of information.

00:43:20.443 --> 00:43:27.103
So superimposing things may be a good idea, but it's not the perfect solution.

00:43:27.103 --> 00:43:34.063
Can you do better? So I did a lot of those, that research in aviation and cars

00:43:34.063 --> 00:43:37.863
on head-up displays in the 90s and the early aughts.

00:43:38.843 --> 00:43:43.063
When I got out here to Colorado State University, actually in retirement,

00:43:43.443 --> 00:43:46.423
I started collaborating with a computer scientist.

00:43:47.383 --> 00:43:53.643
And he was very interested in the extent to which the same issues with head-up

00:43:53.643 --> 00:43:57.403
displays now holds true with head-mounted displays.

00:43:58.536 --> 00:44:04.436
And, you know, head-mounted display can present any type of information overlapping the visual field.

00:44:04.656 --> 00:44:10.776
But then when that information is synced, synchronized to my direction of head

00:44:10.776 --> 00:44:13.216
movement, now we're talking about augmented reality.

00:44:13.616 --> 00:44:20.016
You know, we're talking about putting a picture of something that is overlaying

00:44:20.016 --> 00:44:23.716
the actual thing no matter which way I'm turning.

00:44:23.716 --> 00:44:31.536
So, my current interests are particularly in just head-mounted displays in general,

00:44:31.696 --> 00:44:38.636
but also head-mounted displays that either represent AR information,

00:44:38.636 --> 00:44:43.456
which therefore requires sensors of head orientation so you can move things

00:44:43.456 --> 00:44:45.976
across the display to keep them supervised,

00:44:46.196 --> 00:44:50.376
or simply AR of other types of information.

00:44:50.376 --> 00:44:54.316
For example, a chat box, you know, this up there and you can read the chat at

00:44:54.316 --> 00:44:57.436
the same time you're monitoring the traffic perhaps.

00:44:58.476 --> 00:45:01.636
Or since we were doing a lot of this work for the Marine Corps,

00:45:01.916 --> 00:45:06.916
you know, looking out at a cluttered environment to try to find an adversary

00:45:06.916 --> 00:45:08.556
out there hiding in the bush someplace.

00:45:09.716 --> 00:45:12.876
And you know will the head-mounted display

00:45:12.876 --> 00:45:15.536
help that process because you

00:45:15.536 --> 00:45:19.096
don't have to move your eyes or will it

00:45:19.096 --> 00:45:22.016
hurt that process compared to say

00:45:22.016 --> 00:45:26.816
a tablet display and all you got to do is scan up and down and now you see an

00:45:26.816 --> 00:45:30.816
uncluttered view when you look up and you see an uncluttered information when

00:45:30.816 --> 00:45:37.276
you look down is that them yeah this is a trade-off in design between scanning

00:45:37.276 --> 00:45:39.416
as a cost and clutter as a cost.

00:45:40.612 --> 00:45:43.712
Correct. So that's really the main sell.

00:45:43.932 --> 00:45:49.972
So in other words, the selling point of AR is largely that you don't have to

00:45:49.972 --> 00:45:56.192
scan because the pieces of information are superimposed within the same part of the field.

00:45:56.692 --> 00:46:02.212
And you're saving energy or effort from glancing away at things.

00:46:02.392 --> 00:46:07.392
And then I guess, therefore, working memory to some extent gets...

00:46:07.392 --> 00:46:12.032
Well, and the advantage of AR, and this is actually came out of the HUDs,

00:46:12.152 --> 00:46:16.752
is that in your brain, you're actually fusing the image on the glass with its

00:46:16.752 --> 00:46:18.172
counterpart in the real world.

00:46:19.152 --> 00:46:24.432
It's creating a three-dimensional fusing. So, yeah, that's, I mean,

00:46:24.472 --> 00:46:26.772
that's an advantage, definitely.

00:46:27.772 --> 00:46:32.112
And, but then also the advantage is that you can see text, no matter which way

00:46:32.112 --> 00:46:35.632
you're looking, or instruments, and so forth.

00:46:36.652 --> 00:46:43.632
The other advantage of really AR VR or mixed reality, virtual reality and augmented

00:46:43.632 --> 00:46:51.632
reality is with virtual reality, I can simulate a world 360 degree world in which I'm operating.

00:46:52.032 --> 00:47:01.132
And then I can also put a HMD image on that world that I'm using VR to simulate

00:47:01.132 --> 00:47:04.372
what we call the far domain of the world of operations beyond.

00:47:06.179 --> 00:47:10.579
I love this. Um, and then could you tell us a little bit about decision aids,

00:47:10.879 --> 00:47:16.499
um, in general, and then as they relate to the wildfire management, um,

00:47:16.859 --> 00:47:20.759
stuff that you work on and it happens to be extremely relevant right now,

00:47:21.019 --> 00:47:23.059
uh, given what's happening in LA.

00:47:23.719 --> 00:47:29.139
Yeah. Yeah. And this is, you know, I going way back in history again,

00:47:29.139 --> 00:47:32.979
I was inspired by Kahneman's work on attention and effort.

00:47:32.979 --> 00:47:39.979
I was also really inspired by their Kahneman Tversky's work on decision making,

00:47:40.359 --> 00:47:48.099
decision heuristics, where we fall short in decisions making because we simplify

00:47:48.099 --> 00:47:53.979
the way we make decisions and how occasionally that falling short can get us

00:47:53.979 --> 00:47:56.319
into hot water, big trouble.

00:47:57.219 --> 00:48:03.579
And I followed that stream all the way through my career, wrote quite a bit

00:48:03.579 --> 00:48:05.519
about it in some of the textbooks,

00:48:05.859 --> 00:48:10.459
fascinated by its role in medicine, poor medical decision-making,

00:48:10.679 --> 00:48:13.299
as well as poor pilot decision-making.

00:48:13.299 --> 00:48:22.159
And, again, as AI began itself to become more intelligent,

00:48:22.579 --> 00:48:33.659
there was more research on how the decision aid and the human can work in concert.

00:48:34.799 --> 00:48:40.679
A classic and tragic example recently was the crash over Washington,

00:48:40.919 --> 00:48:46.279
D.C., where there is a decision aid, and for some reasons, well,

00:48:46.379 --> 00:48:48.539
having to do with the reliability of the aid,

00:48:48.799 --> 00:48:53.019
it was not operating at that time in the cockpit,

00:48:53.339 --> 00:48:58.099
telling the pilot to climb, climb, or descend, descend to avoid the contact.

00:48:59.184 --> 00:49:02.124
The collision a very simple form of decision

00:49:02.124 --> 00:49:04.964
aid that you know pilots come to rely

00:49:04.964 --> 00:49:11.544
on as long as it's reliable the problem is at very low altitudes with lots of

00:49:11.544 --> 00:49:18.044
traffic the decision aid for collision avoidance becomes unreliable it's telling

00:49:18.044 --> 00:49:23.404
you to maneuver when you don't need to and we all know what happens to us,

00:49:23.604 --> 00:49:29.584
our trust in automation with unreliable automation, we turn it off or we don't

00:49:29.584 --> 00:49:31.824
pay attention to it. The boy who cried wolf.

00:49:32.244 --> 00:49:37.004
Okay. So that's a little detour, but it's an example of how the pilot and the

00:49:37.004 --> 00:49:41.024
collision avoidance system usually work very well in tandem.

00:49:41.544 --> 00:49:47.564
Certain things the collision avoidance system can see and understand the pilot

00:49:47.564 --> 00:49:49.864
can't and conversely the other way around.

00:49:50.484 --> 00:49:56.364
So I've become so interested. And then when the nature of the decisions made

00:49:56.364 --> 00:49:59.164
or the advisory made become more complex,

00:49:59.824 --> 00:50:06.224
it's more challenging for the human, the issues of understanding the best synergy between the two,

00:50:06.884 --> 00:50:12.164
how aggressive should the decision aid be, how much training should the user

00:50:12.164 --> 00:50:16.544
have of that decision aid, these become critical for understanding the fluency.

00:50:17.404 --> 00:50:23.544
So let me, I'll continue the train of thought because it leads directly into

00:50:23.544 --> 00:50:24.984
the firefighting project.

00:50:25.784 --> 00:50:33.424
And, uh, I've had a longstanding interest in wildfires, a little bit of a fascination

00:50:33.424 --> 00:50:38.324
by them, but also, you know, the real challenges of wildfire,

00:50:38.504 --> 00:50:40.324
uh, Suppression management.

00:50:40.644 --> 00:50:44.164
And we've had some big fires out here in Colorado.

00:50:44.624 --> 00:50:51.624
Nothing obviously compared to the LA fire or the Laguna fire in Hawaii a couple

00:50:51.624 --> 00:50:57.444
of years ago, you know, but one of the big characteristics of these fires is.

00:50:58.023 --> 00:51:02.403
Their unpredictable nature. There's a lot of forces. You look at the dynamics

00:51:02.403 --> 00:51:08.503
of fire modeling, and there's a lot of uncertainty there in terms of what a fire is going to do.

00:51:08.763 --> 00:51:12.603
Part of that is because fires generate their own winds.

00:51:12.863 --> 00:51:18.463
So you have fire-generated winds that can blow the fire in a direction that's

00:51:18.463 --> 00:51:21.183
opposite the prevailing winds, and so on and so forth.

00:51:22.583 --> 00:51:30.723
And so we reasoned. And the other characteristic of fires is any sort of automation

00:51:30.723 --> 00:51:34.963
has got to work fast because the automation is,

00:51:35.083 --> 00:51:38.623
if the automation is going to predict the trajectory of a fire,

00:51:38.843 --> 00:51:45.423
it needs to make that prediction with much advanced notice that the human has

00:51:45.423 --> 00:51:48.383
a chance to either evacuate or call forces,

00:51:48.623 --> 00:51:53.103
call resources before the fire has already reached a place again.

00:51:53.103 --> 00:51:57.443
And so we looked at some of the fire models that were out there,

00:51:57.483 --> 00:52:01.763
and they are pretty accurate at forecasting where a fire will go,

00:52:01.883 --> 00:52:04.863
but they have to take a half an hour of computer time to run,

00:52:05.083 --> 00:52:09.803
which is way too much time to make use of in the scene.

00:52:09.803 --> 00:52:13.823
And so I, with my student,

00:52:14.283 --> 00:52:20.723
Rob Spensley, who was very interested in it, and he had been a resident of Fort

00:52:20.723 --> 00:52:22.563
McMurray, Alberta, where they

00:52:22.563 --> 00:52:29.103
had a huge fire about 10 years ago that destroyed half of City of 100,000.

00:52:29.883 --> 00:52:36.683
Came interested in developing this, basically a display where you would show

00:52:36.683 --> 00:52:38.823
a prediction of where the fire is going to go.

00:52:38.823 --> 00:52:41.503
And people aren't very good at prediction.

00:52:42.203 --> 00:52:50.623
And as well as a decision aid that could make recommendations of whether a certain

00:52:50.623 --> 00:52:55.663
area needs to be evacuated or not within a certain period of time.

00:52:55.663 --> 00:52:58.663
So that's the project we're underneath in.

00:52:59.143 --> 00:53:06.203
It is an automated decision aid. It involves graphics of representing the fire.

00:53:06.823 --> 00:53:14.463
And so we're looking at how effective it is, but also, and this is where we

00:53:14.463 --> 00:53:18.683
get into this issue of automation reliability, what happens when a decision

00:53:18.683 --> 00:53:20.683
made of this sort makes a mistake?

00:53:21.383 --> 00:53:26.063
When it says the fire is going here, and in fact, the unpredictability means

00:53:26.063 --> 00:53:28.503
the fire is going north rather than east.

00:53:28.903 --> 00:53:35.463
How do people, how well do people, we like to say, process the raw data that's

00:53:35.463 --> 00:53:41.783
going into the decision aid to know when the decision aid is making a mistake?

00:53:43.224 --> 00:53:46.064
These are very important applications uh i'm sorry

00:53:46.064 --> 00:53:49.204
i think i think you're about to say uh mention something

00:53:49.204 --> 00:53:52.404
else nope um uh so we've

00:53:52.404 --> 00:53:55.844
got uh the aviation the wildfire um

00:53:55.844 --> 00:54:01.944
some of the dod use cases i know medical use cases have a lot of overlap uh

00:54:01.944 --> 00:54:06.744
would you like to mention some of the examples um for the audience of some of

00:54:06.744 --> 00:54:12.784
those you know where this stuff uh impacts um medical scenarios. Yeah.

00:54:13.384 --> 00:54:16.804
Well, they certainly impact medical. Oh, let me just throw in,

00:54:16.904 --> 00:54:20.964
done a lot of research on boats, looking at Navy collision avoidance tools,

00:54:21.204 --> 00:54:25.064
where the trajectory is a hurricane and a boat rather than a fire.

00:54:25.464 --> 00:54:29.284
But in the medical area, there's really two big issues.

00:54:29.584 --> 00:54:32.984
Any type of medical alert is based on automation.

00:54:33.244 --> 00:54:39.224
It's integrating information, alerting the nurse or the anesthesiologist that

00:54:39.224 --> 00:54:41.944
there's a condition that needs to be taking place immediately.

00:54:42.764 --> 00:54:48.784
How accurate is that alert? How frequently does the alert start getting false

00:54:48.784 --> 00:54:52.404
alerts and saying there's a problem when there's not?

00:54:52.824 --> 00:54:57.544
That issue is huge because there are so many alerts that are given that are

00:54:57.544 --> 00:55:04.344
essentially false alerts that people are beginning to distrust them and not pay attention to them.

00:55:04.684 --> 00:55:08.924
So it's pretty simple automation, but it's got huge consequences.

00:55:08.924 --> 00:55:15.464
In, for example, the intensive care unit, the neonatal intensive care unit in particular.

00:55:16.324 --> 00:55:22.164
But also the online surgery, particularly for the anesthesiologist who is getting

00:55:22.164 --> 00:55:24.704
alerts about the health and safety of the patient,

00:55:24.864 --> 00:55:31.984
and the ER as well when you've got critical patients arriving.

00:55:32.324 --> 00:55:38.164
So that's one big area that we're interested in. And what happens when people

00:55:38.164 --> 00:55:44.144
stop paying attention to, there's the word attention, to the alerts,

00:55:44.144 --> 00:55:46.844
whether they're visual or auditory, when they come on.

00:55:48.864 --> 00:55:54.264
So then the other, and that's been a pretty big focus of my interest,

00:55:54.444 --> 00:56:01.024
we're maybe starting some collaboration with University of Michigan Hospital to do some work there.

00:56:03.244 --> 00:56:08.744
But the other issue is just general, you know, AI programs in diagnosis.

00:56:10.024 --> 00:56:13.344
You know, again, AI, just as the physician is,

00:56:13.724 --> 00:56:18.404
is getting information from a lot of different sources, integrating it together

00:56:18.404 --> 00:56:23.844
with various AI algorithms to come up with an inference about how critical is

00:56:23.844 --> 00:56:28.204
the patient and what might be the patient's condition.

00:56:28.744 --> 00:56:33.564
Again, how well is that presented? How gracefully, fluently is it presented?

00:56:33.644 --> 00:56:34.664
How well is it displayed?

00:56:34.864 --> 00:56:36.784
And how accurate is it?

00:56:38.012 --> 00:56:43.332
And the real concern I always have is when designers or computer scientists

00:56:43.332 --> 00:56:49.112
become too optimistic about how powerful their algorithms are and say,

00:56:49.272 --> 00:56:53.272
we can give you this algorithm and look, it's 90% accurate.

00:56:53.852 --> 00:57:00.012
And I come back and say, how good is a human going to want to use a tool that

00:57:00.012 --> 00:57:04.012
is wrong 10% of the time in safety critical areas?

00:57:05.372 --> 00:57:08.332
Yeah, that's so, and that reminds me of some of the, uh

00:57:08.332 --> 00:57:11.772
just the work on bias and you

00:57:11.772 --> 00:57:16.252
even phrasing it as well it's 90 accurate well

00:57:16.252 --> 00:57:19.732
that means one out of 10 times it's also making mistakes that's

00:57:19.732 --> 00:57:22.912
called that's called framing how do you frame the probability

00:57:22.912 --> 00:57:25.792
that's right and accuracy it makes

00:57:25.792 --> 00:57:30.052
a big difference in people's choice yeah and

00:57:30.052 --> 00:57:32.872
so for folks out there who want to learn more of your insights of

00:57:32.872 --> 00:57:35.932
course there's uh the books that you've co-authored the

00:57:35.932 --> 00:57:39.152
ones i've got here um applied attention theory

00:57:39.152 --> 00:57:42.392
um and then the other one designing for people

00:57:42.392 --> 00:57:45.792
in introduction to human factors engineering but um

00:57:45.792 --> 00:57:49.292
in terms of talks that you give or folks want okay you've got it you've got

00:57:49.292 --> 00:57:52.372
another book you're showing here and this is the other one engineering psychology

00:57:52.372 --> 00:57:58.492
and human performance uh fifth edition um this is kind of the big textbook that

00:57:58.492 --> 00:58:04.552
integrates all of these aspects of cognition decision making motor control.

00:58:04.892 --> 00:58:06.712
Oh, that's kind of the major textbook.

00:58:07.072 --> 00:58:09.832
So I would throw that in the recommendation list too.

00:58:11.339 --> 00:58:15.139
Or would you like folks to, if they want to contact you, is it just your standard

00:58:15.139 --> 00:58:20.659
university email or I know you're not too heavy on LinkedIn or anything?

00:58:21.759 --> 00:58:30.959
Yeah, I mean, you can use, you know, Wiccans at colostate.edu or you can say

00:58:30.959 --> 00:58:35.999
the email that I'm here, pandawiccans94 at AOL.com.

00:58:35.999 --> 00:58:39.019
So that would be the you know two places

00:58:39.019 --> 00:58:42.699
that i'd be more than happy to to you

00:58:42.699 --> 00:58:45.419
know chat with yeah chat with folks who

00:58:45.419 --> 00:58:50.119
uh want to reach out to you um uh chris wickens it's been an absolute pleasure

00:58:50.119 --> 00:58:54.819
talking with you today uh thank you very much for being on the show well you're

00:58:54.819 --> 00:59:00.679
very welcome um thomas and uh you've been a great interviewer too so yeah appreciate

00:59:00.679 --> 00:59:04.319
it