great so if we could start with where
you were born
sure i was born in tehran iran and
what kinds of schools did you go to well
i was born in iran but to parents who
were traveling a whole lot back then so
they actually had been educated in
england and the us and they went back
and forth back and forth
so i went to pre-primary primary levels
of education in iran but by the time i
was in
second or third grade i was in missouri
of all places so i moved from a
metropolitan area in iran to actually a
rural family farm
area of about 300 people called platt
city which was near kansas city
and then i actually traveled in a car
back down to a private school
called pembroke country day in kansas
city missouri
and actually went there all the way
through high school
where did you do your undergraduate um
when i finished at pim day
i went straight to stanford university
and uh the reason is it was of all the
schools i applied to the only one that
accepted me so
it was not a hard decision to make and
so i went there for undergrad
during my undergrad experience there i
did a whole lot of work in solar race
cars in solar electric cars
and in computational molecular biology
work protein folding stuff
and then in robotics and then i just
stuck there for the masters in psg
so in fact the only school i ever went
to outside of missouri was stanford
all the way through postgraduate work
and you were an ee major
or i was a computer science major i
started as a comparative literature
major
and then did plenty of that and in doing
that i got
to fall in love with organic chemistry
of all things because of the geometric
nature of organic chemistry and
chirality
and so i got really interested in that
and then i got really interested in
evolutionary molecular computation and
how we could use
evolutionary algorithms to figure out
how proteins fold which is kind of
related to
the organic chemistry world and
sort of fell in with ai from there the
idea that you can use computers to solve
problems that are hard for people to
solve
without computers and so then i at that
point i assumed they'd become a
molecular chemist of some kind
using computer technology just as a tool
box
in my back pocket but then i met a
professor
from whom i took a really outstanding
class called agents that was michael j
nazareth
and he convinced me to start applying my
artificial intelligence work to
robots so then we actually went down the
path of
him buying a robot for me and me
starting to program it
and that's what ended up causing me to
stick around at stanford
in the community science department and
get my phd there what was the robot
so it's an interesting question um you
know back then there weren't very many
mobile robots on the market there was
only two companies around back then that
sold robots to professors universities
rwi which stood for real world
interfaces and in fact
the dna of rwi really ended up becoming
irobot
over time call it angle and such but it
was rwi back then
and it was nomadic technologies and i
used to visit nomadic technologies
because they were local so they were
right there in mountain view
in literally a garage just like the
story you hear about hewlett packard and
all those sort of startups that became
huge companies so literally in a garage
jim and john slater who were brothers
had started this company where they were
hand building robots from essentially
scratch using 68332 boards and 6811s and
such
all motorola processors and they were
hand wire wrapping these things so you'd
look at the bottom of the boards
and there were thousands of little wires
running across them this is before you
could do fabs and send things out for
layout
and so i talked to them and then mike
jones
the professor and i visited their garage
and finally bought the second robot they
ever made
so we bought this little robot it didn't
even have a it was
nomadic technology serial number two
that was all you could call it it's a
very short squat robot
and back then we put power books on it
so we put power bug 150s and 170s on it
too
macintosh powerbooks running common lisp
so all the robot programming i did in
fact for my phd was in common lisp
nearly all of it running on two
macintosh computers on top
on a piece of foam on top of a robot
platform
and that was the first row that we
bought fast forward a few years and we
ended up doing a lot of educational
robotics at stanford
and for that purpose we ended up
suggesting design changes to them as
they went into more of a mass production
mode
and together we came up with this idea
of the nomad 150
that jim and john then built which is a
very nice robot a very robust robot we
ended up buying i think four of those
and used them for education at stanford
for many many years
and then the fast forward one final time
when i got to
carnegie mellon we made a cheaper
version of that at nomadic technologies
that i bought six of for here and in
fact they're still being used here after
12 years
as the introduction world priming class
which is kind of
astounding that after this many years
the hardware works
and those are called nomad scouts and
they were
simpler smaller versions than the 150
because they didn't have an
independently rotating turret
the whole body of the robot rotates all
together
what year was that first robot that
first robot would be 1991
and our job with that first robot was we
wanted to figure out
how we can apply artificial intelligence
to the problem of robots
interacting with people and moving
around in the world so we did work with
that indoors
but the most exciting thing we were
trying to do was called the bookstore
project we wanted that robot in 92
to be able to go to the bookstore and
fetch a book for a professor and bring
it back
and so we started this thing called the
bookstore project back at stanford and
had the robot navigating the quad that
really beautiful arched area the inner
quadrangle of stanford
that's a repeating area it's easy to map
and so i was out there for six months
with a tape measure
we got tenth of an inch accurate mapping
of the quad it never occurred to us the
gold look at architectural drawings we
just measured it all and made it all
over again
students um
did you have an interest in robots prior
to this certainly um i was of the
generation
you know there's a there's a class of
people you'll meet that will say
what i'm about to say which is the
honest truth this is how it happened
i went to see her be the love bug
because it had just come out of the
theaters
and you go to the theater in kansas city
missouri and it's sold out because
everybody wants to see her be the love
bug
and the next theater in the same complex
is showing this new movie nobody knows
anything about called star wars
and so there's a generation of people
like me who went to see her be the love
bug
failed to get in the door cried and
their parents took him to star wars
because
what else we're gonna do go home and let
him cry and then you come out of star
wars dazed and confused going uh-huh
r2d2 i want i want that robot how do i
build that robot
so um part of my interest began indeed
with fiction
the other part of my interest began with
this idea that my parents really enabled
me to
build and take apart anything i wanted
at home so i typically would buy things
like a simon which is an old game that
you press the buttons on
and you could program it you could put
it in program mode where you decide what
sequence the buttons turn on in
well once i could program it i could
connect electric motors to it instead of
lights
program it and then the motors would
turn on in succession so i'd make little
simple robots
with assignment as its brain and i could
program it and then watch it move
so that was the way a non-programmer
back then who doesn't have access to a
trs-80 or something could actually build
something that's programmable
that moves and so at the same time
fiction inspired me
but taking apart toys and remaking them
into robots that also inspired me
i was also part of the generation that
had the type x sinclair
zx821 and and the
trs-80 so in fact i had a trs id and
then i got a timex sinclair zx81 and i
was writing little basic programs
of course they weren't connected to
physical machines but they do teach you
the idea the power of programming
so that timex which i still have at home
and the trs-80
and physically taking things apart and
the movies are probably the three
tripods of the stool that made me
kind of stand up and go oh i can do
robotics growing up however
everybody around me explained to me that
robotics was my hobby not my job
not my profession and that in fact
computer science and programming was
only a hobby and no more than that and
not worth more than that
so i really was disabused of the notion
that this is a job and it wasn't until
late in college
that it even occurred to me that i could
do computer science rather than compare
to literature organic chemistry
or physics as a profession as something
that you can actually make a career out
of
why was there such a vision of computer
science
well part of it has to do with the
immigrant community and the dynamics of
immigration
you want your children to be very
successful and usually that means that
your children should become doctors or
lawyers
because at an international scope those
are the two obviously
recognized professions so everything is
a hobby
except for those two and i was not from
a
household where there were professors in
my immediate family they were in my
mortise and family but not my immediate
family
so the idea that you can choose any
profession you want and then go into
academia in that particular subfield
that was not on the table right you're
going to be working a job
and so you're going to have to become a
medical doctor or a lawyer
so when you when you started working
with the robots
i'm curious why did you start with this
bookstore project i mean was there
some other type of application in mind
or
we the thing that i had started doing
with mike nazareth
that we'd gotten excited about was this
idea of planning with incomplete
information
what does it mean to have a computer
make decisions when the computer doesn't
have all the information and can you
make decisions in a way that's really
effective
even with limited information but really
fast
so the first thing we did was come up
with ways where we have
standard planning systems terminate
early and make decisions early
and once we had a way of doing that we
wanted to apply to a problem so i picked
a problem which was a oneplus rule it
was a video game
and we used the planner to play lumpus
in doing that we actually named the
program shallow thought
because this is when deep thought was
being designed and deep thought was
about
using a lot of computational power to
make decisions and this was the opposite
it was using
some cleverness in algorithms
and very shallow conceptualization of
the problem at hand to make a quick snap
decision
but making many many snap decisions over
time and trying to show that the overall
behavior ends up being almost as good as
making deep decisions once in a while so
shallow thought was this video game
thing at first but we wanted a real
problem
that has incomplete information and
neither of us got
just how hard robotics was we thought if
we buy a robot it could move around no
problem
but we did get that once the robot's
moving around its wheels accumulate
error as it spins
and so we did get that oh there'll be
this constant
rush of incomplete information coming
into the robot it had sonar sensors and
that's all back then
no no vision cameras nothing like that
so
to us robots were simply at first a
practical application of the planning
system
it wasn't until we got deeper and deeper
into using the algorithms on the robot
that we started realizing
my goodness robots stink they don't just
they break all the time the electronics
breaks the motors break everything goes
wrong with it
they're incredibly incomplete
information and then people interact
with them funny you know you have to
roll on the quad moving around
and a guy came up with cowboy boots and
started kicking at one time
and it then it occurred to us oh so
there's interaction that we didn't
expect people will come and kick you
this never occurred to us and our planet
wasn't designed to incorporate or digest
the conceptualization that people
might come and kick me and so that's
when we became more excited about this
idea that robots are so
complex and such intricate machines in
how they are woven into our world
in a social level that they are
themselves
a really interesting focus of artificial
intelligence research
so after the the bookstore what did that
lead to so there was a guy named ben
dugan
and i who did the bookstore project
together we were equals and worked hard
on that and he ended up going to uw
and in fact you should talk to him he's
a fantastic fellow
benjamin dugan d-u-g-a-n once
ben dugan and i had worked on the
bookstore project we had big demos we
had
in fact jean-claude latom and barbara
hayes roth and john mccarthy and all
these people come see the robot moving
around
in the quad then
this was exactly when people started
having contests around robotics
in fact triple a i just started a
contest series
and so we went to the very first contest
i believe that was the one in san jose
and from then on we started more
actively being part of that contest
circuit
and at first working on the contests by
submitting an entry
and then later i started being one of
the judges designing the contest and
running the contest
and that's the only time when i started
actually
weaving into the raj community up until
that time i really wasn't aware of what
was happening across the country i was
doing planning research so i understood
the planning problems really well around
the country but not robotics itself
that was a mere application area for me
in fact that first contest
dave miller was there and i don't think
i think
i don't remember where he was he was on
the east coast somewhere at that point
but dave miller had this fabulous robot
called scarecrow
and he saw this interesting collision
right there in that first contest
because
there were poles with barcodes on them
and the idea was you were supposed to
visit them in a certain order and have a
light light up on your robot or
something so we knew that you visited
them in the right order
and that you recognize these poles and
there was an entry i think it was kurt
cottledge from sri who had a very
complex entry
and there was dave miller who came in
with this circuit-based robot that
didn't even have
any programmable matter in it it was
just a circuit it had a barcode reader
which was a bunch of bristles of metal
bristles and it was
jumping around having a little green
light bulb light up as it randomly moved
around the space
and it won and it was this sudden
collision between
hardcore conventional ai we try and
solve the problem by having cognitive
cognition and planning and thinking and
problem solving
and clever engineering where the
engineering demonstrates that in fact
at the behavioral level we can achieve
such good results
with very little explicit reasoning or
decision making
and that battle of course only grew in
time as broad brook started publishing
papers
about the idea of using the world
instead of internal representation
and then people shot back and in fact
that was where i saw the seeds of that
debate start
how do you see that debate having been
resolved
well that debate will stay unresolved
probably for all time
the problem is that fundamentally
each group is able to point at successes
and failures in themselves and in the
other group respectively
so people who try and design robots
bottom up and use
circuits and use simple kind of
progressive techniques
are always able to say look it's much
much cheaper and simpler to build robots
this way
and people who do high level cognitive
work can always pick problems in which
they can make a plan or a decision maker
do really well
and then they can point it out and go
look for this problem our method rocks
the only partial resolution i've seen is
this thing that in fact david kortenkamp
really pushed hard
in his time and i'd say he was a major
proponent of it
which was the idea of what he called
tiered architectures or 3t
the idea that in fact perhaps a robot
should have at the same time
a low level competence that is fast and
reactive and witty to its surroundings
and a higher level competence running in
parallel that's
reflective and thoughtful and has a
little more ai and planning and
reasoning going on
courtncamp was probably the first person
that really nailed that down
and what he did on that really stemmed
from something jim furby had done
earlier
jim furby had designed a whole
simulation system that many of us used
back then
and then corton camp used that
simulationism to demonstrate a tiered
architecture
and so at stanford there were other
groups that were also doing robotics
what were your contacts with them or
because i was a student it was really
easy to cross the walls and talk to many
groups
but it was always odd for me because the
groups didn't talk to each other and in
fact i'll describe to each group to you
in detail but it was funny because
i even had this big meeting where i
tried to get all the group leaders in
one room at one time and it worked they
were all in one room at one time and
then they didn't do that again
but there was a number of groups doing
robotics and in some cases using even
the same hardware
we had barbara hayes roth who had robots
and she was working unbelievable robot
personality
we had jean-claude latone who was doing
landmark based navigation at that point
and he had robots with cameras on them
pointing up at the ceiling
and they had special types of barcodes
on the ceilings and they were moving
around
trying to track those barcodes to tell
where the robot was
so we had jean-claude laton barbara
hayes rather we had kurt connelly at sri
next door virtually next door he was
doing his own robotics work on
navigation
and then we had mike nazareth and i the
least robotics
of the crowd we were least robot savvy
but we were deep in the world of
planning and reasoning
and navigation using planners and making
decisions of that nature
so why do you think they didn't didn't
meet up more often
well there's a fundamental trope you
will find across universities
which is that universities love to talk
and tout the fact that there's great
collaboration
but by and large it's easier for
university professors to collaborate
with people at other universities
because in doing so they don't threaten
the description of credit
within their own universities of their
work and because of the
incremental nature of decision making
around tenure and around
promotions everybody wants to carefully
protect the idea that they can
gradually further a piece of research
every year and show modest
gradual increases in research stature
over time
so because of that dynamic socially
people tend to work better with people
from berkeley than from their own
you know university stanford of course
there's exceptions over time both with
people
and for twilights of time when an epic
pass is during which people can
cooperate with one another for four or
five years but these pass
and so in your in your own work when you
were doing the
bookstore were you uh a master's student
at that point or a phd student
i got my message as part of the phd
program okay so the robot was called
dervish the project was the bookstore
project and we infected all that work
while i was a phd student except that we
started it all about a year and a half
before i became a phd student at all so
i was an undergrad in fact i got to
graduation decided okay i'm done i'm
graduating i better go find a job i
interviewed at bell labs i decided i'm
going to go there and mike said you're
not going you're going to stay here and
get a phd
and i said but i can get a job now and
he said no you're staying
and of course he did a very good thing
because then the labs essentially
uh collapsed because of the economic
recession
and so what other projects did you do
during your your phd
during my phd i at the early part of my
phd i finished up the computational
molecular biology stuff i was doing
where we were matching dna sequences
after working on the planner
the shallow thought style planner and
then doing the dervish robot
we entered a number of contests where we
had navigation systems on robots
and that became probably a common thing
that i would work on
we used assumptive programming and a
couple of different navigation
contests at triple ai
everything that i did was around
planning with the information in fact
that ended up being the
title of my thesis it was really this
question of should we make assumptions
about the world
and then operate under those assumptions
until they're violated and then if
they're violated we change our
assumptions so
how do you choose the right assumptions
so that they can be violated and you'll
find out
instead of getting killed before you
find out they're violated
and then how do you choose new
assumptions that are consistent with all
the observations you've seen
that assumptive direction and then what
we call property-based
planning where we didn't think about the
states of the world anymore we thought
about broad
overarching properties that encumbered
many many states of the world
and so we partitioned the world into
sets of possible properties that are
true or false
that's the finishing work that i did
during the the graduate program
i was also juggling back then i was um
absurd as it may sound getting a pilot's
license
and then an instrument flying license
for flying in it under instrument
conditions and a helicopter
pilot's license rotorcraft license so i
was actually
doing planning and execution work on
robots
and then going and studying and working
on flying
did those two ever get together
they never quite got together no uh no
in fact there was actually the third
thing in fact my time was more or less
split
split in three i started taking uh very
seriously a classical guitar
and in particular some spanish and south
americans douglas guitar
and so that's in fact i started all
three of those pretty much with the
first year of my doctoral program and i
continued all of them up until now
except that for the last four five years
since i had because i really haven't
flown at all
and so what were who were the other
important people that you worked with
during your phd
i was a number of professors that i grew
really fond of and really close to
one of them was john mccarthy i ended up
having countless
chats with him i worked on a lego robot
with he and his son
and i just really enjoyed his mentality
i enjoyed having conversations with him
about robots and vision
he and i were both of the opinion that
computer vision had to get much better
before we could really make robots work
and i think i still believe that so i'm
still waiting
but john mccarthy ended up being
somebody that i really got close to
i had many conversations also and had as
a supervisor
junk little tom and
i'm thinking of the third person carlo
tomasi who's also one of my co-advisors
carlo was a vision guy and we did this
odd project where david andre and i
did something with carlo tomasi back
then it was fun it had to do with this
concept called depth from defocus or
depth from focus
we took three cameras and put them on
one of my robots and put it in the same
direction but
adjusted the focusing positions to be
slightly different so that each one had
a different focal plane
and then by comparing the imagery and
the three you could start to categorize
how far away and how close things were
in the scene so we literally had three
frame grabbers
in fact no we had one frame grabber we
synced them and had an rgb
one was r1 with gene one was b so we
turned them into black and white cameras
on the rgb
lines of a single frame grabber but this
is really fascinating
because you could at light speed very
very quickly move around the world
detect obstacles detect staircases
detect bushes and we even had
demonstrations where we put it out and
invite faculty to bring their kids
and give the kids treats and have them
try and get the thing to go hit
something
and have the kids run around with the
robot that's part of what got me
interested in educational robotics
because working on this idea of depth
from defocus
yielded a robot that people could
interact with safely well when the
robot's safe then you suddenly
realize the children playing with it are
adapting to the robot they're
finding the lowest common denominator
and playing
the adults playing with it are trying to
figure out how to make the robot break
they're trying to demonstrate that
they're better than the robot and the
robot is
is incompetent at some level of
performance
and the children in fact as soon as they
discover some way that otherwise
incompetent
cleverly and quickly learn how to avoid
that particular stage of interaction
so it doesn't do that and so it doesn't
actually and so you can actually
interact with them in a more
sophisticated way
that was fascinating right and that
really got me interested in this
question of educational technology and
what does it mean to have children use
technology and how
genuine and authentic are they with
technology as compared to how
adults are and so
after stanford what happened next
well when i was at stanford doing
everything i said i also started a
company
called blue pumpkin software blue
pumpkin was an interesting company
because
there was a particular regime where my
friend of matan and his friend duran
aspies
had suggested that planning with
incomplete information is really useful
and it was a particular tech place
called scheduling and the big challenge
is
uh in particular any situation where you
have a ton of people
who need to have some agreed upon
schedule together but they have
preferences and requirements and i want
to take a holiday and i want to work
three days and be off for five days
etc so you have all these crazy
constraints coming into a system
how is a system to take many many people
and come up with a schedule
that actually maximizes everybody's
happiness you know
achieves as many of the goals and
desires and preferences that they have
and maybe there's some hard rules too
like union rules that say you can't work
more than 30 hours in the hospital you
have to go sleep after 30 hours of work
so we were really interested in this
direction and before there was even a
company there was this problem
of scheduling uh prospective phd
students to meet with new faculty at
stanford
and so in fact mike jones and i made an
interface that was used for years and
years after i left in fact i used to fly
back to stanford and run the interface
for them which was
absurd to go back from carnegie mellon
to do that it was still running in
common lisp so we had to go back and
deal with it and list on an old computer
but we had this really nice system where
all the faculty would put in the
students they want to meet with
all the prospective students coming in
for the open house would list the
faculty they want to meet with
and then we would find a schedule that
maximized the number of meetings that
were desired
and that was really fun and it was
really application of the same kind of
cleverness
in making quick decisions building a
partial schedule and then undoing some
of the decisions that were
causing problems and redoing some of
them kind of like you might with post-it
notes swap things around
trade a couple people and get a better
schedule so we had quite a nice system
for that
the company happened because of matan
who was a phd student and my office mate
advised by the same person mike nazareth
he and his friend decided let's do this
for hospitals
so we made the same system for the call
schedule of the radiology department at
stanford university and in fact they
used that for many years
i think for nearly a decade so all the
radiologists put in their preferences
they have different subspecialties you
have rules about which sub-specialties
have to be available how much of the
time what hours of the day and night
and the system comes up with two week
schedules two weeks at a time so
everybody knows what to do
this got so extreme that at one point i
think they had new macintoshes running
simulations of old windows systems
so they could use our old software on
their new computers
to do the schedule because that was
everybody dependent on this as the
schedule
now the things to concern for the more
commercial when it became apparent that
call centers have the same problem
call centers are these places where
hundreds sometimes thousands of
employees are answering the phone
united airlines may help you book your
flight today
it's the same thing they have cigarette
breaks they have everything you can
imagine
impeding upon their time infringing upon
that
and then on top of that you have skills
you have people who speak spanish other
people who speak english some people
know windows some people are mac
you know a macintosh operating system so
you have
a huge morass of skills and requirements
and preferences
and then you have varying call loads
over time the day before nov before
thanksgiving
huge call volumes you need lots of
agents uh but
you know the day after christmas very
low call volume you need very few agents
and so you want to have just enough
agents so people don't have to be on
hold too long
but if they're not on hold at all you
have 20 agents
we want people on hold two minutes let's
say but not 20 minutes because then
you'll lose business
so it's this massive optimization
scheduling problem and so the last two
years that i was a doctoral student
we created software on that for call
centers
started a company got some angel funding
and
in fact that became quite large and as i
graduated from the doctoral program
i was the chief scientist of that
company but then i got the offer from
carnegie mellon
right as i graduated and again this is
the only offer i got so just as with
graduate school
and and with undergraduate in fact
graduate school staff was the only place
i got into it was the only place i
applied
undergrad only place i got into except
berkeley and and then faculty position
the only place i got an offer from so
i came to carnegie mellon and then flew
every week back and forth to blue
pumpkin
in mountain view for i think two and a
half years
and then eventually we sold the company
to a much bigger company and then i was
able to spend more time
trying to email them and so
how did you start at carnegie mellon who
was around
when i started here that was 1996 i
think january of 97 in fact
i graduated in 96 and we went straight
to smith hall
and back then the old guard of robotics
here was fully enforced the automated
highway system project was running
takeo canada was director of robotics
chuck thorpe was running the ahs system
and i started immediately working on
that project
and matt mason was here he was one of my
mentors early on mike erdman was also an
early mentor
and those were the gangs that i really
hung out with my neighbor
next door to me was andrew moore so my
office was right next to his which was
great because when we moved to this
building i was also neighbors with
andrew until
he left for google and so those were the
people
and i was a young assistant professor in
robotics and i was a bit of an
experiment
almost everybody had been hired not
almost everybody
that was the robotics institute was that
was either research faculty
or they were co-appointed across
multiple departments
because robotics was a department but it
was a relatively young department
it didn't have any tenure-track faculty
that were solo that were
only exclusively in robotics that didn't
exist yet
so that was really interesting and they
hired me as a sort of experiment because
i was
entirely in robotics i didn't teach any
computer science classes
i didn't have any computer science
responsibilities i wasn't even courtesy
appointed in any other departments
and yet i was tenure track which was
really unusual everybody else that was
purely in robotics
was research track and so that that was
97
and of course over the years we've had
more and more faculty enter and the
department has really swelled and grown
well and has far more funding even than
it had in early years
how many people were there then both
faculty and students around
we'd have to ask suzanne i don't know
okay i don't know probably no more than
two-thirds of what there are now
probably even less than that
especially if you count nrec nrec i
don't think it has even started yet
so in that sense we were probably at
most half the size we are now
so what kinds of you mentioned the hhs
project or something thank you
the automated highway system right yes
for ahs
right yeah ahs so what other
what were some of the issues that you
dealt with there what were some of the
other
projects that you decided to work on ahs
was a really straightforward application
of the work that i had been doing at
stanford with a student
fantastic fellow now founder of a
company here in the area called
paraguatavia
the interesting question there was could
you have cars
making good quick decisions about
whether to change lanes or not and
what's safe to do if they're autonomous
cars
by using some simple sensing around them
without going through a huge amount of
planning
and in fact we ended up making something
called a universal plan where we
actually
programmed all the possible plans we
would need and were able to select the
right plan and execute it very quickly
and that was exciting because then on
the simulator we could show dozens of
cars
and show autonomous cars with very
little computational effort making
appropriate decisions
changing lanes without being dangerous
and that was very early work that was
97-98
the first big project that i did at st
at carnegie mellon university was this
museum project we had
had this interesting situation so
sebastian thrun had just done a robot at
the university of uh with university of
bonn
in the museum called rhino and that had
gone really well
in a small museum area and
from what from the phone call i got
basically somebody called me from
the field robotics center and said red
whitaker has this project with the
director of the museum of natural
history they want a robot
and we tried reed simmons and he's too
busy he can't do it we tried sebastian
and he's too busy he can't do it
we tried matt and he can't do it right
now either so you're our fourth choice
would you be willing to do it
so i was like oh uh maybe let's let's
talk and so i met with red whitaker and
with jay apt who was then director of
the carnage university of natural
history
their idea was let's make a robot that
just tours of dinosaur hall
which is interesting because dinosaur
hall is is full of paleontological
specimens that are very old
and i love the irony juxtaposition of a
new robot giving you tours of
paleontological data
so that was cool it was an interesting
idea and it was one of the few
examples i've seen that's compelling of
a robot
doing something useful in the human
social sphere because
in a physical place like a museum you
actually want to take people to the
exhibits and show it to them
whereas there's many cases where people
have mobile robots like delivering faxes
where you can say well we don't need to
deliver faxes we can email them to you
or robots delivering coffee where we say
well you don't need to deliver coffee
it's better for you to get up and walk
over and get coffee because
take a break from work but in a museum
it almost makes sense
for a robot to provide multimedia
exhibits
that increase your awareness of the
physical exhibits around you in the
dinosaur hall
so we did we took on that project in my
lab and got some outstanding swiss
students to help with it and
that really worked out well for us it
turned out that at nearly the same point
in time
in fact sebastian did do a museum tour
guide as well
which was minerva which is the
smithsonian
american museum of national history and
that was neat
because suddenly you had the paper here
publishing stories about the museum
director here the museum director in the
smithsonian the robot here the robot
there kind of the tale of two robots a
tale of two cities i tailed up to
museums
and so that became interesting for us
that robot
project was about human robot
interaction so we gamed away
all the problems of having robots be
robust we put big pink markers in the
room
used a vision camera a la jean claude
latum's memory based planning system
so the robot couldn't get lost it's hard
to confuse anything with a big big pink
square on the wall
so the robot couldn't get lost it
plugged itself in and plugged itself out
of the
wall and we really wanted a robot that
could run for years and years without
help
the reason we did the project that way
was that niels nielsen at stanford who
was also an advisor to me
had published a paper talking about what
he called i think the robot challenge
and the idea was can you make a robot
that in fact
um and i could probably dig this up
somewhere in my email but you can get it
straight from him
can we make a robot called the robot
factory
i think it was called the robot fact
totem paper he said we need
a new grand challenge of robotics can we
make a robot that
runs for a full year with its original
programming without humans making
changes to the program
and does something in our world and so
we thought well this is the perfect
robot factoid example it's a
constrained space we can put pink marks
on the ground on the floor
on the ceiling i mean excuse me and
indeed we can plug it in and plug it out
by itself so we literally hooked it up
with an analog pager system
we all wore pagers for five years that's
five
years okay it would pay just if it
needed help
and it got to the point where we
wouldn't go there unless a page does so
it had a level of autonomy where
it only asked for help and we would help
it we no longer had the situation of
going there because it stopped
communicating and you can imagine the
number of
just technical hurdles you have to
conquer to have something be that a bit
that sort of industrially available
we also did surveys of the students and
children and adults in the area
about dinosaurs to see if the robot
caused educational impact in terms of
their understanding learning or
excitement about dinosaurs in general
and we got really good results and so
that was that taught me two things that
that huge project
five year project one was that in fact
robots can be made reliable enough to be
in the physical social sphere
second thing was that and by the way
they don't crush things they don't smash
babies or anything they can make them
safe enough
the second thing it taught me was that
we can quantitatively evaluate the
educational impact of the robot
and wow if we can quantitatively
evaluate then we have a feedback
mechanism now we can go and try and
refine
how we make robots to make them as
educationally impactful and positive as
possible
and so that's how i got into this whole
area of social robotics and
the you know formative and summative
evaluation of a robot in society
can you tell us a little bit about the
robot itself and how you how you
designed it or where you got it and
also exactly what it was doing in the
museum so the robot itself
was something that we cooked up with
nomadic technologies my old friends jim
and john slater and their company
we wanted a robot that was holonomic
they could go sideways because people
are effortlessly able to go down a hall
backwards forwards go around people by
sidestepping them
and so we felt that aesthetically that
was important for the motion profile of
the robot to be
appropriate to the level of cognition
and respect we wanted people to give it
in the museum
so they built some new robots right then
called the xr 4000 which were
astoundingly cool robots they were
huge they had 24 sonar ring on the
bottom another 24 on top
two 24 ir rings the entire
skin of the robot was touch sensitive so
you could tell if somebody was pushing
on it it was an
amazing robot and we worked with red
zone robotics locally to build a shell
on top of the robot
a head and torso if you will
so that was the robot itself and we
delivered one from them for something
like 35 thousand dollars which was a lot
of money for robot back then
and it was holonomic and fantastic and
they were very helpful
what the robot did was we put a
laserdisc player in it so this is before
we had digital
encodings outside of laserdisc domains
so we had a local company called magic
lantern create educational
shows about various aspects of
paleontology from what do
the scientists do to why there's
disagreements about how the bones go
together
to visualizations of how the dinosaurs
lived to what the extension of
extinction event theories are for that
dinosaurs so everything you can imagine
was packaged up into two and a half
minute presentations on the dvd
and then we had a computer we had dvd
player the computer would trigger the
dvd player
i'm sorry the laserdisc player the
laserdisc player would seek and play
on a little television screen inside the
robot's head and so
people would follow the robot around and
then it would take them to exhibits that
they don't usually visit
like the aquatic marine mammal exhibit
on the side of the building because
people would look at the tyrannosaurus
rex and go wow and they'd leave the room
the whole idea was take them from that
experience of excitement and enthusiasm
and pulled it over to tell them about
marine dinosaurs
so then they realize there's much more
in the room to look at than just the big
t-rex bone
and so we were taken to the side show
them videos and then even
in the future years in the out years we
had quizzes that would come up they'd
answer some questions
and that increased audience engagement
tremendously so we even ended up with a
touch screen so people could
answer questions on the screen by
pushing on the screen bars
and that was the whole idea was giving
them a tour with multimedia experience
from the laserdisc player
how did you design the initial
interaction and did you change it
with timer we changed it hugely over
time we made tons of changes in fact the
whole system was this feedback cycle
so in a way we failed niels nielsen's
test because we kept changing the
software to make it better
we weren't trying to keep it completely
frozen in static the way we designed it
was we came up with an architecture
for how we could modulate the voice of
the robot and
how we could have some affection in the
robot and
parameterize the affection over time
based on the interaction it had so he
could be happy sad tired
excited so we would change the valence
of its effect
and then have that impact its voice then
we designed an architecture for how it
could dialogue with people what
could it say what kind of pauses could
it have what videos could it cue
and then we presented all this to the
education division of the museum of
natural history
and said you guys are educators you
train docents
so now you decide what the script is for
the robot
what do you want to choreograph what do
you want to say when it's tired and
somebody's interested in t-rex's
what do you want to say when it's
excited and somebody asks about t-rexes
so they create the content including the
formative content of how it behaves
and then with them we went to magic
lantern and actually cut the videos
onto the laserdisc and then we
programmed it all into the robot
and then using the education division
again we tracked people did
you know micro uh what do you call it we
did
microgenetic social studies of the
people interacting with each other with
a robot
did some ethnographic work did some
quantitative evaluations and then we'd
go back and change it all again
and so really it was this interesting
collaboration where the media
technologies made the hardware work
we were just the programming glue in a
way but then you had this whole
education division of a museum of
natural history dedicated to
tracking and updating the content as
appropriate
and why did you what why did it end in
the end of five years
j app left the museum and when jh left
the museum they
had raised the capital funding to do a
major renovation of dinosaur hall
so they closed dinosaur hall and took
out the robot and in the new version of
denisov hall there was no room for that
robot
and it was dated by them i mean it was a
long time for a
research robot to be installed in the
museum
so what was what was next or what other
did you do some other project at the
same time as the
oh yes we did tons of projects
if the tour
robot in dinosaur hall taught us the
potential for education
it also taught us the expense of taking
on a really big project like that
so we started trying to do projects
where we engaged with the public
with less expensive robots but in much
more massive ways outside of pittsburgh
and so we did many projects back then we
started something called the palm pilot
robot kit
which is just a simple recipe online for
how to convert a palm pilot which was
all the rage back then
into a robot into an actual moving robot
on your table
and that was interesting because we
could see you know this is before make
magazine but we could right away see
that millions of people and back then
that was a lot are going to our website
downloading the recipe getting going to
home depot getting parts and making
their own palm palette robot and to us
that was super exciting
it also started to teach us about the
idea of licensing because companies
would come along and say
we know you haven't patented this thing
but we want a relationship with you can
we build the palm pilot rocket for you
and we'd say sure go ahead and and would
you pay some royalties so we can get
research funding and they'd say yes
even though we had no patent no ip no
protection of any kind in the business
sense
and so we learned that the crowd was
excited about the idea of getting their
hands
dirty with robots we learned that
companies are excited about marketing
them for us
and we learned we don't have to try and
go down patent or protection paths
instead we can just keep things open
open source it and still have relations
with companies that generate funding for
us
so it taught us a huge number of lessons
that turned out to be critical
we went on to do cmucam which was a
small vision system
backed with a processor that made it
really easy for artists and hobbyists to
put vision on their robot
because you could ask it things like
just tell me where the orange ball is
and it would just tell you where the
orange ball is so you don't have to be a
computer vision researcher
to have a camera just like you don't
have to be an acoustics expert to use
sonar
so it made it that simple in a semantic
level and again we licensed that to
three or four companies and they started
selling thousands of these little
cameras
without us having to ever bother with
patents and
copyrights and any of that nonsense so
again we're able to
openly share the design create company
empowerment so that companies could
actually sell it to people and create
empowerment in people because suddenly
they can make different kinds of robots
from what they could make before
that became the recurring theme of the
lab as we took on more and more projects
where we were essentially in the
business of empowering people by
innovating some new technology
and then designing a way to deploy it
out into the world
and so i know you've done a lot of other
educational projects and also
some other museum projects could you
tell us a little bit about
sure there's too much to tell you
but i'll give you some examples it's
hard to pick
i mean it's really hard but i'll give
you some examples um the music natural
history had the
entomological specimen collection of
terrariums with bugs in them
and they were bothered by the fact that
kids would pass by these terraria
and not pay much attention to the bugs
they would just walk right by
and so they asked us is there something
we can do with technology that changes
the relationship the underlying
relationship between the visitor of the
museum
and the madagascan hissing roach and the
scorpion and snake and the tarantula
so he came up with this idea of a
robotic terrarium and we call it insect
telepresence
this is before telepresence was the rage
the idea that we had was well the
presence is going to take off people are
going to be
you know logging into a robot to go to a
meeting but what if you could log into a
robot to change your scale
what if by being in a robot you were
able to interact with a bug
at its scale level so it in other words
could a robot make you feel
like you're the same size as a bug or
that a bug is the same size as you
and if that was the case could you
interact as a peer with a bug
and if you were interacting with a bug
like with a pure relationship would that
change the
degree to which you paid attention to
its microstructure to its microbehavior
so we built a little raster arm with a
panasonic remote camera head back then
the
amazing thing that had just come on the
market was an analog camera from
panasonic
it was smaller than this pens cap the
camera
and it was a little wire and it went to
a special control box so it was this
tiny camera
that was full color very very very
highly accurate
camera so he put this on the end of a
raster arm
clutched it so you couldn't squish the
bugs and designed the whole thing into a
terrarium
where you could put madagascar hissing
roaches or other bugs and then in the
corner of the room we designed a kiosk
where you have a three degree freedom
joystick and a huge tube television this
is before lcd panels
it's a huge tube television on which you
could see the bug
blowing up to be your size and we put a
stool there just the right
distance away so you'd fall into this
experience with the joystick
and we made the terrarium completely
clear and positioned it so that you
could
you could sit down and drive inside the
terrarium where you could stand up and
look at the robot
interacting with the bug and see the
person and the television screen
so you had two different views and
displays one
was the telepresence display and one was
essentially an art piece
that showed a human a robot and a bug
all interacting with one another
in a very unusual unconventional way and
that really worked beautifully in
measuring time on task we saw people
jumping from seconds walking past the
display
to two to three minutes spent on the
display until the parents say really we
have to go ricky come on get up we have
to go enough with this let's go to the
next exhibit
and so we saw this wonderful experience
of people interacting with it in
different ways
looking at the bugs looking at how the
bugs eat and especially looking at the
mandibles on the
madagascan roaches and how they would
actually ingest the pear and cut it
apart
so we saw that kind of peculiar
attention to natural detail
forged by a relationship that you have
to a robot that disappears
the relationship with the rule is not
explicit and in fact it becomes per
background
and that also set an ethical stage for
our research this idea of
using robots as a tool for causing
a bridging of the gap between people and
the natural world
where we have the robot somehow
disappear into the woodwork and have
what is surfaced to be a relationship
between person and nature
odd to hear because ordinarily the way
you've achieved a relationship between
humans and nature is you go outside you
go walk
you go use your senses so the challenge
for me was as a
as a innovator or engineer is there
something we can do with technology
that in fact can amplify that feeling we
get in nature
and is useful rather than deleterious
the fact that we want people to spend
more time outside and less time with
technology
and what kinds of technical challenges
or innovations
came up with once you started working
very much more closely with people and
hearing about interaction
part of the challenges that we started
facing were that neither human factors
nor human computer interaction had
really figured out
the aesthetics and the social design of
robots physical models
had done it for the flat screen of the
computer but it doesn't apply directly
to robots
that are social inhabitants of our space
human factors had worked out for complex
machinery like a boeing 747
but that doesn't work for untrained
humans interacting with an object for
the first time that's really quite
complex
like a 747 so one part of the challenge
is that we saw the need for
multidisciplinary design teams to work
together to be able to yield
a product that had true value and to
that end we had to put together a lab
that had this absurd collection of
skills and talents we had
to put psychologists curriculum
designers in the same room as a graphic
designer
a material science engineer a mechanical
engineer electrical engineer
and we have all of that in one lab at
one time and of course we had to have
firmware coders and hackers and software
enthusiasts and ai experts
so that was the biggest challenge was
how do you get all these people in the
room at the same time and then talking
the same language well enough
that they can in a cross-disciplinary
sense build a robot
they gaze on all of their disciplines
equally
rather than being this rousing
demonstration of some one thing
like look it can navigate really well
and we're going to ignore everything
from aesthetics to human form to factor
to long-term educational impact
how do you do all of it that was one big
problem that we faced the other big
problem we had was sustainability
we didn't want to do projects in which
you demonstrate a robotic artifact
for your funder and then put it on the
table or the shelf and then move on to
your next project where you demonstrate
a new artifact to a new founder
that's a common theme in many of the
engineering and science disciplines
where you're essentially
driven by the need to do a good demo
because the demo brings in the money
and then the money brings in a new
project and you forget about the old
project
our question was fundamentally if we're
trying to measure social impact
it has to be long term so how do we
create a funding
architecture in a lab that can
sociologically support
people taking risks doing work that
isn't milestone or deadline driven
and then rolling things out that are
affordable and so the tack we took was
no defense funding no contracts only
gift funding
and so that was really difficult because
i had to go out and raise a whole lot of
gift money from foundations from
corporations
that ordinarily want some kind of
contractual arrangement explaining to
them
i don't want your contract i'll take
your gift money but don't worry i'll
open source it so you can use it
but then anybody can use it and that was
really the challenge
of running the thing put the skills
together in terms of people and then
get the money to be clean money not
encumbered by milestones
driven by the defense department so they
have clean money
interesting group of people and now you
can take on new high-risk projects
without
anybody telling you what to do on the
outside and that's where we've gotten to
over the years that's really exciting
and different
so what were the challenges of
convincing the companies to do that kind
of gifting and
what were the arguments you made that
changed their mind it's interesting
because the ch the companies
everybody assumed always wanted
first-rate reviews for the licenses
so the supposition by the
technology folks and the legal folks at
carnegie mellon from day one was
the companies want license agreements
that's why they fund us
they want to use it and own it or at
least have exclusive rights to it so
they can have a
competitive advantage with it but we'd
meet with them and literally i remember
with microsoft we met with them one time
and they said we'd like to first
write a refusal on a license to your
educational analytics technology that
we're going to fund
and our lawyers were in the room and
their lawyers when they are in the room
and i said actually we don't do that in
my lab
but tell you what we will do we'll make
it a gift you guys give us an
unregistration gift we'll get to use all
the money as we see fit for educational
robotics
and i promise to open source it all so
you can use it anyway
and i could see my lawyers feeling like
oh no now we're going to have this
debate
about ownership intellectual property
and the microsoft folks said oh yeah
that's fine we just want to make sure we
can use it
and we all nearly fell off our chairs it
was this realization that the companies
aren't trying to establish competitive
advantage with universities
they're trying to further themselves
they're trying to lift their world
their industry themselves they're going
to have a competitive advantage because
they have an interaction with us because
they have a personal relationship
so they're going to have that anyway and
they get that they're investing in
people not ideas
and so their relationship with people is
all that matters
and so when you tell them we'll open
source it and you can use it don't worry
we'll make sure you can use it we'll
help you use it they were fine with it
and this happened with intel with
microsoft with google
and to me what was interesting was we
had to change a bit of
mind step mindset we had to change a bit
to the mindset
of the tech transfer folks here but as
they saw
things like cmu cam and palm pilot robot
kit and the museum work take off
they got that we were a low-cost
investment because they didn't have to
go after patents for us they didn't have
to
file suits against anybody or create
complex legal agreements
and so in a way letting us run it this
way was cheaper for the university
and they could see that we're
establishing good relationships anyway
so what um so what were some of the we
have a bad news though it's 10 o'clock
i'm sorry guys okay there's like 50
other projects the problem is i mean
the lab has been extremely productive
it's impossible to cover everything you
can't go to what i was curious about is
nasa because you spent some time at nasa
um and you know doing work
so what kind of work did you do there
and also who were the people that you
interacted with there well i've been
lucky to get offered multiple
sabbaticals
over the years from carnegie mellon on
one of the sabbaticals i went to nasa
ames
i was recruited by dan clancy back then
he was uh running a large portion of the
intelligence systems work at nasa ames
and we moved there for a year and a half
and in fact i had my first daughter
there
my first child my daughter there and
lived in downtown san francisco which
was fantastic
it was an exciting time to go there
because of two things that were
happening one
the mars rovers were landing and so what
a perfect time to be in the excitement
of nasa
two we had started a major project where
we wanted to
actually create a mars rover-like
experience at museums all over the
country
so the people could see what it's like
interacting with a robot doing science
and again it was the same ethos
we don't want you to just play with the
robot we want you to be a scientist wear
a white lab cut
imagine looking for signs of life in a
rock yard and so we wanted to create a
robot
experience that gave you that
scientific experience as closely as
possible that you could feel
the quest of science looking for life
doing a
chemical test on a rock so it was
perfect because i was at nasa we were
doing the educational deployment
we worked with the exploratorium which
was right there in the national
aerospace museum on this side of the
world
and i was able to be in the middle of
the hubbub of
the nasa landings my job there was to be
lead of robotics for nasa ames
so i was directing a group called
intelligent robotics group which is the
principal robot group within nasa ames
and so as part of the lead position
there i was trying to chart a course for
how our group grows
what projects we do i was working on the
30-year
strategic uh planning effort for nasa as
in general at nasa headquarters
and so it was really an exciting time to
understand nasa learn about the goods
and the bads of government work
and government funding but also be so
close to the action
that i could do my best possible job
directing our educational robotics work
with the per the
personal exploration rover that we ended
up deploying to a number of science
centers
and who who did you work with nasa the
robotics group included
fabulous people randy sargent anne
wright matt dean's liam peterson
susan lee the list goes on and on i had
something like 25 or 30 people in my
group
and then above me there was jimmy
crawford and above him
david corsmeyer and above him dan clancy
so that was my chain of command so to
speak
and it was fabulous they are people who
are genuinely interested in furthering
human knowledge and
figuring out how humans and robots can
together go back to places like the moon
and mars
which is constantly in flux and the
chance of this happening
varies from one percent to five percent
goes back and forth bouncing around and
what i mean in terms of the 30-year plan
what were some of the points that
the end of the 30-year plan at that
point was that we were just about to
take off and go to
mars with human flight the exciting part
for me
was we were going to go to the moon and
take the
lunar regolith the lunar soil center it
with a laser and that makes concrete it
makes great concrete 2000 psi concrete
so we're going to make concrete
structures on the moon all with robots
so then when the first next astronaut
lands on the moon they go inside and
take their helmet off and
they're home that was the the dream that
i just found remarkable was that we're
going to have a
robot factory making housing for people
on the moon
that was good science fiction
i really have to go i'm so sorry no no
this is great