just start by asking where you were born
and where you grew up
okay um i was born in akron ohio
lived there for two years and then moved
to buffalo new york where i spent
most of my formative years i was an
undergraduate university of buffalo
so hometown and uh then went on to
uh spent a year working for a company in
an art
in ann arbor uh doing uh computer
design graphics and then moved to
mit to do my graduate work in artificial
intelligence
and when did you first encounter
robotics
when i first encountered robotics so rod
brooks was at mit at the time
and he was very active doing robotics so
a number of
his grad students were contemporaries of
mine and uh
you know i saw what they were doing and
they had robots running around the lab
but i was kind of a good old-fashioned
ai guy
and didn't actually do anything in
robotics
during my graduate work when i
when i graduated um i was
intending to do to go into into
industrial research and that's where i
spent most of my uh
time looking for jobs and then
towards the very end of my job search i
get a
a call out of the blue from tom mitchell
who
along with red whitaker and takeo canada
had landed a
fairly large nasa contract to build a
prototype mars rover
and red was going to do the mechanism
and control
and takeo was going to be doing the
perception
and tom was going to be doing the ai so
he was looking for someone who
you know a postdoc who could come in and
do ai
for robots and it was such an intriguing
idea
something i'd never really considered
but such an intriguing idea that i ended
up taking the job
um i figured i'd be in pittsburgh you
know
two years for uh for the post doc and 22
years later i'm still here
and uh it's funny actually a couple
years ago
one of my students came up to me um with
a copy of one of my
papers that i'd written when i was a
graduate student and he said
is this you or is it a different reid
simmons because he couldn't believe that
you know that what i do now that i had
worked in this in this very different
area you know 20 years ago
but uh yeah so i basically completely
changed uh
um you know focus of attention once i
got to carnegie mellon
so what was in your graduate work what
was your thesis and what was this
um the thesis work was on combining
what's called causal reasoning first
principle reasoning and
rule-based systems and i did the work
in the domain of geological
geologic interpretation so basically
trying to understand
given what the earth looks like now what
the
forces were that that caused it to be
that way so where
you know where were the earthquakes and
the falls and
there were volcanoes that split the
rocks like this
and sedimentary rock so it must have
been originally
ocean bottom that then rose up and
things like that and so it would reason
about
patterns and first principles in order
to come up with these interpretations
what year was the the lunar project uh
it started in 88
that's when i got here um went on for i
think
four years um
um the the nasa work was just great we
did
did lots of work with nasa over the
years and
thoroughly enjoyed all the work with
them they just
had the best problems and you know
really good people that worked
at the nasa centers that i could
collaborate with
yeah that was that was that was a really
good time
how had you gotten to know tom mitchell
who invited you over
so tom and my advisor randy davis were
good friends and uh i guess tom just
you know put out uh you know question to
people he knew whether
they knew of anyone who was graduating i
don't think i had met him
at conferences beforehand but uh um
but yeah it was made basically through
my advisor
and did any of your work on causal
reasoning come in handy on the
liver project not on that project but
subsequently
you know some of that work has been has
been applied um
the uh a lot of what the
uh the work
well yeah i guess some of it has been
applied a lot of the work
was um um
um what was i saying
a lot of the work that is a graduate
student was involved in uh
planning and uh so i've done a lot of
work since then
in planning not so much in the the rover
project but uh
subsequently to that and so so things
have come come back but even there
um things have changed dramatically
as a result of some of the graduate
students that i worked with
um we all got very interested in
probabilistic reasoning probabilistic
planning and so that's basically
um kind of taken over the way that i
look at how to deal with robotics
problems so there's a lot of
uncertainty in in robotics and reasoning
about uncertainty
in a probabilistic way is something that
that we find very important so that's
what we've been doing a lot of
since then um when i first got here
um there was a lot of um
so it's robotics in those in those days
were basically
kind of one-off mechanisms you know you
i remember very distinctly that
[Music]
at those times that red's group
when they built a new robot before they
started programming if the first thing
would do is
uh build their own operating system
real-time operating system for the robot
because there really wasn't anything out
there
that was suitable you know lightweight
real-time they could
they could do what they wanted so they
would they would do a lot of that on
their own
and you know every project was different
and so they they do that again
and so you know i felt that there was a
need for for
tools that would help make putting
together
robot systems easier in particular at
the higher level
so you know a lot of work can be done in
controls and and real-time systems
but not much work at that kind of what
we call the task level control
so for the first
you know um seven or eight years
that i was here um that was my focus was
on uh
designing uh robot architecture software
architectures
and um that was completely different
than anything i had done as a graduate
student um
but it was it was more out of necessity
than anything else that we
embarked on that um and since then um
i've been kind of going back to to my
roots ai roots
in terms of planning and reasoning um
but
um yes that was that was something that
was important to do
what were the big challenges in
designing a robot operating system
in architecture the the
the big challenge was that uh there was
this
uh gap between um the
kind of real-time control that you
needed to make the robots
um operate in the world and
uh the um kind of
unbounded computational nature of of the
ai parts of the system so
uh you wanted the the the robot to react
quickly to contingencies
but still you wanted it to plan and that
planning could take
you know a large amount of time so um
it was kind of bridging that gap is
where we spent most of our efforts
and what were some of the the things
that you worked on during the
the first rover project did you mention
nasa has all these interesting problems
so uh designing the the robot
architecture was
was a large part of it um
we did work in gate planning so where to
where the rover should put its feet it
was a legged rover so where should put
its feet
um that was uh basically a combination
of
of using perception to understand where
good places were to put
for the rover to move and um
planning planning paths um and that led
to a
a a long probably a decade-long
set of work that that we did for nasa in
terms of uh
uh path planning for for rovers
um um navigation planning i should say
navigation planning for rovers
um uh which culminated actually one of
the
one of the graduate students who uh who
worked with us
in the early 90s went on to nasa
and took the ideas that uh we had
developed
here at carnegie mellon and uh ported
them to the uh
mars rovers the spirit and opportunity
and so
basically they've been running our
algorithms for
you know seven years now which is really
cool
it was interesting when they when they
first launched
we also i we all said that you know if
if the rovers work work well you know
it's
it's a great win for nasa and if they
don't work well carnegie mellon's going
to get blamed but
they worked they worked flawlessly and
uh um it was it was really great to see
so
that was that was that was really
exciting because we actually had some
impact on um on an actual mission
and who were the people that that went
from here to there
and he's he's still a jpl and he's still
working on he's working on the next
generation rover
so um and he was in here
he was not he was not a student of mine
we hired him as a postdoc
um to to do some work and uh then when
the postdoc
ended he went on to uh uh to to jpl
to to work with nasa
how many uh phd students have you
trained i have been here
um i i think i've graduated
like 12 or 13. i i don't i don't have
the exact count
i'm sorry so
all right um i'm going to be interrupted
in a minute because
my son's here has to pick up with some
pimps and paperwork
okay yeah we can just wait
all right um so yeah
when he'll he'll knock in a minute um
sorry yeah so your uh phd students um
what are they going on and what are they
doing um
basically all over uh my first phd
student
is still here he
graduated in 1995
actually funny story about that um
uh my wife was pregnant with our third
child at the time
and uh we worked really hard to make
sure that
his defense would not coincide with
the the delivery um but she ended up
being three weeks early
and it happened to she happened to go
into labor that day
so um i was in the um the
labor room with her on the phone with as
with his defense because you know his
external member come in everything been
set up
and it was you know just easier to do it
that way and uh
the nurses couldn't believe that i was
doing this but
it worked out it worked out fine that
the defense ended
and you know and then she went into the
delivery room and
everything worked out fine but i always
remember exactly when
um his defense was because uh
it was um the day my my son was born
uh so anyway so i was 1995. so he's
still here
he's uh he's a professor as i am in
robotics
what was his name uh sanjeev singh
and uh i have one other
um no two other students in academia
one is fenn koenig who's now at
university of southern california
and chris erinson um who
i co-advised with red whitaker did the
urban challenge
and he's he's on leave at google but we
all expect him to come back
um the rest of them are scattered around
uh no large number of them are actually
at nasa
both at nasa ames and and at jpl
um you know that was a it was a really
good synergy for um
for for nasa and us you know they would
provide us with funding and we provide
them with
well-trained students so i think four my
students are still at nasa now
and within jpl and ames are they all in
the same research groups
it's kind of spread out yeah
and did you work with the same nasa
groups
uh every time or were there different
groups that you've worked with uh
there were a number of different groups
so um there was a couple
two groups at ames that i was involved
with
um and uh like three groups of jpl
and who were the groups who were there
uh so there was a a
group at ames that was doing
um uh robot work uh terry fong
mike sims um dan clancy uh there was a
group at jpl
that edited ames doing uh program
verification
uh charles brosher and claus havilland
uh the jpl group was uh
um i worked with steve chan uh rich
doyle
um um
i don't remember his first name al jabri
was his last name
um yeah i can't remember first name and
uh isa nesnus
um uh
mark well did actually i guess i never
really worked with mark mamon
so that was that was that was kind of
the group i'm sure i'm forgetting names
but um it was it was spread out over you
know
different groups i mean they're all they
all kind of work together and everything
but
uh and after your post talk have you
continued to work with red whitaker
and collaborations um
[Music]
some not not a lot but some yeah well
some of your collaborations
um
there was some some nasa work um
uh so there was a lunar rover
project that we were involved in
together um
not not a lot i have been working a lot
with
sanji singh though who's as i say was my
my first student
and we've been involved in a number of
activities over the years
the most recent one being a
multi-robot coordinated assembly
so one of the things that i've done is
you know a lot of people will um
kind of do basically basically have the
same path
for their whole career and they just
like pushing these these this big idea
and
as far as they can push it and
i don't have the patience to do that so
i've been kind of bouncing around so
so you know every five or six years i
you know i start up a new effort and you
know old ones kind of die out so the
architecture efforts that was you know
kind of the first thing i did
that's died out um don't do don't do
much that much anymore
uh the rover navigation stuff was big
for a while that's died out
um mostly because nasa is not funding
any work in that area
all right
okay so uh so you did the architecture
the navigation
oh and then um oh and then
then there was a there was a period of
doing uh
indoor robot navigation um
and uh most recently we've been doing
multi-robot
assembly large-scale assembly so
coordination of multiple robots
and i guess my current passion is human
robot interaction
particularly human robot social
interaction so we've gotten a number of
uh projects in in in this area
both for uh conversational interaction
how robots can
uh engage people and and and talk to
them in a socially acceptable way
mostly dealing with nonverbal
communication
and
navigational interaction so how you move
through space in a socially acceptable
way
things like how to get in on alpha
elevators uh
passing people um and always things like
that
um so so every every number of years
a new thing comes up how'd you get
interested in the human robot
interaction
good question uh this is this is this is
a good story
because um so
the um in the
mid 90s um i had a student who is sven
koenig who is very interested in
probabilistic um reasoning
and uh
he and i uh put together a uh
a system that did navigation using
markov models and it was
different than what most people right
now nowadays it's
it's probably it's very common because
it turned out to be
a very a very useful thing to do and
sebastian thrawn came up with a much
better way of doing it than we had done
it and it's it's very popular now
but back in those days it was it was not
rude
it was not really very uh well accepted
um so eli norbush had done a little bit
of work using
probabilistic planning but uh
not not in any real big way
so what we wanted to do is we wanted to
demonstrate that this thing was
more reliable than um you know than
competing technologies
and so we did an experiment where we
would have the robot wanting
the halls for hours a day as long as his
batteries would last every day
and it eventually accumulated like
300 miles of driving indoors
and while we're doing that we noticed
how people would react to
the way that the robots uh reacted to
the robot i mean
this was you know 15 years ago
um seeing you know the idea of seeing
robots
actually driving around you it's one
thing to go into lab and you see it and
you know
you know put in a in a protective
setting but to see them actually driving
around you was was very unusual in those
days
um and uh we noticed that people
would react to the robot in very
different ways than they would react
other people
um and realized that a large part of
that was prop would
was probably because the robot wasn't
driving around the environment in ways
that they expected
so it was it was very unpredictable so
in
particular things would things like
the robot would be driving towards
someone someone would be walking
um the robot had this algorithm that
would
when it would when its path would be
blocked it would go to the largest open
space
um and so more often than not that was
um
to the person's left so the robot would
start moving over to the left
but the person who is assuming that the
robot would be doing the
socially acceptable thing would also be
moving over to
would be over to his right and they
would come at each other
and then the person would move out of
the way in the meantime the robot saw
its path is blocked so it would go this
way and so there was this
uh dance between them until finally
invariably the person
gave up and the robot would win and go
on its way
um and so eventually people would just
get in a habit as the robot would come
they would just kind of move to the side
let the robot pass and then and then
continue and uh we
we realized that this was you know kind
of socially unacceptable and if
robots were going to um
actually be
accepted in society they would have to
abide by people's rules
so that that got us into this human
robot social interaction
and it's it's something i think that's
uh is going to turn out to be more and
more important as
robots become more common in society
what were some of the first uh studies
that you did
with robots and social interactions what
kinds of robots were you using
so the um
the the the we started off by by using
xavier um which was a
robot that was built in the very early
90s to study study uh
navigation um it was kind of a 24-inch
base it was pretty big
kind of a cylindrical robot um
and uh the studies the two studies that
we did then were
having the well three actually one was
having a robot
stand in line so that turned out to be a
really
cool project and
led to some nice results the second was
having the robot pass in a
on the right which turned out to be
fairly straightforward because the way
that the system was architected
we just needed to add a little bias for
the robot to go one way or the other
and the third which was actually never
published was getting having the robot
move on and off the elevators in a
socially acceptable way
and those all culminated in
2010 no 2000
2002 no the the
mobile robot challenge yeah
yeah the triple
okay um
i'm pretty sure the first one was 2002.
you probably know better than me
yeah um and where we had the robot
um we entered we entered grace which was
a robot that we had
designed specifically for this
competition
in the aaa mobile robot challenge the
challenge basically was to have a robot
attend the conference and the idea was
you drop it off at the entrance to the
convention center
it would have to find its way to the
registration booth register
uh once it got there we get a map of the
environment
and a location where it was supposed to
give a talk
and they would navigate there and give
its talk and um
that was uh um
that turned out to it we were shocked at
the publicity that
they came out of that i mean there were
hundreds and hundreds of people who came
to the convention center to watch the
robot perform
basically the robot needed bodyguards
because it didn't couldn't navigate
through really thick crowds so we would
have to
push people away so that the robot would
have any chance of getting anywhere
it was really it was really crazy
but uh so we did we did rather well
there and and participated in a couple
more
after that um
and that you know that kind of set the
stage for a lot of the work that we did
subsequently to that so that that one
but that one at
um involved getting on off elevators
um and standing in line uh we didn't
actually
use the socially acceptable passing in
corridors because
we didn't expect that we you know we
expected that the robot would be in
these large open spaces
um and that was a multi-institutional
effort we
involved with us
northwestern i'm going to forget someone
metrica
naval research lab and swarthmore i
think those are the five that were
involved in that
in that project and uh
a large part of what what we did was
uh um integrate all this
um software that had been developed over
the years by different
uh researchers into to kind of one one
system
and uh that turned out to be quite an
effort um
but well worth it in the end uh
what since then uh the main uh
there there's been a large focus on
conversational interaction
so we developed these uh uh this
roboceptionist
project where basically it's a
stationary robot
that uh with a uh graphical face
on a in a monitor and a pan tilt head so
it can move its head around but it has a
cartoonish but but three-dimensional
face
rendered in graphics and this is a joint
effort with the school drama
so they develop a character backstory
and
continuing episodes in the life of the
robot
and we can program that in and when you
you can talk to the robot about its life
you can ask it about
its parents or you know its siblings or
its love life or
what it thinks about its job or its boss
who happens to be me
and the robot robot for whatever reason
thinks that i'm a evil guy um that's a
continuing storyline no matter what the
character is i'm the evil guy
um so uh
uh but it's it's the the the the
carnegie mellon school drama is
is world class and uh the writers that
we get for this who's
mostly uh um uh graduate in the graduate
writing program
are just really really really good and
so there's been a lot of really
interesting uh
story lines that come out of that and
very interesting it's
it's a very different type of
human robot interaction study because uh
most of studies had been and still are
controlled so you go into a lab and you
sit down and you talk to the robot or
you interact with the robot
and this is what they call in the wild
the robot is just
along the same with with xavier the
robot's just there and it's it's an
uncontrolled experiment
so uh we get reactions that
differ wildly from what you would get in
a laboratory
um i mean you can't it's it'd be hard to
imagine
uh people going into a lab where they
know an experiment is
taking place and swearing to the robot
but we get that all the time or
propositioning the robot we get that all
the time
um so there's all sorts of interesting
things that that we're discovering about
the way people interact with the robot
um and some of the things are for
instance we had one of the students
rachel kirby
uh did a study where she uh
implemented a an emotional model on the
robots the robot could express
uh emotions and moods and it turns out
that
just showing different facial
expressions so if the robot looks sad
people would interact with it
differently than when the robot looked
happy
and you know maybe that's not surprising
in retrospect but
you know the fact that that you know
people who have absolutely no idea
what's going on
would just you know one day they're
walking in the hallway and the robot
looks happy and they go over and they
interact with it next day they're
walking in the hallway the robot looks
sad they just avoid it
that's what happened so um you know
those
those are some of the things that were
been very interesting
discoveries what are some of the
technical challenges that have come up
with the social robot
so in my mind the main technical
challenge
is that people are infinitely variable
um you know the robots
even with even with machine learning the
robots basically
adhere to rules you know there's certain
rules about how to
how to interact how to behave in certain
situations
and people just present an infinite
variety
of of the way that the ways that they
interact and um it very quickly you know
so
so the robot could do the right thing
first for
for a little while but then invariably
the interaction breaks down because
the the person asks question the robot
isn't
isn't uh capable of answering or
um they you know they try something like
sarcasm and humor that the robot just
doesn't understand
or um you know some something happens
like
um they turn their back on the robot for
a second
the robot doesn't realize it and
continues to talk to them
and they get pissed about that and they
walk away so
all those things happen with with high
regularity
but none of the things happen regularly
enough that
that we can actually capture that and
say okay here's a new rule for the robot
um and we do that but but still there's
there's a large number like
a quarter of the things that people say
to the robot the robot doesn't respond
to in a reasonable way
so um and and it's
it's even more so i think when you um
when you're interacting with the robot
spatially so the way people
move through space is
predictable for us as humans but it's
really hard to
kind of explain
just rationalize so that you write it
down in a set of rules that a robot
program would understand
and who have you been collaborating with
in this direction of your research
um
um mostly people at carnegie mellon
there's uh
uh drama professor anne mundell and uh
writer michael chemers um we've been
involved with um
ela norbosch manuela veloso has gotten
very interested in
in this area now aaron steinfeld
jody for lizzie
haven't had a project but have
interacted with sarah keisler
how did you decide to build a
roboceptionist
um well
uh
so there had been um
people people have been um kind of
complaining for a while that we're the
robotics institute
and most of the time you come in to the
robotics institute this is you know
in the 90s
you'd go into the robotics institute
you'd never see a robot
and so there was a push to um
to develop a a robot presence that would
be
there all the time um and uh
we're concerned about having mobile
robot because of the
the the the danger and also the fact
that
it would not be available full time it'd
only be available for a couple hours a
day as
long as batteries lasted but mostly
because the danger of having a robot
wandering around completely unsupervised
for
you know 40 hours a week
so we came up with this idea of
a robot receptionist
uh randy pouch actually suggested that
uh to me
that the uh a great way of keeping
people engaged would be to have uh
the robot involved in a soap opera so
that they would have
stories to tell and that people would be
engaged with that and keep coming back
to hear its stories
and so that basically was the genesis of
the
interaction with the uh the theater
department
um and yeah so we decided
that we would we wanted to so the other
the other impetus
was that um uh
there had been a number of
uh technologies that were
had been put out in the environment so
uh
hci uh human community institute
um had this really cool
piece of technology it was a bubble
machine where
we're
it was a bubble machine and so basically
what it was is that it
it had these um it was tubes of water
and these air bubbles that would come
out periodically from the tubes
and they would form patterns
and you could program the patterns and
it would it would
do the patterns that you wanted and it
was really cool
and people i mean you know in the early
days when it first came out
there were lines of people you know
waiting to use it everyone liked to play
with it
and then over time people stopped using
it and the mechanism got old and
it started leaking and they took it down
but i think by the time they took it
down
like it was hardly used at all
so the question was how could we
develop a technology that would maintain
people's interest over
very long periods of time um and so
you know the idea was to have a robot
with a character and a personality
and as randy suggested uh uh
story lines that would capture
people's imagination and and uh you know
make them interested in coming back and
hearing more about the the robots lives
so that's what we did and it turned out
to be a fairly successful experiment uh
the first year um that the
roboceptionist was there we had a huge
number of people interacting with it
which then the second
third year kind of calmed down but but
this has been going on for
seven years now and it's been pretty
even over the over the
the length um doesn't mean that the same
people come back all the time but
uh um you know there there's enough
repeat
visitors that uh it keeps the the
interest level up
on a daily basis and what were some of
the other
robots that you worked on
oh gee um so let's see so we had xavier
who
which basically did the robot navigation
there were a couple mars rovers ambler
um
rattler was the lunar rover
did that work with both both the ambler
and rattler work with eric krokov
marshalli bear was also involved in
those projects
um we have a robot called bullwinkle
which is a mobile manipulator um
and it it coordinates with this
robotic crane that we borrowed from nist
10 years ago and haven't gotten around
to returning yet it's
huge i don't think they'll ever want it
back because it would
you basically need a trailer to uh you
know
an 18-wheeler wheeler to cart it back um
one of the grace and the roboceptionist
the latest robot that we're just
starting to use is
called companion it was uh developed
by rachel kirby who is co-advised with
jody for lizzie at the hci institute
and it's a omnidirectional
humanoid robot so it's got a
fiberglass shell that's shaped roughly
like a human
it's got a graphical face like like the
roboceptionist does
but the face rather than being on a big
screen is embedded in a
three-dimensional head
so it looks much more organic and
it's going to be used for social
navigation
so one of the things that rachel found
in her thesis was that
robots that um um
the the uh um holonomic growth i'm sorry
the non-holonomic robots that basically
have to
turn their whole bodies in order to
change
orientation um turns out to be not very
socially acceptable
um as the robot turns people are get
confused about
uh what the robot is going to do so
people as they're moving down the
hallway
tend to keep their body aligned with the
direction of travel and they just
sidestep it they want to move away from
someone
so we've developed a robot that can
actually side step like that and
we think that it'll prove much more
socially acceptable
and how does the robot side still what's
different about the design compared to
that well it can it can
move instead of just moving in the
direction that his body is facing it can
move side to side as well
so it can keep his body in this
orientation and just move to the side
and so is it we were just talking to
ralph yesterday
with the ball so is it on something like
that or
do the wheels just kind of it's
yeah it's um it's this thing
they're called mechanum wheels or
swedish wheels and basically they're
the wheels have have rollers that are
kind of set
at the 45 degree angle so that as the
wheel turns
these rollers turn as well and they can
just slip so depending on
on the direction which you if you push
in this direction they go forward if you
push in this direction
it just slips to the side so it's yeah
it's not like ball bot but
yeah could you tell us a little bit more
about the different
um nasa projects that you worked on
um there are a lot of them
um so we worked on
um um
the first one was was ambler which was a
a very large eight-legged
six-legged
six-legged i think six-legged walking
robot
um very unique design this is designed
by john berries as his phd thesis and
john joined the faculty here has been
very successful since
had some projects with architectures
i had some projects with navigation both
for
lunar rover and for mars rover
um
had a project oh actually uh i had a
project for about this one i had a
project that was uh
uh develop an autonomous um
spacecraft so this was basically ai
in space so this was actually a mission
to set
this flew um so we developed we i was
part of the team that it
worked on this project it was to control
autonomously control spacecraft
normally they're controlled uh from the
ground it tells them
you know when to turn on the motors how
long to lan how long to burn everything
this was all decided on board using ai
planning technologies
it was a very very fast-paced
very intense project
very large product probably the largest
project i've ever worked on
um and uh it was really cool because it
flew we went down to
um cape cape kennedy and saw the launch
and then six months later it actually
did its thing it was quite cool
what was the spacecraft and what was the
mission uh
the mission was to visit some asteroids
um
it was called deep space one it's got
deep space one
so it was a technology demonstration
mission so it
demonstrated a number of different
technologies the only one the only other
one that i remember was an ion
propulsion system
um which is i believe since been used in
other missions as well
as far as i know the air in space hasn't
been used again but
it was it was a great development effort
um
what when was this
uh geez i don't remember 98 somewhere
around there
trying to read it
um
yeah all right
okay i'll believe that that's close
enough
um and then and then we've had work on
uh
uh multi-robot assembly most recently
and as far as you know is that the first
spacecraft that was completely
controlled by ai
yeah yeah definitely and there haven't
been any
sense not that i know no i mean there
there's been
there have been spacecraft that have had
ai components in it but
as far as kind of being um completely
controlled
by ai system now i don't think so so can
you tell us about
how you design that system and how it
looks
um i don't know it's it's
it's not bad sorry
oh crap
oh jeez
hold on
um yeah it's not it's not interesting
um what about is amelia another
yeah um
so amelia is the base that became grace
so there was a meal it was she was
called amelia for for a while but
um there wasn't there really isn't much
to
talk about amelia it was mostly
we used the same base um put a face on
her and it became grace
and so she's been she's been known as
grace since then
so what did you i see a nasa peer award
um sorry okay
[Music]
um
i don't know a couple of them um oh
i remember what this was so um
we uh the other one another pro so
practice i
forgot to talk about was ben actually
was involved with for a number of years
was uh
uh robot architecture problems so nasa
wanted to
put together a common architecture that
they could use for
kind of but a lot of their
uh robot projects and uh i was involved
in that
and uh i was i was i was involved in
that for a number of years
uh we basically
took the navigation work that we had
done
and integrated into this architecture
and it turned out to be you know
it was an interesting effort the
architecture
was fairly successful
they ended up open sourcing it and
people can could use this called clarity
and it was uh run by uh issenes
and yeah so there was uh rather
successful
nasa um gave out awards to people for
that so
what about your work on um mixed
autonomy
and the you mentioned the crane and
coal and what was could you tell us a
little bit about that
uh yeah so so one of the things that
that
we had noticed in in doing some of our
multi-robot coordination work is that
the robot
would um actually this is this is
another area that
i should mention that i've been very
interested in um
one of the things i've been very
interested in is
error error detection default fault
detection air recovery
um so one of the things that robots do
very poorly
is noticing when they're in trouble and
so you
i mean this this came from i was
involved for many years in the uh
aaai robot competitions and you would go
there and you would
you would watch and see the robots do um
things over and over again um basically
get stuck in loops so they would
like um there there would be let's say
they have to collect
orange balls and so they'd see an orange
ball over there
and uh they would turn towards it to go
and and there'd be a rock in front of
them
so they couldn't go over the rocks so
they'd turn away from the rock
and then the cameras would detect an
orange ball over there
and so they turned towards the orange
ball and there'd be a rock in front of
it so they
and they would do this forever you know
literally forever you know sometimes we
just had to
um stop the robots because you know it's
like it was clear they were just never
going to get out of this this
loop um and similarly you know
these the mars rovers would would get
themselves into trouble
um where they would uh you know the more
they would try and
um you know not even noticing they were
in trouble the more they would try and
and
um move according to their plan the
deeper and deeper they would literally
dig themselves in
um until you know eventually
they would they would end in catastrophe
more often than not so uh very much
interested in trying to
detect these types of uh
um uh behavior changes and and be able
to react to them in
reasonable ways and one of the things we
realized is that
again you know i was talking before
about humans being infinitely variable
um the environment is not as variable as
people but it's still
pretty variable you know different
things can be thrown at you
and we realize that um um
you know to to get robots to be 100
autonomous you know do it all by
themselves
was just not going to happen anytime
soon
and so we were very much interested in
looking at how people and robots could
work together
to to help this out
and so uh what we're interested in is
having
uh the robots be able to um detect when
they're in trouble
and be able to determine for themselves
where where
when they could get out of the trouble
themselves and when they needed to
ask people for help so a lot of my my
recent work
uh with students like brennan sellner
and laura hyatt
have been in terms of uh recovering from
from failures so either from how to
detect
that the system was failing or how to
recover once the system was failing
and then in some cases how to ask humans
for help to to help the robots when
they're failing
and what kind of tasks are you doing
mostly assembly tasks so
putting together large structures like
space structures
and this is also an is it a nasa
a project um it's it has been supported
by nasa
yeah uh we've since been looking for
other areas of support since nasa's
sport has kind of dried up but
you know there are other manufacturing
companies that are interested in this
type of
of behaviors
has most of your research been supported
by nasa or have you had other funding
sources
um in the
that's i would say for the first kind of
ten years
this i've been here for 22 years the
first half of the time i've been here it
was almost exclusively nasa
um but since then the kind of branched
out
i've had a little bit of darpa support
not much
[Music]
onr support nsf
support industrial support
um we just got an air force contract so
you know
but uh um yeah nasa had been
my big support uh up until you know
a few years ago you mentioned that
nasa's support was
running out is there a particular
reason are they like not doing so much
funding for robotics anymore
[Music]
let's not go there okay it's political
okay what sort of industrial partners
have you had
um good question uh
general motors is um um
trying to remember has industrial
wow
i guess not a lot of industrial support
i'm sure there i'm sure there are others
i just general is the only one that
springs to mind right now have you had
other outside collaborations apart from
nasa um
um
no not really
what was the attack of a desert truck
oh um this was this was another nasa
thing this is this actually something
that was led by
by red whitaker where they sent a robot
to the atacama desert which is the
driest desert in the world and
basically um you know
had it had drive around for um
weeks at a time so it was basically
trying to show that
robots could operate in in very harsh
environments
so i don't know if you have something at
11 but it's
okay it's getting there um so we were
we have kind of an ending question but
dude is there anything you'd like to add
or
something that we missed before no i
think
i think it's you covered a lot of things
yeah
so our final question was um if you
do would you have some advice to give to
people who are interested in getting
into robotics
um
yeah let's see um
so when when i got my um
i got this this offer to come to
carnegie mellon
um and as i said i had not had any
experience in robots before
so i asked rod brooks who was at mit and
uh you know very well renowned for his
work in robotics
um and i knew him but not well and
basically asked him for his advice
and his advice basically was forget
about this planning stuff you know the
action is all in perception
and you know
i thanked him for his advice and i
wasn't going to get into you know i
wasn't going to go that far afield
but uh it's still it still is good
advice i think i mean the
you know a very large part of what
makes robotics hard is is
getting good understanding of what's
happening in the world um
i mean as i said that i said before that
people are very
very variable but one of the things that
makes social interaction hard
robot human role with social direction
hard is that robots can't pick up on the
type of cues
that uh that people give out you know
uh expressions and nods and prosity and
all i mean people are working on all
that but it's just not there yet so
you know it's still it's still a very
large kind of open area
but so that's kind of passing on rod's
advice
from 20 years ago my own advice though
is is that i think it's important to
to do something in robots the robotics
that you're passionate about
there's an awful lot of work where
people will um
kind of flock over after the next fad
and you know this decide what to work on
based on what's what's hot at the time
and invariably i well at least i don't
think that's a very satisfying way of
choosing uh research topics i think you
want to choose research topics that
that interest you that you're passionate
about that you believe will have an
impact on society
and uh and the field in general um
and then and then go with that and if
you're right and you do good work
then eventually you know the the funding
will come
and and uh people will notice um
you know there's the social robotics
work is is one thing i mean
i wasn't the first person by any means
to get involved in it but
but um when we started there was
relatively few people doing work in
in hri period and and even less work in
in social interaction and it's become
much much more
prevalent now you know 10 years later
i got into it because i saw that it was
an important thing that
the robots were missing and realized you
know
myself that figured that that unless we
had this type of of
interaction the robotic robots wouldn't
be well accepted in society
um you know so
but but it's important to as i said to
kind of choose topics based on on what
you think is important what you're
really interested in rather than just
following following the latest fan so
besides hri what do you think are some
of the other
um important directions in robotics in
the next
um mobile manipulation um
and um you know
better perception particularly
multimodal perception
because we get we get a lot of um
feedback from
from all our senses and able to put them
together very well and
you know that those are all going to be
very important
and given your background what do you
see as the relationship between
ai and robotics and over that time
period
um so
as robots have become more and more so
there's two things
one is that as robots have become more
and more capable
um mechanically um and
and you've been able and we've been able
to um you know computing power has gone
at the point where you can
embed uh very very powerful computers
on board
it's given it's given it's made it's
made ai
feasible um to do i mean ones with
robots really couldn't do much
you know planning didn't really make any
sense because the robots
you know didn't survive for more than a
few seconds you know a few minutes
whatever
um it wasn't very important but uh now
that robots are very capable and you
know you can
think about heavy robots that are
available
you know all the time then the ai
becomes much more important
the other thing i think is more more
fundamentally is that
thinking about ai and robots has
pushed the the field of ai in a
different direction
so when when i was starting out
when i was getting my graduate degree ai
was all symbolic reasoning
and there were you know a few people who
were doing
um non-symbolic reasoning uh you know
probabilistic reasoning but they were
very few and far between
um and and since then
uh once once you start taking um
the world seriously and that there's so
much uncertainty in the world
um it's pushed ai into
much more into uh non-symbolic um
representations of reasoning
probabilistic reasoning
non-deterministic reasoning
um the machine learning has been
become much more non-symbolic
than than than it used to be um
you know control learning things like
that have become very important
so so um
putting ai into robotics together has
has opened up a
huge new um set of problems for ai to
solve and
and and has led people into very
different types of
uh solution techniques than than they
had involved in in the past
great okay sure