we can start with your name and where
you were born and when
i was born wow okay um well my name is
terry fong
i was born in pittsburgh pennsylvania
um and did you go to school there as
well or
uh school at what age kind of
just quickly oh throughout so well um i
grew up
outside of chicago i went to uh to grade
school
junior high high school in the chicago
area
and then i went to do my bachelor's and
master's in boston
and my phd in pittsburgh and which
school were you in boston i was at mit
and in pittsburgh it was carnegie mellon
and what major were you pursuing at mit
mit i did a bachelor's and a master's in
aeronautics and astronautics
and then at carnegie mellon i did a phd
in robotics
and how did you get interested in
aeronautics
well when i was a you know when i was
growing up as a kid i always wanted to
to to build and then fly airplanes that
was sort of like my dream i thought i
was actually going to be an
aircraft designer and then i went to
college and i realized that
well actually there's a lot more to to
uh
to building aircraft than i imagined
especially a lot more
math if i uh actually got interested
in robots because they did uh things
that were um you know
quite a bit more interesting more
interactive with people so did you see
any robots while you were at mit
i did i mean there were robots all over
the place um
in the aeronautics department double e
computer science um
lots of different kinds of robots too
small big ones built for contests
and that was you know really kind of fun
and interesting because there were so
many different types
did you participate in any contest at
the time or no actually i didn't
um and it's kind of funny because you
know i've
these days here at nasa ames i have lots
of people that work
on you know various outreach projects in
this course they they work with lots of
kids
lots of students of all ages and you
know i've certainly
been judges for those kinds of things
but my myself have never actually
participated in a robot contest
what was the first robotics project that
you were involved in
the first robotics project boy let me
think about that um
it was probably um when i was a
sophomore in college
and uh at that point in time i was
working
basically part-time in the space systems
lab at mit
doing work on a i think a large
underwater
robot arm and who did you work with on
that so that was in dave aiken's lab
dave aiken was a professor at mit at the
time he had a
lab the space systems lab that did
neutral buoyancy robotics
as a way of developing and testing space
systems
and then he moved from mit to the
university of maryland uh
pretty much just after i finished up at
mit and what kind of work did you do
with the arm um so um i've always been a
software
nut and you know from a very early age
and so for me
uh robotics was primarily a thing that
allowed me to to use software to make
well honestly make cool things move and
what was the next thing that you did
that was robotics related
so um you know i actually worked in the
space systems lab at mit
uh throughout my pretty much the rest of
when i was an undergrad and then i
decided to stay for grad school and i
did my thesis
my master's thesis in the space systems
lab and i built a large
underwater uh robot arm it was designed
to be similar in scale to the space
station arm
but it was an interesting system because
it had a
a serial component so a three-joint
configuration that was serial and then
a parallel large parallel end effector
that was a steward platform so it was a
steward platform
augmented manipulator or spam and
it was it was a fun project to really
try to look at this idea of
very large scale manipulation
that had sort of like a coarse large
positioning component so a large
very large three-joint arm with a very
fine precision um parallel thing at the
end so you know correspond
mobility or manipulation what were some
of the challenges in
designing that system well um it was uh
it was
built by me and a team of undergrads so
i had uh you know freshmen sophomores
and
a couple seniors working on this and so
you know we had no idea how to how to
build anything like this this was
it was actually pretty large complex
thing for us
and it was interesting because it was an
underwater arm
and it was uh which meant that we had to
decide well how are we going to
to actually you know power the system
it's very large and it was
not something that you could actually
build large motors and seal it
underwater because we wanted to be able
to move very very large things
so we ended up looking at a combination
of hydraulics
in this case it was actually pushing
water not oil or anything
and air so lots of pneumatics and it was
really interesting i think
just trying to figure out how do you
actually build the system
and size it in a way that you could
actually move people
um because the lab actually did a lot of
simulated spacesuit work as well
underwater um and so it was uh you know
trying to take all these things so i
remember calling lots of like
plumbing supply places and asking for
you know various manifolds
and things and they want to know what
i'm using it for so i'm building a robot
and you know at that time it wasn't all
that common to see
you know sort of these uh these research
underwater robots built like that
um although these days you know you see
you know kids even from like fourth or
fifth grade building underwater robots
and they have all these contests and i
think it's a lot more common now
when did you finish your master's degree
i finished my masters in 1990
and then after that i i came out to work
at nasa for the first time so
i've been actually at nasa twice the
first time i was here from 1990 to 1994
at nasa ames
um and that was a i think a
for me it was it was really a break
because i wasn't sure i wanted to do a
phd
i decided i wanted to come out and
actually do some work in a research
environment and i came out here to nasa
ames
what were you working on when you came
here oh boy so uh
nasa ames has always been an interesting
place because it's uh
the work here has been very very very
diverse i do recall that
the first time i was here so 1994 worked
on everything from
high performance computing we actually
did some parallel
computing on a system called the iwork
which actually grew out of a project at
carnegie mellon called the warp
um basically a high power distributed
parallel computing system they actually
called it systolic computing kind of
like
like basically like the cardiac system
because data would be pumped through the
system
in lockstep um so we did some work uh
there with the parallel processing
we also actually got famous for i think
at that time
um virtual reality interfaces this was
back in the early 90s and there was a
lot of interest and hype about
you know oh we use vr and virtual
environments for all kinds of things
um the group here at ames really was one
of the the pioneers of trying to use
vr interfaces uh for robotics it was a
real kind of different thing and at that
time computers were
like silicon graphics octane computers
these you know you know refrigerator
size literally refrigerator size um
computers
that cost several hundred thousand
dollars and at that time that's what you
needed for doing real-time
interactive 3d graphics so we used
virtual environments for doing a lot of
things
for controlling robots in our test areas
here
we also did robots that went to the
antarctic there was an rov that we
deployed under uh the ice the sea ice at
mcmurdo sound that was remotely operated
with a virtual environment
that led into additional work we ended
up working on dante 2
with carnegie mellon the robot that
walked into mount spur
and so it's really interesting trying to
look at you know how you use these
virtual immersive kind of environments
as a way of representing the remote
world and for you know understanding
what the robot's doing
and what were some of the challenges of
designing these environments and
interfacing them
as an interface for the robot well if
you you know you have to recall that
back in the early 90s virtual
environments really
were sort of cutting edge um people were
looking at these for all sorts of things
from
from simulation of robots to financial
markets to
just social interaction and there wasn't
a whole lot really known about
i think the psychophysics of how people
should really be in these environments
just whole questions of how do you put
say stereo displays and a head mounted
display
in such a way that people could stay in
there for more than a few minutes at a
time because you know it's really tiring
um if you don't have the uh the graphics
right um it's also really tiring
at that time because head-mounted
displays were pretty heavy i mean there
were like 20-pound helmets
that you put on with these big you know
you know actual crt monitors in them
um and then of course the you know head
tracking was done with these uh
magnetic electromagnetic systems like a
polymes tracker
or there was something called the the
ascension flock of birds
and these were extremely expensive and
very non-linear sensors so you know you
move your head and you know the scene
goes like this and
that was a real big challenge i think
trying to create environments that
people would be comfortable in for a
long time and
be productive in was a real real
interesting research
area so who was in charge of the
robotics research here then and who else
was working in that group
um so when i came to nasa ames there
were a couple people involved with that
one was mike sims he was here
running the group when i first arrived
and in fact he's still here at nasa ames
but i think about maybe 10 months after
i arrived
he stepped down from that and then
butler hein took over the group
and butler was the one who really got
into
virtual environments virtual reality
user interfaces
we actually worked a lot with steve
ellis who's a pretty well known
researcher in the area of
spatial displays and actually steve is
still here i actually used to carpool
with steve to work so
we get an extra two hours of work every
day and just talking about virtual
environments
were there any other people that you
worked with who were related to the
virtual environments or the robotics
group
uh well actually so we did uh i think
one of one of the things that really
distinguished the robotics group at the
time and at that time it was called the
intelligent mechanisms group
um and we tried to look at using virtual
environments
to operate these robots but they were
really motivated by using robots for
scientific exploration
so we actually worked pretty closely
with a number of scientists planetary
scientists who were interested in trying
to see
you know what did the robot actually
find and trying to understand what the
remote environment was like
so for example the project that we
worked on with the uh
the rov that went to antarctica that was
a project that we worked very closely
with
carol stoker carol is a planetary
scientist here at nasa ames
she does a lot of work uh in a pretty
broad range of things she's worked in
underwater environments she's done a lot
of drilling work
she's pretty well known for a lot of
mars related planetary science
and um you know because of that it was
interesting it was a to me it was a
blend of really
you know engineering because we're
trying to build robots and trying to
build uh interfaces
but it was motivated primarily by those
robots being used for science
and so that was an interesting thing
from nasa we hadn't actually seen within
nasa this this real sort of
blend of science and engineering around
things like virtual
reality interfaces and robots and what
were some of the motivations and
constraints that came from this
interaction with science and the science
scientists well part of it was that we
were trying to find ways of representing
the data that was meaningful to both
engineers and to scientists
especially data that came from robots it
was pretty unclear at that time really
what's the right way to to represent
that
you know you have to think back to the
early 90s and as i was saying the the
best
computer systems graphical systems at
that time were silicon graphic systems
but they were nowhere near what you see
on the market today i mean we're we're
still talking you know flat shaded you
know
uh polygonal models of perhaps uh tens
of thousands of polygons at most
you know so these days uh you pick up
anything even your cell phone and it's
probably got 100 times the graphical
performance of what you saw back then
so trying to represent scientifically
meaningful
data and engineering data in a
meaningful way with such
limited resources was a huge challenge
you know i do recall that time it was it
was sort of a real cutting edge to look
at the idea of texture mapping
you know that you could actually take
images and put them on top of
flat polygons and that was
you know our way of increasing the the
realism
of those scenes but it was very very
limited it was very hard
to really create things that were i
think meaningful and accurate
um that was i think probably the biggest
challenge
what made you decide to go back to get a
phd
um well you know i i worked here for
four years and i worked on a lot of
different
remote operations projects a lot of
different kinds of software
and i decided that i i really wanted to
to spend time and be able to focus one
area
because i think one of the things that's
probably true just about anywhere
um you know in in a research environment
especially in an applied research
environment is that you have projects
after project after project every
project
and you never really get a sense to to
just focus on just one thing for a long
period of time
um just because that's the nature of i
think research in these sort of settings
so
i wanted to go back and get a phd
because i wanted to actually
be able to spend multiple you know years
working on something
um and so at that time it was a decision
of well do i stay in california
because clearly there are a number of
good schools here or do i go to
to someplace else and i chose to go to
carnegie mellon because it uh
it was and still is in my mind um you
know the the best
place for you know learning about robots
and you mentioned that you had worked
with some of the carnegie mellon folks
on dante
and so did you already have some
connections with people there
from that project yeah so i actually
knew a lot of the people
at uh at carnegie mellon extremely well
before i went there because we had
worked together plus i'd you know on
visits there
i would go and of course hey you know
hey the guy from nasa is coming so let's
go
you know show him around and meet people
so i was uh you know
extremely um you know familiar with a
lot of the research at cmu and i
actually knew
many of the the professors there
extremely well before going back as a
student
in particular you know i had spent quite
a bit of time uh
working with reid simmons and red
whitaker and chuck thorpe
and so knew them all extremely well
before going i also because of the work
that we had done on the iwork
knew a lot of people in the computer
science department
thomas gross and john webb
i can't remember who else but it was
an interesting transition for me because
of course you know having worked as a
researcher
and really knowing all these people and
then going back as a student it was a
little
you know kind of odd actually because
you know i went there and i
i knew all these people um and extreme
actually knew them really well
um so it was an interesting transition
how did you choose an advisor once you
got there
um well actually you know i um i went
there um
knowing that i really wanted to work
with uh with chuck and with red
because i had worked with them before we
had actually you know the course of the
projects we had done together you know
talked a lot we had a lot of similar
interests um and of course
you know having come from nasa i wanted
to do something space related
um or something that was really in the
field and both chuck
and red really you know did projects
like that um
i did also talk to other people too i
mean obviously i i
i knew uh that reid simmons did a
tremendous amount of work at that time
you know the ambler project was a big
project at that point
eric kotkov was at cmu at the same time
and
i really um i really almost uh decided
to go with with eric because
when i was talking to him during the
sort of marriage process they have at
carnegie mellon where students arrive
and you spend time
you know meeting all different people
and the sort of like a matchmaking
between
you know students and advisors one of
the things that really struck me about
eric
was he said you know his goal in doing
research was to be able to do robotics
research in all the five senses
so he wanted a robot that could taste
and one that could smell and then one
could use its vision
and i thought that was a really great
you know um you know goal
um and i was i was this close but then
you know
i think that the lure of really working
on on field robotics really drew me to
chuck and red and what were some of the
projects that they had going on at the
time
uh so this was back in the early 90s
the nav lab project was still going very
strong at that time i think there were
only
maybe two nav labs by the time i left
there i think maybe half a dozen or even
more than that nav labs of various
sizes and shapes the antler project had
wrapped up the dante project had wrapped
up but there was some work
in building some lunar rover prototypes
that i recall
being done um and i ended up in my
in my initial work at cmu focusing on
trying to
to add proprioception to nav lab
really just trying to figure out you
know could you drive by feel
which is an interesting thing you know
obviously cars
even back then but certainly today have
all these traction control systems and
you know they try to the cars themselves
can take some action to safeguard
the driver if you start running off the
road or skidding around
you know anti-braking is a good example
of that
but with nav lab what we're trying to do
is figure out well could we
feed some information back in the system
that when it's driving off-road
would allow it to perform better so you
know realize that oh this is really
really bumpy or okay i'm shaking left
and right i'm
mainly going right therefore i should
try to correct and go left
um and that was the the research that i
started working on um
uh really primarily with chuck and i
think it was driven by the fact that
nava at that time was moving more and
more towards off-road
scenarios and what were some of the ways
that you
kind of implemented these solutions with
matlab so what we
were looking at was i think a
combination of the sensing because this
was really meant to be proprioceptive
so you know i start off by saying look
i'm not going to look at anything that
looks outside of the robot i'm only
going to look at
you know say accelerations changes in
orientation
trying to pick out patterns and for
example
you know how rapidly we're we're
bouncing up and down or maybe how
strongly we're doing that
it was a really you know interesting
challenging kind of
area of research because it wasn't clear
well do you need just one sensor or do
you need lots of sensors you have to
look at
for example a wheel slip as well as
you know sort of inertial measurements
you know i think for
us as humans it's clear that you know as
we move we have a lot of proprioception
just because of our limbs and our joints
and our head
motion and everything um but the same
time you know you close your eyes you
think about yourself like on a roller
coaster
you know because whether or not you want
to look out on a roller coaster you
certainly feel yourself going up and
down and you can infer a lot about
at least what you you can think you
think you know what's going on in the
outside world
and you make a lot of judgments based on
that and i think that
the work that we do with nav lab
certainly show that you know you could
tell a bit of what's going on but the
problem is that just like
humans you know you you close your eyes
move around you think you know what's
going on
and that's only part of it because
you're not grounded in the world
um you know i think uh whether you're on
a roller coaster or actually if you're
actually an airplane i think in the
cloud bank is a good example
you know you may think that hey i'm
level because my body tells me i'm level
but in reality actually when you come
out of the
the fog and you find out oh we're
actually banked steeped over
um that's actually a bit of a surprise
we had the same sort of i think
challenges with nav lab too
what was your thesis on so
i um i have never followed the the usual
path through life
um i guess i've been kind of an unusual
person in that respect i've always
chosen to take the uh not even the road
not travel but the road not yet
built um and uh i think that was
reflected in the fact that you know i
had worked
uh in this industry or i mean in
particular at nasa and then went back
and so i was
already a kind of unusual grad student
being an older grad student somebody
who had done research and had published
papers already and certainly knew all
the faculty at more of a pure level
but then what happened was after my
first
year and a half because my wife
who was also doing a phd at carnegie
mellon
she had joined a research group that did
all their research in switzerland
so after a year and a half we left
pittsburgh
and moved to switzerland and i ended up
doing my
my phd research at swiss university
this was federal institute technology in
lausanne or epfl
and at the same time the research was
done at epfl but my degree was from
carnegie mellon
and during the first year my advisor at
that time was
was chuck at that point in time i
decided to just focus
on my research with chuck he actually
wasn't able
to support me because most of his
research funding came
from from darpa and it was difficult for
him to support
a researcher working out of the country
completely alone
so i was in this the situation that okay
i'm in switzerland
i'm trying to do my research for cmu and
where am i going to get funding from
but i was fortunate because at the same
time some people that i
had worked with at nasa ames actually
had decided to leave nasa and create a
startup company and they asked me to
join it so
for a period of time i was a cmu grad
student doing my research in switzerland
paid for by a california startup company
it was a company called fourth planet
that
actually did work based on the virtual
environment research we had done
creating these virtual environment
visualizations of real-time data
things completely unrelated to robots
things like computer network monitoring
just looking at say bandwidth usage and
you know connections for various places
on the internet
um and so like i said it was a kind of
you know different
approach i wouldn't recommend anybody go
through grad school with the goal of
doing your research you know 6 000 miles
away from your university six thousand
miles away from your
your advisor and being paid for by a
startup that you created
in california but it was it was a path
so it's at least two jobs and
one it was at least it was at least two
maybe three jobs but um
you know it was it was fun and uh you
know
you know like i said i've i've always
tried to take the you know the road not
even
you know yet seen did you work with any
of the people at epfl
at all yeah so at epfl at that time
there actually was an institute of
robotics there were
focus that was primarily on very small
scale systems
there are a lot of you know insect and
behavior based robots
as well as a lot of industrial systems
they did a lot of manipulation
for very you know precision high speed
pick and place type work
the group that i actually joined there
was the virtual reality and active
interface group
and that group was interesting from the
standpoint of really looking at
interfaces broadly so there was
computer-aided surgery
there was uh just visualization
visualization of uh
remote environments and there was a
little bit of robotics in that group too
that was within this larger institute
that did you know those other
robotic you know things that i talked
about
and um you know when i first arrived i
did spend some time uh
uh talking with uh
the time it was uh raymond clavel he's a
professor there who's famous for this
robot called the delta
it's widely used in especially drug
manufacturing lines for very high speed
manipulation
i also worked a bit with jean
daniel nico who was
his group actually created a number of
small robots there was one called the
koala and there was the kepera
which then led to a startup company
called the k team for a while
i actually don't even know the k team's
still in existence but certainly
their robots live on many research labs
these really nice small little tabletop
robots
i actually think i have one even on the
table behind you someplace or
maybe it's it's around here um
but yeah so so epfl was certainly a big
change for me from from carnegie mellon
it was a
big shift uh in terms of the way people
approach research
in terms of uh just the facilities you
know
going from a place that like carnegie
mellon which had hundreds of researchers
working in robotics to place which had
maybe
a dozen was a big change um but it was
also i think uh
you know really kind of liberating in
the sense that um you know
there weren't any expectations of of oh
you can only do this research because
you know there were just there's so few
people and everybody just did you know
what they thought was really interesting
so what did your thesis wound up being
on
so my thesis so you know actually having
moved to switzerland um
i of course couldn't continue my work on
proper reception with the nav lab
because the nav
at the time was a a retrofitted humvee
it was pretty hard to put a humvee in my
suitcase and take it to switzerland with
me
so i ended up
going off a completely different area
and uh the
advisor that i had in in switzerland his
name was uh charles bower
um and so you know it's kind of funny
that i actually had two advisors named
charles
although one was charles and switzerland
the other was chuck in and uh and
pittsburgh
and we ended up getting this whole area
of human robot interaction
at that time the hri community was
really very nascent i mean there wasn't
really any sort of recognition of the
term human robot interaction they were
sort of like oh
they're robot interfaces and you know
humans and robots
there was a conference at that time it
still runs today called called roman
uh for human you know robot uh you know
communication but they
it wasn't this notion that there really
was human robot interaction what that
did that mean
um but it was starting a lot of people
around the world were starting to look
at that
and i was really interested in you know
this idea that
as you get humans and robots working
together in similar tasks
how do they communicate not because i'm
curious
interested in you know you know
semantically how do they express
themselves or even necessarily
the display but really you know what is
that communication useful for
and so my thesis ended up working in
this area that i eventually end up
calling
collaborative control the notion that if
a human is working on something
and has a problem uh it should be able
to ask a question to the robot
and vice versa because it was clear to
me and
being a part of this was driven by the
fact that i was in switzerland which was
a very collaborative environment
that nobody has all the answers and so
therefore
maybe if you could ask questions you
could benefit from the other's knowledge
especially when you look at robots and
humans which are situated in the world
and have different perspectives because
there are different locations have
different sensing modalities
uh different areas of expertise
different levels of precision or
you know ability for doing repetitive
tasks that you could actually
benefit from having a team where the
human robot are exchanging information
and working together from you know sort
of a collaborative point of view
and that was quite different i think
from what people have been doing at that
time
were you talking to me too because he
has that concept of mixed autonomy
um which seems somewhat related but i
don't know if it is
yeah well it's it's uh it's the case too
that
you know for a long time within the ai
community there's all been all these
notions of you know a mixed initiative
and adjustable autonomy and sliding
autonomy and i don't know how many other
autonomy kind of terms like that
the time but those were really focused
on this idea that the robot has
some level of autonomy and you're just
trying to dial in
you know the right bit of autonomy for a
particular moment in time particularly
tasks particular scenario um and i
wasn't interested in that because
to me that is really sort of like
tweaking knobs on a control system
i was more interested in this notion
that humans and robots could work as
partners or as peers
um you know just like you know we're
talking that when i have a question i
should ask the view and i could you know
benefit from from your knowledge
so to me it was it was a whole question
of okay
you know how do you use this and when
would this be useful
so i was trying to look at scenarios
these are very task driven scenarios
it's not
not in terms of like social interaction
but it's more of like well we've got a
job to get done
therefore we have some common ground
because we're trying to do the same
thing
and therefore i know that if i have a
question i can ask it of you
because we're working towards the same
sort of goal
and so i don't have to worry about
trying to really you know completely
model you or try to understand you know
what you're doing i'm we're trying to
work towards the same
objective uh and therefore the uh
sort of the dialogue the the the range
of questions the range of knowledge that
we need to have in the system
can be more limited because it's very
task specific so what were some of the
tasks
um that you tried out with this so the
most basic one was just the
just moving from from point a to point b
just sort of the the classic navigation
task
um which we know um is is still a
challenge for for robots of any
size shape or form um in particular
i think that uh humans because of our
you know ability to look at the world
and relate
you know our prior knowledge about
objects to things we haven't seen which
might be similar like
i see for example you have this bottle
here of some trick i've never seen
before but i recognize it as a bottle
because it has these characteristics
which say oh that's plastic and
it's got a label on it and therefore you
know it looks like a bottle to me and so
i'm going to say well it's a plastic
bottle there
are really really hard still these days
for robots
and back when i was doing my research it
was clear that if you're trying to drive
especially a robot through a natural
environment
that the perception of obstacles the
perception of terrain hazards was a huge
problem
so i was interested in looking at the
question of could we have the robot ask
questions of the human and say hey
is this dangerous you know can i drive
through this can i do i have to drive
around it
and so my my research was really focused
on sort of navigation issues
trying to see could we allow a simple
robot to actually make progress in the
natural world
if it has the ability to just you know
ask a question whenever it's uncertain
so i would do these interesting studies
where i would take for example
you've probably seen these uh these flat
paper cats you know they look like a cat
but it's actually just a piece of paper
um so i put those in front of the robot
and have people remotely operate the
robot and the robot would drive until it
would see something
and say i don't know what this is is it
a rock because it clearly couldn't tell
if it's a cat or not
or is it a caps is it a real cat and it
would send a picture back to human and
say hey
can i drive through this and of course
depending on the user then the answer
would be well yeah it's a flat plastic
paper cat so yeah just drive over it or
maybe no i don't know if that's a cat or
not sort of drive around
and that was that was a really you know
sort of very basic
thing basic skill that robots need to
have being able to go from point a to
point b but still something that could
benefit from the sort of interaction
with with people
and was it mostly natural language that
in a sense the robot was using so it was
asking
questions or were there other types of
communication that you were working on
it was um
you know it was the dialogue was very
very scripted from the standpoint that i
had chosen a very very
uh specific task and because of that um
i ended up trying to encode basically a
number of different
uh you know i guess uh primitive
pieces of communication so you know a
predefined list of questions or at least
topics to focus on
and the challenge there was then trying
to figure out okay if you have like you
know in your vocabulary
maybe call is not the right word but
basically say uh your ability to ask 100
different questions how do you choose
which question to ask
because you know you might want to ask
about a lot of different things at the
same time
but you actually want to end up choosing
you know the question that's the most
important for this moment in time
it's also the case too you don't want to
be annoying because
you don't want to you know ask question
after question after question pretty
soon the human says ah this is just
too annoying i'm going to stop asking
questions so trying to do a bit of user
modeling
was important trying to say well okay if
i have an expert in navigation
i should ask a certain type of questions
if i have a novice
um you know maybe maybe they don't
normally drive robots but they're
they're still pretty skilled in some
areas because they're human after all
trying to figure out how do you choose
to communicate with those different
you know types of people and that was a
big part of my thesis
really just trying to figure out how do
you choose the questions how do you
decide when
to ask these questions and for different
users you know do you ask different
questions
and what were people's reactions to the
notion of robot as peer
it was uh i think it was something that
within the community and as i said the
human robot interaction community was
really getting started at that point in
time
was really a thing that people didn't
you know quite have
a good sense of whether this is a good
thing or a bad thing
in fact even i would say today nobody
really knows uh there are notions that
that robots are certainly in everyday
environments and well is it a peer is it
a partner is it a tool
it's a great place to have you know long
and lengthy discussions and debates
about
but to me it's that's not even so much
the issue i mean i don't i don't
necessarily care whether or not a robot
is a peer or a partner or tool
if for a particular task because i'm
very task driven especially here at nasa
that uh however we work with it makes us
uh or gives us the ability to actually
achieve something better than we could
without it
and to me that was really the sort of
fundamental thing whether or not it's a
tool or not i don't know
and so for eddie pfl you used small
mobile platforms generally for this kind
of work
yes i actually did work with uh
the a bunch of pioneer robots at that
time it was active media i think they've
since changed her name into mobile
robots and i think recently were
purchased by adept
but at that time mobile active media was
selling a line of pioneers they had some
indoor ones and eventually created these
these kind of pseudo rugged outdoor
robots and i did a lot of research in
the lab and
a bit of research outdoors and terrains
which were actually you know
pretty benign but still used in a
natural environment
and did any of this research then carry
on when you came to nasa
yeah so yeah so when i came back to nasa
after having been gone for 10 years
i came back and the group had changed
from intelligent mechanisms intelligent
robotics
uh and certainly had continued doing
research in the area of using robots for
remote exploration
but when i first came here um i actually
came back
to to work with elon norbos because ela
had decided to take some time off
from carnegie mellon he was a professor
there for a while and he and i had
known each other for years and years and
years but never worked together
at that time he actually came out to
nasa ames and was running the group here
wasn't sure if he was actually going to
go back to carnegie mellon he eventually
did but
when he was first here i decided hey
this is a great chance
to actually get to work with ela who i
think the world of and
when i arrived here we created this
project called the peer-to-peer
human robot interaction project and that
was really sort of an outgrowth of my
thesis this whole notion that
humans and robots could work as peers or
as partners really exchanging
information
and there we were trying to go beyond
the kinds of things i had done in my
thesis which were you know limited to
pretty simple navigation tasks
to things that really were able to get
into the area of spatial dialogue
so you could say to robot hey you know
shine this light here
and because it would have some knowledge
of you and what does here mean
and the object and the task at hand be
able to react more appropriately
and that was a great project because it
was a project involving
the group here at nasa ames some
researchers
from the robonaut group at nasa johnson
involved carnegie mellon because reed
simmons was part of this and alan
schultz from the naval research lab
so it was a really i think great project
for
about a year and a half and looking at
this whole notion of can you really push
towards sort of more pure interaction
at least focus on the the area within a
within a specific task
um just not in general i mean we we
never had the goal of trying to create
robots that could just you know freely
interact but really
interact within the context of a
specific area of work
and what were some of the results and
things that you learned from this
project
well uh that project was also i think uh
you know much broader than my thesis
because we were trying to use natural
language
so there was speech recognition involved
we were trying to use a spatial uh
reasoning
so there was some computational uh
cognitive modeling involved as well
we're trying to to really do some
perspective taking trying to figure out
okay if i'm here and the robot knows
that i'm here but it's over there and i
say move this there
you know what am i talking about is that
reference to me into my body frame and
my
reference to the object and the task i'm
working on
so it was a bit of a challenge trying to
figure out what area we really wanted to
explore first which area wanted to make
the most progress on
and i think that for us the big
contribution was that we were
able to show nasa that robots and humans
could try to work together
in a more autonomous manner
it wasn't like the traditional nasa
approach that hey we've got a robot
we're going to send it off we're going
to command it
and we're going to monitor it but really
this notion that humans and robots could
work
more closely as independent peers
and that was a huge interesting i think
notion for nasa at that time i do
remember doing lots of demonstrations to
people we have
you know managers from nasa headquarters
come by and they think oh wow that's
great that's just like science fiction
and i thought wow okay great so now
we're making science fiction into
hopefully some small bit of fact
that was a lot of fun how did you get
them interested in this because
we have heard a lot about the kind of
command and control approach and also
the need to be very conservative when
you're working on actual missions
and so this is obviously kind of a more
research oriented
um project i i guess i would say so
could you tell us a little bit about how
you kind of got it passed through and
how you got people at nasa to accept
this kind of different way of looking at
things
yeah well i i i don't want to take a you
know a tremendous amount of credit for
that because i think it was a lot of
good timing you know when i had finished
my
my thesis and i came to work here with
ela at that time
nasa had started a very large new
technology development program
and as part of that they were really
looking at a very very broad range
of research areas some of those things
were near term but a lot of them are
very far term
really sort of open-ended hey you know
what might be possible under 10
15 20 years and because of that we had
the freedom to really do something which
was very different from like you were
saying this
this sort of you know mission focused
okay well
let's reduce the risk as much as
possible because we're sending a you
know 2.3 billion dollar rover to mars
like we will in a few months
and because of that we you know had the
freedom to really do more
i guess really fundamental um very very
you know
different uh very risky actually i would
say research
and so what are then you said it took
about a year and a half was how the
project ran and
what happened to it then and what did
you do after uh well the usual sort of
thing i think
um uh that brand new program that i
mentioned um
got cancelled and uh because of that uh
you know nasa ended up you know
shuffling around the various programs i
was doing and uh
ended up creating a whole different set
of research projects which are
much more i think driven towards
supporting near-term missions
uh so we wrapped up the peer-to-peer
human interaction project and
uh uh moved on to other things
can you tell us about some of the things
that you did then yeah so over the past
i would guess i would say maybe five
years we've actually moved
from peer-to-peer human interaction at
least with here at nasa ames
to what i'm now terming robots for human
exploration
and this is the notion that that robots
unlike the ones that we use
on mars right now really can be used to
improve the way that humans do
exploration
it's it's a little bit of a subtle
difference here because if you think
about the way that we use robots today
spirit and opportunity and eventually
curiosity those are robots that have to
do the exploration end to end they've
got to do everything because there are
no humans there
therefore we are extremely conservative
in how we operate them
we don't want to lose them because
they're the only things we can use um in
a place that we probably are not going
to get to
again even with another robot for a very
long time
um and then contrast with that
with the the approach that we've taken
towards human exploration and
it's it's it's interesting if you if you
think about this the last time we really
had a human
explorer and i'm not talking about just
like low earth orbit uh you know
in space around the earth but on another
planetary surface
that was almost 40 years 40 years ago
1972 was uh
when apollo 17 was on the moon um you
know jack schmidt
who's the actually the only scientist to
actually go into space the only
geologist
i mean trained as a scientist that was
in december 1972
it's been a long long time since then
but
we've certainly learned to do a lot more
in terms of how do we use things like
robots whether they are rovers or
landers
or spacecraft orbiters satellites that
kind of thing
and so there's been a big you know
change i think in what you can do
from a technology standpoint since the
last time we had humans on another
planet
and one of the things that we're
interested now is this whole question of
can you use robots to improve the way
that humans do operations
can you improve the things that humans
do
by for example helping robots work
before humans
doing things like scouting uh setting up
equipment
setting up communications relays perhaps
even doing initial survey work
um and then can you have those same
robots perhaps support humans while
they're there in a different manner
you know as uh you know automated
transport or maybe safeguarded
transports you
you have a robot which is working before
but now humans can jump on it
and because it has sensors it can avoid
hitting obstacles while they're driving
so they don't have to worry about that
and then if you take those same robots
and after humans leave do follow-up work
the idea that you can complete tasks
that were started by humans or do tasks
um you know that are complementary um or
supplementary to to what humans are
doing
and for us that's a real big change
because here now the robots aren't
focused on doing the expiration end to
end here they're trying to do those
things which robots are good at
um but which would be you know very
unproductive for humans to do
uh systematic survey is a good example
that a lot of the work that we
we need to do to understand an area
involves making thousands and thousands
of repetitive measurements in a very
structured way
well sending a human to another planet
which is extremely costly
which is extremely risky to go make
thousands of measurements in sort of
these lawn mower patterns is a huge
huge waste of time um and people would
say that you know why would you do that
well you do it if that's the only way
you can you know understand the
the environment but if instead you would
say hey you know this sort of repetitive
thousands and thousands of measurements
in a lawnmower pattern that's perfect
for a robot well
so you should use a robot for that um so
as i was saying looking at the idea of
robots for human exploration is
something i think is very powerful for
nasa and we've
spent a lot of time the past few years
trying to to understand how do you build
robots that work before
in support and then after humans
and um is this is there a view to
implementing this
anytime soon and one of the missions or
well as i said we're driving towards uh
supporting human exploration so of
course the basic question is
you know when are humans going to get
off the planet and actually set foot on
some other planet
uh or an asteroid for example and up
until last year the answer was oh we're
heading towards the moon we think we're
going to be there in the 2020 time frame
um that's changed again uh i think
because of political realities but also
because of just
you know the economy um there's some
things that we're just not able to
afford right now um and so
i don't know um you know when we're
gonna see humans
back on the moon or on the surface of
mars or on an asteroid
um i firmly believe it's gonna be
sometime you know during my lifetime
um but we don't know a specific date um
at the same time however that means that
nasa still is spending a lot of uh
you know effort to actually try to
figure out how do you create robots that
can support humans in those sort of
exploration
you know scenarios so we are continuing
to do a lot of work
um primarily testing here on earth in in
planetary analog
environments so places on earth that
have some characteristics that are
similar
to the moon or mars in terms of terrain
in terms of geology for example
and because of that that's really where
our focus is these days
um and you also became the director and
correct me if director is the wrong term
because i've been corrected before
no direct direction it's okay in jpl
they don't use directors so
um of the robotics group um how did that
happen and what what was the group
like when you kind of took it over and
how how are you
well you've been developing it since
yeah sure as
as i said when i when i came back to
nasa ames i came here because
ela was here running the group and
you know i'd always wanted to work with
him like i said we had known each other
for a long time
but never really worked closely together
other than um
actually it was interesting just before
before coming here we actually had put
together
uh a survey paper human robot
interaction um which is
incredibly dated now and i'm actually
horrified that people still cite that
because it's so
old now um but
but it was the first it was sort of like
the first one out there um
and uh because of that we you know we
really thought hey this is great we
should work together
um so i came here to back to california
came back to nasa ames uh
working for ela and uh six months after
i got here he actually found out that he
got tenure
um at cmu and he made the decision that
well
um bye bye california back to pittsburgh
so he left um so i moved back here to
work with them working within six months
it was actually it was great six months
and we continued working together while
he was in pittsburgh on
the peer-to-peer human robot interaction
project but when he left i i
basically inherited the group from him
the group at that time had i think about
15 people in it
and uh we've really grown over the past
uh
i guess i was about six years ago past
six years where we now have 32 people
um and we do uh research that's that's
pretty broad um some of the work we do
now people would say oh why is that
robotics and
it's been a real you know interesting
journey i think over the past six years
and so what are some of the priorities
for the robotics group
so we are still focused on you know
being able to create
technology that really improves the way
that you can
explore remote environments that's sort
of like our core mission you know trying
to find ways to improve remote
exploration
but that doesn't mean that we have to do
that exploration only with robots that
can drive on a surface that have wheels
and
instruments and have to be really sort
of in situ
but we're looking at how can you really
approach
exploration more globally and part of
that is that that nasa and other
international agencies have
all these spacecraft orbiting around
other planets other planetary bodies
providing lots and lots of lots of
detailed mapping
information and the challenge there is
really how do you take that information
which is
very very large these days i mean you
know petabytes of data
and visualize it in a way that anybody
whether they
are a scientist an educator a student
you know or your grandmother can
actually look at that data because this
is wonderful rich data of places that we
just can't go
as humans but which we care to know a
lot about
so over the past few years we've taken
some of the software that we originally
developed for our robots for navigation
basically looking at say stereo images
and creating these 3d maps that are
useful for navigation
and turning it into software you can use
for building planetary scale maps
so a large part of my group now works on
automated planetary mapping
we take lots of data from from mars
orbiters we've been working with the
highrise imager
on the mars reconnaissance orbiter we've
also worked a number of other data sets
over mars
and over the moon there's the lunar
reconnaissance orbiter right now which
has a camera called the
lunar reconnaissance orbiter camera lroc
we've been processing that as well as
historical
data sets um some of the richest data
some of the best data believe it or not
is actually uh from apollo apollo 15 16
17 had these cameras called the apollo
metric cameras mapping camera
and during the past several years um
there's been a
project led by mark robinson at arizona
state university to scan the original
film
with a photogrametric film scanner so
this is one that can actually scan
things down to film grain level
you know so you end up with these
enormous really enormous images
of film film strips that are perhaps a
meter long and
maybe 20 or 30 centimeters wide
so we take that information we apply
computer vision that we originally used
for our robots to
create these large area mosaics so image
base maps and then
areas where we have say stereo overlap
we create 3d terrain models
and then the thing that's been so
exciting for the past few years
is that our next door neighbor i mean
literally across the fence our next-door
naval
neighbor is google we've actually worked
with them
to create versions of google earth for
the moon and mars
so today for example if you go download
google earth there's a little icon
on the toolbar that looks like saturn
which apparently is the universal
planet at least in google's eyes you
click on that and uh
you can switch from looking at the earth
to actually looking at the moon looking
at mars and i'm just
incredibly proud that just about
everything you see there was developed
by my group
the base maps the the images the tours
which allow you know people of all of
you know from all domains
um you know all areas of interest all
areas of learning to actually look at
some of the data collected by nasa
um to me that's a phenomenally you know
interesting thing
and i know that or if i remember
correctly eli had been working with the
googlers folks
at some point too so were you also
connected with that
yeah so just before eli left he uh
and randy sargent started a project
called global connection
this was really focused on the idea of
how could we use
tools like google earth or systems at
the time
there was only a prototype a gleam in i
think randy sargen's eyes of a system
that would eventually come to known as
gigapan
and that was something that we really
talked with google about the idea that
you could
interactively explore images in a way
that you hadn't been able to do before
that you could use tools like google
earth to explore environments in a way
that you couldn't do before
and that led into a very long you know i
think series of
research projects and development of
different pieces of robot hardware and
software
to sort of realize that mission and now
these days uh you know i've got to say
that ila
and randy you know have created this
truly amazing wonderful thing called
gigapan
um which just really allows you to
really browse and
study uh images in remote environments
actually
in ways you couldn't um before um
it's also interesting for the gigapan
project that the real inspiration that
uh
believe it or not came from mars there
were some uh
there's a camera called the pan cam on
both the spirit and
opportunity and some of the original
pictures that came down from that
that camera system were things that that
randy who was working
here at nasa ames at the time in in the
robotics group
really wanted to be able to just
visualize because he had little bits of
pizza people had to stitch them together
and say photoshop and yet with this
this panorama but it was a static
panorama i mean you could zoom in but it
was hard
so he actually started creating what
became the first version of the gigapan
browser being able to actually work with
panoramic data that came from mars
and that was the inspiration for what
became gigapan
and you mentioned that one of the things
that you're working on really excited
about is
having these visualizations and this
different kind of data and exploration
open to more people yeah
um is any part of that there are a lot
of people who are now looking at how
you could use like crowdsourcing to
actually explore things is that part of
the vision at all or yeah it it
certainly is
um so it's it's nice the way a lot of
things come full circle
um that you you may start off in one
direction and create something new and
then eventually it wraps back around
and comes back so gigapan is a good
example of that
uh you know i started with mars data as
a way of trying to
visualize and allow scientists to browse
these these panoramic data sets
eventually became a robot camera and
actually spin-off company as well
now selling these things commercially
and then last year what we did is we
actually took
some commercial systems we went out and
used gigapans to basically do scouting
ahead of a simulated human mission
and then we used that set of data these
panoramas
we put them online and let the general
public look at these
and took their input to help plan the
simulated
human mission that we're going to do in
an area this happened to be in the
arizona deserts
so here we started out with you know a
robot system that was built on
exploration from data from a robot on
mars eventually led to the creation of a
commercial platform which was used for
us to then
plan a simulated human mission that
we're trying to test on new technology
and techniques for exploration
but then involved the general public and
we had the public involved in you know
doing analysis of these panoramas
we also had them involved in just sort
of voting on hey what's the best place
to go stop
at so we had this this large voting
project online
and based on what the general public did
i mean what they decided is hey this is
the right place to go we use that
incorporated into part of our
traverse planning for that mission and
how did that work out
it worked out really well on two levels
one was that i think it really
uh raised the awareness of some of the
future
you know mission work that nasa is doing
right now trying to plan out for future
for that day in the future when humans
do go back to other planets um
and it was really rewarding to me seeing
some of the comments because you know we
had
we had an online voting system but of
course as a place to also add comments
and you know some of the comments we got
were just phenomenal especially from
from young students saying oh hey you
know
this is why i want to grow up and be a
rocket scientist um
so to me that was really nice to see
that connection to
you know students we also also um you
know had a number of geology students at
the
arizona state university um look at
these images and actually
try to do field geology um you know from
the comfort of
their their uh their classroom so they
use these panoramas to to do some
analysis that we then fed back into the
the planning for the mission so
you know you mentioned crowdsourcing of
course before and of course i think
crowdsourcing can be on different levels
depending on the
level expertise depending what you're
trying to do so one case we're looking
at a very broad public just hey vote on
what you think is most interesting and
it could be
interesting by a number of different
criteria
but then also try to get more detailed
scientific information in this case
people undergrad students for example
who have some geology
but you know still very limited uh
you know field work field experience um
as a way of sort of triaging these very
large data sets and helping us
you know plan better where we should go
um do exploration
so what was chuck thorpe like to work
with as an advisor chuck was great
he was um i was i was so
phenomenally lucky to have him as an
advisor and then actually charles
bauer in switzerland i was absolutely
convinced they were twins separated at
birth
um you know they would they'd probably
be horrified to hear that because they
they they they look so different and
they obviously have such
different backgrounds but they have a
very common approach towards
research they were very very um
you know open to new ideas they were
very you know
willing to give you as much rope as you
want to hang yourself on
but then you know help to actually
support you when things didn't go right
so
they were uh phenomenal they were really
you know the kinds of people that
you know i don't think i could have
actually done a uh
a thesis where i was in switzerland
doing a cmu
phd thesis funded by a startup company
if i hadn't had those two
so that was you know phenomenally lucky
and
what do you see as the the major
challenges facing robotics over the next
five to ten years
well i think that these days you see
robots in a lot of different places they
certainly become
you know much closer to the average
person um a lot of people have robot
vacuum cleaners
you know the roomba is you know hugely
successful the military of course is
using
robots more than ever before and that's
that's both expected
and somewhat disturbing i think too um
especially because
you know whenever you have these sort of
new technologies that kind of
radically change things there's always a
question of well is that technology can
be used appropriately
um it's not to say that you shouldn't
use robots for for military operations
just a question of well
you know how do you do that and what is
the implication of doing that
the same i think could be said about
just any sort of new technology
for space it's a real question of you
know can we use robots to improve the
way that humans do exploration because
that's really i think
the thing that drives most people to to
come work at nasa
um it's true that we've learned a
tremendous amount from using robotic
explorers whether they
are you know rovers or landers or
spacecraft but at the end of the day
there's this
fundamental urge i think for humans to
get onto exploration
and it's clear that if we're going to
have humans spending more and more time
off the surface of the earth
you know how do you do that in a way
that becomes
more productive that's less risky and
perhaps is more cost effective
and clearly you know robots um you know
offer a
you know possibility of actually
addressing all those things
and for young people who might be
interested in a career in robotics what
kind of advice do you have for them
um well you know i'm a software guy
so my advice is you know learn how to
write code early
learn every single language every
platform out there
because so much of robotics these days
is based on software i mean
this is not to say that that creating
new mechanisms isn't important
i'm continually surprised by the the way
robots evolve
mechanically and electronically but to
me the thing that really
you know makes robots different from say
just remotely controlled vehicles
is that they can actually think and they
can be independent
and that's all software so you know
for people who want to get into robotics
it's you know learn everything you can
about software engineering everything
you can about
um you know perhaps ai learn everything
you can
about creating user interfaces
um and that's all based on software
could you tell us a little bit about you
mentioned that you were there
at the beginnings of the hri community
could you tell us a little about who
else was
you know you communicated about that or
there's an eli at the time
what it was like how it's kind of
developed
in the last i don't know 10 years yeah
you know it's
uh to me it's kind of interesting
because um
a lot of people contact me and they say
oh boy you know you were there at the
start of hri
and well how come you're not there now
and
it's not because i have a lack of
interest in that i mean that's really
been you know i think for my thesis work
one of the things that i've been most
interested in this whole
question of how do humans and robots
interact it's just that over the past
few years here at nasa we focused on
like i said this notion of
robots for human exploration and so that
does require some interaction
between humans and robots but it doesn't
have to be you know proximal it doesn't
have to be real time
it's more on the human robot
coordination level
but in terms of human robot interaction
as a community as a domain
you know as a specialty of robotics um
you know
i do feel like i was there you know
certainly in the early days of hri
um there was the creation of the human
robot interaction conference
as sort of like you know the core
conference that focuses on
you know issues of of human robot
interaction and
human robot teaming and human robot
coordination and all that sort of thing
um and when that was first getting
started the people that i think were
most
really involved were people like ela
like mike goodrich at brigham young alan
schultz
at nero research lab there were other
people who were
i think still in school at that time
even like like holly yankel
yeah sorry holly yanko there were
certainly people um that you still see
involved today
um uh maya matarik was was certainly a
key person there and then there were
other people uh in government labs like
gene schultz
at the time who was just getting towards
the end of her career
uh being really interested in trying to
figure out well if we're gonna have
human right interaction how do we
measure that
how do we assess um at some level in
some way
the way that humans and robots interact
and that to me was something
that i think really led to the creation
of the human robot interaction
conference which has really i think
spurred
um you know this sort of growth of human
interaction as a subdomain of robotics
and how do you think hr does hri feed
into
um the rest of robotics or is there
still kind of a
division there what are some of the
challenges to having those communities
well you know it's it's the case the the
robotics by definition is so
interdisciplinary multi-disciplinary um
i've even heard the word
transdisciplinary recently
i'm not quite sure what that really
means um and it's basically
you know it's just something that to me
is is inherently broad
um and so it's a question of really you
know
is there a boundary uh around robotics
and in some sense this is the same
question that people asked years ago
about artificial intelligence
oh because clearly oh everything that
you in robotics that's artificial
intelligence well not everything you do
is robotics and artificial artificial
intelligence is little tiny piece
so um it's hard for me to really try to
wrap my arms around that and say yeah
there's
this is within robotics and this is not
um we were talking about hri but um
i don't know if you wanted to say
anything else about yeah i guess the
only thing i would say about
about hri is that because it is so
multi-disciplinary
um i think it has the effect of pulling
others into the robotics domain or at
least increasing awareness about
robotics
people uh from design for example who
may not have even thought about robots
before people interested in ethics
for example um just looking at you know
how do these things that we've now
created who that are more or less
autonomous or be
semi-autonomous how do they interact
with us and to me
perhaps the biggest contribution of hri
is the fact that it is pulling more and
more
people and more more domains into
robotics in general
and that's a good thing because that
means that robotics touches other things
and can learn from other things and can
give back to other areas
and to me you know whether or not hri
exists within you know this bubble
called robotics or something that bumps
into it from various places
that's actually irrelevant the fact that
it's there the fact that people care
about it and actually learning about
things together
is perhaps the most important thing
great
thank you anything else i don't have
anything
unless there's anything you'd like to
add or something
no i don't think so great robot stories
yeah great robot stories
yeah um i don't know i think uh
i just been uh you know i think really
fortunate in uh you know having worked a
lot of great people in robotics people
who really care about uh
you know trying to create these things
which you can go out and actually do
things meaningful
um and you know certainly uh
if uh if you haven't yet talked to them
um you know i'm sure that you will uh
people like chuck thorpe and uh
and reid and red and and others uh you
know
they are the people that really you know
carry the torch and really
create this great thing to happen called
robotics
thank you sure