okay i was born in saskatchewan in uh
town of north battleford in canada um
it's the oldest of five boys uh so i
you know did my education up through
bachelor's degree in saskatchewan
and i went to university of waterloo for
a master's in computer science
and then carnegie mellon for a phd in
computer science
and how did you get interested in
computer science
actually when i was in grade 12 i took
the grade 11 level computer science
class
and up until that time i was interested
in chemistry and i just enjoyed
programming i found it
a fun challenge decided to try that in
university and
never looked back uh who did you work
with uh could you
get over a little bit who did you work
with at crowning uml
i started with um hans morrowback
but he was not in the computer science
department per se he was in the robotics
institute
so i needed to have a computer science
professor as a co-advisor which was
takeo canadi
uh and then along the way i switched
from has morovec to steve shaffer but
still with takeo as a co-advisor
so in fact takeo was really my main
mentor all the way through
and what was your first sort of project
with them
as a graduate student i was working in
hans morvex mobile robot lab on
stereo vision for autonomous navigation
of ground robots
so there was a problem of trying to
estimate the motion of the robot
as it drove the environment by using the
imagery to track features in the scene
and so i made improvements to that
algorithm and
at the time i didn't actually believe
that that was the most important thing
to be doing but it's turned out
30 years later that that's now called
visual odometry
and it's actually very valuable and very
widely used and
many people have worked on the problem
since how did you get interested in
vision actually as an undergrad i got
interested in artificial intelligence
and then for my master's degree i ended
up
studying computer graphics i still
wanted to do
artificial intelligence and i thought
the best places to do that were
the big schools in the us that were
doing that so that's how i ended up at
carnegie mellon
uh in my first year at cmu
[Music]
i got a little disillusioned with
artificial intelligence um
but computer vision still had some of
the same appeal
but it had also some of the appeal that
i'd found in computer graphics
so i tried that and in particular uh
ponzo was working on computer vision for
robots so
that was exciting uh and i enjoyed it so
i
that appealed to me because it had some
of the same
visual gratification you get from
computer graphics but it had a strong
theoretical foundation
which i was looking for as well okay um
pardon me what about ai disillusion
well um at the time you know i don't
want to be unfair to the field but
at the time ai was kind of a collection
of problems that we didn't know how to
solve yet and
once we knew how to solve them they were
no longer considered ai and
the very fact that we didn't know how to
solve the meant we didn't have a good
theoretical foundation in some cases at
least not a strong mathematical
foundation
and so at least the particular things i
had been looking at at the time
i was not happy with the degree of
theoretical basis to them
so in computer vision it's really
founded in
strongly in physics and mathematics and
uh
you know much of the curriculum you
would find in electrical engineering for
example
uh it's it's pretty cross-disciplinary
in that respect but it draws on
on engineering and applied math
disciplines quite a bit
so what year did you get to carnegie
mellon and what kind of
systems were you working with as far as
the vision
i got there in the fall of 1981 and i
started working on vision in the fall
of 1982. um
when you say what kind of systems like
what was the technology like that you
were working with
well as a as a robot to do the vision
work with um we had something that
looked like a little tricycle
about the size of a tricycle and we were
working with
computers that uh don't remember exactly
what we had then but i think the vax
1170 might have been about the right
time frame
and they didn't have a lot of memory so
they've been a lot of work you know in
in
years just before then in writing vision
software packages that could
swap images in and out of memory so that
you could actually process something in
reasonable amounts of time
so things were a lot slower there were
central mainframes the pdp-10 that the
department had that
i don't think we did much of the
research on but you know everybody had
terminals we didn't have you know your
own desktop computer in those days
the kinds of cameras that we use this
was before
solid state cameras really so i think
viticom's basically a much lower quality
imaging device than we have now
you know in i work on vision for
robotics so it's a multi-sensory problem
so you're combining
the vision sensors with inertial sensors
in those days inertial sensors were big
and heavy and very expensive so you
basically didn't use
anything but the cameras
that's kind of a quick overview i guess
and uh where did you go then after that
after cairo and i came here so i came
here in 89 i've been here ever since
and uh i did a lot of young reeves when
i graduated uh
12 to 14 um including
university schedule labs and companies
and
what i was enjoying at cmu was robotics
for outdoor applications
so i was looking for a place to do that
and this was basically the best
opportunity to do that
i wasn't necessarily interested in
in space per se um
but once you get here it's nice to have
a ringside seat on the space program
and it's certainly exciting to work on
space applications and so
it so happened that i dropped into a
perfect niche for me
the research i'd been doing was exactly
what was needed for mars rovers at the
time and it's gone to mars since
so it's really panned out well in that
respect i'm curious since you mentioned
you were in field robotics at the time
at cmu did they have a field robotics
lab
or what were the what kinds of labs do
they have
and particularly so i so i was in the
computer science department
i think the robotics institute was
founded uh
a year or two before i got there 79 or
80.
um there were some smaller labs so
hans murray was running the mobile robot
lab
this was before the field robotics
center before
that the the term field robotics had
been popularized
and red whitaker who started the field
robotics
center and
i think there's two institutions there
but anyway
he was still a professor of civil
engineering and
i think while i was there was when the
three-mile island incident happened
and that was a big thing that propelled
him along in robotics
and the darpa autonomous land vehicle
program happened while i was there and i
think red was involved in
building up test bed vehicles for that
so
basically um red grew
in in mobile robotics while i was there
and i think frc got set up while i was
there
uh toward the end of my time there maybe
just after i can't remember now
so then is that when the term started to
become popular
uh it probably started then i think it
grew
probably through the 90s as uh you know
red
and others were addressing uh field type
applications
in uh space military and mining
agriculture what were the big challenges
for field robotics as opposed to
just mobile robotics well there's
basically a contrast of people working
indoors um on level floors
where um there's there's only two types
of terrain there's smooth level floor
and then there's discrete obstacles
you can represent the world in two
dimensions and do all the path planning
in two dimensions
uh the weather's always good the
lighting is always good
whereas in field applications you've got
uneven terrain
you've got ground cover you've got
different soil types that
some of them are drivable some of them
are not you've got
a variety of lighting conditions you've
got a variety of weather conditions
you've got dust that can
follow the sensors or you can't see
through
you have to represent the world at least
in two and a half d
if not in 3d uh you have to make
inferences about physical properties of
the world that you don't have to bother
with indoors
it's a more complex problem um
it's more expensive to work in that
domain you know typically uh
you need a bigger robot you need just a
lot more
logistics and infrastructure to to do
field applications so
um it's not studied as much sure
fewer places have the wherewithal to do
that
and how do those play out in your first
projects at jpl
uh when i got here uh nasa and jpl were
working on
a program called mars rover sample
return
so the acronym was mrsr the vision was
to do
a sample return for mars they had a fair
bit of money
actually cmu and jpl were funded under
that program so jpl was working on
wheeled robots
with stereo vision we had quite a big
robot that was
not exactly sure of the size but it was
probably 15 to 20 feet long and
to the top of its camera mast it was
probably 10 feet high
and carnegie mellon was working on
alternatives basically a legged vehicle
using laser rangefinders so we had quite
a bit of money in a big team to do field
work
and the field work that we did was
there's a dry river river wash next to
jpl called the royal seco
so we would load up the vehicle on the
trailer and go out there and do test
runs
so i don't know what the budgets were
but uh
that takes a lot of financing to run
that kind of experiment
compared to working indoors you know you
buy a little robot
two feet in diameter and you know work
with one or two grad students and that's
all you need
so uh you know very much a difference in
scale between what you could do in a
university or at least most universities
and what you could do at a federal lab
or you know one or two companies
like martin marietta that were working
on that at the time
what were the challenges of the vision
system for the sample return
uh the algorithms were very very slow so
um you had to have 3d perception
since uh that's how you could uh
you know determine the where the terrain
was was smooth enough to drive and where
the obstacles were
um and
so we were working in that with stereo
vision and cme was working on that with
laser rangefinders the stereo algorithms
were very slow
uh so one of my first big breakthroughs
was
designing a new stereo vision algorithm
that had much better computational
complexity
and basically reduce the run time from
half an hour per image to
six seconds per image so that was a huge
breakthrough at the time
in feasibility of outdoor robot
navigation
did you work a lot with cmu still at the
time
uh not directly you know we were funded
by the same sponsor
working on the same kind of mission but
we were working independently uh
since then i've had collaborations you
know when the 22 years i've been here
had collaborations with cmu people a
number of times but uh when i first got
here
it was not not directly collaborative
what were some of the later
collaborations
um i may not get these in
in the right order but or at least
chronological chronological order but
um cmu managed to get funded by nasa
to set up an organization they called at
the time the i think it was the
nasa robotics engineering consortium
it's now called the
national robotics engineering consortium
and so that was a
vehicle to try to
channel nasa-funded research into
commercial applications
and there was a mechanism whereby you
could propose
joint projects you know between say us
cmu and a company that would try to take
a piece of nasa technology and
expand it into commercialization so i
worked with sanjiv singh on a few things
on that
and then along the way
the army research lab has funded a
program called the robotics
collaborative technology alliance
and so i've collaborated a little bit
with martial aber
along the way on that
al kelly was a grad student
[Music]
let me think darpa had a program called
demo2
so we and cmu
were part of the demo 2 program working
on different parts of the problem and
so we collaborated a little bit there
and i hired alkali for one or two
summers as a student
to bring his path planning capabilities
and merge it with our vision
capabilities here
and after that that was taken to uh i
don't remember it was still locking my
steel martin marietta then or lockheed
martin but taken to denver to put on the
robots for a demo too
we've worked some with tony stanz who
does a lot of path planning work as well
i'm probably forgetting people but
those for me personally have been the
most direct
and the most substantial collaborations
so we heard a little bit of yesterday
about the
the sample return sort of getting
cancelled and then evolving into
the the other mars rovers the next
generation
so were you working on those and what
any of those systems
that you worked on according well so the
mars river sample return program uh i
was working on when i first got here
um and i don't remember exactly when
that got cancelled but it was
fairly shortly after i got here you know
within the first two or three years it
was simply unaffordable
and then the pendulum swung to the other
extreme of looking at micro robots
and that was far far less capable but
also
far far cheaper and so we managed to
sell this mission called mars pathfinder
which was the first lander that
that nasa jpl did with airbags
and it was you know every mission has to
have some science but
it was as much i think a technology test
and demonstration mission as it was a
science mission
and the same was true with the rover so
the rover was a technology test and
demonstration
it had to have some science you know it
was fairly minimal science
and it was cheap i think it was 25
million dollars just to do the rover
so i was involved in that
a little bit i didn't work directly as
part of the mission team but
um it had a sensor called a light
striper for detecting obstacles
that was having some performance
problems and just coincidentally in
parallel i was working on a next
generation
light striper and so i got involved in
uh
essentially debugging what was wrong
with the one that was headed for mars so
there turned out to be a problem with
the optical design that
that i was able to isolate and then we
got an optical engineer to come in and
redesign it
so can you tell us a little bit about
what a micro robot is like compared to
the previous idea
well um it's behind me i suppose we
don't want to turn the camera but um
uh so the uh mars pathfinder rover which
was named sojourner
weighed about 11 kilograms i'm not sure
if that's the exact number
uh the rovers that had been looked at
for mars over sample return were
probably
uh you know close to a ton
i don't know the exact number but you
know we're talking
two orders of magnitude different in in
weight
uh and uh whereas the
sojourner rover never went more than six
or eight or ten meters
from the lander the original concept for
the sample return rover was to go
i don't even know you know hundreds of
kilometers um
and be a self-contained mission whereas
sojourner was dependent on the lander
and the cameras on the lander and the
communication system on the lander
to be told what to do and to communicate
with earth
but nevertheless that mission uh
did firmly convince the whole mars
science community of the importance of
mobility
so essentially the vision of how to
explore mars was never the same
after pathfinder because now everybody
realized you've got to have mobility
because
there can be stuff just out of reach
from the lander that
is is where the big discoveries are
uh and what we found in the mars
exploration rover mission which
those two rovers have covered uh
i don't remember off the cuff but you
know it's um
20 order of 20 kilometers in
in seven years um you know we're several
kilometers away from where they landed
and that's where the big discoveries
happen so
um i guess i'm straying from the
original question but
um you know the difference between the
micro rovers at
10 10 to 11 kilograms and and staying
with inside of the lander versus sample
return at
let's say uh uh well
the uh mars science lab mission that
we're working on now the next rover
um i think it weighs 900 kilograms
um and it's supposed to drive in order
of 20 kilometers
um that's the difference
and were you involved in the mars
explorer program
uh the mars exploration rovers uh yeah
so my whole career
i've been in research so i've never
worked on the flight software or been
formally part of the mission team
but key elements of
you know the capabilities in that
mission came from either me personally
or my group
so those rovers use stereo vision to
detect obstacles
that basically is the outcome of work i
did in my first year here
they use visual odometry to help keep
track of where they are
and more importantly to detect when
they're slipping
that basically grew out of my phd thesis
research
the path planning software that's on
there
i didn't have any personal hand in
developing but that was done by somebody
in my group
and uh i basically convinced him that he
should as a career move
get involved in the mission uh
later on there were some things that
were added uh after they
already landed as kind of mid-mission
software uploads
so there's software on board that does
basic image processing to detect when a
dust double goes by
uh that software was developed by
somebody in my group
um that mission halfway through
developing the spacecraft you know like
uh two years before launch
they realized that they had
underestimated what the
wind velocities might be and there was a
concern that the horizontal velocity of
the lander might be
so large that the airbags might rip on
on impact so
you know some people came to us turned
out that they were retro rockets on
board
that could be used to reduce the
horizontal velocity
if you knew what that velocity was and
so then the problem became can we get
a velocity sensor and the traditional
way to do that is with a doppler radar
but there wasn't time or money to put
that into the mission at that point
but coincidentally they had designed in
a camera that they weren't using and so
the question was could they put that
camera back in
uh and aim it down it was originally
designed to be on the rover looking up
to
to track the sun but so could we aim it
down instead of up
track features on the surface during the
sound and estimate horizontal velocity
so we mounted a
you know a crash effort pun intended
to to try to develop that capability and
that was successful and that was used
for both landings and
uh for the spirit rover um from the
after the fact reconstruction it may
well have saved that mission because it
did lead to the retro rockets being
fired and it reduced the velocity by
about a
factor of two so so for the mirror
mission
um uh
i also had a hand in helping
keep track of where it is besides this
visual odometry thing that
operates kind of over short distances
i work with a photogrammetrist named ron
lee at ohio state university
on mapping and so there's bundle
adjustment software that ron's group has
developed
that is used to keep track of the the
multi-kilometer traverse
and is part of a whole mapping package
that ron's group has developed to
generate terrain maps as as the rover
goes along
i think that's pretty much what i did in
that mission
basically uh most of the autonomous
navigation capabilities in that mission
came out of my group what were some of
the
most important breakthroughs that you
feel you've had
well um
real-time stereo vision so um
in the 1980s
the dominant paradigm was to try to do
stereo vision by matching
edges because that was thought to be
much faster
and when i got here i could see that for
off-road navigation
you know in desert like terrains edges
was just the wrong way to do it
so you really needed to use what was the
alternative paradigm which was
area-based
cross-correlation
so finding an algorithm that was fast
enough for real time
was something i did and that was a key
breakthrough
this whole visual odometry business was
something actually i did at
cmu before i left there
[Music]
i think that was not not at the time so
i developed the first accurate
uh visual dominance algorithm even
though we weren't calling it visual
donald trump
and um you know at the time you know
that was a
good advance over what had been done
previously but it wasn't that important
i think you know over the years since
then we've recognized it is a very
important capability
um
[Music]
along with some collaborators so my
advisor takeo and rick zieliski we did
a well-known paper that was one of the
first to
to show how to process images in an
incremental fashion to build 3d models
so you know much better techniques of
course have
emerged but that was one of the first to
to show that you could incrementally
build 3d models from image sequences
[Music]
i've been well the thing i just
mentioned about
vision systems for mars landers so
terrain relative velocity estimation
during landing um
when i got here in the mid 90s i managed
to get funding to
work on landing hazard detection so
essentially i built a team
uh that has been working on various
approaches to landing hazard detection
with imagery and radars
none of that has has gone to mars yet
but um
over the years we've built up quite a
bit of capability to be able to do that
and i believe that someday that will
go to mars precision landing has been
another
important thing especially for mars and
so
more or less the same team that i built
up here has been working on
how do you recognize landmarks during
descent
so that you can reduce the position of
uncertainty
from say plus or minus 20 kilometers to
plus or minus 100 meters
and that would make you know far more of
the planet
accessible uh and if we could land
you know 100 meters from interesting
terrain instead of
you know 20 meters some interesting
terrain or 20 kilometers some
interesting terrain
um you know now you can get there in a
day instead of getting there in months
other things that i've worked on for
space application include
to explore asteroids that have a lot of
craters we've developed algorithms that
can use the craters to
navigate a spacecraft around asteroids
uh jpl's been working on balloon
missions for
titan which is the big moon of saturn
that has a thick atmosphere
this question of how do you know where
the balloon is so we've been working on
on you know taking algorithms that
fundamental algorithms that were
developed by others and applying them in
this domain
so that you could take imagery from an
ordinary camera on board
and register that to radar data taken
from an orbiter
so that kind of cross-registration of
different sensor modalities
can uh tell us where the balloon is
and then so that's that's the nasa side
i've been working on
uh defense robotics for you know more
than 20 years and at grad school most of
my work was funded by the defense
department
so um you know as a funding strategy
i've tried to
to find things to work on that had
relevance to both nasa
and other agencies so off-road
navigation
fits so the focus on
terrain understanding so even while i
was at grad school i collaborated with
hans moravec and alberto welfes
on um they had a fundamental
breakthrough in an algorithm called
occupancy grids
for um world model representation and
alberto and i wrote the first paper on
on combining data from multiple sensors
in occupancy grids so that was the
sonoran stereo
and then with funding from darpa and the
army research lab i
was the first person to use
near-infrared imagery for
terrain classification and vegetation
for off-road navigation
the first person to recognize that you
could use
radar to classify tall vegetation which
can be difficult to discriminate and you
can drive through it
but it can be difficult to discriminate
that from a real obstacle that you have
to go around
there's something that we call negative
obstacles in this business so basically
any hole in the ground they're they're
very hard to see they're still very hard
to see you know the
the only good way that's ever been
discovered to find those is
from the air but uh you can't always
send an aircraft ahead
of you so along the way i discovered
that um there's uh just kind of a
physical property
in uh the way heat transfer works that
the interior of a negative obstacle
tends to keep itself warm
that has not completely solved the
problem but it was an interesting
advance that makes it somewhat easier to
detect those than it would be otherwise
in looking for things that i could work
on that other people weren't doing
[Music]
water bodies in off-road navigation that
are hazards to navigation
we've myself and collaborators here put
a lot of effort into how do you detect
water bodies from a ground robot and
you know we've made a lot of progress
towards solving that problem so that's
you know it doesn't have wide impact but
uh it's something
fairly distinctive that we've done
um
[Music]
i think that's a fair selection of
highlights
has optic flow concept that you've
used in some of your vision visual
events certainly have used it
so um what we do in visual odometry
tracking features through the image
sequence is
you know there are many different
flavors of optical flows so that's a
flavor of optical flow
uh the collaboration i had with the chao
canadi and rick silski was essentially
using a flavor of optical flow
in in my work on ground robotics
i focused on on stereo vision for 3d
perception and motion estimation whereas
there have been other people in the
community that have tried to use a
single camera
and so the problem with a single camera
when you're driving forward is
you don't have depth perception directly
in front of you because you have no
visual parallax
so optical flow per se has not been as
important part of my work
because of that but in the last few
years i've started to work on
microair vehicles under funding from the
army
which have very important applications
in military reconnaissance and you know
applications in
the civil arena and inspecting bridges
and so forth
but it's also very relevant to things
for nasa so if you're trying to explore
a comet or an asteroid
or even fly a balloon in the atmosphere
of titan or venus
you know they're a very different scale
of a machine
but the vision problems have a lot in
common or if you want to have a micro
inspector like a little ball
you know a foot or two in diameter on uh
on a human spacecraft like the space
station or if in apollo 13 if we had a
micro inspector that could have gone
outside and had a look
that's a lot has a lot in common with a
microarray vehicle for
for earth applications so there you
can't really fit a stereo system
and expect to see very far so optical
flow is quite important in that
in those domains so we're using it we
haven't
developed new approaches to optical flow
but we're using it for perception for
those kinds of
systems so for these various like
near-infrared and lidar and
vegetation detection uh what were the
robots that you put these systems
on and demonstrations
uh some of that work was done under the
darpa demo 2 program
which was using humvees that were turned
into robots
then there was the darpa tactical mobile
robotics program
which um essentially that was before
irobot became big and successful
and they developed a little tracked
vehicle
which initially was called the urban
robot and later became the pack bot
so that program was the the genesis
of their packbot product line and as you
probably know those are now
used in the middle east for bomb
disposal um
so we put sensors including the
well-known sick radar
single access scanning laser rangefinder
on pacquat and use that to
classify vegetation
the near infrared was on the humvee
the thermal infrared uh that came out of
another program
called preceptor where we were basically
using all terrain vehicles that were
outfitted as robots
uh you know the quad all-terrain
vehicles
um and uh general dynamics robotic
systems has been a big player in
in army robotics uh they built some
custom vehicles that are
i think around three thousand pounds
they call them experimental unvent
vehicles
so our work has been integrated on those
vehicles
you know sort of volkswagen size vehicle
um what else
i think those are the highlights of that
can you tell us a little bit about
the organization of the groups that
you've worked in and how they've changed
their time
how different groups interact with each
other within nasa
what did it look like when you got here
okay
uh well i was originally in a robotic
vehicles group
run by brian wilcox um
[Music]
somewhere along the way the organization
decided it was time to have a vision
group
and i became the leader of the vision
group and when that was first created
actually there were a small number of us
working in vision
which they merged with a slightly larger
number of people working on star
trackers
so star trackers are used in
interplanetary crews to keep track of
the orientation of the spacecraft
so i really was not interested in star
trackers
and felt that spending a lot of time on
them was going to be detrimental to the
stuff that i was interested in so so i
didn't really take care of it very well
and eventually management recognized
that and moved that off somewhere else
and left me with just the vision part
um and that grew from you know
originally a half dozen people to as
many as 18
and then we you know had succession of
reorganizations and the group would get
smaller and then it would get bigger
again
and um along the way we
uh we recognized that from you know
strictly a
funding strategy perspective there's
actually a lot more money in visual
surveillance than there is in in
robotics or at least vision for robotics
so
we thought we should participate in that
somewhat so we started to
get a toehold in airborne surveillance
work
and then there was another
reorganization where we formed a
separate group that
that is carrying on that work and i'm
focusing
again on on vision for autonomous
navigation
i have uh uh avoided
uh moving further up in management
because i didn't want to be purely
managerial
um so if the question is
you know how has the organization
evolved within jpl
that's it and just within the last year
year and a half
you know i felt that it was important to
try to get
more visibility with top management here
and more appreciation by top management
of the
importance of unmanned systems outside
of nasa and
therefore it should be important to
address those applications within jpl
uh so i've taken a sort of a day a week
level of responsibility to try to
provide some
strategic coordination
of robotics for non-nasa applications
across jpl
you mentioned earlier that
with some of these missions like the
idea that mobility is important
became clear what's been this what has
been the status of
uh of robotics sort of um unmanned
vehicles within
nasa has it changed through time
well uh i think nasa's been interested
in
in surface mobility since uh since the
60s
with the lunar surveyor program
so there's always been the interest
there but i
i think it wasn't until the the mars
pathfinder mission
that uh that it was really appreciated
as
as essential uh and also that it was
recognized that it was feasible
so i think there was a watershed there
around 1997
[Music]
and then
with the mars exploration rover mission
we've shown that it's not only feasible
for short-range motion it's feasible for
long-range motion and that it's
crucial to find rock-out crops that are
many kilometers away from where you
landed
but you know as the ebb and flow of
research funding goes it's been a little
bit of a hand-to-mouth existence because
um nasa there's been swings of the
pendulum within nasa where
for periods of time there is very little
research funding per se
you know there's very focused money on
specific things needed for the next
mission
but that's another reason why it's been
essential to be
you know having other sponsors at the
same time in order to
manage the ebbs and flows of research
dollars within nasa so to maintain
you know a steady cadre of
of people working on these problems so
that when the opportunities came back
within nasa we were still here and still
equipped to address them
so how does the funding work exactly how
much of it are
does the group have to kind of create
for itself
or most of the funding that we get we
have to raise ourselves
so there is some funding that comes
through what's called focused technology
programs that's
you know assigned to jpl to solve
specific problems for specific missions
that will
kind of trickle down through management
to us but
at the you know more basic and applied
research level
almost all the funding that we get is
competitively ordered
so i liken it to you know in
universities there's a
concept of a soft money position which
is non-tenure track
funded by research contracts it's very
analogous to that
so the size of the group also in a sense
depends on the ability
to raise money right yes
and how much does the funding shape what
you decide to do research
is how much is driven by your interest
well they go hand in hand and you know
you have to be pragmatic and realize
that if you have to have
a charge number you know and a contract
behind that charge number
uh you have to work on things that
people will fund
but that's a bit of a chicken and egg
thing if you bring an interesting new
idea
to a sponsor then sometimes they will
apply some funding to it
so you need to sort of be working uh
under the radar
on things that you believe in even if
you know the country is not ready to
apply funding to them
and there are there are ways to get
funding for that at you know a small
level
so jpl has some internal seed funding
pots you can apply to
um you can collaborate with people
elsewhere so there could be different
agencies like the national science
foundation
that fund basic research that maybe nasa
or
the defense department aren't ready to
fund but if you collaborate with those
people you can kind of get the ball
rolling
and then move those ideas into other
agencies
and have you had much interaction with
companies as well
yeah so the
robotics collaborative technology
alliance that the army's been funding
we've always been
part of a team led by companies so
general dynamics the darpa demo2 program
was led by martin marietta we were uh
i don't remember if our money came
through then i think our our money then
came directly from darpa but
we were basically working on a team with
martin marietta
um when we were working with cmu
and the nasa robotics engineering
consortium we worked a little bit with
some companies so
toro was interested in automation for
some of their
you know lawnmower products
we've had some of our vision software
licensed by small businesses so
no small businesses that have made it
big yet that we've had some of our
software licensed
i actually collaborated with people here
a little bit
who pioneered the development of cmos
imagers
so the cameras that are in all the cell
phones these days some of the
pioneering work on that technology was
done here so i was part of
a patent for um
on-chip block averaging for
multi-resolution readout
which was part of a package of patents
that were licensed
that particular capability hasn't had
much impact in the market yet but you
can buy it
let me think we've collaborated with
other companies
in other dod funded programs so we're
doing work
for the navy now where we're a
subcontractor to other companies doing
unmanned votes for the navy
there's probably others that i'm not
remembering so irobot you know we
collaborated with irobot in some
programs
we're collaborating a little bit with
irobot's connect competitor kinetic
in the um robotics technology alliance
these days kinetics part of that team
we work for boston dynamics on some of
the darpa programs for
legged squad support system basically
for legged robots we provide
vision systems for legged robots for
boston dynamics
so what do you see as the the big
outstanding problems facing
field robotics over the next few years
well one of the things that we've been
uh working on which there's been
progress i don't think it's done
but um for
many years all of robotics research was
done in in static environments
so the problem was hard enough when
nothing else was moving
that you didn't want to have anything
moving in the environment because we
just didn't have the perception
capability to
understand that yet or the the path
planning capability to reason
about things moving in the environment
and a lot of that was just driven by
limitations and computational power so
we've been working on
the perception and representational and
planning capabilities to
operate in dynamic environments
for mars there is still an issue of
understanding the difference between
hard soil and soft soil so
the rovers have gotten stuck in soft
soil and
there are problems yet to be solved in
in detecting that before it happens and
making sure we get out of it
back on earth um
you know in the mainstream computer
vision community
uh there's been a lot of work on object
recognition you know that's a
dominant theme in computer vision these
days um
that will play into uh
mobile robotics it hasn't been as
significant yet but it's going to become
important as we get the
more basic problems of of safe mobility
solved then you need to do more purpose
of mobility and that means
you've got to understand things around
you as more than just 3d structures
you've got to understand what they are
and what they're for
so even just you know going through a
door
helps to understand what doors are in a
semantic sense
not just in a geometric sense
and there's a growing trend to
work on mobile manipulation so robots
with arms
so now you're combining all of the
problems that come with manipulation
including object recognition
with the problems of mobility so
you know if you have arms on a mobile
base now you have to reason about the
degrees of freedom of the arms and the
mobile base
let's see
i think that's fairly good what do you
think is the
future of robotics and nasa in nasa
um
[Music]
well the holy grail for mars is a sample
return so rovers will be part of that
uh right now for mars we're limited to
terrain that you can go to with a
wheeled vehicle
um but some of the more interesting
science may be in terrain that you can't
access that way
so we're looking at robot systems that
can say repel down cliffs
nasa would like to to enable sample
return from comets because comets
may you know be where some of the
chemistry arose that gave rise to life
and you know an unmanned spacecraft
is not the same as robots that you see
you know running around on earth but
they have
many of the same requirements for
autonomy
so free-flying spacecraft that can
operate in close proximity to other
objects whether those objects are
planetary bodies like comets
asteroids or moons
or other spacecraft so like a micro
inspector
and then long term for nasa
i i believe that there will be value to
robots for satellite servicing so this
has been a debate
you know in the commercial world is that
really economically viable
and uh just within the last six months i
think uh there was announcement that for
the first time
a big satellite operator has awarded a
contract in this case to a canadian
company
to do satellite servicing but what's
made that
economically viable is that this is for
communications and
and intelsat has many satellites
uh essentially in geostationary orbits
on the equator
and so you can go and refuel those with
one mission that just kind of drifts
along the equator servicing many in one
mission
so that becomes an economically viable
business model
but for nasa so nasa spent you know
five billion dollars or more on a
mission called the james webb
space telescope which is going to be on
the far side of the moon
and it is not designed to be serviceable
and i
i really know very little about the
mission but i have to wonder
if we could do robot servicing
could we do a better job of greater
great observatories in space
and perhaps make them more affordable
because they don't have to be so
fail-safe
up front you know servicing missions
aren't cheap
but you know if you can spend 500
million on the servicing mission as
opposed to
you know throw another one and a half
billion in to greater reliability so you
don't have to service maybe it's a good
idea
and we've seen that with the hubble
right we extended the life of the hubble
many times
through servicing missions how do you
see
the um not the competition but
they're of course autonomous robots and
there's also a lot of tele-robotics
at nasa have you worked on any of the
tele-robotics then
how do you see those two working
together against each other
um i haven't had much involvement in
tele robotics
you know there was a lot of work on that
when i first got here but i chose to
work on rovers
i don't see that as a competition it's a
matter of an evolution
and so you do telerobotics
where it's feasible and where autonomy
is not yet feasible
and you do autonomy where that's the
only possible way to do it
and therefore you've invented a way to
do it so you can do telerobotics in
earth orbit
or potentially on the moon because
communic basically what makes
telerobotics possible
is relatively low communication delays
and where it must be autonomous
is where the communication delays are
too long so
potentially mars and anything beyond the
moon has to be autonomous
and then you know there are analogies on
earth the same thing goes so
you know applications that you would do
telerobotically on earth
you could think of surgery as a
telerobotic kind of application where
you wouldn't dream of not having a
surgeon involved in today's
with today's technology um
but uh at the same time you'd much
rather send a robot into fukushima
than have to send a person and it's sort
of unfortunate that we aren't ready
as a society to have robots that could
be sent in there
to do that job but i think you know
that's a
a rare occurrence that that disasters
like that happen
and therefore the even though
technically i think it's feasible the
investment had been made
you know prior to that incident to have
the robots ready
to do more so they've certainly taken
robots in there but we could do a lot
more
the capability exists to do more
it's just not mature to the point that
it's quite deployable yet
and i noticed that you were also on the
editorial board of field robotics
the journal the journal of field
robotics yeah autonomous robots
yeah wondering how long you've been on
those boards and how you've seen the
field kind of change while you were
there
yeah um
i've been on the editorial board for
autonomous robot i can't remember how
long i've been on either of those
editorial boards i've been on autonomous
robots longer
um um
i haven't been the most active editorial
board member
um you know for uh autonomous robots i
review papers occasionally
for jfr
i've been a
um guest editor for special issues
occasionally particularly in space
robotics
um so how has the field evolved um
[Music]
since i can't remember exactly when i
started those roles i don't really have
a good answer for that
it can be more broad than just from the
journal itself
well um
we have examples of things that have
actually been fielded so you know we've
been talking a lot about
space you know prior to 1997 there had
never been
uh an autonomous planetary rover mission
and so since then we've had substantial
missions for that uh on earth um
things like the pacquat and the the
talon from kinetic
are the first deployed military robotic
systems
um
since i really haven't been that
involved in
commercial applications i can't say as
much about that side but i think there
are
applications in agriculture and mining
and um
material movement in uh shipping ports
and things like that where we're seeing
autonomous vehicles um you know we've
been seeing a lot of work in the last
decade or more
so the whole robot road following thing
which you know got a big start in the
darpa alv program
in the mid to late 80s
you know that's gone from something that
was just barely possible
in a very big very expensive research
vehicle to something that actually works
pretty well
and is you know doable
in an automobile now and
now we have cars that can park
themselves
and so this has been driven by advances
in sensing
and especially by advances in computing
and by reducing the cost of those things
a big driver that i believe we're going
to see in the future is cell phones
so you know small cameras for cell
phones
which lead to now you've got images in
cell phones you need to be able to
process those images so now we're
getting
very substantial image processing
capabilities
in cell phones uh the next thing that's
going to happen is now you've got two
cameras in cell phones so you can do
have a camera and do you know skype on
the cell phone
um but uh there are also
uh things coming out so you can have two
cameras facing the same direction
to aid with teleconferencing so that's
that's a stereo system that you can use
for a robot
um and because it's a cell phone it's
not only going to have a lot of
processing power it's got to be very
uh low power dissipation so
i think we're going to see processes
develop for cell phones eventually
go into small in fact not eventually
it's already happening
go into small mobile robots both on the
ground and microwave vehicles
so robots robotics by itself is too
small a market
to drive substantial innovation in
sensors and computers
but commercial electronics is a huge
market that is driving those
and robotics will basically
ride that wave in sensors and computers
i i have a curiosity question i was
curious
since you worked with both hans moribeck
and takeo
what was it like working with them
they are polar opposite personalities uh
you know hans was a very creative guy
i have to say somewhat eccentric and
didn't like to manage
and uh you know he was fine in
communicating one-on-one but he didn't
like to give talks in front of big
groups
so he was a pioneer in his day and did
some radically new things but takeo
um you know meaning in a positive way
takeo
is an empire builder and uh and he's a
great communicator and he's a great
educator
and and he doesn't mind management and
he goes out and he builds large
organizations
and so he's had because of those
differences in personality
far more impact
so what were their labs different
well um
[Music]
when i got to cmu i think was probably
not that long after hans had got there
he is a faculty of course in me as a
student and
i think he had a 700 000 a year grant
from the navy at the time which for
at the time for university was a very
large grant
so he had half a dozen or more people
and was probably
the biggest robotics lab at cmu at the
time
um but because he wasn't really a
manager
it sort of it carried on for several
years but
eventually it kind of wound down whereas
takeo started at cmu really as a vision
researcher and he got involved in
robotics
i guess i don't really know but in
mobile robotics he got involved through
the darpa alv program
and he was involved in robot
manipulation as well but i
don't really know when he started that
[Music]
and he ultimately became the director of
the robotics institute
um and you know he was very adept in
raising money from many different
sources
and uh you know given his ties to japan
he was able to get
he was always getting you know new
equipment somehow which you know as a
student you never understood really how
this happened but we'd have
the best state-of-the-art printer which
he somehow got
as a gift from some company in japan um
and i think he would get other
you know fairly unrestricted funds
resources like that
so
[Music]
and we would have a very nice
group retreats for his whole lab so now
we're talking you know 60 people
as opposed to six people um once a year
we'd all go off to a local ski resort
and it wasn't just skiing it was you
know you'd spend two or three days where
you'd spend
half the day meeting and half the day
skiing talking about
where do we go next and um
and giving the students a chance to
present you know what have they done so
it both helped the students to
you know polish their presentation
skills and it helped the whole
group to crystallize their thinking
about where are we now and where do we
need to be going
and you know
part of the ability to do that was his
ability to
raise unrestricted money that could be
spent that way
so i owe a lot to takeout
um so our final question this
generally um if you could give some
advice
to young people who are interested in
robotics what would it be
um get a solid grounding in mathematics
but not just that you need to be
hands-on so you need to get a
solid grounding in computing
and the ability to program
it given the multidisciplinary nature of
the field it might actually be useful
to start with an undergraduate degree in
an engineering discipline
uh and then possibly do
you know with some computing options and
then possibly move to a computer science
department in grad school um
you know i've talked to students
recently who are grooming themselves for
grad school
who are getting a very diverse
background in computer science and
controls and elements of electrical
engineering
uh and i'm very impressed with you know
the level of
maturity and forward thinking i see in
people like that because it's a very
diverse field and when you get to grad
school
you know you can't be completely sure in
advance which direction your
interests are going to go and which
direction your opportunities are going
to go
so if you come in with that breadth
of background as an undergrad not only
are you more likely to get accepted to a
good place
but you are more prepared for whatever
opportunities arise
[Music]
you know if possible at the high school
level so you know through the first
program it's it's good to get
hands-on experience with projects at the
high school level
you know if you can manage to attach
yourself as an undergrad to
faculty doing research you know get
summer jobs or you know part-time winter
internships not internships really but
uh
you know work with faculty who are doing
research as an undergrad
on whatever they need to do to to get
that experience and exposure
um but also remember that it's
it's at least today it's still a fairly
small
field um so while it's growing
uh there are more jobs in other fields
than there are robotics um
and yes it's exciting um
but it is still a small video although i
believe it's going to grow
so thank you you're welcome did we was
there anything we missed that you wanted
to tell us about or
there's nothing burning in my mind i
think that was pretty good
thank you all right thank you