i was born in and grew up in oklahoma
city
born in 52 grew up
there went to mit
in 1970
just a few days before my 18th birthday
what'd you study as an undergrad
um i started majoring in physics and
math
and shortly thereafter
switched to computer science
and uh what what motivated you to switch
well i had taken a class at stanford so
when i was still in high school
i did a summer session at stanford
and discovered computers there
there was a very nice class in computer
programming that i took that summer
i had to sneak in and kind of crash the
course you might say with my brother who
was attending stanford
so i knew computer science was cool this
was 1968 and 1969.
so everybody else didn't know it was
cool
you know some people would say well you
know computer's just a tool
um kind of like you wouldn't study
calculators
you know telling somebody you study
computer science in those days might
have sounded to some people kind of like
you're going to study calculator science
or something
but uh
i think just from hanging out and
studying
uh physics and seeing some of the things
that are going on i just
just confirmed what i already knew in my
heart which was that computers
really were cool that was fun
when did you first become interested in
robotics
uh yeah that's interesting uh it's an
interesting question
when did that happen
when i was at mit i came into contact
with the people in the artificial
intelligence laboratory
and at that time they had just done
something called a
copy demo which was the idea that you
could set up a bunch of blocks and then
the robot would look at it and build a
copy of it
and so i was hearing some
research talks from that and
and that's what got me into the lab
initially
i written i didn't really start working
myself on robotics
until a few years later and that came
about
mostly because i just happened to be
sharing an office
with uh mark rayburt and
john hollerbach and shimon allman
all of whom were doing robotics of one
form or another
and uh they were just having too much
fun
who were some of those first people who
did the copy
yeah the copy demo um yeah as i say it's
a little bit before my time
um and i'm trying to
recover the names now some of them were
graduate students or young faculty that
were there
and then left before i really got to
know the people in the lab
i think one of them was eugene charniak
but it seems like he was doing
linguistics
yeah well i don't know you should ask
jerry sussman
you should ask birth old horn certainly
berthold horn was
involved tom binford who later went to
stanford
i think was involved
and kind of been
along related i mean there's related
work going on by
uh sorry i should have uh
should have done some little exercises
to work through all the names
uh there's a very famous guy another guy
who's well known for linguistics at
stanford
who might have been involved
what was the first robotic project that
you worked on
i guess the first because the first
thing i did was a
project in compliant motion and force
control
this was my master's thesis so
i started working on that a few months
after i started grad school
so i started as an undergrad in 70 but i
started graduate school in 76
so it would have been right around 76 or
77
and compliant motion and force control
that means
you know if you've got a hand and you're
trying to interact with things in the
real world
um you have to do it gracefully you want
to be able to
touch hard surfaces and interact with
them without
hurting yourself or or the surface
so it makes sense to try and control
force rather than just
controlling position what kind of
machine we're using
uh you know what that's that's funny
that uh we
we did have machines and i was messing
around with them
my master's thesis was
um theoretical
um i don't think
i did any actual force control until
after i finished the thesis
immediately after finishing it i did do
a visit an internship at ibm yorktown
heights
and worked with russ taylor there
worked for my
i think my um the boss of the group was
peter will
and uh another manager was dave grossman
russ taylor was the one who i was
working directly with
and i did implement a force control
system for their arm
that was called their experimental
system is called the rs1
and they released a product a couple of
years later called i think the
7565 or
the arm we had in the lab maybe that
gets back to the original question
what did we have in the lab we had
at a time we had the mit arm it was
called so this had been designed by
victor shineman i think at the request
of minsky
and uh i think the original one had
aluminum gears there would be a little
pile of aluminum dust under it if you
if you worked it too hard raver did his
phd thesis on that
and then we had an arm that we were
developing a lab ourselves called the uh
per brick arm john purbrick was a guy
who was designing an arm
and i actually did the sort of the
programming and control system for that
arm
and then there was a third system called
the silver arm
there had been a visitor at mit named uh
hirochika inoue
who went on to be a
very famous guy in japan and he had
worked
on assembly
and worked with tomas lozano perez
and that was on the silver arm
which was uh yeah another arm that was
developed at mit
any more hardware questions
so uh well what was your phd thesis
and how did that research progress yeah
so the phd was on
uh basically how to push things around
um
let's see i i think
maybe the right way what's the right
perspective for all of this so there's a
few things going on so my advisor was
berthold horn
horne was very well known for
using sort of careful i'll say you know
mathematical or engineering models of
the image formation process
and and and inverting them in order to
develop
really hard-nosed uh approaches
to constructing the shape the
three-dimensional shape
from an image so
there were a lot of us that were
interested in assembly and using
arms which may seem very distant from
vision
but i think i was probably inspired
by horn's work to try and do something
similar that is can you
from a very careful
engineering model of how things
move when you
when you touch them can you invert that
then to
come up with the plans and the controls
for a robot uh
um in the abstract that seems pretty
simple then you start looking at real
cases so i mean usually you think of
things at a rather abstract level you
know i want to put block a and block b
i'm going to pick up block a
you know where's the subtlety in that
where's the engineering model in that
[Music]
but when you look at very carefully what
happened you know
when you go to pick up an object
usually there's some error often there's
some error
and because of that error you make
contact
not exactly as you planned things move a
little bit
or maybe your fingers deform a little
bit there's all kinds of that of little
things like that going on
and uh from looking at this and sort of
thinking about some problems i'd seen
over the years
i realized that those little those
little motions can
actually be very important maybe the
best example are several examples the
best example was that i'd seen a
movie that had been made at stanford
called
i think it's programmable assembly three
short examples
um maybe this is going to show up at
some of your other interviews i believe
the
lou paul richard paul was one of the
authors and bob bowles
i don't know if he's on your list uh at
sri
and a guy who i never met named pingle i
think was part of it
um and they pick up a a hinge
they're going to put hinges together
they're going to assemble a hinge
and when they go to pick up the hinge
instead of the two fingers coming in
like this
and grabbing like you sometimes see
instead the fingers sort of
start over here and they came along and
pushed the hinge like so
and as they squeezed and if you're like
wow why did it do that well that
pushing process eliminated the
uncertainty and if you continue to push
as you squeeze you end up with those
fingertips right in the
corners of the hinge with
i guess i'll say perfect precision right
i mean you're in absolute mechanical
contact with
both features of this corner so it's a
precision that you can't get
just by raw positioning and you realize
when you watch people program arms you
see them doing these kind of things all
the time
russ taylor's phd thesis had a great
example which was that if you squeeze
a box and a lid at the same time
i think you might have squeezed it twice
you don't want in the x direction when
the y direction
then you've perfectly aligned the box
and the lid
and then makes the assembly easier
so you see this stuff all the time you
watch people work and if you've got an
eye for it you see people doing similar
things right is that when you shuffle a
deck of cards
you know you don't get the precision
with which this
card deck is stacked by open loop
positioning
actually if you watch an infant messing
with cards you will see that right
they'll pick up the cards one at a time
and put them in a stack and it's a mess
but you know somebody with a few years
of experience will squeeze it and tap it
on the
right or or tapping chopsticks right so
you just see these tricks over and over
again and you begin to realize it's
actually
an important part of manipulation so i
guess that's a pretty long answer
the question was what did i do for my
phd thesis
so i decided to work on
[Music]
on these pushing motions and uh that's
what i did
so when did you start applying them to
robotic systems
well that was from the beginning i i
don't think uh
i i don't think
uh it's not as easy to you don't see
these things i mean the beauty of
robotics is that
you know you're trying to make something
happen
and you can't appreciate the problems
until you're trying to make it happen
yourself
right so if somebody's out there and
they're doing something and you think
wow that looks easy and then you could
try and do it
then that's when you discover it's hard
right
so if you're trying to build a machine
to do something you're trying to program
a computer to do something
you really have to understand the
something
and so that's one of the very
interesting things you know there's all
kinds of
problems of a very fundamental nature
just
in geometry or in dynamics or in simple
mechanics
that could have been of interest even
100 years ago
which really never got that much
attention
perhaps weren't noticed at all until
they became relevant in the context of
robotics
so i certainly never worried about
pushing
until i was worrying about how to get a
phd
what did you do after your phd well
there was a few things so
there was one project that started
before i finished my phd
the guys at mit had this great idea they
had something called the year or the
robot
i think that's what they called and they
invited a bunch of people up
i think john liu
was there mike brady had been there for
some time in fact who knows maybe the
year of the robot was his idea
but russ taylor came
and uh so russ taylor
and tomas lozano perez and i
had a project to try to
i was on fine motion planning there was
an idea that had been around a long time
which was that
you can sort of divide manipulation into
fine
and coarse so if you're just moving your
hand from here to there
that's one thing but then when you're
trying to pick things up there are these
little fine motions that happen that are
perhaps more sensor mediated and
more involving compliance and pushing
and things like that
so um we were sort of trying to have a
very
develop a geometrical
perspective on how this fine motion
planning happened
how do you you know the errors
and uncertainty and the task
can be greater than the precision with
which you can control
things so then you want to use the
compliant motion
and maybe force feedback
to uh to help you guide the
system and into the into the desired
goal
so anyway there was that was a project
which
we called fine motion planning and
later on i think it was uh lozano
perez's
graduate students who coined the uh
turned it into uh lmt lausanna perez
mason taylor so a lot of
times people call this lmt sometimes
they call it pre-image back
chaining so there was that project i put
a lot of time into that
some before and a lot after working with
tomas and russ
um i don't know
it's a lot of time after phd so yeah the
answer to that could go on for a long
time
um when did you come to carnegie mellon
i came straight to cmu from
from mit so my my first year in graduate
school i mentioned i was in an
office with mark gravert when he
graduated
which was just at the end of that first
year
he went to jpl at caltech ultimately he
came to cmu
around 1980 or so and then in 1982
he convinced me i should come here
so what was cmu like in 1982.
uh
well let's see
it was an exciting place i was firmly
convinced it was the best place in the
world to do research
the money situation was good
didn't have to spend a lot of time
writing proposals
and the main reason for that was alan
newell and
raj reddy and people like that were you
know were
doing a lot of that work for us and so
um and there was and the teaching load
was light
there was no undergraduate program
but we kind of shared responsibility for
an undergraduate program
with the math department so the teaching
load was rather light
and the students were awesome
and the faculty that were around were
great
and it's just a terrific place
so was it a full-fledged robotics
program at that point are we still part
of it
well you know the robotics institute was
uh
had been going for just a year or two um
my uh i identified much more at that
time with the computer science
department than
i do now and uh you know i think i can
admit to
some prejudices that i arrived with from
mit
uh you know there's people as i was
leaving mit they go into carter
carnegie mellon telling me to watch out
for this guy and that guy
and you know maybe you shouldn't
maybe you should be careful about that
robotics institute you know
but in any case i mean it was kind of
you know people refer to those as wild
and woolly days
in some ways i guess they were there was
no robotics phd program
the robotics institute was
just a research institute anybody that
had a
tenure track appointment was joined with
another department so i was joined
between computer science and robotics
so it was interesting it's you know it
brought some great people here
hans moribeck was here because of it
and mark and i wouldn't have been here
without it i don't think
who else was here at that time well it
was a really interesting thing going on
uh ivan sutherland was here so
sutherland was
one of these people that i you know had
been part of
mit uh many years before i had been
there
and then uh rayburn got to know him
in in california and then sutherland is
a
an alum of the ece department here and
he was back here and he was building a
walking machine
and so he had attracted a really
interesting group of people
including mark and their walking machine
was pretty interesting and
they uh he produced a book which i still
pull out once in a while it's a very
unusual book and that it kind of
covers everything from how do you
create the little fillets in the middle
castings
you know all the way up to how do you
design gait patterns for walking
machines
it's very uh it's interesting
so what kinds of projects did you start
when you came here
well let's see so i told you about the
fine motion planning project
um
what else was going on here at the
beginning what was i doing
more pushing i kind of started uh
well i guess probably the most
interesting thing was
a thing called trey tilting
and that was a work with mike erdman
um so he was still he was a graduate
student at mit and he came here one
summer
and we were just trying to figure out
how
far could you go understanding
uncertainty or eliminating uncertainty
in the world
but not by using sensors and maybe this
just
started as a sort of contrarian thing
but you know i think
almost anytime you pick up a paper on
robotics research
it might have said well there's
uncertainty in the world and therefore
you've got to use sensors you need force
sensing and tactile sensing visual
sensing et cetera et cetera it all seems
perfectly obvious if there's uncertainty
in the world
you know you need sensors to get more
information
and then um you know but we knew some
some examples uh where people were
getting really interesting information
without using sensors
and uh i'll cite an example from the ibm
group peter will uh
had told me some stories that were
pretty interesting about these things
and then
grossman and blastian had a paper
where if you want to know where a part
is in a tray
well you just or i'll say it this way if
you want to know where a part is
let's say it's kind of an irregular
shaped part you show the
you have this tilted tray you throw the
part in the tray
and if there are no symmetries or
anything and if maybe
even shake the tray to eliminate
friction it's going to settle into one
of some finite number of poses
and and you can disambiguate that
by just having a sort of a cat's whisker
and so you go in and kind of do a binary
search you say is there stuff there
no well is there stuff there no well
then it must be
configuration number three and uh
so that seemed kind of interesting and
mike and i started thinking
gee i wonder if we could even get rid of
the cat's whisker you know
can we can we actually figure out where
something is with
zero sensory information so
the first idea was to use
actually to kind of use pushing so you
could imagine
imagine you have a part that's like in
the center of a table
and let's say you like push it like that
and then you push it like that
you're not looking at it or anything you
just know if it was somewhere in the
center of the table you do a couple of
pushes like that you're going to know
approximately where it is
but you know maybe you want to do some
other things maybe you kind of come in
with a needle and do this
or push this way or that way you know
can you actually get all of the
information about its orientation and
position this way
well that turned out to be too hard but
there's a kind of simpler version of
that which is you go back to the tray
toss the object in the tray and now you
tilt the tray this way
and then you tilt it that way you tilt
it that way things sliding around you
make it slide from this corner to that
corner back and forth
a few times and at the end of that
process you know
its position and its orientation so uh
i don't know it's a long answer are
these no should i be
okay okay well anyway yeah so
uh uh that was one project that was with
erdman and i don't know that one
actually we still pull it out and show
it to people a lot
erdman had just done some work where he
could automatically
predict the behavior of rigid bodies in
her fictional contact and so he had a
system
more or less lined up ready to go and so
he could basically simulate all the
actions that you might contemplate
in order to give you the information
that you need in order to do automatic
planning of that
and then but i think maybe the most
interesting bit
was that you could
you could identify a finite set of
actions so
subject to some you know these things
are always full of assumptions to try
and narrow things down but subject to
these assumptions
you could show that there was a a finite
set
of actions that covered everything that
you could possibly do
and uh and so that made a an exhaustive
search actually
feasible for that for that tray tilting
problem
so what other kinds of main questions
were you
interested in you know that's a good
question
you know a lot of it was i mean it gets
back to that original inspiration
from horn uh
i guess i've always said sort of that
maybe the main thing i'm interested in
is the mechanics of
manipulation um
i think you know the work on how things
move when you push on them
uh got enough traction or you know i got
enough sort of positive encouragement
from people
and you can see there's a lot of
interesting things kind of along those
lines
um so that was something that
uh and of course i'm working with
graduate students and
who were doing most of the work by that
time so
randy browsed ken goldberg
probably ken's thesis
was the one that
was the most uh
noticed you know out of that whole body
of work that was going on
shortly after i'll say shortly after my
phd within the first 10 years or so
so yeah mechanics of manipulation that
was
one element
uncertainty how to plan for uncertainty
and how to eliminate uncertainty
with i don't want to say without sensors
it's not that you know they did get to
be a joke for a while that i hated
sensors
and um uh
but you know the positive side i mean
the the truth of it was that we were
interested in getting rid of uncertainty
by
knowing how the world behaves and what
you've done with the world
um and then there was a time when i
started working on learning
so at some point rayburn left and tom
mitchell showed up
and uh you know not exactly a
coincidence uh
i got interested in machine learning and
uh
alan christiansen another graduate
student did some work there
and uh just recently coming back to that
the original idea was that you know
we've been doing this work in mechanics
as if we were
what's the right word as if we were 19th
century philosophers trying to prove
everything from first axioms and
it became an interesting question you
know instead of trying to create a model
of the world that was
based on newtonian mechanics
could you do similar things with a more
empirically derived
model of the world so could you just
watch how things proceed
and abstract from that and develop
models of how the world works
with which you could then construct
plans automatically
so what were some of the the
important projects that you worked
um i don't know let me think uh maybe
there aren't any
of the important ones that's a tough
question
there are so many often people work on
so many so some that you
yeah um i don't know i think uh
you know the more important the more on
the periphery
um i mean most recently you know the
urban grand challenge
uh and i was just on the periphery of
that i
probably wouldn't have been involved in
that if i wasn't the director of the
robotics institute but
uh certainly that was i think one of the
biggest days in the history of robotics
and uh you know when i got to be there
but it was mostly just kind of
supporting in an administrative way
the work of red in his gang
gee i don't know um
i mean some of this stuff is you know
it's hard to say how the technology gets
into place
uh i mean when i started it seemed like
there were 100 different companies that
were
talking about doing manufacturing
building manufacturing equipment
i remember we went out and consulted
once with bridgeport
who i think were you know who were
working on
building a robotic arm for manufacturing
you know some of the things we were
doing
but i'll say kind of mundus what seemed
like mundane things like you know
gee why don't we have uh why don't we
have
uh robots whose
control computers include the ability to
have subroutine calls
or you know functions with parameters
you know very very mundane things
some of that stuff uh you know you never
know how it got to where
to where it got in most cases who are
some of your phd students
uh let's see uh this is gonna be really
bad if you put this online and i forget
anybody
so
you wong who was actually the first
person i ever met
from mainland china so
they were pretty rare in those days
he sometimes goes by michael wong
randy brost
ken goldberg alan christensen
kevin lynch
wes wong
um garth zeglin
devin balcom
[Music]
um amir degani
graduated they got his phd like two days
ago
and so now let's see uh who have i
forgotten
i'm really sorry do you have a list
was it a test yeah
um well devin right now
is at uh uh is it dartmouth
and kevin lynch is at northwestern
ken goldberg is at usc ellen christensen
is it uh mitre
randy brost i forgot the name of the
company they're working on solar
mirror technology he worked for quite a
while at
sandia national labs and then worked for
kodak
um
you know see him for getting even more
people you spent some time at sandia as
well
i did a couple of summers i i went there
for a couple summers
did you do that i was trying to help
randy brost
and his group uh one summer i
added some
a uh of pushing uh to a system that they
were working on
and what did i do the other summer i
don't remember must have been
wasted
what are your projects right now
well uh
simple hands there's this whole there's
our whole
our whole uh plan and all kinds of
crazy ideas most of which aren't going
anywhere
but uh you know maybe a few will
um when is this going online i gotta
publish everything yeah at least
oh great i'm planning to do this the
rest of my life
um yeah the simple hand idea
is that uh well it's interesting if
if you look at how people design hands
there's a few ideas that dominate so one
idea is that
you know this hand works pretty well
let's build something that's kind of
like that so the anthropomorphic idea
and it's uh i don't want to make it
sound superficial i mean obviously
people know a lot about the human hands
and everything we learned about it can
inform the design and the people that
are designing those things have done
some really amazing things
there's another idea which is that
maybe we should have be able to
manipulate something
just using the freedoms of the hand so
um
if you're going to do that you sort of
need to have at least nine
motors because if you just grip
something at two points
you can't very well control that
rotation
where you have three fingers you can
well if you want to give arbitrary
motions into this thing with three
fingers then you need to be able to give
x y and z motions to each of the finger
tips and so the
three times 3 gives you 9 motors
so both of those
stories give a very
tidy approach to the question of how do
you do manipulation in general
if you can grab an arbitrary shape by
having an anthromorphic hand and even
have some
general i'll say fine motions again you
know using the fingers
and then if you've got a general purpose
arm meaning it's got like six degrees of
freedom or more
then that seems like you've got a
general purpose manipulation system and
that's pretty tidy right i didn't give
you a lot of assumptions or anything
but there are a lot of hidden
assumptions in there
like for example the world isn't
composed of rigid objects
there's lots of soft things in fact
maybe most of the things you want to
work with are soft
and there's liquids there's bulks things
there's just multiple manipulations all
kinds of stuff
like that
[Music]
and you know but then
the other thing is that these hands are
really
uh complicated and we really haven't
mastered them i mean
i've mastered that one and you've
mastered yours
but we roboticists haven't mastered the
robotic copies of these hands
and uh and
uh why is that
there's a funny thing which is that
if you look at at practical systems
if you ask what's the most sophisticated
autonomous
manipulation system that's that's been
developed
um it's pretty hard question to answer
but i'll just say like you know in i
think one of the most impressive ones is
a system that mike erdman did one time
which involves two
arms each arm has just a palm
okay and by uh varying the angles and
pushing this way and
it's automatically reasoning about lines
of force and where the gravity is and
how much friction there is and what the
shape of the object is
and it's able to do
manipulation of objects all reasoning
and just using palms
and uh you know you can look at some
other systems a system
done long time ago by lozano perez
called handy a system
done at uh um
called freddy at edinburgh university
many years ago both of those had
relatively simple hands well handy
sandy might have to sell it i don't
remember about handy but
anyway and then if you look at what
people do
well you think well people have these
kind of hands
but look at somebody that's lost their
hand that has a prosthetic hook
right they've got a very simple hand and
they're still doing all kinds of
you know of stuff not as well but
they're doing pretty well
and uh or you can look at a human being
using a
a tool you know a simple gripper
a pair of chopsticks or
one of the most interesting things to
look at
is if you look at a surgeon
using a da vinci surgical robot
those grippers are very simple there's
almost no haptic feedback i think for
all practical purposes there is no
haptic feedback
that's what people have been telling me
there is visual feedback
uh they do really interesting things
they're folding you know if you
you should you should youtube robotic
origami and you'll see
somebody folding a paper crane with two
simple little
grippers you know as simple as alligator
clamps really
and so
so what we're trying to do is to say
well let's uh
let's look at devices that are so simple
that they're very affordable they're
very light
they're rugged we can use them
but maybe more to the point they're so
simple that we can understand them
and then try to build up the autonomy
you know from from that understanding
arising from the fact that we're using
such simple devices
now it makes a lot of things a lot
harder i mean the tidiness you have with
uh you know with saying well i'm going
to have a device that's so complicated
that i can conform to the shape of the
thing
no matter what shape it is and no matter
where it is
right we don't get that tidiness anymore
so if i have a very
simple gripper that's going to go pick
up let's say a
bottle of apple juice you know
by just squeezing it good and then
picking it up
that's something a human being would do
like that
but our theories of what constitutes a
stable grasp
don't work for that because the pose
from which
that grasp would be stable may not be
available with a very simple device
and what you see the human doing is to
grab the thing even though it's not
exactly stable where it is right now
but you just squeeze it anyway when you
pick up it adjusts
you know and you've got it it's not a
problem
so um
so that's what we want we want you know
we want something that i guess
in a way has to be smarter
but can but can use
simpler devices and develop an
autonomous manipulation
ability that way so you have a book with
ken salisbury manipulation yes
collaborated on other things
with ken um that's interesting ken and i
have been such good friends and
uh we really worked together
i mean even that book wasn't much of a
collaboration it was kind of a staple
job
it was uh ken's phd thesis in mine
and then ken tossed in some of his
papers as well
i don't even remember if we wrote a
forward for it or not
that book's been uh
book's been very good there's lots of
great things that ken did in his phd
thesis and sometimes
when people cite it it's cited as mason
and salisbury because him comes before
us so thanks ken that was
pretty nice you can see some people who
absolutely don't want to cite me for
ken's work and they'll
they'll cite ken's tr or something else
instead just
just to make sure that's understood but
i sorry to your question about working
with ken
um you know i had an opportunity and
when he first did the whole arm the wham
arm you know because i'd been kind of
working on related things
we talked about it and i couldn't really
figure out any way to help
let me think about other things i need a
chance interested by telling
wait is this running again
yeah yeah but with ken uh so we were
talking about
collaborations with ken salisbury uh
funny i don't remember
i don't remember ever actually doing
something
professionally with him but i remember
always visiting him at mit when he was
there
and uh and we used to
we did used to hang out together so
there was a there was a
i think there was a group that was a bit
more fun
and i don't know if this was a stanford
thing or was it a west coast thing or
something but all of a sudden i fell in
with a
gang at conferences that were going uh
going bar hopping so
i think i can tell you at least that
much yeah that was fun
salisbury was in that group mike
mccarthy uh
was maybe i shouldn't be telling these
things huh
mike mccarthy i met him at uh at a
workshop in minnesota
and i had no idea that in those days
there was this really awesome music
scene
in minneapolis and uh mike knew about it
and
i think i met him and then like five
minutes later we were headed down there
to
check out some of these bands so yeah it
was a lot of fun so
i don't know mike's on your list or not
but he was uh
a student of bernie roth i think and
there's a lot of interesting history
around you know
mathematical kinematics which i assume
bernie gave you all that uh let me think
what's the question other collaborators
yeah
like that kind of social network as your
student and then
that's also very interesting yeah
uh yeah there was this group of people
at mit
so when i got there
mark rayburt and john hollerbach
and then tommaso zana perez was doing
one year
at yorktown heights and i believe he
came back
after halfway through my first year in
graduate school
and between them and eric grimson
ellen hilldrath uh
uh it was quite an interesting group
there was also a guy robin stanton that
showed up from australia
visited for a while and a
japanese visitor koji fukumori
and i'm going to be forgetting a few
people
there was something going on i guess not
that you would know it
necessarily at the time i mean how who
knows maybe you suspected but you
wouldn't really know or maybe the people
that are kind of the more arrogant
or you know or visionary you know
actually
believe it but but it did seem like
uh maybe maybe there should be
a field of robotics or maybe there was
something new going on
um there was you know in the way of
publications
uh there weren't many so my master's
thesis was published in the ieee
transactions on system man and
cybernetics
there weren't any robotics journals
there wasn't really a conference there
was the
there was an industrial conference
[Music]
mainly we were trading technical reports
we were reading the technical reports
from stanford and we're reading our own
technic
our own technical reports
and phd theses
we actually put a book together that was
a bunch of reprints
we kind of thought well you know we're
just trading these things amongst
ourselves if you
don't have the good fortune to be
to have landed in that group or a
similar group at stanford
how would you ever hit on this stuff so
we
we created a collection and i should say
that by that point in time mike brady
had showed up
and he was uh you know he's a real doer
i mean he's somebody that
when a lot of people are thinking maybe
we should do this maybe we should do
that he's
you know he's like okay i'm doing it and
uh and you better jump on the bandwagon
if you want to if you want to be part of
it because
because he didn't waste any time uh
but man i mean that combination really
worked i mean um mike
and i wasn't really privy to the inner
dealings of this but
but you know mike together with lou paul
put together the international journal
of robotics research within a few years
and um
and the international symposium on
robotics research
which was uh certainly uh at that time
the premier conference in robotics
uh and uh and then
i don't know exactly how it came about
but then it wasn't that much longer
afterwards that the ieee
international conference of robotics and
automation came along and uh
you know so things kind of did evolve in
the way that you kind of
i guess it gets sort of imagined that
that we had an idea that maybe something
like that was in the offing
but i don't think of it as the birth of
robotics you know it's a certain step in
the maturation
of robotics where it becomes
uh maybe more academically
conventional and people can have you
know conventional tenure track
you know positions and and uh
and succeed and before that you know an
awful lot of the work that was being
done
were things that you heard about yeah
this guy had an amf versatran arm
and you know and all the things they did
with it's not written down anywhere or
if it was you
it was hard to find an awful lot of
those guys
were just these you know were people
without phds without academic positions
kind of you might say kind of
you know moving around from one group to
another
and uh
so that so so that's i mean that was a
real change was it changed
i'm concurrent with that like how did
the robotics institute at
carnegie mellon evolve well uh
let's see so yeah i got here in 82
so it started in 79
when i got here
raj told me that
you know i mean the question is i think
you know everybody recognized at that
time
that mit and stanford were at least as
far as academic institutions sort of
uh the two that were doing robotics in a
big way
um there were some other places
draper labs in boston close to mit sri
jpl
and then the academic circles there was
some stuff going on in illinois and
purdue but
but you know it seemed like stanford mit
well okay i was at mit what do i know i
had my you know my
uh maybe not the best vantage point but
that's the way i understood it at the
time
and when i got here raj was saying well
uh cmu's
a more systems oriented place a more uh
less theoretically oriented and you know
we're going to make things that work
focused primarily on manufacturing
systems
uh that's kind of what i heard and i'd
you know that may not be accurate i
don't know if you're gonna interview raj
or not you should
get the original vision from him i think
but it really changed
uh if it really did start that way it
changed quickly
when moravec came uh
and when red whitaker joined up
and when mark rayburt showed up i would
say
within a few years we're already the
number one place in
mobile robotic systems and
with takeo canada coming here we're you
know one of the top places in computer
vision
and and these guys were doing
um work not just with industry not just
manufacturing applications but also
doing
work with darpa and
so with the defense department and with
the national science foundation
and then a very strong sponsorship from
nasa
over the years and in the office of
naval research
so our kind of manufacturing orientation
was a very strong element of the
robotics institute for a long time but
it started broadening out
uh right away
and how did you decide to
have a robotics phd
as part of the institute yeah that's a
good question
and uh you know what i don't really know
the answer you know i can only speculate
uh there are a few reasons
i guess i i you know things have to come
together
all the reasons have to be right uh
so i mean it's gonna be intellectually
right and it has to be
uh and you have to be able to solve the
kind of logistical financial
structural issues as well
and in this case everything was was just
right i think people saw robotics
uh maturing and were excited about it as
a field
and that goes back to the origins of the
robotics institute it's a very strong
support for it
from our most
uh valued uh leaders
uh alan newell in particular and then
you know the president of the university
and so forth
so intellectually it just kind of worked
i think we've always had people who are
interested
in new things you know in creating
new enterprises and uh so that certainly
fits
and then on the structural and financial
side it just it just kind of
works better it worked better for us to
have uh our own phd students rather than
to be
borrowing so to speak phd students from
other departments
so
so but i don't know the details
raj would know when did you become
director
and then about six years ago
how did that happen yeah well that's
interesting
most of my uh i think people would
i haven't been in close touch with year
by year were
maybe shocked to discover that i was
doing this
because in the old days i always kept my
head down i
i i thought you had to be insane to want
to be a department head
and so
i was always the guy who would try to
duck out of
every administrative job you know
committee
work and so forth i really wanted to
focus 100
on on research and well you know you
want to do a good job teaching so i
would try and do all right with teaching
as well
i had no interest in management uh
and so what happened well part of the
problem is that for deep
coast law uh i used to hang out around a
lot with badeep
and uh
and he was always
talking to me about this obviously he
saw things in a different way
as you might imagine from his his rise
in the academic administration
and uh so that actually he did have an
influence on me um ko
tricked me into running the robotics phd
program
uh and you know and that has its rewards
so i guess i uh i fell into over to the
dark side
you know bit by bit influence of
former friends good friends
and uh i don't know and then as far as
the actual event
you know it's funny because uh yeah
chuck thorpe was a director just before
me
and i imagined that chuck would be the
director for 10 years
so i wasn't really thinking uh about it
at that time um i mean i had when uh
when takeover stepped down i thought
well maybe i do want to do this
but when chuck after chuck had been
director for a year or two
he ian ousty was going to step down and
go be the dean of the qatar campus of
carnegie mellon
and and then at that point well you know
it's more or less
the faculty voting on who's least likely
to
give them trouble so that's an actual
vote and what's your vision for the
future of the institute
uh oh boy i should have i should have
prepared for this question
where do you see robotics going uh well
robotics is
let me know let me try that one that
one's easier
okay so what how here's my view of
robotics
robotics is about the idea that a
machine can do
the kind of things that people do and uh
and you might extend that and say oh
people and animals
depends on i guess what you think
animals are doing but
i you know can can a machine
have purpose can it be perceptive can it
be aware of what's going on around it
can it actually interact meaningfully
purposefully with the physical world
um it's a question that has actually
been
on everybody's minds for centuries right
this isn't a new question
uh there are stories about uh you know
the golem being made from
mud and uh you know machines that have
been built through the ages
that have mimicked
living things one way or another
somewhat superficially i would say
but now we're trying to do it in a much
deeper way right and maybe the most kind
of profound way
can can we do the things that people and
animals do
and uh and robotics
what makes robotics different and this
is i guess this is the 50-year thing
that
that brings us up i mean i think what
happened about 50 years ago
was that the final pieces were in place
where you could see that this might
really be possible
the final piece is being well computers
electronic cameras
and intellectual elements feedback
control theory um
and so about 50 years i think
about 50 years ago is when there comes
the first
plausible
idea that we might actually have the
kind of technologies we need
and and i
think that's actually been proven now so
if you
watch a
uh robocup soccer you know if you watch
a bunch of little sony ibo
robot dogs or cats whatever they are
playing soccer they're they know where
their players are they know where the
ball is they know where they are
they pass they shoot
they play defense
i don't think it's possible
in in ordinary in an ordinary sense in a
common sense
way to watch that and
without getting the sense that machines
can be aware of their surroundings right
and it's
it's easy to forget that you know
it was not at all clear that a machine
was
that a machine could do those things and
now i think it is clear it's not that
they're as good as humans
you know humans are still way ahead of
them and cats and dogs are still way
ahead of them
but i think they've crossed that
boundary where you could say
yeah you know you gotta you gotta admit
machines can be
aware they do perceive they do have
purpose they do
act purposefully so
um that's where i think
we've we've gotten to okay
well let's see yeah so i was on to the
spiel about uh
robot about ibo and robocup and uh
yeah yeah well so uh
one of the things that's happened is
that
our sense of of what use
this technology can be has also
changed uh it's a strange thing
so when you start out you you you start
with well we want to build
you know we want to build a dog well you
got to be able to see you got to be able
to run around you know
so let's work on seeing let's just take
seeing for an example we need a vision
system
right you build a vision system
and well in the process first off you
discover that it's very
very uh it's a rich problem there's
numerous different technologies that
can come together or be used
independently and
um and then you start looking at all the
applications of them so right now we
have machine vision systems
uh but mostly they're not in robots
right mostly they're out doing other
things probably the most common thing
for a machine vision system to be doing
is the systems that recognize faces in
your digital cameras
to do the autofocus to help with
autofocus right
people look at that and they you know
they don't
think of a robot right there's nothing
in there that's necessarily a robotics
technology
it's a technology that might have been
inspired by trying to emulate what
animals
and people do
and it has this other use well i mean
that's just the beginning
the most some of the most
interesting ones are mind-bending ones
that that i'm aware of is
for example teaching kids to read
so there's a project here called project
listen that jack mossdau is involved in
and uh it's simple they're using
voice recognition software
to build reading tutors they're using
the kind of cognitive models that our
psychology department
has come up with over the years there's
an algebra tutor program well they
developed a reading tutor program
the program knows what the text is
and it has models of how humans you know
how kids learn
and what kind of issues they're dealing
with as they develop
and so with that you can have an
informed
interaction with the student to to teach
them to read
i don't think you know i don't think
people ever really thought that much
about how profound that kind of thing
could be
a long time ago i mean when you start
having the technology available
you see these things the excitement in
it is not that you're doing something
that wasn't being done before
i mean a human teacher at least you know
a skilled human teacher
is still better than the computer but
the computer doesn't have to compete
with a human being right most of the
kids
if they're learning to read at all
they're reading classrooms where
generally they're reading alone to
themselves
right and the teachers may be going from
student to student and their interac
individual interaction with the teacher
is rather rare
the computer is competing not with
humans is competing with inattention
right and and computers are very cheap
they're getting cheaper all the time
uh microphones that you can do speech
recognition is uh
very cheap so it
it's an opportunity to
to provide that's the service namely i
mean a really important one namely
education
in ways it couldn't be into people where
it couldn't be delivered before
so you know another example of that
would be
another i'll say kind of unexpected
application
would be watching mice the um
there's a company in manhattan named
psychogenics they
uh test drugs by giving them to mice
and watching to see how the mice behave
right
so ordinarily a human being would be
doing this with a clipboard
and uh watching how the what the my
mouse does and writing it all down and
so forth
and it's very expensive now because of
the expense of the mouse but because of
the human being
well they came to us
we built uh some instrumented cages for
them that can measure
uh i think there's a force plate on the
floor i believe there's a puffer that
can blow air at the mouse
there are cameras so that it can see
where the mouse is in the cage
and behavior recognition software
that classifies the mouse's behavior
tick by tick
and then machine learning software
that's
classifying the behavior of the mouse
and running the drug testing protocols
and all that can happen
now autonomously
the point of this is again it's not it's
not that it's doing something that you
couldn't
do before but it's it can do it
it transforms the economics of these
experiments which in turn has the
potential to transform the economics of
drug discovery
so these things weren't on the list
right what was on the list before home
butlers
right this seems
i don't want to say you're relevant i'd
love to have a home butler you know
but but compared with you know
delivering education
across the digital divide or
dramatically changing the cost of lab
experimentation
right those things seem seem very small
so that's one answer
it still doesn't say where it's going
well how do you see uh the robotics
institute's role
in the future uh well let's see so
let me give you my excuse first why not
answer the robotics institute's
you know we're a university department
we have
uh depending on how you count somewhere
between 40 and 100 faculty
and none of them are looking
to me for direction so
the title director is uh
more of a courtesy title um
so they're all uh doing their own thing
they're all visionaries
a lot of times strategic planning is
more of a sense of
looking to see what they're doing and
maybe trying to extrapolate a little bit
connect the dots
and then run out in front you know grab
the flag and run back out in front of
the mob and pretend like you're leading
so you know five years ago we went
through one of these exercises
and we discovered there had been a big
transformation which
is uh what we call robotics in life
so in the old days when the robotics
institute started
we were focused on
motivating robotics by
by looking at things that people
wouldn't want to do or couldn't do
so space exploration you know hard
vacuum
long distances cosmic radiation etc etc
right it makes a lot of sense to
send machines to space mines
mining even manufacturing a lot of times
is very tedious and dull and people
would rather not do it it's hard to
sometimes fill some of these jobs
agriculture would be another example of
that right so this was what people call
the 3ds dole dirty and dangerous jobs
right and that that was how we used to
motivate
robotics but
you know implicit in that is the idea
that we're replacing
you know we're trying to figure out how
to get machines to do things that
otherwise people would have to do
replacing humans with robots and what
we've figured out within the last five
or ten years or so is that the most
exciting applications of robotics are
working with people
so medical robotics socially assistive
robotics
quality of life stuff helping elderly
stay out of the nursing homes live
independently in their own homes
longer and longer rehabilitation
education that i was talking about so
there's a long long list of these things
and that is i think
where the most rewarding applications
are and where some of the most exciting
work is
so this is what we figured out about
five years ago the advisory board came
there was an interesting faculty meeting
one year where this was when chuck was
director and he said well uh
oh and the president university was
visiting and checked without having
prepared just said well let's have a
show of hands how many of you are doing
right now some application that involves
human interaction or medical robotics or
something like that and we were all
stunned because more than half of the
hands in the room went up
so this is uh an interesting change and
not just at cmu all over
all over the world of robotics very
interesting stuff
now did you ask me about five years ago
or about now
the future yeah i don't know it's
interesting
i don't know is maya matarik on your
list
i heard her saying that
socially assistive robotics might end up
being the most significant application
that i think is a really interesting
idea where is it going
uh
i guess what i've maybe my conclusion
from all of this the
kind of the proliferation of
applications you see
the appearance of applications that are
completely
outside of what anybody imagined
i i have a pretty humble attitude i
think about
trying to project what's going to happen
in the future
if you ask yourself what are the
applications of robotics
well let's see so if you just think back
a few years in fact maybe this hasn't
really
changed i mean if you just look at a
computer and you ask
you know what are the applications of a
computer well it's a lot
but you know still you look at it and
how does it get sensory information well
it's not that much right it's getting
stuck over the internet but other than
that
it's key strokes and you know what kind
of actions can it take well it can't do
much
you know if you imagine that you
yourself are in that
situation you're locked inside of this
box
you know interacting by putting pictures
up on a screen and getting keystrokes
well you wouldn't really be able to do
much right so
comparing an animal or a human being
with that is i think
really mind mind-blowing
uh what are the applications of
a human being um
isn't you know i mean if that's what
we're doing if we're
finding machines that can do the things
that human beings do
it's like asking what are the
applications of a human being
but without all of the constraints
right so i mean the reason the reading
tutor works is because
they're so cheap i mean they're just
impossibly cheap right you can't
train the human beings and get the human
beings in place that can
that can do that it's the same thing
with the mice
right why are they successful in space
uh red
i think i think i've heard red say every
space mission every robotics based
mission is a suicide mission
right you can do things in space
obviously you can't deal with human
beings they can be small
they're not all soft and squishy so
uh well okay so that's just another
excuse for why i don't have an answer
but uh uh the immediate future
uh is great one of the best arguments i
have for the future of robotics is to
just show people the
uh level of activity in the robotics
institute which
started at a million dollars a year in
1979
and it's running at 63 million dollars a
year right now
and uh almost almost without exception
it's been exponential growth through its
whole history despite
the so-called failure disappointment of
robotics in the 80s
despite ups and downs the trend seems
very obvious
and it's an interesting phenomenon it
depends on several things working right
but one of them is
that there's just no limit inside to the
applications
sorry do you see any limitations to
to where robotics can go
i mean uh not fundamental
no no
everything looks everything looks good
people are excited about it that's
the that's the main thing the
excitement's justified
what would be your advice to uh young
people who are excited about robots
you should do it sometimes
sometimes they're slowed down so you
know we do have people that come in and
say well
i don't know you know is the robotics
industry really going to take off
are there going to be jobs and that's a
too narrow way to look at it right if
you say the robotics industry well
there is a robotics industry and it's
doing great but that's just the tip of
the iceberg
right the real payoff is in uh
how it permeates all industries and
so you know if you look at where the
robotics phd students have gone over the
years
most of them haven't gone to places that
you think of as a robotics place
a lot of them are at microsoft research
that go to microsoft
google and intel
financial companies
all over the place nasa of course
and you asked how do they get started
well
we're again having we're going to have a
problem because it is so broad right
there's many many different things to do
in robotics
if it's your aim to do a phd in robotics
then you should study
hard you should get good grades you
should study math
and you should go home and
build um build stuff
um and uh what
stuff you know in my day people were
working on their cars
that's not as easy as it once was i
started building radio controlled cars
about 10 years ago that's pretty
great radio controlled model airplanes
some some people i know used to build
helicopters
and then these days the really amazing
thing is suddenly you know robotics as a
hobby is
developed into something really
interesting subscribe to make magazine
learn to build flame cannons in your
backyard
don't tell the cmu lawyers i said that
you mentioned the the failure of
robotics in the 80s
or the the so-called so called yeah yeah
could you just tell us a little bit
about that and how actually people
managed to kind of
in a sense get out of it yeah i don't
know
uh you know it's it surprised me when i
was looking at our
charts and i realized that you didn't
see a dip in the
in the uh funding uh because of that i
mean what happened was that
in the 80s or late 70s
the hype about robot
about revolutionizing manufacturing
was really over the top
and and a lot of venture capitalists and
others got
excited about it and there were many
many companies
building basically arms and hands for
for manufacturing
and at some point the floor fell
out and uh almost all those companies
went belly up
or a lot of them were companies that
were trying and
had other things to do they just stopped
um
why why did that
not have a serious impact
at cmu i think the reason is because
well there's there's two reasons i mean
one reason is because the technology
is so
fundamental well that's right we're not
so basic to so many different
applications i mean the ability to
perceive your environment
you know if you put it that way it's
like does that have anything in
particular to do with manufacturing no
that's just everything right has to do
with everything
so uh so there's that and then there's
the fact that the faculty here are
extremely agile
adept even proactive at anticipating
downturns i'll cite another example
which is that
we were we did have an enormous
run-up in funding from the defense
department for unmanned ground vehicles
and when there was a downturn
the faculty had basically anticipated it
and have replaced that with
funding from other sources mining
agriculture and other things
so you know i guess one way of looking
at it i mean i look at this
as evidence that there is an untapped
reserve that there's you know
there we haven't seen the bottom of
of the resources that are available for
developing the technology
is there anything that you'd like to add
that you think we
missed i don't know history of robotics
and 50 years ago and stuff like that
um
well i don't know i'll say this yeah
i'll say something which is that i think
i think the most important thing
robotics uh can tell us is about
ourselves
and this gets back to a question that
was asked earlier you know
did i had i pondered the mechanics of
pushing
before trying to do it with a robot and
the answer was no
i don't think we have a sense of how
hard it is what we do
or even in detail what it is that we are
doing
um until we try to do it ourselves
and uh you know when people ask well
what's the number one lesson that we've
learned from robotics
maybe it's we've learned just how hard
it is to do these things
so you know it wasn't so long ago
everybody was impressed with how
difficult chess was
and then uh and then you know computers
started competing at the
grand master level in chess well we're
still not competing with
with humans in what we generally regard
as the most trivial of
physical tasks and
as you ponder that you start asking
questions like gee you know how hard
is it how hard is it to let's say
figure out what's the best way to jump
you know how high could a human being
jump
what would be the best strategy to jump
uh
these are really hard questions right i
mean the world of jumping was
transformed several years ago
you know with the fosbury flop you think
wow
you know after being in these bodies for
i don't know how you count you know
somewhere 500
000 years or 5 million years we're still
discovering how to operate them
you know in fundamentally different ways
that kind of fits in with the robotic
experience which is that
that motion planning and control and so
forth uh that these problems are
fundamentally uh very difficult
so our progress you know with with
robotics i think
not just at such an overall level but i
think in detail
can can help us to understand what we're
doing
do you think once we build a robocup
team that can beat the world cup team
that that will
diminish our appreciation for soccer ah
diminished appreciation for
no i don't think so i don't think i mean
if it turned out that somebody just
somebody said oh my gosh you know there
is this we just didn't think of it we
should have been using left-handed
screws instead of right-handed screws
and then everything's simple you know
and
unless it's something really stupid like
that or like you know
oh you know frankenstein he hit the
dead body with electricity that's the
secret of life right
it turns out to be something that's
simple and i guess that diminishes our
appreciation
but like most people i don't think i
don't think knowing
the nature of stars actually diminishes
our appreciation of stars
and i think human beings are going to be
way more interesting than stars already
are
great thank you