so you just start by uh telling us
your name where you were born and where
you grew up
my name is chris axon i was born in
washington dc our nation's capital
in 1959 it was just two years after
sputnik
and uh i grew up in washington spent
some
time towards the end of the 60s in the
philippines
and then moved back to washington in the
70s
in the 80s i went to college at harvard
and then graduate school at mit uh
actually harvard was late 70s so i was
in
cambridge during that time i moved to
georgia tech in 1994 and i
moved to carnegie mellon in honor of the
millennium
in the year 2000 and i've been here ever
since
what did you study at harvard
biochemistry i was a biochemistry major
i also got a uh
i also got a master's in applied math
harvard didn't acknowledge that
computers were worth studying
so uh that you had to sort of sneak it
in and applied math at that time
and then when you uh you went to the
graduate school what department did you
graduate
when i entered it was called psychology
when i graduated it was called brain and
cognitive sciences
and what did you specialize on what did
you study
i don't think they had internal
divisions or titles or anything
i my advisor was a neuroscientist called
emilio beetse
he was interested in how the brain
worked obviously
but in particular how the brain
controlled movement
and how you coordinated movement at that
time uh
robotics was starting to happen uh at
mit
uh i guess in a more visible way and he
reached out to two people a guy named
neville hogan in the mechanical
engineering department
and a guy named john hollerbach who was
at the mit ai lab
and they collaborated trying to ask the
question
what what can robotics tell us about how
the brain works because clearly the body
is a robot
so the brain has to be a robot
controller and maybe
some of the theory that's being
developed for robot controllers
could be useful and that was the context
in which i opera
i was in graduate school uh first couple
years
i first year i studied human movement
uh second year i
yeah it's all coming back to me now i
said war or human movement in a slightly
different way i'm happy to go into more
detail
on that kind of stuff later
and we also looked at did some monkey
studies
and then at that point i decided
oh let's go try to understand
robots they're a lot simpler uh and
that's what i did
so my thesis was on the question of
if you pick something up you know how do
you know how much it weighs
the example i love to use was
when you reach into a refrigerator and
pick up the milk how do you know it's
frozen
now that's not a question i actually
answered my thesis but that was a
motivating question
so what were some of the human movements
that you studied
so a big underlying question
at that time was it seemed that robots
or the way to make robots useful was
develop something called a general
purpose controller which meant
that whatever you were asked to do you
had some planning machinery that
basically operated the same way and just
did it
and a signature of that kind of approach
is
no matter how you moved you're always
going to do things in the same way so
maybe you move in straight lines or
something like that
so the obvious thing to do is to then
look at humans and
apply that kind of test and see
when we ask humans to do different
things
do is there some commonality or are they
coughing up some sort of special purpose
solution to everything they do
and it turned out most of the time you
ask people to move from point a to point
b
and we develop some measurement system
to do that
they move with a hand and a roughly
straight line and more importantly
with what's called a bell-shaped
velocity profile so they
start up in a particular way and they
slow down symmetrically
so it's a nice smooth movement
and that was actually an argument that
whatever's going on in your brain
there seems to be a general purpose
movement system at least for
move from point a to point b in a human
with your arm
later the bell-shaped velocity profile
is extended to all sorts of things like
jaw movements
tongue movements leg movements
so it actually seemed that whatever's
going on
it generalized across limbs as well okay
so uh what were what were some of the
things you discovered in terms of the
control structures that are models that
you then try to
apply to robotics
i'm going to focus on what i think is
the the last part of your question
because i could parse that as two
questions uh
you know how did the study of humans
affect what we did with robots
at about that time there was a lot of
emphasis on
industrial robotics and doing things
fast
and so the idea was if you want to go
from point a to point b
you want to hit your arm as hard as
possible turn the motors maximally on
accelerate maximally get up to your
maximum velocity
do what's called coast move at a
constant speed
and then when you're close to your
target you want to reverse the process
and blast your arm with your motors
in the opposite direction so you slow
down as fast as possible and then you
stop on a dime
okay the public image of robots
at that time was that they moved jerkily
like this okay and if you ask anybody to
imitate a robot
you'll see them do jerky movements and
in fact i believe there's even a robot
dance called do the robot
which is all about that that had
nothing to do with robots it had
everything to do
with this choice of movement strategy
humans could move like that but they
don't want to
they instead use this bell-shaped
velocity profile which means we move
with a slow acceleration and in fact
let's see there's something called
position you're changing position
there's something called velocity which
is how fast you're going
there's a derivative velocity called
acceleration which is how fast you're
changing your acceleration then there's
a derivative of acceleration
you want to guess what that's called
it's called jerk okay because when you
change your acceleration rapidly you
induce a lot of vibrations of the system
and makes it look like you have high
jerk
what humans seem to do is whatever
they're doing in their head
they seem to be moving as if they're
minimizing the square of jerk
when you apply that to robots and get
them to move in the same way
they look alive they look like a human
and they do not look jerky
it's time to end this mindless
discrimination against robots
they move just like we do okay i mean
did you was there
you can control the church just get the
jerk under control
exactly so to speak okay there's another
joke i have to share with you
the higher derivatives people ask the
question okay if minimizing jerk is so
good
let's minimize the derivative jerk you
want to know what that's called
that's called snap can you guess what
the two higher derivatives are
crackle and pop i like that
okay moving right along okay
so what was the first robot that you
actually worked on
so we actually have one of these still
downstairs it doesn't work
but it was a puma the large size puma i
forgotten what kind of number that is
it's like a i don't know maybe it's the
250 or maybe something but it's you know
it's it's link was about that big uh and
that's that's what i did
and the first thing i made it do is
i wanted to avoid having to have a hand
it didn't really have a hand
so i screwed on a weight to it and then
i have it
waving around and it would tell me how
much the thing weighed
and what were its moments of inertia in
other words sort of what
is its shape it's sort of like going to
a carnival and guessing your weight
by having something grab you and shake
you
so that's what i did and was that your
thesis project
it led it was part of the thesis
it turns out that we can take that idea
of
holding some in your hand and figuring
out how much it weighed and then
pretending that you're holding your
forearm
in your upper arm and we can move that
around you can now know how much your
forearm weighs we do the same thing with
the upper arm
so and we were very interested in this
idea that humans self-calibrate right
you're born and no engineer measures you
or anything like that
and somehow throughout your life you're
always figuring out how much do i weigh
what does it take to move me around
and so that's what we were doing with
robots we were strapping things onto
them and moving them around
or just having them move around by
themselves and having them tell us
here is a good model of how we work
and then we can control ourselves more
effectively
and is i mean were you particularly
interested in making the robots
be more human-like be more human-like
or was there a different motivation for
taking the human as a model for robot
okay
there's a basic split there are two
kinds of robotics people
they're people who want robots to be
useful and they're people who want
robots to be like humans and those are
pretty much
disjoint sets of people and they fight
with each other
and i'm clearly on the side of i want to
understand what it means to be a person
robots are a way for me to
do that understanding and my idea of a
good robot
is it's like a person behaves like a
person looks like a person
um isaac awesome off
and of course many many other science
fiction writers
and movie generators so metropolis the
robot looks like a person
it's a lot easier to you have an actor
be your robot so they naturally look
like people
but a lot of these people believe that
that's what a robot
should look like and
so they're all on my side
in the 1970s i can remember as a grad
student or i guess 1980s
a professor named warren siri in the
mechanical engineering department said
wrote an article for i guess the mit
alumni magazine
technology review which basically said
if you want to build vcrs
your robot should not look like a person
it should look like a vcr making
device and he was absolutely right
but the politics of this message were
really bad
it basically said don't fund all these
crazy people who want to build people
or artificial people you know
fun people who are going to do something
useful to society because at that time
the reason robotics was hot was because
we were
you know trying to deal with the
japanese and the perception
that automation automation was making
them much more competitive relative to
us
and our trade balance was therefore
terrible
uh that's obviously that fear is receded
now
and we have terrible trade balances with
places like china which has
little or no automation so
there are other issues going on as well
okay so um uh
who was your you mentioned your advisor
immediately yes
okay so um so your robotics work then
got you the degree in
cognition or brain and cognitive
sciences that's right i was now
accredited
to go up to robots and say how are you
feeling
and if they said well you're feeling bad
i probably with a little extra work
could have been accredited to give them
therapy
and people did do that in my department
they went out and got accreditation to
give
therapy except humans not the robots
um so were there other people in that
department working with robotics
at that time in the media's group yes
there were other graduate students uh
tamar flash who's now in israel uh
and uh you know several others
as well i'd have to go through the
records to actually figure out who was
there
and were you working closely with people
in the ais
so john hollerbach was in the ai lab and
i was working pretty closely with him
but you were physically separated in a
different lab
no i pick myself up and walk across the
railroad tracks to
the eye lab you know i went wherever i
needed to go
okay i was kind of like the guest who uh
stayed too long you know she shows up
but
what's that guy doing on the ninth floor
you know that kind of thing
and who were the people that were doing
robotics in the ai lab at that time
well there was a huge number right so
they had gone out and gotten funding to
build a robotics group
from i believe something called the
systems dynamics foundation
and got a lot of money and you know
there was a big effort at that time
to sort of track down the details of
that you really should talk to patrick
winston
or zano perez who are both still there
to sort of flesh out that vision
i think bertolt horn is still there but
i think most
everybody else has moved on to one place
or another
but which ones did you interact with
specifically
from the people in the ai lab
uh well i had a fellow graduate student
named cheon and i worked very closely
with
i had some other graduate students dave
siegel and sundar nara simhan
i worked very closely with i was a
graduate student so i worked
very closely with other graduate
students
great um so uh
what did you do after you graduated i
became a professor
or assistant professor at mit in the
brain cognitive science department
and also in the mit artificial
intelligence lab and so what were what
was the
kind of work that you started developing
there when you were able to have your
own
lab in your own research direction
um i'm just pausing to think
since you're asking about things that
were
alone a long time ago 34 years
okay um so
again i'm motivated by
things that we think are going on in
humans
um so a lot of people have very
been very interested in something called
content addressable
memory and the idea is in her computer
if you shove things into memory
it doesn't speak to you you have to
remember the the box number
that that thing went into if you're ever
going to get it back
uh it's like going to the lockers at the
train station and shoving things into
the lockers
if you don't know that you put your hat
in 852 you will never see that hat again
worst case what you have to do is open
every door until you find your hat
there's a different kind of memory where
you shove things into the lockers
but then you can speak to the lockers
and say hat
and 852 opens
and the belief is that's a much better
model of how biological memory works
so we spent a lot of time thinking about
different ways
to implement that in engineering systems
and how you would use it to improve
control of a robot
and there were other people who were
using it
in the general memory based approach or
exemplar based approach
or the instance based approach this is a
very popular sort of meme at the time
where they were using things like
something called the connection machine
a big parallel computer
and they would just shove a lot of
things into it and
you would say i'd like to know about
baseball
and it would figure out what memories
are relevant to baseball and spit them
out
the modern version of that of course is
google
okay so we were asking the question how
could google be useful for controlling
robots
about 34 years or 30 years before there
was a google
and so what were what were some of the
specific projects that you started
working on to address this larger
problem
well we would do simulations which
always worked really nicely
and occasionally we'd actually try to
drive a robot arm
um i mean i'm not sure i could
fractionate it more than that i mean
that that's
those were the daily activities were you
still using the puma or was it something
no okay so
as a graduate student i started the puma
but
a guy named john hollerbach was very
interested
in something called model based control
and the claim was
what are called models of the robot that
essentially predict what forces to apply
to make a particular motion are too
expensive
for to use in real time
and he believed that was wrong and he
believed that we could demonstrate that
was wrong
but we needed a better higher quality
robot arm to do it
at about the same time that cmu here
was developing what's called a direct
drive robot arm
uh he thought that's a great idea let's
build one for ourselves
uh and we and that is the arm i used uh
for a lot of studies what was it called
uh we call it the mit direct dive arm
there might be a one or a two after the
name i can't remember which
and uh so what was the differences in
the direct drive
and how did that allow you to do more
precise okay so this
this gets a little technical
the puma robot and if you look at for
example the japanese humanoid robots of
today
each joint has a motor and a gear
or essentially for our purposes
equivalently something called a harmonic
drive
and then the joint and what that gear
does is has a gear ratio of 100 or more
and it's like your car or your bicycle
where you put it in very low gear and
you pedal really really fast
and the bicycle goes really really
slowly or the car goes really really
slowly
and the big problem with a heavily
geared arm is it's very stiff
it's easy for the motor to move the arm
part it's almost impossible for the arm
part to move the motor
that is completely different from humans
i can grab your arm or shake your hand
whatever
it's relatively easy to draw dry
people's arms we're back drivable
whereas these humanoids and the puma is
not
it also means that you don't really need
to accurately control the forces
or what the motor is doing because the
gears
there's a lot of sort of slop in what's
called the dynamics
so models are much less important
that's the advantage of heavily geared
arms that's one of the reasons the
humanoids have been so successful in
japan
but if you want a robot that acts like a
person
you've got to get rid of those gears and
so that's what we did
and then you can easily grab the arm and
move it around
but the downside is it's very weak
because it doesn't have gears so
we had to compensate for that by having
putting a thing on steroids
and having really big motors if you want
i can show you i still have the motors
it's kind of kind of frightening
so um so who were some of your first
graduate students
i should have brought my cv man this is
embarrassing
um okay
i had uh
a very successful undergraduate david
rinkensmeyer
i think he was the first guy to sign on
he's now a professor in the university
of california system i think
riverside when i'm or irvine i'm not
i'd have to go check so he does now
prosthetics and rehabilitation so he
shared this interest in people and
that's the direction he ended up going
i had a chinese woman named ying zao
she was interested in machine learning
which was another thing i was doing at
the time and
i jointly supervised her with a
statistics professor
she did very well i must remark
that she basically defended her thesis
had a baby and moved to california
basically all within a week
which was kind of amazing and she ended
up working at
for ibm out california
um i had an egyptian named sharif
beautros
he also was interested in humans work uh
he ended up working for a consulting
company in boston
i had a guy eric above
but we did some early learning work
something called task level learning
with him he ended up
saying okay i'll take a masters and
i went to business school and i imagine
now he's making a lot of money
somehow he's forgotten to send me the
checks so
anyway uh
tag i really need to have my cv to
get them all and i apologize to
everybody i'm forgetting
i had two very successful postdocs
one is andrew moore
and he was interested in machine
learning he was interested came
to mit interested in these memory based
models
and he actually after a couple years
with me ended up as a professor here
and now he's working for google
so you know he
is moving out and it's a real world
another very successful
postdoc was stefan shaw who you i
believe talked to
so i don't need to really say a lot
there he was also
i think fundamentally motivated by how
people work and
his work now reflects that um
i had a guy i co-supervised paul viola
interested in sort of how we see which
is
you might ask questions what the hell am
i doing advising somebody about that
since that's not what i do
but somehow it worked out um
i think that's all the mit folks but i
again i
really need to check my cv did you start
incorporating vision
into the system no no we ruthlessly
cheated
uh we didn't know where stuff was but we
really didn't we couldn't wait around
for the vision people to solve the
vision problem because it never will
uh so what we did is we marked things
and there are two ways to do that one is
very simple you paint things bright red
or
bright green or something and then you
you look in the color
channels of the camera and the bright
reds shows up in red and the bright
green shows up in green's vision is
pretty simplified
i actually had an undergraduate who
worked with me we she developed special
purpose hardware to do that really fast
and they went out and started a company
which was successful so that's nice
another way you can mark things as
bicycle safety tape
so you'd wrap things with bicycle safety
tape
and you shine a light at it and it would
reflect right back at you and the camera
was right next to light
and so it'd be really bright and that's
how a lot of motion capture systems they
used to make movies like avatar work
like that
we used it in real time to do vision
and you know 30 some years later they
still haven't solved the vision problem
and i'd still use those tricks
to do vision it's very sad
what other kind of sensors are you using
sometimes i need to know when i'm
touching something
and we we don't have a good way
we don't have good robot skin and we
don't have good robot tactile sensing
so a lot of my work has used what are
called
force torque sensors so you put
something in between your hand and the
re
and the rest of the arm and that
measures
the forces in different directions and
something called the torques in
different directions so
it's called a six axis force torque
sensor so that's a sensor i used a lot
what about uh measuring the position of
the arms and things like
that i'm sorry i sort of took that for
granted
um after this is over i'll show you our
current robot
it like everything else has a position
sensor in each joint
which we work with a company called
sarcos which is now sarcos raytheon in
utah
they love potentiometers so that's what
it has uh
if you want to be compliant it's often
very useful to have force sensing
on each motor or actuator so we have
that
as well and i'll show you that
you have a inertial system
and a bunch of gyros in your head and
that's what makes you sick
if you you know move in a way that
doesn't match your vision
for example so we
not to make the robot sick but to help
the robots know where they are and which
way is up
we put in the same sort of system of
vestibular system
for the robot so accelerometers and
gyros so what were the first robots that
used those or
the ones that you're using earlier
everything i'm talking about
has been around long before i existed
when was the first time that you used
those kinds of sensors
and your robots so then they might
i mean the puma robot had whatever
sensors had and we used a force torque
sensor on that
um
the force sensor on every actuator has
the idea has been around a long time the
humanoid we have now has it
so um what were some of the other
applications that you
you were working on at mit before you
okay like i said there are two kinds of
people people want to be useful
and people who just you know want to be
human
you know as much as i'd love to do
applications here are the tasks that i
do
paddle a ball up and down throw a ball
into a trash can
uh that kind of stuff so i would hardly
dignify that with what applications am i
going to do i don't do anything
what about juggling
okay so batting a ball is a
simple form of juggling again i wouldn't
really call it an
application it's not going to make money
uh
though yeah i suppose we could get it to
make money um
yeah so we did a couple things
ah let's see i'm trying to remember
so i talked about the ball juggling
there was ideological warfare about the
ball juggling would you like to hear
about that yeah okay
so we did a ball juggler
which was a classic example
of sort of the
let me use classic twice in the same
sentence that a classic ai paradigm
which is we see things we
reason we do things so in the bowel
batting case
the vision tracked the ball and said
okay it's going to land over here get
over there
tilt your thing like this you can get it
back and then hit it
oh it's coming down here move over here
okay
now we were very proud of ourselves and
the system worked
uh it is true that if you had a crappy
vision system and you guessed wrong as
to where it was going to go
you're going to do something screwy the
ball is going to fly away true enough
so there's a guy down at yale named dan
kodechek and he
i don't know how well like he's now a
department head of u-pen
or dean or something like that you can
certainly track him down
how can i characterize his view his view
was
well we we're interested in dynamic
systems in this business about
predicting
where things are going to go that's
really bad
um it's a little hard for me to explain
why he thought that was bad
because it involved having a model of
the world and all that kind of stuff
he'd rather have a system that had less
internal cognitive abilities so he
and his students developed a system
where yes you watched the ball
but you didn't think about how to hit it
instead you just he talked about
reflecting the ball through a
mirror so as the ball comes down the
hand comes up
and so his students developed this
system and then they basically said to
us
our thing is better and how is it better
uh well you know they claim first of all
the formalism behind it is more
attractive
uh and i think there was an implied
claim that was more robust to noise
well we uh we had to
deal with this so you know we sat and
thought about it for a while and said
you don't really need to see the ball at
all
uh because you have a ball coming down
if there's a ball that's too high it's
too much energy it'll come down later
right
that's too little energy you'll come
down sooner so what you want to do is
hit the balls and come down sooner
harder
and hit the balls that come down later
softer and then they'll all have the
same amount of energy and that's what
you want
so that says as you move up you slow
down
so you're going fast and the early balls
get hit hard
and then you slow down and the later
balls get hit softly
so we made our robot do that and voila
it could juggle balls without actually
seeing it was totally blind
so this is our retaliation one of those
guys from the kodacheck team came by
gave a talk at mit and we dragged him
off the lab he said look
here's our robot you make a big deal out
of juggling or
batting two balls simultaneously we can
bet as many balls as you want
simultaneously okay why because you
don't have to look at any of them
we just have separate paddles a row of
paddles and then they're all moving the
same way
and the walls are bouncing on each one
of them so we showed him juggling three
balls and
everybody had a good laugh and we moved
on but nevertheless
that was our little ideological warfare
and did you actually learn anything
about
was this outside of the view of how to
understand humans
so a deeper fundamental question
is to what extent do biological systems
choose strategies that minimize sensing
versus what i call the classic strategy
you measure everything typically with
vision
you model it you reason about it and
then you do something whereas this
you know slow down you don't see
anything you don't reason about anything
and the stuff works if we were to sort
of say
what happened later that sort of fit
that paradigm there's something called
passive dynamic walking
which is building these walking robots
that don't measure anything
don't see any you know don't reason they
just
do the task so you know the the deeper
intellectual thread
was you know what what's how much
sensing and reasoning do we
need or want to do what are some other
applications or uses of this and
manipulation and grasping like where do
you see this principle popping up again
so uh there have been people who studied
walking
that looked at what's called swing leg
retraction so when your leg swings
forward
they actually say that when you hit the
ground you should be moving backwards
for the same
reason if you come down early the legs
out
far and that slows you down more you
come down late the legs here
here and that gives you more energy so
that's just
the same kind of phenomenon
were there any other uh things of note
that you did at mit
before it's like asking
yeah what how is your life summed up
here you know okay
okay i mean i know you want to hear the
story so i'll just tell it to you
um so we're interested in the control of
very high degree of freedom systems
okay that makes them really complicated
hard to sense hard to reason about
hard to push around so cloth is a great
example
and so i had a student
get a robot to crack a whip because
that's a very
complicated high dimensional system if
you do it right it's kind of fun
break the sound barrier make a lot of
noise
and he did and then he came to me
and he was saying well how do i show
this off he said
you know if you're in a circus you put a
cigarette in your mouth and
sort of sit there and whip knock the
cigarette out of your mouth
i said there's no way that i'm going to
permit you
to put yourself anywhere near that thing
what kind of medical shirt insurance do
you have
okay so we compromised he wore a
motorcycle helmet with a cigarette
sticking out and the whip knocked it out
somewhere i have that video i'm not sure
where it is at this point but
it's a very amazing thing
um let's see what the hell did i do at
mit i think that's pretty good summary
but
i mean again you know in the academics
right i hand you my cv it's my list of
all my papers and
you know there it is that's what i did
any other
good parlor tricks
i need to get reminded of them okay
and so when did you leave mit in 1994
94. and you went to georgia tech yes
what kind of projects were you working
on
well similar sort of thing um
you know we were kept pushing to make
this memory based thing
more real
there was an intellectual war on about
uh neural networks at the time
uh neural networks were very popular
the statisticians were horrified because
you know they're hard to analyze and
formalize and i'm sitting here with
these memory based models
and i think the fundamental property
that the memory based models have which
the neural networks do not is
there's something called interference if
i teach you
how to play tennis and you get really
good at it and then i teach you how to
play squash
chances are you're you'll get confused
that's called interference and your
ability on tennis will go down
well memory based systems don't have
that problem right because they remember
one and the other and as long as you
figure out which
task you're doing your memories are all
there neural networks suffer
horribly from interference train them
over here
train them over here they basically
forget everything about here
so i was you know
fighting that fight at that time so
you're you're
against the neural networks that's
correct
why did you decide to go from mit to
georgia tech
i got an ultimatum
my then girlfriend future wife
got a job at georgia tech and said very
clearly and slowly
either you come to atlanta or forget it
so there you go what were some of the
differences between mit and georgia tech
at the time
in terms of the environment
it's not as hot mit is not as hot
i mean they're very different places uh
mit
is i think a much bigger place i might
be wrong about i mean in some ways
georgia tech might be a bigger place but
at least in
sort of the world that i was in there
there was a whole lab of mit
ai lab for artificial intelligence and
at georgia tech there was
there was a group but it was like three
people or four people or something
and the computer science department was
similarly
much smaller uh there was no
comparable brain and cognitive
department
uh so it was a smaller place absolutely
what was the robotic situation like when
you got there
were there a lot of roboticists or any
robots running
there were a couple people there was a
guy in mechanical engineering wayne book
who was doing a lot of stuff
ron arkin was doing a lot of stuff um
but it wasn't organized
one of the big wins of the mit ilab was
that there was
so mit had four
robotics or robotics relevant places
but the mit i lab was a big
concentration
now ultimately cmu beats everybody with
having a much bigger
critical mass in the robotics institute
but anyway
in terms of comparing it mit and georgia
tech
the you know they had critical mass in
the ai lab now that dissipated later
you can ask them why that happened but
um
and interesting enough georgia tech
after i left
said hey we've got to really get our act
together they hired a guy named henry
christensen
and now they have a robotics i don't
know what they call it center
or entity or something
so you point to the places that have
made the commitment to robotics georgia
tech is definitely on the list
but it wasn't when you just got there no
no that was a long time ago
so who was there when you got there
you said that there were a few people
that were
wrong well there was ron arkin
my future wife jessica hodges was there
or i'm sure there are other people i'm
forgetting does she work in robotics
also
she did you're gonna really have to talk
to her to get her story
but she was a student with a guy named
mark rayburn
is she still working in robotics i think
the most
accurate description is she's come back
to it
okay and she's here in the faculty
yes her offices over there
she's in europe right now so okay
so what did you start doing it when you
came to georgia tech
what were some of those okay as an
academic you just
move you don't really necessarily change
what you do
right but what kinds of what kinds of
tasks did your robot do did you bring
your robot with you
did you have to get another one
i think we're i'm a little more
abstracted
at different times i've had different
robots there isn't any
robot or you know a lot of stuff i do in
simulation
um one of the things
in fact one of the things that happened
around the time i moved to georgia tech
was i started
spending a lot of time working in japan
and in fact
what happened was that the my
collaborators in japan
invested in a lot of robotic stuff
which i then worked with or helped them
work with
who are you collaborating with a guy
named mitsuo kawato
a place called atr and so you were
working with the dynamic brain robot
yes as a matter of fact i was
that was actually not the first robot we
worked with
um what we did is we went to
this company sarcos and
said okay how are we going to get to the
point where we have something human-like
which at that time was defined as two
arms and a head
and some kind of torso and sarcos
i believe wanted to reuse technology
they'd already developed
they had a robot that they had used for
actually
i believe power line repairs or
something like that
and it was pretty bulky and heavily
heavy but it
did what we wanted and i think stefan
has it now
um and so we had we started out with one
arm
and we used cameras to figure out where
stuff was
and we did a lot of good stuff with that
and then db was actually this
the next generation and so did you have
more of an input in terms of how to
design db to make it
amenable to the kinds of things that you
were studying
have you ever worked with an artist no
or commissioned art for your house
i'm not that rigid yeah okay so you walk
up to the artist you say
you know i really think it should be
more pink
typically the artist says yes of course
and then doesn't do what you want so
there is that element to our
relationship with sarcos that you know
they
they had a vision we certainly gave them
input
and then they you know tried to make the
best of it they could but they did have
their own vision
and they were also basing on existing
technology it's very expensive to make
technology out of nothing
how did you initially develop your
relationship with sarcos
i inherited that from john hollerbach he
had worked with sarcos on something
called the mit utah hand
there may well have been previous
projects before that
uh but you know the mit utah hand was
you should talk to hollerbach and or
steve jacobson about this
but it was an attempt to really push
hand technology
and so jacobson was also he was head of
suncoast yeah it was his company
okay and so db was also was he partly
the
artist that you're talking about yes
i mean you should really talk to them my
understanding was that they had done
what are called animation figures for a
couple companies
uh and db was a modified version of
those
animation figures and after we're done
here i'll take you over
and i'll show you some of the technology
help you understand where it came from
and so what kinds of things did you get
tv to do i mean i've seen it
for example playing was it playing what
are those things called
air hockey okay so that was a student of
mine during pennsylvania
there's a whole host of stuff if i have
my laptop i just start running videos
um i was interested in
tasks that people do dynamic tests
um we had done ball batting
and that was nice but when i say
juggling that isn't what people think of
so we got a step closer to that
by doing what's called three ball
juggling
in what's called a shower pattern that's
like always getting confused anyway it
was symmetric and the balls were
crossing over rather than the
asymmetric pumping with one hand yeah
exactly
um now
there are some facts of life one fact of
life is
hands are heavy and we didn't have any
so i had to cheat on that so i used
kitchen funnels
um so it's not quite where i wanted to
get to but it was close
and you know the thing you know
juggles three balls we were interested
again in the role of sensing
so we did it blind and that worked
because of the kitchen funnels
and with vision and that also worked
surprisingly it was more reliable
blind than it was with vision
which was a surprise to us
there was another task called devil
sticking
which is you hold two sticks you sort of
batter center stick
back and forth which is a very nice task
to study it's it's clearly
dynamic uh yet you know you never grab
anything
you're always on the move so we did that
staphon did a whole host of things which
i hope he told you about
there was some learning to balance
pendulums
learning to do something called swing up
so there was an interest in learning
i think that's what i remember at this
time how did you meet
kawato
he's uh i think about five years older
than i am
uh we share interests
in how a
robotics or an engineering approach
can be used to sort of understand how
the brain works
and because we basically had that
interest
we met each other at conferences and
things like that
what were some of the conferences that
jointly attended or i believe
so easy to be caught in an error here
because confidence might not exist at
the time
uh i believe there are two important
conferences one was
nips neural information processing uh
that at the time was pretty much
always in utah yeah snowbird
and then six months later there was
something called
snowbird which is a very similar
conference
i don't know what it was really called
uh
and i think for a while both of us are
going to that
i'm sorry who were some of the other
people you collaborated with um
i'm trying to figure out how strongly to
apply the word collaborate
you have collegial interactions with all
sorts of people
you know you say things to them in the
hall um
i think in the robotics end of things i
did my own thing
i had a huge collaboration with the
japanese folks spent a lot of time there
because i didn't have a lot of hardware
at georgia tech
and also i was kind of interested i
developed a
new research area for me
that you could argue is not robotics or
you could argue it is robotics
and that is uh i think the simplest
thing to call it is intelligent
environments
um it's mostly about sensing
it's about using sensing to figure out
what people are doing and how you can
help them
and i had a lot of collaborators there
and in fact we
ended up building a group
a key guy early on is
gregory above
and you know we had a
uh aaron bobek join us from mit
beth meinet um
i'm spacing on some of the others
but if you send an email let me give you
a more complete list what were some uh
intelligent environment projects that
you weren't done i think
the main one the one with okay there are
two main ones from my point of view
one so i teach for a living right
and i would like to make that work
better
and one of the things that's always
amazed me is
that a classroom lecture
can hold people's attention but the
exact same lecture videotaped
is incredibly boring
what's the deal you know do we need to
smell people
to to be engaged
uh it was a total mystery to me
so i in the end went totally hog wild
uh on let's capture everything let's
pump it out to the web
which was just happening at the time and
uh let's uh you know allow students to
skip lectures
and then feel they were really there
as best we can sort of the sort of max
technological approach
so the culmination of this effort i
would wear the robe of knowledge
which was a bright orange rope it's like
a highway safety vest
and that would make it easier later to
figure out where i was what i was doing
we would have four cameras in the room
some zoomed in on me some zoomed in on
the boards we had electronic boards or
everything was captured
and then i would wear cameras looking
out so you could see what i was looking
at and then there's a camera look at my
mouth so you can read my lips
um yeah
and the amazing thing was people still
came to class
even this crazy guy was up there you
know sort of technology
exploding if you've watched uh the star
trek uh deep space nine
it was like the board you know with the
things sticking out your ear and all
this other stuff
did any did people go to the virtual one
yes at least i think they did
um it's an interesting that is you're
asking a very interested
evaluation question uh to be wholly
honest i was never interested in
evaluation
i just wanted to do it it turned out
actually that people's parents would go
to the web
and use the camera on my head to figure
out if their kid was in class
and if they're not that bad things have
happened
so you opened a different kind of
privacy related
it had never occurred to me that that
was an
issue but
if you were interested in the in this
issue of presence and
what makes people feel like they're you
know really
right they're really there i mean did
you evaluate it in any way
did you oh i had other people that
collaborate that's what collaborated
before right
so what happened what do you mean what
happened i mean what did it work
everybody was very enthusiastic yeah of
course it worked
it's doomed to work
i'm very cynical about this kind of
thing okay yeah no because it seems to
me like it would be this huge
information overload
oh no everything's cross-linked right so
you could sit there
you know it's like the old movies where
the little dot is bouncing on the words
and you sing along
you're just watching the lecture you've
got video here and the guy's talking he
got zoomed in and what he's writing
you know the little words are skipping
by
you can make a very integrated
presentation
it doesn't smell
and you know maybe it's the element of
fear
at any moment the lecturer can keel over
that's what makes it exciting
or they can get angry or
do something unpredictable
i don't know there's also the other
people
really they shame you into staying awake
yeah maybe maybe you have teams of
people watching this web thing
okay so that was one thing
the second big project and this was
actually a really big project
is we built a house
uh and it was very fortuitous
the people interested in wi-fi and radio
were very interested in having a testbed
that would allow them to figure out
you know how to design better wi-fi and
radio for computer networks
and we latched onto that and said well
this would be
a fantastic place to do experiments on
people
with sort of futuristic technology how
they reacted to it and stuff like that
and if we could get rid of the lawyers
we could even have them live there
turns out the liability issues go
through the roof
if they spend the night
so during the time i was there
we never had anyone legally live in that
house a lot of people lived there
illegally
but nobody legally lived there what was
the house called
the aware home
so was this one of the first kind of
living labs that were
it's a type of living you can actually
if you look historically there are
been a lot of people who built houses
for research and you know if you if
you're
willing to include you know world
expos and stuff like that people build
houses to demo stuff all the time
there's been a long history of solar
power houses and stuff like that
so i'd be very reluctant to say this was
the first of anything
i think at about that time i think a lot
of universities
jumped on the let's explore
how futuristic technology can help old
people
and i have no idea you know who is first
i have no idea
and so your focus there was also on the
elderly yes yes it was
very explicitly we're going to help
improve the lives of all people
what were some of the sensors and
technologies that you incorporated
so i spent a lot of time building
infrastructure like you know how would
we put
100 cameras in a space like that and
collect all the data and things like
that
we had vision people who would actually
do the processing on the data
we had a set of people interested in
interfaces
and actually probably doing very simple
things that turned out to be very useful
so one thing that'll get you into a
nursing home really fast
is if your adult children lose faith
that you are safe okay like
you burn a hole in the floor so the
house i bought here had a big
burnt hole in the floor and very clearly
the guy who lived in it
who you know he's a widower he lived
alone he's probably i don't know how old
he was
he was clearly shipped off to the
nursing home a minute that happened
um so
one very simple thing you can do is have
a house that reports to the adult
children
things are okay mom got up this morning
mom seems to be doing the normal pattern
of room-to-room movement
and stuff like that you can go
a step up from that you've seen on tv
the life alerts which
rely on you have to press a button
and then help comes often people are old
people are too confused
or too injured to actually press that
button maybe the house can press a
button for you
it's like onstar so did you
were there particular insights that you
got
oh did you did we learn anything after
spending approximately a million dollars
of the georgia taxpayers money did we
learn anything
yes i think we did
we go for launch yeah yeah
okay
one of the things we struggled with was
the issue of privacy
and as far as i'm concerned people have
very strange attitudes about privacy
if you walk into a room that has a lot
of cameras bolted to the ceiling that
you can see
a lot of people react very negatively
they don't like it
interestingly enough in georgia tech
every single classroom had such a camera
uh because uh you know they had a
vandalism or security problem so that
we lived in a world that had everything
in a camera
so we spent a lot of time trying to
figure out how we gonna get people to be
happy to have
monitoring technology in their lives
because if you if you want a servant to
take care of you and the servant is
deaf and blind that servant is not very
useful
well in the classroom example i gave you
where we recorded everything
the students were totally happy to have
a gazillion cameras
because they saw an immediate benefit so
you can sell people
on an immediate benefit uh it's a much
easier thing
and what you know the tsa's problem is
they can't convince people there's an
immediate benefit to these strip
searches
oh well a second issue
is people's perception
can be sort of shaped
for example if i set up some cameras in
the roof here looking at you
you have one reaction if i rolled in a
mobile robot that had eyeballs
the same cameras wouldn't react
at all why that is well i speculate
that you know you're sitting there
thinking well the robot needs to see
so of course it's going to have cameras
and then you project all sorts of
human-like characteristics
like it'll forget and it doesn't store
stuff and stuff like that but that is
nonsense
the cameras in the ceiling and the
cameras in the robot are exactly the
same
and have exactly the same system behind
them
yet people's willingness to have them in
their lives
is totally different so privacy was a
big
issue the second issue was just
how are you going to handle this
mountain of data and we
developed techniques to do that
what were some of those techniques like
to manage all the data flow
they mostly had to do with figuring out
what the big problem was
and then compressing like worrying about
the big problems not small problems and
then compressing like crazy
and there's a lot of analysis you can do
in real time
and then you compress the vision data
for example or you throw it away
were there other kinds of sensors
besides cameras
microphones were very helpful we would
have
okay so everybody in
our day and age knows what gps is very
excited about
it but they don't often realize is the
gps doesn't work indoors
plus it's not very accurate we would
love
to have a gps system that worked indoors
and everybody had a little tracking
thing on
that would make life so much easier so
we spent a lot of time
basically buying systems that promised
that and trying them out and then
discovering that
they had problems
were there any controllers of the
environment as well or just sensors
okay there were things like
dim the lights i don't know what level
of control we got over hvac
but our interest wasn't really in that
aspect of smart houses or controlling
those things
our interest was in monitoring the
people and understanding what they were
doing
and there actually was no robot in the
house either
and so did you figure out something
about human dynamics what were the
particularly
interesting things that that you looked
at
so again on the privacy end of things
this is actually a project that happened
here at cmu
um we asked the question well how
simple can the sensors be and still
figure out who is who and what they're
doing and
you know most people are familiar with
motion detectors
and that doesn't bother them
and they're happy measuring when doors
open and closed when cabin is open and
closed when the toilet flushes and stuff
like
that so these these are referred to as
binary sensors
and then you can ask the question well
with all these binary sensors can you
still figure out who's who and what
they're doing and the answer is
yes if you use sophisticated machine
learning techniques
so in reality we probably don't need to
put cameras in there
so you can just learn people's patterns
and behaviors correct
and not allow you to recognize them
so i had a student named daniel wilson
uh
and he instrumented his house
with a bunch of sensors for his thesis
work
is this the daniel wilson of the goofy
book
what was it called okay survive a robot
yeah okay you guys there's no no no no
no
you are totally missing the bus
this is the daniel wilson who is my most
successful and probably most famous
student
in addition to the fine books that you
mentioned
he has written four or five other books
his latest book to be released in 2011
will be made into a movie directed by
none other than
steven spielberg so as a measure of
success
he's far more successful than i am or
any of my other students
in terms of reaching people and getting
a message out
now i agree with you it is unfortunate
that the message is mostly revolves
around
robots attacking people
that's not good press but you know okay
that's his thing he's doing it or the
robot is being actually
quite easily incapacitated with small
well okay that is what he took away from
being in the robotics institute
and it's not inaccurate it's not hard to
incapacitate a current robot so how did
you get to cmu
from georgia tech uh
i followed my wife
they called her up said uh
yeah can you help us get computer
graphics going again
and uh in the end we came and
you know see did that and
here i am how did you meet her
initially you guys getting real personal
he seems to be directing your career
yeah sure that's life
you know opportunity fate you know
stuff happens that's how it works um
so a hundred years ago
back in the dawn of time i
had to get a thesis committee mark
graver was on my thesis committee i
visited his lab
to tell him what my thesis was really
about i think that's when i first met
jessica
i hope so yes
they're gonna scratch this for the
record yeah
and uh it can only be used against me
this is really why we're doing it yeah
we got to this point here yeah go yeah
it was love at first sight or uh did you
start dating right away
no no no it took a while yeah
and then but how many years were you
together before you followed her to
georgia tech
eight
back to cmu so when you get to cmu i
mean
you mentioned you know first you heard
of mit there was a lot of robotics
georgia tech
not so much but you were still able to
work on it and then here
you came to cmu and then there is
obviously a great deal of
robotics activities right how did that
affect
okay the thing cmu did that these other
places
didn't do although georgia tech has now
done it is you know they made a
commitment to robotics they said you
know we're gonna
instead of distributing in around a
gazillion different departments computer
science we can't go doing
electrical engineering aeronautics and
astronautics operations research
wherever else
we're going to concentrate in one place
uh and they made that decision i guess
1979 or something i
i don't know exactly but that was fairly
early on
um so clearly
getting here you know suddenly wow
a lot of people doing robotics
and you know not only does that make you
feel good but uh is
there a lot of people you can get help
from or talk to or colleagues
things like that who are some of the
ones that are
that you interact with the most
so at my age one of the
very important activities is helping
young people
get going
so i we have a system of older people
mentoring younger people i mentor
something like somebody named yoki
matsuoka
who's now at the university of
washington i meant towards something
somebody named james kufner who is now
at google
uh i'm mentoring somebody called hart
mcguire who's
we decided recently he's still here um
and i spent a lot of time talking to
them
uh in addition to the sort of humanoids
work i also because of the aware
homework got involved in something
called
care media which was a spin off of
something called infra media and
basically digital libraries
so they were interested in if you
collect buckets of data how do you make
sense of it and they
they were the care media was well if you
collect buckets of
video and audio and data and stuff like
that
from real environments how do you make
sense of it and our emphasis there was
going
into uh nursing homes and helping take
care of people with alzheimer's disease
and stuff like that
uh i helped
get together a what i'll call a training
grant
for people in that area from the nsf
and more recently there's been a bunch
of us who've done
something called it's a national science
foundation
engineering research research center
which is one of their big deal grants
in something called quality of life
technology which is again
how can we use technology to help people
so you are actually now more on the
application side
in a sense as well
i guess i mean i still also have an
activity in humanoid robotics
and for the humanoid robotics i mean did
you go to
did japan end up being this big
collaborative collaborator because
specifically your you knowing
mitsuoka or was it is there
is there some reason that you're doing
this specifically with japan
you got to know who you're collaborating
with you have personal connections
right right but you know you're they're
i don't know they're humanoids
in a sense in the us as well so what is
it no they're not
there's nothing like it but the sarcos
humanoid that they're using
the other two are in japan
they're really i mean i don't want to
okay all you folks out there in tv land
i don't want to insult anyone but there
really is not that much going on in the
united states
in terms of humanoid robotics it's
nothing like japan
and you know korea i have a lot of
interaction with koreans
coming here
and korean companies and
they're working hard to catch up and
they've got a lot more than what's going
on in the united states
what are some some of the centers of
robotics research in korea
it's very confusing so they have
something called kaist
korean advanced something
then they have something called kissed
both of which have efforts they have a
lot of big companies there some of whom
have efforts
um and then there are other places as
well
i'm curious so if japan japan obviously
has a lot of
humanoid activity like in terms of
designing them there's the theta
there's all these places yet the
the in in atr they're using
the sarcos humanoid why not probably
the only japanese lab that's using a us
thing right why did they end up using
the us thing is it
mainly your connections or is it
i think it's partly due to my
connections but it's also due that we
wanted a human-like robot and these
heavily geared electronic things are
just not
they're just it's very difficult to make
them human-like
okay so i mean they're useful but
they're not
human-like and in your estimation db
would be
is probably kind of the most um
convenient platform to do the human-like
okay so
db has been retired okay and i'll show
you
the new one the new one why do we have
it
we can make the joints compliant we have
higher max velocities at the joints
and we have bigger forces at the joints
all of which by the way are still not
close to what humans can do
but they're closer than any other robot
i know about
what do you consider the biggest
accomplishments or contributions that
you made
on the algorithm side the work in memory
based learning
i have uh
let's see
a lot of people my age are managers
okay they don't do anything they manage
projects
early on i figured out that that didn't
make me happy
so i've always tried to maintain some
activity where i
actually program and unfortunately it's
very hard to have time to work with an
actual robot so i
i i program so i'm actually very proud
of my work
in planning for dynamic systems
which has sort of been my side activity
to keep me happy
all these years
and i'd be happy to launch into a spiel
about it so it's like
having a drunk guy in a bar talking to
you can't shut the guy up
uh but only if you want to so i'm proud
of that and i
i'm also proud of the stuff that we've
done with these various
uh robot systems which you know the
the key difference between them and and
everything else is
these are compliance systems
and so in some ways that makes it more
human-like
in some ways it makes it harder actually
because if something goes wrong
everything goes wrong
i'm planning for dynamic systems
okay there's been tremendous
progress in what's called robot motion
planning
and getting complicated robots
to move slowly through complicated
geometry
and that's a tremendous accomplishment
what we don't know how to do is find the
best path
under those situations and also
how to be more dynamic move faster
forces matter so
i've been spending a lot of time trying
to figure out how to do that
and i think the best way to do it is to
combine
a lot of greedy planners
together in such a way that when you're
done
you actually find a globally optimal
answer
and what does that mean it means a
greedy planner means you go straight to
the goal so if i'm here
and you yell fire i run straight to the
door there
but if i'm there and you yell fire it
means i run straight into the table
that's not a very good plan
but if we had a bunch of people all run
to the door
some of them would make it then those
guys
we could think about how they catch up
with the people who made it
so in the end by combining a lot of
fairly stupid
planners we can get a plan for everybody
that's pretty good
so you know that sounds good but the
challenge is to make it real
to find the final question so our final
question is
um if you if you were
giving advice to somebody who's
interested in
getting into robotics these days what
would your advice be
so i think we have to separate out the
robotics part of that question
from the what's a good life strategy
part of that question i think the good
life strategy is you should do something
that you fall in love with
you should just do it whether it's a
good idea or not it could be a terrible
idea just do it anyway
that's my engineering applies to young
people
um i mean there's something we didn't
talk about about robotics and that is
suppose you walked in the door and said
i wanted to
make fundamental progress
what should i work on in robotics i
think we have to recognize that robotics
is a kind of screwy field
in the following way
i drive a 20 year old car
there hasn't been that much progress on
the mechanical side
of robotics we
have been building new mobile robots we
use something called the barrett arm
from vera technologies
that was an arm that was developed when
i was a graduate student
okay that's the beginning of the 1980s
so if i have my math right that was 35
years ago
we're installing a 35 year old
arm in on our robot
and saying state of the art
contrast that to computers which is sort
of the other half of robotics
the idea that you would have a 20 year
old computer or a
35 year old computer is a joke
we throw away computers after a few
years because they're totally obsolete
so on one side on this maybe the sensing
and intelligent side
you have this mind-blowing explosion of
resources you know the computer stuff
just
blasting away
on the other side we have slow or almost
no
progress on the mechanical side so
someone coming to me for
career advice i say well you got to
recognize that it's unlikely that you're
going to make 100
improvement on the mechanical side
because you know lots of smart people
have been beating themselves
to death with these same systems and
it hasn't happened on the other hand you
could be an
idiot on the intelligence side
but because computers get faster pretty
much whatever you do is going to be
better
that's a good idea or not
so i just point that out to them
is there anything else you'd like to add
i don't think so