um so i'm just going to ask you to start
with
a little bit of your background on where
you were born where you grew up
and how your education started so i'm a
child of world war ii
i was born essentially three months
after the battle of midway
on september 4th 1942 and
it had the war of course had profound
implications for the world but
for me in a very selfish way putting
aside all the geopolitical implications
indeed the way the war turned out if it
had turned out differently we wouldn't
be here having this interview so
uh that itself is uh
is a point to ponder but uh
what was really important for me was the
fact that when the war ended
uh it unleashed on a civilian
marketplace an enormous amount of war
surplus radio equipment
and as a ten-year-old okay growing up
outside new york city
i could go down to a place called radio
row which was on a
street called cortland street where they
eventually built the world trade center
and i could go into this warren of dusty
buildings
and buy dirt cheap war surplus radio
equipment
now it had the following implications it
was all vacuum tube electronics
all right and that was very important it
turns out because it made it possible
for a ten-year-old to understand the
underlying physics
all right that explained how any of
these devices work now
uh admittedly it's probably hard to
believe but i used to have hair on the
top of my head
and when i had hair on the top of my
head i could take a comb and run it
through my hair as a 10 year old
and i could charge the comb with a
static charge and i could let water
flow out of a faucet gradually and cause
the water to deflect due to the static
charge on the cone
and this is basically the principle on
which electric fields interact with
matter
and how one can think about how a vacuum
tube works a vacuum tube amplifier
where you have an electric a weak
electric field controlling a strong
current and you get
amplification so as a 10 year old i
learned the fundamentals of
radio electronics and as time went on i
got a radio amateurs license and went on
to
got more and more interested in
electronics and went off to cornell at
age 18
and studied electrical engineering so i
was at cornell for eight years i got my
bachelor's master's and phd at cornell
and then i went off to yale where i had
a postdoc
and then eventually i an assistant
professorship for a couple of years
where i taught control theory
and electrical engineering subjects and
i did the same at the university of
illinois for a couple of years
until i came here in 1974 where i've
been
ever since so i'm starting my 38th year
here at penn
and i started out in what is now called
the electrical
uh systems engineering department it was
called something else uh
several years ago and uh eventually uh
in meeting uh a person who became very
pivotal in my life
rooja nabaishi uh she enticed me to
join the grasp lab and turn my interest
in control theory and decision making
under uncertainty
in the direction of robotics so uh
and that would be 1984. so i
i joined the grasp lab as a professor in
electrical
in what was then systems engineering now
electrical and systems engineering
and uh two years later i actually joined
the department of computer and
information science
and switched my departmental affiliation
so
in 1984 i joined the grass lab in 1986 i
became a professor in the
cis department and i've been doing
robotics and related research and
decision making under uncertainty
machine perception in robotics is the
way i refer to it decision making under
uncertainty i should say
in machine perception and robotics and
that's what i've been doing
essentially ever since what was your phd
work
my phd work was in essentially control
theory
uh with stochastic systems i was doing
work in stochastic control theory
uh which is something we use today in
robotics but
we use in a lot of other subjects as
well and your postdoc research
my postdoc was all related to that in
fact until i got involved in
the grasp lab all my stuff was basically
doing
control theory the mathematics of
control theory
and uh so what years were you at the
university of illinois uh 1972 to 1974 i
was with the
csl group and the department of
electrical
engineering do you ever run into heinz
fund forester and
biological computer though i certainly
knew who he was
but i never did get to meet him yes
i did some work on my dissertation
so um what would you say was your first
robotics project
well i to understand my interaction
my in getting involved with projects in
robotics one understands that when
i did it collaboratively i can sit and
do my
own thinking about mathematics and such
but
when you want to do things with robotics
it's not a bad idea to have some
extraordinary graduate students
okay and the grasp lab in some sense
is the home of extraordinary graduate
students and i'll name a few
okay who i had the pleasure to interact
with and it was my good fortune to uh to
have this interaction
so uh first was a guy named greg haeger
who's now a professor in computer
science at johns hopkins
and greg got his degree as i recall i
think it would have been 1988
here at penn and i as i say i got
started in the grasp lab
in 84 but as i when i joined the
department in
86 i started supervising students
okay and greg was my first phd student
in
cis uh following that
okay uh there were students who did work
either in robotics or worked in decision
making under uncertainty
so i'll say a little bit about what
greg's work was about and i say it's his
work i
my job in life as i tell people is to
supply the applause
okay the students who do the brilliant
work and i get to
make you know hopefully helpful
suggestions from time to time but
the students are the ones who are uh
really quite pivotal in making all this
happen
so greg in try to summarize it briefly
greg's work involved seeking information
in other words we have sensors in the
world
and we gather information through
sensors and one really can't do
robotics today without thinking about
sensors okay sensors are
extraordinarily important part of the
story and the importance of sensors
has grown i would say exponentially
in uh in the time when we started doing
a lot of this work in the mid 80s
okay to the present day where now
sensors are everywhere in sight
okay and not just in robotics all over
the place
all right so that's a story for perhaps
another day
but uh in any event uh greg's
research idea was to
work out how we acquire information
in an intelligent fashion by using
sensors so sensors
don't want to remain static you don't
want sensors just to remain static in
the world
you want sensors to be controlled in
such a way that you go out and use
current information
to get even better information so it's
how do we
search for information that's helpful
for solving a problem
you're going to grasp an object you need
to know what the object looks like
and if you just look at the object from
one position you won't get enough
information so you're searching for
in an intelligent efficient way good
information so you can achieve some kind
of
task solution which would be say
grasping an object moving an object
or even just identifying an object by
looking at it from various
points of view so greg worked on that
problem
and there's a lot of interesting
statistical decision theory that goes
into this
and uh he and i had a lot of fun
uh working on this um other students
worked
on uh actuation where greg's
greg's where where there was robotic
actuation there was a robot arm with a
camera
okay so the robotics was a robot arm
with a camera
okay as opposed to a mobile robot
running around
in the grasp lab achieving some kind of
other task solution
so in in the case of say robbie
mandelbaum who
graduated in 1995
he robbie's now the chief technology
officer at
lockheed martin after he graduated he
went to
sarnoff research in princeton
and then later left sarnoff and
went to darpa okay and after he
left darpa he uh joined lockheed martin
where he is presently
so uh greg had a couple of postdocs to
go back to greg haeger he had a couple
of postdocs
went to yale after he was after he left
yale he went to johns hopkins where he
is presently
in the department of computer science uh
so robbie uh would have been the first
student i worked with where we actually
had mobile robots
and uh i smile when i think about the
what mobile robots looked like in the
early 90s to 1995 uh and this is uh no
joke that
uh we uh had very clunky platforms okay
uh they were called trc lab mates and
they were big heavy uh ponderous objects
that could heft a very large payload but
were pretty hard to control indeed the
control for
what we did with these things where the
students did
really happen by having a off-board
computer
connected to the robot using a serial
line
okay so to understand research
then versus research today the big
breakthrough the big
change that has occurred in the uh
nearly 20 years when this whole thing
was going on
the the fundamental word two fundamental
words miniaturization and wireless
okay now things can be done very small
computers
okay powerful calculations are possible
powerful computations are possible with
very small devices
and enormous amounts of information be
gotten with sensors
and all of this stuff can be done
wirelessly so now we don't have
the embarrassment of having a serial
line
connecting a desktop computer to a
mobile robot and you can only
laugh when you hear some of the things
that would occur when the uh
serial line ran out of slack and the
line uh
effect parted from the from the pro from
the uh robot itself it was
it was uh well amusing in some sense and
uh
not so amusing in other senses do you
remember any particular cases
uh well if you promise not to publish it
too widely i'll say that
i was near one of the computer one of
the robots when it happened and
they i had to shall we say be nimble to
get out of the way of the machine and
hit the what's called the panic button
or mushroom okay to shut the thing down
okay when the the thing basically uh
lost its computer control
so uh this was uh this is something i'm
not going to forget
uh although i'm not not proud of the
moment okay it's a
it's an amusing uh moment now uh today
as i say
everything has been miniaturized if you
go into the grasp lab
you'll see uh people working on devices
that are very small
okay that do amazing things that we
could
not have even dreamed of doing uh back
in 19
in the interval of 1990 to 1995.
so you need to go into the grasp lab and
see some of them quite quite wonderful
things they're doing with these
quadrotors
that can fly through hoops all right and
do really intricate maneuvers all right
so the control
okay is exquisite and uh it's done
wirelessly
and it's done with uh very very
intricate sensors
okay that we couldn't uh couldn't
imagine having
at least i didn't imagine having let me
say it that way
in the interval 1990 to 1995. so in some
sense my research program had been about
decision making under uncertainty in
machine perception and robotics
and that included how to understand how
sensors work
how to understand the uncertainties in
sensors how to use different sensors
together
okay in a fashion so they complement one
another
so that i get more information okay
and it isn't just that i take the sum of
the two but i combine them in an
intelligent way
to learn more about the world which is
what i need to do and to do so using
the underlying fundamentals of
probability theory and
statistics in a reasonable way so that
where we go out and actually measure the
uh the real
uncertainties about the sensor behavior
as opposed to just making wild guesses
so uh the students in the lab in that
era would tell you don't talk to max
about gaussian he'll tell you you have
to go and establish that the
noise really is gaussian as opposed to
just making a uh
a broad assertion so uh i have a there's
a joke in the lab it probably still
exists okay that i'm the guy who
was a pain in the neck and would would
raise that point uh
perhaps too often so uh
the the story of decision making under
uncertainty is important
but as sensors get better and better and
better okay
now the problem transforms to huge
numbers of sensors
how do you organize them
in a clever way you don't want them all
reporting to a single
central processor you want to do things
in groups you want to do things
cooperatively you want to do things
independently so there should be some
intelligence embedded in one
collection of objects and intelligence
embedded in another
they do their own thing separately
communicate where needed
okay and the strategy of deciding when
to communicate and what to communicate
these are still open research problems
all right
and what's happening is as you get more
and more of these sensors
all right the problem becomes uh very
very very very
uh i wouldn't say well complicated is
the right word
but requires
requires a careful analysis of what
one's doing
in other words one is not beating the
problem to death with a hammer
okay one needs to think cleverly about
how to put these things together
that's what the folks in the labs lab do
today i'm
somewhat backing away from it truth be
told i suppose i can admit this to you
although you're interviewing me about
robotics
i've developed over the last uh 10 plus
years
a new love that has a little to do with
robotics it has to do with quantum
computation
so today i spend most of my time
thinking about quantum computation
and how i'm going to teach that to
undergraduates which i've been doing for
over
11 years and how i will you know
continue to do this as
it becomes possible someday to get
quantum real quantum hardware
as opposed to just talking about uh
mythical machines
so in a sense my interest has always
been in mathematical physics
at a very simple level how things work
in in a robotics environment and i don't
mean
trivially simple robotics physics is not
simple because you have complex
interactions
with the world just to make a sort of a
trivial point
when we first started getting little
robots that would roll around on the
floor and i was
working with a grad guy who eventually
graduated from penn doing working
machine vision
but as an undergraduate and as a first
year graduate student worked with small
robots
okay we used robots that would roll on
rugs in the lab and the rugs if you look
in the rug in my office here
you'll see it has different behavior
it's not isotropic
and so there's a directionality to the
weave that affects the way the wheels
would slip
and so the physics is by no means simple
the physics of the interaction
is in fact quite sophisticated one has
to work on that to understand
what that does to the odometry because
odometry is
is a is a place where one can get into
trouble if one's trying to figure out
where you
were what position the robot is in from
merely the rotation of the wheels
where slippage could occur so the
slippage issue is a non-trivial issue
and
we learned that through lots of good
experiments and try to understand
what can go wrong i guess in some sense
my job in the lab were to break other
people's experiments
and i mean that in a positive way to
you know find ways for things that might
go wrong
find out where things where they're
sensitive to uh the word that we used in
those days which is now gets overused
today is the word called robustness
okay robustness means that you want your
system to work
in spite of your assumptions okay where
you've made some assumptions the
assumptions may not be entirely right
you want the thing to gracefully degrade
as opposed to fall off a cliff
when the assumptions are no longer valid
so robustness is something that
we strive for in a variety of ways and
uh
still very important today and becomes
uh
when you put huge numbers of systems
together in a complicated fashion
where your networking systems but the
network doesn't mean a totally connected
network
where there's local communication local
communication or local communication
and then the occasional communication
between the local groups
okay one has to be one has to understand
the robustness issues loss of signal for
example
i move behind an object where i don't
get line of sight
okay and i might be trusting on the
possibility of having line of sight
people are using radio communication now
something we never did in 1990
all right and trying to understand the
nature of how to connect objects
together
using radio signals
so you formulate your work and the
probability theory of stochastic models
um
did you draw on biological models
economic models and
you mentioned some physical okay so um
others did
rojana baishi always drew in biological
models and
so uh you ask economic models and of
course people are interested
in when you talk about teams and such
team theory which was
which started in economics okay is
something that people brought into
robotics
okay uh so my connection
okay is somewhat more limited i'll say
how it how it happened
to decision making under uncertainty uh
it sometimes pays to use mathematical
game theory
because it turns out there's a entire
subject
of decision making under uncertainty
called statistical decision theory
which has game theory built into it it
goes back to the
1940s with a guy named wald who
actually pioneered in the subject of
statistical decision theory
and using game theory so he made use of
uh
very fundamental ideas in minimax theory
so
i as a young graduate student got
interested in game theory and in control
theory so i got interested
something called differential games and
here uh this has nothing to do with
robotics at that time this was
uh 1968 69 as a postdoc 1965 to 68 as a
as a graduate student
uh got interested in uh multi-person
games okay but these were zero-sum
games where one person's gain is another
person's loss
and you could think of it loosely as uh
uh
how would uh two aircraft uh engage in
combat so that one c one aircraft
could defeat another aircraft or uh how
would an aircraft
try to defeat a gun system that was
trying to shoot it out of the sky
okay so i worked on problems in
statistical decision theory where i had
uh two dynamic systems all right or at
least one dynamic system and a
gun system which can be thought of in
some sense as a dynamic system
where you would develop strategies
okay where uh where game theory turned
out to be a good way to
sort out what was what were good things
to do so this was minimax decision
theory with
two person zero sum games
and that is work we did as i said in the
late 60s early 70s
and then that carried over to things
that happened in robotics but
i uh truth be told tended to think more
about the more elementary stuff as
opposed to the
real team theory stuff which other
people certainly have not engaged in so
game theory
does play a role in robotics in in at
least in the theoretical aspects of
robotics
and uh game theory is is still a good
way
of at least getting a qualitative idea
okay
about how to develop good strategies um
i think it is still
fair to pretty fair to say that the
computational problems associated with
serious differential games is still
beyond what we can do in real time
robotics
so what one wants to do is figure out
from
looking at solutions that are done
in off not in real time but done offline
give you
some sense of what are nominally good
things to do
now the the game theory that we also
engaged in
which has again nothing to do with
robotics okay
dealt with how do you deal with things
like multiple missile evasion
and that's work i did in the mid 80s and
here the problem could be easily stated
it turns out that our experience and our
experience meaning the u.s experience in
vietnam
led it led the air force to understand
that
a fighter aircraft might be attacked by
another
fighter aircraft or another two or more
fighter aircraft
okay where they would launch missiles
against
our fighter and they would do it what's
called out of envelope and that loosely
speaking means that when you have a
missile engagement
an air-to-air missile engagement between
one uh
one weapon system and another weapon
system uh there is a
region in space where it makes
sense to fire a missile if you're too
close to the target
it doesn't work if you're too far away
from the target it does work
crudely this is all quite crude uh you
can think of
the target having a doughnut shaped
distorted doughnut shaped
space around it okay i'm just giving the
physical space and not considering
velocity issues
other things that have to be considered
so airspeed plays a role in this it
isn't just position
all right but to simplify the discussion
so we can you know
come to a reasonable point uh the
uh fact of the matter is that american
aircraft fighter aircraft were
threatened by multiple missile launches
sometimes out of envelope meaning
not good launches but you have to
understand what did the pilot have
on board the pilot had his eyeballs okay
and could see maybe a flash when the
missile left the rail on the
enemy aircraft and that was it all right
maybe there was a
uh a homing and warning receiver uh in
the
in in our in the american aircraft that
would give it a threat warning
okay that it's under attack but
basically the pilot looks over
his shoulder to see if there's a missile
bearing down on him and
you to a vote to evade a missile is a
very tricky business
okay you have to time a maneuver if
you're going to do it
so that you outwit the seeker system in
the missile that means making a high g
maneuver
timed at just the right moment so that
you can make it possible for the missile
to miss
your aircraft now this gets to be much
more difficult
when you have two missiles bearing down
on you even if
even if one of them is not going to be
effective you don't know it you can't
know it at least in that era you
couldn't know it
and uh so you needed
means by which you could maneuver your
aircraft to blunt the effectiveness say
of a multiple missile launch
so i worked on heuristic algorithms for
doing this now heuristic algorithms
are simply algorithms that have a
reasonable strategy underlying them but
are not
optimal in some sense the the notion of
optimality in control theory
in robotics okay is in some sense still
a dream
we simply don't have
clean pristine optimal problems to solve
that are
large and make you know interesting
problems to work on
small problems we may be able to do
things often but when you have the big
picture
okay optimality is hard to come by all
right
so optimality is a guidance
idea a sort of a thought process as
opposed to an
absolute prescription as to what you
want to do because you can't achieve it
you don't even know
what the components ought to be
accurately enough to achieve the ends
you want so you have some guidance you
look for things that you're sensitive to
things that might be important so
heuristic algorithms give you
a way to do that and so we worked out
heuristic algorithms for maneuvering
our aircraft to blunt the effectiveness
of a multiple missile
threat and we used game theory to do
this and
uh this was something that we did
between 1976 and about 1986 so about 10
years
i worked with the air force and with
some really good
graduate students who were very
interested in this kind of work
and that's it's not a robotics story
okay but a story about control theory
game theory
and it sort of informed me in thinking
about problems in robotics
where you want to avoid collisions okay
where you're not
air-to-air collisions but ground-based
collisions there are ideas
that come from avoiding
problems in the air-to-air environment
that makes sense to talk about
uh to a degree on the ground so we've
made use of these ideas
so you mentioned control theory and game
theory did you find cybernetics
influential on your work at all
so cybernetics is one of those words
okay that i probably couldn't define if
my life depended on it
okay um so cybernetics is due to a guy
named norbert weiner as i understand it
and he
brought together a whole lot of stuff
okay um i would say that
i'm more of a fundamentalist when it
comes to how i look at things so
i have a computational engine i have
information
i have probability laws and i tr and i
have some
crude ideas about what is a good um
cost or payoff function okay to try to
optimize in the crude sense so
for instance if i can make an analogy
a lot of work goes on in decision making
even today
not in robotics necessarily but in lots
of other places where
mean squared error is an important
criteria that is an
average of a square of an error
now not to make a joke of it but
one thinks about it i need i have a
robotic
arm here and i have another robotic arm
here
and i need to put the pen into the cap
or the cap on the pen if you prefer
well if my system just to make a
simple story out of it my system is not
working well and i'm missing
here i've missed the cap i'm not getting
this very well and i miss it over here
not at all clear unless i poke someone
with the pen
that my errors over here are any worse
than my errors over here because i've
missed the pen
okay and getting into the cap so what i
need to do is get the pen
and the cap lined up properly so i can
get the pen
into the cap and then let compliance
solve the problem so there is a
cross-section
area that's important and then in the
sense of manipulation
it becomes obvious that you have to get
into the zone and once you're in the
zone
it doesn't much matter where you are in
the zone as long as you're in the
area of the cap you'll get it to work
right okay
and that's that's the idea i want to
sort of bring out here
now uh for aircraft
and uh say gun engagements the gun
misses the aircraft the bullet
around misses the aircraft and it
doesn't much matter if it misses it by
uh
by 5 meters or 100 meters because it
didn't hit the aircraft
so unless this thing has a what's called
a tracer round which glows as it passes
over the aircraft
and the pilot knows that he's that the
gun system on the ground is
is is starting to close in on him
okay uh that missed round doesn't do
much good
unless the round can be visible to the
or at least the effect of the iran can
be visible to the pilot
and that might make the pilot less
aggressive okay
cause them to you know back off from
attacking the target
okay the missed round has very little
value if it's at five meters versus
a hundred meters the only rounds that
are important to those that strike the
aircraft
and even in the case of hitting an
aircraft you've got to hit the right
part of the aircraft if you're going to
destroy it
okay or make it fail to function
properly you can fly an airplane with a
lot of holes in it and
aircraft pilots who engage in combat uh
have done this and there are
plenty of stories to to buttress that
case
so um again here it's a
how close are you if you're close enough
to actually hit the thing
then you're in the zone so we talk about
what are called zero one loss functions
where i either get it or i don't get it
if i get it i'm good it doesn't matter
where i am in that region
and if i miss it it didn't matter by how
much i missed it so
we made use of these ideas in robotics
and in
machine perception and that was uh that
was one of the things that
uh i thought was important in terms of
trying to understand
how do we do good decision-making
so um so you're here at penn for quite a
long time can you tell me about meeting
regina veggie for the first time yeah
psychotics program
okay so well russian will give you the
history
and better than i can but i can tell you
that
i met her first when i was uh here as an
assistant professor in 1974.
she was in computer science i joined the
department of computer science a year
before i got here
to become a member and a member of the
faculty in systems engineering
and uh so i met her okay as you know
part of the
then called the moore school of
electrical engineering which had three
departments
uh and electrical engineering systems
engineering and computer and information
science
and then rogen has started the grass
blab and i'm gonna say it was 1979
although she may upgrade me and tell me
i'm uh
i've got it wrong okay but she started
her first grasp
work okay uh with a single finger
um touch sensor in a uh basement lab
in this building okay uh and she did it
on her own okay had collaborators
elsewhere in the world
but there was no one else at penn that i
can recall other than her students
who are working with her and uh what she
did which was quite enormous
is she took this lab which was herself
and her students
and she drew in other people okay
from 1979 through the mid-80s okay
through uh the uh uh early 90s before
she left for nsf
okay and she built an interdisciplinary
lab
all right which had a huge number of
people working in it okay
modulo the fact you know it's uh
it's part of a larger engineering system
of
faculty group and she drew in
loophole she got lou to come here i
think it would be fair to say that
and lou you know very talented wonderful
guy
uh and he and roger uh you know
worked wonderfully together and the rest
of us benefited in in really great ways
uh when uh
rogen also got other faculty to join the
uh department
costa stan elitist is a person i would
would decide
okay cj taylor okay uh and others
okay and so rogena i think it would be
fair to say
was pivotal in many ways in bringing
robotics to where it is today
and it wasn't just robotics she was a
great believer in vision
machine vision okay and yet she did a
lot of other things that you'll
no doubt hear about from other people uh
and uh so i would say she had a very
major influence on all of us not just
myself
but on on a huge number of people
including
not just the faculty but her students
and other colleagues in the department
and who are some other people that you
collaborated with
i had the good fortune to have other
students i worked with that were not my
formal students one of them was hugh
durant white
uh who was here in systems engineering
who grew into
the grasp lab as part of his research he
worked with lou paul lou was his faculty
advisor
and hugh and i would talk about decision
making under uncertainty uh
at length and we had a lot of fun doing
that so
i would certainly mention hugh as a case
in point
other students of mine gerda cambarova
would be one jeff agnell would be
another
uh raymond mckendell and
uh kevin addison are all people who
worked
either directly in robotics or in
decision making under uncertainty
in things that were abstractly related
to robotics for example
dimitri churkowski worked on problems in
decision making under uncertainty that
have a robotics connection but it was
largely minimax decision theory applied
to
uh some some very interesting
distributional
problems what do you think is the
the problem that you solved that might
have had the biggest impact or the
problem
with your students well i don't know
that we ever solve any problem
completely i think it would be
presumptuous to assume that we get
better at what we do
my students may disagree with me and uh
and say yes they were it was solved i
think we we learn
uh iteratively uh we learn it's like a
progressive jpeg
uh we see uh uh we see the we see the
shape more
clearly as time goes on as things get
sharper things become
clearer but what happens is that
problems don't in fact totally get
solved because what happens is that the
technology changes
so uh what looked like a solved problem
today
becomes less solved if you bring in the
rest of the technology and all the other
things
that have to happen so the notion of
sensor integration might in some sense
be solved easily with just two sensors
that are static
looking at some object in space and
trying to decide what it is
using two complementary sensors but when
you now have a huge number of sensors
and they're distributed and they're
mobile and they have to communicate
and so on and so forth then that problem
okay which is still a problem of sensor
integration so if you say we've solved
the sensor integration problem
our answer is i don't think so we've
solved pieces of it
at certain scales okay and it keeps
coming back again and again and again
all right as the technology changes so
uh
i guess in some sense uh i think i think
the part rather than saying i've solved
a problem and it's now
settled okay i would say i had a lot of
fun thinking about how to do statistical
decision theory
and a wide range of problems with some
extraordinary people
who did the real work and i got to
supply the applause
what do you think are the big
outstanding problems or
the things that might really advance
robotics if they
it's i think it's the scale problem it's
the problem of uh
how do you get all these pieces to work
together how do you integrate
huge numbers of disparate uh independent
uh sensor-like systems together to make
them
make them work and do so with a great
deal of reliability how to reason about
a very complicated world
and reasoning about the world is
everything
in machine vision and now i'll get shot
by others in the grasp
who perhaps see it differently i claim
the vision problem is still a very hard
problem
that vision itself is not easy that
vision
vision remains to be settled in some
sense if it's ever settled
okay uh so uh vision is hard
and i believe it's important that we
don't think about
uh cheap solutions where we go out and
barcode the world so now i know where
everything is
and i can just scan it and figure out
what it is
okay i want to be able to do it from
scratch i want to be able to do it or
have other people do it from scratch
and i don't want to have to go out and
engineer the world i want to build a
deal in a world that
where i haven't been but i should be
able to send sensors into the world
on robotic platforms and learn about the
world
so uh being able to do things from
scratch
in a complicated environment where i
cannot control the
the lighting for example you put lights
here to have a
have this uh interview all right well in
many situations
i can't control the lighting lighting
changes and uh
the students learn this okay uh when
they're doing their uh
early machine perception work that uh
lighting is a is an important issue
so uh and that's just one of gazillions
of other issues so uh
it's the robustness thing coming back
how do you uh effectively
uh uh control for the things that are
uh not going to be neat and clean and
something that you can
figure out a priori all right you have
to be able to deal with a range of
conditions and the range of conditions
are not necessarily
neatly delineated all right and the
conditions
don't interact in a simple way so there
may be a lot of non-linearities
and how these various factors come
together and so that's a
those are issues so it's the scale of
the problem and
the fact that one wants to do things
quickly real the word real time is
everywhere
okay so everyone everything has to
happen quickly
has to happen reliably uh reliability is
important
how do you define reliability when you
have complex systems
okay lots of things that have to work
what is
what are what are the are there can you
look at the system and figure out
is this the piece that if it fails
everything falls apart
or can we do without this can we shift
the burden of the decision making or
robotics work to another piece of the
system can we cooperate how do you
define even cooperation
cooperation isn't easy to to to
to cleanly define all right and
a lot of these problems have been
attacked by a lot of by folks in the
grasp lab and have done some really nice
stuff
but i would say that uh trying to sort
out
these problems and figure out what uh
that there has one got a clean answer to
uh
to a problem getting clean answers is
hard
all right and i think that's the problem
we're we're we're
it's it's what's it is what separates us
from mathematics mathematics
there may be if you pick a good problem
a clean answer
i can prove a theorem is the theorem
useful maybe
maybe not okay and uh does the theorem
apply to the world well that depends on
whether the assumptions that went into
the theorem
can be mapped into a physical world and
it's often the case the physical world
is much more complicated than the
mathematical model
okay uh which represents it uh uh
you know actually achieves
so uh it's the question of how do we
build good models
and as a model builder and that's what i
do really for a living in some sense
i realize everything i'm going to do on
a given day is wrong
all of my work is wrong now i learned to
live with that
okay it's not as depressing as it sounds
uh
what one has okay to understand is it
isn't that i'm wrong it's
how wrong am i what are the consequences
of my errors
learning how to deal with errors okay
and not let them kill you
is the is an important deal and uh
that is something that uh we need to
ponder how do we
how do we live with our mistakes and
make the system
uh suffice this idea of optimality is
again
something i don't really think we
achieve we try to achieve it i think
when we're practical
uh something called sufficing we want
something to work
okay that it works perfectly it doesn't
have to as long as it works well enough
so it's the well enough how does one
define well enough that's not easy
and that's where we have to spend a lot
of time particularly given the
complexity of the systems that we have
these huge number of different systems
with
different kinds of communication
different kinds of information all right
and uh and so on so it's the scale of
the problem
okay that makes this thing uh very very
very very
challenging and i believe that's where
students today and the people working in
the future are gonna
gonna have to figure how do you do this
distributed problem i mean there's a
this story about distributed systems in
computing
okay and uh so it's not a new story it's
an old story but
how do you make the thing function how
do you know what you don't know
yeah well that gets us to another story
too
exactly um
have you also done work in multi-agent
yes the multi-agent story is the game
theory where we we did this
we had uh uh some early work this is uh
baishi vijay kumar lupaul
myself and very talented graduate
students and we had
some of the some of the i'm smiling as i
say it because i'm recalling the trc lab
mates which were these clunky robots
which we used and we had multi-agent
systems this is
i'll be forgiven for for saying this uh
in today's world but
in the 1990s early 1990s when we were
doing
this work the robotic platforms were
very crude and the the sensors the multi
agents
we had a upper level agent not just in
the hierarchy but one that was up on the
ceiling
looking down on the world talking of you
know in a sense getting a sense of where
the robots were
and what was going on and this was being
communicated to the robots
so uh we had different scales different
what were you trying to get the robots
to do uh something as simple as uh
organize themselves to move through a
narrow passage or to
communicate or to work together to move
an object
it was pretty primitive but you uh you
got it
well my colleagues okay i will give them
the credit my colleagues uh
you know did wonderous things with some
very crude equipment okay
and uh i guess if we had uh
had smaller computers more agile
platforms we could have done
more but we had to learn something i
believe along the way
uh if we had somehow either gone from
1985 technology to 2010 technology
overnight i'm not sure we would have
been able to do it
because i think there's a learning curve
we don't talk about
all right and we start out here with
some very crude stuff
we learn what doesn't work so let me
comment on this this might be
useful and it has to do with a lot of
stuff that goes on
in research not just in robotics but
just about everything
including pure mathematics i had a
professor who taught me probability
theory many years ago and
he used to say to us gentlemen there
aren't any women in this particular
class gentlemen
if you want to understand this paper and
this mathematician's paper don't read
the paper
look at the mathematicians wastebasket
because what didn't work
is often more illuminating than what did
work
and we never there's a whole ethic that
we don't publish
our failures although i believe we
should there should be a journal about
failure
because until we learn about failure we
won't understand success
and there's no nice way to say that
because the entirety of
the community's ethic is to show how
good things are and how fast things can
be done and how well things get done
and we never like to talk about what
didn't work although we should
trial and error yeah um
so who uh who else was influential on
you early in your education during your
graduate work um folks in the statistics
department in the math department at
cornell
got me interested in minimax decision
theory
okay and roger farrell would be one of
them one of them was
jacob wolfowitz who was the professor
who said look at the wastebasket
he taught me probability theory and
james thorpe who was my phd advisor who
taught me control theory all right
much of what i did in my early days
dealt with filtering and decision making
using
kalman filters and kalman-like filters
so i spent a lot of time
on finite dimensional systems
linear and non-linear systems doing uh
doing filtering
and so that's how i gradually move from
control theory to robotics because
in a sense these decision-making modules
are
were important in robotics so that's how
i in a sense got to that
at the time were there other
applications in mind or were they
were you working in a pretty abstract
level oh uh there were always
applications and the applications were
driven largely by
the aerospace industry just about
everything had to do with
military systems military or industrial
systems
industrial systems where you're doing
chemical process control where you have
uncertainty
that's one example um or you're dealing
with aircraft and missiles that's
another
situation so those were the were the
situations that were typical
um where we where we you know had
examples
systems that got very complicated early
on that
i didn't have anything to do with what
others did dealt with
things like power systems you're
probably too young to know this but in
1965 as i recall in
uh mid november mid to late november
probably something like november 22nd we
could check the date i don't
i'm not absolutely certain was the 22nd
22nd 23rd november 1965
uh there was a
major power failure in the northeast of
the united states
and uh what happened was there was a i
mean the story goes that it was a
squirrel that got into the switchgear in
ontario hydro
i don't know if that's true or not i may
be totally wrong but in any event there
was a
fault in the switch gear at ontario
hydro that led to a circuit breaker
opening
which caused another circuit which was a
transient which caused another circuit
breaker to open and we were in ithaca
new york at the time
my wife and i were in school at cornell
and uh we're sitting in a basement
apartment having dinner and all of a
sudden the lights go out
and it turned out i wasn't
thinking this way but it turned out that
the lights had gone out over most of the
northeast united states
all the way from northern new york and
southern canada all the way down to
perhaps
even as far away as virginia and
certainly up towards boston
and it was all due to a collapse of a
power system
and there you had an example of
a complex dynamic system a power system
is a dynamic system
their flow is of power in the system so
that's what makes it dynamic
it's a non-linear system because it has
the com
the physical components that make it
non-linear
so it's not easy to analyze it's
stochastic meaning
there's a lot of uncertainty the
stochasticity is not easily modeled or
at least it wasn't in those days
all right and the system collapsed quite
literally new york city was without
electric power for a goodly period of
time
and when you think of shutting down the
entire city and having no electric power
okay it's a probably it's a breathtaking
thing
and it was on a cold you know november
night it just was not a
nice summer's evening okay and
basically it affected a whole lot of
things aircraft flying into then i guess
it was called kennedy
okay at that time
about to land okay having just crossed
the north atlantic
okay as the story goes okay uh the field
disappeared now they knew
that they were still on in the same
universe
because i could see cars
driving on the highways in queens
because they were running their
headlights
but the entire all the strobe lights all
the
navigation aids for aircraft to land
okay
disappeared because they were all
making use of this power system that
failed
so people caught under the east river in
subway cars
aircraft had to be rerouted okay it was
decla
it was it was a tricky business the
problem is again
complex dynamic system hard to model
hard to understand
and nobody had thought about could this
happen
okay this notion that that the system
was vulnerable
to this kind of an event okay uh
i don't think it crossed anybody's mind
or at least if it did
they didn't make it out of the uh
someone wrote a report or a memo
and it didn't go anywhere okay now you
would think well that settles it now the
problem will be fixed and it hasn't been
that way at all
we're still seeing uh occasional
blackouts that happen regionally not
just you know
uh around the university here but i'm
talking states we in
uh i think it was 2003 there was one and
that basically started in ohio
progressed to michigan went to canada
came to new york and it was just by the
grace of god that
uh philadelphia electric uh through the
pgm interconnection disconnected from
the network before
pjm crashed so uh it
getting this stuff to work is uh complex
systems are
are a problem okay i wanna
okay um so as an engineer
how do you design for all of that
complexity and uncertainty i mean
what would you what do you tell your
students how do you go about
design well i don't think i know enough
about
huge systems to actually address that
thing from beginning to end i
would be dealing with parts of it and
parts of it would be the
decision-making structures where i have
uh
muscle models that give me
you know a range of possibilities for
the dynamical system i'm dealing with
and i would train them not to assume
linearity not to assume gaussianity
but to deal with the real system uh
that is physically cut away the
bits and pieces that don't help us get
to a good solution
the oddities that increase the dimension
of the thing but don't do so usefully
okay and so so to come down to a minimal
a minimal size system that is realistic
enough
to handle the problem at hand or at
least the component or modular component
at hand
so i'm a believer in doing things in a
modular way not everyone
sees it that way i think there are
people today who believe
we can do this thing or to do some of
this work that goes on
organically we build a huge network
and we use machine learning ideas to
seek to make the whole thing function
and i'm
i'm still not converted to this view and
maybe i'm mistaking the view maybe no
one
actually believes what i've just said
but
my personal view is is that i like to
think i like to understand
component pieces i'm still a
reductionist i'm perhaps a little
old-fashioned but
i believe in trying to understand
component pieces
and uh making uh making that
uh making that work my phone just went
off may i uh
take an interlude okay um
okay um so
what what would you recommend to young
people or
who might want to get into robotics or
study robotics in school
how do you what would you suggest
yeah so i'm i have some strong views on
education
and i'll be delighted to recount a few
of them say if you
state a few of them i believe in
fundamentals i believe there's no free
lunch
i believe that we don't start with the
magic and the high level stuff
but we get we get good grounding in
physics
good grounding in mathematics good
grounding and engineering science
and that that's what we build from we
can't have one of these huge
razzle dazzle systems until we
understand
how how the fundamental pieces fit
together and that's there's no
substitute for learning the fundamentals
physics is absolutely essential part of
the computer science ethic and
this troubles me a little bit but i have
to show a little bit of
appreciation for why it is the way it is
is that
computer science it can be in some
circles largely divorced from the
physical world
computer science is a science of the
mind where we think
algorithmically we think in terms of how
bits are manipulated and
the manipulation of bits doesn't have to
happen in a on a physical platform we
can just think of it as a mathematical
abstraction
so in some sense we can be divorced from
the physical world i think it's a
mistake
to divorce computer science entirely
from the physical world
i think computer science has to has to
uh intersect the physical world
surely uh in in in important ways the
to put it to quote rojana baishi okay
the bits have to get into the machine
somehow okay and if you're thinking
censored systems
sensorized systems the systems with the
sensors bring the bits to the machine
so we have to uh in effect
connect to the physical world so physics
is important all right
uh whether or not it's robotics or not
it could be something as simple as
building a good automobile
okay where no one would doubt that
physics is important
mathematics is important what kind of
math now there's a dichotomy in
mathematics that troubles me
in computer science the mathematics is
largely something called discrete
mathematics we're dealing with the
discrete structures and the notion of
calculus
notion of mathematics where we have
limits and we talk about things like
integrals and derivatives and on and on
and on
okay place uh no role at all in core
computer science so core computer
science is about discrete structures
and has essentially nothing to say about
uh has no connections to
what i'm going to call core calculus
ideas elementary calculus
now elementary calculus is important in
robotics because you're now talking
about a physical world you need to
be able to talk about the robots as
physical entities so
dynamical systems requires calculus and
calculus is a serious subject but if you
talk core
computer science as divorced from
the application of computer science to
robotics or even
graphics where you're talking about
animations okay where calculus could
play a role
all right uh the mathematics of computer
science is largely discrete whereas
and they and computer scientists work
all the time with things called
automata which are primitive models for
computation
okay so finite state uh systems
okay well if you go to a mechanical
engineering for the
as another or electrical engineering
not not entirely but it is somewhat true
in say mechanical engineering that
finite state systems play little or no
role in their lives and the mathematics
is largely continuous
so there's a divide between the between
different disciplines in terms of how
they use or don't use mathematics
and i think this is wrong and that we
should get the we should have
uh appreciation for different kinds of
mathematics where to use them where not
to use them
what one where where where how how
things fit together
mathematics is a fabric and we can't
just say
it's this piece it's this piece it's
this piece it's the high
it's the entire thing okay we have to
see how the plate
pieces fit together and at one level
i'll need to think discreetly at another
level i'll need to think about
continuous systems
okay and in some cases i'm going to need
to think about both discrete and
continuous systems at the same time
i have a colleague george pappas who
deals with hybrid systems
and you know favorite examples of hybrid
systems are things like aircraft
where uh you're using finite state
automata to control the
uh the way fuel flows to a jet engine as
it functions at different altitudes
so you're going from one control system
to another control system
as the aircraft uh moves in space uh
from one altitude to another altitude as
a
as a case in point so it's both
continuous systems and discrete systems
playing together and it's a very serious
matter so uh
this is stuff we need to do better and
there's room for this totally room for
this in robotics
it's all over the place in robotics so
these notion of hybrid systems how to
make them work
okay is a big deal so teaching people
about math
physics okay the language of computer
science
is different from the language of
mechanical engineering
very different language okay and
we need to be able to learn other
people's language and we need to be
comfortable
and there's no free lunch we need to
work hard we need to dig
in if i want to learn a subject
i need to work on it for a while
and i tell my students the best way to
know if you've learned a subject or not
is can you teach it to someone else
and you get up at the board and teach it
okay without
reliance on looking at a set of notes
can you just walk to the board
and talk about a subject then if you can
do that then you know you know something
and uh and if you can't then maybe you
need some more work
so it's how do we learn the bits and
pieces before we get the grand ideas
that we're going to build the
huge systems that will have these
miraculous
properties so talking about the
transactions between electrical and
mechanical
and control theory and different areas
of engineering
sort of historically what do you
remember the sort of first time people
talked about robotics
as robotics or as its own field and
how do you see that field emerging from
these other fields
well uh rojana
still has this dream okay i don't think
it's happened yet
but she stated this dream to me in the
early 80s she said she wanted to someday
have a
dinner party where they were
robotic-like devices that would clean up
after the dinner party and
totally take care of everything we have
yet to build service industry robots
that do that
so my feeling is we still have a long
ways to go
in terms of making that happen and that
has to do with the complexities of the
environment in which things are
the notion of being able to actually
determine what an object is
what state is it in okay it's not easy
to do
and um so there's still there's still
plenty of work to happen
so i'd say roojina laid the groundwork
for thinking about uh what i'm going to
call
service service robotics
okay as opposed to military robotics um
i i'd say that the idea of being able to
combine
vision combine actuation okay were the
early days
okay uh there's a story that i'm told i
don't know if it actually happened that
marvin minsky said to a graduate student
i think
at mit in the might have been the late
50s early 60s
one summer day or whenever you should
connect that
television camera to a computer and see
if you can get the computer to see
i mean and it's laughable today that one
would make such a statement if in fact
it was made
okay that you know be so naive to
suppose we will just simply connect a to
b and
miraculously we will have have something
that
works terribly you know particularly
well
and so this began i'd say in some sense
a notion of machine vision
so but it happened at a very crude level
and uh it took a while and uh uh
again i think we we cycle back this
notion that
things are settled and that there's a
substrate and that we never have to go
back to look at the bottom of the
substrate
is wrong we come back we cycle as we as
as other dimensions change the
complexity dimensions of
multiple sensors speed with which things
have to happen
okay accuracy requirements i need an
accuracy requirement that works fine for
one thing and another accuracy
requirement
quite different for something else
reliability for one thing can
you know one thing is okay to be this
reliable i can lose this many machines
it doesn't matter
okay other machines are very expensive i
don't want to lose them
i don't want something to go wrong
we still have very complex systems and
i i mean in today's or yesterday's wall
street journal and in the new york times
it's
alleged i mean i have to let the but the
organizations that are responsible for
this
weigh in on it but uh i i note with
sadness that
it may be that that air france jet that
crashed
on its way from brazil to france uh
crashed because of uh pilot error okay
because the pilots got
uh distracted okay and weren't taking
uh paying enough attention to their
airspeed okay
and the the aircraft basically uh was
lost as a consequence of
machine uh complexity where there were
too many sensors and alarms
okay some set of sensors failed caused
the autopilot to be shut off alarms were
going off in the cockpit
okay they know this apparently because
they've now transduced from the
black boxes that recorded the
information at the last moments of the
flight
and so in a sense here we have a man
machine
system okay which was too complicated
to control by the people who were in
charge of controlling it
and it's not to say that they couldn't
have done it but their training didn't
outfit them to handle this particular
bad instance so it's trying to think
one's way through bad instances
where one has complicated systems to
work with
and that's not easy to do and i'm not
going to point fingers at anyone because
i could see it happening again i mean
again the power systems fail
okay different things go wrong okay so
maybe we solve one problem but then we
don't solve it sufficiently to
take care of all future instances where
things will be a little different
okay it's being able to understand what
the differences might be
we we tend we've also come to rely on
systems in ways that
are sort of breathtaking uh we have an
enormous amount of cockpit or
automation in aircraft and uh
it's probably necessary to have that
okay but
um in a sense when uh when people rely
too much on automation
okay it can be uh it can be dangerous
and so one needs to
always keep alert and keep an eye on
what the machine is doing
all right so again i this is not
robotics per se
but uh it's complex automatic systems
okay that can in fact defeat us if we're
uh if we rely on them too much
well i know the uh more sponsored by the
robotics and automation society of the
ieee
do you see a difference between robotics
and automation uh in any
significant way i'm one of the outliers
in this
i don't know i wouldn't be able to tell
you what everyone thinks but um
i see a blend between them um i mean
there may be
i don't want to say political reasons
for picking uh you know
drawing a circle around this piece of
turf and circle around this piece of
turf and saying this is x and this is
why
i'm not a great believer in turf
definition i'm a great
believer in trying to understand basic
problems and if if
part of it is you know if i'm doing
something with a with a missile okay
is a missile a robot well not in the
sense of the robots running around in
the grass blab but
is there a is there a sensor system is
the thing uh making decisions based on
sensor data
is there a control system you know what
what differentiates a cruise missile
okay and has to follow a uh trajectory
at low level okay uh to get in under
enemy radar versus
a robot okay doing something uh
uh to move from point a to point b okay
i think there's some issues that are in
common but there are also things that
that make them different and maybe it
may be the
people who do the work are themselves
different have different outlooks as
opposed to
that there is something absolutely
separable
intellectually i doubt the absolute
separability of the intellect
i'm very easily convinced that people
will want to draw boxes
or circles or whatever around domains
because it's easier to
to define a domain that way this has to
do with organizations
as opposed to dealing and people as
opposed to dealing with intellect
and that's why we have departments and
universities you could imagine
we're all thrown together in one uh huge
department within the school of
engineering it would be unmanageable
okay it couldn't be done at least i'll
say it couldn't be done the dean may
tell me i'm wrong but
i don't think it could be done and it
can't be done or can't easily be done
because it has to do with
the uh nature of human beings and
how we could how we like to think about
things how we like to organize ourselves
and having an inside and outside in some
ways which isn't always a nice thing to
say
in terms of the institution when was it
first possible at the university of
pennsylvania to get a degree
in robotics that happened recently it's
a master's degree
and we have a master's degree in
robotics that george pappas and
vijay kumar and costa stan elitist are
primarily responsible for starting
what year was that oh goodness it must
be three four years ago no more than
that i
we could check for sure but it's within
the last four three four years
and so most of the members of the grasp
lab have been in the electrical
systems engineering no so the grass lab
is broken into bra the grass lab is a
multi-disciplinary laboratory so it
includes
it's a lab in a sense it's the grasp lab
of the department of computer science
but that misses the point
it's computer it's supported by the
department of computer science
but it couldn't function at all without
close collaboration
integral collaboration intertwined like
this you can't
you know break them apart mechanical
engineering
electrical engineering and by electrical
i mean electrical systems engineering
and uh other other divisions uh the
psychology department has people doing
work in
machine perception or in perception
anyway and they work with folks in the
grasp lab
okay we have folks in the grass lab who
work with folks in the statistics
department
okay so uh there's machine learning
going on here
which affects work and grasp machine
learning interacts with folks in the
statistics department
i have an appointment in the graduate
group in statistics having more to do
with my
interest in decision making under
uncertainty okay but i wouldn't call it
robotics
okay and i wouldn't call what the people
in the statistics vote
in the statistics department do as a
having a connection with robotics
they're interested in
statistical theory and application okay
so uh in a sense the phd students came
from
computer science in no particular order
computer science electrical engineering
mechanical engineering
uh and uh
there might even been people uh who were
postdocs
who had connections with other things
but let's talk about the phd students
so they came from graduate groups it has
to do with how things are organized
students get degrees at penn through
something called a graduate group
so there's a computer and information
science graduate group which is roughly
con
connected with the group of people in
the department but may have some other
people as well from mathematics and so
forth
okay and from philosophy and so forth
and from
other places and then there's the
electrical
engineering uh graduate group electrical
systems engineering graduate group and
they may have people from
computer science as part of their
graduate group but it's a graduate group
that
husbands and runs the compute the end
the phd program then there's a
mechanical engineering
graduate group so the students come from
those graduate groups but they work in
the grasp lab
and they work interactively so you have
mechanical engineers working with
electrical engineers working with
computer
and information science graduate
students okay
to uh achieve some very interesting
research
so i would say it really is a
multi-disciplinary department
with graduate students who have their
home base
in because of the way we're organized
specific departments could we all be one
department to do this
i suppose we could i don't know how it
would be organized and my instinct is it
probably wouldn't work as well because
by making it a little more focused this
way
it makes it a little easier to know
who's in charge of what
um so i think that's most of my question
is there anything else you'd like to add
uh when selma sent me an email i
had to take my take a deep breath i
hadn't thought about
this but yeah i've been involved in the
grasp lab for 27 years
i hadn't been counting so uh it's been a
it's been a good time
and been very i've had a lot of fun and
if i had to do all over again i wouldn't
change a thing
great thank you