you can start by just telling us where
you were born and
where you grew up and where you started
school well i was born in england
and at the age of seven moved to new
zealand and at the age of nine moved to
sydney in australia which is where i
went to high school and my undergraduate
work
okay do you want to speak english i'm
sure you're sure
okay well my my career i should first
clarify that i don't really consider
myself as a roboticist my career is
overlapped robotics from time to time
and i think perhaps
influenced her from time to time but i
wouldn't consider myself
one of the robotics people as it were so
i really got started
in moving away from pure mathematics
which was my first love in college
in this direction by reading norbert
weiner cybernetics
which showed me that mathematics could
be applied in a
different way from classical newtonian
stuff
but thinking about parallels between
brains and machines
and about the same time i came upon grey
walters book
which introduced me to his tortoises
and so that was probably the first real
robot i knew outside
reading science fiction like i i robot
by asimov
um but as a mathematician i was very
interested in
automata theory and mathematical theory
of neural networks and things of that
kind
and then in my final year
there was a neurophysiologist working at
sydney university on
the visual cortex of the cat named bill
levick
and he wanted to learn more mathematics
and so he taught me about the brain and
let me see what they did to cats
and i helped him with this mathematics
and it was he who introduced me to a
paper called what the frog's eye tells
the frog's brain
which come out only a little bit earlier
in which
um it built on some of what i've been
reading
of the cybernetics book for example the
formal theory of neural networks by
mcculloch and pitts
and it turned out that ledfin and
maturana had worked with mcculloch and
pitts
and had taken not the classic 43 paper
but the 47 paper on how we know
universals
which showed how pattern recognition
might operate over a layered
neural network and they tried to test
that in the frog
and in the end they did find an
interesting layered network although it
wasn't
the one predicted by the theory and so
that got me
interested in the frog as a place in
which one could bridge between
the actual biological data and the more
formal theory that my mathematics had
gotten me to
it also told me that mcculloch and pitts
were at mit and
i until that time since i've been
reading their classic papers i thought
they were still in chicago
and also as part of my automata theory i
read a book called automata studies
which was edited by shannon mccarthy
who were on the faculty at mit and had a
very interesting paper by marvin minsky
before he became virulently anti-neural
networks
and so all these people weiner mcculloch
and pitts
shannon mccarthy minsky and so on were
at mit
so that became the natural place to go
for my
phd work i started working with
wiener because i wanted to know about
cybernetics but i also had a research
assistantship with mcculloch
and so i was involved with that group
but then it turned out that weiner
wasn't really into cybernetics anymore
his big thing was statistical mechanics
so i was his phd student for a while but
when he left for
um his sabbatical and we didn't get much
correspondence i switched to another
mathematician working on statistical
mechanics
and in the end i decided that wasn't
very interesting type of mathematics i
ended up with a phd in probability
theory
stochastic processes that has nothing to
do with robots or brain
in terms of my own career but
midway through the phd i went back to
australia
and gave a series of lectures at the
university of new south wales and that
became my first book brains machines and
mathematics
so that really set me up for my future
and the one other thing i would add
about mit was
that and it just shows how much
serendipity affects one's career
a friend from
sydney wrote and said he'd read this
book
on control theory and the authors newton
gold and
kaiser were at mit would i go and talk
to them until that time i had known
nothing about control theory beyond what
was in weiner's cybernetics book
so i didn't talk to them and they said
well since you're a mathematician
you'll be interested in these two people
working out at
lincoln labs michael athanasiadis
and peter phelp and i went to see them
and it turned out
they were developing the theory of
rudolph kalman who invented the kalman
filter
and so after my phd i spent four months
driving all the way around the united
states
visiting people and national parks and
so on
confusing the hell out of them because
they all assumed if i was visiting them
i was looking for a job and then they
start talking about jobs no i'm just
wanted to see what you're up to anyway
one of the people i visited was kalman
who was then at the university at
maryland at a research center but he was
just
then moving to stanford so he asked me
to go there as a postdoc
which i agreed to do so after about six
months
traveling around europe no actually six
months in london
working on neural nets
and then about four months traveling
around europe and the soviet union
i went back to australia and then
finally got to stanford
and so during that time i probably had
my
first real contact with robotics
research
because just down the road from stanford
was sri at that time still called
stanford research international
before stanford associated itself and
then niels nielsen and his group
were working on uh shaky the robot and
the strip
system for robot planning so i think
that's where i got
first involved with real work on
robotics
but my own research
was divided between automata theory and
mathematical theory of neural networks
and then getting more
involved with um real brains in real
animals
and in particular building on the work
with
what the frog's eye tells the frog's
brain i worked with a student enriched
today
and we asked what does the frog's eye
tell the frog
so the idea is okay here are some
signals in the brain the original paper
said that might be correlated with prey
might be correlated with predator but
how does the animal decide if there are
several prey for example which one to
snap at
so we came up with a model of which i
think might have been the first winner
take all network you can check
with everybody else and find out whether
they're earlier dates from the late 60s
but and then with other students we
modeled
mammalian systems with poverty dev
um looked at cortical networks for depth
perception
with kurt boyles looked at the
cerebellum
and its relation to that that's when we
started interacting with
david ma and then i developed something
called
schema theory where the idea was
that just as in computer science doing
everything in machine code is death you
need higher level procedures
so i felt that the brain needed not just
a structural account in terms of
networks and
neural columns and so on but a more
functional
level of parallel interaction
distributed in corrections that was
those were the schemas and the focus
there was to really think about vision
not in terms of just recognizing what
was going on in the world
but in terms of how that recognition
could extract the parameters needed to
guide action
so what i was doing then it has been
characterized as computational neuro
ethology ethology the study of animal
behavior computer models of the brain
mechanisms underlying that
so in a sense i was
learning about robotics not contributing
to it but developing a framework
that was very much relevant to robotics
as if you will embodied computers
which had both sensors and effectors so
that gets me through stanford
there any questions you want to raise
before i
jump into the next phase so you were
conscious of the applications to
robotics and how much of your work was
sort of driven by an interest in being
able to
translate these neural models to an
artificial system
and how much was just trying to model it
at some stage
i think that the robotics people i was
interacting with at that time
were either looking at the visual front
end
um what computer vision algorithms might
influence the thing or in terms of
extracting a model of the environment
or they were looking at the high level
planning
in a very abstract space so i think
that my work was probably more
complementary than
tightly geared to it at the time i
wanted to know really how
you have a system that is extracting the
information it needs
to control its behavior
inspired by the brain talking to the
roboticist but not
looking for a platform on which to to
actually implement it
and in terms of the cybernetics scene at
mit when you were there
was there a growing interest in trying
to
build robotic systems there or to
continue grey walters work in any sense
or
were they sort of working i left mit a
long time ago
1963 so at that time
um under minsky's leadership there was
quite a lot of stuff in what you might
call
symbolic a.i um
minsky and papa were writing their book
on perceptrons
which many people took as the death
knell of neural networks but i think
just
introduced complexity theory into the
study of neural networks
um we had some early work on
computer vision um
but i don't think that was what you
might call live robots at that time i
mean i've always claimed since that i
was a
a robot created by marvin minsky but um
whether that's true or not i don't know
i don't know whether you know this but a
few years ago
a group in england published a paper on
a very unfortunately a very simple robot
but it was called arbi but the good
thing about it was not the robot itself
but the acronym because it turns out
that i am
an autonomous robot based on inspiration
from biology
so what you're looking at now is not
actually a human but a
an artifact that marvin minsky created
in the early 1960s
but based on biology based on definitely
inspired by biology yeah
um did you ever encounter like shannon's
rat
or any of the other cybernetic robots in
the early days you saw the tortoises
first hand
no i never saw them uh yeah i knew
shannon
moderately well i once went rowing in
the
pond behind his house and capsized and
he cycled out to rescue me because he
had a
bicycle on pontoons that he could take
out onto the pond
but but i talked to him more in terms of
his mathematical theory of communication
so i knew about the rat study in
relation to
as you say grey walters robot and other
efforts but
i don't remember that as something that
we were actively engaged in discussing
at that point the mathematical models
and the neural models were more
interesting and so they
it seems that cybernetics had that sort
of moment where they built a lot of
robots and then they sort of stopped
for a decade or so and then they started
rebuilding them with
the more symbolic ai except for maybe
the hopkins beast
which was another cybernetic one yeah no
i'm just saying that
i mean there are obviously a lot going
on at mit so i can't claim to be giving
you an accurate picture of the whole
spectrum but
certainly in terms of the people i
interacted with there were the
formal theory of neural networks and
reliability theory over in mcculloch's
group there was the
move into symbolic ai but the interest
in perceptrons
in minsky's group and then a lot of
control theory communication theory
mathematics
so i was not actively engaged in
robotics as such at that time so as i
say
i think the first serious conversations
i had on robotics were with
um niels nielsen when i went to stanford
where
it was at the level of planning so in
some sense it was complementary to my
interest in
what the neural networks were by a
a computer model of the brain
can recognize objects in terms of
interacting with them navigating around
them
and things of that kind what was the
extent of your interaction with the
robotics is there did you mostly have
conversations or did you work with them
on any of the projects
nsri conversations
what was the first project that you
worked on that involved a robotic
component
that's at the next stage the university
of massachusetts
so in 1970 i left stanford
to i was recruited to start the phd
program and chair the the new department
of computer and information science
and so in setting that up um i
explicitly
added cybernetics to
systems and theory as the three
tenets as it were of the of the
department
and so
the i think the turning point came at
a 1979 conference
on the brain which was arranged by
amongst others david ingle who was one
of the people i had been
interacting with who did frog
experiments and
at this meeting there were two very
influential papers
about human and monkey brains mark
genero
from france had been studying
hand movements in humans
and looking at the way that the
pre-shaping of the hand is coordinated
with the movement
of the arm and um
so that led me to think through the
issue
of um
how to take the schema theory that had
been developed more in relation to my
visual control of frog behavior and
think about how it could be applied to
the visual control of hand movements
and so i put out a review paper
that included my initial thoughts about
that
and fortunately or unfortunately it gave
the impression that i knew about
hand control and so i got invited to an
international meeting in australia
and then i had to produce i had two very
good phd students theaberroll
and um damien lyons who uh
work with me to develop a theory of hand
control and that work
finally led into robotics
um in two directions one is that
it raised the question of well can we
apply this to control of a robot hand so
we got hold of
what was then one of the first dexterous
hands produced the salisbury
hand and started applying her algorithms
to that
and damien in addition to working with
me on modeling the human hand took
schema theory and developed a
programming language called rs
for robot schemas and that's seen modest
application in the robotics community i
would say not
massive but some and then
we established something called the
laboratory for perceptual robotics
and i think this was somewhat
novel at that time in that a lot of
concentration had gone into the things
of path planning for robots
and then of course at industry by that
time they're already a lot of robots
but they were doing stereotyped
operation
so the notion of really thinking through
on the basis of my animal studies how to
close the loop
of perception and action i think that
was still relatively novel
and so um some of the things that came
out of that i had a couple of students
um john prager uh
and daryl lawton who worked on optic
flow so
that as you moved uh here i'd
been influenced by the work of a guy
named dave lee from edinburgh university
who'd been looking at the role of optic
flow in
animal behavior so we said okay can we
go from his observations to actually
think about
algorithms for extracting optic flow in
a way that could be useful whether it's
a brain doing it or a robot doing it and
then
a fellow named ken overton um who
unfortunately left academia to work at
general electric and has never been
heard from since but he
was quite the operator and so
he developed tactile sensors that we
applied
to the um salisbury hand so
we spent time picking up eggs with the
souls we had and things of this kind
um and then there was one other student
um steve
begish i think who uh continued the work
on tactile
census so there was quite a bit um
i would say that was my main phase of
being as it were within robotics
um rather than having ideas that
interacted with robotics
the lab for perceptual robotics is still
going strong the
current director is a guy named rod
groupen and
in addition to having a lot more robots
than we ever had
um he has developed these ideas of
perceptual and motor schemas that were
part of
the schema theory another way we
developed
schemas at umass university
massachusetts amherst was
that we had two people ed reismann
and al hanson who had a computer vision
group
and they developed some of my ideas in
schema theory
and i interacted with them to think
about how to recognize complex scenes so
that you would have different schemas
specialized in recognizing different
objects in the scene but passing
messages to each other so that they
could
the different schema instances could
compete and cooperate come up with the
interpretation
of that scheme and that architecture was
related to the hearsay
architecture for speech understanding
that had been developed in the early to
mid 70s at
cmu and we hired victor lesser
who had been one of the members of that
group to join us so
that that got cooperative computation
distributed
artificial intelligence call it what you
will up and running
at umass and then the other thing which
has had an influence both in
neural networks and in robotics and also
in machine learning
is that i met a guy named
harry klopf who was working at the air
force
research center in cambridge or outside
cambridge
bedford massachusetts and he put out a
little book called the hedonistic
neuron and his idea
was to think about brain learning by
asking what's in it for the neuron
so his idea was how could you have the
neuron was in some sense adapting
changing its synapses to satisfy its own
needs
and so i got some money
from him to do research on seeing if we
could come up with a real theory
of that and um
i had two colleagues working on brain
modelling at the time
nico spinelli bill kilmer and while i
was on sabbatical after i got the money
they hired
a postdoc to come in and work on the
project his name was
andy barto and when harry klopp
gave us the money one condition was that
i know this very bright student at
stanford
just getting his bachelor's degree you
have to hire him as a research assistant
on on this ground and that guy was rick
sutton and
as they say the rest is history battle
and sutton came up with
the mathematical theory of reinforcement
learning that has been
incredibly successful and influential so
i i think that oh and then the one other
person i should mention
from my umass days is uh
ron arkin who um became very intrigued
not with the handwork but with the frog
work
uh in terms of the the models we devised
of how the frog recognizes its prey how
it avoids um
its predators how it can detour around
obstacles and so on and so he took that
and implemented that as a controller for
a robot
and that became the basis for his very
successful career in robotics that he's
pursued for many years at um
georgia tech so yeah so i'd say my my
peak involvement with the robotics the
history of robotics is during that
period at umass
and then i moved here in 1986
and i was recruited in part by george
becky
who was then chair of computer science
here
and he had a dexterous hand as well
perhaps
not quite as beautiful as the salisbury
hand
that had less independent finger control
but it was the usc belgrade hand
and we brought over thea ibero
as a postdoc to work with us on that
collaboration
um and
uh so there was quite a bit of work um
between the three of us then on
dexterous hand control in relation to
studies of human hand
control and then we had several
phd students work with us
tony lewis um did a thesis
which we jointly supervised on
evolutionary algorithms for uh
locomoting robots hexapod robots and so
on
um but i think it became clear after a
while that
most of the students we were getting oh
and just one other student in relation
this andy [ __ ] did his phd with me
on um what's going on in the monkey
brain in relating parietal and frontal
cortex
in taking visual information and
converting it into hand control but he
was also
actively involved with becky's group
so that there was a bridge there from
the the brain modeling we were doing at
the time
into the the robot control and then
tony lewis was looking at the evolution
of neural networks
and that actually lays lay in some sense
a basis for a
later thesis at the university of
edinburgh by alki iceberg
on how to evolve a
a fischer's a a lamprey's
spinal generator for swimming into a
salamander's spinal network that could
do both swimming and locomotion on land
and and he later came here with his uh
then wife
um to do a post-doc but that's that's a
different thing but so there were some
successes i would say from that
collaboration but it also became clear
that we were not making as much progress
as we should
in part because both of us were doing a
lot of different things
in part because the students tended to
be perhaps too purely
computer science students so we we had a
few lucky students
like lewis and [ __ ] who who did enjoy
tinkering but we had others who
didn't really know what to do if the
robot stopped
and so at that stage we
were successful in recruiting a couple
of very bright young people
then young people to come in as
assistant professors
and basically handed over robotics to
them
so um stefan schaal in terms of the
the hand control and now that's moved on
to a lot of
other issues where you're really
thinking about how
various learning algorithms can be
applied to really look at the dynamic
control of complex
manipulators skeletal systems and so on
and then maya mataric who came to us
from rodney brooks at mit
who came in more to look at the social
robots where
you don't really care you don't really
need to know much about the guts of the
robot
it's it's got a simple control it'll
scoot around but how do you
how do you work on those controllers to
make them them interact
and then there was a a good phd student
of george becky's
named gareb sukhatmi so around
1999 2000 we promoted him from
phd to postdoc to assistant professor
and he has become
very dynamic since so with matari shah
tsukame and and some interaction with
law eti who i hired
to work on the bridge between computer
vision
and and brain vision we really have a
very strong robotics group here but i
would say
by now i'm just a brain guy and
leaving the working with the real
machines to them
um my own interests were captured i
would say
in the late 1990s
we got some money to collaborate with
um an international group of people in
japan france
and italy on
the neural mechanisms controlling hand
movements
on the basis of visual input and
anti-fag thesis was a part of that
and then this group discovered something
called mirror neurons um
czecho maritzalatti and his colleagues
in parma which bridge
between firing both to code the action
the monkey is doing and decode
an action the monkey is seeing someone
else do
and then we here at usc um with a
brain imaging expert called scott
grafton image the human brain
so we weren't there looking at
individual neurons coding
the action two ways because at least are
there chunks of brain that light up more
for interacting with objects and
seeing someone else interact on it's
rather than just looking at the object
alone
and we found that that seemed to be in
broker's area which is traditionally
thought of as a language area
so ever since i've been continuing
modeling the macaque brain but also
developing a theory of language
evolution that
suggests that the mirror neurons might
be at the heart of
an increasingly elaborate set of
circuitry that allows the human to
learn and use language
and that is my life have you seen that
mirror neuron model
being used in robotic learning systems
erhan aztop who um was the first student
to model mirror neurons with me
uh after his phd went to atr in japan to
work with mitsuha kawato who's
probably the leading person on the brain
robot
interface in in japan and so
they they have done some work though i
would say their
their mirror neurons in the robotic case
are not as interesting as the
the brain mirror neurons but there was
also a collaboration between them
and daniel walper in london
on mental state inference where they
were looking at how you'd have internal
models for different actions and how you
could deploy them in different
combinations on the basis of the mirror
neurons
we'll be running a workshop here at the
end of july
where we're bridging between models of
action and action recognition
neuro-linguistic models and the
neuro-informatics that can support that
and one of the
people we'll be inviting is janice
demiros from
london imperial college who is working
on imitation in robots
because in fact some of the modeling he
has been doing
overlaps with what oztop and i did
in thinking about the relevance of
mirror neurons for
larger systems that could do imitation
the mirror neurons by themselves just
say i'll recognize an action i already
know
so you need a bigger system to say oh
there's something i'd like to do but i
don't know how to do it
how can i add it to my repertoire
who are some of the other partners in
this international collaboration you
mentioned
parma yes okay so
i i mentioned that my interest in hands
started back in 1979 with the work of
mark genero so he was the french anchor
for this effort um ritz salati in
parma
who looked at the parietal cortex he was
basically looking at them
recording from the monkey brain to look
at how vision was coded in a way that
was relevant to action
and retaliati's group was more looking
at the frontal cortex to say okay if i
get that visual input what do i do with
it
and then general was more in terms of
okay those ideas are great let's see how
i can apply them in analyzing human
behavior
and then we had another modeler at that
time who's since gone on to do other
things but michael jordan
who did his phd with rommelhart in
ucsd where i met him because i was on
sabbatical there while he was finishing
his phd
and then he for a while was very much
engaged in motor control
and then he basically handed over to two
of his
phd students dan walput i've already
mentioned and garamani
and then he went into graphical models
of machine learning
and is now at berkeley i think where
he's done very well
so that was the the initial group and
then we brought in
other people to do some brain imaging in
later years
and so kawato came became involved and
saw this
just later on when aircon now kawato is
a different path
so uh back at umass i had something
i founded something called the center
for systems neuroscience with
funding from the sloan foundation
and in the first year of that one of the
fellows who came to spend
a year with us was sunichi omari
who has gone on to become the the most
influential
person in the mathematical theory of
neural networks in japan i mean that was
many years ago so
he came in 1977 i think so amari and i
organized a meeting in kyoto
in 1982 on competition and cooperation
in neural nets
that in part built on the work we've
done together in amherst but
also introduced a lot of other people
and one of the bright young
things at that meeting was mitsuhata who
just finished his phd
so as a result of that i got to know
mitsuha quite well
and visited him in atr quite a few times
and so that led to the fact that
stefan shall who had been doing a
postdoc
for several years with kawato when he
was looking for a university position
it was on the basis of that
long-standing arrangement with koato
that i came to be able to help recruit
him here to
to usc but also on that basis um
several of my phd students went on to do
post-docs with kuato so
the nikolai schweigerfer who did a phd
with me
on the cerebellum uh
worked with kawada for several years and
he's actually now on the usc faculty but
over in
biokinesiology and physical therapy
looking at how to apply
his modeling to rehabilitation how does
the
neural net how do the neural networks of
the brain change
uh after a stroke how can we help the
patient recover their functionality
um erihan ostop then followed in those
same footsteps and then the third person
i think from my group who went after his
phd was
jacob spolster another person who worked
on modelling the cerebellum
so that's that's a long-standing
relationship
and so it's included discussions of the
work
he does on robotics but i would say the
the serious interaction has probably
been more on general modeling
of control of hand movements without
worrying too much whether it's a
a monkey brain a human brain or a robot
brain that's that's doing the
controlling
whereas as you know stefan shall
really got involved in the the detailed
algorithms for control of the sarcos
a robot very beautiful humanoid robot
what do you think are some of the most
significant
neural models or mathematical models
that have been influential in the
evolution of robotics so
common filters specific forms of neural
networks
what's your personal impression of
what's been some of the breakthroughs
well modeling the cerebellum has
certainly had quite an impact
um there was the the cam model which was
directly influenced by that
jim albus who did a phd thesis
published in 1970 on the cerebellum not
only came up with that model but was
very influential at the national
um institute of standards whatever
it's missed these days i forget the
acronym
um so thinking about the cerebellum um
and then michael paulin a new zealander
was the one who really pushed using the
kalman filter as a model
of the cerebellum so that ties in with
that
brain influence work but on the other
hand the kalman filter is
such a general mechanism that it's
certainly been
well applied
i mean clearly i would think optic flow
algorithms in terms of robot navigation
depth algorithms more generally whether
they're static stereo or
dynamic optic flow have certainly been
important
the work of charles is typical of those
people who are using
um complex non-linear learning models
to to really get at the problem of if i
have a robot that is
very flexible i can't afford to uh
use a a sloppy algorithm uh
the way i can with a robot with a few
degrees of freedom and very rigid
joints and so on so i think that's
another area
and then people have continued work on
planning
and that's probably an area where neural
networks to date have not been as
important as more traditional
ai models um
from my point of view another thing
that's very important is thinking about
distributed control
so you're not just looking at the um
a serial flow in this case you're really
thinking about how to distribute control
to different parts of the system and
it's interesting there that if we look
at today's motor cars they already
have vast networks of
of of computers on board doing
everything from
optic flow to avoid collisions to
self-parking of the car at the other end
so um the interesting point there i
suppose is
that the field of robotics
has moved perhaps out of its
specialized niche into computer science
more generally as we think about
um embedding computers and systems which
have sophisticated sensors and so on so
that
the skills that go into robotics are not
just for
animaloid or humanoid systems but for
very diverse vehicles as it were which
which can be platforms for integrating
sensation and action
in terms of optic flow did you find the
work of jj gibson to be
informative or influential well jj
gibson is where we started
well jj gibson in terms of the general
concept of
optic flow and then part of our work has
been influenced by the general notion of
affordances
david lee in terms of looking at actual
use of optic flow and controlling
the dive of the gannet towards the water
to catch the fish and
the the judgment of pace length by
people doing the long jump and things of
that kind
so those those two set the stage in
which our particular
quest for how do we come up with brain
like parallel algorithms for extracting
optic flow
took place the debate with that man
and actually was a debate when i was at
umass we invited gibson
to come and debate me because he had
this idea of direct
perception the notion that somehow the
brain just picked up affordances
including optic flow and so
where the debate was joined was
basically i was saying yes i agree with
everything you're saying about
we pick up this information but look
it's not direct
and we have to understand the neural
networks that
that carry out these performances so so
conceptually
gibson was tremendously important
computationally
he wasn't in the running and uh so we we
built on his work in that way
what was his reaction to you he was a
nice guy
no no i i mean it was just we we had a
debate it was very well attended by
the psychologists as well as the
computer scientists and i think
we we both enjoyed the interaction and
uh
i don't think either of us budged an
inch but i think the audience
benefited from getting a deeper
understanding of what gibson was about
by seeing how he responded to my
questions but perhaps
getting also a fuller understanding of
what neural networks were about if if
they were pure gibsonians who had just
looked at
what information is being picked up
was it your sense that the people
interested in robotics were less
concerned about the
symbolic ai versus neural network sort
of paradigm struggle
and we're more willing to engage with
neural networks than the more
traditional kind of cs people
who might have been more interested in
the debate
i think one has to be careful because my
sample is completely biased so in other
words i interacted with people who were
interested in interacting with me and if
they wanted to interact with me
then they wanted to see what they could
learn
in the way of biologically inspired
robotics
so um i wouldn't claim any typicality
there i think other people were
if you look at the people doing mobile
robots there
they probably put on some laser range
finders didn't worry too much about the
um
the details of the visual processing um
the the controllers were very simple for
steering
the robots and so a lot of their work
would be more at the planning
stage an exception to that would be
um ronakin's work where he really was
interposing a more
potential field approach based on the
frog stuff and
other people did pick up on that and
then of course
in the early 80s rodney brooks came up
with
what i thought was
a reduced form of my schema theory and
what unfortunately most people thought
was something brand new
i'm sure you've got a lot of this in
these interviews and um
where he had a layered hierarchy but but
to my mind
the whole point was that you had to have
representations to close the loop in
general if you
if you were looking for example at
navigation where you had to know the
layout of
a complex environment so but but
the point is that that work of rodney
brooks neatly complemented what ron
arkham had taken
from a more biological point of view and
created this
sort of field of behavior-based robotics
and so on
but probably if one writes the history
of it rodney's
strict hierarchy of representation-free
schemas was more influential at the
beginning at least
i think in the end everybody realized as
as ron himself
pushed that you had to see how to
integrate
the delegation of many functions to
perceptual and motor schemas
uh but still that had to interact with
with long-term knowledge about the world
and in the end i think it's a moot
debate you just said well in some cases
the robot
doesn't need much of a representation of
its world another case it does
and so to be good at robotics you really
have to integrate the
the stuff which is more easy to relate
to neural networks of the perceptual and
the motor ends
to networks that may tend to be more
abstract
interacting schemas or even high-level
programming languages that can
bridge into into representations and
we've done some modeling in the years
since
of rat navigation and looking at models
of the hippocampus
and those models actually go all the way
back to the same year that sunichi omari
was a visitor we had an israeli visitor
named israel lieblick
and we were looking at
finding behavior in rats which had gone
out of fashion so we went back to the
classic literature
and came up with something called the
world graph the idea
that the rat was on the one hand
worrying about the affordances it won't
go through
if there wasn't space to go through but
it had to know where it
got food where it had got other things
and that
coincidentally came out a year before a
classic book called
the hippocampus is a cognitive map so in
fact we had a model of the hippocampus
as a cognitive map before
the book which declared this as a major
theme but we were already influenced a
little bit by the
early findings on place cells and
epicampus so i think that's another way
in which
i wouldn't say that study has been
very influential outside a relatively
small circle but for me it
it gave a way of thinking about a hybrid
system where we had
certain parts of the system were
developed as neural networks
for the as it were the immediate if you
will the bruxian
interactions with the environment to
avoid collisions find a place to go
things of that kind
and a representation this world graph of
knowing about your world knowing where
food is
and so on and another thing interesting
about it is that
i mean just trivial personal history but
this
in some sense was reinforcement learning
before reinforcement
learning existed because we had the
animal was hungry then the reinforcement
of food was what mattered that the
animal was thirsty
the reinforcement of water was what
mattered in any case avoiding a painful
electric shock
was important and so in fact much
later after the theory of reinforcement
learning had been well established on
the basis of
of the sutton work then we were able to
improve our original model
by formally using some of the theory of
reinforcement
learning to fit things that have been
done in an ad hoc fashion on the first
on the first pass so
that was mostly sort of perceptual side
but
did you see this also on the motor side
in terms of the central pattern
generators
being controlled by representations and
or
grasping and trajectories of them
okay so um the
central patent generators were always
interesting
um we had some interaction
and i think the at the ucsd sabbatical
where i met
um actually no it goes goes way back
okay so way way way back at stanford
uh there were two people working on uh
central pattern generators and
insects um don wilson
who died rather young in a boating
accident on the snake river i think
and don kennedy who went on to be
chairman of the fda and president of
stanford
and so the study of central pattern
generators was always an
interesting theme for us but apart from
tony lewis's thesis and alky i spirits
um
postdoc that hasn't been something that
we've really
developed as this thing from being part
of just the general culture that i share
with my students
um and then with the the hands
the whole point was that it wasn't a
central pattern generator
in the sense of an ongoing rhythm to be
modulated it really was a discrete
movement
and in fact one of stephen charles
papers is where he claims to have
differential brain imaging for discrete
movements versus rhythmic movements
so um the idea of okay you've got a
target and then you're
coordinating two controllers one to get
the arm to the right place
to carry the hand the other to get the
hand in the right shape to grasp the
particular object
that has been a research issue that
we've looked at both in terms of neural
networks
and in one phd thesis with a guy named
bruce hoff
in terms of adaptive control theory
where we
again interacted with the genero group
in terms of data
on what happens when you perturb the
position of an object or perturb
the size of an object how can the
controller
adapt to that and we came up with a very
interesting model that explains some
rather strange anomalies so that if the
hand is coming out like this and you
suddenly switch
the size of the object it may
in some cases not stay like this
and go into the new size the audio but
it may actually sort of shrink down and
open up again
and sorry that's if you displace the
distance to the object so we came up
with an optimality criterion
for the coupling to say basically it
costs you to hold the hand open
so if you if you're suddenly told i've
got further to go before i grasp then it
may be worth your while to close the
hand for a while
in terms of that optimality criterion so
that allowed us to explain what it first
looked like some very
strange behavior that genero and his
colleagues have observed in their work
in france
you discussed a lot of different ways
about how your
kind of work on different biological
systems
um influence things and robotics i was
curious if your
interactions or bodies that's in your
work in robotics what kind of
effect it had on um on the rest of your
your work
well as i say there was a specific
period
at umass with the lab for perceptual
robotics
and then when i came here interacting
with becky where we were actually
looking at
robots as well as continuing the brain
modeling
and there there was a real conversation
thinking about the optic flow in terms
of how the robot could use it
fed into our thought about the the other
system and then thinking about
hapsis of the sense of touch was
influenced
by the robotics but but i would say that
in my personal
history it is more trying to understand
the brain
and the robotics is a source of
stimulation and occasional application
rather than many other people you'll
talk to for whom they're
roboticists and they learn a little bit
from the biology occasionally rather
than
centrally and now there seems to be
quite a push for cognitive robotics as a
new area of
research and expansion do you have any
comments on that or is that something
interesting that's what i've done for 40
years
if they want to say it's new good on
them but it's the same with
embodiment and so on i mean this is what
we've been doing ever since we started
work on
on the frogs and suddenly people in ai
and cognitive science are talking about
embodied ai embodied robotics but
i mean not everybody robotics embodied
cognition but really this has been
a central theme ever since the late 60s
in my work
so what year did you create the
lab for perceptual robotics
1980-ish
um yeah probably
might have been a couple of years
earlier but 1980-ish would be right and
what year did you come to usc
1986.
and the perceptual lab is still going
yeah well
when i was last at umass about three
years ago it was and a guy named rod
groupen
was at that time leading it and not only
as i said before with a much wider range
of robots than we had when i left
but nonetheless building on the schema
theory that i had initiated so it was
nice to see that those ideas that
i developed at ronak and developed the
damien lions who came up with our rs
schema language that that work that
had been from the early 80s basically
at umass that the person who had taken
over the lab for perceptual robotics was
in some sense building on that tradition
rather than just doing any old robotics
as it were
and did you actually collaborate with
ken salisbury at all or just
worked with his hand well in terms of
getting the hand and
learning how to use it we interacted
with him um
but we didn't i think feed back into the
design of a new generation of salisbury
hands or anything like that it was more
that we were
customers and he was pleased to be
informed of what we were doing with it
and i'm sure we showed him the touch
sensors but
i cannot remember at this stage whether
he said oh great i'm going to add those
touch sensors to my hand or whether he
said that's nice thank you
that that is gone from my synopsis
do you what do you think are some of the
the future
challenges um in this in a field of
brain
robotics
what do you think are some of the things
that people are coming across
now as big issues that they might want
to deal with or
some of the things that are going to be
coming up in the next five to 10 years
just about everything i mean in terms of
visual
perception when i worked with hansen and
reisman
and this work peaked probably about 1980
on the visions
system for scene perception i thought
that in the next few years we would have
a general system for analyzing objects
in the scene and how they move it's not
happened
the funding has all gone into build me a
very special purpose vision system that
does a particular thing
so i think general scene perception uh
is still remote
um we've seen a lot of progress lately
in object perception
picking out a particular object in the
scene
mirror system stuff is how do you
recognize what a hand is going to do to
an
object more generally how do you
recognize actions
i think we have very little going so
especially if we're concerned with
robotics for social assistance of of
patients and children and others being
able
to flexibly recognize
what's going on around you not just what
objects are there but what
humans are doing so that you can either
get out of the way or
protect them from harm or assist them in
some way i think that's
maybe so i'd just say on the on the
vision side just
that general thing of dynamic what we
might call dynamic scene perception to
be able to recognize
agents objects and the actions that link
them
on the motor control side
i would suspect that the sort of work
that i associated before with stephen
charles humanoid control and so on will
continue
a lot of work in mechanical engineering
of different systems
um as i mentioned before
the question of how much of the robotics
of the future will be
so strictly humanoid and how much we'll
be able to
as it were exploit different mechanical
systems i don't know so that's a
the mechanical engineering end and the
control theory and is tremendously
important but that's
probably rather separate
everything in between you know
planning is still pretty primitive
it's fine if you've got a very
constrained environment
but going beyond that and then language
i mean that's not really robotics but
if we're going to have robots that are
built
as some of them will be for human
interaction
then a decent language interface that
can pick up on ambiguities
and use the current context to
to remove some of those ambiguities are
going to be very important
so
i think the general point is i i see
robotics as blurring
it's not really a separate subject
anymore just as computer science has
blurred when i when i set up the
department at umass
um when i persuaded some skeptics
to allow becky to hire me here
the view of computer science was pretty
narrow um
you know we do operating systems to do a
little bit of theory of turing machines
and beyond that
it's not computer science and now
we have one of the things i initiated
here
was a seminar for a phd student just
each week to have one of our faculty
come in and talk about their research
and i for the first year or two that i
would sit in on the lectures
and uh you know the barriers have gone
with we've got paul debovek who
uh as his computer science work came up
with a system
for facial motion capture that powered
the
success of the avatar movie and
we've got people doing sensor networks
so gaurav sikatmi is both looking at
mobile robots but he's also looking at
what happens if you scatter a few
thousand
sensors in a bay to monitor the
ecosystem
there's no line between those two two
projects
um so the idea that computer science is
no longer
symbolic manipulation in a box but
but embodies all these techniques of
perception all these
diverse forms of effector control and so
on
means i think that although there will
still be
a certain class of
computerized devices that people
recognize as robots probably the more
humanoid
ones the the expertise that
is in robotics will not be segregated
from
this very dynamic network of
of computer science research areas
that's emerged in the last
20 years let's say and you yourself have
had a very
um trans inter or trans-disciplinary i
don't know how you would call it
um career and even
kind of belonging to different parts of
the institution how did you
kind of navigate that
because if it makes sense
is it challenging talk very fast
yeah i mean i think there is always a
danger um
that if you do too many things you end
up not being good at any of them
so i think the answer has been not to
try everything all at once
i mean in fact when i came here
25 years ago i dropped my work in
in algebra theory of computation for
example which was
a sad thing to do but i just couldn't
establish the program here and then as
we said later on
it became prudent to hire these great
younger people
like shaolin matarich and then sukkatmi
and eti
to take over the robotics so i think
that
and now i'm obsessed with the evolution
of language i've always worked on
linguistics off and on since i knew
chomsky as a graduate student when i was
a graduate student not here
um but i think that's probably that in
any particular year
um there have been real foci
grounded by having very good phd
students
and but at the same time the base of
knowledge is increasing so
that opportunities can't later so for
example i had a
group of four phd students at umass
working on schema theory approaches to
language that came to an end when i left
there in 86
but then when about 11 years later
we did the brain imaging which said oh
look these mirror neurons and the human
appear to be in
broca's area language area i was then
ready to be able to
to land on my feet running and returning
to to language but now
tying it into an evolutionary scenario
as well as trying to think through
whether this scenario gave us a new
approach for thinking about the human
brain and neurolinguistics which it has
done
i had it okay just a quick one i was
curious about you mentioned kind of
changing places and the availability of
students and things like that as having
an impact
what about funding did
is that answer your question yes no i'd
say funding has been very difficult
because i think what happens is that
the funding agencies prefer narrow
focused proposals
and they also prefer proposals except
for very young people that are based on
a track record in a particular area
so that i would say that that's the one
dimension of my career that has not been
as successful as i would like it to be
there have been periods where
i have been well-funded other periods
where it's the
teaching assistantships and fellowships
for the students that have basically
kept things
things going so i wouldn't say i would
recommend
this diverse career to most people but
uh and there are times when i i sort of
envy some of those people who stayed
very well focused and gone
gone deep uh
but overall i think despite that
nastiness with funding from time to time
it's been worth it
because having this broad perspective
has been very exciting
yeah do you find that the
interdisciplinarity was inspired by the
cyberneticians who were also very
interdisciplinary in their approach
oh well that's how it started i would
say because until i read wiener's book i
was a pure mathematician and my
in fact my freshman year at university i
founded a little group called the
useless mathematics mob with the idea
that if mathematics could be applied it
wasn't worth doing
and then i was changed by cybernetics
and that got me reading um and then as i
said i
you know met neurophysiologists and
started sitting in on their experiments
and
learning about the frog's brain and that
led to
mit as an interdisciplinary hotbed as it
were so
by the time i finished my senior year i
would say i was already
mocked for for this career
in terms of actually working with weiner
and
interacting with them what were they
like as people
and what was it like to be a student
learning from that
i have a story but i'm not going to put
it on tape
no if you send me an email i'll send you
a writer but i'm not doing it
um well they were very different people
weiner was on the one hand
terribly brilliant and on the other hand
terribly insecure
and if you read his
autobiography you get to understand it i
think
as a child he was pushed by his father
to be a child prodigy
so he got his bachelor's degree from
tufts i think at 14 and harvard at 18.
but as i understand it his father
instead of congratulating him
on how bright he was would sort of say
see
norbert my theories really work even you
can do
well so he always had this
need for praise and i i think when it
came to graduate students
he was more tuned to the graduate
student who would
appreciate him overtly rather than being
able to pick what were necessarily the
best graduate student
so that's weiner
um probably a little autistic
very bright he wrote a novel called the
tempter
and if you read that novel and look at
the love scenes that novel you could be
pretty sure the guy did not really
understand
the human emotion of being in love
so um mcculloch described the difference
between them
i don't know where this helps you in the
british civil war
back in the 1600s there were the
cavaliers
who were the proponents of the king who
wore
hats with feathers coming out of them
and were very lavishly dressed and
the round heads who went with oliver
cromwell who were very dowly
dressed inside so he characterized
himself mcculloch as the cavalier to
wiener's round head
and uh we mcculloch's problem
what was i thought there was a problem
but he was a romantic
and and so when you talk to him it was
tremendously exciting
but you could never be sure what parts
of the account really were anchored in
the current
neuroscience and what counts were his
extrapolation of how it had to be
so if you knew him well enough or were
thoughtful enough
to say i'll take the inspiration but i'm
responsible for
sifting through it then he was great so
he had a very nice group around him he
had an office
which was lined with books and filing
cabinets and then there was a
blackboard about yay big as he would say
and there was a
an old packing case and so all the best
people would come into this office
and they'd stand up on the packing case
and scribble and chalk on this little
board so just
sitting in a chair in the corner of that
office around the many tables that were
there
was an education in itself so i would
say that
although i respected weiner as a
mathematician in the end i didn't get
all that much out of interacting with
him
um whereas mcculloch although he didn't
contribute so much to my
formal development uh was in some sense
my guru and
thinking seeing how he thought about the
brain and then being critical about it
not in terms of accepting it but rather
saying okay that's a starting point now
all these people are coming into the
office what are they saying how can i
understand that
that really i think created a broader
intellectual framework
um that that has become part of the way
i approach things
did you interact much with let finn or
pitts
okay let van i interact with quite a lot
um
pitts unfortunately by that time already
had the dts
and as an undergraduate i had
written a paper which came out in the
journal of the acm in 1961
just after i started at mit but it
included a reanalysis of some of the
ideas in the 1943 paper and i went to
discuss those with pips
and unfortunately um he just started
shaking
and so we never had a real conversation
but talking to latvian about the brain
allegran's problem was that he was also
a sort of romantic so he didn't believe
in rigorous he
he claimed that he once got a grant from
nsf
but he would only accept the grant if
they promise he didn't have to publish
anything
that he sort of felt that just grappling
with the brain and coming to understand
it was
was great that one shouldn't be bound by
this so i feel that um
people i talk to as a graduate student
over at harvard medical school were
huble and weasel especially huble who
had done the basic work on the
edge detectors in visual cortex
and
they were rigorous and
spawned a lot of people who were working
in the rigorous development of their
original findings and they got the nobel
prize
latvin's strategy i think i thought his
results were actually more
important in and of themselves
but his approach just blocked any real
coherent development of his group
into something that had a rich body of
results rather than these few very
tantalizing and important results
so lots of interesting personalities
yeah yeah
and and shannon what was he like as a
personality
well shannon had a very fertile mind so
on the one hand he had these rigorous
mathematical theories of communication
but he was always coming up with
interesting ideas so one of his ideas
was an assisted olympics where for
example
you would have large elastic bands that
would be around the thighs of the
athletes so that
the return stroke would be augmented
as they ran and so they would perhaps be
able to run faster than would otherwise
be possible
and as i mentioned earlier he had this
wonderful collection of
things on the pond behind his house
outside boston
including the the bicycle honor pontoons
so interesting guy and colin
what was coleman like when he visited
him
well i did my post doc with him and we
put out a book together
um combining some of my ideas and
automated theory
with some of his work at that time he
was getting into
applying algebraic geometry to linear
systems
and then a guy named peter falb who had
who was the person i first heard about
him from
um he was a somewhat difficult person to
work with
because if you came up with a result
either it wasn't interesting or he'd
already thought of it so that was a
little frustrating
but what he came up with i mean he
really is
probably the most important person in
mathematical contributions to control
theory
uh after wiener wiener came up with with
hopf with the wiener hop
filter and kalman really pushed it into
the
into the age of computation by replacing
tell me the infinite time
series till now and i'll tell you what
comes in the future just saying here's a
a finite state system that will
record in compact form what you need to
know about the history to make
good predictions about the future so
that was fundamental
so final question um
so would you have any advice for young
people who are interested in
robotics
no you want me to say more or just leave
it alone
you've warned them
[Laughter]
well i mean i think in a sense it's
implicit in what we've been discussing
that um the boundaries between robotics
and computer science and neuroscience
and cognitive science
are no longer impermeable
they've always been a little porous but
now
they're they're really porous and that i
think the
the big breakthroughs are going to be
made not by saying i'm a robotist
in some narrow sense the word i don't
think it's good advice to anybody to say
okay
there are so many connections i'm going
to master them all and work on them all
but i think rather by finding an
interdisciplinary conversation rather
than the whole schmear
that's going to be the way to get new
insights but
you know it's going to depend on
discipline you're going to have to learn
a lot about
the mathematics underlying it or a lot
about the
mechanics underlying it or a lot about
the neurobiology so whatever you do it's
not going to be
any more what it might have been 40
years ago that there are some
easy picking some low-hanging fruit it's
probably going to require an intense
amount of work
at one of those interfaces to to bring
back really new ideas to robotics
or as we've discussed to export ideas
from really working with the embodied
systems
because for example now there are there
are people who are
really into computational neuroscience
but suggesting that by using robots not
as devices to be technologically
important but by
showing that your model of the brain
really works with real senses and real
effectors so that's where the influence
is beginning to go the other way so all
these exciting specialities each of
which is interdisciplinary each of which
requires a lot of rigorous hard work but
that's where the future lies i think
and as a sort of follow-up was there a
point in your recollection where people
started talking about robotics
as a field instead of the sort of
different disciplines that control
theory or something like that when
there's
suddenly some moment where there's
robotics labs or robotics conferences
the 13th of june 1976