and could you just start by telling us
your name where you were born and where
you grew up
i'm gary bradsky um i was born in
arizona basically grew up in silicon
valley in palo alto
a long time ago where did you go to
school at palo alto high school
and uh your undergraduate at berkeley
what did you study there electrical
engineering
what motivated you to do that i
was uh originally a physics major and i
decided
uh at that time i didn't want to have to
go all the way through
on a phd to get a phd to do anything
sort of useful
so i thought with ee i could
work for a while i always intended to go
back to graduate school but i didn't
want to go right away
and were you involved in robotics
applications and projects at that time
no
is involved in wind turbine design and
professor otto smith then
when did you first sort of encounter
robots or become interested
in a way so i my background as
uh well my graduate studies were in sort
of
biological vision modeling the human
perception system and again when i got
out since
since i took my time i was an older
student at the time i decided i didn't
really want to hang around
and do docs and
keep getting paid little and so i
decided i'd switch into
computer vision and sort of formally
and i ended up after a bunch of detours
i actually worked as a quant on a and a
wall street kind of operation for a
while i didn't like it although it paid
well
and so then i went back to research
until research and that's where i really
started computer vision because
i had some background and they wanted
to work on this image processing library
and so that's where i
started there with this library group on
the promise i could go into the research
group and that happened a year later and
it turned out to be
a great connection because that's where
i started the open source computer
vision library
and you know that's that's one of the
things i've been running or it's become
very popular over time
but i wouldn't have been able to do that
without the sort of knowing the people
in the library group
who weren't doing anything like that but
i just started this as my own initiative
but
and and so where i met finally
intersected with robots was
sebastian thrawn came by well he's
coming to stanford
i knew he was so so i don't remember how
it happened
actually but he came he split a
sabbatical and spent some days of the
week at
intel research and
you know so he had a little row of an
erratic you know that would
wander around and we were doing some
things there but then i had a sabbatical
coming up
and so i he talked me into going to
stanford
for the sabbatical i was going to go to
switzerland and i ended up
not during the sabbatical but right
after you know the darpa grand challenge
had run
and stanford wasn't in it thrawn didn't
go the first time
and the second time was was then queuing
up
and i was i was just visiting him and he
said oh let me show you
this stanley what i mean it wasn't
it wasn't stanley at the time it's just
a uh
a a volkswagen
suv and so i looked at it and said well
i'm in this looks like fun so
so that was really the first robotics
project
i'd been on although i've been doing
computer vision and it you know dabbled
around in
in little stuff that has robot
capable but you know so i organized the
computer vision team
and and we won the race
where did you do your graduate work at
um in boston
at boston university under grossberg
this was biological modeling of the
brain
so i guess grossberg had broken off from
mit
and and i decided i could become
interested in the brain because
prior to way back i had worked at this
place called neuroscience which was a
computerized
one of the first computerized eeg
companies and then i got interested in
the brain
and i figured this he was doing neural
networks
biological neural networks sort of
modeling perception i thought well
this is the best way to combine these
interests and so
you know i went i went to him i was on a
business trip and i talked to him and i
go okay
let's just go to graduate school here so
so did you really
learn a lot from the biological systems
about how to build better computer
vision
no i mean i learned a lot i keep
hoping to put it to use there's maybe a
few principles but
i don't think biology
has has informed computers
much i mean in a general sense uh
uh you know von neumann whatever his
first
things he was thinking of were neurons
and
and and then there's you know some
things now there's this deep belief
learning and some some things in ai
you know are influenced by some
conception of what the brain is doing
but
it's been pretty weak link and
everything i found to be successful
sadly is with robotics is
uh you really engineer the system so i
haven't really done
you know the bio-inspired robotics and
whatever
i also don't see them as being such a
big success
yet i mean it hasn't happened you know
what's a big success of these industrial
arms and
stanley and you know the slam and all
these are very
sort of engineered systems so you know
it seems like
you know that's what we're able to make
successful and
whatever maybe the hardware that we have
is just too different from the brain
to to make some connection
i can go on and on you know like people
in ai
they're always thinking if we had enough
processing power that's when ai is going
to happen well we have
we have essentially equivalent in
different modes to
you know to compete with the brain or at
least approximate it so that hasn't
magically made ai appear and then you
know
they go well if we had enough data well
often
you know if you calculate it out when
google trains on an image they often
train on
you know to get some object recognition
they often train on more images than a
human sees in a lifetime
and yet you know we're arguably
quite a bit better so and so none of
those things have panned out i mean
there's fundamental things we still
don't
understand and you know kind of people
on the cutting edge are
trying things so uh you know they have
this sparse coding theory
and you know like andrewing at stanford
is is
uh working on these things and maybe
some of that will pan out but
and so when you were at intel um what
motivated you to
really go into the open source
computation uh computer vision library
like what
why was it important to make it open
source because
so i joined intel and at the time
i guess their charter was you know write
papers and get
their thought was let's get this on the
map until research
and but i looked around and said well
you know i could
i could be in a postdoc or whatever and
write papers maybe even better you know
and
people in academia are pretty good at
writing papers i said like what
can i do here that's more impact or more
leverage
and i was also you know at the time
visiting
mit media labs and i saw i was looking
at the students there
you know they were doing a lot more
because they had so much infrastructure
built up like
when you come into the media lab you
weren't just okay i'm going to write my
own
you know gradient detector and then i'll
build that up
they had like this whole system they had
stereo working they had everything
working
and then they come on and and built on
top of that
and so you know one of my ideas was i go
well
you know what intel really wants is is
to drive the need for
you know processing power in the world
you know like computers were getting
fast enough for spreadsheets and
whatever
and and so i go well that combines with
you know that sort of charter
if i can get i can speed up people doing
computer vision by giving them the same
kind of infrastructure that a student at
the media labs would start with so i
would make that universal
and so you know that was the the idea
at the time and you know
you know in life very few of your ideas
pan out but that original conception for
opencv
i think is really panned out like it did
boost computer vision it did get
uh increased the need for mips or
you know out in the world and you know
spread around
so i think you know it it did answer
that problem and still does you know so
when did you start on that
1999. well no
no no 1999 i think i think you know it
started as a little project on my hard
disk and then
at the time intel had these russian
contractors who then became employees
and i
sort of cobbled them in
you know before i actually had
permission when someone went on
sabbatical i took their group and
started them working
by the time they came back it had too
much momentum behind it and so they made
it official project
and the i think 2001 was when we
actually came out with it at cbpr
and what were some of the challenges of
building a unified vision library
uh the same challenges we had i mean so
i wouldn't say it's unified i mean
that's the challenge
i mean so there's a certain
you know how complex you should be and
how much you should include
so i mean opencv is over 2500 functions
now and
you know not all of them are used and
not all of them should be in there
but you know so what to put in and what
to keep out
is a challenge i think you know how to
design
and architect the library at the time
you know what the
c was uh c plus plus had this name
mangling problem so we stuck with c
of course you have a lot more tools in c
plus plus and we finally have switched
the whole library over
to c plus plus the c interface remains
probably forever there's no reason to
get rid of it really
but you know i've seen other libraries
that just went
that came later and so they had more
like boost tools and
they just went what i think was
overboard and templatization
and so what i really i think one of the
good things in the library and what i
wanted is i didn't want you have to
learn
like a system before you learn to
function i want if you wanted to do
candy edge detection
you just take and do candy edge
detection and so
i think to that extent that was good and
that's one of the reasons the library
spread because a lot of other
libraries are really frameworks and then
you go well i have to learn the
framework and get all this buy-in
and opencv you know you just i mean
people use it in bits and pieces and
that's how it was intended
but we have now tried to organize it
into modules so it's getting more
categorical but there's
there's no there's still no overriding
framework
that you have to learn but you were also
involved with the open source machine
learning
yeah so well there's a machine learning
library in opencv
that came out of this it's statistical
machine learning
that i i did with um i
when the the dot bust happened i figured
like everything's falling through to the
floor
maybe computer vision like who are you
gonna cut you know the microprocessor
designer this computer vision guy i
thought i better get a little more
relevant to the
the core of intel and so
what i did is i started doing uh
projects with intel manufacturing and
involved this statistical machine
learning so
i sort of do both things and and
we were i actually came out with a bayes
net learning library that's still out
there somewhere but i just ran out of
cycles to run all those projects and
and i guess the guy who was on it never
really drove it as much that
it takes a certain personality to keep
open source going
but in manufacturing there was just no
way they weren't gonna
we had to get them on to decision trees
and things not to base nets so we gave
up on that we just used these
statistical techniques and and there's a
pretty successful
project in basically reducing the need
for testing so you'd have faster
throughput and
subject to some pretty hard constraints
on
they didn't want to see they didn't want
to lose the statistics so we had to
impose random sampling on it and they
didn't want to see the numbers change
at the end despite the fact that we
showed that it it's sort of
counter-intuitive that it's actually
better in some cases to have the parts
fail at the end and just
run them through and then bake them in
even though that seems like it's more
expensive it actually
turned out with all these steps not to
be in the slow downs but but they
wouldn't hear of that so
but i mean that that thing is now
running in the manufacturing plants so
so when i after i went back to
the computer vision library i drew in
those techniques we actually had to
rewrite them because now they were
you know part of the manufacturing
process and so top secret
right so we had to rewrite all the code
for the decision trees and
and all that stuff and so that library
is still in there it's meant
now it's focused on object recognition
but it has
a whole bunch of statistical routines
and and lately it's
added from marius muja david lowe's
group
it's it's added flan which is a fast
approximate nearest neighbor so
we use that quite a bit now and
how is that all being used within uh
ross
so i mean opencv stands alone but it
also
is wrapped very lightly in ross as
all things are so
if you're going to call in and build the
camera modules or things that depend on
calibration or anything
like that the cmake file will will
call in and build opencv and so you'll
just have it
seamless to you you know if you need it
it will be built and pulled in
and so so it's the people are using the
perception parts will end up
with opencv
so let's jump back to uh stanley and uh
what were some of the the challenges for
the vision system
and being the leader of that project so
i mean the main challenge is this is a
car it's going to run in the desert for
n hours seven hours or something like
that and so when you look at the failure
rates what can they be it's really like
zero so we tried
i mean we started meeting at night and
hashing out a bunch of approaches
and we tried a lot of stuff and
so you know i i did these convolutions
with
line finders and then you know roads
dirt roads tend to have lines and and
pave roads with lines
so you'd find these converging lines and
and and we'd have some idea of the
horizon and so we could identify
like these things seem to go to horizon
points and so we could identify roads
and and
we did many things like actually just
doing machine learning on what is a road
and those things kind of worked in you
know
sometimes on in high 90s and actually
one of the machine learning techniques
uh beat as far as we could tell
what we actually use but the thing is
you know the sebastian's very pragmatic
about what to do and so the vision
system on stanley there was a laser
system
and the lasers had enough range and
and enough range pointing out sort of
straight you know as you
glance the road at some point the lasers
do you no good they're too flat to the
road so every little bump
seems like you're changing you know many
meters
so the vision system was really designed
to extend
how fast the car could go and still see
so the lasers could
go to about 25 miles an hour but to do
this race
you realize that you and the flat
sections you have to go faster
so so it was really
the vision system really came and
clicked on when it could see enough road
and say
look um you can really go faster than
you can stop with the lasers because i
see it clear for a long ways
and so it allowed the stanley to speed
up and go fast when it could go fast in
the
tight turning mountain roads it wasn't
really the vision system was on but it
wasn't
it wasn't able to say go you know so
so when you look at the race the vision
system was on
a fairly small percent of the time which
is true in any race
whenever you have a race and you have
the fast parts you don't spend much time
in the fast parts because you go fast
right you spend most of your time in the
slow parts so
you know we backtracked that out it was
absolutely critical for
stanley to win i think it would have
come in third place without
without doing those speed ups and so
that was the role of the vision system
is when
to see further than the lasers and you
know these were both merged into a
bird's eye view the laser map and then
and then the the vision system what we
ended up doing after
thrashing around and and before these
two groups were separated and then
they're put on the car
now we actually had the laser data
available to us
and so we decided like the lasers would
say well this is flat
like we you know the laser thinks the
car can drive on this and so we just
took that chunk and learned the color
model and extended that out
and and so it's sort of like the laser
would say i like this
and the vision system's job is to find
more of this
and that's how it worked and in the end
we had just a very simple color model
it our system got simpler and simpler
and we had this machine learning
model and that was kind of a regression
system that actually
did better but i think well partly
i don't know it was the intersection of
the code freeze but i also think
sebastian
thought you know like this color system
was kind of it came down to a decision
tree and what does it mean that a color
is
one you know it's above 127 or below it
it's hard to tell whereas we did a
gaussian color model
and you'd you could you could make some
reasoning about that that i'm
this far away from the color
distribution so i'm going to trust it
less and less and
and so yeah he wanted to go with the
simpler system and that
you know because he figures we can
always slow the car down
you know and and but we want to be
really sure when we can go fast that
we're not going to run into anything and
so that's how it worked but
and you know so essentially the vision
system
didn't fail you know seven hours running
full time
even though the laser system did fail i
think three times in the race
because of some timing problem where
things got out of order
it would think there was a wall in front
of it
what was sebastian like to work with i
it was great i mean it was a lot of fun
it was it was actually
you know there i guess there was this
competition between
red and sebastian and and
and and uh i like sebastian style more
so
it was kind of two styles and and i
thought like
so red whitaker was was um
he's more command and control he's like
he's an ex-marine he tells you what to
do you do it and
that works often if if the person is
smart enough and whatever it doesn't
work if the person's not smart but
but in this case both leaders were smart
enough right
and so that works but i but sebastian
was more like you know just
keep up with me he's not uh directive in
that sense it's just like you know well
you know you're not keeping the pace
like
i'm going fast you keep up and and so i
i wanted
part of my joining the team was like i
want to just establish that that
approach
also works right and so in this case
it did you know that this uh more open
style of leadership
uh yeah not that there weren't their
occasions of falling out and screaming
but but uh
you know i mean
that team when it was coming together
you just have this feeling that we're
going to win i mean it's just like
it it clicked in many ways so uh you
know mike monroe
was he was sort of you know the guy
who's the type a personality holding
everything together
but i mean all the personalities there's
hendrick and whatever
you had a good mess and you just felt
like this team is just hitting on all
cylinders so
any funny stories i don't there's
many but i don't know the i guess the
funniest one is we had
i'm actually friends with this reporter
john markoff
and so i saying i'll cover this cover
this uh whatever so he came in and
and he took a ride in the robot and that
was the one time it crashed
and i think what what it was was an
overreaction i think
i think it was i might get the story
wrong but i think mike montemerlo
wanted to be extra careful and so he
i think he put a limit on the speed to
be below 15 miles an hour
but i think it ended up introducing a
bug because i guess they went down this
berm
and i'm not sure exactly what the
sequence of things happened but i think
i think it slowed down to go down and
then saw that
it's not 15 miles an hour and it's sped
up to go 15 miles an hour and then
crashed down into some bushes or
whatever but
no one got hurt the car didn't get hurt
but uh
i don't know there were other funny
things like
i think the cmu team had professional
drivers
because they might have asked their
lawyers like can a student be in a robot
car
but sebastian's the kind of guy he's not
going to ask that question because you
know what the answer is going to be a
lawyer in the school will say
no like what's the what's what's the
lawyer's benefit in saying yes
only a disaster could follow so but it
was never asked and i think it was
important that the students were in the
car
because you know a professional driver
if some glitch happened yeah
you know just ignore it after a while
well the car straightened out but
the guy who's coding that will say what
you know like why did that glitch and go
look into it but
so you know as far as debugging i think
that was important but
i don't know the whole project was a lot
of fun
and how did you get involved with willow
garage and decide to make that career
so in a weird way i had to i sort of was
an incentive i don't know at what level
to
forming willow garage but not in the way
so i had a
at intel i had a darpa subcontract with
the cognitive architectures and i
thought
that's a crazy program like i don't even
know what a cognitive architecture
is i don't think anyone really does so
but
and then you know i saw that the program
manager was going so this was an intel
subcontract but i saw the program
manager was going to leave
and it was retiring whatever they only
keep them for a certain time
in six months and i go there's no way
this program is going to survive six
months the new guy is going to come in
and
no new guy says oh i'll take the old
guy's stuff and run with it
and so i go this is only going to last
six months and so
what i did is i had money to hire some
students to work on cognitive
architecture instead i bought a robot
arm
which caused some consternation between
the lawyers at usc who we were
subcontracting and intel
but i smoothed it over somehow and so he
had this robot arm and one of the things
on the robot arm is it has to be
transferred off as government property
to some other
darpa contract or government entity and
i go well stanford's full of darpa
contracts so
i went to andrewing and i said well
here's an arm and stanford had all this
early history it was shaky and whatever
and there ought to be some learning
based
perception based robotics and my idea
was let's put some cameras and
do it with the arm and start working a
project but but
andrew took i guess that as a kick and
expanded into the stair
program stanford ai robot that's still
going on
right and and i guess ken salisbury i
don't really know
the causality chain here i went off
soon after that i quit and went to a
startup i quit until
so the contract ended i transferred the
equipment that arm is still there
and in andrewing's lab and it's a
harmonic arm from canada
and and so he started this whole project
putting wheels under it and making it
basically a learning robotics project
and got a bunch of department
involved and i guess that triggered ken
salisbury and some of his students
eric and keenan who were originally here
to start building a robot torso and i
while i was at stanford i was hearing
about no there's a robot taurus like oh
wouldn't that be cool to see
you know like instead of this stair was
like sort of bolted together
little pieces and and then i went off to
the startup
and and i guess during that time i was
away
doing the startup which was video surf
it's still there
at video indexing um
i guess they built up pr1
and then scott their founder came in and
and said well that that'll be one of the
robotics projects and brought those two
in that's formed the nucleus here now
later on after a year of doing the
startup i decided it's not the right
place for me
because i wanted to keep working on
opencv and other things and so i was
looking around and then andrewing
called me up and i was starting to
interview i interviewed with facebook
and all these places and
and i was going to get ready to take a
job and andrewing
said you know i really think you should
see this new place
and so i i said like i can't like
they're getting on to me is like a 15
person startup you know it's pretty
obvious i was spending a lot of late
lunches and whatever
i was interviewing and he said no i
think you should go
so i i looked at the map i said well i
could get there in 15 minutes have a
sandwich shake some hands and get back
and within a half an hour they'll never
know
and so i came here and i think i stayed
for like three hours and and then
i i just uh you know i got to work here
so
since then you know they've supported
opencv and now we're
we're working on the
so we've had a lot of perception
packages now we're putting it into a
framework that i think will be pretty
efficient
so uh you know we're looking for a way
where it's very fast
to code object recognition and
we'll also keep the code in shape and
maintainable
so we want fast development speed but we
also want
uh that the code itself is fast
and so we have kind of a python
framework that wraps
modules which wrap opencv
and basically you know when you're done
you build up all these modules and you
can just
put them together mix and match so i'm
doing a color based object recognizer
but in once it's done we have this
texture based object recognizer and
it'll be
simple just to say put them together so
you know that that's what we're doing
here is sort of the perception side of
robotics
but that's what i'm doing but what are
some of the real challenges for object
recognition
just that it's an unsolved problem
uh i mean yeah i mean it's actually kind
of uh
an insane i've always worried about
being in computer vision like
you know i'm in a field that doesn't
really work and
and i get these encourage these students
and they go but they all
stay employed so i mean it really is
making progress but
it's it's daunting in many ways because
humans have this sense of vision they
think it's easy but
but when you look at a camera it's just
the flux of numbers and it really is
this flux of numbers because it's
the world's field of noise and you know
how do you how can i
segment me from the background and and
you know how do you be discriminated
enough so if you have something that's a
very tight match well you're not going
to match anything because you never ever
see the same view
twice right you just don't do it unless
you're some
very precise industrial machine so you
need flexibility in templates well that
means they're sloppier
and that means they get confused and so
you really have to put on a lot of
modalities and so
some computer vision techniques that do
well
are extremely slow and then it's like
well when will they be fast or like
never
and so we're looking more at you know
things that are real time rather than
just trying to get out academic papers
that
do good jobs and recognition so i also
started this contest that's called
solutions and perception
and we ran the first one in icra a month
or two ago
and this is to establish what problems
are
solved in vision and start with easy
objects and work our way up
unfortunately no one no one got above
threshold
but uh i think uh i i think so we
started with textured objects
that are things you might find in a
grocery store like a box of tea
or cereal box or detergent bottle
not not things that were that were
transparent
or or shiny
and so basically things with a lot of
texture on
it texture could be anything
manufactured pictures or actual text
and and so we i think that is solvable
that and
uh it was just that the contest didn't
have enough time
it was sort of a rough we were doing it
with nist and the white house office of
technology
was liked it uh thomas khalil was
spurring it on and so it was a bit of a
rush we did it in four months
so this time we'll have a year and and
and so
i hope will solve some categories of
vision we're actually going to loosen up
the problem a little bit to be textured
and textureless rigid non-shiny
non-transparent objects so
lots of the stuff around here the tape
dispenser over there
would be part of this we also were a
little bit limited by the connect
and so the connect is a great device
it's really a price breakthrough
revolutionary in that sense
but it's got a really lousy camera
because it was designed
not for robots but for you know
consumers in the living room and they
don't really
care you know that the camera is full of
noise and
and whatever it's a really cheap camera
so
now you know we have calibration to put
a better camera
with the connect ultimately we want
connect
a device for robots that will
you know have higher resolution so
because it's so low resolution
that hurt the texture recognition we
decide to standardize around this
connect device
and it's a struggle because you have to
turn it on it's
it's one megapixel mode which is sort of
hidden in it there is such a mode
and it's really really noisy so this
texture they're trying to find gradients
and yet
half the image is just fictitious
gradients of noise so
so that was a big challenge and and so
this next year we're going to lift the
say bring whatever you want just
whatever sensor you want whatever
computer you want
you never just get the job done so
that's one of the learnings for next
year but uh do you focus mostly on the
static images or i mean robots have the
ability to move through the world and
use optics so in this case it was
multiple
views it it was also you know we made a
lot of mistakes we
were going to be the master serving the
data and checking the code so that other
people
my fear was things like the pascal
challenge itself reported
it's not that i think people lie but i
think
there is the possibility that you you
run it once you go i forgot to set the
threshold you know and then you set it
well now you're
now you can't really run things twice
and get an honest score
so i was afraid of that but now i'm
going to say well i've lost my fear
it's too much work and also it's too
constraining on the people
so we're just going to put out here's
the kind of objects that we're going to
do
and and at the conference where this is
going to run we're just going to bring
in a pile of these objects
of new objects that of the sorts that
people saw and they're going to have to
learn them there and recognize them
there
have two days to train and test and then
the third day will be a contest
and they'll be able to move around so we
think
we didn't right now you know robots are
still expensive and
and whatever and and so we're just going
to have however you want to transport
your camera around it can be
on a robot or you can just move it by
hand
but the idea is you will have the
ability so
unlike this last time we were asking per
view we're going to
set up stations where you'll have a
certain fixed amount of time to run
around the room and recognize
objects and at these stations you can do
whatever you want
so we just want the report at the end
what objects are there so
one of the problems we had while making
these fixtured scenes was things would
occlude well now how do you code that
occlusion and
sometimes it's half an inclusion is it
fair to ask them to recognize the back
objects i mean
these were all such judgment calls that
we didn't like
so now we're just going to say a robot
could move around but the human might
actually be moving the camera but it'll
just be
do whatever it takes to recognize this
and the idea for this contest is once we
establish like plates and cups let's say
are solved then then this
then people can rely on you know the
planners and
can just say oh that that part is solved
and we can go on and think of
more complex things i got that from like
the darpa grand challenge which i think
you know people thought of doing that
because things like navigation i mean
gps was mature
and slam was getting mature and so that
allowed people to think well we could
actually run for hours in the desert
before then
i mean it would be ridiculous i mean the
early robots could barely get across a
room
and now i can hear robots wander around
all the time they don't bump into things
or people
and so that allows you to think well you
know i can think of them now operating
around people so i want with objects to
say like these classes of objects are
solved
so now we can move on you can say well
can i clear a table yeah
if i can get cups and dishes and
whatever that would be something
someone would think of having a grand
challenge or maybe
building you know robot that can clean a
cafeteria or something like that but
but i think it will be unleashed when
the perception's unleashed
you know that's that's one of the gating
functions
you know there are many other gates on
using the arm and planning and stacking
you know but but
we're going to see what we can do how do
you
interface with other parts of the team
we argue no and
so i mean more or less they're the
manipulation
people and and planners are
consumers of the perception so
we've had a long running demo that sort
of gets some bowls and whatever from the
early stuff i did with marius muja
and now we're trying to replace it with
more sophisticated stuff
you know that and and so
they're sort of our customers or
whatever
you know our user base is the people are
going to use it for manipulation
you know we have a small team so i run
this opencv
team that willow supports and those are
five uh
russian developers but they're mostly
responsible for opencv
not the object recognition in particular
i mean there is overlap they
they did help on the plugging in and
many other things
but mostly they're on the library
maintenance and debugging
and advancing certain techniques and
so what we have here in object
recognition the real core team
is me and two others so uh vincent robot
and ethan rubley
are working on this infrastructure and
then there's a programmer here
troy who's who's helping us with
trying to get this seamless
infrastructure of um
these modules and things that are easy
to string together
i was programming on it just before the
interview
and you mentioned i mean you mentioned
object recognition you mentioned the
plugging in
um what are some other you know
applications of pr2 that you've
worked on well i mean there was these
milestones right so that was uh there
were a couple
phases of plugging in for the first mile
a milestone two that was the big one
we didn't actually have the cam well the
camera in the forearm just came but way
too late for the effort
so we stuck with the high resolution
camera but later on we did another
effort where we moved it down now it
uses the foreign camera
and doorknob recognition but the
perception's been used in
absolutely every video you know you can
look on youtube
there's the beer getting well it was
using the stuff to find the beer there
is the pool well it was using
you know finding balls and whatever you
know using opencv functions and whatever
and
and and on and on we've had picking up
demos
and and clearing demos and
you know they're they've they've used
this on and on
the problem has been that this object
recognition has been kind of fragmented
like
for i had a i teach a class at stanford
which is like perception
for robotics and and the first year we
called perception for manipulation and
you know i wrote a module that was
pretty good at recognizing
cans and things and it just wasn't
maintained
so that's why we're building this
infrastructure as a
kind of polished piece of software
engineering where the stuff once it's
locked in
it stays usable and stays live
so you know that's what we're aiming at
now and and that
hopefully will become more consumable by
other people as
i don't know if a perception person i
mean a manipulation person will actually
use this module
the guy sophisticated people like mate
here might but
they would have the ability to actually
construct their own object recognition
by putting oh i know how to filter the
background that's just
this what we're building you can
actually do graphically
or in python but when you're done it's
all c plus plus
because these are just very light python
wrappers over c plus plus code
but they can string this stuff together
and then they have a standalone c plus
plus module
this you know that's for speed that you
can run
as a python script but you can run it
standalone and then we hope
to actually make it even easier for ross
developers
at the end we can say well once you're
done designing this
module it can either stand alone or it
could
you could just say i want this as a ross
node and it will make a message type for
it automatically
or we're off service so you know that's
the goal
have you had feedback from people who've
been using it
with frost well i mean this framework is
new
so the feedback's all the people
is internal users but yeah from
well plenty of feedback on opencv and
and
you know feedback from the other systems
as i think they want a more modular
system that's more of a package
you know like we're researchers so we're
moving ahead and
and we're not the software engineers so
there hasn't we haven't put enough
effort into
released code that that's this released
object module
and so i i think that would probably be
the complaint
the answer to that complaint is this
system because every module is very
simple
and will have test code and and be
essentially released and so
on a new system you'll just string
together modules and they'll be the core
one you're working on well you write
test code for that and and
the documentation is forced because we
have these
getting into detail virtual functions
you have to fill out and one of them is
a documentation function
so the documentation is just another
method in the class as a result the
python can say document yourself you can
put all this stuff together and you'll
have this whole flow of what this thing
actually does
and and so that's another thing like and
it spits this out
in swig you know so
it's a language that can be turned
immediately
to uh to an html or a pdf document
so i know you have to get to graduation
yes
um so we should probably wrap up but do
you have anything
else you want to ask other than our
fabulous wrap-up question okay so
um so we just asked everyone this to
wrap up but um
do you have any advice for young people
who would be interested in robotics or
computer vision
do your math homework i mean uh you know
it's not always so mathematical as
people think
uh that well i mean some of the things
are
extremely mathematical but i'm not sure
they have as much payoff
as just simple intuitions and sort of
persistence
so i think you know there's a big role
for persistence and creativity
there's you know getting into robotics
and computer vision
there's always this thing like you sort
of want to know
what you're doing and by that i mean is
how do you know if something's working
or not
and that's sort of a talent or skill
that you have to build up
a lot of times you have these complex
systems that have lots of trade-offs and
you can't test
everything so you have to develop a lot
of intuitions i'm building a colored
base detector well there's a million
things i can do i've
i've using color ratios because that's
the only thing that's more or less
stable and but it's not so stable so how
am i going to
counter that out and the thing you know
i run an early version it kind of works
but it has a bug in it how do you know
that it has a bug in it because a lot of
the time when you're doing pattern
recognition
things will work but not as well as it
should and so how do you tell
you know that you have to think of ways
of testing that
what you're seeing is really what you
thought you
wanted to do and and so
i don't know um creative play i'm all
for that uh
you know i i think you should just
really
go for what you're passionate about find
if you find perception
interesting then just start doing it
hacking it
you know the there's a lot of code out
there
i'm i'm now just relearning hardware
design i used to work in it
i've long since forgotten it but now
there's all this open source arduino and
you know you can just get in there and
start designing chips and and now
there's like eagle you can lay out
circuit boards it's getting pretty easy
right
so there's all these tools i mean that's
the good thing is computer vision is
just start learning
the tools and it gets you a long ways
there
and you know i guess one of the easier
ways is to pick up matlab and that's
that you know easy to develop in and
then
you know start adding maybe python and
then c and c plus plus
um how do you think the availability of
these kind of more open
both um hardware systems and also
software systems is affecting the
development of
the field i mean it's
it's training up this new generation of
you know people maybe we thought we lost
these old like
designers and hardware because i mean
people are just doing wild stuff
on you know you can find almost anything
you think of like
i'm looking at spectrum analyzers well
people build this little chip into a
spectrum analyzer not in the way i
really want to do it but
but i mean it's getting easier to start
combining
these arduino chips and all this stuff
uh
it's getting easier to start doing
hardware and software again i mean
it's sort of like building up these
packages and so now you can really
start doing hobby robotics you can make
an arm
pretty cheap with these little servos
and controllers
and you know maybe that's what it takes
i i've heard of
you know very clever things that people
are doing so you want
force control and touch and you know how
are you gonna do that well people have
taken off the touch screens on like the
iphones or whatever and use those like
sliding together you can detect
movement and force and and contact
and you know so i mean a lot of creative
stuff happening out there so
great well thank you very much thank you
very much