okay so we're rolling we're yeah
okay my name is ralph wallace i'm at
carnegie mellon university in pittsburgh
uh i'm 69 years old means i was born in
1941.
i was born in hutchinson kansas but grew
up
most of my life in wichita kansas
so um let's see
my earliest
recollection of being interested in
robotics
was probably around 1954
and i had read a pair of scientific
american articles by w
great walter i think it was the 1951 and
1953
uh issues of scientific american
anyway as is well known uh walter uh
built a couple of small mobile robots
which could move around the house and
could sense uh
objects and avoid obstacles and and so
forth so
i thought that was like the neatest
thing ever like that was
that was it so right away also
i read somewhere um
about sensing things using capacitance
so uh so let's see i would have been
about 13 i guess at that point
so my friend phil roberts and i
decided to build a robot that could move
forward
until it could sense a wall and then it
would stop based on
measuring the capacitance to the wall
so i had gotten i think as a christmas
present a big old lead acid battery
so that was the first ingredient and
then we went to the local surplus
place in wichita called the yard it's
very famous
it's still there and bought some motors
and so forth
to make this thing but
it wouldn't really go because the
battery was too heavy and uh
so eventually we we gave it up
but we we began to understand the
problem of uh
uh you know carrying around the battery
with wheels
uh he had that was a trade-off between
having the battery too big
and not being able to move the battery
too small not being able to move
so uh that was a pretty cool lesson
so of course this was back in vacuum
tube days actually
as well so that was our
my first sort of introduction to
robotics
later in about i think 1957
my friend phil and i uh spent a lot of
effort
and we built a small robot that was in
the shape of a
cylinder basically
it was about 10 inches in diameter and
maybe
10 inches high it had lead acid
batteries
and was mainly made out of old erector
set parts
as i recall but it
used telephone relays for a brain
actually
so we had lots of surplus telephone
relays on this
and it also had a pair of feelers that
could extend from
either side and it could roll and
feelers had wheels and it could roll
along the wall
and as it rolled along the wall it kept
an equidistant
from the wall when it came to an opening
like a doorway
the feeder would spring forward and hit
an electrical contact which would
trigger a relay to cause the
the robot to turn and so forth so it
could occasionally find its way out of a
room by following the wall
around the room which was pretty cool
and i still actually have parts of that
robot although i don't
have all the robot back in those days
once we
watched something work we took it apart
and did something else with it
so we don't do that anymore we always
keep what we have
if it works that's it
so uh well that was
basically high school days i guess
i went to uh university of
wichita which is now i think called
wichita state university
for two years and majored in
mathematics there and uh
one of the uh big things was the uh
my first introduction to the digital
computer actually i'd started
a year earlier in 1959 when i was in
still in high school
programming the ibm 610 computer which
had a drum memory and paper tape
and so on vacuum tubes and by the time i
was done with
doing all kinds of calculations with
that computer a couple years later
it was i logged more hours in the
logbook than
any of the faculty and other students at
the university so
i was quite enamored by computing
i also uh at that time
the bendix salesman came around to which
is a
university with the bendix g15 computer
the size of a refrigerator and used this
cool language called intercom 500 which
i
tried to learn and so i
programmed that i was much much faster
than the 610 and
it had von neumann architecture as
opposed to the 610 which
actually did not have ability to store
programs
they were all on the paper tape or on
the wiring panel
so i programmed the g15 and
for many months until the
vending salesman came and took it away
because it was just a demo
so people took pity on me and
then i went the local aircraft company
beach aircraft had three bendix g15s
so they let me use their computers from
midnight to 6am
so so i would spend all the nights over
at beach aircraft
doing calculations almost all of the
things i actually computed had to do
with
uh rocketry which was my hobby really
i built a lot of rockets and and lots of
stuff like that
i did a lot of calculations on both the
external
rockets flight and the internal flow of
gas dynamics and so forth
i read a lot of books on fluid dynamics
so well anyway then i went to
after that to kansas state university
where i
switched to majoring in physics
and uh got a master's degree
bs and master's degree uh didn't do much
work in robotics i guess at that point
moved to california
worked for north american aviation on
space
flight vehicle simulation and
the reason i got the job there was
because of my experience with
doing calculations related to amateur
robotics
but this was uh with the minuteman 3
system so
i formed a team of eight people and we
wrote this computer simulation of the
minuteman
flight from takeoff to impact to the
warheads
and it was sort of a robot if you will
uh it did a lot of measurements it had
initial measuring unit
and so on uh so a lot of inputs and
outputs and computation and so forth
very much like a robot
so um that was uh
up till um 19
early 1970s um
and uh but during that time i still was
very interested in
robotics so i started building a
computer at that time based on
old parts from from wrecked ibm 1620
computers
as well as the very newly available
texas estimate ttl logic
as it turns out the minuteman missile
was the first use of
ttl logic from texas instruments
and they had a local office there and so
i would visit the texas instruments
people and they would give me
circuits and so forth at that time there
were only about
four different circuits available from
texas instruments hard to believe now
with the millions of
chips that are available but using those
basic circuits i built a computer that
could do
like a couple hundred thousand additions
per second and
also i was able to get a surplus of
magnetic drum memory which was part of
the sage
early warning system and
a freedom flex rider for input output
and then the idea was to control the
robot remotely so
so i hung out as it turns out my my
friend phil
roberts the same one that we built the
robot with in the 1950s
uh was a grad student at cal tech so
i was in anaheim california so in orange
county
so i hung out a lot at caltech
the two of us built uh another robot
which sort of got half built
the idea was to interface it to all the
stuff that i had
the computer and so on so that never
really
happened never really quite got together
but the id a lot of the ideas were there
so let's see that brings us up to about
maybe 1970 i think
so actually this that wasn't quite right
it was like i was in
north american aviation for from 19 1965
to 1970.
so how am i doing so great questions
okay
so uh i went back to school to get my
phd at university of colorado again in
physics
where i succeeded in putting together
a superconducting magnet system with
lasers
laser optics at the time it was uh
became the highest field magnet in the
in the world
with optical access which was
kind of neat and i got a lot of results
in solid state physics light scattering
from semiconductors
but i still had a hankering to do
robotics
so so i started putting together
a small robot that had stepper motors
which for some reason i call the newt
because
my girlfriend thought that would be a
good name
comes from shakespeare i have nude toe
frog or something like that
so that became alpha nude and uh it
was able to uh roam around and
and see lights and and avoid lights and
go to lights and so basically in 1970s i
was reproducing what gray walter did
in the 50s or late 40s
so uh but uh from that
we there was another friend i had dennis
toms
who was another physics grad student
we convinced the university to let us
have a lab room
and with a group of about four of us we
started doing all kinds of robotics
things
building cellular automata and things
like that
out of hardware and
so we wound up building
a robot which we called betanit the
second version
which was uh
had an onboard computer which is a
significant advancement and
so in about that time i guess 19
so i graduated with my phd in 75
in 76 i went on a
exchange scientist fellowship to
university of paris
for about 10 months also got married
so and there
i decided to write an article about this
robot we had created
that had the onboard computer which was
an 8080
until 88 i think it was and i wrote that
for bite magazine
and at that point i had already
submitted
several articles to physics journals
like
physical review and so forth i was
amused by the fact that it cost
actually costs page charges to publish
in some of these
journals
i submitted this article on newt
the robot to bite magazine and
it became the cover story
and they actually gave me like over a
thousand dollars for the article
and uh as it turned out it
was the most popular article
byte had published up to that point they
had a little
thing that they rated user responses
and apparently it was several
standard deviations from all the
previous articles
so that was cool uh and do you know when
some of the recipes or responses were
did you ever read them or yeah yeah oh
yeah yeah
what were people saying well they
thought it was really cool and uh
i don't know they it's a long time ago
i don't remember exactly what what the
responses were but it
but i started to think well you know gee
there must be a lot of interest in
robotics you know
compared to physics for example
so maybe there's something to this why
was that robotics i mean you've
obviously had a big interest in it from
very early on
so why did you decide to go into physics
rather than
something there was no there was no
robotics
right it just wasn't there wasn't even a
radio shack around
to go buy parts uh i used to travel all
over the country to
surplus places all over to get parts
and uh but there's just almost nothing
there's no digikey to order parts from
or new egg or
any of these kind of things of course uh
so uh and i i did like physics too
for sure and i was reasonably good at
experimental physics so um
yeah so uh
the bite article had a big response
and sometime after that i guess
the wgbh nova broadcast
came out to colorado and they spent
three days
filming uh newt alphanute and betanate
and uh they had a program which i think
came out about 1978 called the mind
machines which
which featured my robots which i thought
that was pretty cool
and wall street journal had a small
article on the front page
i got interest from just all over the
place
like companies from china that wanted to
make them
things like that so
at that point i i moved to new york
yeah i got a job with ibm research
at yorktown heights which was good
and i worked in in physics
and applied physics doing
magnetic field effects and thin wires
doing microwave acoustics and what have
you
so that was great and a lot of fun and a
terrific place to work
fantastic place in fact i wound up
spending 16 years at
ibm research i don't regret any of them
so but because a change of management
and a few other things
a change of projects i wasn't able to
continue
what i was really had started which is
the microwave acoustics
i built the first acoustic microscope in
use in industry
there working with cal quay that
at stanford who was the real pioneer
uh in that area um
i wanted to continue that but as it
turns out
i sort of got a cease and desist order
because ibm isn't a university really
so at that time i was kind of looking
around for something to do
and there was a newly started fairly
newly started robotics effort
at ibm so i decided to uh
after all this work in education
i decided to leave the field of physics
and do robotics full-time
and that was a really really tough
decision because i invested a lot of
energy
into physics and i loved physics
but
i also loved robotics so so all along i
guess there's been this
duality that has followed me
what year did you make that switch uh it
would have been about 1982 i think
who else was working in the robotics
group at ibm at that time
so uh larry lieberman was one named
russ taylor who you're doubtlessly going
to be interviewing if you haven't
already
uh
i do remember but i don't because i'm
getting
see now what were the projects or the
the applications they were looking at
right
janine meyer has another name
so i'm trying to think of mike wesley
david grossman
david grossman was the senior manager
so i wound up uh reporting to russ
taylor
eventually after he became manager a
manager
there so
at that point ibm had already developed
a series of hydraulic
robots cartesian robots which became
products that were productized in the
ibm boca raton facility
and they used an interesting uh sort of
hydraulic stepper motor design that
a guy named pat panacity had developed
pad had about 50 patents in ibm which
was pretty
pretty nice so uh
russ taylor was responsible for
developing the
programming language or at least mostly
responsible called
a manufacturing language or aml
uh that was based on earlier work that
he had done at stanford using the
shineman arms and so on and so forth
so that's another story i guess
uh so uh
they had also recently introduced a
scara robot with a shoulder an
elbow and a four-axis scare robot
mechanism from japan but the software
from ibm
so uh so it was pretty exciting time
because these robots were starting to be
used in many different manufacturing
applications
within the company so uh
my personal effort was in developing
mostly and developing
fine motion devices the robots any kind
of robot you
had they were not very accurate
maybe um
half a millimeter or maybe a little bit
better maybe 200 micron
accuracy so so a lot of ibm products
required a lot more precision than that
so the question is how would you do that
and so what people mostly did is they
used
very high precision stages that were
very expensive with
granite bases and laser interferometers
and all that to
to get down in the micron range which
was uh
was out of reach of robots basically so
uh i developed uh at the suggestion of
russ
i developed a fine motion device which
we called the fine motor fine positioner
fine positioner ibm fine positioner
which
uh you could mount on the last link of
the robot arm
and you could then enhance the precision
of the robot because the
robot could move roughly to the position
you wanted to go
and then the fine motion device could
take over and move
from say a millimeter on down to one
micron or even
better than one micron so
we were able to do a lot of things like
precision assembly
and precision testing where we could put
an electrical probe on a
circuit that was very very tiny which
would be missed entirely by the
robot itself so we called this
paradigm course fine positioning
and i think we were among the very first
to
to really do this kind of thing where
you had redundant degrees of freedom
on the robot the robot the standard
robot arm would form the course
robot and then the fine motion device on
the end of the robot would
take over and do the rest
of this with rust or yeah
all right
so also mark levin that's another name
who did some vision
work with us as well
so we built a number of these based on
flexures metal springs
i have some with me here actually
it was a big hit it got on the cover of
the ibm annual report
and like four million copies and i don't
know how many languages
things like that which was cool
that early fine positioner was able to
move in two degrees of freedom
x and y let's call it the robot still
had to do the z
motion later i developed
another kind of fine position to use an
air bearing and could
move in x y and theta that is
transversely and also a certain amount
of rotation
that was based on a flux steering
motor i got a an electromagnetic theory
so
that was easy so
and that worked quite well and became
sort of an internal product to ibm
and we made a bunch of them i don't know
maybe 12 or 15
and started to use them in a number of
prototypical applications in
japan and france and all over the u.s
and so on so so for example one could
put a disc platter onto a spindle
by having the robot bring that disc
platter down to the spindle and then
having
having the fine motion device which was
this air bearing fine positioner
sense the spindle inductively and and be
able to
guide itself on so the robot could be
off by millimeters but still the spindle
would go
straight on to micron tolerances and do
it fast
so so uh
and all kinds of other applications
stacking uh thin
laminates like circuit boards for really
high
super high density circuit boards things
like that
so we built a number of systems around
the
fine positioner using using mostly the
scara robots
and uh the fine fine motion devices or
fine positioners
uh let's see about that time i also
started wondering you know
it would be really nice to have six
degrees of freedom instead of just three
degrees of freedom
x y and theta why not you know up and
down and
rotate around your whole pitch and yaw
i couldn't quite figure out how to do
the linkages i guess i could have used
like a steward platform kind of thing
but uh i thought about it and thought
about it for a long time and then one
evening
it occurred to me that why not just
levitate the fine motion device in
magnetic fields
then you could move it any direction you
want there would be no
suspension or no linkages no air bearing
no anything
so i sat down i did the calculations and
thought of a few
geometric configurations and at that
point
there was recently available
magnet materials neodymium iron boron
which had just come out
and then also there were
digital signal processors available that
could do really fast computation that i
thought we
needed in order to keep the levitated
part from
falling basically of course
there's no stable equilibrium points in
the magnetic field
due to a theorem by ironshot
1845.
so so it has to be a closed-loop system
with sensing
and i'd already used position sensing
photodiodes
for doing the sensing on the
three degree of freedom fine positioner
so anyway
we started putting together a system
that could levitate
we call it well it was
something sitting on the desktop
actually
and hooked up to some surplus
power amplifiers and we had like a 9 000
signal processor from with chips from
texas instruments
and ibm pc xt or something like that
i had hired a
fellow from a new graduate from berkeley
tim salkudian and
he got his degree in advance control
theory
from berkeley and so
he teamed up with me and also
an intern from duke university named
peter allen
student and
we made the thing work
tim did most of the software and
algorithms for control and
so once it took off
uh and floated above the desktop stably
uh we were pretty amazed and we could
also move it
in x y and z and roll pitch and yaw over
limited
angles so
one kind of funny story that i tell
people
we thought it was cool because we
weren't aware of any
any other device like that anywhere
and it used lowrance forces
rather than the usual maxwell forces
that are used in magnetic bearings
so lorentz force being that force on a
coil and in a fixed magnetic field
so the part that floated which we'd done
float or
f-l-o-t-o-r an analogy with a rotor of a
motor
that contains six coils which interacted
with
magnetic fields produced by
fixed permanent magnets
the sensing of the position and
orientation then was done
with light-emitting diodes attached to
the moving part and then
position sensing photodiodes based on
the lateral effect
on the stator so
we were able to not only stably levitate
it but move it in
in all those degrees of freedom so
we thought well okay so now it's time to
show the big boss
there's boss of the whole i don't know a
couple hundred people
the big department manufacturing
research department
to show him that the thing worked and so
forth
so uh we did we
had got on his calendar of course he's a
busy man
and uh he uh he came to the lab
eventually on the appointed day and time
uh just before he came uh tim
was worried about everything because it
was just a spaghetti massive wire and
everything and
amplifiers and computers and barely
hanging together and uh he said well
you know maybe i should reboot the
computer to make sure
and it rebooted the computer
which was the fatal mistake so
uh uh the big boss i i don't even
remember his name he came in
uh we pushed the button and it was
supposed to
levitate up off the desktop and instead
it just didn't do anything i just sat
there
and he he leaned over kept leaning over
looking at it more and more and then
smoke
came up so
so uh
he jerked his head back
uh i looked at his watch and said that's
very interesting and turned around and
he
he walked out and uh
that was he never did see it ever work
that was our big chance uh it's probably
a career limiting
demo but it turned out it was a software
uh kind of sign of a hardware software
problem
that caused the device to go with
maximum force downward instead of upward
but anyway we got that fixed and things
worked after that
so the next thing with them with that
levitation was we built it uh uh
a a model that could work
on a robot that we could actually attach
to the end of a robot and do all these
amazing things we had done before with
fine positioners so again we could do
micron level things but now we could do
with six degree to freedom
that's really important for things like
putting a peg in a hole where you need
additional degrees of freedom
to let the whole peg slide into the hole
in case the robot isn't perfectly
aligned and so on
so uh as it turned out uh
that worked really well and it were it
was like magic
actually it could do all kinds of things
it would act as a forced torque sensor
we could vary the stiffness of it we
could emulate different mechanisms with
it
that would formally be required to go to
the shop and actually build a specific
mechanism for a specific compliance
we could do all by software
so we tried to think of a name and we
finally came up the name magic wrist
so so that became the ibm magic wrist
and we we proved with a lot of
experiments that we could do
lots of assembly and test kind of
operations even with a robot that wasn't
accurate
and do them faster and
with less force than you would do with
using just say a robot even with a force
torque sensor on itself
so it became pretty
pretty well known i guess
another little story
is that i think it was 1987
i was invited to go to the international
symposium for robotics research in santa
cruz
california and it's a very prestigious
conferences
especially back in those days there
weren't very many people
that were invited and i gave the
chose to give the paper on the magic
wrist but that was before the magic
wrist actually operated
so uh i was you know the deadline came
up you have to go you have to give the
talk
which was mainly on the design but you
know
when showing that it would work and uh
there's one famous roboticist there uh
who uh asked the question that wouldn't
be
easier if you did three degrees of
freedom or four degrees of freedom and
i tried to argue no it's actually
simpler to do all six degrees of freedom
because then you eliminate all mechanics
and so forth
he was not real convinced of that
bernie roth
and then as i was uh
and my recollection here is a bit hazy
but as i
left the podium and uh
there was a by the way at this
conference the
the tradition at the time was to give a
uh
an award or something sort of a little
award to the technology that
was deemed to be most promising
technology
and uh well there were a lot of really
good
talks and it turns out vic cheiman got
the award that year
for another project called robot world
so as i was
leaving the session i guess
i heard one person say
after my talk that maybe it should get
an award for the project least likely to
succeed
the technology least likely to succeed
so that was a funny thing
but anyway
we did get it to work though and it
worked really really well so it became
pretty well known
but we only built four of these and we
never quite
applied them to uh to actual
applications in ibm
why does it not apply well
i don't know if you know anything about
technology transfer but
either i mean we did transfer a number
of things
but if you're a little bit early too
early it's not good
if you're too late it's not good at all
if it's the wrong color it doesn't have
a line card it doesn't have a manual
doesn't have an 800 number that somebody
can call when it breaks in the middle of
the night
then you know it doesn't work
okay so uh uh
we never developed it into a hardened uh
what we would call a turnkey system that
somebody that didn't know how to program
could push a button and make it all work
that would have taken additional
resources and so on
additional time and money and what have
you
as is the case with many things remember
i was at ibm research
but we we worked a lot with the
development labs and manufacturing
labs and different places in ibm so
anyway
uh well anyway so eventually i became a
manager of advanced robotics russ taylor
moved on to a different
position and
so i managed a group there and we did
all kinds of applications and
sensing and we became interested in
manufacturing based on planar motors
which is a thing called the sawyer motor
which was invented in the 1960s by bruce
sawyer
i actually met him once
and we thought they would have a lot of
benefit to manufacturing so we started
working with that sort of thing
what made those motors different than
previous motors
they could move in x and y directions
and not just in a linear direction and
they would direct drive a single moving
part
and uh sawyer had licensed things to a
company called
xynetics or zionitics i don't know how
you said and they
built plotting machines that would do
drawing using these these motors which
could travel in x and y so you could
hook a pen to it or something and you
could draw
that company is long gone that same
exact
motor design became uh
the prime mover in shineman's robot
world
work for which he won that best paper i
guess
yeah as it turns out we were already
working on
our own version of robot world at the
time
as well as a spin-off from bell labs
i was working on that same thing two
spin-offs two different spin-offs from
bellat
one was a company called mega nation
they were all working on that
and we were too so uh
what we decided to do
these were all these planar motors let's
call them they're stepper motors
that they could move in x and y how am i
too detailed here no no okay i
just have no idea okay okay
they were open loop stepper motors which
means the computer would give
pulses to the motor it would move but
the computer had no way of knowing
actually if it actually did move you
know how far it moved and so on
plus the resolution was rather poor and
there was no disturbance rejection like
if you
pushed on it it moved the computer never
didn't know that so i started focusing
on making it a closed-loop system
so sensing its position on the
platinum surface that it was flying on
let's use an air bearing to fly
above that and
people in my group um yehuda is shalom
dennis manzer and i and i had
engineers and technicians we started
working on that
pretty hard and we had lots of ideas and
we got some
interesting results and so on
about that time ibm started going
downhill
i think starting about 1990
and in 92 i guess it was
ibm decided to reduce its workforce in
the research division
by about 400 people so i think there
were about 3 200
people in research in yorktown heights
in
san jose california and zurich
switzerland at the time and maybe tokyo
research lab so they offered a buyout to
everyone
uh no matter who you could
you know it was voluntary but it was a
buyout
and i had no intention at all of
leaving actually i was perfectly happy
and
i had a nice group and uh
i had a postdoc coming in from france
that would spend time
with me and we had customers in our
various divisions that were
counting on us uh developing solutions
for manufacturing
and so forth lived eight miles from work
and drive through the woods and see
nature i mean it was
fantastic ibm research yorktown heights
building is located out in the country
and it's this fabulous it's beautiful
designed by aerosarin and the famous
architect
it was a fabulous place
but it was going through hard times
especially for researchers
and so i told my wife well i don't know
maybe i should
leave
and send my resume out and so forth and
so i did i sent it out to a few places
and i was invited here
to carnegie mellon on thursday
i was met for breakfast i gave uh
interviews with different people gave my
talk and you know ended up like 10 30 at
night
i liked what i'd seen i had been to
carnegie mellon a couple of times before
giving
giving talks i went back home
on friday morning and i think it was
saturday morning a fedex truck pulled
out in front of our house with a written
offer
so
[Music]
a lot of the faculty here i know exactly
that mason um
so uh well i had that
sort of in my pocket but it wasn't until
may that i
gave him the final decision that i was
going to come here
and then i had i've spent until october
actually november
at ibm trying to
tie up all of those ins and place people
at different places some people retired
some people quit some people
i've moved to different places and all
that kind of stuff
also it's a long story but i also
managed to pack up
about 80 percent of the lab and bring it
here to carnegie mellon that's what you
see here
lots of equipment and instruments and so
forth
so ibm was digesting itself of the
robotics
that was as it turned out that that was
the case because i was it at that point
basically russ
taylor was still at ibm but he was 11
miles away in the hawthorne division he
was
focusing on medical robotics which they
also
decided they didn't need either you have
to understand ibm got out of so many
different things
like it does almost nothing now but you
know hold
customers hands probably have to edit
that out but anyway
if
partly because uh partly because uh they
got rid of uh
they did turns out get rid of about 400
scientists
almost all of whom went to universities
they did save a little money and they
did
emphasize technology much less and
that's probably why they made it through
the dot-com bust
and they've been extremely successful
and i'm glad they are successful for
various reasons yeah before that
that all happened what kind of
applications were you
do i mean you were looking at these
manufacturing applications but were they
really for
manufacturing various products that ibm
was doing or were they thinking about
also selling this technology to other
companies for other kinds of
manufacturing
uh the fine positioner we considered
selling it to other companies
to use uh and that didn't
seem to work out for some reason i don't
know why
but all the other technologies was for
in internal use
any hard drive manufacturing hard drive
most
for for the technologies that we had
circuit board manufacturing
ceramic substrate manufacturing hard
drive manufacturing
printer manufacturing
all the things they don't even do now at
all and
you have to understand dial back to
those days ibm developed
most of its manufacturing equipment
in-house it's very it was very
vertically integrated
company now it's a lot more
sears roebuck each
because there are hundreds of companies
that that
produce manufacturing equipment that ibm
can just buy
but that didn't exist back in the day
so so kind of what we were doing became
obsolete in that respect
as well so
yeah have you worked with ibm since then
no i really haven't i still have friends
back at ibm uh
and uh but i haven't had the opportunity
to
actually i've been pretty busy doing
other stuff actually
did you bring any of the people no no
just the
some of the equipment in the lab
well so i got here at carnegie mellon in
1993.
i by the way when i was back at
ibm i continued to work on uh the robot
newt and doing a lot of things with that
uh
got it to the point where i could
navigate around the basement
workshop and uh had an on-board gripper
and used ultrasonic sensing to sense
walls
i could do a routine like in the corner
of a room where it could sense the walls
and precisely locate its position
in orientation it could pick up
wooden blocks and build simple
structures at that point
i was very
enamored with a number of phd theses
that
were written out of mit i guess in
stanford
uh scott falmond's
work and his system called build was
very
influential for me uh
other other work lots of work at
stanford
uh some work at carnegie mellon you know
i came here a couple of times
to give talks and jim crowley was a name
that uh
who built mobile robots han samara back
here
patrick muir who i later hired
so so i i immediately connected with the
mobile robot
work that was starting here at carnegie
mellon but
it was in a very crude state compared to
to what it is now of course
uh so uh well i came here
uh 17 years ago
and uh time has gone by really fast
uh and i don't know what i've done
exactly
but i've loved it here it's tremendous
place
absolutely wonderful i think they like
me so that's good
so we raise our two boys here
we're now off in the world so we're
empty nesters
my wife beth has been extremely
supportive to
all my endeavors all these years so
uh well anyway so getting back to your
question about
what here well so i came here i had the
the beginning ideas of how to make
closed-loop
planar motors and possibly make it into
some kind of manufacturing system
i also had ideas about
how to use the magic wrist technology
to enable a person to interact with a
computer through the sense of touch
haptics and that whole notion was
started with tim sakuti in it as well at
ibm
so we kind of co-invented i guess the
the notion of using magnetic levitation
for for haptics
uh about a year or two before i came to
carnegie mellon tim went to university
of british columbia where he was a
professor
doing many things a lot of it in medical
robotics
we're good friends and keep in touch
so so i had those two basic kind of
notions what to do with magnetic
levitation what to do with this planar
idea
uh can we really make it into a close-up
servo motor because if we could we could
dramatically increase increase the
performance
so so uh as it turned out i
fortuitously got two brand new phd
students
the moment i came here i don't know
whether their arms were twisted to work
with me as a startup
thing or what but i also
uh as i mentioned brought
all this equipment from my band from my
lab at ibm
not all the equipment that i've been but
a lot of it
so at the time
right when i moved the robotics
institute was
taking over this building smith hall and
remodeling it
so i thought wow that's a good
opportunity
so my brother happens to be an architect
uh he designed the lab we're sitting in
uh
uh sent a bunch of plans to me i handed
them to carnegie mellon and carnegie
mellon build it
like i wanted it so
that was nice um finally i was able to
ship the equipment from ibm it took
three 18-wheel tractor trailers
coming from new york to pittsburgh 65
crates 40 000 pounds
of stuff which my two new graduate
students had to
learn how to unpack and plug in and
make work so uh
so they did so uh peter berkelman was
one of the students who decided to work
in a haptics area
and arthur quaid worked in the plein air
motor area
and i got another student a year later
zach butler
who joined us and so on some master
students lots of undergraduates
and so on so uh
peter and i took the magletic magnetic
levitation technology we remodeled it
into a haptic device
and it worked quite well and uh
so that was his thesis
arthur did create the closed-loop planar
motor that we were looking for
and it was terrific and it showed that
it performed much better than anything
out there and we started to develop
this planner manufacturing system called
mini factory
and i was able to get a large amount of
funding several million dollars from the
national science foundation
to work on the mini factory and
i've been working i'm still working on
it after
14 years and i just have a new grant
starting in
january which if
it's six we've already created i don't
know graduated a number of students and
i've written like 48 publications and i
think 40 people have worked on the
system now
but we still believe that it really
could revolutionize manufacturing
especially precision assembly
in the u.s what are some of the ideas
behind the mini factory
oh wow so just summarize those 48 papers
well uh so first of all we had some
goals for the mini factory
uh the first goal was to reduce the time
it takes
people to design
and deploy or program and deploy a an
automated assembly system
for assembling a product of some sort
like imagine a cell phone or
a small center a medical device or
something like that
a disk drive
there are automated assembly systems out
there that build these
but all of those are designed and the
assembly system themselves are
pretty much designed and built by hand
and that takes a long time and a long
time translates into a lot of money
so a typical rule of thumb is that if
you put all the robots in the cost of
all the robots
the robots make it flexible multiplied
by three and you got the total cost
so you have to add all kinds of stuff
you have to add the vision you have to
add
the communication you have to add
sensors you have to have conveyance for
the product and
blah blah blah blah so it takes a long
time
to deploy something so so our first goal
is to reduce
maybe the time it takes to deploy an
automated assembly system from
four months to two weeks so
so the big that's the number one goal
and
the question is what set of hardware and
software technologies
need to be invented to reduce that time
down to a week or two
okay that was the big question but we
didn't
exactly know but we had some ideas of
what needed to be invented software wise
and hardware-wise
the second thing is we wanted to
increase precision over
traditional robots about 90-some percent
of the precision assembly systems in the
world are based on the scare robot the
four-degree freedom robot
which has improved enormously over the
years
but still is not micron level
kind of accuracy so we want to increase
precision anywhere in this
mini factory to micron level or below
anywhere and do it sort of as a natural
consequence of the architecture
which we were able to do
and because of the success of the closed
loop planar motor and many other things
we wanted to also reduce the floor space
that the thing takes up
and so we're about 1 6 the floor space
of the conventional system
so that's kind of a lot of hardware and
truly there is a lot of hardware and
then a lot of software had to be
developed as well so
so um we
decided to make the system extremely
modular
as the key to to rapid deployment okay
so so each of the robots in the system
we call an agent
it has software that makes it
self-describing and self-representing
and the whole system sort of plugs
together like lego blocks
in a way that requires no central
controller and no central database so
each of the agents talks to their
neighboring agents and exchanges
information and
so forth and the product gets assembled
okay so so far we've we've done partial
assembly of small things like
microphones like small optical devices
and that sort of thing
we don't have enough agents built yet
and we don't have quite the right
software
yet to build full
you know products which may require say
25 robots or 30 robots
we have enough infrastructure in the
factory to do that but
we lack the funding and wherewithal to
make that happen at the moment
so that's kind of the status of that
where does most of your funding come
from most of my funding comes from
national science foundation
we're trying to demonstrate scientific
principles rather than do engineering
but
all of the all my projects involve
building hardware
and then what you don't see is all the
software and algorithms that
go on top of it in my lab we don't
really do
fundamental theory i would say and we
don't do
software only kind of projects
but we always have a hardware component
do you collaborate with anyone i
collaborated a lot
with many different people on the mini
factory
collaborated with mark kreider in
electrical and computer engineering and
with
sachit narayanan and computer science
and so forth and other other project
areas i
i collaborate with other people
but in the mini factory we're
pretty posed to move forward now
and i've had for the past year i've had
two students working on
their visiting students from germany and
from china
we're making good progress there so
uh
so in my lab we tend to develop what i
would call fundamental robotic
technology
we actually don't do applications
until just very recently we're starting
to do a little bit of applications but
we typically don't
respond to rfps that say we need
such and such and then we deliver it you
know
almost everything in this lab comes
straight out of my head
like just because i want to do it and
then of course i have to go
make the sale to somebody okay
so we don't really do applications and
we don't really do theory but we do
this what i call fundamental robotic
technology actuation sensing control
real-time control so those are the
things that
we i think excel in as a group
so they i call it agent-based
micro manufacturing this mini factory
project
there's a whole it's not really a
project i call it a program and then we
have many sub-projects within it
in fact in the lab we have three
programs
agent-based micro manufacturing as one
haptics and tele operation
as another and dynamically stable mobile
robots as the third
area unfortunately here's an example
here's another example
okay so each one of these is a
what i call a program and then it has
multiple funding sources
and then within that there are
individual projects to demonstrate this
or demonstrate that or you know graduate
this student or graduate that student
that's that's how the lab kind of works
so in the area of haptics we
as i mentioned peter and i developed
this prototype
we also we collaborated with david
barath
who was now for many years now at pixar
and uh when you see toy story and so
forth at the end that he's
on the credits rolling by he's done very
well at pixar
drives a good car has a nice house
so so uh
but we were able to to interface
everything
to a three-dimensional virtual
environment and where you can feel
forces and torque in your hand and so
forth so the fundamental advantage of
that technology is that it
it's there is no mechanics
all the mechanical issues are thrown out
the window
so conventional haptic devices
are basically back driven robot arms
that the user grabs the end of it
puts a position or position and
orientation into the computer than the
computer
feeds back forces and torque to the hand
and just
yeah you were talking about the magnetic
yeah the advantage of the magnetic
levitation
is that it uh it doesn't have any
mechanics
so kind of like graphics
if you have a dirty screen or low
resolution graphics and so forth you
don't
see the real thing really you need
really lots of pixels and so forth
the same thing is true in haptics and
you have some machinery between you
and the virtual and the
three-dimensional virtual environment
and if that machinery causes
friction or backlash in the joint or
inertia effects or cogging in the motors
you tend to
feel that sometimes rather than
feel what the computer is intending you
to feel
with the magnetic levitation that goes
away and you essentially get a pure
connection
through magnetic fields between what the
computer is doing and what you're doing
so we actually believe it is the most
direct connection you can have between a
running computer program and a person's
hand
other than you know go straight into the
nerves which is
fairly invasive probably not going to be
a big hit
so we're a little bullish on it so in
a few years back i got a major research
instrumentation grant from nsf uh to
further refine
the technology and we did that and we
built 10 of them and we sent them out to
stanford and purdue
and harvard and lots of places
we have some here as well and
it's they work great people like them
and so on so
uh in a little less than three years ago
we've spun off a company which is called
butterfly haptics
llc and my wife
runs the company and we have a network
of
vendors building parts for us and
two local companies doing final assembly
so
my wife and i sort of just run the
company she does all the paperwork and
stuff and i'm sort of a consultant
actually
and that's working out well we're
selling systems in
u.s and canada we're expanding to japan
soon and europe so that's kind of a side
effort and that's the subject of the
article i guess in robotics and
automation magazine that the editor
asked me to write i guess he's starting
a new series on spin-off companies or
something
that do robotics so are they all for
research
sold for research purposes yeah they're
pretty expensive
48 000 each which is competitive
actually with other devices that are out
there other mechanical devices
[Music]
but we have about 25 times the
performance of some of the most popular
ones
which is a big jump in performance so
what you feel
is exquisite
let's say and a lot of people don't
understand that
until they actually try it so
because of that my wife and i we go to a
lot of conferences we've been to eight
so far i think where we set up a booth
we put up our banners and we have our
equipment there and we run demos
for days to hundreds of thousands of
people
and most of people like it and uh
you know we get orders and we sell them
so so it's been a lot of fun
for both of us so that's a side effort
we're still producing or we're still
doing haptics
here but we're no longer developing the
product
we're not developing the hardware really
we're trying to develop a few little
applications
actually so try our hand at doing
something real
so one is we have a us navy contract
to do bomb disposal robotics so working
with a local company here
re2 is called so we've interfaced one of
our
butterfly haptic systems to their robot
their bomb disposal robot
and you can dig in the sand and you can
feel
the sand grains and you can feel if it
hits a solid object you get an immediate
response
in your hand and imagine that that's
several hundred yards away or miles away
for that matter
and it's a kind of a non-visual thing
because when it's buried under the sand
you don't really see it until you hit it
and then you're going to hit it too hard
it's going to blow up and
maybe that's not good so
also we're developing a grasping
capability
which is soon to be demonstrated which
will allow
the operator of the robot to remotely
pick a wire or a boston cap and pull it
out
without squeezing it too hard and
crushing it which will blow it up
the the objective is to be able to
disassemble some of these
previous explosive devices to find out
how they're made and so forth so
that's one application i have a phd
student that's doing
work related to that okay as well as
doing that
um so uh and then i have a post doc
who's doing a biopsy needle insertion
so pushing a simulated needle into your
belly and going through the skin and the
fat and muscle
and each one of those interfaces you
feel a little bit different
sensation if you feel a fourth thing
that's not good because that's the
intestine
and that's not good and uh
so so uh so that's going along really
well and
uh doctors are saying it feels really
good so that's
that's another sort of application um
i had a young lady here a phd student
from china until just last month
doing a haptic dental training so
scraping and picking treat teeth finding
cavities carry dental caries put the
probe in the artificial
you know the virtual math and if the
probe sinks in just a little bit
instead of scraping across the tooth and
that sticks
and when you pull it out there's a kind
of sticks a little bit it means it's
decayed area of the tooth and then you
can go in and drill it
using a voxel-based approach the
different geometric approach and so on
so so that's another application we have
a lot of interest
in developing simulators for eye surgery
and ear surgery
those two things
but there's a big step going from the
hardware and software that we have
to turn key application again this is
like the magic wrist so
there's a huge amount of software that
has to be developed related to the
physical modeling and the graphical
display
so we're seeking partners to to try to
bridge that gap because we can't do it
all
ourselves here with my students but
we're we're doing a few things
taking a few steps in that direction i
would say
so are there any other researchers
looking at maglev for
this kind of half day well so tim saw
cutting
my cove co-investigator back at ibm
uh also pursued magnetic levitation
haptics at
university of british columbia for uh
several years and did some very
interesting things there with it
he's no longer doing that partly
because he can just buy it now
but uh
there is one of our systems at the
university of british columbia but it
it's not in his lab i don't think but uh
so uh uh so yeah that's the only other
okay yeah yeah there's about
maybe five or six companies in the world
that are producing
systems we interviewed ken salisbury as
well
and he's been doing haptics well ken
yeah ken
started uh well at mit
uh
he uh his student had the spinoff
company sensible technologies
i don't think ken has real
long gone from that having a
relationship with him maybe but
but uh yeah so he developed all these
incredible cable driven
devices robots and haptic devices and so
on
it's interesting he gave talk in 1993
again it was one of these international
symposium for robotics research
uh same place i same time i gave it
on on the magic wrist actually and the
magic risk use of the haptic device
so that was before we actually developed
a real happy device but we
temporarily we used the magic mist as a
haptic device
so he's turned into a company and now
my idea also turned out to be a company
sensible technology is just the world
leader and haptic
devices i think emerging always says
they are but
probably are in terms of money but but
in sensible sells these phantom devices
but
we have 25 times the performance that i
found so
we're hoping before but
you can try ours if you want if you have
time sure
absolutely yeah
so uh so uh as a company butterfly
haptics
is uh you know we're selling it on the
on the performance
you know we view it as a general purpose
scientific instrument that can be
interfaced to almost anything
back at ibm we actually interfaced the
magic risk to a scanning tunneling
microscope and we were the first to
feel atomic scale surfaces in real time
with your hand
by moving the scanning tunneling
microscope tip over a surface
you could feel the bumps which were due
to small clusters of atoms
we were first to do that that was with
tim sakunian
as well also
the technology same technology that we
developed from magic risk and the
haptics
can be used for vibration isolation i
tried to sell it for nasa
to nasa for i went to about five
different nasa places and tried to sell
it and
couldn't do it tim managed to convince
the canadian space agency and
bjarney trigbass and a canadian
astronaut
to pursue this lawrence levitation we
call it
lawrence force levitation as a vibration
system in space and so they did and uh
it flew for three years in the russian
mir space station
it also flew on the sts-85 shuttle
mission
and it was very successful a third
version is being prepared for the
iss hasn't been launched yet and who
knows if it ever will be but
um the idea is to isolate a payload
from vibrational disturbances in space
on the space station and on the shuttle
there's a lot of vibration going on
fans are running thrusters are
astronauts are pushing off the walls and
all this stuff so it's really
a microgravity environment but it's not
very good
unless it's isolated which we can do
with this technology
so uh another spin off
okay what do you see like time i guess
almost out of time we were just gonna
that's one last thing but we haven't
gotten to dynamically stable model
robots and this is the very first one oh
no
yeah let's uh let's do it over there
so uh so a few years back i had the idea
of uh
replacing the three or four wheels that
a conventional wheel robot has with a
robot that has only a single wheel and
that single wheel
would be a sphere and that's what this
is
so this is the world's first one here
called balbot it is
it can uh when the three legs
come up it can balance and it can move
about from point a to point b
very nicely it senses gravity using
fiber optic gyros and mems
accelerometers and so forth
we're adding a pair of arms to the robot
people around the world
are building ball bots actually eth
in switzerland and also in japan this is
a different version of balboa
over here it's built by my colleague in
japan
masaki kuma guy
there we have license the technology
and there will be commercial versions
available in
a year or two probably do you have
a lot of international collaborations or
so not so many international
collaborations
but i do have a lot of students coming
from
like switzerland and germany mainly and
some from china
some from holland from netherlands
some from france so
so what's the big advantage of this kind
of mobility
so a ball bot like this allows
with only one wheel the robot can be
tall and skinny
it also can move in any direction so
it's omni-directional
so something another balancing machine
like a segway
has two wheels but it must turn first
before it can move
so a ball bot like this can be in your
cluttered apartment building or
something and
it's almost impossible to have it be
trapped where it can't move because it
can move in any direction
it also because we only have one wheel
we can make that wheel be
rather large that's eight inches in
diameter down here
a large wheel means we can easily go
over carpets
changes of height go in and out of
elevators with gaps
no problem ah so that's another
advantage
also one depending on how it's
controlled it can be
very very gentle to you can guide it
around by hand you can push it
and you can do all of this without
having any external sensors
although we do have some external
centers and we're adding more and we're
adding two arms to it
but it's a good potential um
you know everyday life personal robot
personal provide type
yeah exactly so
it has some other with arms it's it can
we haven't demonstrated yet there's some
of the arm mechanisms here
it can potentially lift large loads by
leaning
much larger than a another typical robot
can do can operate on ramps and slopes
that would tip over a regular robot
see most robots in the lab are about
this high which is fine
but if you really want to be person
sized and you want to interact with
people
hri you want to look them in the eye and
everything
you're going to have a considerable
amount of mass up high which means high
center of gravity
means tip you either move slowly
or you tip over okay this robot can move
a meter and a half per second
and do that very gracefully in fact
so i think there are a lot of advantages
as research platforms
uh we don't have a plan for that
although we have shared a design with
many people i mean
various tips and so forth we don't
actually have the drawings online but
we could i guess
um what kind of advice do you have for
young people who
ah well robotics is uh for my life has
come from a hobby to a
what you might think of as a recognized
field
even though still it's not maybe in a
completely recognized field people
seem to know what robots are nowadays
and that's good for
if you want to go into that as a career
in the old days you know everybody just
left when you said robots because
you know that you know science fiction
uh people don't seem to laugh so much
anymore because you have robot airplanes
you have robot
surgeons you have all kinds of robots
they're doing things for you
i mean half your car nowadays a robot
so it's a good time
i i think if you want to do robotics
research that's
great it combines several different
fields into one if that's your
cup of tea if you don't necessarily want
to be an expert in
physics or an expert in electrical
engineering or
software but you want to be a jack of
all trades robots
integrate all those fields into one and
increasingly also uh the human element
which by the way i forgot to mention one
long time collaborator roberta klatsky
who is a psychologist
who uh and that's another we've done
continuously uh a number of
psychophysics experiments with our
magnet
magnetic levitation haptics where we've
measured the human ability to sense
textures to sense deformable objects and
so on and so forth
so uh yeah so well probably a lot more
right than i missed but
those are the highlights maybe the
highlight reel
i don't know that's great great
thank you very much yeah okay wow this
should be very interesting