LESSON 3 - MODERN HISTORY AND DEVELOPMENT
Usually when we think of virtual reality we associate it with computers and cyberspace. This is because the ability to interact with a virtual world requires the use of equipment of such complexity that it would have not been possible prior to the advent of the electronic computer. In fact many of the non-interacting improvements to the synthesis of virtual worlds have only been accomplished with the aid of the computer.
For most of this century the movie industry has been the leader in creating "virtual worlds." Generally films contained two types: those containing real characters with "created" settings and those which were completely created, including the characters. Walt Disney pioneered the use of animation to produce a totally synthetic environment and characters. In 1964 Disney combined both real and animated characters in the same scenes in the movie Mary Poppins. This was done entirely by hand without the use of computers. However the advent of high powered computers has enabled far more complex activities.
The movie industry is one of the largest users of supercomputers, such as those manufactured by Cray. Given sufficient computing power it is possible to create or modify a visual scene; store, alter, and invent sounds; and make possible actions which appear to defy natural laws.
The computer-controlled camera has allowed realistic sequences to be filmed with models which would otherwise have been impossible. George Lucas and Industrial Light and Magic have become associated with the latest and best of such techniques. Powerful graphics programs allow the creation of entire stage settings with no stage or props. In Star Trek II, the landscape of the Genesis Planet existed only in the memory of a computer. The ability of a computer program to reshape or modify a picture, one pixel at a time, has opened entire new worlds to possible inclusion in a movie.
In the same way that photography made possible the inclusion of previously impossible levels of detail in a visual presentation, computers have made possible previously impossible levels of manipulation of visual scenes and their contents. They have provided the ability to add color to movies which were filmed in black and white; to mix real and animated actions; to enable real actors to appear to defy natural laws.
Many of the modern techniques used in VR equipment are not necessarily new ideas - they just had to wait until computer technology made them practical. For example, a 1950s fiction story postulated use of a computer to synthesize sentences from small samples of sounds. Thus someone could have samples of his voice recorded and the different basic sounds used to produce an entirely different statement but given in his own voice. A discussion of the story stated that the idea was sound but that it would require a computer the size of the state of Texas. Less than 40 years later this can be done with a laptop.
Many other techniques and approaches to VR work have been invented in response to the development of newer, faster, and more powerful computers. As is often the case with new technology, many of what will become the most common uses were never dreamed of by the computer developers. Originally computers were developed for and thought of as devices to solve complex math problems. The idea of using them to manipulate words or pictures was not even considered. After the computer was developed someone looking at its capabilities thought: "Why couldn’t they be used to ........." It wasn’t until 1968 that a computer was used to display text and to cut and paste.
Early computers required punched tapes or cards to enter data and produced output on the same medium. A turn around time of 24 hours was often required between submitting a program and receiving the results. Operation of the computer itself required the efforts of trained technicians. Under these conditions it is easily seen why computers did not immediately lend themselves to development of virtual reality.
In the early 1950s the use of cathode ray tubes (CRTs) was first considered for computer output display. Douglas Engelbart, an engineer with a background in radar (which used CRTs for displays) pursued the idea. He also developed the idea of using the CRT to provide program input. Because of the shortage of available computers and funds, it was not until 1968 that he demonstrated the use of a CRT to display a document. Using the predecessor of the mouse he cut and pasted sections of the document, thus showing the interactive capabilities of computers.
Until about 1970 computers required the use of batch files and long processing turn around times. Teletype units replaced punched tapes and cards, but these still required paper output. Also interface with the computer was limited to a single user. Around this time a major breakthrough in the use of computers took place: time sharing. In a time sharing situation, a number of users could input data and run a variety of different programs at the same time. Actually, the computer ran only one program at a time, but the effect was of simultaneous operation. To understand this you need to know that most of the time in running an early program was spent in waiting. The actual computing operations were very fast, but transferring data into and out of the computer - what is now done as a hard disk access - required the vast majority of the time for most computers. Some large computers had smaller computers with the sole job of reading in and printing out data. Time sharing allowed the computer to work on one program until it needed to wait for data - usually a few milliseconds - and while the data was being obtained to operate on a different program. By working on a number of programs a few milliseconds at a time, the computer created the effect of simultaneous operation. This allowed a number (as many as 25 to 50) of users to all run individual programs.
Even with time share the output was still limited to printed text. Also, if a single character in a document changed, the operator could not see this unless the entire document was printed out again. That is, because the output was on paper, no changes could be made to a previously printed copy. In the early 1970s, teletype terminals were replaced with CRT displays. These were monochrome and usually limited to text output, but they offered the considerable advantage that the writing time was much faster than a paper printer. Using a CRT terminal, a change in a document could be made and another copy displayed in a few seconds per page. Improvements to these terminals have added color and graphics capabilities.
Another change which was to have a drastic effect on the development of VR systems took place in the early 1970s. This was the development of the microprocessor integrated circuit. Computers had always been very large, power hungry devices, often taking up several rooms. (A prediction was made in the late 1960s that someday computers might be as small as one and one half tons.) As a direct result of research for the US space program, the techniques of complicated integrated circuits were developed. The result was a computer with the circuit small enough to hold in your hand. By about 1980 these chips had allowed the development of the "personal computer", a desk-top device small and inexpensive enough to be owned and used by an individual. This literally placed the power of the computer in the hands of millions of individuals and allowed them to develop ideas for its use.
Speeds and complexity of microcomputers have increased orders of magnitude during the twenty years since their introduction. Today’s microprocessors are more powerful, faster, and control more memory than the large mainframes of a few years ago. One computer expert has said that when he throws away one of the birthday cards upon which you can record your own message, he is throwing away more computing power than existed in the world thirty years ago.
The CRT display - now commonly called a monitor - provides a high speed output device which could display both text and graphics in color. However, it is still only two dimensional. It also lacks the familiarity of the real world. As early as 1962 Ivan Sutherland began considering these limitations. In the late 1960s Sutherland worked to develop a head-mounted display (HMD) which could present a computer-generated view which the user would see in the same natural fashion as he looked at the rest of the world. Early HMDs were complex and very heavy. Overhead supports were required to carry the weight and large computers were needed to drive the displays. The complexity of the necessary calculations resulted in a low resolution, low speed display. But it still could provide a three dimensional, computer-generated world which the user could view in a normal manner. Today’s HMDs are just advanced versions, not a new concept.
Even with the limitations of complexity and speed, it was possible with computers to create real-time scenes. This had always been a limitation of simulators. Flight simulators were one of the earliest attempts to provide an artificial virtual world. As far back as the 1920s crude flight simulators existed. Although no visual input could be created to use with these simulators, they proved extremely useful in training pilots for instrument flying. Not only was a simulator safer for a new pilot to fly than was a real airplane, it was also less expensive. Additionally it could also be used to place the trainee in situations which were not allowed by a real aircraft.
As aircraft became more expensive and more complicated to fly, the need for more realistic simulators grew. With the advent of the computer and graphical displays it became possible to create real time visual scenes which could change in response to the user’s actions in the simulator. The development of the portable video camera made possible the use of models with the camera mounted above on a control device which allowed the camera to be tilted and moved in response to the pilot’s control stick in the simulator. Later, the use of video cameras allowed real scenes to be photographed and the computer processing allowed the viewpoint of the scenes to be changes in response to the pilot’s control actions.
Today simulators include real time visual displays, sometimes present on screens surrounding the cockpit and sometimes as 3-D pictures in a head-mounted display. These coupled with audio and motion platforms can provide a very realistic experience. Their use has become a necessity in the training of pilots for expensive and complicated high-performance aircraft and spacecraft. Use of these simulators has allowed the development of contingency plans and emergency procedures which have saved many lives in actual flight and space operations.
By the mid 1970s advances in computer technology had made possible another step in the development of interactive virtual worlds. Video of real images, including the user, could be mixed with computer generated images and displayed as a single picture. Thus the operator could interact with non-real , software generated objects which only he could see.
One area which received less attention until recently was the use of tactile interaction. True, the flight simulators used the control forces on the stick as inputs and the motion platform as outputs, but no objects in the display were moved about. In the 1970s and 1980s some systems using a force feedback were developed. One of these, UNC’s Grope system, is described in the text. This system used as its basis a remote manipulator which had originally been designed for handling radioactive material. These manipulators were mechanically linked control devices which allowed an operator to grasp a control handle and operate a remote "hand". Because the linkage was mechanical, the operator could feel the forces applied to the hand when it grasped something. In the UNC application, motors were added to provide this feel in response to computer generated interactions. The manipulator sensed the control forces applied by the operator and moved a computer generated visual object (a molecule) in response. When the object encountered another computer generated object, the expected force was produced by the motors and transmitted to the user’s hand on the control.
Actually, force feed back had been long considered necessary in an electronically controlled system such as would be used in a fly-by-wire aircraft. In all aircraft until the F-16, the pilot’s control stick was physically attached to the rudder and elevator by metal cables. In large or high performance aircraft, the force of the pilot on the stick could not move these control surfaces so hydraulic boosters were added. The control force - and thus the feedback force to the pilot - was transmitted through these cables. Aircraft designers would like to eliminate these mechanical linkages and use electrical signals to control the hydraulic actuators at the tail surfaces. This type of system was called fly-by-wire. The designers realized that part of the information the pilot used to control the aircraft was the "feel" of the stick - i.e., the force feedback from the tail surfaces. From the earliest designs, fly-by-wire systems have included methods to apply force to the pilot’s stick in response to those forces sensed on the rudder and elevator.
Inputs to such devices used joysticks, control handles, or some similar device. A more desirable situation would be one in which the operator’s hand movements could be sensed directly. Thomas Zimmerman developed a data glove which could measure the angles of the various joints of the wearer’s hand. Although these gloves measured angles of only some of the joints and needed frequent recalibration, they provided an entirely new input device. Now a user could make natural motions with his hand which could be sensed by the computer and used to move a computer generated hand in the display. These signals could also be transmitted to a remote location and used to control a robot hand.
A term commonly used for such remotely controlled manipulators as that used by UNC in its Grope systems is Waldo. This term originated in a science fiction story by Robert Heinlein dating from over 40 years ago. In this story he described a system very like modern telerobotic systems in which the operator donned control rings (very like a modern data glove) and operated a remote robot hand or other device. Visual feedback was supplied on screens and force feedback to the control rings.
Much of the development work in these areas has been sponsored by NASA. It is easy to see why there would be an advantage in having an operator use a 3-D HMD and a data glove (or suit) to control a repair robot working in the vacuum of space. Much of the development of the software necessary to present 3-D displays and to be able to move and rotate objects within them is also a direct result of NASA work.
One additional area on sensory input which has recently been developed is 3-D sound. In the real world it is possible for someone to accurately locate the source of a sound with no other clues except the sound itself. Stereo sound was a first crude step in this direction, but it provided only a two dimensional position. The human ear obtains its information from the relative volumes, times of arrival, and shaping (done by the structure of the ear) of the two sounds at the two ears in much the same way as two eyes provide 3-D vision. New computer modeling programs allow a sound to be placed at any location relative to the listener for a realistic effect.
DEGREES OF FREEDOM
To understand what is necessary to, for example, sense a hand position and use it to control a robot it is necessary to understand the term "degrees of freedom". The number of degrees of freedom (DOF) belonging to an object is a measure of the number of ways the object can move. For example, a simple lever control can move only back and forth: thus it has one DOF. A typical joystick can move back and forth as well as right and left. This would then have two degrees of freedom. A steering wheel of an automobile has only one DOF: it can rotate right and left. Add a tilt control and this increases to two.
In general, an object can have six special degrees of freedom. These include three translational and three rotational. Consider a rotary wing aircraft (helicopter) in flight. It can move along three axes: forward-back, right-left, and up-down. These are the three translational DOF. However, it can also rotate about three axes. It can rotate about the front-back axis (roll), about the right-left axis (pitch), and about the up-down axis (yaw). These are the three rotational DOF.
Now if we examine the human hand we see that different joints have different numbers of DOF. The fingers can bend at each of two joints. They can also move right-left and up-down at the knuckle. There is little rotation at these joint so that can usually be ignored. Still we are left with six DOF for each of the four fingers. The thumb has one bending joint and a knuckle that can move right-left, up-down and rotate. This means we have a total of at least 28 DOF just for the fingers of one hand. Add the necessary DOF for wrist, elbow and shoulder and you can see that modeling the human body can become very complicated very quickly. Remember, to reproduce a picture of the hand in a virtual world it is necessary to measure angles or movement for each degree of freedom and then perform a set of extremely complex mathematical operations. If these measurements are to be used to control a robot hand it is also necessary to measure angles or movement for each DOF in the robot hand.
As might be expected this complex model is often simplified. Most robot hands, for example, have only two fingers and a non-rotating thumb located opposite them. Many do not allow fingers to bend at all, much less at two joints. If we are controlling such a simplified hand it is not necessary to measure the extra DOFs of the real hand as they are not necessary for operation of the robot. In its simplest form the hand need only respond to open-close - i.e., one DOF.
If we wish to make our robot perform more complicated tasks, it is necessary to add more degrees of freedom. But doubling the DOF increases the necessary computation by a much larger factor as well as requiring more sensors for angles and positions. As computing power increases and our ability to manufacture miniature mechanical devices improves, the complexity - and thus the abilities - of our models will also advance.
It should also be noted that the term degree of freedom can apply to more than just spatial movement. For example, a light attached to the end of a manipulator could possibly be rotated about all three axes and moved in all three directions. But it can also be turned on and off (or varied in intensity). Thus it would have seven DOF. Or in addition to being either on or off it could have three colors of light which could be controlled individually. In this case there would be a total of nine DOF. Add sonic transducers capable of producing six different tones and we have fifteen DOF. As you can see modeling the real world is not a simple task.