There are several visions of the ideal interface for interacting in virtual or augmented reality. Controllers for demanding applications such as games, hand tracking for ease of use and portability, and gloves for combining natural gestures with a still little used sense: touch.
Stimulating this sense mainly happens today through simple vibrations. An innovation from the days of the Nintendo 64 “Rumble Pak” that has now become commonplace in video game entertainment. Even if the technology progresses (for example with the new controllers of the PlayStation 5), it does not allow today to feel the shape or the texture of an object.
Meta’s research teams are publishing an article on the evening of November 16th detailing their work on the creation of haptic gloves capable of providing sensations good enough to trick the brain while being light and comfortable. A Herculean task, but which seems attainable to them today, after seven years of research on the subject.
The haptic puzzle
Why Herculean? Because it requires innovations in smart textiles, soft robotics, microfluidics and highly personalized industrial manufacturing. Not to mention of course the problems of tracking and rendering of sensations.
The objective is to be able to feel the edges and corners of a cube for example, but also to know if it is made of a rough or smooth material. This same technology would make it possible to feel the fact of pressing a virtual button or typing on a numeric keypad, or even exchanging a “virtual handshake”, among many other possibilities.
“We create practically everything from scratch in this discipline,” says Sean Keller, who leads this research and oversees several hundred scientists within Meta’s Reality Labs division, in the post. The first blocking element encountered by his team was the size of the existing actuators in 2014.
Miniaturized pneumatic actuators
Indeed, the latter (small motors that control a valve) are the key to the haptic feeling: by activating according to the right sequence, they press on the skin, even oppose resistance to the fingers, which gives the illusion of the user to touch a specific shape or texture. To provide a quality haptic experience, you have to be able to condense dozens, even hundreds of these small motors into a glove.
HaptX, one of the hottest start-ups in the industry, uses 130 pneumatic actuators in its glove, which is connected to a large box from which air is sent. Reality Labs researchers have also made significant progress in the field of pneumatic actuators, as well as electroactive actuators, whose size and shape are changed by an electric current. However, they do not reveal how many actuators they managed to fit into a glove.
To control these actuators, they also developed a “high-speed” microfluidic chip, located in the glove, which manages the airflow. The researchers highlight the speed, lightness and portability of the system, whose response time is measured in milliseconds.
Important element: all the equipment to which the glove is connected and which remains essential to its functioning
Fool the brain with partial sensations
In parallel, on the software level, they have developed a haptic rendering engine in order to provide the right instructions to the glove so that the stimuli sent to the user correspond as closely as possible to the virtual scene, and specifically to the texture, the weight and rigidity of virtual objects. The thing is particularly complex because the actuators do not make it possible to perfectly reproduce real sensations. The gloves offer only limited resistance but cannot prevent a user from clenching their fist or allowing them to lean on a virtual surface.
They were therefore interested in cognitive science and in particular in multisensory perception, that is to say the way in which the brain merges different senses to form its perception of the world around it. This realm is all about tricking the brain as best you can. For example, when we see a virtual cube made of a material resembling wood, we already have certain estimations based on our past experiences of the texture, shape and weight it will have.
By grasping the object and manipulating it, these visual data are combined with those of our muscles and proprioception (the body’s ability to sense the position and movements of its limbs) to refine visual perception. We therefore reinforce the impression of realism by “confirming” the visual, even if the haptic perception is not perfect. It is for example possible, if the timing is perfect, to simulate the weight by slightly pulling on the fingers when the user lifts an object, to imitate the effect of gravity.
Manufacturing on an industrial scale, the ultimate challenge
To fit all these technologies into a thin and light glove, Meta has also embarked on the development of smart textiles capable of combining several functions (conduction of electricity, capacitive movement, measurement) in the same fabric. There remains the problem of personalization, each hand being different. At present the prototypes they use are handcrafted by specialist engineers, but cannot be produced on a larger scale.
Researchers are therefore studying new manufacturing methods, because the effectiveness of gloves is largely based on the fact that they fit perfectly to their user. This means being able to insert the actuators precisely where they need to be, no matter the size or shape of the hand.
To be clear, this research remains today far ahead of a hypothetical commercial product (which will not see the light of day for another 10 to 15 years, according to Meta), but it is interesting to see the progress of the company. in the matter given its more or less unlimited budget. The researchers are also enthusiastic about possible outlets in other sectors, for example medical analysis, microfluidic biochemistry, or assistive devices for people with disabilities.