The Facebook Reality Labs division has unveiled one of its most stunning projects: an augmented reality interface based on interactions on the wrist.
Facebook has decided to step up on its research work on augmented reality and virtual reality. Supported by the Facebook Reality Labs, the various projects are regularly talked about, even if the majority remains secret.
After exhibiting its vision for the next 10 years on human-machine interactions and interfaces, Facebook formalized this Thursday its progress on its IT platform of the future. And for that, the wrist is its unexpected physical support.
An intuitive, permanent and non-intrusive device
A suprise ? Not that much, according to the engineers and researchers from the Facebook Reality Labs responsible for the project. The central idea: to find an interface offering little friction, intuitive and easy to use, while offering great reliability and confidentiality. All without being cut off from the real world.
So to wear it all, there are obviously the augmented reality glasses that are in the boxes of Facebook. And for the interaction, the nerve center, both concretely and abstractly, would then be the wrist.
Six years ago, with the founding of Facebook Reality Labs Research, a named successor to Oculus Research, teams began to design generalized entry technology, “able to meet the wide variety of needs of all types of people and be usable in all everyday situations ”, explains Facebook.
So began a reflection around a device “intuitive, always available, non-intrusive and easy to use», But also capable of handling more complex commands such as manipulating virtual objects or modifying a document. And above all, it had to be comfortable to wear on a daily basis, able to operate without interruption and without consuming too much energy.
The wrist, an asset at your fingertips
The voice is intuitive, but does not allow privacy or reliability. A device such as a smartphone increases friction as soon as you put it on a table or in your bag. Thus, FRL had the idea of a bracelet encased in sensors, worn on the wrist and capable of being forgotten at any time. Comfort, ease of use and confidentiality: all the boxes were checked for a platform (system, battery, antennas) with easy interactions with the hands for augmented reality.
But, behind the technological device, there is also a scientific translation called EMG or electromyography. EMG uses different sensors to translate electrical signals sent from motor nerves in the spinal cord to the wrist and hand. The device then transcribes them into commands to communicate and control various functionalities in a customizable way, but also adapted to each person’s situation, valid or not.
And that’s why the Facebook researchers focused on the wrist, an area where it is easier to pick up the signals sent by the neural system. Ask his little finger to stand up, his hand to turn around … the simple intention is picked up by the EMG.
Anticipate your action
« With neural interfaces, we seek to give you direct control of the machine, by using broadcasts from the peripheral nervous system, the nerves outside the brain that move your body», Explains Thomas Reardon, Director of Neuromotor Interfaces at FRL. “Instead of writing or dragging, you express your intention the same way through your system nervous system, and the bracelet uses the electrical activity of neurons throughout your nervous system to create new, intuitive ways to interact with your devices. »
Here, there is no magic or telepathy, there is no question of the firm reading your thoughts, but just of transcribing an electric current in your body which translates an intention. The same signal as the one sent when you perform an action on your smartphone screen or type on a keyboard. Facebook is thinking about several prototypes, with sensors that could be in the form of miniature bumpers against your skin or flat and in sensations.
The possible uses of EMG, on the other hand, are already quite advanced. The bracelet could thus be used to type a text on a virtual keyboard that does not exist elsewhere than in front of your eyes in augmented reality. And that would have several benefits: adapting the “dimensions” of the keyboard to your gestures, the typing speed or the order of the keys. Top-notch virtual customization. The playful aspect is not forgotten with the possibility of “virtualizing” any object taken in hand such as a bow and arrows, a tennis racket, etc. All that often requires additional accessories on which to put sensors to use them in virtual or augmented reality.
And for this, the FRL researchers also evoke a “haptic dimension”, a sort of response of the system to the wrist which will allow better immersion by adding sensations and also guarantee that the action corresponds to reality, whether you have it. successful or not, just like in real life. For Nicholas Colonnese, head of science research at FRL, haptics is also a way to add and transmit emotion. What he called “haptic emojis”, translations of feedback that could be symbolized in the mind by the famous images and allow playful interactions.
The smart click or the AI that supports you
At the level of controls, the individual could interact in a very intuitive way with his hand or his fingers, by sweeping, pinching, moving. Gestures that are easy to perform in any circumstance, while talking, walking or otherwise, with an appropriate response.
Facebook baptized this “Smart Click”. These are micro-gestures combined with an adaptive interface. The AI thus translates the data input and is able to adapt its response to the context. You are cooking because your actions betray your actions. The AI has understood this and can come up with / understand the need for a timer because you might have started a recipe in augmented reality. It proposes an action and you only have to validate or cancel.
The AI found that you went out for a run like every Thursday morning and that a lot of times you listen to music. The adaptive interface recognizes this and suggests the playlist even before you trigger it. Perhaps also simply because the perceived neural signal is the same in this kind of circumstance.
Thousandths of a second gained that accumulate
It is not a question of seeing the AI anticipate your needs, but just to understand them faster and to gain a few thousandths of a second which, accumulated at the end of the day, can prove to be a precious time saved.
For the time being, Facebook Reality Labs has not communicated a release date for its project, which is still in the development of a prototype. The first returns are “promising», Assures Sean Keller, project manager. And to add: “This phase of innovation and discovery is an incredible moment, because it announces an upheaval of the Old World, a change in the rules that we have followed and on which we have relied until then to make technology evolve.. But the change is not yet for now.