“Hey, Meta. Look at this and tell me which of these teas is caffeine free.”
I spoke these words while I was wearing a pair of Meta Ray-Bans at the tech giant’s New York headquarters while I was looking at four tea packets on a table, their caffeine labels blacked out with magic marker. There was a slight clicking sound in my ears, after which Meta’s AI voice told me that the chamomile tea was likely caffeine-free. It was reading labels and making decisions using generic AI.
I was demoing a feature that’s rolling out today for Meta’s second-generation Ray-Ban glasses, a feature that Meta CEO Mark Zuckerberg already promised in September when the new glasses were announced. The announcement was made. AI features, which can access Meta’s on-glass cameras to view images and interpret them with generative AI, were scheduled to launch in 2024. Meta has moved to introduce these features much more quickly than I expected, although the early-access mode is still very much in beta. With the addition of Bing-powered search to Ray-Ban as part of a new update that boosts the power of the glasses’ already available voice-enabled capabilities, Meta’s glasses are quickly beginning to gain a number of new capabilities. Have been.
I was very surprised by the demo because I had never seen anything like it before. I have a few takeaways: Google Lens and other on-phone devices already use cameras and AI together, and Google Glass — a decade ago — had some translation tools. That said, the simple way of applying AI to identify things in the world around me under the lens of Meta seems quite advanced. I’m excited to try it more.
Multimodal AI: How it works now
This feature still has limitations. It can only recognize what you see by taking a photo, which is analyzed by AI. You can hear the shutter snap after you make the sound request, and there’s a pause of a few seconds before the response comes. Voice prompts are also worded: every voice request on Meta glasses must begin with “Hey, Meta.” And then you just have to follow up with “Take a look at this” to trigger the photo being taken, followed immediately by whatever you want the AI to do. “Hey, Meta, look at this and tell me a recipe with these ingredients.” “Hey, Meta, look at this and make a funny caption.” “Hey, Meta, look at this. What plant is this?”
Each AI response, and the picture it sees, is stored in the Meta View phone app that pairs with the glasses. I like this, because it’s a visual/written record for later, like memory-jogging notes. I can see myself walking around somewhere and asking it questions, using it as some kind of mindless Google search for my eyes while I’m shopping or whatnot.
It may also have potential use for auxiliary purposes. I wore a test pair of Meta glasses that didn’t have my prescription, and I asked him what I was seeing. Answers may vary in detail and accuracy, but this may be helpful. She knew I was showing her my glasses, which she said had blue lenses (blue-black frame, close enough).
Sometimes it can cause hallucinations. I asked Gilas about the fruits in a bowl in front of me, and she said there were oranges, bananas, dragonfruit, apples, and pomegranates. It was perfect, except for the pomegranate. (None of them were.) I was asked to create a caption for a large stuffed panda in front of a window. It made some cute ones, but one was about someone being alone and looking at a phone, which doesn’t match.
I looked at a menu in Spanish and asked Glasses to show me the spicy dishes. It read some of the recipes and translated some key ingredients for me, but I asked again about the meat dishes and it read everything in Spanish.
The possibilities here are countless and fascinating, and potentially incredibly useful. Meta acknowledges that this initial launch will be about discovering bugs and helping develop the way the on-glass AI works. I found that there were a lot of “Hey, Meta, look at this” moments. But that process may change, who knows. When engaged in immediate image analysis, asking direct follow-up questions may do the trick without saying “look at this again”, but I’m sure my success will vary.
The future of wearable AI is getting interesting
This AI, which Meta calls “multimodal AI” because it uses cameras and voice chat simultaneously, is a precursor to future AI in which the company plans to mix multiple types of inputs, including more sensory data. Qualcomm’s AI-focused chipset on the Meta’s new Ray-Bans already seems ready to take on more. This is also a process that Meta plans to make more seamless over time.
Meta CTO Andrew Bosworth told me in September that while the glasses now require a voice prompt to activate and “see” so they don’t degrade over the course of battery life, eventually they “will have sensors that are less power-efficient.” Will be what they will be.” Capable of detecting an event that creates awareness that triggers the AI. This is really the dream we’re working towards.” Meta is already researching AI tools that blend multiple forms of sensory data together ahead of more advanced wearable devices in the future.
Now, know that this is an early-access beta. Meta is using anonymized query data to help improve its AI services during the early access phase, which may be a concern for people wanting more privacy. I don’t know the specific opt-in details yet, but more separate controls over data sharing seem like they could be implemented after the final AI features launch, possibly next year.
All of this reminds me of exactly what Humane is aiming for with its wearable AI Pin, a device I haven’t seen in person yet. While Human’s product is expensive and has to be worn over clothing, Meta’s glasses cost $300 and are already on store shelves. As watches, VR headsets, and smart glasses all develop their AI capabilities, things could be very different for the future of wearable tech and the level of assistant awareness.
It’s becoming clear that a new frontier of wearable AI products is already underway, and Meta’s glasses are the first to get here.