AR Minute with Dr. Flora Tasse - How Spatial Maps Enable Smartphone AR
Welcome to The AR Minute! At Streem®, we build cutting edge experiences and technologies that foster human connections, solve real problems, and always put the customer first. In this series we'll interview experts from across our team to learn about the people, processes, and tools that go into building user-first innovative technologies—and even the Metaverse.
This week we talk with Dr. Flora Tasse, Head of AI and Computer Vision at Streem. We’ll learn what augmented reality (AR) is and how spatial mapping gives your smartphone the context it needs to deliver helpful AR experiences.
What does a smartphone need to know to share an AR experience?
- What is the phone’s position and motion in space? To display AR in the context of your room, a smartphone must "know" where it is in space and when it is moving.
- What is it looking at? Immersive AR apps must also understand the context of what they are looking at, in order to display AR effects in the right time and place.
- What is the right experience for the customer? When AR apps understand where the consumer is and the details of their surroundings, spatial AI, can tailor and personalize AR apps to deliver just the right experience.
What is a spatial map?
Spatial mapping happens when a smartphone, head-mounted display (HMD), or other AR device scans a 3D space and creates a digital copy of what it sees.
Spatial maps can be created from 2D images by analyzing the colors (RGB), shadows, and edges against the camera’s position and movement.
More precise spatial maps, sometimes called “digital twins,” leverage laser-precise Light Detection and Ranging (LiDAR) sensors to create photo-realistic 3D models of spaces and environments.
Watch the AR Minute with Dr. Tasse
Video transcript follows below.
Nathan: Welcome to AR minute with Streem I'm Nathan. And today I'm talking with Dr. Flora Tasse, head of Computer Vision and AI at Streem. We've got about a minute to learn all we can about spatial mapping with AR and AI. So let's get started.
Nathan: What is augmented reality or AR and what does a smartphone need to understand to make AR happen?
Flora: Let’s think of an example, let's say that you just got a new washing machine and you have no idea how to use it. And instead of using you are manual, you might want to have a treated experience around your washing machine, almost like a special effect in the movie that tells you how to study it.
How to make sure that everything is working correctly and how to start your machine. And so you want some virtual reality wrapped around your actual physical washing machine that can tell you how to use it.
The smartphone, it's basically a representation of your brain. So it needs to know key things.
- Where is it in your 3d space?
- How is it moving?
- What is it looking at?
And what is the right experience to put around what the camera is saying?
Nathan: At the beginning, we said, we were going to talk about spatial mapping and what spatial mapping means.
Flora: Okay. I think about it as a digital copy of your environment. So it needs to know how far any object is from the camera. It needs to know what is the shape of different objects. And so that's how you can do things like measurements. And it's basically a copy of your physical world that you can edit.
So you can add things, you can move objects, you can replace them. And that's really what the term “spatial map” encapsulates.
Nathan: How does spatial AI use a spatial maps and the camera?
Flora: What is special about Spatial Ai is that it's “spatial”? So miss lo what is best is around it, where objects are, what they look like. And so really to make special AI work, we need to know what this special map of the environment and what you are looking at at any given moment in time.
Nathan: What's some of the cool things you can do when you know what you're looking at?
Flora: One thing that you can do is just, as the camera is moving around, just have different experiences that I really tell a lot to what it's looking at that moment in time. If I'm looking at my washing machine is a different experience.And if I'm looking at my TV, or if I'm looking at my kitchen.
Nathan: Going through those menus???.
Flora: You had those menus, yes. You just want to compare to, you know, your camera, to look at something and just knows what is right.
What is the right experience?