Getting Spooky with iOS 12 Object Detection
Since ARKit’s initial release in iOS 10, I have patiently awaited the arrival of one feature — object detection. With the release of iOS 12, we now have our first taste of the technology in a mobile format. I put together a small experience to showcase the technology in motion.
Getting Spooky with ARKit
Armed with a pumpkin, some 3D models and a love for Halloween, I set out to create an object based AR experience. My journey ended with a spooky AR Jack-o’-lantern. The best part — no knife was needed and I was never elbow deep in pumpkin guts. Check out the vegetables of my labor in the video below.
So What’s Actually Going On Here?
Good question! Let’s take a brief look under the hood of ARKit’s new object detection feature. Below we have a few “behind the scenes” frames from my Jack-o’-lantern experience.
In the leftmost frame, we start with our real world pumpkin. Earlier, I scanned it to create an “arobject” file. This “arobject” is not a 3d model, rather a collection of data points that uniquely define our pumpkin (color, texture, position, acc.).
In the second frame, you can see a visualization of that “arobject”. A yellow cube marks the outer bounds of our identification zone, while yellow dots indicate stored data points that have been identified. A single data point won’t do too much for us. However, once enough points are identified, we can determine the real world location of our pumpkin and use it as an anchor in space.
Now it’s time for some spooky augmentation. In the third and final frame, I use the anchor we created to bring our Jack-o’-lantern to life. I throw a face and occlusion mask on our pumpkin, surround it with candles, and add some particle animations for dramatic effect.
Boo! With a few second of environmental scanning we now have an AR Jack-o’-lantern.
What’s Next?
Though this is a simple experiment, it represents a giant stride for the technology. Placing AR models in void space is a nice trick, but any practical implementation of this technology will require us to seamlessly and intelligently integrate with our physical environment. The ability to digest and identify three dimensional object is the first step in making those mixed reality experiences practical.
Conner Hasbrouck is an Engineer at Perficent Digital Labs. Read more about how Perficient Digital Labs uses Emerging Technology to create innovative experiences that transform the way our clients do business here.
