The GuidiGO team has worked on more than 30 augmented reality (AR) experiences for museums over the last 2 years using Google’s AR platform – Tango and now ARCore. One of the main obstacles we’ve had to work around is occlusion.
What is occlusion?
Occlusion problems occur because virtual objects will always appear in front of the real world on the screen. For example, imagine that a virtual statue is placed in a gallery. If a person is standing behind the statue, the experience works because the virtual object appears in front of the person.
However, if the person is standing between the device and the virtual statue, the immersion is broken because the statue will cover the person on the screen.
Similarly, adding a virtual element inside a real object (such as adding a virtual apple inside a real bowl) can cause occlusion issues. The images below illustrate how the AR illusion works only for some angles. Again, the virtual object will always go in front of the real one.
So how can we solve occlusion problems?
Many occlusion problems for AR experiences in museums can be anticipated and avoided early on through design and forethought. Possible solutions to prevent occlusion issues include:
Create a designated space. A helpful way to control occlusion issues can be to contain where an AR experience will take place. For example, in the “Sites Eternels“ (Eternal Sites) application at the Grand Palais in Paris, the devices were physically tethered to a large table on which the AR experience appeared. This limited what could physically get between the device’s cameras and the table.
Another solution could be to physically mark the area where the AR experience will take place, so that other visitors know not to get in the way. Whether or not a museum employs this solution depends on the available space and whether the exhibition is temporary or permanent.
Utilize strategic positioning. The placement of virtual objects can help prevent occlusion problems, as some positions are more viable than others. For example, in the application we developed for the National Museum of Singapore, a 42-foot-long whale skeleton was positioned just under a high ceiling, well above visitors. In this case, because visitors point their devices up at the ceiling, the positioning of the whale ensures that there is nothing to cause occlusion issues.
Position your visitors in a specific place. In a recent project, we placed a virtual mark on the floor to encourage users to stand in a specific position before starting the experience. This way, we made sure nothing would get between the user and the AR scene.
Add masking effects. Environmental effects, such as clouds or mist, can help mask the real world elements that could cause occlusion problems.
For example, in another recent project, we animated a virtual spacecraft that lands on the surface of an asteroid. This experience took place in a small, crowded gallery, which caused occlusion issues. We added environmental effects to the virtual scene, and it helped to foster the sense of immersion in the experience.
Integrate real-world elements in the virtual world. In many cases, we can identify something called “planned occlusion”. For example, GuidiGO’s wayfinding capabilities display virtual blue dots on the floor that lead visitors to points of interest. In the case in the photo below, when the blue dots indicate to turn left, the rest of the path is not displayed until the visitor is in the doorway. This is possible because we integrate the pre-identified obstacles (walls, cases, etc) in our algorithm. The rendering happens in real-time and takes into account the physical world before displaying a digital object.
Position “occlusion masks”. Similarly, for a recent project, we wanted to display a virtual object along the length of a gallery wall. However, the room was filled with furniture that got between the visitor’s device and the wall, making the virtual object appear fake.
To work around this constraint, our programming team created “occlusion masks”. These were virtual 3D objects that were the exact same size and shape as the real-world objects, placed over the furniture using our AR Composer tool. In this way, we made it possible for the furniture to appear in front of the main virtual object.
Delineate museums into different sections. We can also separate museum spaces into unique galleries, so that when visitors are in a specific gallery, they only see the virtual objects on display in that gallery. This avoids the artworks of the galleries nearby being visible “through the walls” due to occlusion, as virtual objects will always appear in front of real objects on the screen. To read more about a project where we used this technique, check out our article about our work with the Clyfford Still Museum here.
Employ real-time programming solutions. Sometimes, occlusion problems cannot be avoided or planned for, such as visitors walking around inside the museum. There are programming solutions to these occlusion issues, in which the physical environment is reconstructed in real-time using device’s sensors and cameras to integrate the real world inside the virtual world.
The main downside of this option is that it requires a lot of processing power, however, this will certainly become a non-issue in the future.
Every AR experience requires custom solutions to specific occlusion problems. And the development of new sensors and more powerful devices will help facilitate new ways to work around these obstacles in the near future. Through creativity in design and programming, it is possible to find innovative solutions to make augmented reality even more real.