When museums think about remote immersive experiences, Virtual Reality (VR) is usually the first technology that comes to mind. While we have created VR experiences in the past – such as an entire exhibition for the Carnegie Museum of Natural History – VR did not seem to be the best fit in this instance. The education team’s brief included the following stipulations:

• Immersive life-size experience
• No application to download
• Usable instantly by anyone on any mobile device
• Compatible with computers and tablets
• Photorealistic rendering

Three years ago we would have replied: “Sorry, this is not possible.” But Mobile AR has changed the game and is now able to tick all these boxes.

When iOS and Android adopted .USDZ and .GLB formats in 2018, it became possible for more than one billion devices to display an AR experience directly inside a mobile browser.

• .GLB is a 3D file format that describes an object according to its surface, with meshes, materials and textures. .GLB is the bundled version of the .GLTF format which was established by the Khronos Group, and is supported by Google, Microsoft, Facebook, and Adobe.
• .USDZ is another 3D file format developed by Pixar and Apple. It’s the only AR file format compatible with Apple devices.

Let’s put it this way, .USDZ and .GLB files are to AR what .PDF files are to documents. They display 3D objects, in AR in your mobile browser (Chrome or Safari), in 3D in your computer browser (similar to Sketchfab).

If you would like to see for yourself what mobile AR looks like, try the following:

• On your mobile device, google one of these words: Tiger, Apollo 11, Neil Armstrong, Chauvet cave, or Tyrannosaurus.
• Tap the “view in 3D” icon, next to the relevant thumbnail.

More AR search examples are available here.

10 exhibits in 3 weeks to be AR ready

The NHMU Education Department requested this AR functionality for the beginning of the school year. Museum staff had already scanned their exhibits in photogrammetry, including a selection of paleontological items, bird skulls, and artifacts of Native American culture.

The GuidiGO AR team used GuidiGO Studio and AR Composer platform to quickly optimize the mesh and textures for Android and iOS devices and make sure the quality of the 3D models was consistent especially in low light environments.

We are thrilled with the result. Check it out for yourself by displaying the baby mammoth skeleton in your room! We are even more excited to hear about the impact of our work after receiving this kind note from Tracey Collins, Interim Director of Education at NHMU:

NHMU was thrilled to partner with GuidiGO to deliver such innovative science education and immersive learning experiences to 25,000 students in Utah this school year. We have received great feedback so far and are looking for new ways, Museum-wide, to incorporate AR functionality into all of our virtual opportunities.

To learn more about GuidiGO AR, visit www.guidigo.com/AR

Other example by GuidiGO – Tap on the image below on your mobile to discover this famous masterpiece in AR, in real size:

This AR experience is part of “Launchpad: Apollo 11 Promises Kept“, a new 10,000 sq. ft. exhibition paying tribute to the past, present and future of space exploration. The exhibition combines the Museum’s significant collection of Apollo-era artifacts with cutting-edge interactive experiences to create an immersive and exciting cosmic journey.

Photo by Fort Worth Museum of Science and History

What is unique about AR Composer is that digital objects keep their original positions, creating persistent AR experiences shareable across several devices.

To position digital artefacts accurately, AR Composer comes with an easy-to-use 6-DOF (Degrees Of Freedom) Editor. 3D objects can be moved from left to right, up and down, forwards and backwards, including three rotation options. With this unique feature, objects can be placed accurately in any space, in no time.

Discover how to try AR Composer at the end of this post.

AR Composer also comes with 2 modes: Creator and Visitor mode. In Creator mode, the museum team can create as many persistent scenes as they want, placing and anchoring their objects wherever they like in the physical space. When this process is finished, the museum simply has to lock the scenes in Visitor mode to make them available on any device running AR Composer. Visitors can then use the devices provided by the venue or their personal device to enjoy the content.

Another perk of AR Composer is that there is no limit to the number of 3D assets that can be uploaded, which gives digital and education teams at museums the opportunity to design scenes with the creativity of a theatrical production. Objects can be animated, interactive and can display data to showcase digital storytelling or contextual information.

There are a number of creative ways to use the app. Examples include:

• Sharing works that are missing from a collection, whether they are in another museum, being restored, or on loan.
• Rebuilding archeological monuments – for example, making an entire temple appear where only an ancient column remains.
• Comparing works to previous versions, or to the artist’s preliminary sketches.
• Taking complex mechanisms apart and then putting them back together to see how they work. Examples include machines, the human body, the solar system, etc.
• Making digital characters move around in front of the visitor.
• Recreating the decor and atmosphere of an entire room in real size, by superimposing its former or future state on top of its current reality.
• Adding flesh and skin to the skeletons of extinct species, e.g. dinosaurs.
• Creating digital collaborative artwork.
• Creating a portal to access a virtual world.
• Creating a digital character guide to provide specialized tours for children, teens or other visitor groups.

AR Composer saves museums from having to build their own customized application and CMS (Content Management System) from scratch. That way, the majority of the budget can be allocated to creating content and designing compelling visitor experiences.

Try AR Composer now! We are offering AR Composer to museums and heritage sites in private Beta, with several demo assets included. If you are interested, please contact us.

With the addition of the full AR app, MUSE is offering visitors an experience that harmonizes perfectly with their innovative spirit – in total fluidity and transparency.

A Fascinating Content

The app has what it takes to please even the most directionally-challenged of us. The days of losing your way in exhibitions are over: with Go!Muse visitors can see their precise location on each of the six floors of the museum, and are guided in Augmented Reality toward the object of their interest.

As for the educational content, the app includes over one hundred AR hotspots that can be activated by the visitor to unlock multimedia content (text, videos and more) that provides additional context for the various attractions.

Last but not least, the app makes it possible to view realistic virtual portrayals of ten dinosaurs and a whale by adding life-size animated bodies to their skeletons in Augmented Reality, right in the middle of the museum.

An Emotional Experience

On the day of the launch, over one thousand visitors came with their families to check out the new app. It was the ideal opportunity to confirm that the app would be as magical as we had hoped. It was touching to see children, parents and grandparents equally surprised and enthralled by the illusion of a partially-digital reality. No matter their age, every visitor had the happy face of a child thrilled by an amusement park or magic show.

A Synergy of Multiple Talents

The obvious joy of the visitors made me proud of the work our team had accomplished. This project took no less than six months to complete and incorporated the work of at least a dozen individuals: an illustrator, a designer, a 3D animator, a game developer, front and back-end developers, a project lead and a program manager. The synergy of all these talents makes it clear that our field has changed. Cultural mediation is no longer about making a connection between historical and scientific facts, or simply coming up with a captivating narrative. It is also about creating an unforgettable emotional experience for every type of visitor.

To meet this objective, we collaborated with MUSE’s digital department. Fabio Manucci, the paleontologist of the museum, worked closely with Davide Bonadona, an illustrator celebrated for his work on dinosaurs, to create a scientifically accurate portrayal of each specimen.

From a scientific point of view, there is always a risk when you attempt to bring to life creatures that have been extinct for millions of years. But museums also have a responsibility to provide accurate representations of dinosaurs, in contrast to the fantastical portrayals in cinema and video games.

We then made it possible for these virtual animals to come to life inside the museum. Each 3D model was superimposed on its actual skeleton to within a centimeter of precision using GuidiGO AR Composer.

Seeing these dinosaurs in the flesh – and not just as bones – is impressive enough. But watching them move before your eyes makes the experience even more stunning. Since the skeletons themselves obviously remain static, we had to find a way to suspend disbelief when the animals are in motion. To do so, we added a Virtual Reality mode that hides the skeleton when the dinosaur begins to move.

A Large-Scale Experience

With the launch of Go!Muse, MUSE has become the first museum of Science in the world to offer an Augmented Reality experience at this scale, with ten animated dinosaurs and a whale as part of their permanent collection, and 80 AR-compatible tablets available to visitors.

MUSE official video trailer

While the app is designed for a wide range of users of all ages, it is sure to capture the imagination of young visitors most of all, due to their interest in novelties and interactivity.

Since they’ve been immersed in the digital world from birth, the younger generations are already used to learning and discovering outside of the traditional learning environments of libraries and museums. Touchscreens are already part of their daily lives. Virtual Reality headsets are soon to follow, and any museum planning for the future should be aware of that fact. It is essential for cultural institutions to pursue immersive, innovative technologies if they want to form lasting ties with the adults of tomorrow.

Experiences like this one will help young ones to discover the magic of museums and the joy of interacting with works of art and science along with other visitors.

What is occlusion?

Occlusion problems occur because virtual objects will always appear in front of the real world on the screen. For example, imagine that a virtual statue is placed in a gallery. If a person is standing behind the statue, the experience works because the virtual object appears in front of the person.

However, if the person is standing between the device and the virtual statue, the immersion is broken because the statue will cover the person on the screen.

Similarly, adding a virtual element inside a real object (such as adding a virtual apple inside a real bowl) can cause occlusion issues. The images below illustrate how the AR illusion works only for some angles. Again, the virtual object will always go in front of the real one.

So how can we solve occlusion problems?

Many occlusion problems for AR experiences in museums can be anticipated and avoided early on through design and forethought. Possible solutions to prevent occlusion issues include:

Create a designated space. A helpful way to control occlusion issues can be to contain where an AR experience will take place. For example, in the “Sites Eternels“ (Eternal Sites) application at the Grand Palais in Paris, the devices were physically tethered to a large table on which the AR experience appeared. This limited what could physically get between the device’s cameras and the table.

Another solution could be to physically mark the area where the AR experience will take place, so that other visitors know not to get in the way. Whether or not a museum employs this solution depends on the available space and whether the exhibition is temporary or permanent.

Utilize strategic positioning. The placement of virtual objects can help prevent occlusion problems, as some positions are more viable than others. For example, in the application we developed for the National Museum of Singapore, a 42-foot-long whale skeleton was positioned just under a high ceiling, well above visitors. In this case, because visitors point their devices up at the ceiling, the positioning of the whale ensures that there is nothing to cause occlusion issues.

Position your visitors in a specific place. In a recent project, we placed a virtual mark on the floor to encourage users to stand in a specific position before starting the experience. This way, we made sure nothing would get between the user and the AR scene.

Add masking effects. Environmental effects, such as clouds or mist, can help mask the real world elements that could cause occlusion problems.
For example, in another recent project, we animated a virtual spacecraft that lands on the surface of an asteroid. This experience took place in a small, crowded gallery, which caused occlusion issues. We added environmental effects to the virtual scene, and it helped to foster the sense of immersion in the experience.

Integrate real-world elements in the virtual world. In many cases, we can identify something called “planned occlusion”. For example, GuidiGO’s wayfinding capabilities display virtual blue dots on the floor that lead visitors to points of interest. In the case in the photo below, when the blue dots indicate to turn left, the rest of the path is not displayed until the visitor is in the doorway. This is possible because we integrate the pre-identified obstacles (walls, cases, etc) in our algorithm. The rendering happens in real-time and takes into account the physical world before displaying a digital object.

Position “occlusion masks”. Similarly, for a recent project, we wanted to display a virtual object along the length of a gallery wall. However, the room was filled with furniture that got between the visitor’s device and the wall, making the virtual object appear fake.
To work around this constraint, our programming team created “occlusion masks”. These were virtual 3D objects that were the exact same size and shape as the real-world objects, placed over the furniture using our AR Composer tool. In this way, we made it possible for the furniture to appear in front of the main virtual object.

Delineate museums into different sections. We can also separate museum spaces into unique galleries, so that when visitors are in a specific gallery, they only see the virtual objects on display in that gallery. This avoids the artworks of the galleries nearby being visible “through the walls” due to occlusion, as virtual objects will always appear in front of real objects on the screen. To read more about a project where we used this technique, check out our article about our work with the Clyfford Still Museum here.

Employ real-time programming solutions. Sometimes, occlusion problems cannot be avoided or planned for, such as visitors walking around inside the museum. There are programming solutions to these occlusion issues, in which the physical environment is reconstructed in real-time using device’s sensors and cameras to integrate the real world inside the virtual world.
The main downside of this option is that it requires a lot of processing power, however, this will certainly become a non-issue in the future.

Every AR experience requires custom solutions to specific occlusion problems. And the development of new sensors and more powerful devices will help facilitate new ways to work around these obstacles in the near future. Through creativity in design and programming, it is possible to find innovative solutions to make augmented reality even more real.

The use of augmented reality paintings in this exhibition is a creative solution to the fact that most of the selected virtual artworks would have been difficult, if not impossible, to borrow. Works by da Vinci and de Kooning are thus accessible in the galleries, as digital facsimiles.

Unlike traditional digital tours, this augmented reality experience doesn’t provide interpretation material about the artworks on display. Rather, the digital content is an integral part of the exhibition itself. The augmented reality assets become museum objects in their own right.

The Visitor Experience

The digital aspect of the exhibition experience is integrated simply and effectively into the visit. Prior to entering the galleries, visitors are equipped with Lenovo Phab 2 Pro devices. As they walk through the exhibition, which is spread out over the nine galleries of the Museum, they are notified about the presence of virtual artworks by a message on their devices.

Visitors are free to explore the galleries, finding the virtual artworks and comparing them to the physical ones. What is so unique about this experience is that the physical setting was designed with the digital components in mind right from the get go: there are blank spaces on the gallery walls with a text that lets visitors know that virtual works will appear when they look through their tablets. These walls are only populated when the virtual artworks are displayed in AR in real size. No markers, iBeacons, or WIFI are needed as the devices recognize the space using Google’s VPS (Visual Positioning Service) announced at Google I/O in May.

The high resolution digital reproductions allow visitors to explore the virtual paintings, discovering the vibrant details of texture and color, as if the paintings were physically in the gallery. Visitors can get extremely close to these virtual works (closer than would be allowed with real paintings), examining Van Gogh’s bold brushstrokes or Rauschenberg’s monochromatic topographies.

How the AR Experience Was Created

The creation of the digital component of the experience involved a collaboration between the Museum and GuidiGO to create high resolution 3D models of different paintings to be virtually hung on the gallery walls.

Google’s augmented reality team came to set up the VPS (Visual Positioning Service) by mapping the galleries. This allows the devices to recognize where they are in the museum space, which makes possible the placement and display of virtual objects. The Clyfford Still Museum is among the first 10 museums in the world to have embraced VPS technology to enable accurately prepositioned, life-size AR experiences.

After this step, GuidiGO’s team came on-site to place the virtual paintings on the walls using GuidiGO’s AR Composer, an application we developed to manipulate 3D objects and place them in mapped spaces. In the Clyfford Still Museum, this meant hanging the virtual paintings so they fit perfectly with the real artworks on the walls.

GuidiGO’s AR Composer was also used to delineate the museum space into unique sections. This step made it so that when visitors are in a gallery, they only see the artworks on display in that gallery. This avoids the artworks of the galleries nearby being visible “through the walls” due to the occlusion effect, as virtual objects will always appear in front of real objects on the screen.

The result is a seamless experience where visitors are engaged in a new kind of visit in which real and virtual complement each other. AR becomes social when groups of visitors see the same thing at the same time, sparking interesting dialogues and shared ‘wow’ moments.

The exhibition “Still & Art” is generously supported in part by Daniel Benel and Lena Fishman. Additional support is provided by U.S. Bank and the citizens who support the Scientific and Cultural Facilities District (SCFD). The augmented reality experience is made possible at the Clyfford Still Museum through a partnership with Google and media company RYOT.

© Featured Image courtesy Clyfford Still Museum

Lumin content was created by the DIA’s interpretive team leveraging GuidiGO’s augmented reality platform, AR Composer, built specifically for Tango. A pilot of the device took place January –April 2017. During the pilot evaluators conducted user testing and observations. Below are some of the results.

Visitor responses are taken from Lumin Short Survey Card (n=442) and Lumin Online Survey (n=110).

GuidiGO was presented with the Gold MUSE Award in the category Games and Virtual/Augmented Reality during a champagne reception on Sunday, May 7, 2017, at the 2017 American Alliance of Museums Annual Meeting in St. Louis, Missouri.

This award recognizes GuidiGO’s high achievement in the application of media and technology to Gallery, Library, Archive, and Museum (GLAM) programs.

Lumin content was created by the DIA’s interpretive team leveraging GuidiGO’s augmented reality platform, AR Composer, built specifically for Tango. This allows the user to interact with real size 3-D animations, information and display directions in the visitor’s field of view. (Learn more about Lumin.)

The MUSE Awards competition received more than 200 applications from a wide variety of institutions in North and South America, Europe, Australia, and Asia. This year’s entries included videos and films, interactive kiosks and installations, VR experiences, applications and APIs, digital communities, websites, audio tours, and more.

Over 90 GLAM professionals from across the globe participated as jurors in the process of reviewing and scoring entries. Winning programs were expected to demonstrate outstanding achievement in their content, interface, design, technical merit, innovation, utility, and appeal.

Now in its 28th year, the MUSE awards competition recognizes outstanding achievement in GLAM media and technology efforts. The competition is administrated by the American Alliance of Museums Media & Technology Professional Network.

For more information about the MUSE awards, please visit this page.

Singapore’s oldest museum now allows its visitors to experience the building through an enhanced architectural tour. It tells the story of the building’s long history and its transformation from a Natural History Museum to the most significant historical museum of Singapore.

Visitors are given a Tango-compatible Lenovo Phab 2 Pro device, which will transport them back in time and enable them to see the architectural evolution of the building, how it looked like over a century ago and onwards, and the artefacts that used to be displayed in it. The digital tour, led by volunteer guides, evokes the museum’s imagery in a way that helps visitors obtain deeper understanding of its contents.

There are currently six tour stops that incorporate a range of AR experiences, such as:

The Building’s Evolution

The evolution of a 130-year-old building is bound to be an interesting one. Combining Tango technology’s ability to both augment and recreate reality, visitors are able to see the replica of the museum’s transformation throughout the decades.

The Museum’s Rotunda

The old rotunda used to display a natural history collection, as well as a marble bust of Sir Stamford Raffles, the founder of Singapore. Through the lens of their device, visitors can walk around a life-size model of the room, surrounded with the original artefacts that once graced it, and see the clay tiles instead of the current marble flooring.

The Indian Fin Whale

This iconic 42-feet-long skeleton used to be displayed in the museum in the early 1900s, until it was returned to Malaysia in the 1970s. Thanks to Tango technology, visitors can once again see its fossil under the glass passage on the second floor, where it once stood. However, that’s not all – they can also witness the reimagining of how it looked and sounded like while it was still alive.

Lumin content was created by the DIA’s interpretive team leveraging GuidiGO’s new augmented reality platform, AR Composer, built specifically for Tango. This allows the user to interact with real size 3D animations, information and display directions in the visitor’s field of view.

Lumin connects visitors with artworks in a global context beyond the DIA,” said Salvador Salort-Pons, DIA director. “Augmented reality allows the user to see the unseen, imagine art in its original setting and understand how objects were used and experienced in people’s everyday lives. It is an exciting way to incorporate the latest technology into the visitor experience.

Visitors will hold a Lenovo Phab 2 Pro, the first Tango enabled phone, to explore different works of art. The initial phase of the project includes seven tour stops with a variety of AR experiences, including:

A Mummy, from Ancient Egypt

Mummies give us insights into ancient burial rituals. With the app’s AR overlay, you can peer beneath the sarcophagus and even the mummies’ bandages to explore an X-ray-like view of the skeleton.

The Ishtar Gate, from Ancient Babylonia

The Ishtar Gate once stood six stories high. Now, with the power of Tango’s motion tracking technology, you can explore the gate at scale, walk under its arch, and see where the Detroit Institute of Art’s 3×4-foot mosaic piece fits into this architectural wonder.

A limestone Relief, from Mesopotamia

The limestone relief was originally painted with vibrant colors, which have long since faded. By looking through your Tango enable device, you can tap on the relief to see what it would have looked like when it was first created thousands of years ago.

A Cylinder Seal, from Mesopotamia

Small round cylinder seals were once used to make impressions on clay for items like jewelry and signatures for administrative purposes. Now, instead of just seeing the seals in a case, you can roll the seals in AR to see the written characters and figurative scenes that they create.

WANT TO SEE LUMIN IN ACTION? CLICK HERE TO WATCH THE VIDEOS.

This is just the beginning of how you’ll be able to use Tango in museums to see more, hear more and learn more – in other words, to experience museums around the world in a whole new way.

“Lumin” in the Press:

• Hyperallergic: More Screens, More Knowledge: Testing the Detroit Institute of Arts’ New Augmented Reality App >>
• Fortune: Google Wants To Give Museum Visitors A Pokémon Go-Like Experience >>
• Venture Beat: Google opens Tango augmented reality platform to museums, starting with Detroit Institute of Arts >>
• The News Herald: Augmented reality adds dimension to art at DIA >>
• The Economist: Futurism: museums and technology >>
• Geek: Detroit Institute of Arts Enriches Museum Experience with Google Tango >>
• Tom’s Hardware: Google’s Project Tango Heads To The Detroit Institute Of Arts >>
• CNET: Google’s project Tango AR lets museum visitors delve deeper >>
• Tech Crunch: A Detroit art museum is leveraging Google Tango for an AR history lesson >>