A 3D design and visualization tool with phenomenal potential in the AEC world is coming from a collaboration between Microsoft and Trimble.
Imagine designing an object, in 3D and color, using only gaze, gestures, and voice (GGV) commands. Imagine designing an entire building the same way—or modifying an existing one. Imagine moving in a physical space and overlaying tags, menus, applications, and virtual reality objects on the physical objects around you. Imagine studying the surface of Mars by immersing yourself in a 360-degree, 3D image of it, created from imagery taken both above the surface and on it.
All this is now possible with HoloLens, a wearable holographic computer with see-through, high-definition lenses that Microsoft is preparing to release. It incorporates a CPU, a GPU, a new processor that Microsoft calls a holographic processing unit (HPU), spatial sound, and sensors that map the physical environment. The uses of this new device are endless.
HoloLens blends physical reality with AR and VR to create MR. What does this mean? Physical reality (say, an air conditioner) blends with augmented reality (AR, when the filter was last changed in that unit) and virtual reality (VR, a bracket you designed to attach that unit to a particular window), creating what Microsoft calls mixed reality (MR).
Blending Realities
MR is such a new concept that trying to describe it in words is akin to trying to describe a flavor to someone who has never tasted it. Even the distinction between physical reality and virtual reality is not accurate, because data—such as 3D images—is also a physical reality in a computer’s circuits.
HoloLens, explains Adrian Ferrier, a civil engineer working for Trimble, covers a spectrum from VR, “which is all about the imagination and being immersed in something that does not exist,” to “an MR component, where the world really matters, because you are aligning information to objects in the world.”
“In the context of the building industry,” explains Aviad Almagor, an architect at Trimble, “MR is the phase in which digital and real content co-exist, where architectural design collides with reality, and where construction teams transform digital content into physical objects. We believe that by bridging this gap that exists today, we can bring efficiency and transparency to our customers’ projects.”
When a designer—say, an architect—is imagining something, it doesn’t exist in the context yet, Ferrier points out. “So, they are free to invent, understand, and visualize. They are naturally thinking about 3D forms, so there is only so far you can go with a flat screen.” Likewise, they need 3D tools to communicate their vision to stakeholders or owners. Prior to an actual construction project, however, all of that is still in the imagination.
HoloLens users can see the position of other users who are either physically or virtually near them.
This is when the HoloLens overlays the architect’s vision onto the actual construction site, creating an MR in the user’s imagination, Ferrier explains.
The next step in the process is construction, where the focus shifts from imagination to instructions for efficient assembly, which HoloLens can display.
Finally, once the building is completed, much of the data collected during design and construction can be used for facility management. This is when, Ferrier points out, HoloLens could help workers navigate and assess needs, say, by reading model numbers and maintenance schedules for components, which are spatially located. “For example, if I see three A/C units in front of me, which one is the one that fails most often?”
The moment when VR and physical space overlap perfectly is when a 3D model is registered correctly to the physical space by aligning a digital monument to a physical monument.
Trimble and Microsoft Collaborate
Adrian Ferrier is an engineer who has been working on virtual reality since the 1990s.
Civil engineer Adrian Ferrier began working on VR in the 1990s as a student at Georgia Tech. In graduate school with colleagues, he built Interactive Construction Visualizer that showed surfaces and vehicles and was interactive and immersive, at least in the crude ways in which it could be back then on Silicon Graphics machines. Later, he worked on interactive visualization and 3D for engineering in various industries.
In the late 90s, Ferrier worked for Spectra Precision Software on a product called Terravista, which was a 3D visualization of land surfaces from the engineering product Terramodel. Since 2007, he has continued his work on 3D visualization at Trimble in a subgroup called XYZ, which he runs. It is focused on special problems to solve within the company, typically having to do with 3D visualization, optimization, or simulation.
Architect Aviad Almagor joined Trimble a couple of years ago when it acquired the company in which he had been a director of product design, Vico Software. They made a product that integrates 3D models with cost and schedule data for the construction industry. By picking a line item in their costs list, for example, users can highlight all the related items in a 3D model of a building. Since the acquisition, he has been working in Trimble’s design, build, and operate group to develop a similar process of integrating the different stages of the construction process.
In the past two years, the group has been researching AR and VR technologies and testing the relevant devices on the market. “We developed tools for Google Glass and for Google’s Project Tango,” says Almagor. “We tested VR solutions, such as Oculus Rift and Samsung’s Gear VR. As part of this process, we built very good relations with Microsoft, testing their Kinect sensor, exploring how depth camera technology can help us sense the environment.”
Aviad Almagor is an architect who has been working on 3D modeling of buildings. Almagor and Ferrier worked together with their team to develop AEC-related applications for HoloLens.
As part of this research process, Microsoft invited Ferrier and Almagor to see emerging technologies that were not yet available to the public but were applicable to the type of work they were doing. They went to Microsoft’s campus in Redmond, Washington, and met with Alex Kipman, the principal creator of Microsoft’s Kinect motion controller and the leader of the HoloLens team. He explained his vision and demonstrated the hardware. Trimble’s team then worked side by side with the Microsoft HoloLens team to produce a proof of concept. Trimble formally created an MR program group and made Almagor its director.
Almagor’s group tested HoloLens to support Trimble Sketchup (a desktop 3D modeling application to design buildings), Trimble Connect (a Web collaboration environment project), and V10 (a 360-degree, survey-quality, imaging device). If they could prove that it could work with these three devices, he explains, they could apply it to other tools in the company’s portfolio.
Who Does What
Microsoft is the owner, developer, and manufacturer of the HoloLens hardware and its software development kit and plans to use it for such markets as education, medicine, and marketing. It’s a Windows 10 device; therefore, any program developed for Windows 10 will work with HoloLens.
Trimble’s part is software development, to extend the capabilities of the existing solution by adding holographic technology. “I expect that other vendors like Trimble from multiple industries will work with HoloLens to develop applications for the device,” says Almagor. “Trimble has a unique position here, because we have the software for design and analysis that can be the source of information to users of HoloLens, and we also have a hardware solution—specifically, positioning devices—that can help improve positioning for HoloLens users.”
HoloLens for AEC
“You can use HoloLens to understand where you are in a space relative to other objects around you, or you can use HoloLens together with positioning devices and get very accurate absolute positioning, which is required for some engineering tasks,” says Almagor.
To test HoloLens, his team followed a design-construction process. First, they asked Foster & Partners, a large UK architectural firm, to send an architect to work with them on the project. They created a physical model of an empty site in downtown Denver and asked the architect to design a community center on it, using Sketchup and HoloLens.
The architect, Almagor explains, “created some 3D models, put the HoloLens on, looked around, and experienced the design in a table-top MR view—to understand the spatial relations between his design and the physical model of the site—or in immersive mode to experience the design from the point of view of pedestrians on the street.”
At some point in this process, they added imagery collected with a V10 to enable the architect, who was never actually in Denver, to make decisions about the materials for the project. After several iterations, once he was satisfied with the design they added a Trimble Connect Web-based collaboration environment and invited additional team members, such as a structural engineer, to collaborate using holographic technology.
“This is one of the strongest capabilities of this technology,” says Almagor. “I have a hologram on my table, and you—on the other side of the continent—can have the same hologram on your table, and we can both see it, each of us from his point of view, and I can also see where you’re standing and what you’re looking at. When I’m looking at something and I’m talking about it, you see the objects highlighted or you see a ray coming from my eye and touching this specific object. It’s a fantastic way for remote collaboration.”
Next, they took the 3D model created for the design and projected it onto the real site. Because the project was about adding an annex to an existing building, they were virtually standing in a room of the existing building, and the HoloLens enabled them to see what the new opening in a wall of the existing site would look like and what one would see through it in the new annex.
This process could be used to convince the stakeholders about the value of the design and show them how it would function once implemented. “I actually demonstrated this to a customer in our Atlanta office,” Almagor recalls. “He was just 1 cm from a solid wall and looking through a virtual opening to see what was behind it.”
The MR program is working with other Trimble divisions—such as mining, geospatial, and architecture—to deliver MR solutions to their respective markets. “The idea is that we will be able to create a common core technology,” says Almagor. “HoloLens is very strong in 3D visualization, and its visual aspect is a valuable tool when you want to communicate data and explain complex situations.”
Almagor and Ferrier give several examples of HoloLens’ possible applications:
An architect could use HoloLens to explore a design and then convince an owner about it. “I already have the data and a 3D model,” Almagor says, “Why can’t I just take it out of the screen and put it on the table and walk around it and move between a table-top view and an immersive view, to better understand the experience?”
Before digging for a new project, an engineer could use HoloLens to look down and “see” the existing underground piping and infrastructure, captured in a 3D GIS model, so as not to damage it.
HoloLens’ ability to create a digital model of the surrounding surface could enable a facility engineer to look at an air conditioner or a projector, see virtual tags on them, pick one, open a menu, and see that unit’s maintenance history or manual or a video of how to replace parts.
Mechanical installers could use HoloLens to see a building information modeling (BIM) system rather than carrying a paper plan or an electronic version on a tablet. “They don’t have to make that transformation—from a plan to the actual environment—in their head,” Ferrier explains. “The opportunity to use an MR projection into the environment while they are working is key to improving their production work flow.”
HoloLens could enable mining engineers to visualize and analyze data from lidar and other sensors that are constantly mapping and monitoring a mine, make design decisions, and communicate them remotely to other team members, without having to actually visit the mine.
HoloLens’s positional and environmental sensors could help navigate construction workers to positions inside buildings or away from danger, such as a crane with a heavy load overhead.
The Future
If you use HoloLens with positioning devices, you get accurate absolute positioning. (Used with permission from Microsoft.)
Ferrier predicts that as HoloLens takes hold, other devices around it—such as safety glasses and other safety protection equipment—will start to change so as to fit more efficiently with it. Likewise, software originally written for tablets and PCs that have traditional mouse- or touchpad-based inputs will change to dovetail with GGV inputs. Additionally, machine-readable codes, such as bar codes and QR codes, will routinely provide environmental clues.
Because HoloLens is produced by Microsoft, which sells to both the business and consumer markets, its use is likely to extend the professional tool set down to consumer activities on a personal scale—for example, to design objects that consumers can then make with a 3D printer.
“I do believe that augmented reality is going to change quite a lot the way in which we—in general, not just engineers—consume, interact with, and communicate data,” says Almagor.
“There are moments when we realize that you can do certain things only in this technology and you could never have done that before, and this would be truly transformative,” Ferrier concludes.