If there was one future technology group that really stood out as being key at the MINExpo 2016 tradeshow (and I’m sorry to keep banging on about it, but the event really is a bellwether for industry direction), it was virtual (VR), augmented (AR) and mixed reality.

A number of the largest technology players had headsets at their booths that allowed miners and a handful of intrepid editors, such as myself, the chance to explore an underground mine or walk around and inspect a mining truck, all from the safety and comfort of Las Vegas convention centre. And, while this is just the tip of the application iceberg, it was just enough to rouse some serious interest.

Collectively known as ‘immersive virtuality’, high-end versions of AR and VR have been kicking around the technology sector for decades. However, thanks to a slightly more open mindset, lower-cost headsets and perhaps the entry of multi-industry players such as Dassault Systèmes, IBM and Hexagon into mining, miners are slowly starting to see the potential for thinking and working outside the box (or their mine).

But before we go any further, let’s get our definitions straight: in a February 2016 vlog (video blog) for Goldman Sach's Profiles in Innovation report, Heather Bellini, business unit leader for telecommunications, media and technology, explained the difference very well.

“VR is what we believe will be the next-generation computing platform along with AR,” she said. “With VR, you wear a head-mounted display and it completely blocks your field of view so you can’t see through it. With AR, it’s like wearing a pair of glasses which you can look through and see the environment around you, with a display highlighting whichever images it’s going to project to you.”

Mixed reality is therefore a combination of the two.

Goldman Sachs predicts that by 2025, the market for immersive virtuality products will approach US$80 billion, roughly the same as the desktop computing market today, and that it will transform the way we interact with virtually every industry, from both consumer and enterprise perspectives. 

While 2016 revenues are expected to come almost exclusively from hardware sales, by 2025, Goldman predicts a US$45 billion hardware and US$35 billion software split. The company expects 75% of spending will be focused on VR, with 25% going to AR; 54% of spending will be on consumer applications, with 46% spent on enterprise and public-sector applications.

What’s different about VR today versus 10 years ago is that the technology is starting to catch up. For example, processor speeds (Moore’s Law) have vastly increased, and graphics cards have become more powerful, which allows the correct number of frames per second to be displayed in order to prevent queasiness while wearing the device.

Price point, which has been a major barrier to adoption, is also starting to adjust. “As with any technology, we expect to see economies of scale improve over time, and we would expect price reductions to follow what we’ve seen with other hardware devices, as in the laptop or desktop markets,” said Bellini. “At the end of the day, we think they [VR and AR] will be as transformative as when we used to see people walking around New York city with a big brick, talking to people we couldn’t see at the other end.”

In July 2015, research firm Gartner classified both AR and VR on its annual Emerging Technologies hype cycle as having reached the “trough of disillusionment,” a dip that occurs when overhyped technologies don’t meet expectations. However negative this may sound, the trough actually indicates that technologies are poised to achieve real productivity once businesses gain enough experience with them. And, just a year later, when Gartner released the 2016 hype cycle, Oculus, HTC, Google, Samsung and others had used low-cost smart-phone technology to make head-mounted displays (HMDs) widely affordable. Gartner moved VR to its “slope of enlightenment,” reflecting the technology’s expanding practical applications; AR remained in the trough, but moved closer to the slope.

The next step on Gartner’s cycle is “the plateau of productivity,” which indicates when adoption rates of a technology will reach 20-30% of market potential. The firm estimates that it will take VR and AR just 5-10 years to become a routine part of everyday business, and judging by how much we’ve seen of both in mining over the past 12 months, I’d be willing to bet Gartner is correct.

Packing potential

Until relatively recently, VR was seen as the preserve of hard-core gamers, and the reason why it worked so well in that market (and why it will work well in this one) lies in the fact that immersive virtuality produces an emotional response; people remember their experience much more vividly than if they had just watched it on a screen in 2-D. It creates a new way to interact with information.

Imagine sitting in a classroom on a mine site while a trainer instructs you on how to inspect an excavator prior to your shift. Perhaps you’re new to the industry and don’t have much experience as an operator. However good the teacher, you were probably bored and lost concentration after 20 minutes of sitting still.

Now, imagine putting on a VR headset and standing in a virtual environment which is an exact replica of the mine you work in. You can walk around the machine with your instructor, perform the inspection step by step and, using handheld controllers, point at things you don’t recognise or understand. Information pops up in front of your eyes with key points on what to look for.

Which of those experiences are you more likely to remember and find useful?  

“The use of VR calls for changes in the way a company works, and changing working habits and processes takes time,” said Pascal Therond, co-founder of Kalista, during an interview with Dassault Sysèmes’ Compass magazine.

“A lot of organisations still work in separate silos, whereas VR promotes a collaborative approach. Collaboration requires a change of culture, and this takes even more time than a mere change of organisation. So the earlier you start, the better it is.

“Beyond that, we’re still in the early stages of VR and AR, so of course there’s a lot to be gained in a test and-learn approach to remain ahead of the competitors and potential disrupters on the learning curve.”

And of course, immersive virtuality doesn’t just hold potential in employee training and development.

Paul Snowdon, senior manager for mining at consulting firm Accenture, hit the nail on the head in a recent interview. “AR enables you to push a lot more real-time information to those that need it, so basically pushing the right information to the right people at the right time," he said.

“That information might be to help them understand more about what they are looking at, to help them make a better decision through their operations or, more importantly, help them get out of an emergency situation very safely, quickly and with a lot of support coming in through that mechanism.”

Research background

While the mining industry may seem like a newcomer to immersive virtuality, researchers have been looking at ways to leverage the technology for nearly 10 years.

At the second International Future Mining Conference in 2011, CSIRO researchers outlined the development of ReMoTe, an AR system for remote guiding using a touch-table interface and a combination of AR and a wearable operator system. The aim of the technology was to provide expert guidance to operators and technicians from a remote location, thus improving operator safety and boosting productivity through greater uptime.

The team also developed Mobile ReMote, a fully mobile remote guiding system where both the expert and operator use a near-eye display and wear a light computer in a back pack. The expert is able to guide the operator using his/her hands on the virtual display, while the operator is able to see the instructions on the rear eye display.

VR has also proven useful in block-cave mining. At a 2014 AusIMM conference, researchers explained how they applied VR and scientific visualisation (VRSV) technology to analyse complex, multi-dimensional data sets gathered in a block-caving environment and visualise how the system would react to changes in certain variables. A software module was developed at the University of New South Wales in Australia for use in the advanced visualisation interactive environment and was subsequently applied at Newcrest Mining’s Ridgeway Deeps operation with the aim of improving the understanding of the operation’s geomechanics.

In Europe, meanwhile, as part of Horizon 2020, the EMIMSAR project employed immersive virtuality to provide underground equipment operators with augmented versions of their machines using handheld computers and helmet-mounted displays. The operator can move through images of the machine and its components, call up data on the history and present condition of the machine, or be guided through the repair or maintenance of components.

Sensors on the machinery provide insights into the wear on heavy-duty components such as gears and chains. For instance, temperatures and rates of acceleration are recorded and analysed. Sound sensors sample noise from the sprockets on mining plows and flexible conveyors – alterations in noise during their operation indicate their level of wear.

The sensor data is fed into a knowledge-based maintenance system, where it is combined with background data on components and libraries of machine and component visualisations. That information is combined to create information-rich, real-time visualisations of the machinery that can be viewed by the miners as they work on the machine. There are also voice-overs that provide step-by-step instructions on maintenance procedures.

Back at the mine’s control centre, managers can combine the real-time sensor data with information on resources, staff skills and capabilities to improve maintenance planning.

The EMIMSAR system proved so successful that it was deployed by German coal miner RAG. RAG and EMIMSAR worked closely with German engineering consultancy DMT and the Polish Mining Automation Institute, KOMAG, on the development of the system, and it is now used for maintenance planning on longwall equipment, belt conveyors and loaders.

Interest in immersive virtuality doesn’t appear to be confined to specific markets either; the trend seems to be global. In August 2015, the University of Pretoria, in conjunction with Kumba Iron Ore (part of Anglo American) opened the US$1.3 million Kumba

Virtual Reality Centre for mine design. The centre houses an all-in-one 3-D stereoscopic, immersive visualisation platform comprising a lecture hall, 3-D stereoscopic theatre and 3-D 360° cylinder. It also offers access to virtual reality simulation products and modules representing a range of existing environments and practical scenarios.

The centre aims to improve mining education at the university using different packages (such as hazard-awareness and mine design) which were co-developed with Pretoria-based VR simulation company Simulated Training Solutions.

Anglo American also has its Centre for Experiential Learning in Crown Mines, Gauteng, where it uses VR technology to train its staff. The CEL is a dedicated in-house facility, supporting operating-model skills development. It is targeted at managerial and supervisory levels, and is operated as a professional adult learning institute, where teams learn to apply operating-model tools using a combination of VR, modelling and real-time scenarios.

Bringing mining to the masses In May 2014, Rio Tinto Diamonds partnered with New York-based Bravo Media to provide an interactive virtual journey into its Diavik diamond mine in the Northwest Territories of Canada using Oculus Rift technology. The Oculus Rift VR headset provides an immersive, computer-generated environment that allows the user to ‘fly above the coast of Canada and land at the Diavik Diamond mine’ before the user is taken on an interactive journey down the mine. Brandee Dallow, manager of Rio Tinto Diamonds North American representative office, said at the time: "The interactive technology brings to life the mining experience in a very compelling way. Our goal in showcasing Oculus Rift at the JCK [jewellery] Show is to start the dialogue with our customers and other members of the diamond jewellery industry about the new approaches available to better capture the next-generation consumer."

Technological developments

On the technology front, AR specialist Scope AR teamed up with Caterpillar in late 2016 to provide an AR-based live support video calling platform built specifically for the unique remote-assistance needs of industries using heavy machinery. The idea was to support the Caterpillar dealer network in repairing, troubleshooting a problem or conducting maintenance on equipment; something that would be ideal in the mining industry.

"This is about delivering an entirely new, interactive interface for remote workers to more effectively and efficiently communicate with experts in real-time, wherever they are," said Scott Montgomerie, CEO and co-founder of Scope AR. "With machinery and equipment becoming increasingly complex, we are excited to partner with Caterpillar to provide the technology tools they need to share expert knowledge more quickly and deliver the remote support needed to get equipment back up and running faster."

Technology firms HiScene, Inuitive and Heptagon also teamed last year up to launch HiAR Glasses, HiScene’s next generation of AR glasses.
The companies worked together to develop a complete solution for advanced 3-D depth sensing, AR and VR applications. The glasses incorporate Inuitive’s NU3000 Computer Vision Processor and Heptagon’s advanced illumination technologies to provide high performance even in changing light conditions and outdoors.

As mentioned earlier, I had the privilege to test Dassault Systèmes’ Virtual Mine package at MINExpo 2016. The product, which went on to win Mining Magazine’s 2016 Software Award, provides a virtual platform for modelling and design, simulation, execution and monitoring, and collaboration of every aspect of a mining operation. It can be used at all levels from operator to C-suite and enables ideas to be explored and improved upon in the virtual world that would be impractical or cost-prohibitive to try out in the real one.

For example, you can walk through a virtual version of a mine site before construction, while at the same time reviewing the economic impacts of changes to the configuration of equipment, the plant, or locations of stockpiles or haul roads. Slightly more eccentric ideas such as the use of blimps in lieu of building a railway network to connect the mine to a port could also be explored. On a smaller scale, engineers can use the virtual world to test the assembly of a modular plant before it is fabricated and shipped to site.

Managers can test how different mine plans will fare under changing commodity process, and can gather data on equipment performance with the objective of improving the efficiency a global fleet by, say, 10%. The possibilities are endless.

Competitor Hexagon's Manufacturing & Intelligence unit, meanwhile, signed a US$834 million deal in February to acquire MSC Software, a provider of computer-aided engineering solutions, including simulation software for virtual product and manufacturing process development.

The move strengthens Hexagon Mining’s ability to connect the traditionally separate stages of design and production – integrating real-world data generated on the production floor with simulation data to further improve a customer’s ability to reveal and correct design limitations and production problems prior to manufacturing.

The move also brings it a step closer to providing a virtual environment for the design and testing of mine equipment and operations.

“MSC represents a game changer in our mission to deliver actionable manufacturing intelligence, taking us another step closer to realising our smart connected factory vision in discrete manufacturing industries such as automotive and aerospace,” said Hexagon president and CEO Ola Rollén. “We can now leverage the data our MI division is generating to improve design choices and processes upstream in the workflow. The acquisition will also open up new markets and touch points for MSC via our PPM division.”

Gamification

In April 2016, Codelco announced a collaboration with Microsoft on a contest to see who could design the best Chuquicamata underground mine using the online game Minecraft. The company is spending US$3.83 billion converting Chuquicamata from surface mining to underground by the end of 2019.

Oscar Landerretche, chairman of the world’s biggest copper producer, said the event could expose “thousands of children and young Chileans” to the mining industry. And the general manager of Microsoft Chile, Oliver Flögel, added: “Today more than ever we need to engage new generations in the importance of the mining industry in the country’s development”.

Microsoft bought Swedish programmer Markus Persson’s ‘sandbox video game’ – developed and published by Mojang – for a cool US$2.5 billion in 2014. It is said to have more than 100 million registered users who play Minecraft on any game console, mobile devices and Windows.

Microsoft and Mojang have continued working together on a version of Minecraft that will use Microsoft’s HoloLens VR headset to cast the user into an immersive game environment.

Grey Properjohn, previously software development manager at Snowden and now working at Vix Technology, says HoloLens takes “the maligned Google Glass project, combines it with the coolness of the Oculus Rift, the technology from Microsoft’s Kinect motion sensor, and a new holographic processing unit to provide a fascinating glimpse into the future of AR – or mixed reality in Microsoft parlance”.

“Early prototypes of the HoloLens are impressive,” he says. “It seamlessly blends the virtual 3-D content with the physical world, allows users to view, control and interact with the content using their hands and voice. With examples ranging from building Minecraft worlds on walls and table tops, to NASA’s Sidekick project that sees the adoption of the HoloLens in the International Space Station, providing astronauts with virtual assistance from earth, the future of mixed reality looks to be just as bright in the hands of teenagers as with professionals in the office – or space, as it were.”

What impact could the HoloLens technology have on the mining industry and on those involved in providing innovative solutions to its challenges?

Properjohn wrote in a newsletter that it was “clear to see that the ability to mix virtual content with reality will provide countless opportunities to improve operational safety and efficiency, as well as bringing the corporate offices even closer to the operations on the ground by connecting people in ways never imagined before”.

He said: “Imagine, for example, standing in an underground development heading and becoming visually aware of all the adjacent headings, stopes and declines in the immediate vicinity, or being able to virtualise the trends in ground features such as fault-zones or stress-zones ahead of the current development heading.

“Extend this to drilling from underground and you can virtually eliminate the potential to inadvertently intersect other existing openings.

“Application to open pits also sees similar benefits. The assessment of mining an open pit through underground workings takes on new meaning if you could actually ‘see’ the workings. The safety implications of such vision are clear. Indeed, even from an office environment, users can be fully immersed in the underground operation, tracking the locations of all people and assets within the mine in real-time.”

Properjohn said the technology would provide new opportunities for resource geologists and engineers to ground-truth their models and plans – literally by standing on a bench in a pit and visualising the block model beneath their feet to see the latest grade-control drilling results.
“Or, take the existing world of the traditional borehole logs and core image analysis,” he wrote.

“Rather than simply viewing a 2-D representation of the logs – or even a 3-D view – on a monitor or paper, think about being able to physically ‘lift’ the core from the ground and see the geology, geophysics and imagery overlaid on a 3-D representation of the hole in its physical location, with the relative geology of all other boreholes in the vicinity.

“The potential is amazing and set to revolutionise, if not redefine, current thinking.

“One interesting aspect of the HoloLens is its acute awareness of the physical world around it. Imagine looking at a stockpile, not only could you visualise its genealogy on top of it but with HoloLens’ Spatial Mapping, it could generate a mesh over the stockpile and undertake a survey in real-time, by simply walking around and looking at it.”

Away from the mine itself, remote operating centres are already becoming a reality and demonstrate significant efficiencies to operations.
“Imagine enhancing the efficiency by being able to communicate with people on site, not only using voice – using Skype’s built in real-time translations so they don’t even need to speak the same language – but actually seeing what they’re seeing on site and interacting with them as they undertake their work,” Properjohn said.

“This could be as simple as providing geological assistance when logging core samples or assisting with specific maintenance operations. Microsoft’s proof of concept provides remote assistance in installing a light switch.

“The opportunities are limitless considering the remoteness of the operations we’re used to dealing with.”

Properjohn said it was clear that with the impressive list of corporate partners already signed up with Microsoft on the HoloLens project, including the likes of Trimble, Autodesk and NASA, that the technology was destined to be “much more than a gimmicky fad headed for the technological junk pile”.

“If the early prototypes are anything to go by, this could be the one technology that inspires not just us propeller heads to stand up and take notice but the engineers and geologists of the world as well,” he said.

And perhaps now the next generation of industry professionals, too.