Augmented Reality (AR) and Virtual Reality (VR) are both part of a wider field of technology called Mixed Reality (MR). MR describes different technologies that can blend the physical world with the digital world. VR replaces the real-world environment with a simulated environment, where computer simulations influence the user’s senses and perception through the production of images, sound and other sensations. The user can look and move around the artificial environment and interact with its virtual features. Similarly, AR is a blend of the real world and the digital world, which is achieved by overlaying virtual computer-generated objects onto real-world objects. Where AR alters the user’s ongoing perception of a real-world environment, VR completely replaces the user's real-world environment with a simulated one.
Several devices are used in the field of MR including head mounted displays (HMD), AR tablets, Cave Automatic Virtual Environments (CAVE), and AR projectors. These devices provide MR visualization and other functionalities for user interaction. In addition, tracked controllers, object recognition and tracking, or haptic feedback can be integrated to support intuitive usability. MR therefore provides an enhanced interface between digital data and users, due to improved visualization and controls.
MR technologies have several potential uses across industries. These include:
- Simulating environments
- Creating immersive experiences
- Visualizing and interacting with data
- Enabling maintenance
- Enhancing education
- Improving public health and safety, and
- Supporting investigations and case management.
MR technologies are well suited to support organizations across the Project Delivery, Operations and Maintenance stages of infrastructure delivery. They have been successfully used to deploy training programs and can assist workers in performing asset inspections and maintenance works. As VR creates an entirely simulated environment, it can be used to place a user in an immersive training environment, where they can experience real-world situations without any of the associated safety risks. AR, conversely, is better suited to building on the job skills and assisting workers to complete tasks. It does this by overlaying digital information on to real-world assets. For example, the system could project the maintenance history record and relevant maintenance manuals for an asset, whilst also displaying information regarding on-site parts, so that a worker can perform the task with minimal distraction. These technologies can also be used to connect an on-site worker with a skilled off-site worker, who can support them in undertaking the task (see Knowledge Access Platforms for Construction and Maintenance use case).
These project stages have high safety risks for workers. Workers with limited on-site safety knowledge or a lack of safety awareness and/or training, can further contribute to these high-risk environments. Traditional construction safety training takes place in a classroom style setting using presentations or videos. This setting is limited in its ability to represent real site conditions and prepare workers for on-site hazards. VR is instead able to place each worker in a virtual replica of the site and effectively familiarize them with existing and potential hazards in advance of them stepping on site.
MR technologies can also be used to assist in performing inspections. Traditional inspection processes are timely and require one or multiple resources to perform a detailed on-site inspection. This requires specialist resources to be available on-site, often across multiple days, and has the potential for errors due its manual nature. Documenting and communicating findings is also a timely process. AR can be used during both the construction and maintenance phases to enable inspectors to compare a site or asset with the original designs/BIM model, perform quick measurements, add notes, and document findings in real-time. This can be done by an on-site inspector using HMD or Connected Glasses, or by an off-site inspector utilizing drone technology to facilitate a virtual walk-through of the site or asset. This is particularly useful if the inspector cannot be on-site, or if the safety risk is high (see Knowledge Access Platforms for Construction and Maintenance use case).
Organizations that have implemented MR technologies have experienced reductions in the time required to perform tasks and train staff. The Aveva Group, who work with companies including the Kuwait Oil Company in the Middle East utilizing MR systems for staff training, have seen a 30% to 40% reduction in time required for training and a resultant reduction in the cost to perform maintenance by as much as 3% per year. In the rail industry, MR technologies have produced a 30% time saving and a 30% to 50% cost saving. The technologies have also benefitted employee’s satisfaction and productivity levels. Additionally, the tracking of information is more accurate and better documented, resulting in infrastructure being better maintained and utilized.
MR technologies can provide insights into preventative measures, the consequences of which extend further and further into the future, such as enabling pre-emptive maintenance. They can also be used to build construction and Building Information Models (BIM) directly from the images captured (see Virtual and Augmented Reality for Planning and Design use case).
Improving efficiency and reducing costs:
- Reduce time and OPEX costs through increased efficiency and speed in completing tasks. For example a Skylight trial showed that eliminating the seemingly minor inconvenience of glancing back to a book for information resulted in a 34% increase in speed to accomplish a task.
- Improve productivity by accelerating and enhancing experiential learning in comparison to traditional classroom-based learning and reduce errors, resulting in time savings, safer procedures and lower operational costs
- Reduce fixed costs, such as those related to the purchase of various items, as virtual objects can replace physical ones and enable remote training and inspections thereby saving time and money associated with travel costs and trainer fees
Enhancing economic, social and environmental value:
- Prepare workers for stressful or hazardous circumstances without putting them at risk and reduce the number of on-site accidents and injuries through effective training
- Improve employee satisfaction through effective training, mentorship and on the job support and by enabling workers to begin work quickly due to speed of training and begin utilizing new applications with little prior information
POLICY TOOLS AND LEVERS
Legislation and regulation: Governments will be responsible for establishing certifications of the VR/AR content information and operations, as well as regulating the appropriate and safe use of VR/AR systems.
Effective institutions: Specific training organisations and accredited trainers will need to be put in place in order to upskill staff in the use of such devices. Therefore, accreditations should be delivered by specific institutions, following standardised validation and accreditation processes.
Transition of workforce capabilities: There is a need to transition worker skills from a site-based skillset to a remote operations based-skillset, supported using new technologies. Furthermore, human factors processes will need to be set up.
Procurement and contract management: Performance specifications and required experience and expertise in procurement processes will need to be revised to consider those new services. Contract and expected contractors Key Performance Indicators regimes will need to reflect the new performance expectations.
Funding and financing: Investment in VR devices and licensing costs for the analytics of data provided to and from the devices will need to be included in the investment costs and funding models of such services for training and inspection, in addition to consulting services.
RISKS AND MITIGATIONS
Risk: For businesses to access the benefits associated with the processing of large amounts of data, AR and VR systems typically need to be integrated with other company software. In practice, this often means integrating with a multitude of legacy systems which can be difficult and may pose many issues. Furthermore, it can be difficult to coordinate legacy simulators to produce an accurate, complete training experience, and systems can be unintentionally influenced by human bias.
Mitigation: Attention should be given to ensure multiple simulations can operate in the same virtual space. Organizations must ensure their algorithms continue to produce reliable results each time new data is added, in order to understand if additional human layers add biases. They must act to resolve this when inconsistencies are discovered.
Risk: For emerging technologies like AR and VR, getting stakeholder buy-in can be challenging either as a result of a lack of understanding of the benefits; conflicting stakeholder priorities; or miscommunication about the goals of the technologies. There is also a potential for information overload which can cause stress, indecisiveness and can lead to inaction. This defeats MR’s purpose of enabling quick action using real-time information.
Mitigation: It is important to ensure all stakeholders are aligned in their expectations of the technology prior to its implementation. Furthermore, the amount of data that can be accessed through MR applications should be regulated to secured authorized data for organizations seeking to apply this technology.
Safety and (Cyber)security risk
Risk: Like other connected technologies, AR and VR are vulnerable to security threats and unauthorized access by hackers and malware. These attacks can result in a denial of service or the overlaying of incorrect information. Any errors in safety or cybersecurity that could potentially harm customers or employees can significantly impact businesses by destroying trust, brand, reputation and future prospects.
These technologies also present their own unique privacy challenges in how the technologies interact with users, and in the types of data they interpret. They actively assess how users react to new experiences including eye movements, pupil dilation, and other reactions. These are used to generate data points to improve future programs. It has been suggested that such information could be utilized by hackers for identity theft.
Mitigation: It is importance for any major information system implementation that an assessment of the system’s security strengths and weaknesses is conducted to ensure the sensitive data of the users is protected. This type of data collection needs effective controls to protect the privacy of users.
Example: Sydney Metro AR Safety Induction
Implementation: An immersive safety induction program for the Sydney Metro project rolled out to more than 500 users.
Timeframe: The development of the AR training took 10 months to complete. This included user design development, filming, animation, scripting, solution building and testing, user feedback, final build, training to deploy solution and final deployment.
Example: University of Cambridge Trial
Implementation: Use of headsets to perform inspections on a bridge. High-resolution photos are mapped to 3D models and using a HoloLens connected to the cloud, the user can zoom in and out, rotate, and move around the structure from anywhere in the world.
Cost: The solution enables more accurate diagnoses of structural issues, which resulted in fewer large-scale repairs, less downtime, and reduced traffic delays and congestion.
Timeframe: The trial took place in 2017.
Example: BP Virtual Reality Training
Implementation: BP collaborated with Maersk Training to deliver VR training to their offshore drilling teams.
Cost: The training enabled BP to safely complete a drilling task 40% under budget.
Timeframe: The training enabled BP to safely complete a drilling task 114 days ahead of schedule.