A comprehensive review of augmented reality-based instruction in manual assembly, training and repair

Inspired by Ref. [1], this article focuses on the latest research progress of AR-based assembly instruction in visual representation, interaction method, user cognition and ergonomics. These research advances explain in detail the role of AR display technology in promoting industrial intelligence. In detail, the visual representation in human-computer interaction provides a detailed research basis for the design of assembly instructions, and further illustrates the design characteristics of AR instructions. In a sense, AR instructions are a series of visual information reflecting product assembly intention [2]. So as to guide users to assemble all parts into high value-added products [3]. This process requires the user’s cognition, perception and motor skills, with the focus on sensorimotor function [4]. Nowadays, based on the highly competitive industrial environment, it is required to introduce innovative products into the market in a short time. This also requires a collaborative manufacturing environment that requires stakeholders to exchange real-time information during the product development life cycle [5].

For the assembly of complex products, the description of assembly information in physical tasks is an important factor to ensure the smooth progress of assembly activities [6]. In order to guide manual assembly, the AR assembly system must communicate effectively with users through AR instructions. These instructions need to correctly explain the core intent of the assembly task to users [7]. At the same time, the role of assembly instructions has quietly changed from simply providing assembly information to actively providing heuristic visual clues to meet users’ cognitive needs [8]. In fact, the importance of communication and cooperation between AR instruction and users for effective assembly is embodied in the fact that it is more efficient than ordinary operations.

It should be noted that because of the fundamental disconnection between digital assembly information and physical tasks, it can be concluded that this is the gap between the digital world and the real world, which limits our ability to perceive and understand information semantics [9]. Augmented Reality (AR) [10] is a potential human-computer interaction technology. It embeds assembly information in the cyber physical space in the form of virtual objects [11] to coexist and interact with assembly parts in the real world, which greatly enriches the way users experience the real world [12]. It must be admitted that this technology has changed the way individuals interact with the real world in the field of assembly. This change has well supported the concept of intelligent manufacturing [13], and ultimately led to a revolutionary change in the corporate competition model [14].

In recent years, AR assembly has changed from a stepwise guidance model [15] to a content-aware guidance model [16]. The latest information-level AR assembly [17] can introduce assembly intentions into the assembly workshop through a visual interface. In addition, thanks to the unremitting efforts of the NUS AR Group, Computational Mechanics Center of Excellence, Columbia Computer Graphics and User Interface Laboratory, and Empathic Computing Lab in the field of AR assembly, this new technology has been successfully applied in design [18], manufacturing [19], assembly [20], maintenance and repair [21], medicine [22], and education [23].

AR assembly depends on AR instructions, which are visual information that describes part features, assembly relationships, assembly process and assembly level of objects to be assembled. The implementation of the AR directive is to solve a series of problems in the assembly stage of the product’s entire life cycle, such as assembly planning, assembly process design, ergonomic evaluation, assembly guidance and training. AR instructions create virtual visual objects and interact with real objects. Individuals can obtain real feedback and virtual content to analyze the behavior and characteristics of assembled products and gain the benefits of physical and virtual prototypes. Commented on the latest development of the AR Directive. Mizell et al. [24] and Fiorentino et al. [25] demonstrated that using text describing product manufacturing information [26] (PMI) as assembly instruction can improve task performance and affect users’ cognitive awareness. The results of Henderson et al. [27] and Zhang et al. [28] showed that using text, 2D pictures, and 3D pictures as assembly instructions can further improve user assembly efficiency. Stork et al. [29] and Uva et al. [30] showed that text, 2D graphics and wireframes as assembly instructions can also improve user performance in the assembly process. Hou et al. [7] and Wang et al. [31] used text and 3D graphics with dynamic effects as assembly instructions, which can also improve the efficiency of manual assembly. Funk et al. [32] and Doshi et al. [33] demonstrated the feasibility of using 2D images, videos, 2D graphics, and 3D graphics as AR instructions.

Our team hopes to summarize all the aggregated information of augmented reality-based assembly instruction’s characteristics, limitations, operability, and achievable fields. The semantic processing of assembly information is analyzed and discussed from the perspective of man-machine integration,and our conclusion is generally described. Therefore, the contribution of this paper is reflected in the following aspects:

It reviews the main research work of AR instruction in the field of manual assembly training, and repair.

It discusses the advantages and typical running framework of AR instruction in assembly training and repair.

It describes the core characteristics and key technologies of AR instruction.

It introduces the research status and future development trend of AR instruction.

In part 1, our team will summarize AR instruction from two aspects: (1) comparing conventional instruction and AR instruction in manual assembly, training and repair; (2) architecture of AR-based operating instruction. Part 2 summarizes the current situation of AR instruction research. By reviewing the existing experimental theories and scientific basis, analyzing and summarizing them and applying them to the experiments of our team, and solving some technical loopholes and limitations of the previous experiments. Part 3 summarizes the current issues with AR instructions. This part introduces the future development trend of AR instruction. Through the use of the structural framework diagram to explain the article arrangement plan in detail, it is obvious that through the analysis and integration of the review, the current popular AR instruction-related technical topics and the future AR instruction development direction are derived (see Fig. 1).