[2210.07730] DroneARchery: Human-Drone Interaction through Augmented Reality with Haptic Feedback and Multi-UAV Collision Avoidance Driven by Deep Reinforcement Learning

We propose a novel concept of augmented reality (AR) human-drone interaction
driven by RL-based swarm behavior to achieve intuitive and immersive control of
a swarm formation of unmanned aerial vehicles. The DroneARchery system
developed by us allows the user to quickly deploy a swarm of drones, generating
flight paths simulating archery. The haptic interface LinkGlide delivers a
tactile stimulus of the bowstring tension to the forearm to increase the
precision of aiming. The swarm of released drones dynamically avoids collisions
between each other, the drone following the user, and external obstacles with
behavior control based on deep reinforcement learning.
The developed concept was tested in the scenario with a human, where the user
shoots from a virtual bow with a real drone to hit the target. The human
operator observes the ballistic trajectory of the drone in an AR and achieves a
realistic and highly recognizable experience of the bowstring tension through
the haptic display.
The experimental results revealed that the system improves trajectory
prediction accuracy by 63.3% through applying AR technology and conveying
haptic feedback of pulling force. DroneARchery users highlighted the
naturalness (4.3 out of 5 point Likert scale) and increased confidence (4.7 out
of 5) when controlling the drone. We have designed the tactile patterns to
present four sliding distances (tension) and three applied force levels
(stiffness) of the haptic display. Users demonstrated the ability to
distinguish tactile patterns produced by the haptic display representing
varying bowstring tension(average recognition rate is of 72.8%) and stiffness
(average recognition rate is of 94.2%).
The novelty of the research is the development of an AR-based approach for
drone control that does not require special skills and training from the
operator.