From laser simulation to real combat realism

The context

Live military training with real weapons is essential to prepare soldiers for missions, but it comes with major limitations. Existing laser-based training systems, while widely used, fall short in realism and tactical accuracy, limiting their effectiveness in preparing soldiers for complex, real-life scenarios.

To overcome these constraints, one of the world’s leading manufacturers of firearms and military equipment set out to design a next-generation live training system. Their objective: leverage computer vision to accurately assess where a real bullet would have hit a person or a vehicle, without actually firing live ammunition at targets.

The logic

Achieving this level of realism required combining visual perception, ballistic data, and real-time processing. The system had to operate reliably in dynamic, outdoor environments; process live video streams with minimal latency; and provide immediate feedback to trainees, while accounting for real-world constraints and edge cases.

The Solution

We developed an end-to-end computer vision-based Proof of Concept that processes live camera feeds mounted on weapons and augments them with a tactical HUD (head-up display).

Using image data, the system estimates the distance to the target, up to 900 meters. The weapon-mounted camera also provides gyroscope data, enabling to compute bullet trajectory and flight time (assuming no wind), combined with ballistic speed parameters. Together, this information allows the system to determine whether a target would be hit or missed, without explicitly predicting the full trajectory itself.

Built in Python and deployed on Google Cloud, the solution leverages advanced neural-network libraries for perception, inference, and visualization. As the project evolved, we continuously optimized inference time, improved prediction quality, and addressed an expanding range of edge cases commonly encountered in real training conditions.