HoloLens goes surgery.

In March 2018, HoloLens made its official début in an operating theatre using a new procedure developed by up-and-coming medical innovator apoQlar in Hamburg together with surgeons including from Hamburg’s Marienkrankenhaus medical centre. A state-of-the-art digital assistant layers a virtual, three-dimensional image onto the patient’s body. The underlying Virtual Surgery Intelligence (VSI) software runs on Microsoft’s HoloLens mixed reality glasses delivered by Bechtle.

The project launched with attending physician for head, throat, and facial plastic surgery/ENT medicine, Dr Hans-Jürgen von Lücken, and his colleague, Dr Kathrin von Usslar aboard, both with the Marienkrankenhaus. As members of the Medical Advisory Board, they worked with apoQlar to develop the revolutionary procedure. The VSI software employs artificial intelligence to project a three-dimensional MRI or CT scan onto the glasses, which surgeons can then layer on top of their patient’s body with stunning detail, rendering a complete anatomical image with all physical structures that can be moved freely in the room. Surgeons can still see and interact with their actual environment and have their hands free to perform the operation.

Advanced technology.

The Hamburg start-up, apoQlar, specialises in innovative medical applications with a particular focus on mixed and augmented reality. apoQlar sourced the required glasses from Bechtle, Microsoft’s only HoloLens partner and exclusive distributor in Germany. HoloLens is a self-contained computer running the Windows Holographic operating system. The 3D glasses are Wi-Fi-enabled and do not need to be connected to a PC, console or mobile device to be controlled.

“The glasses are wireless, allowing the user to move freely about the room. They recognise the position of the patient in the room and fix 3D objects in place where they are needed, even when the wearer moves their head. In an OR, this offers a huge advantage over traditional methods,” says Sebastian Schwab, VIPM Microsoft, Bechtle Logistik & Service GmbH.

Intelligent control.

At the Marienkrankenhaus Hamburg, mixed-reality applications are used before, during and after operations. Surgeons can adapt and remove layers and structures in the air using gestures and voice commands. The new technology makes it easier to perform operations, enhances the training of physicians, and can also help inform patients ahead of a procedure. Of course, the glasses don’t mean the attending physician or surgeon carries any less responsibility, but they no longer have to continually refer to a previously created 3D model on a separate screen. Initial technological development focused on the head and throat, and has since been expanded to include other parts of the body.

In addition to MRI and CT images, HoloLens also allows users to view other important patient data such as operative and other medical reports, CT findings and lab results, with data protection built in. After the operation, patient data are purged from the glasses and only reside on the hospital’s servers.