Spatial Computing in Surgery & AR in the Army - MetaVisions #21

Hi all, hope everyone is doing fantastic! No Newsletter last week as I was in the Swiss Alps recharging my batteries (although I was active most of the time).
XR seems to continue gaining ground this year and not just Apple!

Spatial Computing for surgery!

The use cases of spatial computing are well documented. There was a boom of interest by large organisations such as the NHS to deploy Immersive Training and Remote Assistance tools during the pandemic. It served as a way to reduce contact between staff, reduce the number of staff needed to do a task and decrease the time to action for newly qualified staff.
Insight UK x Alder Hey NHS - Remote Assist

When it comes to surgeons in specific, they have been using Spatial Computing (XR) to:
•Prepare for surgery by connecting remotely with peers that are located around the globe, going through MRI scans and patient data to go into the operation fully prepared.
•Teach students by showing live ‘demos’ of their operations - students can ask question on the go and get instant feedback, which is quite different to watching video recordings.
•Placing holographic patient data and 3D models in the operation room to assist them whilst they work.

Last week, a medical team at Cromwell Hospital in London, used the Apple Vision Pro in two spinal surgeries - the feedback was positive, two surgeons from the Complex Spine group, successfully used the technology to carry out two microsurgical spine procedures. The AVP leveraged an AI-Driven app by eXeX.

The eXeX software gives nurses and technicians holographic and touch-free access to the surgical setup and the procedural guides from within the sterile field of the operating theatre. They can access data and visualisation that has been previously been unavailable, thanks to the technology. Since being introduced to the Cromwell Hospital, it has been able to significantly increase the efficiency of surgical delivery, through equipment and workflow accuracy, which is improving patient outcomes.

A number of other developers plan to bring healthcare solutions to the AVP, such as: Fundamental Surgery, CollaboratOR 3D, and Complete HeartX.

Is the AVP the best device for this use case?

From a health and safety aspect, I am quite surprised that the doctors were able to use this in a live operation environment. Although the headset has incredible pass through capabilities, which allows the user to see their ‘real world’ even when wearing the headset, it is done through software and cameras. This means that at any point during the procedure, there could be a small latency issue or the OS can jut stop working, which could create a massive risk factor depending on what the doctor is doing.
At the moment, AR devices such as the HoloLens 2 or the Magic Leap 2, which have a see-through display, would be great alternative, as it removes the risk of the doctor becoming effectively ‘blind’ in the middle of a procedure.

US Army AR ‘Robocops’

Since 2018, the US army has been working with Microsoft to develop an AR Headset aimed at the defense industry. The project is knows as the Integrated Visual Augmentation System (IVAS), it is based on Microsoft’s HoloLens technology - Microsoft won a $480 million contract to fund the development of the device, with a further $21.9 billion over 10 years to acquire units.

Microsoft reported that: The devices, using what is called the Integrated Visual Augmentation System (IVAS), will allow soldiers to see through smoke and around corners, use holographic imagery for training and have 3D terrain maps projected onto their field of vision at the click of a button.

The work done so far appears to be a mixed bag, with initial reports in 2022 suggesting that there was some concern over the performance of the device, this was after the Army conducted some field tests with the device and alleged delays and quality problems.

However, at the start of this month it was reported that the Army is requesting $255 million in the 2025 budget to acquire another 3162 units of the IVAS 1.2 system - this suggests that there is still excitement and promise around this project.
Another $98 million was requested for development, which I see as extremely important to keep pushing what is possible with XR. We should see some of the innovation in this project come into Microsoft’s next HoloLens device, some of the development efforts have been reported to be the below:

The Army’s R&D plans for 2024 included developing software for IVAS “that enables soldiers to intuitively relay reconnaissance intent to a team of autonomous sensors and quickly interpret feedback from the systems and make targeting decisions,” according to budget justification documents.

The Army also wants to explore algorithms that use voice commands, eye movements, and hand gestures to interact with the system for “relaying intent and closing the targeting cycle more effectively,” as well as develop “feedback mechanisms” in the Android Tactical Assault Kit and IVAS to improve the AI algorithms “once soldiers recognize mistakes by the autonomous sensors.”

Plans for 2025 include making improvements related to the heads-up-display design, thermal and low-light sensors, IVAS extensibility, form factor, reliability, reducing weight, and developing AI data integration and applications, per the budget justification documents.

Great to see that the IVAS project is still moving - it is an important win that showcases that XR is not a gimmick or only for gaming use cases, but it can provide real value, even in the riskiest of environments!

See you next week,
Davi, MetaVisions