www.popsci.com
Minimally invasive surgeons at UC San Diego Health are the first in the nation to evaluate the potential use of spatial computing apps on Apple Vision Pro in the operating room.UCSD ShareApples $3,500 Vision Pro sounds like a bargain compared to the price of a fresh, medical-grade cadaver. (A body alone can be up to $10,000 before refrigeration and transport costs, in case you needed to know.) And some medical institutions have started practicing surgery using the spatial-computing headset, which doesnt require a physical human body. Replacing cadavers is just one example of how the Vision Pro has made its way into the medical field since it hit the market in February 2024.On January 30-31, 2025, Sharpe Healthcare hosted the inaugural Spatial Computing Health Care Summit, where medical providers gathered to discuss their use of spatial computing, which embeds digital objects into a live feed of the real world. The same tech that allows people to play virtual Battleship with each other has moved into applications that include everything from training and education to full-fledged operations on human patients.Operating room in a boxThe OR is a very kind of cluttered place, explains Dr. Ryan Broderick, minimally invasive surgeon and interim director for the Center for the Future of Surgery at the Department of Surgery at University of California San Diego. Theres anywhere between five and six screens in a room during a case. Those screens also need to exist between other equipment like vital sign monitors, oxygen tanks, and other medical professionals.The current crowded layout can prevent communication and make checking important vitals a time-wasting and body-punishing task. The way the screens are positioned isnt very ergonomic, which can lead to neck strain, shoulder strain, and injury in our surgeons, says Broderick. By moving the displays into a spatial-computing environment, the information exists wherever the surgeons and nurses are looking at a given moment.The UCSD investigational review board has overseen more than 50 live surgeries in which the primary surgeon wore Apple Vision Pro hardware to provide virtual monitors. Right now, Ive got Vision Pro set up so that myself and my assistant surgeonwho is often a traineeand our scrub tech can all wear the headsets during a case, says Broderick. We each have our own virtual monitors that we put in physical space throughout the room so that were ergonomically positioned and can do the operation without these extra monitors that we normally have. Reducing fatigue is a good thing, especially when surgeons could eventually wear these devices for procedures as long as 12 hours.Training in a spatial-computing environmentWhile the Vision Pro live surgery trials are still in the early stages, the headset has been popping up in training scenarios for medical professionals. Boston Childrens is using Vision Pro to train nurses on using infusion pumps, says Susan Prescott, vice president of Apples Worldwide Developer Relations and Enterprise & Education Markets. The app shes referring to is called CyranoHealth, and it provides impressively thorough and detailed training scenarios for common procedures and situations. Prescott also touts the emergence of digital twins, which are highly detailed 3D models of humans that can, in some cases, take the place of cadavers in surgical training.The team at Boston Childrens Hospital created CyranoHealth, an app that offers immersive, comprehensive training on new medical equipment for frontline workers. AppleWhile it may seem odd to add a layer of technology between patient and surgeon, that already exists in many cases. Part of what surgical training is, especially for laparoscopic surgery, is disassociation of what your hands are doing with what your eyes are seeing, says Broderick. You have to make hand movements that do certain gestures on the inside that might not be the same as what youre doing outside of the body. But with this, I can position the monitor really closer down on top of the operative anatomy. And it feels much more natural with how my hands are moving. And that setup doesnt need to change from room to room since the monitors exist inside the spatial computing environment.Instant replayPracticing doctors have also embraced spatial computing for continued learning. Reviewing surgeries after the procedure can take up a lot of time, says Tommy Korn, a Sharp HealthCare ophthalmologist and one of the biggest Apple Vision Pro proponents in the medical field. It takes about 12 minutes. But now, AI software can actually parse it and allows me to jump to specific points and observe them with all the other crucial data presented right along with it. Students can then look back on those surgeries and observe the process with data to contextualize it.Providers can also have their peers review a surgery with a direct feed from a tool like a 3D microscope, a tool Korn commonly uses during a procedure. It will tell me how long it takes me to do a surgery, says Korn, and I can tell if Im taking too long on one part or another. The headset can display notations, timers, synced vital signs, notes, and other data that makes the review sessions more productive.The Vision Pro headsets used in universities and operating rooms are the same models you could pick up off the shelf at the Apple store, and thats a crucial part of Apples plan for the device. We make Vision Pro, Apples Susan Prescott explains. There are software differences that most consumers dont need, like MDM [multiple device management] or VPN, but the Vision Pro is the same for everyone. This single SKU approach may contribute to the Vision Pros somewhat tumultuous path in the consumer market, where the $3,500 price tag and unfamiliar ecosystem have led to reports of slower-than-expected sales. And while the consumer market varies broadly from the enterprise customers, some advantages come with a consistent product across all users.The first variable is patient recognition. Every instrument in an operating room or any medical facility can look scary or intimidating. When a doctor puts on a Vision Pro, patients may recognize it as a device they tried in the store. How scary can a device be once youve played a fun AR robot board game on it?Apps like Cinematic Reality from Siemens Healthineers offer access to detailed, high-res anatomical scans that can be used for training and education. Its real medical data, but its presented in the slick, intuitive way youd expect from a consumer-oriented app.The Siemens Healthineers Cinematic Reality app provides detailed interactive models. Siemens HealthineersBroderick clearly explains the procedure to each patient before getting their consent. Weve showed them pictures of how it kind of looks in the OR and what it would look like in the headset, he says. I think weve had one patient who was just very nervous about the surgery and didnt want to do anything different than what has already been practiced in the past, but everybody else has been all on board and super comfortable with it. Putting the headset on the patient is an effective way to get patients on board.From the providers standpoint, having a single device is crucial for standardizing research and building on a hardware platform that wont change quickly and require hasty, expensive upgrades lest they linger on outdated and insecure systems. A shocking number of medical facilities and programs still rely on wildly outdated hardware and software (including Windows XP).The future of spatial computing in the medical fieldWith just a year in the wild, the Apple Vision Pro is still in its relative infancy, but many providers see potential use cases that havent been tried yet. Broderick sees potential for continued mentoring, even during a procedure. If a surgeons wearing a headset and gets in a jam, they can phone a friend and that video feed could still be seen by that expert surgeon and provide feedback, he explains. Or a surgeon could rewatch a surgery with a colleague after the fact for further analysis.Broderick also sees the potential in overlaying medical imaging over a view of the real world. So to be able to take CT scans or images and create three-dimensional models that you can then overlay on real anatomy could really guide surgery, he says. You could see it as like a surgery by numbers or a paint-by-numbers situation that could be incredible in the future.While apps and AI guidance will play a large role in spatial computings entry into the medical field, Apple and the providers doing the testing are confident that these technologies arent set out to replace providers with automation. The AI might make some suggestions or streamline some processes, Korn explains. But ultimately, the doctor or the clinician or, in this case, the human being still has control. Like the other robots in the operating room, theyre there as tools.The Spatial Computing Center of Excellence at its Innovation and Education Center on Sharps corporate campus is now officially open, and UCSDalong with a litany of other universities and institutionsplans to continue the field research into Vision Pro and spatial computing as a whole. According to Prescott, Apple is listening to the feedback coming from this research and acting on it faster than usual. Weve already done three updates to the software on Vision Pro, which is a pretty rapid pace for us, she says. Each update adds features requested by users in the real world. And with every update, those virtual cadavers will only get more realistic.
