Reading List (blogroll)

Midjourney is an AI image model. If you’ve ever used Nano Banana or asked GPT to draw you a picture, it’s like that, except from a medium-sized startup instead of a tech giant.

Earlier today, they announced a pivot to medical scanners. The new MidJourney Scanner, which they describe as “a bold new kind of machine to reimagine the foundations of healthcare and our relationships to our bodies”, will be a tank of water surrounded by a ring of ultrasound scanners. The patient goes into the tank, the scanners emit ultrasound from all angles, and then some fancy AI reconstructs the echoes into a 3D picture of the body. The result is ultrasound tomography: the same sort of rich data as a CT or MRI, but done via ultrasound, with no harmful radiation, in twenty seconds.

This is cool, and it’s great to be ambitious, but I think the narrative among the SF AI crowd has escaped its basis in the medical facts, so I want to throw a bit of cold water on it. I’m a psychiatrist, which is about as far as you can get from radiology while still being a doctor, so this is speculation only, and you can ignore it if you find an actual radiologist or ultrasonographer with opinions. Still, my take is that this scanner isn’t useful for most current serious medical applications. It could potentially be used to pioneer a new class of low-risk screening applications, but it’s unclear whether these are good, and depends a lot on what other future technology gets invented in parallel.

Why can’t this immediately replace existing medical image modalities like normal ultrasound, CT, or MRI?

Ultrasound is great, but it can’t penetrate bone or air. Many things doctors want to look at involve bone or air in some way. For example, the brain is behind the skull, which is a bone. The bowels are full of air. The lungs are super full of air. This limits ultrasound to the remainder - especially parts of the digestive, endocrine, and vascular systems, and superficial tissue like fat and muscle.

(it’s actually worse than this. Normal ultrasound can be used to image certain organs like the heart or prostate, but only through the technician carefully angling the probe. Midjourney hasn’t given details, but most likely their Scanner won’t be able to match this level of precision, so the heart, prostate, and some other usually-ultrasound-compatible organs will be outside its reach.)

Most MRIs or CTs involve one of the organs ultrasound can’t reach (this would be one reason doctors might do an MRI or CT, instead of just using ultrasound). In other cases, you don’t know what organ we’re looking for, and you want to be able to see everything (for example, if you’re scanning for cancer metastases, you can’t leave the brain and bowels out of the scan!) So this technology can’t replace most MRI or CT.

What about replacing ordinary ultrasound? One of the big advantages of ordinary ultrasound is that it’s a cheap machine you can keep on a cart and connect to a patient who’s lying in a hospital bed. Even though it might work better to put the patient in a giant water-filled tank surrounded by hundreds of ultrasound machines, if you tell your hospital orderlies “please transport this frail 90-year old to my giant water-filled tank, and lower them in slowly” they will stab you with your own scalpel. So this would need to be much better than ordinary ultrasound to capture even a fraction of these use cases. But ordinary ultrasound is already pretty good, this technology is untested, and it will be hard for it to be that much better.

Aren’t there a few edge cases that are poorly-served by existing modalities and ordinary ultrasound? Yes - the classic one is certain types of breast cancer, which don’t show up well on mammography against dense breast tissue, and require too much of a search for ordinary ultrasound. It’s a perfect match for this technology, which is why ordinary medical device companies have already created an ultrasound tomography scanner for the breast and it’s used regularly in medical practice. It’s not quite as neat as the MidJourney Scanner - the patient just lies on a weird-shaped table in a position that puts their breasts in a pool of water, instead of submerging the whole body, and you get correspondingly less coverage - but it works fine for the rare case where this technology actually fills a gap.

There are probably other edge cases I don’t know about, but they weren’t important enough for normal medical device companies - who absolutely know about this technology and have thought about it a long time - to invent devices for it.

Couldn’t this technology enable new, non-specific-diagnostic uses for healthy people?

This is where Midjourney seems to be going. Aware that this doesn’t fill a specific diagnostic hole (and would probably be annoying to get past the FDA), they’re imagining something where healthy people go for one of these scans regularly (let’s say once a year). The scanner can produce an image of the whole body, and if there’s anything abnormal (for example, a tumor), they can send them to the regular medical system to get it investigated and treated. You could even have a longitudinal series - this anomaly was tiny on the last scan a year ago, but now it’s bigger, so it’s suspicious for cancer and needs to be investigated immediately.

Here the question is - why is this better than regular whole-body screening MRI scans, a technology which currently exists?

We can certainly think of the opposite - reasons why the screening MRIs are better. Screening MRIs can view the whole body, including the brain, lungs, heart, and interior of the bowels. They have higher resolution. They’re a real technology that exists now, rather than a cool idea by an AI art company. They cost about $2,000, which is cheap by the standards of the US medical system.

So why don’t people get yearly whole-body MRI screenings? Some people do - companies like this provide them, and some rich people who can pay $2,000 out of pocket consume them. But the medical consensus currently recommends against them because they’re more likely to produce dangerous false positives than helpful true positives, and studies have failed to demonstrate benefit.

(a “false positive” in this context isn’t the scanner hallucinating something that isn’t here. It’s the scanner finding some sort of boring abnormality that doesn’t matter - like a zit but inside your organs - and then making everyone panic that it’s cancer, and causing unnecessary tests, surgeries, etc).

Let’s grant that, in fact, these scans produce a lot of false positives, and that a lot of harm is done by unnecessary tests and biopsies and treatments for these false positives. Still, can’t you just adjust the detection threshold until it only fires for extremely obviously bad findings that are definitely worth investigating? This question has bothered me for a long time, and I’ve never been able to get a perfectly clear answer from the medical literature. Here are the mediocre answers I can sort of vaguely see:

• “Obviously bad” is a medical judgment, not a radiological one. Radiology can tell you when something is very big, or very fast-growing, but sometimes there are harmless large fast-growing things. What clinches a decision of “important to investigate further” are questions like age, smoker status, family history, etc. But if you’re an old person who smokes and has a family history of cancer, your doctor is already recommending some kind of lung scan, and this is the opposite of telling all healthy people to get screened all the time.

• In practice, most problems start producing symptoms before the threshold where they’re so clearly bad on imaging that you should extremely obviously investigate them.

• If you actually set the standard for further investigation high enough, it would trigger so rarely that people wouldn’t want these scans for other reasons, like inconvenience and cost.

• Smart technocrats can set the threshold for investigation wherever they please, but a patient who learns that they have a large mass in their brain isn’t going to accept “no” for an answer, and is either going to get it investigated or else spend the next several years freaking out, which has health costs of its own. And malpractice-suit-wary doctors are going to think about how it will sound in a court case to say “yeah, I knew he had a giant mass in his brain, but it was 0.1mm short of the threshold where we bother checking what’s going on, so I did nothing”. So in practice, patients will demand further investigation, and doctors will agree. And the sort of rich, agentic, ultra-health-conscious people who will pay $2000 for a screening MRI their doctor recommends against are exactly the types of people who would be most likely to fall into this error mode.

So although pushy rich people occasionally get boutique clinics to give them these screening MRIs, normal doctors and the legitimate system are against them. Whether or not you agree with this perspective, I assume they would approach the idea of screening whole-body ultrasounds the same way (ie recommend against them). So this would basically be serving the same population of pushy rich people who are already getting the screening MRIs, which are better. So what does this buy you?

One possible answer is convenience. Midjourney claims these ultrasounds could be much faster and more comfortable than an MRI (which involves ~60 minutes in a giant scary metal tube that sometimes kills you if you forget to take off your jewelery).

Another answer is cost. Currently, the Midjourney Scanner is entirely experimental; its prototype is no doubt be very expensive, like all prototypes. But the sorts of sensors and chips that make up the Midjourney Scanner have better cost scaling curves than the sorts of giant magnets that make up MRIs. So if this ever became exactly as common as MRIs, it would probably be much cheaper.

This, then, is the strongest argument for the whole-body scanning proposition: you could serve the same pushy rich people who get whole-body MRI scans now against their doctors’ recommendations, but they could do it in a nice spa instead of a giant scary metal tube, and it could be cheaper. And that could unlock a whole new demographic of different pushy rich people who then would be willing to try it.

Couldn’t this technology become more useful in the future?

Yes. I think the best way to think of this is as a bet that future technology develops in a way that allows new possibilities for diagnostic ultrasound - or, even better, an attempt to gather the training data / interest / investment that will make this happen.

For example, if you get lots of high-quality ultrasound data (perhaps because you incorporated your ultrasound scanner into a spa and billed it as something that billions of people should be using every year), maybe you could train a really good AI on that data and do better than any existing radiologist in learning to interpret it and figure out what’s dangerous. Then you wouldn’t have to worry about the studies failing to find benefits from existing whole-body screening.

Or - a commenter informs me of Full Waveform Inversion Imaging Of The Human Brain, which argues that bone only makes ultrasound imaging very difficult, not impossible. It scatters the waves in complicated ways, but with enough math, you could reconstruct what’s going on. How much math? Enough that nobody has ever done it for a real human skull. It would take absolutely gargantuan amounts of compute. But it’s fine! The world economy is in the process of re-centering around creating gargantuan amounts of compute! In the glorious AI future, when a halo of space-based data centers has turned Earth into a miniature Saturn, we can get as many brain ultrasounds as we want!

So the best case for the Midjourney Scanner is that they’re trying to pre-emptively jostle themselves into a position where they can benefit from upcoming AI revolutions. If AI drives the cost of sensor electronics to near zero, and gets so good at radiology that it can cleanly separate true positives from incidentalomas, and becomes so good at wave dynamics that it can overcome previously insurmountable problems around imaging through bone, then probably a company that already has a chain of spas fitted with giant rings of ultrasound scanners will be in a great place to benefit from the subsequent medical revolutions.

All that I can say against this plan is that you need to believe AI will benefit ultrasound tomography in particular. By the time the AI revolution has solved all the problems that stand in the way of the Midjourney Scanner, might it also have beefed up normal MRI, or normal ultrasound, or enabled some kind of entirely new scanning modality, or cured cancer so thoroughly that we don’t need yearly cancer screening?

If you think that no - it will most likely benefit ultrasound in particular - then Midjourney’s bet looks a lot better.

Appendix: Highlights From The Comments On Twitter

I mentioned this on Twitter and got some great responses.

The responses from real radiologists were universally negative. Here are some examples:

When I pressed him for details, Dr. Harris kindly gave a longer explanation:

The trivial reason is that due to the limitations of physics ultrasound will always be less capable at resolving anatomy than MRI or x-ray-based methods that we already have

The more fundamental reason is that all of these applications of technology to screening for anatomic abnormalities rest on a flawed assumption that detecting an abnormality indicating cancer earlier necessarily leads to better outcomes from earlier treatment. This is not, however, a dogma that we can just assume, but rather a hypothesis that must be tested for each type of cancer we wish to improve the survival for. The current imaging based screenings we do, whether for breast, colon, prostate, or lung all have robust literature that at least supports the idea that earlier detection leading to earlier intervention improves outcomes. In the case of prostate, it’s still being argued about quite vociferously and I’m not taking a particular position on that myself. But all of these screening regimes depend not just on modality, but the exact protocol you use to enroll and screen the patients in addition to the treatments that are available and how well they work at what stage.

To the lay person, it seems obvious that finding a cancer on imaging earlier is always better, but in many cases, it’s not only not better it can be worse! Especially on a population basis when you account for medical misadventures that may ensue from unnecessary or earlier or futile treatment.

Finally, this guy is a former president of the Society of Skeletal Radiology:

But I also got responses from non-radiologists, who were more optimistic:

Like I said above, I’ve never been able to fully get my head around the argument against screening and setting a very high threshold for action. Michael’s temperature-taking analogy is a good one. But I don’t think this can be the whole story, because even when entrepreneurs set up their own full-body MRI clinics that don’t cost the system anything, the medical system recommends against using them.

Several people thought that no, surely there has to be some way to make this useful, especially if you could take multiple scans over a long time period:

I think a good intuition pump here is the skin. Weird stuff is always happening on your skin - zits, pimples, warts, but also occasionally melanoma. If you couldn’t see your skin, and had to observe it through some kind of medical imaging, a doctor might be able to tell you “There’s some kind of little bump on your elbow”. Then you wouldn’t know if it was a zit, pimple, wart, or melanoma. But since cancer is very scary, you might say “Oh no! There’s a mass on my skin! Maybe it’s melanoma! And it grew since the last scan! You need to do a skin biopsy immediately!” And then you would have to get surgery for a pimple or a spider bite or something.

Using this intuition pump, I don’t think Paul’s argument stands. Most pimples come after birth! Even if you got a skin scan every month and could compare the current skin scan to the previous one, you’d still be at high risk of fretting over pimples and demanding biopsies for them.

All of your organs are constantly growing little pimple-like things that don’t matter. Depending on the location and type, we call these cysts, polyps, fibroids, lipomas, adenomas, hamartomas, etc. We are bad at distinguishing these from dangerous tumors - and so far the studies show that when we try to do it in the entire population on a mass scale, we cause more problems than we solve. That could change in the future! But it’s a separate bet from whether ultrasound tomography will be good or not, and the ultrasound tomography won’t be very useful unless we solve this problem too.

Andrew knows a lot about sports and metabolic medicine, so I take his opinion here very seriously. A DEXA scan is complicated way of arranging x-rays which is specialized for measuring things like density. Its most common use is measuring bone density in osteoporosis. The Midjourney Scanner probably won’t help with that, because some bones are behind other bones or air.

But its second most common use is measuring “body composition”, the relative amount of fat and muscle. Some high-powered athletes use this to make sure they have exactly the amount of muscle they want, and some people on very precise diets use it to make sure they’re losing exactly the amount of fat they want. And then as usual, there’s a long tail of rich people who don’t know exactly why they’re using it, but their boutique rich-person clinics upsell them on it and give them very precise information on their body parameters which they then proceed to ignore. Between these three categories, it’s a 9-10 digit dollar industry.

The Midjourney Scanner would be a good replacement for DEXA scans - there might be a little fat hidden behind air pockets, but it can still give you a pretty good idea. I don’t know whether it has better or worse resolution, but it might be better, and it would save people (a tiny bit of) radiation.

On the other hand, Big Business has done a great job with DEXA scans. The cost has been driven down to ~$50, and you can do them in a van in the parking lot of your gym (or in front of your house if you make it worth their while). The advantage of the Midjourney Scanner over MRI - all you have to do is go to a spa! - here becomes a disadvantage - you mean I have to go all the way to the spa?

So this is a possible application, but it depends on branding and on outcompeting some pretty cut-throat existing businesses.

This is an interesting take. If you could guarantee that the diagnostician would be perfectly rational (because it’s an AI) and perfectly immune to patient pressure (because it’s an AI), would that switch the sign of primary imaging-based screening from negative to positive?

There are many cases in medicine of some computer system outperforming doctors at some specific diagnostic task. But because of the way regulation works, usually the final decision about whether or not to trust the computer system gets put back in the hands of doctors, who then reintroduce some of the biases the system was intended to prevent. So thus far there have been limited gains from this kind of thing.