AI in everyday clinical work: six scenes

1 · “No marker. Can we go ahead?”

This demo shows a fictional hospital morning. You accompany a team that is checking its AI tool.

Nora Can we prepare for the procedure?

Mira Your question goes beyond that.

You are standing at the reporting workstation, where the images are viewed and assessed. Mira, the radiologist, opens the CT images, that is, the cross-sectional images from computed tomography. Lea, the radiographer, has just completed the examination on the scanner. Jan from IT looks after the computer systems. Nora, the surgeon, is standing in the doorway: “We are waiting for the report on the patient who fell. Can we prepare for the procedure?”

The AI processing has run. There is no marker on the image. “Then there doesn’t seem to be anything urgent in the head,” says Nora.

Mira pulls the images closer: “We don’t know that yet. This tool looks for signs of haemorrhage. It supports the prioritisation in the worklist, that is, the order in which I open the examinations. Earlier on, an alert led me to an abnormal case sooner. But I still have to look at everything myself.”

Nora: “And if it doesn’t mark anything?” Mira: “Then it has found nothing relating to its question. It has not looked at yours. Your question goes beyond that. You will get the report, meaning the medical assessment of the whole examination, from me in a few minutes.”

What a tool is meant for is set out in its intended purpose: which task, for which people, under which conditions. Nobody has tested anything outside that. The fact that an alert was right once is no evidence for it.

Nora nods: “Then our ward needs to be clear about what this sign means. And what it does not mean.”

Purpose limitation: A tool is tested for one task. It will still answer any question outside that task, and nobody vouches for that answer.

2 · “Did you also test it with our images?”

So far the AI only runs for some of the examinations. Today the question is whether to approve it for a second CT scanner.

Lea Did you also test it with our images?

Jan opens the documentation: “The results of the manufacturer’s study look good.” Lea points through the window: “Did you also test it with our images? Many people here cannot lie still.”

After reporting, they use the PACS, the system in which the images are stored and displayed, to open the cases used for testing so far. Almost all of them come from the first scanner. There is hardly anything from the second one, and difficult cases after head surgery are missing entirely.

“Then we will check,” says Mira. What that requires is called local validation: testing the tool on your own cases, on your own equipment, with the patient groups and image settings that actually occur here. And defining beforehand what counts as a pass, not afterwards.

“What do we compare against?” asks Jan. Against a reference standard: experienced colleagues assess the same cases without seeing the AI result and resolve disagreements together. The test cases must not be the same ones that were used to set up the tool. Otherwise the result says nothing about new cases.

“Don’t just take the nice images,” Lea adds. The team widens the selection and also looks at the workflow: do the alerts arrive in time, and do other examinations wait longer as a result? Until that is clear, the second scanner continues as before, in the order the team sets itself.

Local transferability: A result applies first where it was obtained. Whether it also applies to your people, with your data and in your workflow is something nobody will check for you.

3 · “I can object. But does my decision stand?”

Mira, the radiologist, wants to move a case forward in the test view for clinical reasons and checks with Jan from IT whether her decision against the AI sorting will stand.

Mira My change does not stand.

Jan opens a test view of the planned reporting worklist, a copy for trying things out that does not affect the live service. A prepared case is to be moved up for clinical reasons, even though the AI has not flagged it. Mira moves it to the top and refreshes the view. It is back at the bottom. She does it a second time. Back at the bottom again.

“So I can object,” says Mira. “But does my decision stand?”

“It works with my permissions,” says Jan, and moves the case up. It stays there. “But on duty I don’t work with your access,” Mira replies. Lea tries it too: she can pass on a change in urgency but cannot see whether anyone has picked it up.

They trace the path back to the RIS, the system for examination requests, appointments and reports. There, the urgency set by the physician is recorded correctly. The new list does not reliably take it over. Mira rejects the extension in this state.

“It only sorts,” says Jan. “The order helps decide who gets looked at first,” Mira replies. How a medical device is classified in legal terms, its risk class, depends on what it is used for and what can go wrong. Not on how harmless the function sounds.

Then Mira states what must apply in the next test: the clinically set urgency stays in place. Whoever changes it can see that the change has arrived. And anyone who is expected to check while on duty needs the time to do so, their own access and the authority to stop the tool from being used.

What happens at your workplace after a no? Is your objection filed away, or does it change what happens next?

Effective cross-check: Anyone who is expected to check must actually be able to change the suggestion or the use of the tool. If a no has no consequences, no effective check has taken place.

4 · “I looked at the sign first.”

Nora, the surgeon, looks at a closed case from the follow-up review with Mira and is asked to say what she looked at first this morning: the marker or the image.

Nora I looked at the sign first.

Nora comes by once more. Mira shows her a closed case from the follow-up review, the later review of examinations that have already been reported: a haemorrhage that was identified by a physician and for which the AI gave no alert. Exactly the task the tool is for, and it did not fulfil it.

“This morning I looked at the marker first,” says Nora. “And then at the image?” “To be honest, hardly at all.”

Mira knows the reflex: “If the display is often right, the next suggestion feels like a confirmation before you have looked yourself.” There is a name for this. Automation bias means that you trust an automated suggestion to the point where you perceive contrary evidence less clearly and cut your own check short. Nobody has to be inexperienced for this to happen. In studies of AI-assisted image assessment, even experienced radiologists were influenced by incorrect suggestions.

In the test view, Mira hides the AI alerts. Now the two of them first discuss the examination request, the patient’s condition and their own assessment. Only then does Mira bring in the alert. “That way you can see what came from you and what came from the tool.”

“Then that is exactly what belongs in the induction,” says Mira, meaning the training before anyone is allowed to use the tool: a convincing wrong suggestion and a missed finding, not just the cases that went well.

Nora phrases her next call differently: “I need the medical assessment of this patient.” You can make the same shift wherever you are presented with something finished. With an AI summary, it means: go to the passage in the source document that is supposed to support the well-worded sentence, and read it yourself. A coherent impression is not evidence.

Automation bias: The more often a suggestion is right, the more cursorily you check the next one. Form your assessment before you look at the suggestion.

5 · “The software is the same. Are the images too?”

Lea, the radiographer, brings cases from the follow-up review where the AI alerts have recently been less accurate, and clarifies with Jan from IT whether the software or the images have changed.

Jan The software is the same.

Lea Are the images too?

Lea has brought cases from the ongoing follow-up review where the AI alerts have recently been a poorer fit. Jan checks the installed version: “We have not changed anything on the AI.” Lea looks towards the scanner: “But since the scanner update, some images are reconstructed differently.”

A lead, not an explanation. If it is correct, the process has a name: Drift. It is not the tool that changes, but what goes into it. Different patient groups, different acquisition conditions, different image processing, and performance in routine use shifts without anyone having touched the software.

So they compare results over time, separately by scanner and type of examination. Lea contributes what has been changed on the scanner and in the acquisition workflow. Jan looks at data input and processing time. The radiologists assess clinical performance on independently rated cases. Individually, each of them sees too little. Only together can they distinguish a transmission error from a clinical problem.

Mira makes one point explicitly: they also look at examinations without an AI alert, and they record separately when a case was not processed at all. “Otherwise an outage looks exactly like an unremarkable finding.”

“Who will be keeping an eye on it tomorrow?” asks Nora. Mira and her deputy take over the clinical evaluation, Jan the technical monitoring, and they review conspicuous changes together. If a deviation affects safety, the team suspends the affected use and involves the manufacturer.

Measurement in operation: A tool can get worse without anything about it changing. Whoever uses it must measure again in operation, regularly and after every change to what goes into it.

6 · “The AI is down. Who takes over now?”

During the agreed failure drill, Jan from IT disconnects the AI service, and Mira, Lea and Nora have to keep the reporting worklist going without AI sorting.

Lea I am doing it without your help now.

During the agreed failure drill, the planned test with the tool switched off, Jan disconnects the AI service in the test view. Images and requests remain available. New AI results no longer arrive.

Lea calls up the complete reporting worklist without AI sorting and cross-checks the open examinations. Mira sets the order according to clinical urgency. Nora tries the direct route and phones radiology. In this scene you work with the fallback path, the previously agreed way of working without the failed tool.

“And what if you are not on site?” asks Lea. Jan takes his hands off, and Lea goes through the steps on her own. In doing so, they notice that the number of the technical on-call service is only in Jan’s notes. It goes into the shared instructions, and these must be accessible even when the systems are down.

Today only the AI fails. If the image archive and order management are also unavailable, the old reporting worklist is not enough. For that, the hospital needs its own downtime procedure, and it must not depend on the system that has just failed.

Switching back on is rehearsed as well. Jan checks the technology, Mira the display. Together they reconcile which cases are open and which alerts arrive late before the team uses the support again.

Nora goes back to the ward. Lea prepares the next examination. Jan stands next to them, and nobody is waiting for him.

This is where the demo ends. Take one question with you: which part of this morning would still be unresolved in your team?

Manageable failure: A tool only belongs in everyday work once the route without it has been practised and the people on duty can take that route without outside help.