AI is everywhere in medical device manufacturing right now — in quality systems, in design processes, in manufacturing lines, and increasingly in the devices themselves. The regulatory landscape is evolving fast, the risks are real, and the opportunities are significant. We thought it was time to get three of Red Hen’s resident experts in the same room to hash it out. Art Meta covers AI management systems and ISO 42001. Norma Tive has spent more years than she’ll admit guiding manufacturers through ISO 13485 and quality system fundamentals. Clara Fication is our QMSR and FDA regulatory specialist. We gave them five questions and got out of the way.
— Amanda
Question 1: What actually qualifies as an “AI tool”? Every vendor in our inbox is calling their software AI.
Art Meta: Great question to start with, because this is where a lot of manufacturers go wrong — they either over-classify things as AI and create compliance work they don’t need, or they under-classify actual AI and miss real obligations. ISO 42001, the new AI management system standard, defines an AI system as a machine-based system that, for a given set of objectives, makes inferences from inputs and produces outputs such as predictions, recommendations, or decisions that can influence real or virtual environments. The operative word is inferences. If the software is just executing deterministic rules — even complex ones — it’s not AI under that definition.
Norma Tive: So my spell-checker is safe.
Art Meta: Your spell-checker is safe. Your AI-powered document review tool that learns from your edits and starts making suggestions based on patterns across your whole document corpus? That’s a different conversation.
Clara Fication: From an FDA perspective, the key split is between Software as a Medical Device — SaMD — and software that supports manufacturing or quality operations but isn’t itself a device. FDA has been working on this since the 2019 action plan for AI/ML-based SaMD, and they’ve been pretty clear: if an AI tool is part of a device or functions as a device, it goes through device regulation. But if it’s a quality or manufacturing tool — think AI-assisted visual inspection on your production line, or an AI tool that flags CAPA trends in your complaint data — then you’re in QMS territory, not device submission territory. That said, FDA is watching the quality-tool space closely. Don’t assume “internal use” means “no FDA scrutiny.”
Norma Tive: Which means the first thing any manufacturer needs to do is classify their tools. Not just “is this AI?” but “what category of AI is this, and what regulatory pathway applies?” Document that classification. Because if you can’t explain your reasoning in an audit, the classification doesn’t exist.
Art Meta: Exactly. ISO 42001 actually includes a nice framework for AI system categorization — intended purpose, risk level, autonomy level. Worth using even if you’re not pursuing 42001 certification. It gives you a defensible, structured way to answer the “what is this, exactly?” question.
Question 2: Once you’ve decided a tool qualifies as AI and you want to implement it — where do you even start?
Norma Tive: You start where you always start: your QMS. Before you touch a single configuration setting or sign a vendor contract, ask yourself whether implementing this tool is a change that requires your change control process. Spoiler: it almost certainly is. Then ask what your design and development procedure says about software. Because AI tools don’t get a pass just because they’re purchased off the shelf.
Clara Fication: Under the QMSR, the answer is ISO 13485 — which means your quality planning has to address new technology risks upfront. As I’ve written about before, the QMSR expects you to demonstrate that your QMS works, not just that it exists. An AI tool that’s humming along making recommendations that nobody has validated isn’t a QMS asset — it’s a liability waiting to be discovered.
Art Meta: ISO 42001 structures implementation around the AI system lifecycle: design, development, deployment, operation, and decommissioning. Each phase has its own risk and governance considerations. One thing manufacturers consistently underestimate is the data governance piece. What data trained the model? Was it representative of your production environment? Is it traceable? If you’re using a vendor-supplied AI tool, can the vendor even answer those questions? You’d be surprised how often the answer is “not exactly.”
Norma Tive: Which is why supplier qualification is not optional here. Your standard supplier evaluation process applies. The fact that it’s an AI tool delivered as a cloud service doesn’t make it less of a supplier. If anything, it makes the evaluation more important, because you have less visibility into what’s happening inside the box.
Clara Fication: And don’t skip the risk management piece. ISO 14971 applies. You need to think about what happens when the AI gets it wrong — because it will, at some point — and whether your process is robust enough to catch that. A human in the loop isn’t just a nice regulatory fig leaf; it’s often what stands between a good outcome and an incident.
Question 3: Model changes seem to be keeping compliance teams up at night. How should manufacturers handle and control changes to AI models they’re using?
Clara Fication: This is the issue right now, and FDA has been signaling it loudly. For AI/ML-based SaMD, FDA introduced the concept of a Predetermined Change Control Plan — PCCP — in their 2021 guidance, and it’s been refined since. The idea is that you tell FDA upfront what types of changes you expect to make to your algorithm, and under what conditions, so that certain planned changes don’t require a new premarket submission every time. That’s a significant benefit if you use it correctly. But the PCCP has to be realistic and specific — it’s not a blank check for “we might update the model sometimes.”
Art Meta: For the quality-tool side of AI — not SaMD — you’re working within your existing change control framework, but with some AI-specific wrinkles. ISO 42001 distinguishes between planned changes and emergent changes. Planned changes are straightforward: version control, impact assessment, re-validation, approval. Emergent changes are the tricky ones. These are changes in model behavior that occur without anyone explicitly updating anything — because the model is learning from new data, or because the input data distribution has shifted, or both.
Norma Tive: Wait, the model can just… change on its own?
Art Meta: If it’s an adaptive algorithm, yes. Continuously learning systems update their parameters as they process new data. Which means your validated state at deployment may not be your operational state six months later.
Norma Tive: (long pause) That is a nightmare from a document control standpoint.
Clara Fication: It’s why many manufacturers are wisely choosing locked algorithms over adaptive ones for regulated applications, at least until the frameworks mature. A locked algorithm produces the same output for the same input, full stop. Much easier to validate, much easier to control. The trade-off is you don’t get the self-improvement benefits, but for a lot of use cases, predictability is worth more than optimization.
Norma Tive: If you are using an adaptive model, at minimum you need: version control for the model itself (not just the software wrapping it), a defined monitoring protocol to detect performance drift, trigger criteria for re-validation, and a clear record of what data the model was trained on at each version. That’s the floor. Don’t accept a vendor telling you “our model updates automatically to get better” as a feature without asking what the change control implications are for your quality system.
Question 4: How do you validate something that’s, to some extent, a black box? Traditional software validation is built on predictable inputs and outputs.
Norma Tive: You validate what you can observe and measure, and you document the limits of what you can validate. The FDA’s Computer Software Assurance guidance — which replaced the old validation guidance — actually gives manufacturers more flexibility here. The focus has shifted from exhaustive IQ/OQ/PQ documentation to a risk-based approach: what is the intended use, what could go wrong, and what’s your evidence that it works well enough for that purpose? For AI, that evidence looks different than for traditional software, but the logic is the same.
Clara Fication: FDA has been explicit that they don’t expect manufacturers to be able to explain every internal computation an AI model makes. What they do expect is that you can demonstrate performance on a representative, independent test dataset. You know what the model is supposed to do, you know what “good enough” looks like, and you have data showing it achieves that threshold. Sensitivity, specificity, precision — whatever metrics are appropriate for your use case. Then you document how you established those thresholds and why they’re clinically or operationally meaningful.
Art Meta: ISO 42001 adds a useful concept here: AI system transparency. Not transparency in the sense of “we can explain every weight in the neural network” — that’s often not feasible — but transparency in the sense of “we can explain what the system is doing, why it was designed this way, what its known limitations are, and what it should not be used for.” That documentation is actually part of your validation package. The system doesn’t have to be interpretable at the algorithmic level; it has to be understandable at the purpose and limitation level.
Norma Tive: And don’t forget bias testing. Your training data reflects the real world, and the real world has biases. If your AI was trained predominantly on data from one type of facility, one type of product, or one demographic of patients, its performance on edge cases may be significantly worse. Test for that. Document that you tested for that. And if you find performance gaps, address them before deployment or explicitly bound the intended use.
Clara Fication: FDA has cited this specifically as a concern for AI/ML-based SaMD — that performance disparities across subpopulations can represent a real safety issue. For manufacturing AI tools, the equivalent concern is whether your model performs equally well across your product lines, your shifts, your facilities. Worth checking.
Question 5: What’s the single biggest risk manufacturers are underestimating right now when it comes to AI?
Norma Tive: Automation bias. Hands down. The tendency for people to defer to automated recommendations even when the recommendation is wrong, and even when they have information that should override it. You spend all this time implementing an AI tool, it performs beautifully in validation, and then your operators stop applying critical thinking because “the system said so.” The human in the loop becomes a rubber stamp. That’s not a control — that’s a liability.
Art Meta: I’d say the biggest risk is treating AI implementation as a one-time project rather than an ongoing operational commitment. You deploy the tool, you check the validation box, and then it runs quietly in the background while nobody monitors its performance, nobody tracks whether the input data has changed, and nobody asks whether the world the model was trained on still resembles the world it’s operating in. AI systems need active stewardship. The governance doesn’t end at go-live.
Clara Fication: Mine is a regulatory one: assuming that because there’s no specific FDA rule yet for a particular type of AI tool, there’s no regulatory exposure. FDA has broad authority under the FDCA and the QMSR. They don’t need a specific AI rule to cite inadequate change control, inadequate validation, or inadequate post-market monitoring for a tool that’s affecting your quality system outputs. The absence of a bright-line rule is not the same as the absence of expectation. Act like FDA is watching — because eventually, they probably will be.
Norma Tive: Also, don’t fire your quality staff and replace them with AI. I am not a biased party in this assessment at all.
Art Meta: Completely unbiased.
Clara Fication: The most defensible approach — for your quality system and your peace of mind — is to treat AI tools the way you treat any other significant change to a regulated process: with rigor, documentation, and a healthy respect for what you don’t yet know. The technology is moving fast. The regulatory frameworks are catching up. The manufacturers who build disciplined foundations now will be much better positioned when the next round of guidance drops.
Art Meta covers AI management systems and ISO 42001 at Red Hen. Norma Tive writes about ISO 13485 and quality management fundamentals. Clara Fication covers FDA’s Quality Management System Regulation — start with her QMSR series overview if you’re new to the regulation. Have a question you’d like the panel to tackle? Leave it in the comments.