Why manufacturing AI must show its working

An AI model can be statistically accurate and still be operationally useless if the people running a factory do not understand why it has reached a particular conclusion. In manufacturing, that gap carries real consequences because recommendations can alter maintenance activity or change the way a process is run. When the output of a model begins influencing physical operations, confidence cannot rest on an accuracy score alone.

Duncan Bain, Principal System Engineer at SAS, argues that the consequences of a poor AI recommendation are much harder to contain in manufacturing than in many other applications. A wrong decision can affect equipment, product quality or the continuity of production, which puts greater pressure on manufacturers to understand what sits behind an automated recommendation. The more authority an AI system is given over a physical process, the stronger the evidence required before operational teams can reasonably be expected to act on its output.

“Accuracy is table stakes,” Bain says. “If it is not accurate, nobody is going to believe it. But you also must show how you are leveraging the expertise of the people on the ground. We know data and AI well, but we are never going to be as expert in the process as the people who work with it every day.”

Explain the decision, not the mathematics

Operators are ultimately being asked to decide whether an AI recommendation is credible enough to influence production. For that judgment to be useful, the system must present its reasoning in a form that relates directly to the conditions they can observe on the factory floor.

“You do not necessarily need to understand the nuts and bolts of how it works under the hood,” Bain says. “What is critical is understanding that if I make a change, what impact is that going to have on the process, and is there evidence to show that? You should be able to see that if I do something here, it has an outcome there and marry those two together.”

Bain points to work with a major pulp and paper manufacturer where process engineers were directly involved in identifying which operating variables mattered to the production outcome. Rather than handing the problem entirely to data scientists, the engineers could see which sensor data corresponded to the conditions they understood from the plant and how changes in those variables affected the resulting model. That involvement gave the people responsible for the process a clearer connection between the analytical output and the physical behavior they were trying to improve.

“I have worked with data scientists who get very excited about a 0.0001 percent improvement because they have done something very clever behind the scenes,” Bain says. “But if that does not have a material effect on rejection rates, shipping or how the overall process works, it has not really delivered a business outcome. A black box that nobody on the floor understands is a much harder sell than something where people can see how you get from point A to point B.”

Manufacturing context cannot be automated away

The limits of explainability become clearer when an AI recommendation carries an immediate operational cost. Predictive maintenance is a good example because acting early may mean interrupting production, reallocating maintenance resources or bringing forward expenditure before a failure has occurred. The value of the prediction therefore depends not only on identifying abnormal behavior, but on whether the manufacturer has enough context to decide what should happen next.

“We are always measuring the symptom, not the problem,” Bain says. “A sensor might tell you that a piece of equipment is vibrating a lot, but it does not tell you why. You still need boots on the ground to look at it and make an informed decision about whether an intervention is worthwhile.”

As manufacturers deploy more AI across production, manual oversight becomes harder to sustain. A handful of models can still be reviewed individually, but once that number grows across multiple processes and assets, manufacturers need a systematic way to understand how those applications are performing. The issue becomes how to maintain oversight across a growing model estate without increasing the human workload at the same rate.

“You have to have domain expertise to tell whether you are automating the right decisions,” Bain says. “Model governance is not about replacing that expertise. It is about making sure that once you have deployed a model, it continues to perform in the way you expect.”

Bain argues that this monitoring should happen as close to real time as the application requires. A computer vision model may need checks that a camera has not moved out of alignment, while sensor-based models may require recalibration as equipment and operating conditions change. In a manufacturing environment, the real test is whether the model continues to produce the expected process outcome as those conditions evolve, rather than whether its behavior still matches the conditions under which it was originally deployed.

Trust must survive the first mistake

Experienced operators are being asked to incorporate AI into decisions for which they remain accountable, often after years of relying on their own knowledge of the process. Any new system therefore enters an environment where established judgment already carries considerable weight, particularly when production, quality or equipment performance is at stake.

“Trust is a currency that is hard to earn but easy to squander,” Bain says. “Run the model in parallel with what you normally do, because people are driven by results. An operator needs to see that an action informed by the model delivers the outcome that was expected. If you can draw a straight line between the recommendation, my action and the result, that is a much better place to start.”

Manufacturers also need a response for the point at which a trusted model begins behaving differently. Excessive false alarms can create a ‘boy who cried wolf’ effect, while silent deterioration may be more dangerous because employees continue acting on recommendations after the conditions that made the model reliable have changed. Controls around drift therefore need to define what happens when performance moves outside acceptable limits, whether that means retraining the model, investigating the underlying data or returning the decision to a person. Without that process, trust can persist for too long in a deteriorating model or disappear too quickly when the problem could have been corrected.

As AI moves from recommendation toward execution, mistakes become harder to contain because the system may influence the process before a person has reviewed the decision. That changes the governance challenge from understanding an analytical output to controlling what an automated system is permitted to do within the production environment. Manufacturers therefore need to think about autonomy as an operational control issue, with clear boundaries around which decisions can be delegated and which still require human intervention.

“Explainability becomes more important, not less,” Bain says. “If you deploy an agent, somebody has instructed it what to do, given it the data and set the incentives. There still must be a clear line of responsibility between that agent and the people or organization that trained it and defined what it is trying to achieve.”

Manufacturing processes are tightly interdependent, with the output of one stage becoming the input or constraint for another. A change made in one part of production can alter conditions several stages further downstream, sometimes in ways that are difficult to anticipate from the original decision. Introducing multiple agents into that environment adds another layer of decision-making to an already connected system.

“One of the biggest risks is setting incentives for an agent in one part of the process and letting it do its best to optimize that part without enough context,” Bain says. “That can have a terrible effect downstream because the next agent may also be optimizing only for its own part of the process. We can get ourselves into a lot of trouble if we do not manage that context problem.”

As AI acquires more authority, explainability stops being something manufacturers need only when a model produces an unexpected answer. It becomes part of the operating discipline around automated decisions, giving engineers a way to understand what influenced an action and whether the system is still behaving within acceptable boundaries. Industrial AI will therefore be judged not simply by whether it can produce the right answer, but by whether the people responsible for production have enough evidence to trust what happens next. In a factory, showing the working is part of what makes intelligence usable.