Firewall Rules That Stop Production: Change Control for OT Firewalls

In an enterprise network, a bad firewall rule means a broken application and an angry ticket. At an industrial boundary it can mean a lost historian feed, an HMI that stops updating, a batch that fails, or an operator who no longer has view of a process they are responsible for. The rule itself is not more dangerous. The consequence is.

This is why OT firewall rulebases become permissive and stay permissive. Someone added an any-any rule during commissioning to get the plant running, and six years later nobody will touch it, because the person who tightens it owns whatever breaks. That dynamic is a governance problem wearing a technical costume. Our OT security best practices guide covers the wider program context.

Why OT change control is different

A change process that survives a plant

• 1. Baseline first. You cannot assess a rule change without knowing current traffic. Monitor the conduit for a period that covers a full operating cycle, including startup, shutdown and maintenance activity. Our guide to building an OT monitoring program covers this.
• 2. Identify the owner of every flow. Not the IP address. The person or function that depends on it. Flows with no owner are the ones to question, and the ones most likely to be a surprise dependency.
• 3. Review as security and as process. A security engineer and someone who understands the process should both sign off. Either one alone will miss something the other would catch.
• 4. Deploy in monitor mode. Log what the rule would have blocked, for a full operating cycle. This is the step teams skip and the step that prevents most outages.
• 5. Enforce in a window, with a rollback. A tested rollback, timed, with the person who can execute it present. Not a plan on a page.
• 6. Record the evidence. What changed, why, who approved it, what the monitor period showed. This is your audit trail for IEC 62443 or NERC CIP, and it is also what lets the next engineer understand the rulebase.

Fail-open or fail-closed

This decision is usually made once, globally, at install, by whoever racked the device. It should be made per conduit, deliberately, with the consequence written down. Fail-open preserves production and loses the control, which may be right for a path where loss of the control is tolerable and loss of the process is not. Fail-closed preserves the control and may stop the plant, which may be right where the traffic should not flow unexamined under any circumstances.

What is never right is not knowing which yours is. Ask the question about your own boundary before an incident asks it for you.

Tightening a permissive rulebase without an outage

The work is slow and that is correct. A realistic sequence for a plant with years of accumulated rules:

• Order conduits by consequence, lowest first. Enterprise reporting paths before control paths. Safety paths last and only with safety review.
• Take one conduit per change window. Resist bundling, because a bundled change that breaks something gives you nothing to roll back selectively.
• Convert one any-any rule per cycle into an explicit permit list derived from the monitor logs.
• Document each flow as you authorize it. The documentation is the deliverable as much as the rule is.

Expect this to take quarters, not weeks. A team that tightens four conduits properly in a year is doing better than one that attempts the whole rulebase and stops after the first unplanned outage. Our OT/IT network segmentation audit guide covers how we scope the starting position.

Why Choose Arista Cyber

We tighten rulebases on running plants, which means we plan around outage windows, insist on monitor periods, and bring safety review into changes that touch protective systems. Our consultants hold functional safety credentials alongside OT security certification, so a change we recommend accounts for its effect on the safety case. See our case studies and our OT cybersecurity services.

Next Steps

Pick your lowest consequence conduit and run a monitor period on it. One conduit, done properly, builds the evidence and the organizational confidence to do the next. Starting with the hardest boundary is how tightening programs die after one incident.

Common Questions

Can a firewall rule change really stop production?

Yes. Blocking a flow an HMI, historian or control application depends on can cause loss of view, failed data collection or a halted batch. In OT the availability consequence of a wrong block is often larger than the security consequence of the traffic.

How long should a monitor period be?

Long enough to cover a full operating cycle, including startup, shutdown and maintenance activity. Many plants have traffic that appears only during a turnaround, and a two week monitor period in steady state will not see it.

Should OT firewalls fail open or fail closed?

Decide per conduit based on consequence, not globally at install. Fail-open protects production and loses the control. Fail-closed protects the control and risks the process.

Who should approve an OT firewall rule change?

Both a security reviewer and someone accountable for the process. If the change touches a path to a safety instrumented system, add safety review. A single approver from either discipline will miss things.

What do we do about the old any-any rule nobody understands?

Narrow it in monitor mode rather than deleting it, and let the logs show you what uses it over a full operating cycle. Deleting on assumption is how tightening programs cause the outage that ends them.