In the past, manufacturing systems went relatively unscathed by Cybersecurity incidents. Simply, older manufacturing systems and processes were seldom networked and even when they were, there was no reason for those systems to communicate outside of the factory floor. Manufacturing company IT departments isolated the manufacturing networks from the corporate networks. Problem solved. The major focus was on physical security alone. The thinking was, if the process network was not connected to any other network, the only risk is someone plugging in directly. This was true at the time.
With the introduction of “Smart Manufacturing” the defined lines between IT and OT (Operational Technology) began to blur. Instead of complete isolation, now manufacturing systems are on the same LAN as corporate systems. The merging of these isolated networks introduced a wide variety of security challenges.
According to a Deloitte study conducted in 2019, the security threat to these manufacturing companies is well understood:
According to the same study, cyber security is the #1 risk to OT systems. More specifically, the #1 risk is the merger between IT and OT. These risks became even more apparent over the last few years as Cyber Security incidents brought with them a massive amount of production loss as well as numerous safety incidents including loss of life. The other major concern is the specialty nature of the manufacturing systems themselves. The companies that install and service the specialty equipment also install and service the specialty equipment at your largest competitor. The risk of data exfiltration is extreme.
Security Challenges
The largest and most obvious risk is to maintain network isolation electronically instead of physically. If a random individual can plug into your HQ network and access a remote plant, you don’t have security. This is typically handled by next-generation firewalls at the HQ and at the plants. Systems that must cross the boundary can either be allowed on the firewall, or jump boxes can be setup either via VPN or directly. This is all relatively straightforward and with a good design, the problem virtually takes care of itself.
The next challenge is access via the factory floor. This security is virtually mandatory and significantly more challenging. This article will focus on this second piece. A NAC solution such as Cisco ISE is definitely the fix, but it introduces a layer of security that can interrupt manufacturing.
ISE For Manufacturing Risks
The main risk is simply, the stakes are much higher in manufacturing and the pressure to have a perfect solution is enormous. If Mary from accounting can’t get on the network for 15 minutes, the world keeps turning. If a PLC that controls your main production line can’t get on the network for 15 minutes, it can and frequently does spell doom. The loss of a single device can stop the line, require a fix to the actual problem, clean-up, and a full restart of the process. This is not a 15-minute outage, it is hours. In addition to the obvious loss of revenue, there are peripheral concerns that may not be immediately apparent.
Reputation Hit: Did the outage cause a missed deadline with a key customer?
Employee Satisfaction: Did the outage cause employee anger and frustration? Loss of hours?
Process Changes: Is there a cost involved with planning for outages? Extra warehousing? Extra shifts?
Lost Product: Does an abrupt outage cause product or material loss?
All of these factors have to be considered with the actual cost of a production line outage. ISE Installation and configuration is somewhat trivial and nearly anyone can learn how to do a basic deployment quickly. However, this truly doesn’t account for 5% of the knowledge and experience required in the manufacturing space.
ISE Deployment Challenges for Manufacturing
The list is long and appears kind of painful, but it doesn’t have to be. Many of these concerns can be mitigated upfront with knowledgeable staff, ample testing, and a slow and steady approach to deployment.
Profiling
Profiling is the ISE process that determines what type of device is attaching to the network. This is a walk in the park for things like Windows PCs, printers, and AV Equipment. However, it is extremely difficult in manufacturing environments. ISE cannot be expected to know everything there is to know about every PLC or Drive created in the last 20 years. These devices must be profiled correctly and thoroughly, or it could lead to an outage.
Cisco does an extremely good job identifying most devices, for example your Rockwell devices are very-well documented. In fact, Cisco has a special Profiling database specific to manufacturing. However, there is zero chance the database will cover every single device in an environment.
WAN Outages and Failover
From a cost perspective it is typically untenable to deploy ISE Servers at every plant within an organization. The good news is, with some changes to the “standard” ISE configuration, this is seldom required. The problem typically lies in the fact that many plants connect to ISE via a WAN link to a regional data center or HQ. Often the plants are located a bit off the beaten path as the best friend of manufacturing is inexpensive land, water, and power. Off the beaten path means few ISP choices, and less than optimal support.
This problem is exacerbated by the fact that quite a bit of processing equipment cannot tolerate an outage, even sub-second. Many devices have built-in safety features that will immediately shut down when even the shortest network outage is detected. For obvious reasons, this is a good thing, but it does make the stakes even higher. From there you can add in Precision Time devices and 100% network availability is now absolutely required.
Along the same lines, many facilities run 24/7. What happens if a PLC goes down and the plant has to plug in a new one at 3am? The answer is they have to call an ISE engineer to authorize the new device before the line can startup again. If the ISE Engineer is a heavy sleeper, you have a sad story until 8am the next day. With a strong process and good design, this issue can be corrected, before the first endpoint logs in.
Finally, what happens if the WAN is down for an extended period? Failover to an authorized connection must be transparent! This is not the default behavior and every facility will require custom configuration and testing to ensure not a single packet is dropped.
Upgrades and Maintenance
The 24/7 nature of manufacturing introduces an issue choosing a time/date for updates. Every system every built has some level of maintenance that has to be done and ISE is no different. This is a significant challenge when plants run 24/7 or are global and the sun never sets on your ISE deployment. In a typical office environment, it is not uncommon to run updates and maintenance in the evening, over a weekend, or even during a holiday, if required. With global manufacturing, there is no good time.
The fix is to ensure you bake in a plan during the initial deployment. A thorough understanding of ISE itself, switch platforms/code versions, and authentication/authorization timers is required. This may not completely eliminate the need for downtime over the course of years, but it does certainly minimize it and all stakeholders will know exactly what to expect.
Closing Thoughts
The key to a successful ISE deployment in a manufacturing space is experience. This experience has to come from previous successful deployments, of course. However, it is equally important that your ISE engineer completely understands your business, your processes, and your expectations. Unfortunately, all of this is frequently overlooked. A solid ISE engineer may whip up a completely cookie-cutter deployment and call it a win. The engineer may have done 50 office deployments the identical way and never had a single problem. Frequently, the engineer or consulting company will bid a project based exclusively on metrics that are not relevant to manufacturing. In other words, by the time the “template” is applied to your business, the engineer is simply out of hours. This either leaves you with frequent outages or worse yet, the engineer has no choice but to start taking shortcuts. In the consulting world, the pressure to close a project can be intense.
It is not uncommon at all for CDA to get engaged in an ISE project AFTER another firm did the full deployment. We’ve seen it all…Issues ranging from frequent and prolonged outages to a previous engineer running out of hours and simply removing ISE anywhere and everywhere that had a problem. The result could be a large global ISE deployment doing absolutely nothing!
Would you like to continue the conversation? Send us a message!