While we’ll discuss AI, the security principles outlined here are equally crucial for all computer-controlled manufacturing systems, whether they incorporate AI or not.
AI’s Growing Role in Controlling Devices:
As AI starts entering more workplaces, it is crucial to recognize that AI will become more interconnected with hardware devices in your organization. You might want AI to control room lighting and air conditioning to make it voice-controlled or adapt to the changing activities in the room. AI can also control massive machinery, including robots and high-powered lasers for cutting steel. We’ll all be surprised at how many real-world tangible controls AI can assist. For AI to control devices, computers must drive the machines. Threat actors could exploit weaknesses to disrupt companies, damage equipment, cause expensive delays, and worse.
Machines Driven by Computers, Including Those Running AI and Traditional Computer Control Systems, Introduce a Security Threat:
As AI becomes integral to your operations, remember: Everything from climate control and identity detection to robots and laser cutters hinges on computer systems. AI’s potential is vast, and its growing adoption means more devices linked to our networks.
However, this surge in AI adoption produces an often-overlooked danger that all organizations with industrial controls must consider. The computer systems hosting your AI and traditional solutions can become obsolete faster than the devices they control. Neglecting to update operating systems and using other security controls exposes your organization to cybersecurity threats. While devices might seem to run smoothly, the escalating sophistication of cyber attackers can’t be underestimated.
Executives: Unchain Your IT Pros from the Security Limitations:
Is your IT Team prohibited from applying critical cybersecurity updates to operating systems or upgrading to supported operating systems on workstations that control instruments, lasers, robots, and other machinery? If they are, those workstations pose a security threat to your organization.
Executives must understand that using workstations with old operating systems or without the most recent critical security updates is a significant security risk. In some cases, executives must ask the IT Team if they have encountered this situation. Sometimes, executives are inclined to delegate decision-making to the IT Pros. Instead, the IT team must alert the executives of the pros, cons, and expenses. The executives need to decide if it makes sense to pay to upgrade the applications that control robotics, manufacturing, or other equipment on a network.
Three Definitions:
In case nobody’s explained these terms, it is essential to differentiate between upgrades and updates:
- Operating System Upgrades: An example is upgrading from Windows 10 to Windows 11. Newer operating systems often have more security features. Microsoft and Apple will naturally be tempted to assign their best and brightest people to develop and update the newest operating systems, so they eventually drop support for old operating systems. Unsupported operating systems are designated EOL (End of Life.) Using an operating system after it is no longer supported is a significant security risk.
- Operating System Updates, a.k.a. Patches: Security updates are rated by the severity of the security risk and how likely an attacker will exploit the weakness. Critical security updates are the most important to apply. Staying up to date with patches can be a significant struggle in many situations.
- Application Upgrades: Upgrades to new versions of the software that controls devices such as CNC machines, robotics, lasers, laboratory equipment, instruments, or any other hardware that connects to a computer.
The Shocking Reality:
Some applications that control devices may prohibit operating system upgrades and security patches. The applications might break if the IT team deploys the patches or upgrades the operating systems. Sadly, as reckless as it seems, some companies that create applications to control machinery will no longer provide technical support to your IT team if the operating system on the workstations is upgraded or has security patches. Their software developers may be too busy to create flexible, secure applications and are forced to focus strictly on functionality.
Depending on the application vendor, paying for an upgraded version of a controller application can be very expensive. Fortunately, sometimes, the upgrade charge is reasonable or free. Sometimes, no upgrade is available to permit operating system upgrades or critical security updates.
Another consideration is the risk that upgrading might interrupt manufacturing flow if the upgrading process requires extensive troubleshooting or potentially interrupt production. When equipment operates 24/7, the IT Team is under more pressure since there is no downtime for maintenance.
If the new application’s user interface significantly differs, shop floor personnel might require additional training. Inadequate training can lead to costly mistakes and safety issues. Scheduling training will affect the timing of deploying the new applications.
So, as you can see, when robotics, scientific instruments, lasers, manufacturing, or other equipment works just fine, upgrading the application offers no valuable benefits, and the IT team is busy, we find during audits and security assessments that many manufacturing organizations have outdated operating systems or need critical cybersecurity updates.
The organization’s executives might accept the risk, especially if compensating controls are in place.
Alternative Tactics Increase Security:
Using compensating controls in networks is essential because systems sometimes have significant vulnerabilities before updates are released or installed. Compensating controls are even more essential to help protect workstations if patches are missing.
Compensating controls include, and are not limited to, isolating the machines that control robotics, manufacturing equipment and scientific instruments on a separate network away from your network. That separate network must have limited connectivity to only allow traffic to and from the specific devices necessary and limit the kind of data and how it traverses the network to reduce the attack surface and make it more difficult for a malicious program or third party to access that instance or device. I sometimes refer to this tactic in keynote presentations as creating filtered subnets.
Another compensating control is to harden the unpatched or EOL machines by removing all applications except those essential for the equipment’s operation. Examples of applications that must be removed include browsers and email clients since they are common vectors for successful attacks. If the employees operating those devices require internet and email access, consider adding a separate workstation that is patchable for email and web access.
EDR/XDR (Endpoint Detection and Response / Extended Detection and Response) technology is another helpful control. It involves installing a small program called an agent on each computer. The EDR/XDR agent monitors the system’s software, services, and behavior for any signs that threat actors might have already compromised the computer. If the EDR/XDR tool detects an IoC (Indicator of Compromise), it can respond by interrupting the process. When tuned to avoid false alarms, the best response is to allow the agent to effectively quarantine the workstation from the rest of the network until the IT team can investigate. This helps prevent attackers from spreading to more hosts.
However, it is common for IT teams to succumb to the danger of relying too heavily on EDR/XDR to protect their organization and, therefore, neglect implementing other industry best practices to protect systems. Threat actors often set up EDR/XDR tools on their test networks to find ways to circumvent the protections. So, even if your EDR/XDR tool says everything is safe, it doesn’t necessarily mean threat actors aren’t active in your network.
To combat this, companies commonly conduct yearly red-team exercises, performed by exceptionally skilled IT teams that regularly perform these exercises and know the tricks and practices real-world threat actors use. These exercises are designed to test the effectiveness of the detection and response process. These exercises look for weaknesses in EDR/XDR and help keep the IT team in practice, ensuring they’re better prepared in the case of an attack.
Depending on your budget, if $20/user/month for EDR/XDR is not feasible, know that the other cybersecurity controls in this article, such as careful hardening and segmentation with very restrictive filtering, are much less expensive than EDR/XDR and have little if any ongoing expense. I don’t want to diminish the usefulness of EDR/XDR tools. If you are on a tight budget, unless your cybersecurity policy requires EDR/XDR, you might choose to focus on other compensating controls.
The IT Team must alert the executives about the expense of upgrading applications, isolating the shop floor instances on a separate network, deploying an additional network for web and email access, training users and operators, implementing EDR/XDR tools, and other expenses. Include time estimates along with financial estimates. Then, the executives can make an informed decision, and IT can follow their instructions and ask for support as necessary.
Step-by-Step Guidance for IT Teams:
Acknowledge that it can be a significant challenge and sometimes practically impossible to ensure that all workstations run with a current OS and that all critical security updates are applied. But keep applying updates if possible.
Inform your executives whether your team has time to make these changes. IT teams must alert executives of the time and expense involved. The executives will have options such as adding more IT professionals to augment the team, postponing other projects, or accepting the risk of continuing with unpatched systems or EOL OSs with the compensating controls listed below.
Explore all technical, training, and expense changes before upgrading applications.
Ask your supervisor to delegate the price checking to someone outside the IT department if feasible. Your IT team is very busy, so checking the prices might cause the upgrade to be delayed. It can be time-consuming to check with the robotic, manufacturing, and scientific equipment vendors to find the pricing for upgrades to their applications that control machinery.
Investigate more than the pricing. Ask about changes in the upgraded applications affecting the user interface and user experience. Ideally, the upgraded application software operates similarly and has the same interface. Unfortunately, some manufacturers significantly change the user experience when they upgrade their applications.
If users will need training, identify a trainer.
Determine how scheduling the training will affect the deployment timing.
Involve executives in decision-making and send them regular reports about the project’s progress.
Implement compensating controls on the workstations because of the high cybersecurity risk of missing critical patches or using EOL OSs. Compensating controls aren’t a replacement for missing patches, but the controls can help tremendously.
Remember that attackers can exploit security risks long before they are discovered. Only when the vulnerability is discovered will the operating system and application developers know to create or release patches to seal that security hole. Refrain from relying on patches as your sole security control for application software and operating systems.
Strongly consider isolating shop floor machines on a separate subnet, especially those you are prohibited from patching and those using EOL OSs. Isolate that subnet completely with an air gap or utilize aggressive filtering at the switch or router to limit traffic to only the required source, destination, ports, and protocols.
Additionally, hardening the workstations against attacks is strongly recommended.
Remove or restrict web and email access. This is one of the most effective ways to harden workstations, as web and email are two of the most common vectors for malware.
If the workers at those devices need access to the web and email, consider deploying a separate workstation to their station they can use for web and email. If feasible, that workstation should not be on the shop floor network. If you put those workstations on the equipment network, you would need to allow email and web traffic, and modifying access control lists to allow more sources, destinations, ports, and protocols can significantly reduce the security you would otherwise introduce to the equipment control network. Strive to exclude TCP ports 80 and 443 on the AI device network while allowing full functionality of the AI and other computer-controlled devices.
Be sure you limit the sources of inbound and destinations of outbound network traffic to the absolute minimum. If you need to run new cables to facilitate the additional workstations for web and email at the workers’ stations, then running new cables might be a significant investment. Deploying a WiFi network for email and web access might be more economical. Keep the key secret. If you share the WiFi password, workers might connect other devices to the equipment network and compromise security. Completely blocking email and web access and access to external IP addresses will hamper the workers on the manufacturing network from exposing the hosts to many threats.
Strongly consider using EDR/XDR tools, along with the Red Team Exercises, to help ensure the configurations’ effectiveness and allow your IT team to prepare for actual emergencies.
Summary:
Protect workstations that control hardware such as robotics, pharmaceuticals, lasers, and scientific instruments, regardless of whether they utilize AI. This helps ensure the safety and operability of your systems, protecting your organization and workers.
Subscribe to maximize your executive potential with Foster Institute’s E-Savvy Newsletter, packed with practical IT security solutions and actionable strategies for success: https://fosterinstitute.com/e-savvy-newsletter/
(Image source: Bing. Learn more at [Bing.com].)