From the diary of an OT pentester

Klaus Mochalski

Hello and welcome to a new episode of OT Security Made Simple. I'm Klaus Mochalski, founder of Rhebo. My guest today is Patrick Latus. Patrick has many years of experience not only in IT, but also in OT security. He has done pentesting and incident response there. But Patrick, just tell us [...] what you've been doing in recent years and how long you've been involved in the topic.  

Patrick Latus

Yes, hi Klaus, thank you very much. My name is Patrick Latus. I've been working in OT for a little over 20 years now, and I worked for almost 20 years for a system integrator for ABB and Siemens control systems. I mainly worked in the energy and chemical sectors, in pharmaceuticals, oil, gas and so on, and have been self-employed in the field for a little over four years now. And yes, I do a lot. Purple Teaming, that is, red and blue, means integrating hardening measures. And yes, attacks on OT systems too.  

Klaus Mochalski

Sounds very exciting. Definitely exactly the area that interests us here. Tell us, you've been around for 20 years. That's a really long time for... I think for many people, OT and OT security haven't existed as fields for that long. Of course, there have always been systems like this, but I think the terminology here is newer.  

If you look back at the development over time, what has changed in recent years in particular? Are there patterns? Are companies maturing? Is infrastructure becoming more secure? Are systems becoming more modern and better patched? What are the trends you are observing?  

Patrick Latus

There are a lot of points that I would like to address. But that's difficult. The word itself has only been around for a few years. It has become a bit of a buzzword, like Industry 4.0. The problem is that the systems have been around for a very, very long time. Many systems are still in use today that date back to the 1990s and simply have a lifecycle of 20 or 30 years. That used to be completely normal.  

But what has been increasingly added for many, many years now is interconnectivity, i.e. interoperability. Everything has to be connected. Values have to be pulled out. KPIs are formed. And these systems, which have a lifecycle of 20 or 30 years, were simply not designed to be secure. They were islands for years. And then someone said: Oh, look, there's a network plug, let's plug it in. Yes, and by the way, management has to be on it too. And remote would actually be quite cool.  

And yes, so in the end some of the systems that don't actually belong there are directly connected to the internet. All the OT people who were responsible for keeping the machines running for years – because at the end of the day, they have to produce – now suddenly have to get involved in cybersecurity. All the IT people who take care of the IT systems on top of it have always said: IT systems, yes, but we don't touch OT systems. If we patch something there, then something will fail immediately. Yes, and so there is somehow a black area and it is actually important to jump in a lot more and solve problems, to do things that are unresolved. Now there is legal pressure behind it. Which is good on the one hand, but on the other hand I see the problem that there is simply a lack of people. As everywhere else.  

Klaus Mochalski

So, what you are talking about here are all problems that I think are often discussed in the OT security community. Are there any changes here, maybe even positive ones? After all, the news is mostly negative. But are there any positive changes that you can specifically observe now? It's always difficult to see the good when you're dealing with the kind of things you're describing. But there must be something.  

Patrick Latus

The good thing is that people are now more interested in it. Of course, there is a downside to that as well. These topics are now also reaching management. I have many customers that I have looked after over the years who always said: Well, everything is running. It can't be as bad as you keep saying down there. All the news that is now coming up has naturally created a big wave.  

Stuxnet. That made a few people sit up and take notice. Of course, it was a super sophisticated attack that required a lot of resources and everything, but it did bring attention to the issue. Basically, the whole thing then picked up a bit of speed and still does. Indeed. But the people are a bit lacking. There is a lack of the right apprenticeships, university courses, people who dare to move out of IT and into OT, put on a pair of safety shoes and a helmet and really go down into production.  

The topic is coming into focus. That's positive. However, it's not only attracting the attention of people who work with it, but also those who want to attack it. So it's a double-edged sword.  

Klaus Mochalski

Yes, okay, so that's the call again. In fact, it's also aimed at young people who are looking for apprenticeships in this direction. [...] I have been working in this field for ten years now, before that I was in classic IT security. And I have to say that OT security has developed into a very interesting field of work. And it still is today, there is still a lot of uncharted territory. That means there are few outdated structures here, there are still real challenges.  

What you describe, namely the combination of IT security knowledge, is something you have to go into in depth to be really well prepared. And I think that's also a major problem in the qualification of skilled workers today. And that in combination with knowledge of the systems that need to be protected. And they are just completely different systems. It's not just a computer. And you can say that a computer is always the same in the end – whether it's a smartphone, a laptop or a server. These are computers with an operating system.  

But here we are talking about machine control. We have sensors and actuators. We have complex networks of such systems with interaction in the physical world. And that is, I think, this combination that makes it equally exciting.  

Patrick Latus

It's really, really cool and I can only encourage everyone: jump into the field, do it with a bit of thought, stick with someone who also knows a bit about it.  

Yes, that's right. You can't just patch the systems, you can't just roll something out. But we need people who are willing to see where medicines are manufactured, in production, where steel is cast, where energy is produced that we take for granted from the socket. But to really work at the grassroots level, to really get down to the plant, where it stinks, where it is sometimes loud. Yes, that's just wonderful, and I love doing it. I'm passionate about it. I'm always happy when someone is interested in it and I can only encourage them to really take a step in that direction. As I said, get in touch with someone who knows a bit about it and who will take you along.  

Help is always welcome. Please don't rush in anywhere with an axe. I also see it sometimes with customers that IT security experts then come along and use a tool like Nessus [note: network and vulnerability scanner software] to scan away any controllers or sensitive sensors. Then production comes to a standstill, and nobody is happy. It sometimes takes days before it can start up again. Be careful, be motivated and interested, but also be careful with.  

Klaus Mochalski

It is a very nice, very nice call. I like that and I would also like to take up the defence of the industry. We are currently talking a lot about critical infrastructure in Germany. And we at Rhebo also have a lot to do with this sector as customers. You always think that these are municipal companies, they are very slow-moving. So, it's either municipal companies or large corporations that act very slowly [... that's what you think].  

But just now in the news – and I want to stick up for them here – the news came that Germany has achieved its climate protection targets. I found that totally surprising. And if you look at why we have achieved this, it is because Germany's energy companies have driven forward the energy transition, so to speak, and not the areas that are lagging behind: buildings and transportation. And that's where we can see what has really become the innovation driver in Germany in the meantime. It is definitely an exciting field of work, regardless of your purely technical qualifications.  

So I'd call on everyone to go into this field. It's definitely exciting and future-proof. To whet the appetite of young people for this area even more.  

Tell us about a specific case that you have personally worked on in the last few months, 18 months, two years, that was particularly exciting, perhaps quite typical or quite atypical. And just tell us what it looks like when you visit customers, when they call you and how you proceed and what you might have found out in a specific case.  

Patrick Latus

The area in which I work is extremely diverse and fascinating. I am currently helping customers to write specifications, which involves including technical details in documents to ensure that the customer gets exactly what they want. I take part in FATs and SATs, which are acceptance tests to ensure that service providers or a large company like ABB, Siemens, Yokogawa – you name it – really do deliver what is needed. You always meet really cool people there. I get called to the tests where the customer says: Here, could you please come by.  

For example, a few months ago we had a coupling at a power plant and they said: We have a coupling to the electricity exchange, we have linked ABB and Siemens control systems.  

I say: What do you mean, linked?  

[The answer came]: Well, an OPC link has been put into operation. For those who don't know what that is, it's a protocol that can be used to communicate with the PLC and the CPU.  

I said: What does it look like? Who was there?  

[They answer:] Two service providers were here who look after both systems and put it into operation.  

Then I say: How long were they there?  

[The customer:] One day. And then it was done.  

And I think to myself: one day is definitely not enough. Can you send me the documentation sometime? Network plans or something?  

[And the customer replies:] Yes, we are still waiting for that. But everything is already up and running.

And I think to myself: Uh-oh.  

We then went there, took a look and yes, it's difficult. Network cables were laid so that it fit. IP addresses set up, but nothing documented. We then found out the passwords relatively quickly, where you could then write values down to the PLC. It's spooky, of course. Yes, but it happens quite often. The networks are super complex, you have to understand what works how, where which roles are in such a control system.  

Klaus Mochalski

It's crazy that something like this still exists. I wouldn't have thought that. I understand that there are established infrastructures that are difficult to protect. But that you could, so to speak, call in a service provider during a modernization project – and that's what it was in the end – and say: Get it up and running!  

And that's an old problem in OT security, where availability or operations always come first. That means the system has to run and in case of doubt it's at the expense of security. You can often find good reasons for that. But in a case like this, it would be really easy to take a step back, take stock first and say okay, what are security best practices in this environment?  

And it's not as if the scenario you describe is particularly unusual or complex. OPC is by no means a modern protocol. It may have been ten years ago, but now it has its own standard. This means that we now know how to implement it securely, even though the protocol itself has certain limitations. But now we know how to do it.  

And then it's crazy that, despite this, service providers are still being contracted to get it up and running.  

Patrick Latus  

Yes, but that happens because the customer – the one who buys – sometimes doesn't have the technical understanding themselves, or they want it to be that way. Then it's sometimes not put in writing. Then the service provider comes and puts it into operation. But very often they are engineering people who program the functions and then write for the PLC. They are not security experts.

You can secure OPC-UA in three different ways:  

a) Without a password – the quickest solution, it works.  

b) Then with a password, which of course is always super short so that it can be easily exchanged, yes. A classic is, of course, a password that is used for other accounts. Like a domain admin or something like that, I've seen that too. Yes.  

Or c), you secure such an OPC-UA with a proper certificate. Yes, even self-signed – better than nothing at all.  

But to find someone who rolls out a proper PKI [note: public key infrastructure] in IT or in OT, with block lists and everything... There aren't many who can do it properly. And then down in the OT, where it's sometimes difficult to store certificates or something like that in ancient devices.  

It's difficult and the easiest thing to do is to put in network cables, no password or a super simple password, and then everyone claps their hands and says: Look, it works! and then that's it.  

Yes, so we need people who really, as you just said, have a deep technical understanding, preferably from IT security, that's fine. As I said, you have to learn the few specifics from OT, which isn't rocket science either.  

For many people, [OT security] still seems like a super-complex topic, some kind of highly sophisticated thing. But these are small controllers that basically do the simplest things, like switching engines on and off and such. They also have protective functions, which are also there, yes. But if I now take a current Siemens portal and download an S7 for the first time, the dialog asks me: Do you want to load with or without encryption? And you can also uncheck the box with the checkmark. And I think to myself: Please do it hard-coded.  

Yes, we are on it. But that will take some time, because technology has to come, people have to get fit. And as I said, we also have far too few people in that area. I see it very, very often that engineering people also install the systems. And if you now say in the Siemens world: Simatic Shell with encryption, please... They have to read their own documentation, and you can't blame them for that. They are engineering people, system people who are really superfit. Unfortunately, they are far too rare. That's the way it is.  

Klaus Mochalski

But I can only confirm that it's not rocket science to look at a controller like that and understand it. That's what we did when we started developing our software at Rhebo ten years ago. We also got hold of controllers. And then people who came from classic IT security and didn't yet know what OPC and Modbus were really got these things on their desks. And we thought, let's see how long it takes. And after a day, they had fully understood the thing and were surprised at how simple the systems are.  

But in the end, it's not surprising when you consider what they look like, what's inside them, how old some of them are and what they do. They're very simple things and relatively easy to understand. That said, most of the expertise needed actually comes from classic IT and OT security. That's really the main effort.  

Patrick Latus

If you understand a few industrial protocols, such as Modbus or Modbus TCP, the successor for network... If you see that okay, you can also put a network on it but then you should also deal with it. What is a VLAN? What is a MAC filter? Just isolate the networks!  

There are also options for making protocols such as Modbus TCP, which by default has no authentication at all, secure and safe to operate. It is possible. As I said, you have to deal with these things, get to grips with them. If you get stuck, please bring in someone who knows a bit about it.  

On websites like Shodan, Censys or something like that, the number of devices still connected to the internet with an S7, with a Modbus TCP, is insane. So, if you get the idea to just briefly connect something to the internet to test it or something like that: Please don't do it! Within two hours it will be available all over the world, and then it will come crashing down on you.  

Klaus Mochalski

You can't be too careful.  

I want to come back to what you said. What triggered the customer you just described. You describe it as a lack of expertise on the part of the customer, perhaps also on the part of the purchasing department, or maybe a lack of awareness.  

There is a debate going on in the OT security community right now: Are we talking about the problems too much and are people getting tired of hearing about them? And on the other hand, there are relatively few major incidents that are also being publicized. And many people say that there is a discrepancy and that we shouldn't go overboard here. But now you're telling me that awareness of the need to address security here is not even there yet. So, what is it from your point of view?  

Are we talking too little or too much about OT Security?  

Patrick Latus

In management, it is definitely there, but not in OT. As I said, this is because in OT, many people or most people have been responsible for keeping the machines running for years. Now someone comes along and says: You also have to do cybersecurity. And they say: Yes, what should we do? The machine is running!

Now to tell someone who has been maintaining, operating and programming a machine for years that they should now set up encryption, change all their passwords and check who is logging in remotely... When I look at a few exhibition stands at it-sa last year, there are still solutions that allow a remote connection directly down to the bus via AnyDesk Teamviewer. There is no monitoring, there is no 2FA.

Nothing. Yes. When I talk to the people or the developers, they say: Yes, it's super convenient, you can get in directly – remote support. It's crazy that products like this still exist and are sold.  

With my customers, we are now trying to establish centralized remote solutions that are ISO-compliant – with monitoring, session logging, emergency disconnect and everything. But it's going to take a while for this to really sink in.  

It's on the right track. And there are also many who say that processes have to be established, this and that. Yes, but at the end of the day someone still has to go out and set the appropriate check marks and provide support if something doesn't work.  

And there is still a long way to go for manufacturers and service providers. So, in OT, everything always takes a bit longer. But it could pick up a bit of speed.  

And yes, the fact that many incidents are not reported is simply because they are not seen. The technical means are sometimes lacking. Then, of course, there is the fear of being reported and exposed.  

If you look at the BSI [note: German authority for informational security] report on the security situation [in Germany], you will see that industrial plants have been included in it every year for many, many years. There is also a significantly higher number of unreported cases in facilities that are actually supposed to report incidents but simply do not have the infrastructure, either technically or procedurally, to report such incidents easily.  

And we have to be a bit more open about this. Because if something like this is reported, we can learn from each other and protect ourselves better. Train people. Also motivate them to support in this area. Because, as I said, it's a supercool area, and I can only recommend it to everyone. Yes.  

Klaus Mochalski

I find what you say very exciting, very exciting. As you just said, I also talked a lot about processes in the podcast. What role management plays, what role the CISO plays, how I manage to use the know-how, especially in larger organizations – because that inevitably exists in larger organizations. In smaller organizations, it is potentially difficult. There may be zero or one person who is somewhat familiar with the area.  

Patrick Latus

The one IT manager, exactly.  

Klaus Mochalski

In this respect, I find it interesting to see this different perspective, that we really do have a problem with skilled workers and also a problem with expertise among service providers and on-site personnel. And we have to compensate for that. And I do understand that of course, just having good processes – an ISO 27001 or IEC 62443 that I put on top of them – won't get me very far if I don't have the people on site. I can get the service providers who might give me the approval, but they would recognize it and then say: You don't get certification for this.  

Patrick Latus

They actually should. Yes! But now I have customers who are critical infrastructure. An auditor turns up and says: Guys, I'll be completely honest. I'm a physics professor. I've just had one like that.  

[The customer] said: We had a physics professor here who said straight away in the first half an hour that he was a physics professor. He did an additional exam, is now an ISO auditor and he looked at all the processes, but didn't check anything technical.

Where the customer says: Well, it's difficult now. On the one hand, we're glad we're through. On the other hand, we're not happy at all, because he should have found something, because then we could have argued to management that we still have real weak points there.

So it's super exciting. It's super exciting, yes.  

Klaus Mochalski

Very nice. That's definitely a very interesting perspective that you're bringing to the table here, Patrick. It's a bit complementary to what has often been discussed in the past here in the podcast. Not only standardization, processes, management, attention is needed, but also expertise at the working level. And that's what we need.  

And I think that's a very nice conclusion and closing words for this episode today. Thank you, Patrick, for being there and I'm looking forward to doing another episode with you, where we can perhaps take a deeper look at the technical topic.  

Patrick Latus

I'd love to! Thanks a lot! It was very exciting. Greetings to all listeners out there. Dare to do it! It's really fun to work at the grassroots level, where our everyday products come from. And yes, it's super exciting. I can only encourage you. Thanks a lot for having me and hopefully see you next time. Great.  

‍