How does OT security in the water sector work?

Klaus Mochalski

Hello and welcome to a new episode of OT Security Made Simple. I'm Klaus Mochalski, founder of Rhebo, and I'm delighted to be podcasting from my favorite location, the Rhebo office in Leipzig, for the first time in a long time. My guest today is Rainer Stecken. Rainer is a consultant for information security at DVGW Service und Consulting. And what exactly he does is best told to us by yourself, Rainer.

Rainer Stecken

Yes, good morning Klaus. I'm the Information Security Officer at the German Gas and Water Association. This is an association of municipal utilities, individuals and authorities, around 14,000 members in total, who look after our gas supply and our water supply. And security is of course a major issue here. Security of supply on the one hand, but IT security on the other. As a consultant, I work for municipal utilities, which means I have a dual function. On the one hand, I am an Information Security Officer for the association itself, but also an Information Security Officer for a municipal utility, where I also have this function and am currently advising others in connection with the implementation of the new legislative activities. NIS2 is the key word, and what they should do as quickly as possible.

Klaus Mochalski

That brings us right to the topic. We are always talking about OT security, i.e. the part of IT security that involves technical systems. I think everyone can imagine that there are a lot of these in water management systems. And you've already mentioned it. This is about security of supply and ultimately cyber security, i.e. OT security is a part of security of supply that needs to be considered. In this respect, it fits very well into general risk management, but is still a new topic for many.

We had already talked about this beforehand - and it had also been a topic in some previous episodes of this podcast - that there is a relatively limited number of real incidents, of real attacks on technical systems in the area of critical infrastructure. At the same time, we know that the threat level is quite high and that the facilities are also vulnerable to attacks. In other words, there is a certain discrepancy. What is the situation in the water industry from your point of view and your own experience? What incidents have you observed in the last two or three years? What do they look like? How many are there and what is their quality? What types of threats do we see there?

Rainer Stecken

Fortunately, it's been only isolated incidents. Or I would say isolated incidents that have occurred in the last two or three years. There were a few incidents abroad. A few cases have been confirmed in the USA. Here in Germany, the biggest incident occurred on June 12, 2022, when a service provider of the Darmstadt and Mainz municipal utilities was affected, which of course caused quite a stir. Incidentally, the DVGW was also the victim of a cyber security attack on the same day, but only to a limited extent. On the one hand, we were well prepared, but on the other, we were lucky. There are occasional reports from smaller water supply companies. One example would be Hochsauerlandwasser, which was affected, but fortunately never in the OT environment.

Klaus Mochalski

Tell us about this one major incident that happened. So was it a ransomware attack against the IT infrastructure, or what exactly happened?

Rainer Stecken

This was a ransomware attack against the IT infrastructure. It was a corresponding cloud provider, i.e. where the services of Stadtwerke Mainz and Darmstadt and Waste Disposal Frankfurt were located. I can't think of anyone else who used this service provider off the top of my head. Their infrastructure was massively affected. Data was also leaked, which means that there were actually attempts to contact customers of the relevant companies afterwards. So it was relatively severe.

As I said, the critical infrastructure itself, the actual supply service, was fortunately not affected because this is generally air-gapped. So you say okay, everything that concerns the office, that has to do with billing, that has to do with customer contacts, we do in this network, and we keep the rest completely separate wherever possible. Of course, there is usually an exchange of data. Of course, because I always have to exchange data for billing or measurement purposes, which then also interests me in IT. But so far, this has not led to an attack in the sense that an attack has been implemented in the direction of the OT. Incidentally, we were also attacked by ransomware.

Klaus Mochalski

Yes, in other words, this example actually shows or confirms what the statistics show, that most attacks do not target the infrastructure of critical infrastructure operators, but rather the IT infrastructure and that OT tends to be a by-product, that poor internal configuration leads to the attack spilling over, which does not seem to have happened in this case. Fortunately.

Rainer Stecken

Yes, exactly.

Klaus Mochalski

And the other smaller attacks? I interrupted you. What else is there? Are there any anomalies that you can observe or are these similar incidents?

Rainer Stecken

These were actually similar incidents. I remember Hochsauerlandwasser, where the IT service, which was operated jointly with another municipal utility, was also affected. But the OT itself was not. And the OT is often even separate when it comes to the area. This means that they don't have a control center where the data accumulates somewhere, but rather each water utility is separated. And some of these can be operated separately - and some are still operated separately. In this respect, security by design, so to speak. Simply because it doesn't actually correspond to the current state of OT.

Klaus Mochalski

So the fact that digitalization is still lagging behind in this area protects us a little. Unfortunately, thank goodness. But it also means that threat scenarios, i.e. the actual risks, do exist. They also happen, and you have to be prepared for them. Such incidents, which do not directly lead to damage in the IT domain, are always a good opportunity to check and test your own capabilities and see how well prepared you are.

What is the situation there, what can you say about the level of preparation in this area? Are the people well trained? Is it clear who has which tasks? The cooperation between the service provider you mentioned and the internal staff? Also the cooperation between the OT, i.e. the utility operators and IT? What worked and what perhaps didn't work well? What needs to be worked on?

Rainer Stecken

It feels like people are better trained to keep IT under control. This simply has to do with the fact that fingers are pointed at it more often. I'm thinking, for example, of the situation reports that the BSI [German Federal Office for Information Security] regularly sends out, 98% of which seem to contain IT reports. Here and there an OT report comes across. These are evaluated relatively regularly and are then also followed up on relatively regularly. As a rule, they are notifications that they should do something. This has taken a while, and it hasn't yet sunk in everywhere that you really have to deal with this issue on a daily basis. But I think that in the IT environment, it's a question of time.

In the OT environment, it is often not possible to do this in such a timely manner. On the one hand, because many of these control systems are simply very old. They have been running for 30 years and never change a winning team. Will they keep running the way they are? On the other hand, the need is simply not yet there or the pain itself is not yet great enough to ensure that the systems are always looked at immediately. In my view, there is also a lack of understanding of what tools are available. So if you say there are advisories and I can get them regularly from the manufacturers or they are made available to me or I get them from public sources and should then automatically check them against what I have in my asset inventory... That just doesn't happen in the way we would like it to.

There's a lot of work to be done, and we have to get some people out of their comfort zone. And we simply have to understand that the people who work on these plants are actually pursuing a different work goal, namely to produce high-quality drinking water or to get the wastewater clean enough to comply with the limit values. These are people who never actually deal with OT and IT at their core activity.

Klaus Mochalski

The question immediately arises: How do we prepare ourselves well in these areas? What do the operators have to do in the water industry, for example? The reflex could now be that everything is working quite well on the one hand. As you have described, this is not surprising because we have been practicing this for years and because there are, of course, standards on how to do this. So this is actually established knowledge. It's just a question of implementation and perhaps sometimes also a question of cost.

Are we now simply transferring this to OT? If you think about it, that's probably not the right approach, because we're talking about relatively few incidents. We're talking about a combination of know-how. You've already mentioned the very old control systems, where of course I need people who understand them. At the same time, you also need knowledge of IT attacks because, as you said, most attacks in OT infrastructures are classic IT attacks. In other words, I need this understanding and I need an understanding of the systems, some of which go back 20 years. Bringing that together is a challenge in itself. And then to get the staff who know what to do in the event of an incident, which systems to press which buttons on, what to switch off and what not to switch off. That's damn difficult.

And that raises the question. If every small operator - and we also know how small the market is in this area in Germany, we have around 1,000 municipal utilities, we have many operators in all kinds of facilities - hires an OT security expert... There simply aren't that many in Germany. That means we have to solve the problem differently. I know we've talked about the fact that you've thought about how to do this in the water management sector. From your point of view, what is one approach to solving this qualitative problem - how do I get people trained - but also the quantitative problem - how do I get the right people in the right place at the right time?

Rainer Stecken

Visibility is exactly what we are talking about right now. Namely: How do I notice very quickly that something is happening on my system that I don't want? Until now, people have usually resorted to saying: Okay, I can control the whole thing manually. I'll take it completely offline, so to speak, and put someone everywhere who will then carry out the control on site. That's self-deception, of course. It works for a day or two, but not for weeks. In this respect, you simply have to make sure that you can see more quickly when you are under attack.

And there was an approach in North Rhine-Westphalia last year, where we set up a Security Operating Center [SOC for short] for the water sector. The idea is that there is actually a qualified SOC with qualified employees. At the moment, there are a dozen of them who collect the individual messages from the systems from the logs or from the network using the corresponding devices, enter them into this SOC and the SOC is then able to recognize whether it could be an attack or not. And this then sends a corresponding message back to the system operators, who are then able to react.

This solves the problem that I don't have to have the know-how about what an attack can be on site, but I have this in my SOC with specialized people who, however, must have actually spent time at the individual facilities beforehand. At the moment, the employees have actually spent a week working with the relevant utility operators to get an idea of the situation: What kind of signals are they actually sending? And what does that mean on site at the facility? So it's not easy to say that I have a SOC and that will solve my problems. But it is of course a step in the right direction.

This SOC has been running since the end of last year, is supported by some large operators here in North Rhine-Westphalia and is of course initially attracting the big players in the industry - from Berlin, Hamburg and Lake Constance - who are interested in getting involved or taking a look at what possibilities this SOC offers beyond what they have already implemented themselves.

For me, that's one approach. The second approach, which we still need to talk about, is the appropriate training. What do I actually do when things go wrong? So that you have a corresponding plan in the drawer and have memorized what you should do.

Klaus Mochalski

Let's stay with the first topic for a moment. We can also talk briefly about the topic of training. But the topic of SOC is very intriguing, because that sounds like the logical approach at first. It's not a new approach. It's been around for a very, very long time. But the idea here is not to run your own SOC as an operator, but to outsource it, so to speak. But not to a service provider who doesn't know my specific requirements. Instead, you organize yourself in this specific industrial sector - in this case, the water sector - and set up a SOC with the corresponding specialist know-how.

And then the individual companies can join in and supply the data that will then provide the visibility you mentioned - which is absolutely essential. And then you have the trained staff on site. And so you have a sharing community. You divide the problem up and make it economically manageable, so to speak. And I think that's a very attractive approach.

Of course, I'm interested in how this will work in the mid-term. I would have thought that the larger organizations might do it themselves and internally and that there might be a certain reluctance to give their data to an external body, which is still uncertain. And that it's more for the smaller organizations. That's why I find it interesting that the larger ones have participated. What do you think is the reason why the big ones have jumped on board with such an external solution and why the smaller ones are still hesitant?

Rainer Stecken

Because “large” is relative. A water supply company that is large in itself is actually a medium-sized company. And there might be a total of 50 people working in IT, OT, configuration and monitoring. If you think about what they need to specialize in, it's still not enough to set up a corresponding [SOC] operation, which then has to be guaranteed 24/7.

In other words, I would theoretically need at least twelve people for this purpose alone. And it's hardly financially feasible to set that up even in a medium-sized organization. That's why it also makes sense for [large companies] to consider: Can we perhaps solve this better together?

Klaus Mochalski

What could be the barrier for the smaller ones to join in?

Rainer Stecken

Well, I don't actually see any greater barrier for the smaller companies than for the larger ones. They are perhaps simply further away from it in their minds, because the big ones have of course already had this requirement in the past [note: since May 2023, critical infrastructures in Germany have had to verifiably operate an intrusion detection system].

We have compared this, for example, how are the water management companies in the Netherlands set up? They have organizations of at least 200 people. They always have specialized IT security experts. We don't have that here. And then we thought about it: Okay, let's put it together accordingly.

In my view, the smaller ones should take the opportunity, if it presents itself, to look into it and take the opportunity to get advice on how to proceed in terms of OT security, because they can't really judge that well at the moment. But I'm glad when it starts with the big players that we have an initialization core, so to speak, to get things going. I think this will actually be a great benefit for the water sector in the long term. We have 6,000 water suppliers in Germany. This means that most of them are so small that there is no way they can do it themselves. And they should get involved.

Klaus Mochalski

Yes, that means you are definitely interested, right? If an operator, a water management operator is listening, they can certainly contact you. And who would they need to get in contact with to get involved?

Rainer Stecken

They can do this at the Competence Center Digital Water Management KDW, which is based in Essen, where the SOC is operated, too.

If they contact the KDW, they will receive information about what they need to do to connect to it.

Klaus Mochalski

That sounds good. We're not a promotional podcast, but maybe it's worth advertising it. And that we display this link in the show notes [https://kdw-nrw.de/]. And maybe that will attract more people. But as I understand it, there's really no reason why smaller companies shouldn't take part.

Rainer Stecken

Exactly. And speaking of “advertising podcasts”: at the moment, this is an operation that is essentially run by the Ministry of the Environment of the state of North Rhine-Westphalia. This means that we also have the necessary backing from the authorities and they are currently ensuring that the launch is successful.

Klaus Mochalski

Is it limited to North Rhine-Westphalia at the moment or could someone from Bavaria also get in touch?

Rainer Stecken

Someone from Bavaria can also get in touch.

Klaus Mochalski

Okay. Good. That's good to know. That this kind of SOC is operated in a specific industrial sector, which is perhaps organized by the relevant industry association. Based on the experience you have gained so far, do you think this is an operator model that could also work in other industrial sectors, for example in the energy sector, in energy supply or in waste management? Or in completely different [sectors such as] the manufacturing industry. We need this sector-specific expertise in such a SOC. Is there anything that would prevent this model from working in other industrial sectors?

Rainer Stecken

Not from my point of view. This could also work in other industrial sectors. The energy sector is simply much larger in terms of financial resources and has of course been operating [SOCs] itself for a number of years. This means that they are not necessarily dependent on sharing what is needed.

Klaus Mochalski

Only the big operators. And I think we still have the problem that they probably wouldn't participate in such a sector SOC. And then the smaller operators are left out in the cold, because they have the same problem that we discussed. They are too small to organize themselves, i.e. to operate their own SOC. And where should they go now? And it would be very important for them to have a parallel model.

Rainer Stecken

That could work the same way. Absolutely.

Klaus Mochalski

Okay, okay. What are the biggest challenges when setting up a SOC like this?

Rainer Stecken

Firstly, to raise awareness of the fact that a SOC can only work effectively if I - to put it bluntly - let my pants down, i.e. provide all the data that is absolutely necessary to assess an attack situation.

We saw at the beginning that people said: Well, we'll give you some of the data from the OT. In case of doubt, you need a view of the entire company, even the IT of the entire company, in order to be able to assess whether there is an attack or not. In other words, a sector SOC is perhaps very helpful in that it is easier to build trust between the customer and the provider in question, because the provider has a bit of a background in the same sector.

Klaus Mochalski

Yes, that's another reason, in addition to the specialist knowledge and expertise that I need to build up in order to gain acceptance. That is understandable. What you say - that I naturally provide very detailed data about my infrastructure - automatically means that it is an issue that has to be organized across all levels. This is by no means just an IT issue, it is clearly a management and executive issue. They have to be involved, they have to support the whole thing, they probably even have to drive it.

In some of our recent podcasts, we have also talked about the best way to organize OT security. And the trend that is emerging in - let's say - larger organizations and - as far as OT security is concerned - more developed sectors, is that it is increasingly becoming an issue that ends up centrally with the CISO, i.e. close to the management. There, it ultimately becomes part of the normal risk assessment for the operation of the company and then it trickles down, so to speak. Through the organization and through the individual departments. Is it the same in the area of critical infrastructure, especially in water management, or do you see other organizational models there?

Rainer Stecken

It's exactly the same. What we see is that at the top management level, managers are actually in favour of using such models because they are fully aware that they are not in a position to do this themselves and independently due to the size of the organization and the financial conditions they have to meet. Sometimes it also takes a bit of persuasion at the levels below to really open up to a SOC. That is the experience.

In other words, you have the go-ahead from the very top, so to speak. Do that. Do it openly. Share the data. Operate it in a way that makes sense. And below that, it becomes a bit difficult to actually get all the data I need to carry out an attack assessment. But the overall model is, as you say, if it's accepted from the top, then it should make sense to implement it.

Klaus Mochalski

Yes, that sounds good.

To summarize, you need the support of the management, you need the support of the technical level through all hierarchies. This is probably an internal challenge as well as an external one. In other words, even as an operator, you have to provide all levels with the relevant information. Why data sharing is secure, what the added value is. And you have to be able to justify this both technically and, of course, economically. So you have to involve all levels.

And then I think the recommendation to the operators is very clear: don't be afraid to letyour pants down. As you said, when in doubt, you're in good company and you can only win in the end. That's why I think your recommendation is very good. To take the first step, to approach this sector SOC, which already exists here, in the water management sector and, as a recommendation to the other sectors, to consider whether this is not a good operator model that is also transferable.

Rainer Stecken

Yes, I mean, you can think about it, there are other industries where this model has been practiced for decades. Telecommunications, for example. The Internet is critical infrastructure that cannot be operated outside the Internet. Nevertheless, they manage to operate it reasonably well and securely. How do you actually do that? With just such models!

Klaus Mochalski

That sounds very good. In other words, the problem is certainly a difficult one but together we know how to solve this problem. It's not an unknown, unsolvable problem. It's no longer rocket science. You just have to take the first steps. And here you can clearly turn to such an operator and follow the recommendations based on experience and the security frameworks - which exist in many areas, some of which are sector-specific - and gradually improve security in the OT sector.

Rainer Stecken

Yes, exactly.

Klaus Mochalski

Great. Then I think we've covered the topic well. Oh, we wanted to talk about the training of experts as well. I don't know if we can go into much depth here. But you also said that it is of course important to build up expertise. What are the challenges here and how should this be done?

Rainer Stecken

So there are various approaches as to how we can simply become faster when it comes to patches, how we can get more knowledge into the organization about the individual components that make up their software. I'm thinking of the BSI. They have launched various initiatives to say:

We need to get the advisory information into the companies more quickly.

We need to be able to automatically compare this with the corresponding asset lists.

Of course, we also need to have software build of material [SBoM] lists where we can really see what the software is made up of so that it becomes transparent.

If I have any weaknesses in my software, I need to be able to recognize them and react to them. That's one point.

And the second point that comes to mind is the Cyber-Range that exists within the E.ON Group [note: the Cyber-Range is a cyber security training center for the German energy sector]. These were announced as publicly accessible at the time. This means that anyone can train there for an emergency. However, they have so many requests from their own company that they can now only offer this internally. I would like to see something like this for the utility industry in general. Because only what you practice can you actually master when the worst comes to the worst. When the pressure of being under a successful attack really becomes apparent.

Klaus Mochalski

Interesting that you mention that. We've already done a podcast episode with the Cyber-Range, where we've also been participating for years as a provider of an intrusion detection system and helping to shape the training. And we learned a lot there too. Of course we have. And what you say is interesting, that there seems to be a demand there that cannot currently be met. And then there's also the question of how to organize it. Do you do it in a similar way to the SOC, in that you organize it on a sector-specific basis, as has now been started in the electricity industry? But that also means, of course, that you need organizations, teams and, ultimately, funding to implement something like this.

Rainer Stecken

So, for me, that is now the goal of what to develop next. Following on from this SOC, that we are actually in a position to train people.

Klaus Mochalski

That sounds exciting. It sounds like you've been successful with it. Once you've taken the next steps, we should do another episode and see how the Cyber-Range works in the water industry and what might be implemented differently there than in the electricity sector. It certainly sounds like a very exciting topic.

Rainer Stecken

I’d love to.

Klaus Mochalski

Rainer, it was a pleasure. It was an exciting discussion. I hope it has also given our listeners some insights. I think it gave us some good ideas, especially in the area of water management, and I'm looking forward to the follow-up episode. Then to the topic of Cyber-Range in the water industry.

Rainer Stecken

Thank you very much and I am also looking forward to this episode. Let's see when it will actually be made.

Klaus Mochalski

I am looking forward to it.

Rainer Stecken

Me too. Thank you.