How to build a SIEM SOC in OT?

Klaus Mochalski

Hello and welcome to a new episode of OT Security Made Simple. I'm Klaus Mochalski, founder of Rhebo. Today, with me is Zeek Muratovic. He is Director of Security Operations with Landis+Gyr. He has been a previous guest on the show. And today we want to talk about a different topic. But maybe for the listeners who missed the first episode, quickly introduce yourself.

Zeek Muratovic

Yeah, thank you. Thank you for having me again. And yeah, Zeek Muratovic here. I've been with Landis+Gyr as the Director of Security Operations for the past year. But the past 16, 17 years or so, my experience was heavily in cybersecurity, penetration testing, data recovery, forensics, et cetera, et cetera, working with a lot of customers, building vulnerability management programs across many different industry verticals. Right. But now, I am heavily focusing on the utility space.

Klaus Mochalski

Yeah, absolutely. And last time I remember, we spoke about how to secure a smart grid, the advanced metering infrastructure. Today I want to tap a little bit into your, let's call it, your past lives knowledge. So you spend quite some time with SIEM – security information and event management – systems that are the core systems forming a SOC – a security operations center. And we also spoke in this podcast about how to build an OT SOC before. I'm interested in your view. You spent [much] time in IT security. Recently you spent some time in OT security. From your observation, what's the state of the usage of SIEM systems in the OT space and [what’s the] status of building successful OT SOC in this space?

Zeek Muratovic

Yeah. So as security is becoming more and more serious in the OT space, you know, companies are starting to invest into products like Rhebo, Into malware [detection], into next gen firewalls. They're in the beginning stages of growing their security arsenal in terms of defense for the utility space. And what that is getting to is them getting many, many applications. And the same problem we had in it a few years ago – not a few years, 10 years ago – is we started getting a lot of applications and very few people managing those applications. So, you have one security analyst or engineer managing 20, 30 applications sometimes.  

The logical next step is [to] bring all the information from all these security solutions into one pane of glass, which, you know, is our SIEM solution, which will make a lot of sense for a lot of these companies that are investing into, into the security products and want to have full visibility right across the entire security landscape.

Klaus Mochalski

Right. In a way, a SIEM system is a system to collect and consolidate information that is relevant to security from many different applications in the infrastructure. Is it the same job for OT environments? And what are the similarities or differences in the data sources that we are seeing in the OT space?

Zeek Muratovic

The data sources are quite similar. On the OT side we have a couple of new protocols. The challenges are that solutions that are built for the OT space... sometimes that data can be very unstructured. When you are looking into a SIEM solution you want to make sure that that SIEM solution is going to be able to read that unstructured data. Else you're gonna be stuck for weeks and months just writing parsers to make the data readable to the SIEM, so that you have actionable data at your disposal.  

If you just dump all these logs in and you don't know what they are, and they're very difficult to read and to identify, then the SIEM solution really is not helping you. Always a good idea is to make sure that you have a SIEM solution that – if you want to bring OT data in – has the capability of being able to make these custom parsers or it can already read unstructured data.

Klaus Mochalski

Well, this sounds like a lot of work. If I understand correctly, we have an issue with the quality of data, the data sources in OT systems. And this is where the custom parsers potentially come in. But it also sounds like for me, as a customer, as an operator of infrastructure, this is quite an endeavor to write these custom parsers or have somebody write these parsers for me.

The problem is basically related to the diversity of the systems that I'm facing and probably also to the age of some of these systems, because that's a problem that has been described. Quite often we have a mix of systems, some of them rather new, recently installed. But we also are up against systems that may be 10 years or even 20 years old and they may provide totally different data, if providing any data at all.

We also see that all the OT monitoring vendors like Rhebo and the others in the market provide rather structured and well defined interfaces to SIEM systems. Is this the reason they are doing this, because the quality of the original sources is so poor in many instances that they try to fill a gap here by providing better data in a better structured way.

Zeek Muratovic 

Yeah, that's a good question. And that's usually a question that comes up when you talk to a little bit more mature clients that already have a SIEM in place. For example, if you go to a smaller client who doesn't has a SIEM, they don't ask that question. “As long as the product does X, Y and Z, we are happy!” You know, monitoring vulnerabilities, monitoring new devices, just giving them information on security. Bigger clients [are different]. That question is [important. They will say:] “That's all good and great that you do that. We expect you to do that. But you know, how about integration with our SIEM? Because all of our data goes to our SIEM?”

So, [for] products like Rhebo, there are quite a few applications out there: plugins for SIEM systems on the market today which make it very, very easy to get the data in. It makes it really easy for those administrators to visualize the data out of the box.

Klaus Mochalski

And I assume there's also a difference in the quality of input data, like what you usually feed into a SIEM system log data, flat messages of events that happen on a regular basis. Usually, you expect them to be in the thousands or 10,000, depending on the size of your systems per day. But if you have a well configured OT monitoring system which is just being triggered if there are anomalies in the network, I assume that the volume of events being sent into a SIEM system is much lower. Is this generally the case?

Zeek Muratovic 

Well, logically yeah. But you want all of the data, right? You want as much data as possible. Because if a breach happens, you want to paint the whole picture, you don't want nothing to be left out.

Klaus Mochalski

You want all of this data in.

Zeek Muratovic 

In a SIEM, you want all of this data in the same [place]. Because you can, to your point, pull out certain pieces for specific reporting and alerting. It will make sense for that. But if you don't pull all of your data into your SIEM, it can happen that you forget about it and then you will need this data for investigation.

One good use case that I've seen that happened in my previous life [was when] I worked at Optiv. I would work with many different SIEMS, from McAfee to Splunk to IBM QRadar to Alien Vault. Also, I was on an incident response team. When we landed to investigate an incident we identified that they didn't have all of the logs being pulled in. When we found out the reason why, the [response] was. “Well, at the time we only cared about these specific alerts”.

Klaus Mochalski

It was not so much a lack of data, but it was a lack of tying or getting the data from the proper sources.

Zeek Muratovic 

Well, it was them filtering out data that they thought they didn't need.

Klaus Mochalski

Okay.

Zeek Muratovic

It was a misconfiguration. […] In a sense they thought they will never need this data. Because nowadays, a lot of SIEM solutions price their solution based on how much data you ingest. So, a lot of companies try to save money by limiting the data that they receive, because they think certain data sets are not needed. And some are really successful with that. They really know what they're doing.

But then again, you never know when you will need that data to go back and reinvestigate something. Understanding your data, what you want from your data, what you want to report on, it's really the key.

Usually the first step is when you get all of your data in, you spend a day or two just going through all the logs and seeing what are these logs telling me. What can I use this log for and correlate it with a firewall log to tell me a story or a domain controller for user access. You can correlate this user log on this computer. He came through this firewall, through this network. If you have all these different data sets, you can paint the whole picture of an attack or for a user accessing a system when he should or shouldn't be [doing it].

And this is something that a lot of utility companies are focusing on. Who is logging into the system maybe after hours, so when they shouldn't. One of the many things they're monitoring for.

Klaus Mochalski

Right. If I understand correctly, from an incident or incident response perspective, having more data is always beneficial, but it also probably puts an operational day-to-day burden on the operators of these systems.

Zeek Muratovic 

Yeah.

Klaus Mochalski

You also mentioned more mature customers. If we look at less mature customers who may just look into setting up a SOC or a SIEM system, what would your recommended approach be? We spoke in this podcast before. I spoke with a couple of guests about the right approach. Is it the best way forward to build an IT/OT integrated SOC to have correlation between events happening on the IT side and the OT side? Because in the real world we know they are probably correlated. Or do you start out with separate silos on the IT and the OT? What have you seen? What would your recommendation be?

Zeek Muratovic 

Yeah, I've seen both. I've seen teams that are very separate IT and OT and both have their own security and they worry about their individual silos. Each individual silo is going to have an arsenal of different security products. IT department [has] products that apply to IT infrastructure and the OT [has] silo products that apply to OT.

And when it comes to the final right step it’s to build a SIEM solution in a SOC.

But my recommendation is always: before you get there [ask yourself], do you have enough data sources for IT to make sense? If you only have a firewall and a NIDS [network-based intrusion detection system] or maybe something that monitors access, like a domain controller. Is it similarly necessary? When you start getting up into this 10-20 application range, then it makes more sense, to have one pane of glass where you can actually pull that information from all these different sources and paint a better picture. With [only] a few data sources, a SIEM is really, I think, overkill.

Klaus Mochalski

So, it takes a certain size before a SIEM really makes sense?

Zeek Muratovic 

Yeah, it's usually when the security engineer’s brain starts to smoke. That's when you're probably going to need a SIEM. Because he's overwhelmed with the applications that he has to monitor, maintain and build on, and he cannot be an expert in all of them. It's very difficult.

Klaus Mochalski

If we look at a specific case, let's assume we have an electricity provider, let's say a DNO [distribution network operator], and they serve a few million customers, so a larger one. They already have proper processes and IT security in place. They run their IT SOC and they have a SIEM system ingesting all the data from the IT systems. But they don't have similar capabilities on the OT side. They have recently deployed OT monitoring systems across their substations and other systems. And now they want to tie this in with their SOC and they want to feed the data into the SIEM system. Would your recommendation be to feed this into their IT SIEM SOC system, or would you first set up a separate system and do the correlation later, once they have the proper operation procedure set up?

Zeek Muratovic 

Yeah, I mean, you can approach it both ways. The approach that I would recommend is if your IT department already owns a very nice SIEM system… Most of them nowadays have the ability to segregate data based on user access. They can give the OT guys access to their own data completely separate from IT data as a first step. And then they will have to come together to see: “Hey, is any of my data usable to you in case of a breach?” It depends, because a lot of utility breaches don't necessarily come through the OT side.

Klaus Mochalski

No, they mostly come through the IT side. That's why it ultimately always makes sense. It's just a question of: Is the first step is overwhelming if I try to tie in all the data together. But from what you're saying, I understand that a modern and capable SIEM system has the capability to separate the data. So, I can basically run my experiments.

Zeek Muratovic 

Yeah, exactly.

Klaus Mochalski

I can feed all the OT data in without messing up my IT dashboards that I already have in place. I can gradually start adding these to the dashboards that I'm showing the operators without messing up the existing procedures.

Zeek Muratovic 

Yeah, and you know, the best case scenario is you having an OT security guy and IT security guy working together. Sharing that information back and forth. That way they're both speaking the same language and understand the implications of what could happen if one of these systems on this side compromise or on this side, how far can they travel into the other network?

They're separate in the SIEM, but they're not necessarily separate on a firewall. The connections are still there. But the monitoring of OT systems requires somebody with very specific OT knowledge. They understand the systems that are running these operations and how sensitive the systems are. Cause if on the IT side, if a server goes down, you know, there could be like a quicker backup than here [in the OT]. A whole production will stop. Power goes out or something like that. It's not as quickly recoverable. I feel like as it's definitely more sensitive. If something breaks here [in OT], it causes much more loss monetary-wise than if something in IT breaks. Not necessarily that drastic, because obviously we've seen what CrowdStrike did in IT systems. It also caused quite a bit of a damage. But yeah, the communication between two similar roles on both sides is, I think, where we need to get. And that's very lacking.

Klaus Mochalski

Yeah, I think that's important advice. We know the attackers are not caring about any organizational separation between IT and OT. For them, it's just systems they can attack. So, I think the advice is very important that whoever is responsible for the systems and for the security operations, they should talk to each other. The OT side can benefit a lot by looking at the security processes already in place.

I also understand that you're saying if a capable SIEM system with all the operational processes is already in place, then you should not set up something separate, but you should tie the data in with these processes and with the system. And then you can still do this step-by-step approach of adding more and more data to the correlation happening in the SIEM system. So that you can correlate events on a higher level that you present on the dashboards to the operational staff.

Zeek Muratovic 

Yeah, yeah. Because you know, from an IT/OT standpoint, IT […] already built a wheel. OT does not need to reinvent a wheel. They just need to plug in these couple of pieces that will help them identify specific OT things in the network.

Klaus Mochalski

Right.

Zeek Muratovic 

The protocols that are running specifically to OT, but everything else is what IT already has. So, don't start over. Work with the IT teams and take advantage of what they already have.

Klaus Mochalski

Yes. Very good summary. I like this. Thank you so much. I enjoyed our discussion again and thank you for being at the show.

Zeek Muratovic 

Yeah, welcome. Thank you.

‍