Lessons Learned from 10 Years OT Anomaly Detection

Anne Reinke

Hello and welcome to the very first episode of the OT Security Made Simple podcast. My name is Anne Reinke and I'm responsible for corporate communications at Rhebo. I'm sitting here today with Klaus Mochalski. We want to start our first episode here and maybe Klaus you want to introduce yourself!

Klaus Mochalski

Sure. I am Klaus Mochalski. I'm the founder and CEO of Rhebo. I have a technical background. I studied computer science many many years ago. Then I did some research at the university for a couple of years before starting my first company in 2005. So my entire professional life I was working in computer networking. I always worked with computer networks, and for the past 15 years I would say with IT security, computer security specifically. And with the start of Rhebo, we worked as a team, particularly on the topic of OT security. But I guess we'll get to this. It’s a central topic of our podcast and so we'll definitely spend lots of time talking about what OT is, what OT security is. So I'm really looking forward to this.

Anne Reinke

And maybe for our listeners, what is the aim of our podcast? What do we want to achieve with it?

Klaus Mochalski

So in the last 7 to 8 years we've met many different customers, well, companies that not always became customers. Some of them did. And over the course of the years, we learned quite a lot about security in general or OT security specifically, but also about the the pains that customers feel in their day-to-day business. And in this podcast we want to share some of these experiences and we want to do this by inviting guests to this podcast. So there will always be probably two people, sometimes maybe more. And we will always do this in an interview situation where I will talk to guests about their attitude towards security. Their position with regard to OT security and this could be people from companies who are in a responsible role for OT Security. It could also be people from government bodies and associations who are responsible from a, let's say from a regulation perspective. So we'll try to make this interesting, uh, give many perspectives and it will definitely not be about Rhebo products. So we'll focus on the general market situation and the awareness mostly in this area.

Anne Reinke

OK, this sounds really interesting, but maybe before we start to talk about what OT and OT Security is. Maybe you also want to tell listeners what Rhebo is doing exactly.

Klaus Mochalski

Brief summary, certainly helpful to understand where we come from. So we started Rhebo as a small team of IT security experts at this time back in 2014. We had a strong background in IT security. In our previous company we built firewalls together, firewalls that were meant to protect IT office, IT environments. And I still remember that in 2012/2013 we started to get first requests from potential customers. They were looking to secure not so much their office IT infrastructures, but they are industrial infrastructures. So they are building automation systems, they are electrical substations or also they are industrial automation systems and factories. So we started looking into this problem, into this challenge and started to work on a solution. And this is how we got to the idea to start Rhebo and none of us in the founders team really had too much background and experience in industrial control systems, which is kind of the core type of system that we're usually protecting at our customers - today as Rhebo customers. So we spend a lot of time probably the better part of a year talking to potential customers, but also talking to companies providing services in these areas to better understand the field and to better understand the specific challenges. And so we had our first pilot deployments in early 2015 at this time at automotive companies, car manufacturers. And here we gained the first real world experience with the prototype that we had built. And the system that we built at this time is still pretty much what Rhebo is doing today. So Rhebo is building a solution, a system to monitor communication in industrial control systems and industrial infrastructures and it uses a learning approach to understand how communication looks like during normal operation and then detecting events that may potentially lead to an outage or interruption of these industrial infrastructures. So that's basically our value proposition to the customer and that's what we're still doing today. There have been many development steps in Rhebps history of course. So venture capital investments, then the takeover by Landis+Gyr about two years ago and of course most importantly the growth in customer base with this the growing experience in different markets and we'll really try to get this experience into this podcast as much as possible.

Anne Reinke

And yeah, we getting in an idea after you're telling what Rhebo is doing. Maybe you also want to tell why OT security is so important to like everybody. Also with your experience you made.

Klaus Mochalski

It's a question that is still ask surprisingly often. And I'm saying this because I believe the answer is quite obvious. There has been an IT security industry for, I would say for at least 25 years, maybe longer. So there was never much doubt that IT systems as soon as they are connected, but even if they aren't need to be protection from cyber attacks. Because cyber attacks are not new, it's something that happened very early on, like when the first computers were connected to the Internet, the first Internet worm started to spread. And so this is a very old problem and when we started putting IT systems in industrial infrastructures, we basically inherited the problem that we always had and IT systems in these OT environments. And maybe it makes sense for listeners to talk a bit about what OT actually means, because it's the term that we use on a daily basis. But not every listener, not everyone may understand what it means because it's not a widely accepted, not yet widely accepted term. So OT stands for Operational technology and it's basically this part of IT of information technology that is used to monitor, operate and control industrial systems in the wider sense. So this could be industrial manufacturing systems that you find in an automated factory, this can be. Industrial control systems that you find in electrical substations that are controlled from a central control room at an energy company. But it could also be the control system in renewable energy production systems like solar roofs or wind belts. So basically we find them everywhere and every domain. And this is also why OT security is such an important topic. If someone falls victim to a cybersecurity attack in an IT environment, the worst thing that usually happens is that we're losing valuable data. And note here the worst thing that can happen is a widespread outage of the affected systems, and depending on what systems are affected, it could be a widespread problem for whole cities even countries and to society in general. Because just imagine a blackout all over Saxony, all over Germany, all over Europe, and you have an idea of what we're trying to protect here.

Anne Reinke

So it's as you said, it's a really important topic, but when you think back then, like 8 to 9 years ago, when you started with Rhebo, how was the awareness or what does the market look like? You also gave a brief inside of it, but maybe you want to share some experience to our listeners.

Klaus Mochalski

Awareness is really something that is very important, I would say, central to our commercial success as a company. Because the customers are not acknowledging that the problem exists, they're not going to the best into solving the problem. And we have seen a significant, maybe huge development with regard to the awareness about OT security over the past eight years. But it's not to say that awareness was not there back in 2014/2015. When we talked to first customers, they understood the problem. I still remember talking to this engineer at this car manufacturer that we were just starting to use a fully automated production facility in their factory. We had deployed our system at their final assembly at the final car assembly station in their factory, and he had a clear understanding that in theory, an attack coming from the Internet could find its way all the way through the manufacturing robot. The downtime of a car manufacturing factory will cost many 10s, hundred thousands of euros. So he understood the problem, but the risk is something that was considered to be rather low and this took many years of basically watching what's happening, how many attacks are happening, how many attacks are successful and how many simple attacks are affecting highly critical infrastructures. And this is really what increased the awareness, not so much for OT security In general, but I would say the risk awareness. To understand how high the risk is and to also make an informed decision of how much needs to be invested to reduce this risk.

Anne Reinke

And when we're talking about the attacks, like maybe you can give our listeners also an insight of what we can see in our in our infrastructures with our security assessments, what we do also on our daily business.

Klaus Mochalski

That's always an exciting topic, although if we look at our real experiences then it it's rather less exciting than what you actually see in the news headlines. This is because the infrastructures that our customers do quite well protected even before we come in and help them with our, we sometimes call it last line of defense to make sure that we even detect cyber attacks that go get through their defense systems that they already may have in place. So at every new customer we start typing assessment. This is the standardized process where we set up our system. We collect data for fixed period and then our experts are doing an analysis to get a clear understanding the status quo of a new customers infrastructures. Honestly, most of our customers can't give us disinformation at the level of detail and quality that we need to train our systems. So based on this activity or based on these activities, we have a long list of findings and statistical evaluation of what's happening most often, and here we really see the most common problem. Being systems, software or hardware that is active in certain environments, meaning that it communicates with the outside world or communicates with other systems in the infrastructure. And those they are not used for any purpose, they are not needed for any purpose. So it's a bit like an IT the forgotten printer. Which is still connected to the network. Nobody's using it anymore, and it may have an outdated web server running on it, which is susceptible to many cyber attacks because it has all vulnerabilities. And these kind of systems should always be disabled from a security perspective. So this is really the most common problem. And then it's followed by things like vulnerabilities. So systems that have known vulnerabilities, if the customer knows about it and has protective measures in place because patching is not possible, that's okay. But quite often this comes as a surprise to our new customers. So this is of course important. And then what we also see is the use of outdated technology. Because in industrial settings the new cycles are longer than an IT. In IT we you used to get new computers every four years approximately. In OT systems are usually running much longer, 10 years, sometimes 20 years, depending on the domain. So it's very normal to find systems that have not been touched for the past five to 10 years. And so it's not really surprising. That you find software versions, firmware versions in systems that have known vulnerabilities, meaning that they are published on the vendors website and they of course need to be handled appropriately. So either by fixing them, so patching, updating. So we are all familiar with software updates these days and if this is not possible, and quite often it is complicated or not possible, then these systems have to be isolated by other means. So this is also a very common problem. And then what also see is quite often, let's call it performance problems in industrial networks. So our system is a passive monitoring system, so we're not interfering with the systems. You're monitoring. We're just learning how the communication works and then detect events. And we're not just detecting cyber related or cyber security related events. We are also detecting if there's a technical problem and instability, communications drop out and we are notifying the responsible staff about these events as well. And here we always find communication problems, disruptions that may be just as difficult and just as important to handle as a security incident.

Anne Reinke

So you also said that some of our customers were surprised. So they not really know what's going on in there like in their network or in their infrastructure.

Klaus Mochalski

In many cases this is true. It's true to a different extent. So we have customers that have very good idea of how their infrastructure looks like. So they may have asset management systems in place and maybe we'll talk about this topic later. So basically systems that give them a full repository of who's communicating in the network and how they are communicating. But quite often this is not the case. Also, we have customers that have a patch management in place to implement security patches, but more often they don't. And many times they don't have a clear understanding about the diversity of the systems they're using, and with the diversity of software and also the right the diversity of communications protocols with all their security applications. Like every communications protocol, so the language is systems are talking to each other, has its own security applications. And the more you have of these different protocols, the more diverse these security challenges are. And so it's important to have a clear picture and limit this as much as possible. So we are really helping our customers getting visibility and awareness about the scope of the problem as a first step and then we are helping them solving some of these problems, but with others we can well consult them, but sometimes they get a long list of homework after our initial security assessments.

Anne Reinke

So you also said there were some or a lot of developments in the OT security sector. So maybe there's some factors like some political regulations or something like that, that contributed that change or what do you think about this?

Klaus Mochalski

Absolutely. So we talked about awareness increasing over the years and this was not a natural process, it is a it was a process which was driven quite a lot by legislation. If we talk about the European Union and Germany specifically, there has been cyber security regulation proposed by the European Commission many, many years ago. And it was actually the so-called Network Information Security Directive (NIS Directive), and this was transposed into national law by all Member States by 2016. So at least since 2016, every EU Member State had security legislation in place, at least for their critical infrastructures. So for the industry sectors that affect our daily life the most. So for instance, these are of course utilities, electricity companies, water, wastewater, utilities, but also public transportation and the health sector. This legislation has been continuously updated, sometimes by the Member States individually, but there are also from time to time, large initiatives like the current initiative to update. They NIS directive with NIS 2 directive which will again pose a new regulation or it will ask Member States to implement a new updated legislation based on what they've already done in their individual countries, so for us. This has been a huge driver and right now I would say we're doing about 70% of our business with these critical infrastructure operators that are required by local law, by national law to implement certain security measures. So here we are really helping customers and this has been a big driver for us.

Anne Reinke

We talked now a lot about the past and the developments that were made. Maybe we also want to look into the future like. But what do you think are the challenges of that the companies are going to face thinking of OT security?

Klaus Mochalski

I think that's a very good theme for many of our future podcast episodes. So we should definitely try to look into the crystal ball and try to predict some of the short term and also long term trends. From a Rhebo perspective I believe that a very important trend that we are seeing is that what we are doing today, so the solution for network and security monitoring that we're delivering will become more and more integrated with the existing infrastructure or with newly deployed infrastructure. So today most of our customers they purchase system from us. So we are a software development company mostly. And they purchased the system that they put in their infrastructure and then we start using the system or they start using the system to monitor the communication, the infrastructure. It's an add-on. This has advantages because it doesn't appear with the existing systems. But it also has this advantages, produces additional costs for the system itself for running it. It may fail. So we in at Rhebo will really see in the future in integrated monitoring systems. Integrated with the existing infrastructure. So new developments make this possible through things like virtualization and container virtualization. So technical terms will probably cover in some of our future episodes. It basically enables us to put our software. As a kind of an app directly onto a system and with this mechanism we could easily distribute it to many of these systems that exist in the customers infrastructure. Potentially millions of devices, and so the task of deploying a network monitoring system could become as easy as deploying an application on many interconnected systems and then the monitoring functionality would become a feature of the system that is already being used by the customer. So it makes installation, deployment and also operation much easier. And that's the trend that we are seeing today, that we are supporting with what we are building in terms of what our R&D team, our research and development team does and that's really where we see the industry's headache.

Anne Reinke

I think to keep our listeners stay tuned, we gave a good overview about the topic OT and OT security and I thank you very much Klaus for giving this nice overview of your experience and also the good explanation about OT security,

Klaus Mochalski

Thank you also very much. And also to our listeners, I want to say stay tuned for the next episodes. We have already lined up a couple of interesting topics. As I mentioned initially, we'll always invite someone where we talk about a specific topic, this could be customers, these could these companies who we work with where we integrate our solution and could also be person's coming from an regulatory background or from business associations, so we really try to keep this as interesting as possible.

‍