Martin Förster
Hi, my name is Martin Föster. Welcome to this new episode of the OT Security Made Simple Podcast. I'm heading development at Rhebo, and I brought you two experts, to give you some insights on how we tackle the security and safety challenges in our product development, especially how we use the Rust programming language and also how we work in the development process to use the language to add additional security measures and how this all ties together to make a secure and robust software product.
Raphael Peters
Hey, I'm Raphael Peters. I'm a backend developer at Rhebo, and I'm working on deep packet inspection, but also on other components in our product.
Ingmar Pörner
And my name is Ingmar Pörner. I'm a senior software developer at Rhebo, working mostly at the [OT network intrusion detection system] Rhebo Industrial Protector. Aside from the senior tasks I'm assigned with, I'm actually allowed to also write a lot of code in Rust. I'm looking forward to this talk.
Maybe let me get this started with a small assessment because, of course, here at Rhebo, protecting the utility networks of our customers is at the core of our business. The Rhebo Industrial Protector, in particular, must, to that end, ingest all the traffic from a customer's OT network in order to perform data processing and analysis. This has several implications in terms of safety, in terms of security, and in terms of robustness of our software. The software must be able to deal with those situations which aim for bringing down individual [network segments] or maybe the whole network of a customer and must stay on top of things. This ties in with robustness. The data stream that we inject is as heterogeneous as it can get, almost. We're still talking about OT networks, but it contains a lot of unexpected elements. We have those requirements. And on top of that, our product also poses a very interesting target for possible attackers. All the information of the network comes together in our appliance, and it occupies an almost omniscient position in the network. To that end, the software needs to be secure, needs to be robust.
Martin
Understood. As you describe it, our product is inside the customer network, and we have to ensure that the product is not vulnerable to the regular malicious attack vectors, like memory-related vulnerabilities, or denial of service attacks, which aim at overloading software or network components. It has to be robust itself, but it also has to be more robust than everyone else in the network, because if the customer network is under attack, for example, via high overload or via some vulnerabilities, it still has to survive to tell the user, "Hey, your network is under attack".
This is why security and safety is important to us and why we have to also consider the entire deployment and the environment where the product works, not only the product itself.
Ingmar
Yes.
Martin
Okay. So one thing that's interesting then is how we ensure that the product itself is robust and safe and how it's up to date with known vulnerabilities, for example. Is there something in the product or how we deploy and handle the product that we need to consider? Or how is the deployment working, and how does it ensure that the product is robust when we bring it into the customer network?
Ingmar
Well, deployment management is a challenge for us because our customer networks are not necessarily connected to the internet, so you can't have those rolling updates that you're used to from your laptop, for example. What we do is we based our software on a widely used, free and open source operating system, and we stripped this down to a minimum to limit possible attack vectors. This allows us to ship the software as a disk image. Think of it as the whole package of your operating system, including libraries, tools, the kernel, our software, of course. This is how we deploy the software to a customer. We ensure that the customer stays up to date on all aspects of the installation.
The alternative would be, for example, to just dump our software on a Windows PC and just update the software itself. But of course, this is not accepted because as we know, the longer you have an [application] running, the more vulnerabilities are known and the more things should be updated. We can't just do that. That's one thing.
What we also do is we peer review any change to the software internally, including changes to the base OS [operating system]. We make sure that everything that we add also gets a second opinion. We don't just blindly pull in the updates that we get from external dependencies and so on. Speaking of external dependencies, there's also constant auditing for known vulnerabilities in the third-party dependencies itself.
Maybe one last thing that I would like to mention is that we actually have formalized tests that we run our software against, and those can be used for automatic testing. Think of it as a specification that we established along all those years developing the product. Any build of our software, we basically can ensure that it still adheres to the specification. The specification, of course, contains various aspects of the software, but it also, most importantly, includes security aspects.
Martin
The product or the artifact we build, which we deploy to the customer network, is automatically tested that it actually fits the requirements, meaning [every] feature works, like this button is red and this button is blue, [for example]. Plus, all the dependencies we use, like the basic operating system, the libraries we also use, they are checked for vulnerabilities. We know they are safe to deploy. But what about the actual code we write for this product? This is written in Rust. How does Rust help us in creating a more robust, more secure product?
Raphael
Rust, like any other programming language, is not just the programming language, but the whole ecosystem that you get with that. We use many libraries that are also written in Rust with our code that we use for certain parts in our programs that we don't want to write or maintain [for resource reasons]. Obviously, like most other companies. Those libraries can also have vulnerabilities, of course, but because they're widely used, other people might find them and also report them.
So as mentioned, we have vulnerability scanning, and we use a tool called CARGO Audit for that. Basically, it runs a continuous integration, like every day, for example, and looks into the dependency graph. So it's a graph because each dependency can also have dependencies. And we then check against a public database if we have vulnerable versions of those libraries, and then CARGO Audit will warn us. Then we have to look into the problem and we can assess if we are even affected by it. Because some problems
[of these libraries] are irrelevant to us, so they are not a big problem. But others have to be fixed really quickly, or else they could be a [vulnerability that can get] exploited by other people. So we also have lots of other tools.
We also want to have a performant product, and when we change stuff in a product, it could get slower. That's why we do a thing called benchmarking. So a benchmark is just to run your program with some static data set. It then counts for every part of the program, how long it takes, and writes that into a database. Then we see, after we made one change, if it got slower, or maybe if we're lucky, even faster. Then we can look at that data and look at the change and see if we made some mistakes and if we can fix that.
Martin
Okay, understood. We have audits of the source code itself when we write it. We have the tests of the source code itself. Then we measure if the source code or the code we write is fast enough for our product to stay on top of the requirements. We know it will survive the high load scenario, for example. It's also capable of monitoring the expected amount of traffic and so on, because this is very CPU or processing intensive, of course. I'm writing the code part. Are there also some aspects in the process that are related to how the people work together when writing the code?
Ingmar
In fact, yes. The mentioned code reviews for example, which we actually made mandatory in our company. This ensures that you have at least two pairs of eyes looking at the code. But not only that, we also encourage pair programming, which means two developers get together and write at the same time the new piece of code. So [we have a four-eye principle in place.
And then we add on top additional reviews, so we have up to four [people], actually, which will look through the code. This is not just a means of preventing bugs being introduced into our code base. It also helps us foster a common understanding of the code base because you want to avoid those islands of expertise and knowledge in your company where you put a lot of responsibility on individuals for making sure that this code is secure or safe. That's one thing.
What I would add here is also that we are an ISO27001-certified company, so we subject our whole code and our whole product to regular external security and penetration tests as well.
Martin
Okay. If I understand correctly, it's at least four people writing and reading the code, so they work together to produce the code. Then the code is tested, then the code is audited, and when the code finally makes it into the product, the product itself is audited, tested before it's released. I'm still thinking about how is this... It sounds like a very complex process. Is there any implication on the people you need? Does the process rely on human experience or skills? Or are there some measures that ensure that all the steps are taken in the process?
Ingmar
Well, of course, the build system that we use ensures that most of these checks get applied when you develop locally on your laptop or workstation. But in order to fully understand the value behind all this, you must also consider that we have continuous integration, which Raphael mentioned earlier.
This means, at some point we have a common code base that we all contribute to. At any point in time when an individual developer pushes a new change to the code base, the whole procedure is processed. All the security checks, all the tooling, everything. All the processes come together in the continuous integration. This is basically a barrier that cannot be circumvented or avoided. We don't allow any code to be submitted to our code base that hasn't run through the full pipeline.
Martin
Okay. Having the pair programming and the reviews and the enforced policies ensures there's some level of quality or that the code achieves some quality goals which are enforced and guaranteed before code actually hits the product.
Igmar
That is correct.
Martin
This is not related to the language itself because this is independent of the language. But the topic today is also about how Rust helps us or how Rust is more robust than other languages. Are there features in Rust, for example, that are more robust or make it more secure than, for example, C or other C++ or other system programming languages?
Raphael Peters
Yes. Rust has a tighter memory model than C or C++.
C will not check if [what] you want to do [is actually permitted or sensible]. For example, you can create a list with 10 entries and then just ask the C-compiler to give you the 11th or the 12th entry in that list. Also an attacker can maybe ask that. And if you, as a developer, don't add a manual check for [prohibiting this query], it won't get checked [in C], and the attacker will get the 11th or 12th entry. But because that is not part of the list, the computer will just give you whatever it finds in the memory at that location. So it can find some random data, but it can also find some credentials or some other secret data. The attacker can even write data in some cases and add, for example, machine code that will do something for [them].
Of course, you can also make mistakes as a developer, And even if there is no attacker, because of those bound checks missing, the program can crash. So many modern programming languages, like Java, and in some cases, C++ or Python, and also Rust, of course, use bound checks and don't allow an attacker to access this data.
But in contrast to [the other languages], Rust is really really fast, and has also some additional features like explicit ownership management. If you place something in the memory, you will get a reference in many programming languages. And with that reference, you can access that data and read it or write it. So if you have one part of the program that is reading that data, that is okay. If you have multiple parts of the program on the computers that are also reading that data at the same time, that might also be okay. But if you change that data, it could cause problems. [For example: While reading data in one place, you might change it in parallel in another place. This change will of course create problems for the reading part of the program, as it is not expecting this.] And then you got parts of the data that don't belong to the same thing anymore or to the same state in time. And your program might do some weird things. Of course, we don’t want that. [Different to C/C++], Rust checks that this doesn't happen. And because there is this explicit ownership management, it's also easier for a developer to run stuff in parallel, which allows us to utilize the hardware even more.
Ingmar
I would like to stress a little bit more that none of those features, or those features that Rust is very prominently known for, are really exclusive to the Rust language. You have plenty of modern programming languages with memory safety, and so on.
But what you get with Rust is the combination of performance and stability or robustness. For example, it also has a small memory footprint if you need it. This has traditionally been a trade-off to say, "Okay, either I'm using a more secure language [with a higher memory footprint] or I'm using those classic and more performant languages [with smaller memory footprints]". With Rust, this is not the case. You get basically the full package. This is what makes it so attractive for us to use. It prevents us from stepping into the common pitfalls that each developer is doing because we are just humans, right?
Martin
You mentioned that we are just humans. I can imagine that it's harder to write certain kinds of vulnerabilities or bugs in Rust because it's harder to write code that reads memory or writes to memory where it's not supposed to. Then it's also due to the ownership model, it's harder to have multiple readers and writers corrupt their data, or each other
Still, I think you can add more complex patterns and vulnerabilities into your code that might not be caught during a review. When, for example, the four people writing or reading the code are less experienced or maybe we don't know about this possible vulnerability, but the code itself is correct. And by correct, I mean it does what it's designed to do, but it might still contain security issues. So do we have something in our continuous integration pipeline which checks the code, that it's looking for patterns in the code to actually detect such vulnerabilities that a human doesn't?
Raphael
When you have some problems, in many cases there's some way to reach them. So of course, if you want to exploit something as an attacker, you would have to find some input for the program so that this unchecked part is executed. And so you would utilize a tool called fuzzer that makes that automatically. A fuzzer is basically a tool that was used in recent years by many security researchers for big projects, [like] the Linux [Kernel], to find stuff in some hardware drivers. And they did find stuff, actually. And we just utilize that for our own product.
The fuzzer basically generates data, that's somewhat random for the program we have. And then we look at which parts of the program are executed. And if it finds some input where a new part is executed that it hasn't seen before, it will keep that specific input and will alter it a tiny bit to find new stuff from there. This doesn't ensure that it finds everything, but it can find lots of things, and it's a very great tool for us to find security problems.
Martin
Okay, great. Thanks a lot. Maybe let's summarize what we've learned today. And what I understood is that safety and security is very important for us, not only because we are a security company providing security services [and products] to our customers, but because our product needs to be more robust, more performant than its surrounding deployment environment to actually alert the user or the customer that something is happening [in their OT network]. So it needs to survive the attack, that the network is possibly suffering.
For this, on the lowest level, we use Rust as a programming language, which features many security and safety measures that make some class or many classes of vulnerabilities harder to implement or impossible to implement, like corrupting memory by reading the wrong or writing the wrong areas of memory, or corrupting memory by confusing reading and writing processes to the same memory.
Then we also have the code analysis, which is automated, but also done by humans.
We have the code, which is more robust per design in the language.
We have four people reading and writing each line of code.
Then each line of code that's changed is subject to the continuous integration process which includes the fuzzing test that actually bombards the entire code with random inputs to find vulnerabilities. That's basically what it's doing.
Then we also have the test that ensures that the code actually does what it needs to do.
Then we also have automated audits that compare all code bases and all the other components like the operating system with public vulnerability databases and detect vulnerabilities before we deploy.
Then we have on a larger level, the review and all the other policies that are not code-based, but that are targeting the entire development process. Like the best practices we apply. Then the ISO certification steps we apply to each code change, to each work item.
On the very highest level, we have a continuous improvement process on top of the development process, which aims at improving the development process itself, where we have retroactive “lessons learned”, where we discuss what we did wrong, what we need to improve, or what went good and what went well, and what we need to foster in the future. Which also aims at improving how we develop the product, how the product development in turn will enable more secure and more robust product development in the future, and also how we can improve the team by fixing skills gaps with training, for example. That's the highest level, how we can improve the team's know-how and how to apply and improve the process in time.
Thank you very much for all these insights. It was very interesting. I learned a lot today. Thanks a lot for attending this episode.
Ingmar & Raphael
Thanks for having us.
Martin
See you next time. Stay tuned. See you. Bye.
