10. An Interview with Lily Sturmann, Security Expert#
An interview with Lily Sturmann, who is a principal software engineer at Red Hat in the Office of the CTO in Emerging Technologies. She has primarily worked on security projects related to remote attestation, confidential computing, and securing the software supply chain. She gave us some great insights into writing secure and clean code, as well as some advice for engineers who are starting out in their careers.
This interview has been edited for length and clarity.
Malhotra: Could you tell me a little about what you do in Emerging Technology?
Sturmann: Emerging Technology is sort of alongside research, which students might be more familiar with. You can think of Emerging Technology as something between product and research. So, we’re trying out new things, but we usually don’t work on them for years at a time. We want to iterate on stuff quickly, and see if it can become a product. And if it can’t, then we move on to something else. Specifically in Emerging Technology, I’m on the security team which is really fun. We get to work on all kinds of different things, and I can’t recommend security enough for students who are interested, it’s a fun field where you’re always learning.
10.1. “And, you know, nobody’s going to know everything right off the bat, and we don’t expect you to. Especially as a junior engineer- having that enthusiasm and curiosity, asking questions, these are all really good signs.”#
Malhotra: Since you work in Emerging Technology, are there any places you like to look to see if there are new things in the security world?
Sturmann: It comes to us from a whole variety of sources. We definitely keep an eye upstream in the security space, and we talk to other internal Red Hat teams to see if there are any gaps that they have where we can work on stuff that can help them. For example, in research, if they have finished a project, and they say, “Okay, this is good enough for us, but we want to pass it along to see if it can be more productized,” we can try to do that. We just try to look for gaps as well as evaluate existing projects that we can build on, so it’s kind of two sides of how we choose projects.
Malhotra: That’s a really cool approach. Now onto our next question, what’s your favorite linux command?
Sturmann: This one is tricky because there are so many. If I had to pick one standard command I want to say it might be diff. I want to say this has saved me more time than I can count when I’m debugging, you know, when I’m trying to make sure that I’m looking at the right key or hash or why something isn’t verifying. If I’m trying to verify a signature or something [ similar ], sometimes the problem will be an extra white space. But, looking through the data in a file as a human is hard. So this command is not glamorous, but underappreciated.
Sturmann: Some other commands I’ve found really useful: echo $ ?, to show the result of the previous command, especially when you’re doing something where the previous command doesn’t show an output or error message. This helped me a lot when I was working on debugging TPMs, which is a security chip, basically. And I’m a big fan of ctrl-r for reversing searching your previous commands. A couple other fun ones that I just wanna plug: one is cowsay, which isn’t included by default [on most Linux systems].
Malhotra: I’ve never heard of that.
Sturmann: You have to install it. A similar command is sl, which if you know ls, which lists all the files in a directory, [ Sturmann goes on to explain how if you’re typing quickly, ou can often write the common command ls, as sl ] if you install sl you’ll also get a fun surprise.
Malhotra: Now I have to download it.
Sturmann: Tell me what happens.
Malhotra: Have you ever worked with juniors, and what was your experience with working with them?
Sturmann: I don’t want to just toot Red Hat’s horn here, but I think we’re doing a really good job selecting our junior programmers because, in my experience, I have found those I’ve worked with personally to be phenomenal. They have been very enthusiastic, intellectually curious, you know, quick to iterate when there are blockers on different projects that we are trying to complete. And, you know, nobody’s going to know everything right off the bat, and we don’t expect you to. Especially as a junior engineer- having that enthusiasm and curiosity, asking questions, these are all really good signs. And I’ve seen all those signs with all the people I’ve worked with. It’s been a really enjoyable experience, actually.
Malhotra: When you’re working with interns or juniors, where do you usually see skill gaps?
Sturmann: So, we don’t expect everyone to know everything. But, one thing that stands out sometimes- is when you’re a student you don’t always get access to collaborative open source tools like Git and Github. And, I fell into this bucket myself as a student. I got some exposure, but not the same amount as I got on the job. That’s one area where we need to help people along a little bit, and that’s why we developed some intro to git workshops to get people up to speed..
Malhotra: I remember that! It was super helpful. Are there any security considerations that young programmers may skip over, but some people with more experience might not?
Sturmann: Yeah, and so the funny thing about this is that there is a huge overlap between secure coding practices and just generally good coding practices. Vulnerabilities don’t usually start out as malicious, they start out as a mistake somebody made that stays in the codebase that someone else can exploit later. Having good coding practices in your repertoire goes a long way towards helping you write secure code.
10.2. “Vulnerabilities don’t usually start out as malicious, they start out as a mistake somebody made that stays in the codebase that someone else can exploit later.”#
Sturmann: Those are things like readability of your code, modularization- are you putting things into functions, or are you writing code that’s sort of mushed together? Are you separating out functionality to make it adaptable, usable, and easy to update? On the other side, are you writing duplicate code? Duplicate becomes wasted effort, and it becomes more difficult to update. Say you’re updating a cryptographic hash due to some new best practices, do you want to update that in one place, and update it correctly, or find the eight places in your code where you wrote it, have to update one by one and hope you updated it correctly everywhere. See, what I mean?
Sturmann: Another huge thing would be writing tests. Again, not so glamorous, but this will go so far to both security and just lessening your headache over time. It makes it easier to build on your code. If you don’t have robust unit and integration testing, people are gonna break stuff, and you won’t know for months. And when you do, you’ll be scrambling trying to figure out how it happened and how to fix it. And in that time period it can be a vulnerability, right? Testing should be one of the first things you think about. Another thing to think about is to try to use robust, well maintained libraries. If you’re using anything upstream, it’s important that you evaluate that they’re using good security practices, and ensuring that those libraries are well maintained.
Sturmann: I could go on, but you get my point, you want good, maintainable, easy to read code, especially in open source, because it has a lot of eyes on it. This means that it’ll be easier to spot issues before they become vulnerabilities.
Malhotra: That’s amazing advice, especially about building testing into the process, because when you’re writing code for classes, you don’t often have to consider stuff like that.
10.3. “Don’t be afraid to ask questions. Because, that’s the sign of a great engineer.”#
Sturmann: And if you’re interested in security, I would recommend looking into threat modeling and the OWASP secure developer guides, which is the online textbook on security, kind of like what you’re doing.
Malhotra: I used the OWASP guidelines in our security chapter.
Sturmann: There are a lot of great resources there. And I just wanted to say, don’t be afraid to ask questions. Because, that’s the sign of a great engineer.
Malhotra: That’s great advice. This next question is gonna be a little weird, but because this book is aimed at new grads and juniors, I wanted to ask a question to help address imposter syndrome, especially for women and people of color. I thought it might be comforting for people who are on the beginning of the learning curve, some mistakes you’ve made in your career, as someone who’s a lot more experienced.
Sturmann: I think this is a great question. Imposter syndrome is definitely real. A lot of us go through it. And if you feel that way it’s totally normal, don’t let it slow you down, we all make mistakes.
Sturmann: One mistake that really stressed me out was sometime during the first few months of my first year as a full time employee here at Red Hat, I was working on confidential computing, and we were under some deadline pressure. I had a really big PR I was rebasing- this goes back to the git skillset- and, I ended up merging my commit into the person’s previous commit. So, not only did I mess up my own PR, I garbled my work into someone else’s. I didn’t know how to pull it back out. I told my team lead, who was also under this deadline, but he had a great reaction. He said “I’m gonna help you fix this, and then I’m gonna tell you how you can not make this mistake again.” And he did. I thought this was a really kind, compassionate response to a new person. After he told me what happened, I took it to heart and I didn’t make that mistake again, but if I had that’s okay- we’re all learning. It’s a good example of what I found really stressful, but in the long run, wasn’t that important.
10.4. “It’s easy to internalize your mistakes if you have imposter syndrome, but it’s not just you making them. It’s everyone who is growing or learning. It’s a part of the ride.”#
Sturmann: I was lucky to have a really good mentor, but it’s important to know that these things happen all along your career. Hopefully, the ones you make twenty years in are very different from that of a new engineer, but you’re still gonna make mistakes as you learn new things. So- ask questions, and know that it’s okay to make mistakes, and to say that I did this, and I want to know how to fix it. It’s easy to internalize your mistakes if you have imposter syndrome, but it’s not just you making them. It’s everyone who is growing or learning. It’s a part of the ride.
Malhotra: That’s a great perspective, and comforting because I think I made literally the exact same mistake two weeks ago, and was lucky enough to have a really great mentor who walked me through it.
Sturmann: Yeah, exactly, it’s easy to feel like a deer in headlights in that situation, but in reality, it’s okay!
Malhotra: Yes. That’s it for my questions! Thanks for talking to us, and sharing your insights.