A Decade at Red Hat
work
The first of March 2020 marked my ten year anniversary at Red Hat.
First Programming Job
I wanted to write programs for living ever since I got my first computer (a used Atari 800XL when I was thirteen, already way past its prime then). Programming was really fun. And wait, you can get money doing it? And apparently decent money? YES PLEASE.
I was really dissatisfied with my then job (which I had during my uni studies), but when I started to look for a software development work, it was really hard. I was apparently pigeon-holed into a QE role and all the coding I did in my spare time or at the previous gig (QE related but coding still) meant nothing.
Red Hat was a thing I knew about, but I didn’t have a strong Linux background. I disliked Windows more and more every day and used cross-platform FOSS tools like vim, Firefox, OpenOffice and Python, but I’d still only just started using Linux (in a VM). It felt that if I wasn’t qualified for any other programming job, I definitely wasn’t qualified to work for the Linux company.
And then I saw an amicus brief Red Hat sent to a North Carolina court. I don’t remember the details, but I think it was speaking against either software patents or the Google / Oracle API copyrightability thing. Either way, it was short, clear and to the point (a legible legal document? WOW!) and it made me realise I’ve got to at least try.
This is a company that a) sells open source products and b) has a stance on that super important thing above (either SW patents or API copyright — I really can’t remember — which admittedly undermines this point) that perfectly aligns with mine.
Turns out, they didn’t do just Linux either! I’ve applied for a Ruby webdev position (having had little experience with either) and waited.
And actually got a reply! And an on-site interview which I thought I bombed. And a phone interview with an American developer which I also thought I bombed. And then I got an offer, which I took. I was elated!
Early Days
These were very formative years for me in general. I’ve just moved in with my girlfriend (now wife), and this was my first ever programming job.
I was still in that phase where I thought software engineers were all really smart and the world would be a better place if they ran it (shudder).
So it was at that time when I was already beginning to feel a bit of disillusionment over the orange website people, when I’d ran into a couple of Red Hatters trying to convince a third one how homeopathy totally works and how at the same time, it is impossible to verify experimentally. Because something to do with coffee allergies.
Yep, engineers can be as thick as the rest of us and Dunning-Kruger is universal.
This isn’t really super relevant to Red Hat other than to say that while pretty much every one I came across there was excellent, it’s a big company and Engineer ≠ Smart In Every Way. It was a valuable lesson for me.
Anyway, it was the first time I was really working on open source properly and with dedication. And the first time I was working with people outside of my country every day. The team I was a part of back then was about 15 people, only two of whom were in Czechia (yours truly included).
We had an American there who spoke so fast I had trouble understanding him. That was frustrating, but really useful to develop an ability to understand voices that are not coming from the TV.
My manager was from the US too. Initially, it felt weird to have a boss in a different continent, much less country. But it’s one of the things you get used to quickly and while I do miss the direct human contact sometimes, it mostly works just fine.
Plus you don’t have someone literally breathing down your neck.
It was the first time I was allowed to work from home on occasion (the Red Hat Czech policy was 1 day a week from home).
In those first few years I’ve learned great lessons about open source development, dealing with a diverse group of people, interviewing (our team was growing — locally and internationally), time zones. And of course git-over-email, code review and collaboration.
I can track all my personal OSS contributions to my time spent there. Opening that first issue or pull request is absolutely terrifying, but having been on the other side of the fence made it much easier.
In addition, I honestly don’t think I would be climbing today if it weren’t for the people I’ve met there. All that transpired when a couple colleagues asked if I’d like to join them as we were coming back to the office from lunch. And despite hating every sport I’d tried and being physically weak, I went in, made fantastic friends and found an outlet that feels like it’s been custom-made for me. Guys (you know who you are), I can’t think you enough for this!
It’s a Job
At the end of the day though, it’s still a job. In one way or another, I’ve been working on solutions for banks, data centres, governments, insurance companies, telecoms and so on. And I found out that just writing software is not the life-fulfilling dream I thought it to be.
In an utopia scenario where we don’t have to work to get by, I would be creating video games while teaching, writing and composing music on the side. That’s where my actual passion lies. But the gamedev industry is still treating their workers like garbage while paying them peanuts so thanks but no thanks. I’ll do this as a hobby.
But! That doesn’t not mean this is a bad gig! On the contrary, the work is challenging, intriguing, mostly fulfilling and at least where I live, pretty well-paid.
The Perks
And I get to work exclusively on open source with a geographically diverse group of excellent people.
Turns out, all those points are extremely important and meaningful to me.
As far as I’m aware, every product Red Hat sells is free and open source software. I have not been asked to write a single proprietary line of code during my time there.
Indeed, when they acquire a company that does provide proprietary products, they eventually open source them. The examples that I can remember off the top of my head are ManageIQ, Ansible Tower and recently Quay. These were closed-source until we got them and opened them up.
This can take time sometimes — the products might be using other proprietary libraries or there may be licensing issues to be resolved. But I’m not aware of any closed-source product Red Hat were offering that wasn’t opened as soon as it was feasible.
Red Hat don’t do proprietary. They don’t do open-core. I believe quite strongly that they fucking get open source (as much as a for-profit corp can).
But they’re a huge company composed of different teams and it’s not always perfect. Some of their projects have close to a 100% of their code written by Red Hatters. In others, Red Hat have joined an existing diverse ecosystem that started to dwindle over time and they became a dominant player by the virtue of companies leaving.
But the engineering mantra at Red Hat is "upstream first". You’ve got a linux kernel bug? Send a patch to LKML before adding it to the Enterprise Linux. Got an issue with OpenStack Nova? Fix it at opendev.org first. Need something changed in OpenShift? If it’s in Kubernetes, fix it there first and then bring it in.
That doesn’t mean they do it out of the goodness of their hearts. I think it’s a good business practice and good publicity. But as far as my experience goes, it’s also a genuine drive people working there have, not just lip service.
And as far as individuals go, a huge amount of people here actually really cares about the health of the open source communities. And if they’re in a position of power there, they’ll fight for what’s best for the community. When they do something on behalf of Red Hat, they’ll make that explicit.
It’s a Trap!
If you’re someone like me and really care about this, it makes it quite difficult to leave Red Hat.
By virtue of having a GitHub account I do get the occasional job offer. And almost all of them do one or more of the following:
- Work mostly or exclusively on closed-source systems (sometimes noting that they allow you to work on open source as a perk!)
- Most or all the team is within a single office
- You are required to work from the office
And all these feels like massive drawbacks to me now. For the last ten years, all my coding output has been open source, out there for everyone to see and use.
I’ve worked with people from the USA, UK, Brazil, Italy, Germany, France, Russia, South Africa, Israel, China, Spain, India, Poland, Switzerland and Ireland.
And yes, it’s sometimes difficult and messy and frustrating and complicated. But the thought of giving that up to work with five Czech dudes in a small office somewhere? Thanks, but no thanks (no offense meant to any Czech dudes out there).
Plus I work from home now. This was mostly brought upon by my medical condition and the meds I take, but it’s also what I’ve always wanted. And again, despite the fact that it’s twenty-fucking-twenty and video conferencing is still complicated and unreliable, and that the online chat solutions range from old and lacking genuinely useful features to slow, memory hungry and gross, I still wouldn’t change this for the world.
Unless I guess I were offered an insane compensation (but it’d pretty have to be the fuck-you money) or something that is even more deeply meaningful to me than OSS + geographic diversity + remote work.
If e.g. CERN or the ESA had a job offer, I would drop everything and move there (sorry!). Except they pretty much need you to have a PhD so fat chance. If ZA/UM, From Software or Larian came calling, I’d be on that train (pending a careful check of the working conditions).
But your Python web scraper startup? I’ll pass. Even if it involves ML. Even if I’d be employee number five. Even if you use MongoDB. Especially if you use MongoDB.
Caveat Emptor
But not everyone finds Red Hat fulfilling or even bearable.
I’ve had the great luck to work with excellent people and possibly even more importantly, to have had excellent managers. Hugh & Angus, if you’re reading this, thank you so much! You have both been incredibly supportive and understanding!
Not everyone felt that way. Some folks didn’t like the same people I’m so thankful for. Some didn’t feel adequately compensated. Some didn’t feel their voice has been listened to. Some plain old didn’t like what they were working on. The timezone-induced delays or the lack of personal contact is too much for some people. And not everyone feels content in a big corporation, which ultimately, Red Hat is.
It is not a great place to come in, start changing everything and expecting everyone to follow your ideas. The people there are the veterans of internet forums and open source communities. They will rip any idea into pieces. We work by putting our ego away and collaborating, often with people from outside of Red Hat! And yet there are people with big egos and little substance there and you will have to deal with that.
I had been a witness of some pretty bad conduct. Sometimes dealt with, sometimes not. I’ve seen people leave in frustration, anger or disappointment.
In addition, Red Hat has large sales and support departments and those places may be completely different. I’ve only seen the tiniest slice of the engineering. My experience is not readily generalisable.
I know people who left, never came back and probably never will. But I also know people who left, went to work for a start-up (or built their own) and came back.
Also do remember this is still technically my second ever real job.
And finally, while being reasonably geographically diverse, there are other forms of diversity where Red Hat doesn’t really seem to be particularly better than the other tech companies even though they seem to try.
Point being, Red Hat might not be for you even if you do care about open source. But you might want to at least consider it!
International Business Machines
In Summer 2019, Red Hat has been acquired by IBM. This caused some concern among some of the Red Hatters, yours truly included.
I have not seen any sweeping changes or anything that would cause me personally look for an out. Working for Red Hat still feels as it did two years ago.
That does not mean there were no changes. I mean, not changing would just lead to stagnation. But my deepest worries — that Red Hat would be turned into something that completely abandoned their principles and/or that the working conditions would change for the worse — that has not happened. It may in the future. But I find speculation of little use here.
For what it’s worth, I personally am cautiously optimistic.
Summary
Ten years ago, a few people took a chance on me and I’m still incredibly grateful to them. It has been a wonderful ride.
I am equally grateful to everyone I’ve met: you have all been fantastic and it’s been a delight working with each and every one of you. I’ve learned loads and I’ve had wonderful time knowing you all. Even the homeopathy kooks taught me a valuable lesson.
I’ve also just realised this kind of sounds like I’m saying goodbye. I’m not!
Indeed, I’ve just found an exciting new challenge at Red Hat that I want to pursue and grow with.
P.S. And yes, if you work there, you’ll get the eponymous hat!
(photo taken indoors due to COVID-19 self isolation)
