Open Source Market Segment LS
Open Source Market Segment RS
Monday, 24 June 2013 11:03

Hacking the kernel: one man's tale


Mel Gorman works with code. Nothing unusual in that, an overwhelming number of the people I interact with in the FOSS arena do just that.

But Gorman (pictured above) is a little more involved than most: he is a kernel developer and hence works at a more basic level than many others.

Gorman is based in Ireland from where he works remotely for SUSE, the Linux company based in Nuremberg, Germany. He is involved both in work for the company and with the upstream kernel community.

In his early 30s, Gorman is not a superstar in the kernel ranks; he is one of the rank and file who contributes to a project that keeps a fair amount of the big iron in the world running. But these are the worker bees of the FOSS community, the silent majority who keep the codebase churning over.

iTWire spoke to him at length to find out how an average kernel developer takes up the job, what he or she actually does, and what keeps them going.

Was there some influence from your parents, a relative or a friend that made you decide to study computer science?

I felt this had to be answered while lying on a couch. My childhood was not particularly technically oriented. I grew up in a rural location in 1980s Ireland. Computers were not common and I do not have much of a reference point to compare with. Maybe my childhood was quite technical to most people and I did not realise it.

There was a C64 in the house that belonged to my older brother as he did have a very keen interest that I guess rubbed off on the occasions he was home from boarding school. Games dominated my computer activity and while I took some interest in programming in Basic and C64 assembler, I do not remember having a particularly deep understanding. I knew some tricks like using the raster interrupt to generate pretty patterns but it was more cargo cult programming based on manuals and magazines than any real understanding. Reading a C64 manual at eight stretches the reading comprehension skills a bit.

Based on that exposure I did decide at 11 that I wanted to work with computers in some capacity so I did find computers sufficiently interesting to make the basics of a career choice at a young age. My decisions in secondary school supported that early choice but as I was in a boarding school with limited technical resources I did not get much access to computers again until I reached university.

When did you first get interested in free and open source software?

I had a Linux shell account when I was in first year but I did not even know what Linux was at the time. It was the latter half of second year before I took an interest in free and open source software. It was encouraged by other members of the computer society.

Is it an ideological thing or are you more of a pragmatic person in your approach to FOSS?

It was a mix of both. From a pragmatic point of view I was in university to learn computer science and software engineering. Even at that point I found that it was very difficult to figure out how proprietary software was put together. The interesting details were simply unavailable and my perception was that  complex application development depended on cobbling together many closed-source toolkits and widgets and working around any oddities. If off-the-shelf components did not work exactly as required, then tough luck. This was drastically different to what I initially experienced with the C64 where it was always possible to poke around how it worked.

When I started working on proprietary software I found that sometimes this attitude still persisted even when the source was available. Poking into other teams' software components had a tendency to, at best, have the emails ignored and/or any changes backed out from source control. This depends on the company but I only worked on one company where this was not the case. I do not know if that still happens today but I have no reason to believe that anything has changed.

Free and open source software was different. The details were always available and to varying degrees developers would discuss different aspects of it. Sometimes the enthusiasm bordered on the manic which could be off-putting but it was always interesting. This did not mean that the software was better. In many cases it was worse that the proprietary alternative, although that has improved drastically over the years. The important thing was that the opportunity was available to learn how it worked and improve it. From a learning perspective, this makes sense and I find that I'm still learning.

Unfortunately, my early career was dominated by proprietary software work. I had not put in the necessary time on FOSS during University to base a career on it.

You were a teacher for a while before you started working in a corporate setting. Which job do you prefer and why?

Actually my road is a little more complex. I went to University and had two summer jobs while there, one proprietary and the other internal-only code. On graduation, I worked on proprietary software for a year before going back to do a masters in which software development work was exclusively FOSS because this is what I found interesting. I was teaching for six months after I completed my masters, but then had to bite the bullet and take another proprietary development job for a few years. I then started a part-time PhD which allowed me to work three days proprietary and two days FOSS a week which gradually became full-time working on FOSS.

I prefer working on software to teaching it although six months is not much of a comparison. I fear that if I was teaching exclusively that I would get frustrated revisiting the exact same topic each year and working with people who do not necessarily want to be there. I know this is not the reality as technology and teaching methods change each year but I do not think it is for me. I might revisit the decision in the future.

Your first kernel-related job was in January 2006 at IBM Linux Technology Centre. Can you detail the steps that led up to this?

I was working part-time on my PhD for a year with IBM when I was contacted by the LTC asking if two of my email addresses belonged to the same person. The kernel team had a number of long-term development goals and my research had the potential to help deliver them. They were interested in hearing what I did within IBM and how long it would take me to complete the research. Martin Bligh arranged an internal transfer with help from Patricia Gaughen, Andy Whitcroft and Dave Hansen so I could work on the research full-time for which I was very grateful. On one hand, I suppose I am lucky but it took a long time to get to the point where I could work on FOSS.

What exactly do you do at SUSE?

My most important role is as the last level of support for SLES customers. Others have the same roles and we are responsible for developing patches or making tuning recommendations to address any bugs reported against the SLES kernel that the support people cannot deal with. Primarily I deal with virtual memory management problems but not exclusively so.

Any bug fixes that are developed for customer bugs are pushed to mainline and backported to -stable where possible. In this way any functional fix I add to the SLES kernel will be removed later when it gets backported to the 3.0-stable series. Some performance works also make it through but it is relatively rare I push complex performance patches through -stable.

To preempt bugs being filed I automate tests heavily and monitor SLES kernels and mainline kernel performance over time. I watch out for the type of regressions that have resulted in customer bug reports in the past and preemptively address them as it can take a weeks to solve the more complex issues and merge them upstream. This minimises the number of bug reports filed but also means in a number of cases where a bug is failed that we have candidate fixes already available. As openSUSE uses mainline kernels the same work indirectly helps them.

If a project is likely to take a considerable amount of time, I'll file a feature request so project management is aware and also so they can priorities if it is not possible to deliver on all of the projects in a reasonable timeframe.

Most of the long-lived development work happens in mainline first and is then backported. I review a number of other people's patches but tend to pay more attention to those that I think I'd want to backport to SLES.

Are you working on the kernel only for SUSE, or are you also working with the kernel devs who work in tandem with Linus?

I work with both groups and am active in the upstream kernel community. I was also program committee member for the last Linux Storage, Filesystems and Memory Management summit and for the Kernel Summit. The SLES kernel policy is "upstream first" which means that in almost all cases the patches in SLES are backports from the mainline kernel. There may be times when patches that have not reached a mainline release yet are in the SLES kernels but this is a temporary situation. The happy result of this is that most of us are active upstream and that most patches in SLES have received external community review.

How does one go about getting patches accepted?

In my particular case it is a relatively straight-forward process. The details depend on the exact circumstance but broadly speaking patches are fixing a functional bug, fixing a performance bug or adding a new feature. There are typically three sets of people involved other than the patch author - the bug reporter (if they exist) or independent testers, the community reviewers and the upstream maintainer. The basic workflow is essentially an iterative development model involving all parties.

If a bug reporter exists then I send them candidate fixes until there is a final version that they report fix the problem. Community reviewers involved in the same area will either add their Acked-by/Reviewed-by or suggest better alternatives. I repeat until both the bug reporter and reviewers are happy and then send that to the area maintainer (almost always Andrew Morton in my case) with all interested parties cc'd. The information in such a patch's changelog will describe what the problem was, what the user-visible impact is and how the patch fixes it. The user-visible impact is important for helping maintainers decide if the patch should be added to the -stable tree or not.

For performance bugs or new features the details are slightly different. This is usually a series of patches and the leader is expected to describe the problem and a short summary of what the series does to address it. In almost all cases I'll include details on how the impact of the series is measured and details of any benchmarks used to validate the patch series. This usually goes through a few rounds of review and when everyone is happy, it's resent to Andrew with everyone cc'd.

It is not unusual for a patch or a patch series to go through a number of revisions before it is finally accepted.

Do you like what you are doing? Do you feel that it improves your perspective on things, or does it drag you down?

Yes, I like what I'm doing. As with everything, there are good and bad days but in general I like what I'm doing.  I'm not sure how much it changes my perspective in terms of working. The primary downside is that I work from home and there are no similar developers living nearby that I know of. Hence, this particular job can be quite isolating but I balance it out with non-work related activities. On the plus side, there are relatively few meetings.

Read 6674 times

Please join our community here and become a VIP.

Subscribe to ITWIRE UPDATE Newsletter here
JOIN our iTWireTV our YouTube Community here


Thoughtworks presents XConf Australia, back in-person in three cities, bringing together people who care deeply about software and its impact on the world.

In its fifth year, XConf is our annual technology event created by technologists for technologists.

Participate in a robust agenda of talks as local thought leaders and Thoughtworks technologists share first-hand experiences and exchange new ways to empower teams, deliver quality software and drive innovation for responsible tech.

Explore how at Thoughtworks, we are making tech better, together.

Tickets are now available and all proceeds will be donated to Indigitek, a not-for-profit organisation that aims to create technology employment pathways for First Nations Peoples.

Click the button below to register and get your ticket for the Melbourne, Sydney or Brisbane event



It's all about Webinars.

Marketing budgets are now focused on Webinars combined with Lead Generation.

If you wish to promote a Webinar we recommend at least a 3 to 4 week campaign prior to your event.

The iTWire campaign will include extensive adverts on our News Site and prominent Newsletter promotion and Promotional News & Editorial. Plus a video interview of the key speaker on iTWire TV which will be used in Promotional Posts on the iTWire Home Page.

Now we are coming out of Lockdown iTWire will be focussed to assisting with your webinars and campaigns and assistance via part payments and extended terms, a Webinar Business Booster Pack and other supportive programs. We can also create your adverts and written content plus coordinate your video interview.

We look forward to discussing your campaign goals with you. Please click the button below.


Sam Varghese

Sam Varghese has been writing for iTWire since 2006, a year after the site came into existence. For nearly a decade thereafter, he wrote mostly about free and open source software, based on his own use of this genre of software. Since May 2016, he has been writing across many areas of technology. He has been a journalist for nearly 40 years in India (Indian Express and Deccan Herald), the UAE (Khaleej Times) and Australia (Daily Commercial News (now defunct) and The Age). His personal blog is titled Irregular Expression.

Share News tips for the iTWire Journalists? Your tip will be anonymous




Guest Opinion

Guest Reviews

Guest Research

Guest Research & Case Studies

Channel News