Author's Opinion

The views in this column are those of the author and do not necessarily reflect the views of iTWire.

Have your say and comment below.

Friday, 24 October 2008 05:14

Life in the trenches: an OpenSSH developer speaks


If you're running GNU/Linux and communicating remotely with other computers, there's a very good chance you have reason to thank Damien Miller.

The soft-spoken developer, now in his mid-30s, has been handling the portable OpenSSH project for some years now; mapping of SSH versions on the internet show that something like 81.25 percent of the SSH servers are running OpenSSH.

OpenSSH is run by the OpenBSD project which is headed by Theo de Raadt. SSH or Secure Shell is a program used to log into another computer over a network, to execute commands in a remote machine, and to move files from one machine to another. It provides strong authentication and secure communications over insecure channels. OpenSSH is a free implementation of the program.

Miller, a Melbourne lad, joined the project in 1999, a month or so after it had kicked off. At the time, he was working for a Melbourne company, called Internet Business Solutions (IBS). "One of the main products which I worked on was a managed firewall service, basically deploying Linux boxes at customers' sites. There was software which enabled us or the customer to handle things like mail, firewall and, in some cases, web caching as well. They had a couple of hundred of these boxes around the country and I worked on the operating system side of things for them, effectively building a Linux distribution and writing the management software. The software helped them run it and let the network operations people administer it remotely as well. That's the place where my involvement with OpenSSH started."

There was a version of SSH available at the time, put out by a Finnish developer named Tatu Ylonen, but the licensing terms were rather restrictive. "It was initially a kind of permissive licence which said that you could do what you liked with it but if you changed the protocol you were not allowed to call it SSH anymore," Miller says.

"Then slowly others (restrictions) were added, like you can't sell it, you can't use it for any commercial use, you can only use it for academic or personal use. We were a small company and we couldn't afford a couple of hundred bucks worth of software to plonk on each of these machines. So I wrote a fairly horrible equivalent using the SSL protocol which kept us going until I heard of the OpenBSD project's work on creating OpenSSH. I had used OpenBSD once or twice back then; it didn't fulfill the need I had so I kept using Linux. But I took a bit more notice of OpenBSD once I heard of OpenSSH because this was something which useful to our company."

For Miller to dip his toes in these waters is rather far from his roots - he is a self-taught developer, one who, like many others, had an interest in computers in his young days but then turned to other interests until some years had passed.

"My parents got me my first computer when I was about eight," he says. "Back in those days, owning a computer was very conducive to learning how to program because they didn't do much unless you programmed them to do something or bought some software which wasn't particularly widely available in Australia.

"I learnt to program fairly young and I didn't really pursue it much after that. I learnt and then forgot about it and then picked it up again in my mid-20s when I was in arts school, studying film sound in RMIT. There I learnt that you could use computers to write music, so I dusted off what I knew and started doing that for a while.

"At the same time, this internet thing was gathering steam and you could use computers to publish information online. So I set up a web server and found myself helping a friend to publish a site. I didn't have any formal training, I kind of picked it up as I went along."

His father, a doctor, encouraged the interest in computers but that apart there is nothing in his family circumstances that led him to this field. "I think my first computer was a BBC Micro which was made by Acorn, a company in the UK which went on to develop the ARM processor which is in many mobile phones these days," says Miller.

"They made one computer under the aegis of the BBC as an educational tool for schools. It was quite a  fun machine on which to learn to program; it had a BASIC interpreter as most machines of the time did but it also had a built-in assembler. You very quickly realised that programming it in BASIC was an exercise in frustration and you'd very quickly have to learn to program it in assembly language which is probably the first introduction I got to how a computer really works."

Once the interest in programming was rekindled, Miller taught himself to program in C and that led to his first real job, helping a friend in Singapore set up a web hosting company. in about 1995-96, the time when the web was beginning to be exploited as a commercial medium. From there he switched to the Melbourne company, IBS.

Once Miller heard of what the OpenBSD project had done with SSH, he decided that this would be a great thing for IBS to use. The OpenBSD developers had taken the code from Ylonen's SSH with the least restrictions and cleaned it up. "Initially that meant fixing a lot of security bugs that had been found in the intervening years between the last free version and the time the OpenSSH guys picked it up. It also included porting it over to OpenSSL, adding some features and making it a part of a BSD system. They were a couple of months into this process when I heard about it and decided that this would be a great thing for us in our company if we could run it on our Linux products as well."

There were quite a few people involved in OpenSSH from the OpenBSD side. "There were Theo de Raadt, Niels Provos, Markus Friedl, Dug Song, Aaron Campbell, and possibly Todd Miller," he recalls. "I wasn't an OpenBSD developer or user, certainly not back then."

He released his patchset to make OpenSSH run on Linux. "Then I probably got an email from Theo saying 'you're doing it all wrong'," he says. "And buried among all the constructive criticism, was one suggestion: 'you're basically re-implementing things which we've solved in our implementation. Why don't you take our implementation and use that?' They were talking about some safe string handling functions that they had implemented well and I had implemented badly for my Linux port of OpenSSH.

"The standard C library defines some functions for copying and concatenating strings; and the standard ones, some of them have got flaws, and some of them lend themselves to unsafe use from a security perspective. Fairly early in the OpenBSD project, Todd Miller devised some variants of these which had an API that is much more conducive to safe use. One of the first things which the OpenSSH developers did was convert over the code to use these safe functions. Most operating systems back then didn't have these as part of their standard C library. These days apart from Linux, most others have them."

Miller then realised that re-inventing the wheel was a waste of time. "So I just copied what the OpenBSD people had done. And that was the approach adopted whenever OpenSSH was being adapted to other operating systems."

Once he had released a version for Linux and posted a message about it to the BugTRAQ security mailing list, Miller started to get feedback. A lot of it was patches that people had devised to make OpenSSH run on their own operating systems. Pretty soon, he says, that one email resulted in something akin to a project. "That's basically how the portable OpenSSH project started. Since then I've basically tracked the OpenBSD releases, and made corresponding changes so that OpenSSH would work on other platforms. Initially, it was just Linux and Solaris but other platforms were added very quickly - AIX, HP-UX, SGI, the other BSDs, and some operating systems which I'd never heard of before. And these were all contributed by other people."

The response did not surprise Miller. "People had kind of gotten hooked on SSH when it was free and were kind of miffed when it was taken away from them. And you know once you have been given something free and then had it taken from you, that's kind of the heroin dealer's model of getting clients. It might work in some cases, but it doesn't work for everyone. So there was a bit of pent-up demand. My experience was mostly with people who used Linux but it was part of a wider culture, all free operating systems. People had gotten used to having high-quality software being available free. And SSH was a piece of software which was first free and then wasn't so it went against expectations."

Not long after, the project got its first legal threat. "I think it was in 2000 when we got our first legal threat from, which was the commercialisation of the original version. They basically threatened us with trademark infringements, saying you can't call your software SSH because SSH is a trademark. It was a bogus claim for a number of reasons: SSH is a contraction of secure shell and the legal advice we had was that one could not trademark an acronym. And then had a history of encouraging people to call compatible products ssh; there was an ssh implementation for the Palm Pilot and the author of that had correspondence with one of the principals behind encouraging its use. Our legal advice also said that if one did not take steps to defend one's trademark, one was basically abandoning it.

"Thirdly, most amusingly for me, they had botched the registration of their trademark. They had trademarked it as a logo rather than as a word or a term. So unless we were using the word SSH on a purple blob or whatever it was, we were on pretty safe ground. Once this was pointed out to them, they backed off."

He says de Raadt received most of the legal threats. "And, of course, he didn't back down. It was good because you hear of a lot of free software projects which stop what they are doing because of some legal threat or another when basically, they have a perfect right to go on doing what they are doing. So it's good that Theo fought the good fight on that one."

By then, the OpenSSH project had a good implementation of the SSH protocol 1. "In the years since there had been an IETF (Internet Engineering Task Force) effort to standardise on a newer version of the protocol which fixed a bunch of cryptographic weaknesses and made it a bit more flexible, which was basically SSH protocol version 2," Miller says. "The commercial versions of supported this protocol. Markus Friedl implemented pretty much all of it himself in OpenSSH in an amazingly short period. I think it was in 2001 or 2002 that we released a version of OpenSSH based on Markus's work which supported protocol 2."

Markus's implementation made OpenSSH compatible with the SSH put out by and fixed some cryptographic problems, "not things which could lead to break-ins but things which scared cryptographers and people like me," says Miller. "It was a moving target because protocol version 2 did not get standardised until 2005."

In 2003, Niels Provos did some remarkable work on OpenSSH to implement what is known as privilege separation. Says Miller: "The typical style of writing a UNIX login process was to run it as root, the most privileged user on the system. This server would run as root for its whole lifetime. The justification for this was it needed to log people in, it needed to be able to write to logfiles, it needed to be able to set the user ids so that joe can log in as joe rather than as some other account. The problem was that that left the server exposed to any bug, it made it a very attractive target and any bugs that could be exploited would give someone highly privileged access to the system. There had been a few bugs in OpenSSH, a couple of which had resulted in break-ins."

Niels introduced some architectural changes. "He split it into a couple of processes, one handles interaction with the network, the cryptography and the passing of data from the network to the computer. All of the complicated and hairy stuff gets done there. And that's the part that is most likely to have the security bugs in it. There's a whole lot of complicated stuff there, you're dealing with binary data which has come from somebody who may or may not be hostile and it's the path that an attacker gets to interact with basically. Niels took this part out, separated it out from the server and made it run without any particular privilege so that if an attacker broke into that they would not get superuser privileges. They would find themselves jailed in a part of the system which really would not give them access to anything."

However, OpenSSH still needs root access to do a lot of things. "So he separated the parts which require this kind of access into a smaller sub-program which hangs on to these privileges and acts as a server to that part of OpenSSH that deals with the network. When this unprivileged network-facing part of OpenSSH wants to log a user in, it goes and asks the privileged part to do what it needs to. And the privileged part performs checks - like has this user authenticated themselves properly? Because it's got a very narrow and tight interface with the unprivileged part, it's a lot more difficult to attack."

Other circumstances led to privilege separation being introduced fairly quickly. "We got notification that a security research company had found a nasty bug in OpenSSH and were going to release the details in a couple of weeks. So Theo and Niels had a choice. One was to wait a couple of weeks and do a coordinated release with these security researchers. They were going to release their findings as soon as we made our release. And if that had been done then it would have been easy to find, by examining the difference between the old release and the new one, what the bug was.

"Or we could release a version with privilege separation turned on by default which would reduce the severity of this security problem from a root compromise of a system running OpenSSH to a compromise of an unprivileged part of OpenSSH. Theo decided to release a version with privilege separation turned on by default. Quite a few people yelled at us for releasing a fairly major bit of functionality at short notice. Two weeks later the security researchers released their bug details and we had saved quite a few people from getting broken into."

Miller says that part of the reasoning behind this release was the fact that OpenSSH, even at this stage, 2003, was used widely. "Theo heard from someone who was responsible for maintaining the university network in Japan that over 2001 and 2002, we'd basically killed telnet and rsh on their network. The use of these old unencrypted login protocols had diminished in direct proportion to the use of SSH which was driven by OpenSSH."

From about 2001, Miller became an OpenBSD developer and started working on the core OpenSSH product as well, apart from being the chief of the portable OpenSSH project. These days he does most of the work on it and has been responsible for making the releases for a couple of years. He still runs most of the infrastructure for the portable version, the bug-tracking system and the mailing list and things like that.

"That brings us up to the present day more or less where OpenSSH is mostly done and we think very hard when we're implementing a new feature, whether it's worth it in terms of stability and maintainability. The curse of having a mature bit of software like OpenSSH is that unless you are scrambling to include features not many people are willing to work on maintenance. We get bug-fixes from various places - some of the people who bundle up OpenSSH with their operating systems send us fixes."

For the last two years, Miller has been working as a software engineer with Google. "I don't have a formal degree as an engineer. The downside of that is that I've had to go back and learn a lot of the boring but important fundamentals that you get by doing a computer science degree. I probably would have been a lot more productive ten years back had I done that. But you can fix these things up in retrospect. I probably could have been a more effective developer earlier had I done a degree. but teaching myself things has given me a perspective which has its own value."

He doesn't find this lack of a formal degree a disadvantage when attending a job interview. "Maybe (it was a disadvantage) 10 to 12 years ago. But these days, I think people look at results and being involved with OpenSSH and a few other free software projects is a tangible thing that I can point to.  I think anyone who has any significant involvement in developing free software is instantly employable. If nothing else, it's a sign of real interest and eagerness and it demonstrates something that a degree cannot."

Though Miller is now an OpenBSD developer, he still keeps Linux around. "I use it for work and on my wife's laptop. I really came to like OpenBSD shortly after I started working on OpenSSH. It's a very developer-friendly project. The whole operating system is engineered as a cohesive whole. It's very easy to polish any bit that annoys you in any way and very easy to get changes submitted. Working on Linux and submitting changes back I've found to be a lot more difficult - you're dealing with half a dozen different projects if you're making a change to anything."

He says Linux gets all the buzz because of the huge headstart the project had. "I think Linux got a huge headstart because of all the legal problems in the original BSD project. When Linux was kind of a toy operating system - and I don't mean that in a pejorative sense - and it didn't really do much, BSD was a more or less complete system. Had it not been for the lawsuits, we'd probably all be using BSD instead of Linux today."

Miller has no problems with the BSD licence which allows people to take a snippet of code, use it in proprietary software and lock it away. "I don't see locking away code as necessarily a bad thing. If they are using our code, then they are not going to make the same mistakes that we made initially, so especially with security software, that's a good thing. If people take what we've made and make a better product, that's fine. They haven't taken anything away from us.

"For example, the version of SSH in the iPhone is our code which has been turned into a commercial product. But it doesn't take anything away from us. We do it because we enjoy it and because it makes our lives and other people's lives better."

He doesn't feel that if someone is benefitting from his work, then others should benefit from that person's work. "There's an ethical imperative that they do that. I don't think that it should be in the form of a legal requirement."

These days, Miller is kept busy by his two-year-old son, Hugo. His wife, Simone, is not a technical person. "I met her through friends. I have a lot of friends, most of them have nothing to do with IT.

"Nowadays my interests are pretty much limited to being a dad. I used to like going running and watching movies, reading, and travelling. That's a bit limited but it's slowly becoming more possible as our son's becoming a lot more self-sufficient."

Simone, he says, has been incredibly tolerant. "She's put up with noisy computers under desks and me getting up at odd hours to deal with people in different timezones. Her tolerance has certainly made it possible for me to do a lot of what I've done."

Right now, work with OpenSSH is pretty much only maintenance. "We had a pretty large release a few months ago because we had two hackathons, one in Japan and the other in Canada, where we got quite a bit of OpenSSH work done. Quite a bit of that was bug-fixing; we made a real effort to bash away at the bug list. Some of these bugs dated back as much as seven years and they were real recalcitrant ones which we had to bash our heads together to fix. We added some new features as well, things which people had asked for for some years and which we finally got around to adding. That was our biggest release in years, we probably won't do another one like that for a while."

Subscribe to ITWIRE UPDATE Newsletter here


It's all about Webinars.

Marketing budgets are now focused on Webinars combined with Lead Generation.

If you wish to promote a Webinar we recommend at least a 3 to 4 week campaign prior to your event.

The iTWire campaign will include extensive adverts on our News Site and prominent Newsletter promotion and Promotional News & Editorial. Plus a video interview of the key speaker on iTWire TV which will be used in Promotional Posts on the iTWire Home Page.

Now we are coming out of Lockdown iTWire will be focussed to assisting with your webinatrs and campaigns and assassistance via part payments and extended terms, a Webinar Business Booster Pack and other supportive programs. We can also create your adverts and written content plus coordinate your video interview.

We look forward to discussing your campaign goals with you. Please click the button below.



iTWire TV offers a unique value to the Tech Sector by providing a range of video interviews, news, views and reviews, and also provides the opportunity for vendors to promote your company and your marketing messages.

We work with you to develop the message and conduct the interview or product review in a safe and collaborative way. Unlike other Tech YouTube channels, we create a story around your message and post that on the homepage of ITWire, linking to your message.

In addition, your interview post message can be displayed in up to 7 different post displays on our the site to drive traffic and readers to your video content and downloads. This can be a significant Lead Generation opportunity for your business.

We also provide 3 videos in one recording/sitting if you require so that you have a series of videos to promote to your customers. Your sales team can add your emails to sales collateral and to the footer of their sales and marketing emails.

See the latest in Tech News, Views, Interviews, Reviews, Product Promos and Events. Plus funny videos from our readers and customers.


Sam Varghese

Sam Varghese has been writing for iTWire since 2006, a year after the site came into existence. For nearly a decade thereafter, he wrote mostly about free and open source software, based on his own use of this genre of software. Since May 2016, he has been writing across many areas of technology. He has been a journalist for nearly 40 years in India (Indian Express and Deccan Herald), the UAE (Khaleej Times) and Australia (Daily Commercial News (now defunct) and The Age). His personal blog is titled Irregular Expression.

Share News tips for the iTWire Journalists? Your tip will be anonymous




Guest Opinion

Guest Reviews

Guest Research

Guest Research & Case Studies

Channel News