Sunday, April 12, 2015

Solaris Admins: For A Glimpse Of Your Networking Future, Install OpenBSD

Yet another proprietary tech titan turns to the free OpenBSD operating system as their source of innovation in the networking and security arena.

Roughly a week ago, on April 5th, 2015, parts of Oracle's roadmap for upcoming releases of their Solaris operating system was leaked in a message to the public OpenBSD tech developer mailing list. This is notable for several reasons, one is that Solaris, then owned and developed by (the now defunct) Sun Microsystems, was the original development platform for Darren Reed's IP Filter, more commonly known as IPF, which in turn was the software PF was designed to replace.

IPF was the original firewall in OpenBSD, and had at the time also been ported to NetBSD, FreeBSD and several other systems. However, over time IPF appears to have fallen out of favor almost everywhere, and as the (perhaps not quite intended as such) announcement has it,

IPF in Solaris is on its death row.
Which we can reasonably be taken to mean that Oracle, like the OpenBSD project back in 2001 but possibly not for the same reasons, are abandoning the legacy IP Filter code base, and moving on to something newer:

PF in 11.3 release will be available as optional firewall. We hope to make PF default (and only firewall) in Solaris 12. You've made excellent job, your PF is crystal-clear design.
Perhaps due to Oracle's practice of putting beta testers under non-disclosure agreements, or possibly because essentially no tech journalists ever read OpenBSD developer-focused mailing lists, Oracle's PF plans have not generated much attention in the press.

I personally find it quite interesting that the Oracle Solaris team are apparently taking in the PF code from OpenBSD. As far as I'm aware release dates for Solaris 11.3 and 12 have not been announced yet, but looking at the release cycle churn (check back to the Wikipedia page's Version history section), it's reasonable to assume that the first Solaris release with PF should be out some time in 2015.

The OpenBSD packet filter subsystem PF is not the first example of OpenBSD-originated software ending up in other projects or even in commercial, proprietary products.

Basically every Unix out there ships some version of OpenSSH, which is developed and maintained as part of the OpenBSD project, with a -portable flavor maintained in sync for others to use (a model that has been adopted by several other OpenBSD associated projects such as the OpenBGPD routing daemon, the OpenSMTPD mail daemon, and most recently, the LibreSSL TLS library. The portable flavors have generally seen extensive use outside the OpenBSD sphere such as Linux distributions and other Unixes.

The interesting thing this time around is that Oracle are apparently now taking their PF code directly from OpenBSD, in contrast to earlier code recipients such as Blackberry (who became PF consumers via NetBSD) and Apple, whose main interface with the world of open source appears to be the FreeBSD project, except for the time when the FreeBSD project was a little too slow in updating their PF code, ported over a fresher version and added some features under their own, non-compatible license.

Going back to the possibly unintended announcement, the fact that the Oracle developers produced a patch against OpenBSD-current, which was committed only a few days later, indicates that most likely they are working with fairly recent code and are probably following OpenBSD development closely.

If Oracle, or at least the Solaris parts of their distinctly non-diminutive organization, have started waking up to the fact that OpenBSD-originated software is high quality, secure stuff, we'll all be benefiting. Many of the world's largest corporations and government agencies are heavy Solaris users, meaning that even if you're neither an OpenBSD user or a Solaris user, your kit is likely interacting intensely with both kinds, and with Solaris moving to OpenBSD's PF for their filtering needs, we will all be benefiting even more from the OpenBSD project's emphasis on correctness, quality and security in the released OpenBSD code.

If you're a Solaris admin who's wondering what this all means to you, you can do several things to prepare for the future. One is to install OpenBSD somewhere (an LDOM in a spare corner of your T-series kit or an M-series domain will do, as will most kinds of x86ish kit) - preferably, also buying a CD set.

A second possibly smart action (and I've been dying to say this for a while to Solaris folks) is to buy The Book of PF -- recently updated to cover new features such as the traffic shaping system.

And finally, if you're based in North America (or if your boss is willing to fly you to Ottawa in June anyway), there's a BSDCan tutorial session you probably want to take a closer look at, featuring yours truly. Similar sessions elsewhere may be announced later, watch the Upcoming talks section on the upper right. If you're thinking of going to Ottawa or my other sessions, you may want to take a peek at my notes on tutorials originally for two earlier BSDCan sessions.

Update 2015-04-15: Several commenters and correspondents have asked two related questions: "Will Oracle contribute code and patches back?" and "Will Oracle donate to OpenBSD?". The answer to the first question is that it looks like they've already started. Which is of course nice. Bugfixes and well implemented feature enhancements are welcome, as long as they come under an acceptable license. The answer to the second question is, we don't know yet. It probably won't hurt if the Oracle developers themselves as well as Solaris users start pointing the powers that be at Oracle in the direction of the OpenBSD project's Donations page, which outlines several useful approaches to help financing the project.

Update 2015-07-07: The first public Solaris 11.3 beta is out, and it contains a port of PF circa OpenBSD 5.5. Sasha Nedvedicky (Oracle's main PF on Solaris developer) offers some more details on his blog.

If you find this or other articles of mine useful, irritating, enlightening or otherwise and want me and the world to know about it, please use the comments feature. Response will likely be quicker there than if you send me email.

Thursday, December 11, 2014

The Password? You Changed It, Right?

Right at this moment, there's a swarm of little password guessing robots trying for your router's admin accounts. Do yourself a favor and do some logs checking right away. Also, our passwords are certainly worth a series of conferences of their own.

As my Twitter followers may be aware, I spent the first part of this week at the Passwords14 conference in Trondheim, Norway. More about that later, suffice for now to say that the conference was an excellent one, and my own refreshed Hail Mary Cloud plus more recent history talk was fairly well received.

But the world has a way of moving on even while you're not looking, and of course when I finally found a few moments to catch up on my various backlogs while waiting to board the plane for the first hop on the way back from the conference, a particular sequence stood out in the log extracts from one of the Internet-reachable machines in my care:

Dec  9 19:00:24 delilah sshd[21296]: Failed password for invalid user ftpuser from port 37404 ssh2
Dec  9 19:00:25 delilah sshd[6320]: Failed password for invalid user admin from port 38041 ssh2
Dec  9 19:00:26 delilah sshd[10100]: Failed password for invalid user D-Link from port 38259 ssh2
Dec  9 19:03:53 delilah sshd[26709]: Failed password for invalid user ftpuser from port 43261 ssh2
Dec  9 19:03:55 delilah sshd[23796]: Failed password for invalid user admin from port 43575 ssh2
Dec  9 19:03:56 delilah sshd[12810]: Failed password for invalid user D-Link from port 43833 ssh2
Dec  9 19:06:36 delilah sshd[14572]: Failed password for invalid user ftpuser from port 52436 ssh2
Dec  9 19:06:37 delilah sshd[427]: Failed password for invalid user admin from port 53127 ssh2
Dec  9 19:06:38 delilah sshd[28547]: Failed password for invalid user D-Link from port 53393 ssh2
Dec  9 19:14:44 delilah sshd[31640]: Failed password for invalid user ftpuser from port 35760 ssh2

Yes, you read that right. Several different hosts from widely dispersed networks, trying to guess passwords for the accounts they assume exist on your system. One of the user names is close enough to the name of a fairly well known supplier of consumer and SOHO grade network gear that it's entirely possible that it's a special account on equipment from that supplier.

Some catching up on sleep and attending to some high priority tasks later, I found that activity matching the same pattern turned up in a second system on the same network.

By this afternoon (2014-12-11), it seems that all told a little more than 700 machines have come looking for mostly what looks like various manufacturers' names and a few other usual suspects. The data can be found here, with roughly the same file names as in earlier episodes. Full list of attempts on both hosts here, with the rather tedious root only sequences removed here, hosts sorted by number of attempts here, users sorted by number of attempts here, a CSV file with hosts by number of attempts with first seen and last seen dates and times, and finally hosts by number of attempts with listing of each host's attempts. Expect updates to all of these at quasi-random intervals.

The pattern we see here is quite a bit less stealthy than the classic Hail Mary Cloud pattern. In this sequence we see most of the hosts trying all the desired user names only a few seconds apart, and of course the number of user IDs is very small compared to the earlier attempts. But there seems to be some level of coordination - the attackers move on to the next target in their list, and at least some of them come back for a second try after a while.

Taken together, it's likely that what we're seeing is an attempt to target the default settings on equipment from a few popular brands of networking equipment. It's likely that the plan is to use the captured hosts to form botnets for purposes such as DDOSing. There is at least one publicly known incident that has several important attributes in common with what we're seeing: Norwegian ISP and cable TV supplier GET found themselves forced to implement some ad hoc countermeasures recently (article in Norwegian, but you will find robots) in a timeframe that fits with the earliest attempts we've seen here. I assume similar stories will emerge over the next days or weeks, possibly with more detail that what's available in the article.

If you're seeing something similar in your network and you are in a position to share data for analysis similar to what you see in the files referenced abovee, I would like to hear from you.

A conference dedicated to passwords and their potential replacements.

Yes, such a thing exists. All aspects of passwords and their potential replacements have been the topics of a series of conferences going back to 2011. This year I finally had a chance to attend the European one, Passwords14 in Trondheim, Norway December 8-10.

The conference has concluded, but you can find the program up still here, and the video from the live stream is archived here (likely to disappear for a few days soon, only to reappear edited into more manageable chunks of sessions or individual talks). You'll find me in the material from the first day, in a slightly breathless presentation (58 slides to 30 minutes talking time), and my slides with links to data and other material are available here.

Even if you're not in a position to go to Europe, there is hope: there will be a Passwords15 conference for the Europe-challenged in Las Vegas, NV, USA some time during the summer of 2015, and the organizers are currently looking for a suitable venue and time for the 2015 European one. I would strongly recommend attending the next Passwords conference; both the formal talks and the hallway track are bound to supply enlightening insights and interesting ideas for any reasonably security oriented geek.

Now go change some passwords!

I'll be at at least some of the BSD themed conferences in 2015, and I hope to see you there.

Saturday, October 25, 2014

The Book of PF, 3rd Edition is Here, First Signed Copy Can Be Yours

Continuing the tradition started by Michael Lucas with the Absolute OpenBSD, 2nd edition auction, I will be auctioning off the first signed copy of the Book of PF, 3rd edition.

Updated - the ebay auction has concluded, final bid was US $3,050.00 - see below
Today I took delivery of two boxes full of my The Book of PF, 3rd edition author copies. They are likely the first to arrive in Norway as well (a few North Americans received their copies early last week), but of course this is somewhere in the range hard to impossible to verify.

Anyway, here is the long anticipated with book selfie:

(larger size available here)

The writing process and the subsequent editing and proofing steps that you, dear reader, will know to appreciate took significantly longer than I had expected, but this edition of the book has the good luck to become available just before the release of OpenBSD that it targets. My original plan was to be in sync with the OpenBSD 5.5 release, but to nobody's surprise but mine the process took longer than I had wanted it to.

As regular readers will know already, the main reason this edition exists is that from OpenBSD 5.5 on, we have a new traffic shaping system to replace the more than 15 years old experimental ALTQ code. The book is up to date with OpenBSD 5.6 (early preorderers have received their disks already, I hear) and while it gives some hints on how to migrate to the new queues and priorities system, it also notes that ALTQ is no longer part of OpenBSD as of version 5.6.

And of course there have been various improvements in OpenBSD since 2010 and version 4.8, which were the year and version referenced in the second edition. You will see updates reflecting at least some of those changes in various parts of the book.

Even if you're not on OpenBSD at all, this edition is an improvement over previous versions, we've taken some care to include information relevant to FreeBSD and NetBSD as well, and where there are significant differences between the systems, it's noted in the text and examples.

It could have been tempting to include specific references to Apple's operating system as well, but I made a decision early on to stay with the free systems. I have written something about PF and Apple, but not in the book -- see my Call for Testing article How Apple Treats The Gift Of Open Source: The OpenBSD PF Example for a few field notes.

But now for the main item. For this edition, for a limited time only, there will be a

Book of PF Auction

You have a chance to own the first author signed copy of The Book of PF, 3rd edition.

The auction is up at - I'll look into extending the auction period, for some odd reason the max offered was 10 days. If your bid is not the successful one, I strongly urge you to make a direct donation of the same amount to the OpenBSD Foundation instead.

I've signed the book, and will fill in the missing spaces once we have the name and amount:

UPDATE 2014-10-26 01:00 CEST: Whatever it was that stopped ebay from listing the auction was resolved. The auction is up at - I'll look into extending the auction period, for some odd reason the max offered was 10 days. If your bid is not the successful one, I strongly urge you to make a direct donation of the same the OpenBSD foundation instead.

The first signed copy, and incidentally also the first copy my wife picked out of the first box we opened, will come with this inscribed in my handwriting on the title page:

FOR (your name)
Winner of the 2014 Book of PF Auction
Thank you for Supporting OpenBSD with your
(CAD, USD or EUR amount) donation

Bergen, (date), (my signature)

That's just for your reference. My handwriting is not a pretty sight at the best of times, and when you, the lucky winner, receive the book it's entirely reasonable that you will not be able to dechipher the scrawls at all.

If you think your chances of actually winning are not worth considering, please head over to the OpenBSD donations or orders page and spend some of your (or your boss') hard earned cash!

My speaking schedule has not been set for the upcoming months, but there is a reasonable chance I'll attend at least a few BSD events in the near future. See you there!

UPDATE 2014-11-26: The auction concluded on November 4th, with Bill Allaire as the successful bidder. He paid via PayPal (as you almost inevitably will on an Ebay auction) immediately, and I sent the signed book to him two days later.

As the lady at the post office said, the package took about a week to turn up in Bill's mailbox. But it took 21 days before PayPal finally made the funds available to me, and after a bit of wrestling with the possibly very intuitive (to someone else) PayPal interface, I transferrred the amount to the OpenBSD Foundation today. PayPal of course racked up fees both incoming and outgoing, to a degree that I think if the fees we paid there ar considered at all competitive, whoever coined the phrase "the giant vampire squid" to describe US banks must have been trying desperately to make the traditional US banks sound all nice and cuddly.

For the unsuccessful bidders, I urge you to head over to the OpenBSD Foundation's Donations page and make a donation equal to your highest bid.

Tuesday, August 12, 2014

Password Gropers Take the Spamtrap Bait

We have just seen a new level of password gropers' imbecility. Or, Peak Stupid.

Regular readers of this column know that I pay attention to my logs, as any sysadmin worth his or her salt would. We read logs, or at least skim summaries because in between the endless sequence of messages that were successfuly delivered, users who logged in without a hitch and web pages served, from time to time unexpected things turn up. Every now and then, the unexpected log entries lead to discoveries that are useful, entertaining or even a bit bizarre.

In no particular order, my favorite log discoveries over the years have been
  • the ssh bruteforcer password guessers, a discovery that in turn lead to some smart people developing smart countermeasures that will shut each one of them down, automatically.
  • the faked sender addresses on spam, leading to bounces to non-existent users. Once again, thanks to the OpenBSD developers, I and others have been feeding those obviously fake addresses back to spamdb, slowing spammers down and generating publishable blacklists.
  • and never forgetting the relatively recent slow, distributed ssh bruteforcers, also known as "The Hail Mary Cloud", who in all likelihood had expected to avoid detection by spreading their access attempts over a long time and a large amount of hosts in coordination that was only detectable by reading the logs in detail.

After the Hail Mary Cloud cycle of attempted attacks, I thought I'd seen it all. 

But of course I was wrong.  Now we have more likely than not found evidence of another phenomenon which by perverse coincidence or fate appears to combine features of all these earlier activities.

Early in July 2014, my log summaries started turning up failed logins to the pop3 service (yes, I still run one, for the same bad reasons more than a third of my readers probably do: inertia).

At our site (which by now serves mainly as my personal lab in addition to serving a few old friends and family), pop3 activity besides successful logins by the handful of users who still use the service had been quite low for quite a while. And if we for now ignore the ssh bruteforcers who somehow never seem to tire of trying to log in as root, my log summaries had been quite boring for quite a while. But then the failed pop3 logins starting this July were for the unlikely targets admin, sales and info, user names that have never actually existed on that system, so I just kept ignoring the irregular and infrequent attempts.

My log summaries seemed to indicate that whoever is in charge of (see the log summary) should tighten up a few things, but then it's really not my problem.

But then on July 18th, this log entry presented itself:

Jul 18 17:01:14 skapet spop3d[28606]: authentication failed: no such user: malseeinvmk - (

That user name malseeinvmk is weird enough that I remembered adding it as part of one of the spamtrap addresses at one point. I had to check:

$ grep malseeinvmk sortlist

which means yes, I remembered correctly. That user name is part of one of the more than twenty-eight thousand addresses on my traplist page, added at some point between 2006 and now to my spamdb database as a spamtrap, and as such known to be non-deliverable. So I started paying a bit more attention, and sure enough, over the next few days the logs (preserved here) were still turning up more entries from the traplist page. We would see local parts (usernames) only, of course, but grep would find them for us.

Now, trying to log on to a system with user names that are known not to exist sounds more than a little counterproductive at the best of times. But from my perspective, the activity was in fact quite harmless and could be potentially fun to watch, so I adjusted my log rotation configuration to preserve the relevant logs for a little longer than the default seven days.

Coming back to peek at the results every now and then, I noticed fairly soon that the attempts in almost all cases were concentrated in the first two minutes past every full hour. There are a couple of hour long periods with attempts spread more or less evenly with a few minutes in between, but the general pattern is a anything from one to maybe ten attempts in the :00:00-:02:00 interval.

The pattern was not exactly the Hail Mary Cloud one, but similar enough in that the long silent intervals could very well be an attempt at hiding in between other log noise.

But that returns us to the question, Why would anybody treat a list of known to be non-existent user names as if they actually offered a small chance of access?

They could be trying to weed out bad entries. One possible explanation would be that whoever is at the sending end here is trying to weed out the bad addresses in a long list that may or may not be the wanted quality. If an attempted login gives an indication whether the user exists or not, it might be worth trying.

They could be thinking the list is all good, and they want access. Brute force password guessing is not limited to ssh. We will explore this option further in a bit.

This could be an elaborate joke. The Hail Mary Cloud got passing mention in some mainstream press, and there are people out there who might be able to pull this off just for the hell of it.

Let's put each of those hypotheses to the test.

First, when you try to log in to the service, do you get any indication whether the user you attempt to log in as exists?

Here's what it looks like when a valid user logs in:

$ telnet pop3
Connected to
Escape character is '^]'.
+OK Solid POP3 server ready
USER peter
+OK username accepted
PASS n0neof.y3RB1Z
+OK authentication successful

At this point, I would have been able to list or retrieve messages or even delete them. But since my regular mail client does that better than I do by hand, I close the session instead:

+OK session ended
Connection closed by foreign host.

And in case you were wondering, that string is not my current password, if it ever was.

Next, let us compare with what happens when you try logging in as a user that does not exist:

$ telnet pop3
Connected to
Escape character is '^]'.
+OK Solid POP3 server ready
USER jallaballa
+OK username accepted
PASS Gakkazoof
-ERR authentication failed
Connection closed by foreign host.

Here, too, the user name is tentatively accepted, but the login fails anyway without disclosing whether the password was the only thing that was invalid. If weeding out bad entries from the list of more than twenty-eight thousand was the objective, they're not getting any closer using this method.

Unless somebody actually bothered to compromise several hundred machines in order to pull off a joke that would be funny to a very limited set of people, the inescapable conclusion is that we are faced with would-be password guessers who
  • go about their business slowly and in short bursts, hoping to avoid detection
  • base their activity on a list that was put together with the explicit purpose of providing no valid information

If throwing semi-random but 'likely' user names and passwords at random IP addresses in slow motion had monumental odds against succeeding, I'm somewhat at a loss to describe the utter absurdity of this phenomenon. With trying to sneak under the radar to guess the passwords of users that have never existed, I think we're at the point where the Internet's bottom-feeding password gropers have finally hit peak stupidity.

More likely than not, this is the result of robots feeding robots with little or no human intervention, also known as excessive automation. Automation in IT is generally a good thing, but but I have a feeling somebody is about to discover the limits of this particular automation's usefulness.

I hate to break it to you, kids, but your imaginary friends, listed here, never actually existed, and they're not coming back.

And if this is actually a joke that has somebody, somewhere rolling on the floor laughing, now is a good time to 'fess up. There is still the matter of a few hundred compromised hosts to answer for, which may be a good idea to clear up as soon as possible.

As usual, I'll be tracking the activities of the miscreants and will refresh these resources at semi-random intervals as long as the activity persists:

At the rate they're going at the moment, we could be seeing them hang on for quite a while. And keep in mind that the list generally expands by a few new finds every week.

Update 2014-08-19: The attempts appear to have stopped. After some 3798 access attempts by 849 hosts trying 2093 user IDs, the last attempt so far was

Aug 17 22:40:58 skapet spop3d[20058]: authentication failed: no such user: jonatas-misbruke - (

I take this as an indication that these access attempts are at least to some extent monitored, and with those numbers of attempts with a total of 0 successes, any reasonably constructed algorithm would have found reason to divert resources elsewhere. We can of course hope that some of the participating hosts have been cleaned up (although nobody actually wrote me back about doing that), and of course you can't quite rule out the possibility that whoever runs the operations reads slashdot.

But then again, the fact that the pop3 password gropers have moved on from my site should not lead you to believe that your site is no longer a target.

Update 2014-08-21: I spoke too soon about the pop3 password gropers giving up and moving on. In fact, only a few hours after the last update (in the early morning CEST hours of August 20th, two more attempts occured

Aug 20 05:07:41 skapet spop3d[2943]: authentication failed: no such user: info -
Aug 20 05:31:58 skapet spop3d[15882]: authentication failed: no such user: info - (

before another long silence set in and I decided to wait a bit before making any new announcements.

But at just after ten in the morning CEST on August 21st, the slow and distributed probing with usernames lifted from my spamtraps page resumed. At the time of this writing, the total number of attempts has reached 3822, with 856 participating hosts and attempts on 2103 distinct user names. I'll publish refreshed files at quasi-random intervals, likely once per day if not more frequently.

Update 2014-09-01: Statistics improved, estimates revised.

A couple of days ago, I tweeted:
which, based on some quick calculations at the time, seemed a reasonable number. Now we've reached the end of the first full month of this extended incident, and it's time to present some preliminary statistics, lightly polished.

For the estimated end date, which will be loosely based on the rate of attempts at new user names, the calculation is as follows: On the spamtraps page, roughly half the addresses belong to domains whose primary MX is not (we're secondary or further out). Stripping those addresses from our total, we're left with 14809 possible usernames. Of course, we have no idea when the list they're working from was sucked in, but this is our latest data. Next, by the time I started writing this update, a total of 2523 usernames had been attempted. We're now well into the 57th day, so dividing 2523 usernames by 57, we get a little more than 44 usernames per day on average. Dividing our average per day with total number of usernames, we get approximately 334 days to try them all, which means that the first attempt on the last username in the list is likely 277 (344 - 57) days in the future. This means that at the current rate, it will be early June 2015 before they run out of usernames. How long they stick around will likely depend on their strategy for selecting passwords to match.

If you want to see some more detail on the attackers' progress, here are some data I generated while actively procrastinating and putting off other tasks with a defined and fixed deadline:

A graph of attempts per day, based on this .csv file and massaged via LibreOffice in this spreadsheet:

(larger .png here). I also tried feeding LibreOffice this .csv file with per hour data, but getting the data graphed in any sensible manner seems to require more effort than I'm inclined to put into that particular task.

Hosts participating with first and last seen dates: Possibly more useful is this .csv file, which has participating hosts in the same order as the List of compromised hosts participating but expanded to comprise the fields Attempts,Host,User Names,First Seen,Last Seen, where the last two are dates in formats that your spreadsheet should be able to parse and use as a sort key with only minor efforts. The host with the most attempts as of the last dump was only first seen on August 29th, while several of the top 10 have been with us since some time in July. And of course, at the other end of the scale, quite a few have made only one attempt to date. I'll be updating the data at semi-random intervals, likely at least once a day. Raw data and generated files can be found here, along with scripts used and even a few temporary files. The data can be used for any purpose as long as proper attribution is included. See the Hail Mary Cloud Data page for details, and the Hail Mary Cloud overview article for background information.

Update 2014-09-04: A separate effort detected Perceptive readers will have noticed that the data now includes traces of activity from hosts in the domain starting on September 2, 2014, which deviates far enough from the general distributed pattern that it's likely it represents a separate, if not overly successful, effort.

These four (so far) hosts have tried relatively long sequences of attempts at single user names at a time, targeting what I assume are 'probable' user names for that part of the world, 22 user IDs so far.  None of those user IDs expand to valid addresses in our domains, and just because I can, I've now added these user IDs to the spamtraps page as presumed members of our domain. If you feel those hosts should be removed from the data for purposes of your own analyses, it's fairly easy to strip them away. In fact, a simple

$ grep -v filename

where filename is  either the raw authentication log or the extracted files that include hostnames will remove them from sight.

Update 2014-12-10: Data from this incident as well as some others have been included in my Passwords14 presentation, PDF version available too, here.

Book of PF News: All ten chapters and the two appendixes have been through layout and proofing, front and back matter is still a work in progress, expected arrival for physical copies is still not set. Those most eager to read the thing could try early access or preorder.

It's almost certain that physical copies will not be available at EuroBSDCon in Sofia, but If you make it to one of my sessions there, a special discount code worth 40% off on both the ebook and print version will be disclosed to attendees. (And I'm told it's active already, so if you guess what it is, you can use it.)

Monday, May 12, 2014

Have you changed your password lately? Does it even matter?

Does enforced password change at set intervals actually enhance security? I want to hear your opinion and your reasoning.

All sites are bound to have some collection or other of rules regarding passwords. In most cases, the rules dictate some level of complexity or at least length, some sites have requirements for various classes of characters involved, and in most if not all cases, site administrators implement some kind of mechanism for making you change your password at intervals.

At some places I've worked, I've been part of setting those parameters, and at others I've done my best to comply. The alternative being, of course, having my access to systems that were in fact crucial to my job blocked.  I can sympathise with policies that require some level of password complexity.

But coming up with a good, complex, password or passphrase that is at the same time both hard to guess and possible to remember is not easy. In fact, whenever I've been subject to a regime that requires password change at short enough intervals that I remember the last one, I've spent considerable energy in the grace period from the 'your password is about to expire' warning trying to come up with a good password or passphrase.

The way out has almost always been to figure out the minimum complexity the regime requires, and in some cases pinpointing the amount of difference needed between two succeeding passwords or passphrases.

So what features of a password regime do actually improve site security? Is enforcing frequent password changes such a feature? I offer this poll, where I want your honest opinion:

In your honest, qualified opinion, do frequent and enforced password changes free polls 

Please also give your opinions in the comments.

In other news, I'm still taking questions for my BSDCan tutorials (see the Upcoming Talks panel (top right in the big screens version) or the post BSDCan Tutorials: Please Help Me Improve Your Experience for further details. I look forward to seeing some of you in Ottawa. Depending on how the ala carte sessions work out, similar sessions may be on offer at upcoming conferences. Stay tuned for developments.

Thursday, April 3, 2014

BSDCan Tutorials: Please Help Me Improve Your Experience

A good tutorial should sound to passersby much like an intense but amicable discussion between colleagues.
In a little over a month, I'll be heading out to Ottawa to attend BSDCan 2014. I've been a regular at BSDCan since 2006, attending every year since except 2008 -- I wanted to go that year too, but other business (actually the business of getting out of a company I'd helped build) kept blocking my preparations even though I had a fresh book out with the first edition of The Book of PF published late 2007.

But I've kept coming back after that, and I've almost always given the PF tutorial at BSDCan, this year I'm branching out a bit to give two separate sessions:

Building the Network you need with PF, the OpenBSD Packet filter


Transitioning to OpenBSD 5.5

Both sessions have been allocated 3 hour slots, and they also share another characteristic: I've invited my attendees to send me an email about what they're interested in learing during the tutorial. The main reason I do that is that I want to improve the experience for you, my prospective tutorial attendee.

Let me give you a bit of background. I've been giving the PF tutorial in various forms quite a few times over the years, and at one point I'd accumulated enough useful PF material that writing a book about PF seemed to be a natural next step. There was always some material that did not quite fit the book format, but a lot stayed at least for a while in my tutorial slides.

Over time I ran into a bit of trouble with the fact that BSDCan tutorials are always 'half day' or 3 hours. My collection of slides and notes have tended to expand over time, partly as a function of more experience, and partly due to the sad fact that the other BSDs have been slow to adopt any post-OpenBSD 4.5 syntax changes and other innovations. At some conferences and events I've done the PF session as a sometimes bit overfull full day event, depending on the number of questions and amount of other interaction.

But this time around it's three hours only, and I think that's quite an opportunity to improve the experience. I have more than enough material, but I've found that I usually know next to nothing about the people who will attend the sessions, and people's backgrounds vary enough that it's sometimes hard to find common ground or even pick out at short notice which parts of the material actually fits the group. I've had groups where some attendees had barely used any BSD at all along with OpenBSD developers who committed updates to man pages during the session in response to my slides and remarks, and most levels in between.

So with a strict limit on time, I would very much like to tailor the event specifically to the people who will be attending and who have something of an idea of what they want to learn. So please send me that email (to, and I probably will end up updating the dense mass of slides anyway, and after the session they will be put in the usual place for browsing at your leisure.

If the format works, it's likely I'll try the short and tailored approach again at future events.

But there's more. As it says in the Transitioning to OpenBSD 5.5 tutorial description, OpenBSD has been the source of a number of BSD innovations over the years, and OpenBSD 5.5 has several noteworty improvements: time_t is now a 64-bit value so time will not wrap anytime soon, we have a new traffic shaping system wrapped into PF and a clear path to replacing the once-experimental but now aging ALTQ, signify(1)-signed install sets and packages, and quite a few more bits. The relase page is filling out nicely at the moment.

It's likely that those changes alone cold be made to fill a 3 hour tutorial slot nicely, but once again I would very much like to shape the session to fit the needs of the people who are planning to attend, so it's likely that more general what to look out for when switching to OpenBSD style material will be useful too. And if you haven't already, your experience will be much improved if you prepare a bit. The OpenBSD FAQ and the website in general is a valuable resource, and Michael W. Lucas' Absolute OpenBSD, 2nd edition is a very good source of information.

This year's BSDCan will be the first time I do the Transition to OpenBSD N.m session (unless I do get a rehearsal run organized with some locals), but it's likely I'll try again at later events for whatever is the just released or soon to be released OpenBSD version. Things are shaping up nicely for OpenBSD 5.6 at the moment, but the details of that future release will be out of scope for the Ottawa session. So please do send me an email (to if you plan to attend the session, and I will do my best to tailor the tutorial to your needs.

For both sessions, my ambition is to have the tutorial sound like an intense, but amicable discussion among colleagues. I look forward to seeing you in Ottawa.

Update 29 April 2014: A few people have asked, and I answer: Even if you're not able to attend the BSDCan session, you're of course welcome to send me questions to indicate what you would like to see covered in the tutorials and in the slides I'll publish afterwards. I won't give a firm promise to cover every question, but I'm happy to hear from you.

In other things, the manuscript for the third edition; of The Book of PF -- which main reason to exist is the new traffic shaping system -- is complete, going through the various editing steps and will be available at a yet to be determined date in 2014. I will be updating here and through twitter, G+ and other channels once more detailed information is available.

If you are unable to attend BSDCan, all is not lost: the EuroBSDCon 2014 conference is still accepting submissions for papers and tutorials, so if you have an interesting BSD-related topic you want the world to know about, please drop us a line (or even better a title, abstract and short biographical description) at (full disclosure: I'm on the program committee). This year's conference is set in beautiful Sofia, Bulgaria in late september.

Thursday, February 27, 2014

Yes, You Too Can Be An Evil Network Overlord - On The Cheap With OpenBSD, pflow And nfsen

Have you ever wanted to know what's really going on in your network? Some free tools with surprising origins can help you to an almost frightening degree.

One question I get a lot (or variants that end up being very close) is, "How do you keep up with what's happening in your network?". A close cousin is "how much do you actually know about your users?".

The exact answer to both can have legal implications, so before I proceed to the tech content, I'll ask you to make sure you understand the legal framework you will be working under with respect to any regulatory requirements or other legal limits as they apply to monitoring in general and your users' privacy in particular before you proceed to setting up a monitoring infrastructure. Legalisms can be tiring to a techie, but illegality can bite you really really hard.

Now for the tech side of things, of course I have network monitoring and a few favorite tools. This article has been brewing, for some values of, for quite a while. While I was collecting notes and anecdotes, last (Northern hemisphere, 2013) summer yielded news stories that showed more pervasive surveillance than most had even imagined, operated by a three letter US government agency, and writing about the relatively benign techniques in my favorite toolbox became less appealing for a while.

But the questions about how to really get to know your network are still relevant to networking practitioners, so I'll let you in on a few not really secret facts about how it's done. Of course all of the things I describe here are easier if you're using OpenBSD, but then you probably knew that fact about our favorite operating system already.

OpenBSD has traditionally had an impressive suite of networking tools, and as we know every release brings new enhancements and sometimes brand new tools for us to make use of.

Enter pflow(4), Yet Another Network Pseudo Device

The NetFlow protocol was invented at Cisco in the early 1990s. It's designed to collect traffic metadata, where the basic unit of reference is the flow, defined as the source and destination IP address pair, the matching source and destination port for protocols that use them, the protocol identifier, time started and ended, number of packets sent, number of bytes sent, and a few other fields that have varied somewhat over the NetFlow versions.

Flows are unidirectional, and a TCP connection will typically consist of a pair of flows, one in each direction. For contexts where you do not need to store the content of the traffic, this is the data you want. A multi-gigabyte file transfer, once it concludes, will produce a netflow record that takes up only on the order of a few hundred bytes, much the same as the almost dataless name service request that probably preceded it.

On OpenBSD, various netflow sensors and collectors had been available for a while when the new network pseudo device pflow(4) debuted in OpenBSD 4.5. As you would expect on OpenBSD, pflow is tightly assosciated with PF, and collecting data from an OpenBSD machine (typically a gateway) involves adding the state option pflow to PF rules that you want to collect Netflow data for, much like you would pick rules for logging with log or log (all) options. To wit, a rule for collecting pflow data would look something like this:

pass out log inet proto tcp from <clients> to port $email keep state (pflow)

But then generating pflow data proved so enormously useful in a lot of contexts that the OpenBSD 4.5 release also included an option to set state-defaults that would apply to all rules in the rule set unless specifically excempted. You guessed it, the most popular set in a number of PF shops became

set state-defaults pflow

more or less overnight after the OpenBSD 4.5 release.

Once you have reloaded your rule set with the pflow option in place, you are generating pflow data (in this case, for any traffic that matches a pass rule in the rule set). But to actually get the data to somewhere you can study them, you need to set up both a sensor and collector. The sensor is the pflow interface, which you configure via ifconfig commands, or for a permanent configuration, in the /etc/hostname.pflow0 interface configuration file. The /etc/hostname.pflow0 on the gateway closest to me right now looks like this:

flowsrc flowdst
pflowproto 10

which means, essentially, that any pflow data generated will be sent with a source address of to the collector we hope is listening at, UDP port 9995. Every flow is recorded, and sent to the collector. The flowproto 10 part means we use flow protocol version 10, the latest one with all the newest bells and whistles (which is recommended on OpenBSD only on version 5.5 or newer).

The Collector

Up to this point, you are free to choose any collector at all, or for that matter, let your pflow sensor send data endlessly into the void. In The Book of PF I spend quite a bit of time explaining netflow via Damien Miller's excellent flowd, mainly because it's damned fine software and very well suited for the purpose, but here I'll go the lazy route and show you the tool I actually use, which is nfsen, which comes out of the OpenBSD package system with a usable web interface as a front end to nfcapd and a host of related tools.

Do take some time to click that nfsen reference, the documentation there is quite usable and provides better illustrations than what I can offer at the moment.

Installing nfsen on OpenBSD is, as expected, as simple as can be. On an otherwise normally configured OpenBSD system, the single command

$ sudo pkg_add nfsen

will get you most of the way there. Do read the package readme as the messages instruct you to. Basically, you will need edit the configuration file /etc/nfsen.conf. Adding data sources is likely the only thing you will need to do at first, look for the stanza that looks like this:

%sources = (
    'upstream1'    => { 'port' => '9995', 'col' => '#0000ff', 'type' => 'netflow' },
    'peer1'        => { 'port' => '9996', 'IP' => '' },
    'peer2'        => { 'port' => '9996', 'IP' => '' },

Here you add the sources you have configured earlier. I give all my sources a distinct color (picking among the CSS-style RGB values you youngsters probably know by heart but old farts like me always have to look up), IP address, type and port, so it's easier to tell them apart.

Then you run a perl script to configure the package, start httpd, start the nfsen package (and add it to the pkg_scripts= line in your /etc/rc.conf.local so it will start at next reboot too).

That's all there is to it. Soon the web interface will start filling in the graphs, and you can point and click your way around address ranges, time ranges and a host of other parameters. You will find that every connection you specified in your configuration is indeed logged, and you have all the metadata you asked for.

After a while you will start appreciating that nfsen displays the command line version of your point and click choices, so you have a better starting point for those wrinkles in the data that are not easily or at all accessible via the web interface.

The All-Seeing Eye Of The Evil Network Overlord

You can tell just who, or at least what IP addresses interacted with each other when, how much data was transferred and to of from what services or ports. It stands to reason that in most jurisdictions there are rules about how data of this kind is to be handled and secured. Make sure you deal properly with the data you collect, staying within whatever limits apply to you. But within those limits, here's your chance to be an evil network overlord. Use it wisely.

Netflow data has been used for a number of things. In his very readable book Network Flow Analysis, Michael W. Lucas relates a story about how they pinpointed the source of entry for a Windows worm into a corporate network using netflow data. I've found netflow to be very useful in a number of contexts myself (as briefly mentioned in the earlier DDOS article, and using netflow data to charge for metered access is not unheard of either), but the most striking example I've seen did not involve an attack, merely an intermittent network nuisance that occasionally cost insane amounts of money.

The setting was this: A couple of years ago, I was a relatively new hire in a large corporation that serves IT services of various kinds among others to an almost equally-sized financial firm. In one part of the financial firm there was a place where trades involving dollar values larger than most of us can imagine were made using a telnet interface to something else, and the 80 by 25 character displays were at times not moving at all. Trades were lost because the tiny packets did not arrive on time.

By the time I joined the company, the regular network crew that took care of that particular arm of the financial firm had been unsuccessfully trying to debug and fix the disruptions for quite a while. A call went out for help, and I proposed setting up a Netflow collector much like what I described earlier in the article.

The proposed budget was pretty close to nothing at all besides my time, so I got the go-ahead. The OpenBSD part of the configuration was done inside half an hour, and after peeking at Michael's book I even fished out the right sequence for the Cisco wranglers to input in their gear so useful data started arriving.

Then came the long wait. Graphs were accumulating, and after a while I would put several weeks' graphs on top of each other and hold them up to a light source. They mathched perfectly. I could tell when people started arriving at work, I could tell when trading started in various cities, I could see the dip for lunch breaks, and the traffic peak for the nightly backups was easy to identfy.

But the source of the random network disruption did not turn up in the overall data volumes.

After a few weeks, I asked the local IT support to send me an email as soon as possible when disruptions occurred, with the name and/or IP address of the computers seeing disruption. Soon after, the first messages started arriving. I used the nfsen web interface to search the data around the reported times and looking at the IP ranges. At first, nothing really stood out. There was no sudden increase in data transferred at my sensors.

But then it occurred to me that the overall data volume was not necessarily the problem, so I started looking at hosts in the likely address range by number of flows (as in, number of open connections). That was all it took. Going back over a handful of reports, I noticed that on every occasion, for a few minutes one particular IP address stood out. For a very short time, a few days every week, one host on the network owned essentially all flows that passed by my sensors. No other host came even close.

It turned out that the machine was used to generate some rather heavy duty reports, collecting data from a large number of data sources. My guess is that the reporting software was one of those things that started small and grew over time, and after a few years it became a marked liability, simply because it was connected to the same switch that the traders were using, and reports were generated during trading hours.

I wrote up my report with graphs taken from nfsen (since destroyed and anyway not for public consumption, ever), and recommended that they find a way to move the report generator off to a separate location, perhaps even one with better connectivity to important data sources. I think they took that advice and acted upon it, but I suppose I'll never know for sure.

If you're interested in network traffic monitoring in general and NetFlow tools in particular, you could do worse than pick up a copy of Michael W. Lucas' recent book Network Flow Analysis. Michael chose to work with the flow-tools family of utilities for the book, but he does an outstanding job of explaining the subject in both theory and practical applications. What you read in Michael's book can easily be transferred to other toolsets once you get at grip on the matter.

I've focused mainly on OpenBSD here, but netflow sensors exist or should exist for essentially anything that has a TCP/IP stack. And nfsen works well on Linux and other Unix-like systems, too, I've heard tell.

As I write this I'm still working on the third edition of The Book of PF. The third edition came to be mainly because of changes introduced in OpenBSD 5.5, and the plan we're working towards is to have the book ready in time for the release.

BSDCan: I will be at BSDCan again this year, offering two tutorials (see the Upcoming Talks panel at top right). More details will follow later, but these sessions will be designed mainly from input I receive from prospective attendees, and so will be critically dependent on your input, or even more so than earlier. See you there!

Update 2014-03-01: Thanks to Sebastian Benoit for pointing out that configuring pflow with flowproto 10 is really only well supported on OpenBSD 5.5 and newer.
Update 2014-04-27: PF tables vs html tags sometimes does not end well. Fortunately fixable.