Monday, July 10, 2017

OpenBSD and the modern laptop

Did you think that OpenBSD is suitable only for firewalls and high-security servers? Think again. Here are my steps to transform a modern mid to high range laptop into a useful Unix workstation with OpenBSD.

One thing that never ceases to amaze me is that whenever I'm out and about with my primary laptop at conferences and elsewhere geeks gather, a significant subset of the people I meet have a hard time believing that my laptop runs OpenBSD, and that it's the only system installed.

A typical exchange runs something like,
"So what system do you run on that laptop there?"
"It's OpenBSD. xfce is the window manager, and on this primary workstation I tend to just upgrade from snapshot to snapshot."
"Really? But ..."
and then it takes a bit of demonstrating that yes, the graphics runs with the best available resolution the hardware can offer, the wireless network is functional, suspend and resume does work, and so forth. And of course, yes, I do use that system when writing books and articles too. Apparently heavy users of other free operating systems do not always run them on their primary workstations.

I'm not sure at what time I permanently converted my then-primary workstation to run OpenBSD exclusively, but I do remember that when I took delivery of the ThinkPad R60 (mentioned in this piece) in 2006, the only way forward was to install the most recent OpenBSD snapshot. By mid-2014 the ThinkPad SL500 started falling to pieces, and its replacement was a Multicom Ultrabook W840, manufactured by Clevo. The Clevo Ultrabook has weathered my daily abuse and being dragged to various corners of the world for conferences well, but on the trek to BSDCan 2017 cracks started appearing in the glass on the display and the situation worsened on the return trip.

So the time came to shop around for a replacement. After a bit of shopping around I came back to Multicom, a small computers and parts supplier outfit in rural Åmli in southern Norway, the same place I had sourced the previous one.

One of the things that attracted me to that particular shop and their own-branded offerings is that they will let you buy those computers with no operating system installed. That is of course what you want to do when you source your operating system separately, as we OpenBSD users tend to do.

The last time around I had gone for a "Thin and lightweight" 14 inch model (Thickness 20mm, weight 2.0kg) with 16GB RAM, 240GB SSD for system disk and 1TB HD for /home (since swapped out for a same-size SSD, as the dmesg will show).

Three years later, the rough equivalent with some added oomph for me to stay comfortable for some years to come ended me with a 13.3 inch model, 18mm thick and advertised as 1.3kg (but actually weighing in at 1.5kg, possibly due to extra components), 32GB RAM, 512GB SSD and 2TB harddisk. For now the specification can be viewed online here (the site language is Norwegian, but product names and units of measure are not in fact different).

That system arrived today, in a slender box:



Here are the two machines, the old (2014-vintage) and the new side by side:



The OpenBSD installer is a wonder of straightforward, no-nonsense simplicity that simply gets the job done. Even so, if you are not yet familiar with OpenBSD, it is worth spending some time reading the OpenBSD FAQ's installation guidelines and the INSTALL.$platform file (in our case, INSTALL.amd64) to familiarize yourself with the procedure. If you're following this article to the letter and will be installing a snapshot, it is worth reading the notes on following -current too.

The main hurdle back when I was installing the 2014-vintage 14" model was getting the system to consider the SSD which showed up as sd1 the automatic choice for booting (I solved that by removing the MBR, setting the size of the MBR on the hard drive that showed up as sd0 to 0 and enlarging the OpenBSD part to fill the entire drive).

Let's see how the new one is configured, then. I try running with the default UEFI "Secure boot" option enabled, and it worked.

Here we see the last part of the messages that scroll across the screen when the new laptop boots from the USB thumbdrive that has had the most recent OpenBSD/amd64 install61.fs dd'ed onto it:



And as the kernel messages showed us during boot (yes, that scrolled off the top before I got around to taking the picture), the SSD came up as sd1 while the hard drive registered as sd0. Keep that in mind for later.



After the initial greeting from the installer, the first menu asks what we want to do. This is a new system, so only (A)utoinstall and (I)nstall would have any chance of working. I had not set up for automatic install this time around, so choosing (I)nstall was the obvious thing to do.

The next item the installer wants to know is which keyboard layout to use and to set as the default on the installed system. I'm used to using Norwegian keyboards, so no is the obvious choice for me here. If you want to see the list of available options, you press ? and then choose the one you find the must suitable.

Once you've chosen the keyboard layout, the installer prompts you for the system's host name. This is only the host part, the domain part comes later. I'm sure your site or organization has some kind of policy in place for choice of host names. Make sure you stay inside any local norms, the one illustrated here conforms with what we have here.

Next up the installer asks which network interfaces to configure. A modern laptop such as this one comes with at least two network interfaces: a wireless interface, in this case an Intel part that is supported in OpenBSD with the iwm(4) driver, and a wired gigabit ethernet interface which the installer kernel recognized as re0.

Quite a few pieces the hardware in a typical modern laptop requires the operating system to load firmware onto the device before it can start interacting meaningfully with the kernel. The Intel wireless network parts supported by the iwm(4) driver and the earlier iwn(4) all have that requirement. However, for some reason the OpenBSD project has not been granted permission to distribute the Intel firmware files, so with only the OpenBSD installer it is not possible to use iwm(4) devices during an initial install. So in this initial round I only configure the re0 interface. During the initial post-install boot the rc.firsttime script will run fw_update(1) command that will identify devices that require firmware files and download them from the most convenient OpenBSD firmware mirror site.

My network here has a DHCP server in place, so I simply choose the default dhcp for IPv4 address assignment and autoconf for IPv6.

With the IPv4 and IPv6 addresses set, the installer prompts for the domain name. Once again, the choice was not terribly hard in my case.



On OpenBSD, root is a real user, and you need to set that user's password even if you will rarely if ever log in directly as root. You will need to type that password twice, and as the install documentation states, the installer will only check that the passwords match. It's up to you to set a usefully strong password, and this too is one of the things organizations are likely to have specific guidelines for.

Once root's password is set, the installer asks whether you want to start sshd(8) by default. The default is the sensible yes, but if you answer no here, the installed system will not have any services listening on the system's reachable interfaces.

The next question is whether the machine will run the X Windows system. This is a laptop with a "Full HD" display and well supported hardware to drive it, so the obvious choice here is yes.

I've gotten used to running with xenodm(1) display manager and xfce as the windowing environment, so the question about xenodm is a clear yes too, in my case.

The next question is whether to create at least one regular user during the install. Creating a user for your systems adminstrator during install has one important advantage: the user you create at this point will be a member of the wheel group, which makes it slightly easier to move to other privilege levels via doas(1) or similar.

Here I create a user for myself, and it is added, behind the scenes, to the wheel group.

With a user in place, it is time to decide whether root will be able to log in via ssh. The sensible default is no, which means you too should just press enter here.

The installer guessed correctly for my time zone, so it's another Enter to move forward.

Next up is the part that people have traditionally found the most scary in OpenBSD installing: Disk setup.

If the machine had come with only one storage device, this would have been a no-brainer. But I have a fast SSD that I want to use as the system disk, and a slightly slower and roomier rotating rust device aka hard disk that I want primarily as the /home partition.

I noted during the bsd.rd boot that the SSD came up as sd1 and the hard drive came up as sd0, so we turn to the SSD (sd1) first.

Since the system successfully booted with the "Secure boot" options in place, I go for the Whole disk GPT option and move on to setting partition sizes.

The default suggestion for disk layout makes a lot of sense and will set sensible mount options, but I will be storing /home on a separate device, so I choose the (E)dit auto layout option and use the R for Resize option to redistribute the space left over to the other partitions.

Here is also where you decide the size of the swap space, traditionally on the boot device's b partition. Both crashdumps and suspend to disk use swap space for their storage needs, so if you care about any of these, you will need to allocate at least as much space as the amount of physical RAM installed in the system. Because I could, I allocated the double of that, or 64GB.

For sd0, I once again choose the Whole disk GPT option and make one honking big /home partition for myself.

The installer then goes on to create the file systems, and returns with the prompt to specify where to find install sets.

The USB drive that I dd'ed the install61.fs image to is the system's third sd device (sd2), so choosing disk and specifying sd2 with the subdirectory 6.1/amd64 makes sense here. On the other hand, if your network and the path to the nearest mirror is fast enough, you may actually save time choosing a http network install over installing from a relatively slow USB drive.

Anyway, the sets install proceeds and trundles through what is likely the longest period of forced inactivity that you will have during an OpenBSD install.

The installer verifies the signed sets and installs them.



Once the sets install is done, you get the offer of specifying more sets -- your site could have a site-specific items in an install set -- but I don't have any of those handy, so I just press enter to accept the default done.

If you get the option to correct system time here, accept it and have ntpd(8) set your system clock to a sane setting gleaned from well known NTP servers.

With everything else in place, the installer links the kernel with a unique layout, in what is right now a -current-only feature, but one that will most likely be one of the more talked-about items in the OpenBSD 6.2 release some time in the not too distant future.

With all items on the installer's agenda done, the installer exits and leaves you at a root shell prompt where the only useful action is to type reboot and press enter. Unless, of course you have specific items you know will need to be edited into the configuration before the reboot.

After completing the reboot, the system did unfortunately not, as expected, immediately present the xenodm login screen, but rather the text login prompt.

Looking at the /var/log/Xorg.0.log file pointed to driver problems, but after a little web searching on the obvious keywords, I found this gist note from notable OpenBSD developer Reyk Flöter that gave me the things to paste into my /etc/xorg.conf to yield a usable graphics display for now.

My task for this evening is to move my working environment to new hardware, so after install there are really only two items remaining, in no particular order:
  • move my (too large) accumulation of /home/ data to the new system, and
  • install the same selection of packages on the old machine to the new system.
The first item will take longer, so I shut down all the stuff I normally have running on the laptop such as web browsers, editors and various other client programs, and use pkg_info(1) to create the list of installed packages on the 'from' system:

$ pkg_info -mz >installed_packages

then I transfer the installed_packages file to the fresh system, but not before recording the df -h status of the pristine fresh install:

$ df -h
Filesystem     Size    Used   Avail Capacity  Mounted on
/dev/sd1a     1005M   76.4M    878M     8%    /
/dev/sd0d      1.8T    552K    1.7T     0%    /home
/dev/sd1d     31.5G   12.0K   29.9G     0%    /tmp
/dev/sd1f     98.4G    629M   92.9G     1%    /usr
/dev/sd1g      9.8G    177M    9.2G     2%    /usr/X11R6
/dev/sd1h      108G    218K    103G     0%    /usr/local
/dev/sd1k      9.8G    2.0K    9.3G     0%    /usr/obj
/dev/sd1j     49.2G    2.0K   46.7G     0%    /usr/src
/dev/sd1e     98.4G    5.6M   93.5G     0%    /var

Not directly visible here is the amount of swap configured in the sd1b partition. As I mentioned earlier, crashdumps and suspend to disk both use swap space for their storage needs, so if you care about any of these, you will need to allocate at least as much space as the amount of physical RAM installed in the system. Because I could, I allocated the double of that, or 64GB.

I also take a peek at the old system's /etc/doas.conf and enter the same content on the new system to get the same path to higher privilege that I'm used to having. With those in hand, recreating the set of installed packages on the fresh system is then a matter of a single command:

$ doas pkg_add -l installed_packages

and pkg_add(1) proceeds to fetch and install the same packages I had on the old system.

Then there is the matter of transferring the I-refuse-to-admit-the-actual-number-of gigabytes that make up the content of my home directory. In many environments it would make sense to just restore from the most recent backup, but in my case where the source and destination sit side by side, i chose to go with a simple rsync transfer:

$ rsync  -rcpPCavu 192.168.103.69:/home/peter . | tee -a 20170710-transferlog.txt

(Yes, I'm aware that I could have done something similar with nc and tar, which are both in the base system. But rsync wins by being more easily resumable.)

While the data transfers, there is ample time to check for parts of the old system's configuration that should be transferred to the new one. Setting up the hostname.iwm0 file to hold the config for the wireless networks (see the hostname.if man page) by essentially copying across the previous one is an obvious thing, and this is the time when you discover tweaks you love that were not part of that package's default configuration.

Some time went by while the content transferred, and I can now announce that I'm typing away on the machine that is at the same time both the most lightweight and the most powerful machine I have ever owned.

I am slowly checking and finding that the stuff I care about just works, though I haven't bothered to check whether the webcam works yet. I know you've been dying to see the dmesg, which can be found here. I'm sure I'll get to the bottom of the 'not configured' items (yes, there are some) fairly soon. Look for updates that will be added to the end of this column.

And after all these years, I finally have a machine that matches my beard color:



If you have any questions on running OpenBSD as a primary working environment, I'm generally happy to answer but in almost all cases I would prefer that you use the mailing lists such as misc@openbsd.org or the OpenBSD Facebook group so the question and hopefully useful answers become available to the general public. Browsing the slides for my recent OpenBSD and you user group talk might be beneficial if you're not yet familiar with the system. And of course, comments on this article are welcome.


Update 2017-07-18: One useful thing to do once you have your system up and running is to submit your dmesg to the NYCBUG dmesg database. The one for the system described here is up as http://dmesgd.nycbug.org/index.cgi?do=view&id=3227

Wednesday, April 19, 2017

Forcing the password gropers through a smaller hole with OpenBSD's PF queues

While preparing material for the upcoming BSDCan PF and networking tutorial, I realized that the pop3 gropers were actually not much fun to watch anymore. So I used the traffic shaping features of my OpenBSD firewall to let the miscreants inflict some pain on themselves. Watching logs became fun again.

Yes, in between a number of other things I am currently in the process of creating material for new and hopefully better PF and networking session.

I've been fishing for suggestions for topics to include in the tutorials on relevant mailing lists, and one suggestion that keeps coming up (even though it's actually covered in the existling slides as well as The Book of PF) is using traffic shaping features to punish undesirable activity, such as


What Dan had in mind here may very well end up in the new slides, but in the meantime I will show you how to punish abusers of essentially any service with the tools at hand in your OpenBSD firewall.

Regular readers will know that I'm responsible for maintaining a set of mail services including a pop3 service, and that our site sees pretty much round-the-clock attempts at logging on to that service with user names that come mainly from the local part of the spamtrap addresses that are part of the system to produce our hourly list of greytrapped IP addresses.

But do not let yourself be distracted by this bizarre collection of items that I've maintained and described in earlier columns. The actual useful parts of this article follow - take this as a walkthrough of how to mitigate a wide range of threats and annoyances.

First, analyze the behavior that you want to defend against. In our case that's fairly obvious: We have a service that's getting a volume of unwanted traffic, and looking at our logs the attempts come fairly quickly with a number of repeated attempts from each source address. This similar enough to both the traditional ssh bruteforce attacks and for that matter to Dan's website scenario that we can reuse some of the same techniques in all of the configurations.

I've written about the rapid-fire ssh bruteforce attacks and their mitigation before (and of course it's in The Book of PF) as well as the slower kind where those techniques actually come up short. The traditional approach to ssh bruteforcers has been to simply block their traffic, and the state-tracking features of PF let you set up overload criteria that add the source addresses to the table that holds the addresses you want to block.

I have rules much like the ones in the example in place where there I have a SSH service running, and those bruteforce tables are never totally empty.

For the system that runs our pop3 service, we also have a PF ruleset in place with queues for traffic shaping. For some odd reason that ruleset is fairly close to the HFSC traffic shaper example in The Book of PF, and it contains a queue that I set up mainly as an experiment to annoy spammers (as in, the ones that are already for one reason or the other blacklisted by our spamd).

The queue is defined like this:

   queue spamd parent rootq bandwidth 1K min 0K max 1K qlimit 300

yes, that's right. A queue with a maximum throughput of 1 kilobit per second. I have been warned that this is small enough that the code may be unable to strictly enforce that limit due to the timer resolution in the HFSC code. But that didn't keep me from trying.

And now that I had another group of hosts that I wanted to just be a little evil to, why not let the password gropers and the spammers share the same small patch of bandwidth?

Now a few small additions to the ruleset are needed for the good to put the evil to the task. We start with a table to hold the addresses we want to mess with. Actually, I'll add two, for reasons that will become clear later:

table <longterm> persist counters
table <popflooders> persist counters 

 
The rules that use those tables are:

block drop log (all) quick from <longterm> 


pass in quick log (all) on egress proto tcp from <popflooders> to port pop3 flags S/SA keep state \ 
(max-src-conn 1, max-src-conn-rate 1/1, overload <longterm> flush global, pflow) set queue spamd 

pass in log (all) on egress proto tcp to port pop3 flags S/SA keep state \ 
(max-src-conn 5, max-src-conn-rate 6/3, overload <popflooders> flush global, pflow) 
 
The last one lets anybody connect to the pop3 service, but any one source address can have only open five simultaneous connections and at a rate of six over three seconds.

Any source that trips up one of these restrictions is overloaded into the popflooders table, the flush global part means any existing connections that source has are terminated, and when they get to try again, they will instead match the quick rule that assigns the new traffic to the 1 kilobyte queue.

The quick rule here has even stricter limits on the number of allowed simultaneous connections, and this time any breach will lead to membership of the longterm table and the block drop treatment.

The for the longterm table I already had in place a four week expiry (see man pfctl for detail on how to do that), and I haven't gotten around to deciding what, if any, expiry I will set up for the popflooders.

The results were immediately visible. Monitoring the queues using pfctl -vvsq shows the tiny queue works as expected:

 queue spamd parent rootq bandwidth 1K, max 1K qlimit 300
  [ pkts:     196136  bytes:   12157940  dropped pkts: 398350 bytes: 24692564 ]
  [ qlength: 300/300 ]
  [ measured:     2.0 packets/s, 999.13 b/s ]


and looking at the pop3 daemon's log entries, a typical encounter looks like this:

Apr 19 22:39:33 skapet spop3d[44875]: connect from 111.181.52.216
Apr 19 22:39:33 skapet spop3d[75112]: connect from 111.181.52.216
Apr 19 22:39:34 skapet spop3d[57116]: connect from 111.181.52.216
Apr 19 22:39:34 skapet spop3d[65982]: connect from 111.181.52.216
Apr 19 22:39:34 skapet spop3d[58964]: connect from 111.181.52.216
Apr 19 22:40:34 skapet spop3d[12410]: autologout time elapsed - 111.181.52.216
Apr 19 22:40:34 skapet spop3d[63573]: autologout time elapsed - 111.181.52.216
Apr 19 22:40:34 skapet spop3d[76113]: autologout time elapsed - 111.181.52.216
Apr 19 22:40:34 skapet spop3d[23524]: autologout time elapsed - 111.181.52.216
Apr 19 22:40:34 skapet spop3d[16916]: autologout time elapsed - 111.181.52.216


here the miscreant comes in way too fast and only manages to get five connections going before they're shunted to the tiny queue to fight it out with known spammers for a share of bandwidth.

I've been running with this particular setup since Monday evening around 20:00 CEST, and by late Wednesday evening the number of entries in the popflooders table had reached approximately 300.

I will decide on an expiry policy at some point, I promise. In fact, I welcome your input on what the expiry period should be.

One important takeaway from this, and possibly the most important point of this article, is that it does not take a lot of imagination to retool this setup to watch for and protect against undesirable activity directed at essentially any network service.

You pick the service and the ports it uses, then figure out what are the parameters that determine what is acceptable behavior. Once you have those parameters defined, you can choose to assign to a minimal queue like in this example, block outright, redirect to something unpleasant or even pass with a low probability.

All of those possibilities are part of the normal pf.conf toolset on your OpenBSD system. If you want, you can supplement these mechanisms with a bit of log file parsing that produces output suitable for feeding to pfctl to add to the table of miscreants. The only limits are, as always, the limits of your imagination (and possibly your programming abilities). If you're wondering why I like OpenBSD so much, you can find at least a partial answer in my OpenBSD and you presentation.

FreeBSD users will be pleased to know that something similar is possible on their systems too, only substituting the legacy ALTQ traffic shaping with its somewhat arcane syntax for the modern queues rules in this article.

Will you be attending our PF and networking session in Ottawa, or will you want to attend one elsewhere later? Please let us know at the email address in the tutorial description.



Update 2017-04-23: A truly unexpiring table, and downloadable datasets made available

Soon after publishing this article I realized that what I had written could easily be taken as a promise to keep a collection of POP3 gropers' IP addresses around indefinitely, in a table where the entries never expire.

Table entries do not expire unless you use a pfctl(8) command like the ones mentioned in the book and other resources I referenced earlier in the article, but on the other hand table entries will not survive a reboot either unless you arrange to have table contents stored to somewhere more permanent and restored from there. Fortunately our favorite toolset has a feature that implements at least the restoring part.

Changing the table definition quoted earler to read

 table <popflooders> persist counters file "/var/tmp/popflooders"

takes part of the restoring, and the backing up is a matter of setting up a cron(8) job to dump current contents of the table to the file that will be loaded into the table at ruleset load.

Then today I made another tiny change and made the data available for download. The popflooders table is dumped at five past every full hour to pop3gropers.txt, a file desiged to be read by anything that takes a list of IP addresses and ignores lines starting with the # comment character. I am sure you can think of suitable applications.

In addition, the same script does a verbose dump, including table statistiscs for each entry, to pop3gropers_full.txt for readers who are interested in such things as when an entry was created and how much traffic those hosts produced, keeping in mind that those hosts are not actually blocked here, only subjected to a tiny bandwidth.

As it says in the comment at the top of both files, you may use the data as you please for your own purposes, for any re-publishing or integration into other data sets please contact me via the means listed in the bsdly.net whois record.

As usual I will answer any reasonable requests for further data such as log files, but do not expect prompt service and keep in mind that I am usually in the Central European time zone (CEST at the moment).

I suppose we should see this as a tiny, incremental evolution of the "Cybercrime Robot Torture As A Service" (CRTAAS) concept.

Update 2017-04-29: While the world was not looking, I supplemented the IP address dumps with versions including one with geoiplocation data added and a per country summary based on the geoiplocation data.

Spending a few minutes with an IP address dump like the one described here and whois data is a useful excersise for anyone investigating incidents of this type. This .csv file is based on the 2017-04-29T1105 dump (preserved for reference), and reveals that not only is the majority of attempts from one country but also a very limited number of organizations within that country are responsible for the most active networks.

The spammer blacklist (see this post for background) was of course ripe for the same treatment, so now in addition to the familiar blacklist, that too comes with a geoiplocation annotated version and a per country summary.

Note that all of those files except the .csv file with whois data are products of automatic processes. Please contact me (the email address in the files works) if you have any questions or concerns.

Update 2017-05-17: After running with the autofilling tables for approximately a month, and I must confess, extracting bad login attempts that didn't actually trigger the overload at semi-random but roughly daily intervals, I thought I'd check a few things about the catch. I already knew roughly how many hosts total, but how many were contactin us via IPv6? Let's see:

[Wed May 17 19:38:02] peter@skapet:~$ doas pfctl -t popflooders -T show | wc -l
    5239
[Wed May 17 19:38:42] peter@skapet:~$ doas pfctl -t popflooders -T show | grep -c \:
77

Meaning, that of a total 5239 miscreant trapped, only 77, or just sort of 1.5 per ent tried contacting us via IPv6. The cybercriminals, or at least the literal bottom feeders like the pop3 password gropers, are still behind the times in a number of ways.

Update 2017-06-13: BSDCan 2017 past, and the PF and networking tutorial with OpenBSD session had 19 people signed up for it. We made the slides available on the net here during the presentation and announced them on Twitter and elsewhere just after the session concluded. The revised tutorial was fairly well received, and it is likely that we will be offering roughly equivalent but not identical sessions at future BSD events or other occasions as demand dictates.

Update 2017-07-05: Updated the overload criteria for the longterm table to what I've had running for a while: max-src-conn 1, max-src-conn-rate 1/1. 

Monday, January 9, 2017

A New Year, a New Round of pop3 Gropers from China


They've got a list, and they're sticking to it. Do they even know or care it's my list of spamtraps?

Yes, the Chinese are at it again. Or rather, machines with IP addresses that belong in a small set of Chinese province networks have started a rather intense campaign of trying to access the pop3 mail retrieval protocol on a host in my care, after a longish interval of near-total inactivity.

This is the number of failed pop3 login attempts to my system per day so far in 2017:

January 1:        4
January 2:    145
January 3:      20
January 4:      51
January 5:      32
January 6:      36
January 7:  4036
January 8:  5956
January 9:  5769

Clearly, something happened on January 7th, and whatever started then has not stopped yet. On that day we see a large surge in failed pop3 logins, sustained over the next few days, and almost exclusively attempts at the username part of entries from my list of spamtrap addresses. Another notable feature of this sequence of attempts is that they come almost exclusively from a small set of Chinese networks.

The log of the failed attempts are in raw form here, while this spreadsheet summarises the log data in a format oriented to IP address and username pairs and attempts at each.  The spreadsheet also contains netblock information and the country or territory the range is registered to. (Note: when importing the .csv, please specify the "User name" colum as text, otherwise conversion magic may confuse matters)

The numbers for January 7th onwards would have been even higher had it not been for a few attempts to access accounts that actually exist, with my reaction to block (for 24 hours only) the entire netblock the whois info for the offending IP address. Some of those blocks were quite substantial. I've also taken the liberty of removing those entries with real usernames from the logs.

Now despite urging from friends to publish quickly, I've silently collected data for a few days (really just a continuation of the collecting that started with last year's episode described in the Chinese Hunting Chinese Over POP3 In Fjord Country article, which in turn contains links to the data that by now covers almost a full year).

Now disregarding the handful of real user IDs I've already removed from the data set, the only new user IDs we have seen this year are:

3f6d...3mb2jbrszf_99ckfnhrrbud3
bsdly....3ef3a9ff


The rest were already in the spamtraps list, as user name parts. As you will have guessed, those two have been duly included there as well, with @bsdly.net appended in order to form a somewhat believable spamtrap email address.

What, then can we expect to see over the next few days?

The progression so far has been proceeding from trap user names starting with 0, ascended through the numerics and have now (January 9) moved on to the early alphabetics. The list of spamtraps is just shy of 35,000 entries, and I assume the entries I see here come out of some larger corpus that our somewhat inept cyber-criminals use.

If you too are seeing larger than usual numbers of pop3 login failures and anything even vaguely resembling the patters of mischief described here, I would like to hear from you. If your logs follow a format somewhat resembling mine, it is most likely trivial to modify the scripts (in the same directories as the data) to extract data to the slightly more database- or spreadsheet-friendly CSV.

From my perch here it is difficult to determine whether the people responsible for the networks that feature prominently in the data are cooperating with the cybercriminals or whether they are simply victims of their own substandard security practices.

If you are in a position to shed light on that, I would like to hear from you, and I will do my best to protect your anonymity unless you specify otherwise.

In the meantime, expect the data, (both the full set starting in January 2016 and the 2017-only set) to be updated at frequent, quasi-random intervals.

If you need some background on what the spamtrap list is and some history of related incidents over the last few years, I can recommend reading these articles:

Hey, spammer! Here's a list for you! (July 2007) - a light introduction to greytrapping
Maintaining A Publicly Available Blacklist - Mechanisms And Principles (April 2013) - on greytrapping principles and practice
Effective Spam and Malware Countermeasures - Network Noise Reduction Using Free Tools (2008 - 2014) - a more thorough treatment of the whole spam and malware complex
Password Gropers Take the Spamtrap Bait (August 2014) - the first time I noticed pop3 logins for my spamtraps, and of course
Chinese Hunting Chinese Over POP3 In Fjord Country (August 2016) - about a previous bizarre episode involving Chinese networks and pop3 activity.


Update 2017-02-08: Another round of attempts at usernames that are likely Chinese user names (much like the June 19 2016 onwards cycle described in the Chinese Hunting Chinese Over POP3 In Fjord Country article) started on February 8th, 2017.

The first few hours brought the following user names, with the likely corresponding real life name in the second column:

Name Username
Luo Chun luochun
Luo Fa luofa
Luo Feng luofeng
Luo Hai luohai

These names have been added to the full data as well as the 2017-only portion. The log file (2016 and 2017 version or 2017-only data) contains the entries starting at Feb  8 15:26:45 (times are CET local time). It will be interesting to see how long this cycle lasts. Look for updates to the data at irregular but hopefully frequent intervals.

If you are seeing similar activity, I would like to hear from you, in comments or (these most recent attempts all originate in the 49.64.0.0/11 network (range 49.64.0.0 - 49.95.255.255, also known as  CHINANET-JS or the CHINANET jiangsu province network). The previous cycle involved several distinct Chinese networks, and as we all know, stretched over several months of low intensity activity.






Thursday, December 22, 2016

So somebody is throwing HTML at your sshd. What to do?

Yes, it's exactly as wrong as it sounds. Here's a distraction with bizarre twists for the true log file junkies among you. Happy reading for the holidays!

As will probably not surprise any of my regular readers, I've spent a bit of time recently browsing and processing SSH authentication logs for some of the systems in my care. As usual I browse logs with a view to extracting insights and hopefully at some future date I will be able to present useful material based on analyses of that material.

But sometimes something stands out as just too wrong. Today while browsing archived logs I came across this entry from July:

Jul 8 12:53:17 skapet sshd[88344]: Invalid user <!DOCTYPE from 187.50.71.54 port 57999

That string is the start of an SGML-style document declaration, basically what you would expect to find at the very start of an SGML-ish file such as an HTML document.

Instruct your browser to 'Display source' for this article, and that exact string is the first thing in the HTML source display buffer. But in the context of an authentication log for an SSH service, it's distinctly odd.

And what's more a little later in the same log I found:

Jul  8 20:59:08 skapet sshd[11083]: Invalid user content="text/html; from 175.143.54.193 port 26240

Again, somebody throwing HTML at my sshd, but this time from a different IP address.

This piqued my interest enough that I decided to take a look at whatever else those jokers had been up to:

[Thu Dec 22 19:40:06] peter@skapet:~$ zgrep 187.50.71.54 /var/log/authlog.23.gz
Jul 8 12:53:17 skapet sshd[88344]: Invalid user <!DOCTYPE from 187.50.71.54 port 57999
Jul 8 12:53:17 skapet sshd[88344]: Failed password for invalid user <!DOCTYPE from 187.50.71.54 port 57999 ssh2
Jul 8 12:53:17 skapet sshd[88344]: Connection closed by 187.50.71.54 port 57999 [preauth]
Jul 8 13:02:15 skapet sshd[85203]: Invalid user PUBLIC from 187.50.71.54 port 58123
Jul 8 13:02:15 skapet sshd[85203]: Failed password for invalid user PUBLIC from 187.50.71.54 port 58123 ssh2
Jul 8 13:02:15 skapet sshd[85203]: Connection closed by 187.50.71.54 port 58123 [preauth]
Jul 8 13:11:13 skapet sshd[25261]: Invalid user XHTML from 187.50.71.54 port 57227
Jul 8 13:11:13 skapet sshd[25261]: Failed password for invalid user XHTML from 187.50.71.54 port 57227 ssh2
Jul 8 13:11:13 skapet sshd[25261]: Connection closed by 187.50.71.54 port 57227 [preauth]
Jul 8 13:20:10 skapet sshd[68619]: Invalid user Strict//EN" from 187.50.71.54 port 25941
Jul 8 13:20:10 skapet sshd[68619]: Failed password for invalid user Strict//EN" from 187.50.71.54 port 25941 ssh2
Jul 8 13:20:10 skapet sshd[68619]: Connection closed by 187.50.71.54 port 25941 [preauth]
Jul 8 13:28:58 skapet sshd[96899]: Invalid user <html from 187.50.71.54 port 48462
Jul 8 13:28:58 skapet sshd[96899]: Failed password for invalid user <html from 187.50.71.54 port 48462 ssh2
Jul 8 13:28:58 skapet sshd[96899]: Connection closed by 187.50.71.54 port 48462 [preauth]
Jul 8 13:37:48 skapet sshd[59363]: Invalid user <meta from 187.50.71.54 port 46496
Jul 8 13:37:48 skapet sshd[59363]: Failed password for invalid user <meta from 187.50.71.54 port 46496 ssh2
Jul 8 13:37:48 skapet sshd[59363]: Connection closed by 187.50.71.54 port 46496 [preauth]
Jul 8 13:46:43 skapet sshd[81970]: Invalid user content="text/html; from 187.50.71.54 port 29652
Jul 8 13:46:43 skapet sshd[81970]: Failed password for invalid user content="text/html; from 187.50.71.54 port 29652 ssh2
Jul 8 13:46:43 skapet sshd[81970]: Connection closed by 187.50.71.54 port 29652 [preauth]
Jul 8 13:55:37 skapet sshd[39952]: Invalid user <title>403 from 187.50.71.54 port 45706
Jul 8 13:55:37 skapet sshd[39952]: Failed password for invalid user <title>403 from 187.50.71.54 port 45706 ssh2
Jul 8 13:55:37 skapet sshd[39952]: Connection closed by 187.50.71.54 port 45706 [preauth]
Jul 8 14:04:33 skapet sshd[68947]: Invalid user Forbidden from 187.50.71.54 port 8465
Jul 8 14:04:33 skapet sshd[68947]: Failed password for invalid user Forbidden from 187.50.71.54 port 8465 ssh2
Jul 8 14:04:34 skapet sshd[68947]: Connection closed by 187.50.71.54 port 8465 [preauth]
Jul 8 14:13:29 skapet sshd[42324]: Invalid user is from 187.50.71.54 port 54112
Jul 8 14:13:29 skapet sshd[42324]: Failed password for invalid user is from 187.50.71.54 port 54112 ssh2
Jul 8 14:13:29 skapet sshd[42324]: Connection closed by 187.50.71.54 port 54112 [preauth]
Jul 8 14:22:20 skapet sshd[83537]: Invalid user <style from 187.50.71.54 port 41269
Jul 8 14:22:20 skapet sshd[83537]: Failed password for invalid user <style from 187.50.71.54 port 41269 ssh2
Jul 8 14:22:21 skapet sshd[83537]: Connection closed by 187.50.71.54 port 41269 [preauth]
Jul 8 14:31:06 skapet sshd[53939]: Invalid user body{margin from 187.50.71.54 port 10587
Jul 8 14:31:06 skapet sshd[53939]: Failed password for invalid user body{margin from 187.50.71.54 port 10587 ssh2
Jul 8 14:31:07 skapet sshd[53939]: Connection closed by 187.50.71.54 port 10587 [preauth]
Jul 8 14:40:08 skapet sshd[24320]: Connection closed by 187.50.71.54 port 58537 [preauth]
Jul 8 14:48:57 skapet sshd[97150]: Invalid user fieldset{padding from 187.50.71.54 port 11375
Jul 8 14:48:57 skapet sshd[97150]: Failed password for invalid user fieldset{padding from 187.50.71.54 port 11375 ssh2
Jul 8 14:48:58 skapet sshd[97150]: Connection closed by 187.50.71.54 port 11375 [preauth]
Jul 8 14:57:55 skapet sshd[38951]: Invalid user 10px from 187.50.71.54 port 43776
Jul 8 14:57:55 skapet sshd[38951]: Failed password for invalid user 10px from 187.50.71.54 port 43776 ssh2
Jul 8 14:57:55 skapet sshd[38951]: Connection closed by 187.50.71.54 port 43776 [preauth]
Jul 8 15:07:53 skapet sshd[72492]: Invalid user \^M from 187.50.71.54 port 58382
Jul 8 15:07:53 skapet sshd[72492]: Failed password for invalid user \^M from 187.50.71.54 port 58382 ssh2
Jul 8 15:07:53 skapet sshd[72492]: Failed password for invalid user \^M from 187.50.71.54 port 58382 ssh2
Jul 8 15:07:54 skapet sshd[72492]: Connection closed by 187.50.71.54 port 58382 [preauth]
Jul 8 15:17:05 skapet sshd[68616]: Invalid user 0 from 187.50.71.54 port 3795
Jul 8 15:17:05 skapet sshd[68616]: Failed password for invalid user 0 from 187.50.71.54 port 3795 ssh2
Jul 8 15:17:05 skapet sshd[68616]: Connection closed by 187.50.71.54 port 3795 [preauth]
Jul 8 15:26:09 skapet sshd[14795]: Connection closed by 187.50.71.54 port 59139 [preauth]
Jul 8 15:35:04 skapet sshd[8499]: Invalid user #header{width from 187.50.71.54 port 16030
Jul 8 15:35:04 skapet sshd[8499]: Failed password for invalid user #header{width from 187.50.71.54 port 16030 ssh2
Jul 8 15:44:12 skapet sshd[17233]: Invalid user 2% from 187.50.71.54 port 2551
Jul 8 15:44:12 skapet sshd[17233]: Failed password for invalid user 2% from 187.50.71.54 port 2551 ssh2
Jul 8 15:44:13 skapet sshd[17233]: Connection closed by 187.50.71.54 port 2551 [preauth]
Jul 8 15:53:05 skapet sshd[36380]: Invalid user 2%;font-family from 187.50.71.54 port 35369
Jul 8 15:53:05 skapet sshd[36380]: Failed password for invalid user 2%;font-family from 187.50.71.54 port 35369 ssh2
Jul 8 15:53:05 skapet sshd[36380]: Connection closed by 187.50.71.54 port 35369 [preauth]
Jul 8 16:02:05 skapet sshd[5384]: Invalid user Verdana, from 187.50.71.54 port 10140
Jul 8 16:02:05 skapet sshd[5384]: Failed password for invalid user Verdana, from 187.50.71.54 port 10140 ssh2
Jul 8 16:02:06 skapet sshd[5384]: Connection closed by 187.50.71.54 port 10140 [preauth]
Jul 8 16:11:27 skapet sshd[80640]: Invalid user #content{margin from 187.50.71.54 port 27941
Jul 8 16:11:27 skapet sshd[80640]: Failed password for invalid user #content{margin from 187.50.71.54 port 27941 ssh2
Jul 8 16:20:24 skapet sshd[71772]: Invalid user <div from 187.50.71.54 port 5467
Jul 8 16:20:24 skapet sshd[71772]: Failed password for invalid user <div from 187.50.71.54 port 5467 ssh2
Jul 8 16:20:25 skapet sshd[71772]: Connection closed by 187.50.71.54 port 5467 [preauth]
Jul 8 16:29:31 skapet sshd[22288]: Invalid user Error</h1></div>\^M from 187.50.71.54 port 2932
Jul 8 16:29:31 skapet sshd[22288]: Failed password for invalid user Error</h1></div>\^M from 187.50.71.54 port 2932 ssh2
Jul 8 16:29:31 skapet sshd[22288]: Connection closed by 187.50.71.54 port 2932 [preauth]
Jul 8 16:38:32 skapet sshd[64659]: Invalid user id="content">\^M from 187.50.71.54 port 44037
Jul 8 16:38:32 skapet sshd[64659]: Failed password for invalid user id="content">\^M from 187.50.71.54 port 44037 ssh2
Jul 8 16:38:33 skapet sshd[64659]: Connection closed by 187.50.71.54 port 44037 [preauth]
Jul 8 16:47:47 skapet sshd[60396]: Invalid user class="content-container"><fieldset>\^M from 187.50.71.54 port 50741
Jul 8 16:47:47 skapet sshd[60396]: Failed password for invalid user class="content-container"><fieldset>\^M from 187.50.71.54 port 50741 ssh2
Jul 8 16:56:46 skapet sshd[84720]: Invalid user Access from 187.50.71.54 port 56868
Jul 8 16:56:46 skapet sshd[84720]: Failed password for invalid user Access from 187.50.71.54 port 56868 ssh2
Jul 8 16:56:46 skapet sshd[84720]: Connection closed by 187.50.71.54 port 56868 [preauth]
Jul 8 17:05:47 skapet sshd[39792]: Invalid user denied.</h2>\^M from 187.50.71.54 port 55262
Jul 8 17:05:47 skapet sshd[39792]: Failed password for invalid user denied.</h2>\^M from 187.50.71.54 port 55262 ssh2
Jul 8 17:05:47 skapet sshd[39792]: Connection closed by 187.50.71.54 port 55262 [preauth]
Jul 8 17:14:42 skapet sshd[2165]: Invalid user do from 187.50.71.54 port 16650
Jul 8 17:14:43 skapet sshd[2165]: Failed password for invalid user do from 187.50.71.54 port 16650 ssh2
Jul 8 17:14:43 skapet sshd[2165]: Connection closed by 187.50.71.54 port 16650 [preauth]
Jul 8 17:23:39 skapet sshd[45938]: Invalid user have from 187.50.71.54 port 15855
Jul 8 17:23:39 skapet sshd[45938]: Failed password for invalid user have from 187.50.71.54 port 15855 ssh2
Jul 8 17:23:39 skapet sshd[45938]: Connection closed by 187.50.71.54 port 15855 [preauth]
Jul 8 17:32:35 skapet sshd[64595]: Invalid user to from 187.50.71.54 port 64962
Jul 8 17:32:35 skapet sshd[64595]: Failed password for invalid user to from 187.50.71.54 port 64962 ssh2
Jul 8 17:32:35 skapet sshd[64595]: Connection closed by 187.50.71.54 port 64962 [preauth]
Jul 8 17:41:30 skapet sshd[99157]: Invalid user this from 187.50.71.54 port 63460
Jul 8 17:41:30 skapet sshd[99157]: Failed password for invalid user this from 187.50.71.54 port 63460 ssh2 

Jul 8 17:41:30 skapet sshd[99157]: Connection closed by 187.50.71.54 port 63460 [preauth]
Jul 8 17:50:27 skapet sshd[60500]: Invalid user or from 187.50.71.54 port 47364
Jul 8 17:50:27 skapet sshd[60500]: Failed password for invalid user or from 187.50.71.54 port 47364 ssh2
Jul 8 17:50:27 skapet sshd[60500]: Connection closed by 187.50.71.54 port 47364 [preauth]
Jul 8 17:59:26 skapet sshd[57379]: Invalid user using from 187.50.71.54 port 60084
Jul 8 17:59:26 skapet sshd[57379]: Failed password for invalid user using from 187.50.71.54 port 60084 ssh2
Jul 8 17:59:26 skapet sshd[57379]: Connection closed by 187.50.71.54 port 60084 [preauth]
Jul 8 18:08:22 skapet sshd[64892]: Invalid user credentials from 187.50.71.54 port 18558
Jul 8 18:08:22 skapet sshd[64892]: Failed password for invalid user credentials from 187.50.71.54 port 18558 ssh2
Jul 8 18:08:22 skapet sshd[64892]: Connection closed by 187.50.71.54 port 18558 [preauth]
Jul 8 18:17:19 skapet sshd[22377]: Invalid user you from 187.50.71.54 port 46996
Jul 8 18:17:19 skapet sshd[22377]: Failed password for invalid user you from 187.50.71.54 port 46996 ssh2
Jul 8 18:17:19 skapet sshd[22377]: Connection closed by 187.50.71.54 port 46996 [preauth]
Jul 8 18:24:50 skapet sshd[98670]: Connection closed by 187.50.71.54 port 40682 [preauth]


The other IP address offered up:

[Thu Dec 22 19:39:24] peter@skapet:~$ zgrep 175.143.54.193 /var/log/authlog.23.gz
Jul 8 16:10:42 skapet sshd[79062]: Connection closed by 175.143.54.193 port 61453 [preauth]
Jul 8 17:01:03 skapet sshd[28839]: Connection closed by 175.143.54.193 port 59520 [preauth]
Jul 8 17:49:47 skapet sshd[1472]: Connection closed by 175.143.54.193 port 39552 [preauth]
Jul 8 18:34:12 skapet sshd[58208]: Connection closed by 175.143.54.193 port 59520 [preauth]
Jul 8 19:19:12 skapet sshd[93151]: Connection closed by 175.143.54.193 port 6465 [preauth]
Jul 8 20:07:33 skapet sshd[84813]: Connection closed by 175.143.54.193 port 39552 [preauth]
Jul 8 20:59:08 skapet sshd[11083]: Invalid user content="text/html; from 175.143.54.193 port 26240
Jul 8 20:59:08 skapet sshd[11083]: Failed password for invalid user content="text/html; from 175.143.54.193 port 26240 ssh2
Jul 8 20:59:08 skapet sshd[11083]: Connection closed by 175.143.54.193 port 26240 [preauth]
Jul 8 21:47:54 skapet sshd[50641]: Connection closed by 175.143.54.193 port 59520 [preauth]
Jul 8 22:38:16 skapet sshd[33990]: Invalid user Forbidden from 175.143.54.193 port 64640
Jul 8 22:38:16 skapet sshd[33990]: Failed password for invalid user Forbidden from 175.143.54.193 port 64640 ssh2
Jul 8 22:38:16 skapet sshd[33990]: Connection closed by 175.143.54.193 port 64640 [preauth]
Jul 8 23:29:47 skapet sshd[84765]: Invalid user is from 175.143.54.193 port 49280
Jul 8 23:29:47 skapet sshd[84765]: Failed password for invalid user is from 175.143.54.193 port 49280 ssh2
Jul 8 23:29:47 skapet sshd[84765]: Connection closed by 175.143.54.193 port 49280 [preauth]
Jul 9 00:18:32 skapet sshd[75290]: Connection closed by 175.143.54.193 port 13952 [preauth]
Jul 9 01:07:11 skapet sshd[15889]: Connection closed by 175.143.54.193 port 16000 [preauth]
Jul 9 01:54:19 skapet sshd[3570]: Connection closed by 175.143.54.193 port 22144 [preauth]
Jul 9 02:38:44 skapet sshd[212]: Connection closed by 175.143.54.193 port 57472 [preauth]
Jul 9 03:24:00 skapet sshd[38938]: Connection closed by 175.143.54.193 port 50304 [preauth]
Jul 9 04:08:22 skapet sshd[60530]: Connection closed by 175.143.54.193 port 21481 [preauth]
Jul 9 04:55:14 skapet sshd[77880]: Connection closed by 175.143.54.193 port 40064 [preauth]
Jul 9 05:45:26 skapet sshd[65360]: Invalid user 0;color from 175.143.54.193 port 20096
Jul 9 05:45:26 skapet sshd[65360]: Failed password for invalid user 0;color from 175.143.54.193 port 20096 ssh2
Jul 9 05:45:26 skapet sshd[65360]: Connection closed by 175.143.54.193 port 20096 [preauth]
Jul 9 06:35:50 skapet sshd[49775]: Invalid user #header{width from 175.143.54.193 port 8320
Jul 9 06:35:50 skapet sshd[49775]: Failed password for invalid user #header{width from 175.143.54.193 port 8320 ssh2
Jul 9 07:24:21 skapet sshd[88261]: Invalid user 2% from 175.143.54.193 port 57472
Jul 9 07:24:21 skapet sshd[88261]: Failed password for invalid user 2% from 175.143.54.193 port 57472 ssh2
Jul 9 07:24:22 skapet sshd[88261]: Connection closed by 175.143.54.193 port 57472 [preauth]
Jul 9 08:16:55 skapet sshd[79482]: Invalid user 2%;font-family from 175.143.54.193 port 57984
Jul 9 08:16:55 skapet sshd[79482]: Failed password for invalid user 2%;font-family from 175.143.54.193 port 57984 ssh2
Jul 9 08:16:55 skapet sshd[79482]: Connection closed by 175.143.54.193 port 57984 [preauth]
Jul 9 09:05:58 skapet sshd[67909]: Connection closed by 175.143.54.193 port 38016 [preauth]
Jul 9 09:57:24 skapet sshd[51227]: Connection closed by 175.143.54.193 port 22144 [preauth]
Jul 9 10:47:35 skapet sshd[89081]: Invalid user 



The sequences become a little easier to read if we extract the user field from the "Invalid user ..." messages:

[Thu Dec 22 20:23:23] peter@skapet:~$ zgrep 187.50.71.54 /var/log/authlog.23.gz | grep Invalid | awk '{print $8}'
<!DOCTYPE
PUBLIC
XHTML
Strict//EN"
<html <meta content="text/html;
<title>403
Forbidden
is
<style
body{margin
fieldset{padding
10px \^M
0
#header{width
2%
2%;font-family
Verdana,
#content{margin
<div
Error</h1></div>\^M
id="content">\^M
class="content-container"><fieldset>\^M
Access
denied.</h2>\^M
do
have
to
this
or
using
credentials
you
 


Now looking at what came from the the other IP address we get

[Fri Dec 23 00:51:28] peter@skapet:~$ zgrep 175.143.54.193 /var/log/authlog.23.gz | grep Invalid | awk '{print $8}'
content="text/html;
Forbidden
is
0;color
#header{width
2%
2%;font-family
<div
Error</h1></div>\^M
id="content">\^M
Access
have
using
you


Well, definitely HTML-ish, but no proper document start. Distinctly odd.

The two machines involved are apparently far apart geographically (one in Brazil and the other in Malaysia if the data from whois is anything to go by). It is of course possible that they're still connected somehow, perhaps compromised by the same group of cybercriminals and run by the same operators.

These operators then fatfingered some command or other, and their charges started pushing bizarre streams of HTML at unsuspecting SSH servers in the great elsewhere of the Internet. What they were actually trying to achieve I suspect we'll never know, but HTML was involved.

HTML is also part of the problem in one of the other bizarre phenomena I find at semi-random intervals in the SSH authentication logs.

Here are some samples from the same preserved log file:

[Thu Dec 22 20:1
6:36] peter@skapet:~$ zgrep "Bad protocol" /var/log/authlog.23.gz                                 
Jul  6 19:03:17 skapet sshd[28549]: Bad protocol version identification 'GET / HTTP/1.1' from 107.179.242.130 port 36147
Jul  8 16:14:07 skapet sshd[89181]: Bad protocol version identification 'GET / HTTP/1.1' from 204.9.214.98 port 49385
Jul  8 20:15:15 skapet sshd[28469]: Bad protocol version identification 'GET / HTTP/1.1' from 189.90.20.100 port 55039
Jul  9 10:56:40 skapet sshd[67430]: Bad protocol version identification 'GET / HTTP/1.1' from 195.88.41.10 port 52504


Again, it's not clear what these operations were attempting to do, but to me this looks like they were expecting to find either a web server or perhaps a web proxy listening on port 22.

Just like the first kind of web-to-ssh stupidity this won't actually get the requester anything and you can safely ignore both kinds of activity if you see traces of them in your own logs.

That is, if you have seen something similar in your own logs and you would care to share, I would like to hear from you, via email or in the comments below.

Even more so if you have any input on the question 'what were these clowns trying to achieve?'.

If the log analyses or related activities turn up any useful insights, you won't need to go far from here to check the results.

Good night and good luck.


Update 2016-12-23: Among the various comments that the initial version of this piece generated, two stand out as particularly useful.

The first came in a Facebook comment on my post about the story there, from my former colleague Egil Mõller, who wrote:

"Maybe they read password guesses to try from a central REST service, but that service has somehow broken, and is serving a default error message (which, since it's REST, is most likely in html)?"

The other came from OpenBSD developer Stuart Henderson, who tweeted:



If the link Twitter gave me doesn't work, here's the plaintext of Stuart's tweet:

"@pitrh the GETs aren't all that odd - could easily be scanning via proxy. I see similar on SMTP too. Not sure about the html though.
1:00 PM - 23 Dec 2016 "

I must admit I had not noticed any GETs in any SMTP related logs on my systems, but now I an honor bound to check. Stuart has given me a task, and I must finish it separately.

Also, I really like Egil's input here, because it fits so well with the data we have. In the meantime I discovered more data from a second host. Unfortunately the actual logs had been rotated out of existence, but it was still possible to piece together data on failed logins from the summaries logsentry sends me.

It appears that one machine apparently located in Hong Kong that had been trying a few logins earlier that month, with no apparent succcess, started spitting HTML at roughly the same time the other two did.

Here is all the activity in early July 2016 from that host:

Jul  4 01:08:19 delilah sshd[13635]: Failed password for invalid user bob from 218.188.213.5 port 38728 ssh2
Jul  4 01:18:28 delilah sshd[26798]: Failed password for root from 218.188.213.5 port 9224 ssh2
Jul  4 01:18:29 delilah sshd[26798]: Failed password for root from 218.188.213.5 port 9224 ssh2
Jul  4 01:27:13 delilah sshd[24543]: Failed password for invalid user ts from 218.188.213.5 port 9224 ssh2
Jul  4 01:35:54 delilah sshd[16906]: Failed password for invalid user pi from 218.188.213.5 port 9224 ssh2
Jul  7 18:30:48 skapet sshd[89165]: Failed password for invalid user admin from 218.188.213.5 port 43498 ssh2
Jul  7 18:40:33 skapet sshd[35698]: Failed password for invalid user lp from 218.188.213.5 port 9224 ssh2
Jul  7 18:50:21 skapet sshd[21112]: Failed password for root from 218.188.213.5 port 9224 ssh2
Jul  9 11:33:25 delilah sshd[10234]: Failed password for invalid user <!DOCTYPE from 218.188.213.5 port 46959 ssh2
Jul  9 11:42:28 delilah sshd[10230]: Failed password for invalid user PUBLIC from 218.188.213.5 port 9224 ssh2
Jul  9 11:51:36 delilah sshd[25489]: Failed password for invalid user XHTML from 218.188.213.5 port 9224 ssh2
Jul  9 12:09:47 delilah sshd[28023]: Failed password for invalid user <html from 218.188.213.5 port 9224 ssh2
Jul  9 12:18:51 delilah sshd[9873]: Failed password for invalid user <meta from 218.188.213.5 port 9224 ssh2
Jul  9 12:28:01 delilah sshd[13890]: Failed password for invalid user content="text/html; from 218.188.213.5 port 9224 ssh2
Jul  9 12:37:01 delilah sshd[10856]: Failed password for invalid user <title>403 from 218.188.213.5 port 9224 ssh2
Jul  9 12:46:04 delilah sshd[19947]: Failed password for invalid user Forbidden from 218.188.213.5 port 9224 ssh2
Jul  9 13:04:32 delilah sshd[22444]: Failed password for invalid user <style from 218.188.213.5 port 9224 ssh2
Jul  9 13:13:14 delilah sshd[15268]: Failed password for invalid user body{margin from 218.188.213.5 port 9224 ssh2
Jul  9 13:22:31 delilah sshd[19611]: Failed password for invalid user Helvetica, from 218.188.213.5 port 9224 ssh2
Jul  9 13:31:35 delilah sshd[15652]: Failed password for invalid user fieldset{padding from 218.188.213.5 port 59101 ssh2
Jul  9 13:40:39 delilah sshd[15607]: Failed password for invalid user 10px from 218.188.213.5 port 9224 ssh2
Jul  9 13:51:00 delilah sshd[18900]: Failed password for invalid user \^M from 218.188.213.5 port 9224 ssh2
Jul  9 13:51:00 delilah sshd[18900]: Failed password for invalid user \^M from 218.188.213.5 port 9224 ssh2
Jul  9 14:00:09 delilah sshd[24794]: Failed password for invalid user 0 from 218.188.213.5 port 9224 ssh2
Jul  9 14:18:23 delilah sshd[22897]: Failed password for invalid user #header{width from 218.188.213.5 port 9224 ssh2
Jul  9 14:27:11 delilah sshd[12713]: Failed password for invalid user 2% from 218.188.213.5 port 9224 ssh2
Jul  9 14:35:56 delilah sshd[32320]: Failed password for invalid user 2%;font-family from 218.188.213.5 port 9224 ssh2
Jul  9 14:44:46 delilah sshd[30676]: Failed password for invalid user Verdana, from 218.188.213.5 port 9224 ssh2
Jul  9 14:53:46 delilah sshd[12799]: Failed password for invalid user #content{margin from 218.188.213.5 port 9224 ssh2
Jul  9 15:02:29 delilah sshd[19535]: Failed password for invalid user <div from 218.188.213.5 port 9224 ssh2
Jul  9 15:11:17 delilah sshd[6404]: Failed password for invalid user Error</h1></div>\^M from 218.188.213.5 port 9224 ssh2
Jul  9 15:20:08 delilah sshd[2837]: Failed password for invalid user id="content">\^M from 218.188.213.5 port 9224 ssh2
Jul  9 15:29:13 delilah sshd[7831]: Failed password for invalid user class="content-container"><fieldset>\^M from 218.188.213.5 port 9224 ssh2
Jul  9 15:38:05 delilah sshd[12172]: Failed password for invalid user Access from 218.188.213.5 port 9224 ssh2
Jul  9 15:46:56 delilah sshd[23460]: Failed password for invalid user denied.</h2>\^M from 218.188.213.5 port 9224 ssh2
Jul  9 15:55:48 delilah sshd[19891]: Failed password for invalid user do from 218.188.213.5 port 9224 ssh2
Jul  9 16:13:29 delilah sshd[5999]: Failed password for invalid user to from 218.188.213.5 port 9224 ssh2
Jul  9 16:22:20 delilah sshd[17535]: Failed password for invalid user this from 218.188.213.5 port 9224 ssh2
Jul  9 16:31:11 delilah sshd[8428]: Failed password for invalid user or from 218.188.213.5 port 9224 ssh2
Jul  9 16:40:00 delilah sshd[2522]: Failed password for invalid user using from 218.188.213.5 port 9224 ssh2
Jul  9 16:48:52 delilah sshd[15034]: Failed password for invalid user credentials from 218.188.213.5 port 9224 ssh2
Jul  9 16:57:43 delilah sshd[14515]: Failed password for invalid user you from 218.188.213.5 port 53073 ssh2


If we again extract only the user name field, we get:

bob
ts
pi
admin
lp
<!DOCTYPE
PUBLIC
XHTML
<html
<meta
content="text/html;
<title>403
Forbidden
<style
body{margin
Helvetica,
fieldset{padding
10px
\^M
\^M
0
#header{width
2%
2%;font-family
Verdana,
#content{margin
<div
Error</h1></div>\^M
id="content">\^M
class="content-container"><fieldset>\^M
Access
denied.</h2>\^M
do
to
this
or
using
credentials
you


From this we see that this host was already busy poking us at long intervals with 'likely' user names, then suddenly started spewing HTML instead of likely user names in roughly the same time frame as the other two.

Seen from my perch here, this serves to validate Egil's suggestion that a common back end system started misbehaving is what caused the odd activity we're seeing.

And the apparent coordination brings to mind the Hail Mary Cloud incidents that we reported on earlier.

I suppose further digging in the logs is warranted.

If you would like to join me in the hunt for more of this, please let me know.

Tuesday, November 1, 2016

The Possibly Russian Fingerprints on the Shadow Brokers' Trick or Treat Package


Whether a result of clumsiness of a bored operator or deliberate subterfuge, there are clues that the supposed NSA front Equation Group operated out of Russia. The question remains:  What were they doing that for?

As you've probably heard already, an outfit calling themselves the Shadow Brokers published a list of what is supposedly hosts that had been compromised by the NSA and subsequently used as staging servers for whatever the NSA wanted to throw at their adversaries, with some other shady outfit known as the Equation Group actually doing the cyber-cyber thing.

The Shadow Brokers' message with links to their material is available in a number of places, among them this Medium article, which seems to be simply the text of the file message5.txt.asc, which is one of three items at both download locations.

The "list" is actually a compressed and encrypted tar archive (familiar to Unix users everywhere), which expands to a directory structure where the first level is the directory trickortreat, with two subdirectories intonation and pitchimpair, both of which in turn have numerous subdirectories with names that follow the convention

    host.domain.tld___n.m.o.p

that is, a fully qualified domain name for a host, three underscore characters and the corresponding IP version four address in the common four octet decimal notation. Each of these directories then have small text files that appear to be data about a specific program or feature.

For example, the directory intonation/hakuba.janis.or.jp___210.232.42.3 contains only the file jackladder, with the content

INTONATION___hakuba.janis.or.jp___210.232.42.3___20000822-135045() {
    ## JACKLADDER Version:2.0 OS:sparc-sun-solaris2.6
}


I take this to mean that INTONATION, whatever that is, contacted the host hakuba.janis.or.jp at the IP address 210.232.42.3, and peeking at the next line which gives us the OS version, it's my educated guess that the last string of the first line is the date in YYYYMMDD and the patch level for the operating system recorded.

The second line, or the body of the curly braces ({}) part if you prefer, tells us the JACKLADDER version and the operating system.

Other subdirectories have several files that appear to follow roughly the same format, and some record other parameters such as trickortreat/intonation/msgstore2.pldtprv.net___192.168.120.3, where the file orangutan

INTONATION___msgstore2.pldtprv.net___192.168.120.3___20021114-120148() {
    ## ORANGUTAN Version:1.4 OS:sparc-sun-solaris2.8
    export CONFIG_KEYS="81733968 69bb0b91 8b6400d6"
}


also appears to record the content of an environment variable, possibly one that the ORANGUTAN software needs to see exported to its environment in order to work as intended on that host.

Basically, this looks like what a fairly well automated system would leave behind while performing a number of operations targeted at various hosts, in a directory structure where humans will be able to find what they need to look at quickly. In my line of work, it's fairly common for such things as system logs to be collected in file system structures much like what we see here.

For your convenience if you want to study the material yourself and can't really be bothered to figure out how to extract the clear text from the gpg encrypted archive, I've put the plaintext tar archive here and the extracted files here for you to browse at your own pace.

My initial plan when I downloaded and decrypted the material was to check whether any of the hostnames or IP addresses in this material matched any of the entries in my records, such as the Hail Mary Cloud cycle that targeted SSH servers or the more recent password guessing efforts aimed at POP3 mail servers.

But then I noticed while reading the analyses of other geeks who had gotten around to doing their thing that the lists of IP addresses all had in them some addresses that should not have been there at all.

The entire 10.0.0.0/8 network was set aside way back in RFC1918 (February 1996) as one of several non-routeable ranges for private use in local area or campus networks. The way the world and IP version four works, if you have a local network at home or at work (even in large multinational enterprises), more likely than not your machines have addresses in one of these ranges:

     10.0.0.0        -   10.255.255.255  (10/8 prefix)
     172.16.0.0      -   172.31.255.255  (172.16/12 prefix)
     192.168.0.0     -   192.168.255.255 (192.168/16 prefix)

and something between those hosts and the Internet that does the network address translation (NAT) so all traffic from your network appears to come from a routable address.

Even on machines that have routeable addresses, it is not uncommon that one or more other interfaces are configured with RFC1918 addresses to pass internal but necessary traffic such as administrator logons, backups and other administrative tasks somewhere that does not interfere with the internet-facing production traffic.

Still, the Shadow Brokers' Trick or Treat package contains a handful of directories for hosts that only give internal, non-routable (RFC1918) addresses:

    intonation/butt-head.mos.ru___10.30.1.130
    intonation/mcd-su-2.mos.ru___10.34.100.2
    intonation/postbox.mos.ru___10.30.10.32
    intonation/webserv.mos.ru___10.30.10.2
    intonation/msgstore2.pldtprv.net___192.168.120.3


If your most convenient route to a machine to a specific machine is to an interface on that machine that has a local area network address, that is a strong indicator that you are working from that local network.

The first four hosts are in a Russian domain which is still operating, apparently out of Russia. The last domain, pldtprv.net, appears to be no longer active but I assume a diligent search will turn up clues about where they were based and when they were operating.
This could mean that the supposed NSA front Equation Group was actually operating from inside Russia, or at the very least had establised a 'forward base' there which was so well connected (via virtual private networks (VPNs) or other means) to their home ground that the least costly route from wherever those scripts ran to one or more Russian hosts was via those machines' internal administrative or backup interfaces.

I repeat, this, that private, nonrouteable IP addresses appear in a place you would expect to find public and routeable addresses, is a strong indicator that the Equation Group ran their activities from a local network in Russia, likely rubbing shoulders with whoever operates the mos.ru domain.

This is also a sign that whoever ran those scripts was a little careless about their routing at some point. But it could equally well mean that somebody, somewhere, is very adept at inserting clues that may in fact be false and misleading.

I probably will go forward with the comparison of this IP address and hostname hoard with my accumulated logs of less than desirable activity at some point. In the meantime I welcome your feedback on this story in comments or (if you don't want your name known and really only want me to paraphrase you in public) via email.

Good night and good luck.



Update 2016-11-02: Added the 'I repeat ...' paragraph to emphasize that finding private and nonrouteable addresses where you would otherwise expect find public and routeable ones is a strong clue that the perpetrators were operating from that local network. A surprising number of security professionals apparently miss that point.