Friday, April 6, 2012

If We Go One Attempt Every Ten Seconds, We're Under The Radar

The Slime Also Evolves: New bruteforce ssh attempts come in at 10 second intervals, and they keep going even if we block them. Is this the warmup to a new iteration of the Hail Mary Cloud?

Note: This post has been updated with a correction, see the end of the article.

Regular readers will remember the activities of the Hail Mary Cloud, which turned up in authentication logs with large numbers of unsuccessful ssh login attempts, apparently coordinated across a large number of source IP addresses and with any individual host in the attacker set making a new attempts at intervals of anything from several seconds to several minutes.


At the time, commentators took these activites either as an indication of a truly inspired idea from a brilliant mind (after all, avoiding detection is essential) or a token of almost unimaginable ineptitude or perhaps just an overdose of faith that if you keep going long enough, even extremely unlikely things will happen.

It's been a litte while now since we last saw the slow, distributed bruteforce attacks at work here at the BSDly labs (we've kept collecting data here), but one curious incident during the last week indicates that somebody, somewhere is still working on ssh cracking scripts that operate on fairly similar methods.

Bruteforce attacks can be fairly easy to detect and head off. In most cases the attacker comes in with a larger than usual number of login attempts in rapid succession from a single IP address, and with modern tools such as OpenBSD's PF packet filter, you can set up rules that use state tracking options to intercept. The phenomenon is common enough that the bruteforce avoidance section is one of the more popular parts of my online PF tutorial (and of course, a slightly expanded version is avavailable in The Book of PF).

I wouldn't publish or recommend anything that I haven't at least tried myself, so just to illustrate,

[Fri Apr 06 14:48:21] peter@skapet:~$ sudo grep bruteforce /etc/pf.conf
table <bruteforce> persist counters
block log (all) quick from <bruteforce>
pass log (all) proto { tcp, udp } to port ssh keep state (max-src-conn 15, max-src-conn-rate 7/4, overload <bruteforce>

The PF rules on BSDly.net's gateway have something much like the published example. This means that a traditional bruteforce attempt will end up something like this:

[Fri Apr 06 15:30:38] peter@skapet:~$ grep 203.34.37.62 /var/log/authlog
Apr 5 17:42:36 skapet sshd[32722]: Failed password for root from 203.34.37.62 port 44936 ssh2
Apr 5 17:42:36 skapet sshd[32722]: Received disconnect from 203.34.37.62: 11: Bye Bye [preauth]
Apr 5 17:42:38 skapet sshd[26527]: Failed password for root from 203.34.37.62 port 45679 ssh2
Apr 5 17:42:38 skapet sshd[26527]: Received disconnect from 203.34.37.62: 11: Bye Bye [preauth]
Apr 5 17:42:41 skapet sshd[29912]: Invalid user db2inst1 from 203.34.37.62
Apr 5 17:42:41 skapet sshd[29912]: Failed password for invalid user db2inst1 from 203.34.37.62 port 46283 ssh2
Apr 5 17:42:41 skapet sshd[29912]: Received disconnect from 203.34.37.62: 11: Bye Bye [preauth]
Apr 5 17:42:43 skapet sshd[30349]: Failed password for root from 203.34.37.62 port 46898 ssh2
Apr 5 17:42:43 skapet sshd[30349]: Received disconnect from 203.34.37.62: 11: Bye Bye [preauth]
Apr 5 17:42:46 skapet sshd[25557]: Invalid user prueba from 203.34.37.62
Apr 5 17:42:46 skapet sshd[25557]: Failed password for invalid user prueba from 203.34.37.62 port 47495 ssh2
Apr 5 17:42:46 skapet sshd[25557]: Received disconnect from 203.34.37.62: 11: Bye Bye [preauth]
Apr 5 17:42:48 skapet sshd[5380]: Failed password for bin from 203.34.37.62 port 48087 ssh2
Apr 5 17:42:48 skapet sshd[5380]: Received disconnect from 203.34.37.62: 11: Bye Bye [preauth]
Apr 5 17:42:51 skapet sshd[23635]: Invalid user postgres from 203.34.37.62
Apr 5 17:42:51 skapet sshd[23635]: Failed password for invalid user postgres from 203.34.37.62 port 48658 ssh2
Apr 5 17:42:51 skapet sshd[23635]: Received disconnect from 203.34.37.62: 11: Bye Bye [preauth]
Apr 5 17:42:54 skapet sshd[2450]: Failed password for root from 203.34.37.62 port 49307 ssh2
Apr 5 17:42:54 skapet sshd[2450]: Received disconnect from 203.34.37.62: 11: Bye Bye [preauth]
Apr 5 17:42:56 skapet sshd[16673]: Failed password for root from 203.34.37.62 port 49910 ssh2
Apr 5 17:42:57 skapet sshd[16673]: Received disconnect from 203.34.37.62: 11: Bye Bye [preauth]
Apr 5 17:42:59 skapet sshd[17522]: Failed password for root from 203.34.37.62 port 50503 ssh2
Apr 5 17:42:59 skapet sshd[17522]: Received disconnect from 203.34.37.62: 11: Bye Bye [preauth]
Apr 5 17:43:02 skapet sshd[4633]: Invalid user mythtv from 203.34.37.62
Apr 5 17:43:02 skapet sshd[4633]: Failed password for invalid user mythtv from 203.34.37.62 port 51218 ssh2
Apr 5 17:43:02 skapet sshd[4633]: Received disconnect from 203.34.37.62: 11: Bye Bye [preauth]
Apr 5 17:43:05 skapet sshd[25728]: Failed password for root from 203.34.37.62 port 51849 ssh2
Apr 5 17:43:05 skapet sshd[25728]: Received disconnect from 203.34.37.62: 11: Bye Bye [preauth]
Apr 5 17:43:08 skapet sshd[10487]: Failed password for root from 203.34.37.62 port 52565 ssh2
Apr 5 17:43:08 skapet sshd[10487]: Received disconnect from 203.34.37.62: 11: Bye Bye [preauth]
Apr 5 17:43:10 skapet sshd[31156]: Failed password for root from 203.34.37.62 port 53264 ssh2
Apr 5 17:43:11 skapet sshd[31156]: Received disconnect from 203.34.37.62: 11: Bye Bye [preauth]
Apr 5 17:43:13 skapet sshd[31956]: Invalid user mmroot from 203.34.37.62
Apr 5 17:43:13 skapet sshd[31956]: Failed password for invalid user mmroot from 203.34.37.62 port 53958 ssh2
Apr 5 17:43:13 skapet sshd[31956]: Received disconnect from 203.34.37.62: 11: Bye Bye [preauth]


And looking up the current contents of the table shows our new perpetrator has indeed been caught:

[Fri Apr 06 15:34:23] peter@skapet:~$ sudo pfctl -t bruteforce -vT show 91.197.131.24
 Cleared:
           Thu Apr 5 20:22:29 2012
 In/Block:           [ Packets: 1                  Bytes: 52                 ]
 In/Pass:            [ Packets: 0                  Bytes: 0                  ]
 Out/Block:          [ Packets: 0                  Bytes: 0                  ]
 Out/Pass:           [ Packets: 0                  Bytes: 0                  ]
200.11.174.131 
Cleared:            Thu Apr 5 19:09:30 2012  
 In/Block:           [ Packets: 1                  Bytes: 52                 ]
 In/Pass:            [ Packets: 0                  Bytes: 0                  ]  Out/Block:          [ Packets: 0                  Bytes: 0                  ]

 Out/Pass:           [ Packets: 0                  Bytes: 0                  ]
203.34.37.62
 Cleared:            Thu Apr 5 17:43:13 2012
 In/Block:           [ Packets: 1                  Bytes: 52                 ]
 In/Pass:            [ Packets: 0                  Bytes: 0                  ]
 Out/Block:          [ Packets: 0                  Bytes: 0                  ]
 Out/Pass:           [ Packets: 0                  Bytes: 0                  ]


The table data show us one more thing worth noting: All of these bruteforcers sent exactly one packet after they were blocked, and gave up right away when they noticed they were blocked.

On Sunday, April 1st 2012, I noticed an unusually high number of ssh login attempts coming from two Chinese addresses (58.214.5.51 and 61.160.76.123), amazingly persistent and for some reason they had not been caught by my bruteforce avoidance rules. Thinking I'd simply adjust my rate settings, I simply added those addresses to the table by hand and started looking at the authentication log versus my rule set. Then a little while later, I noticed that instead of just bowing out after blocking, these two kept going. (I also tweeted about this, however not accurate in all details, at the time)

A little later that same evening, the table looked like this:
[Sun Apr 01 22:58:02] peter@skapet:~$ sudo pfctl -t bruteforce -vT show
58.51.95.75
 Cleared:            Sun Apr  1 22:05:29 2012
 In/Block:           [ Packets: 1                  Bytes: 52                 ]
 In/Pass:            [ Packets: 0                  Bytes: 0                  ]
 Out/Block:          [ Packets: 0                  Bytes: 0                  ]
 Out/Pass:           [ Packets: 0                  Bytes: 0                  ]
58.214.5.51
 Cleared:            Sun Apr  1 14:06:21 2012
 In/Block:           [ Packets: 3324               Bytes: 199440             ]
 In/Pass:            [ Packets: 0                  Bytes: 0                  ]
 Out/Block:          [ Packets: 0                  Bytes: 0                  ]
 Out/Pass:           [ Packets: 0                  Bytes: 0                  ]
61.91.125.115
 Cleared:            Sun Apr  1 03:10:05 2012
 In/Block:           [ Packets: 1                  Bytes: 52                 ]
 In/Pass:            [ Packets: 0                  Bytes: 0                  ]
 Out/Block:          [ Packets: 0                  Bytes: 0                  ]
 Out/Pass:           [ Packets: 0                  Bytes: 0                  ]
61.160.76.123
 Cleared:            Sun Apr  1 14:07:08 2012
 In/Block:           [ Packets: 3262               Bytes: 195720             ]
 In/Pass:            [ Packets: 0                  Bytes: 0                  ]
 Out/Block:          [ Packets: 0                  Bytes: 0                  ]
 Out/Pass:           [ Packets: 0                  Bytes: 0                  ]

The two hosts kept coming, at a rate of roughly one attempt every ten seconds, and apparently ignored the fact that they were blocked in the packet filter rules and would be getting connection refused errors for each attempt.

Looking at the log data (preserved here along with data from various other attempts from other sources in the relevant period), both hosts were busy trying to guess root's password from the time they started until they were blocked. When the block expired after 24 hours, they had both apparently proceeded down similiar lists of user names and were busy with rooter):

Apr  2 14:10:06 skapet sshd[13332]: Invalid user rooter from 61.160.76.123
Apr  2 14:10:06 skapet sshd[13332]: input_userauth_request: invalid user rooter [preauth]
Apr  2 14:10:06 skapet sshd[13332]: Failed password for invalid user rooter from 61.160.76.123 port 46578 ssh2
Apr  2 14:10:06 skapet sshd[13332]: Received disconnect from 61.160.76.123: 11: Bye Bye [preauth]
Apr  2 14:10:14 skapet sshd[30888]: Invalid user rooter from 58.214.5.51
Apr  2 14:10:14 skapet sshd[30888]: input_userauth_request: invalid user rooter [preauth]
Apr  2 14:10:14 skapet sshd[30888]: Failed password for invalid user rooter from 58.214.5.51 port 47587 ssh2
Apr  2 14:10:14 skapet sshd[30888]: Received disconnect from 58.214.5.51: 11: Bye Bye [preauth]

They both kept going afterwards, at roughly the same rates as before. The host at 61.160.76.123 kept varying its rate and at one point sped up enough that it triggered the automatic bruteforce blocking.

After running a fairly familiar alphabetic progression through a list of supposed user names, the remaining host finally gave up during the first hour of April 3rd, by CEST time:

Apr  3 00:36:24 skapet sshd[30287]: Received disconnect from 58.214.5.51: 11: Bye Bye [preauth]
Apr  3 00:36:33 skapet sshd[27318]: Invalid user clodia from 58.214.5.51
Apr  3 00:36:33 skapet sshd[27318]: input_userauth_request: invalid user clodia [preauth]
Apr  3 00:36:33 skapet sshd[27318]: Failed password for invalid user clodia from 58.214.5.51 port 58185 ssh2
Apr  3 00:36:33 skapet sshd[27318]: Received disconnect from 58.214.5.51: 11: Bye Bye [preauth]

Before we go into further details, I have a question for you, dear reader: Did anything like this turn up in your authentication logs during the same rough time frame? If your logs show something similar, please drop me a line at (lightly obfuscated) peter at bsdly dot se.

It could be instructive to compare this last batch with the previous samples. The log format differs slightly, since the previous attempts were aimed at FreeBSD machines, while this last round was aimed at a single OpenBSD host.

The whois information for the two hosts (58.214.5.51 and 61.160.76.123) both point to Chinese networks, as far as I can tell in the same provice and possibly in the same city, Wuxi, which appears to be one of several Chinese tech cities.

The slow rate of the login attempts and the sequence of user names attempted are both similar enough to the earlier distributed attempts that it's possible this is a limited experiment by the developers of the previous bruteforcing malware. The rate of roughly one attempt per host per 10 seconds is a significant speedup compared to the previous attempts, and it fits in the interval where blocking due to the rate of connections would most likely produce an unacceptably high number of false positives.

It will be interesting to see what rate of incoming connection the next full scale attempts will be using. It is possible that the source addresses are somewhere close to the actual whereabouts of the malware developers, but at this point it's pure speculation.

At this point we can only keep watching our logs and make sure that our sshd configurations are the best possible shape. If you need up to date advice on how to configure and use SSH safely, you could do significantly worse than grabbing Michael W. Lucas' recent SSH book SSH Mastery.


Update 2013-04-25: Revisiting the data in preparation for my BSDCan 2013 talk (also to be featured or rather previewed at tonight's BLUG meeting), I realized that a trivial scripting error had lead me to draw false conclusions.

The total number of attempts is correct, but both the number of hosts involved and the number of user names attempted were seriously off. The two hosts I mentioned in the article were the most active, but actually a total of 23 hosts participated, trying for a total of 1081 user names. Full data available here.

It seems the Hail Mary Cloud had shrunk, but not completely vanished as I thought at the time.
Note: A Better Data Source Is Available
Update 2013-06-09: For a faster and more convenient way to download the data referenced here, please see my BSDCan 2013 presentation The Hail Mary Cloud And The Lessons Learned which summarizes this series of articles and provides links to all the data. The links in the presentation point to a copy stored at NUUG's server, which connects to the world through a significantly fatter pipe than BSDly.net has.

8 comments:

  1. I've been seeing increasing numbers of 'slow' attacks against various services we run, generally attempts to brute-force CMSs etc. Spotted one purely by chance on Sunday, swiftly added the annoying IP address to firewall but its leading me to re-evaluate how we catch dodgy behaviour.

    ReplyDelete
  2. I don't watch my logs that much anymore.
    Why? Tarpit.
    To be precise:
    http://xtables-addons.sourceforge.net/

    There is a package in the Ubuntu repo that still has an install 'bug' (the postinst.d run-parts compile into dkms fails if you are running a 3.x kernel) version 1.40.
    The newest version 1.41-1 if downloaded in tar.gz format compiles on my Ubuntu 12.04 beta 2 server.

    This article covers my setup in detail:
    https://dtschmitz.com/blog/linux-server-security-practices-xtables-addons

    This article at Dell does a good job of explaining tarpit:
    http://www.secureworks.com/research/threats/ddos/

    Let me put it to you this way, users have 3 tries at login and then they are put into tarpit where there PC utilization goes up and they hang their connection.

    Give it a try.

    Dietrich T. Schmitz
    Linux Advocate, Human Being

    ReplyDelete
  3. Wow. I just checked one of my company's servers, and there are *thousands* of login attempts all day. Probably millions since the server has been online. I'm getting around 5-6 attempts per second. Here are some IPs:

    189.101.0.183
    96.226.123.22
    223.4.114.1
    218.28.49.62
    187.4.25.90

    They come from all over, does look like a botnet.

    ReplyDelete
  4. On my net with 20 public IPs over the past couple days. Notice one of those has over 15k packets after the ban hammer!
    https://gist.github.com/2339843

    ReplyDelete
  5. I am seeing the same IP addresses in my logs, roughly every two minutes:

    May 8 14:20:31 xxx sshd[1350]: refused connect from 58.214.5.51 (58.214.5.51)
    May 8 14:22:29 xxx sshd[5640]: refused connect from 61.160.76.123 (61.160.76.123)
    May 8 14:24:30 xxx sshd[9816]: refused connect from 61.160.76.123 (61.160.76.123)
    May 8 14:26:25 xxx sshd[18619]: refused connect from 58.214.5.51 (58.214.5.51)
    May 8 14:28:21 xxx sshd[22887]: refused connect from 61.160.76.123 (61.160.76.123)
    May 8 14:30:15 xxx sshd[32129]: refused connect from 58.214.5.51 (58.214.5.51)
    May 8 14:32:10 xxx sshd[4396]: refused connect from 58.214.5.51 (58.214.5.51)

    ReplyDelete
  6. Yes, the low threshold on the PF firewall throttling mechanism is nice for servers. It's managed to bite me though. Today I spent several hours trying to figure out why my web pages only half loaded...

    I noticed that the problem was mainly associated with sites heavy on images, such as Yahoo. It didn't dawn on me until the end of the day that I was using the max-src-conn-rate and max-src-conn (very low) throttling numbers from a server pf.conf script which had been transferred to the internet browser machine.

    I spent sooo much time looking at the block and pass lines in the script, which were correct and so the situation didn't make any sense. Now I need that head-bang emoticon!

    ReplyDelete
  7. You can leave the maintenance of a deny list up to the iptables ipset module, which has a timeout parameter for ip sets. I actually have PermitRootLogin no, but a user section which allows people to try to login as root. If they do, they get added to a blocking ipset. I've posted a blog about it here: http://bneijt.nl/blog/post/honeypot-ssh-with-ipset-and-systemd-journal/

    ReplyDelete

Note: Comments are moderated. On-topic messages will be liberated from the holding queue at semi-random (hopefully short) intervals.

I invite comment on all aspects of the material I publish and I read all submitted comments. I occasionally respond in comments, but please do not assume that your comment will compel me to produce a public or immediate response.

Please note that comments consisting of only a single word or only a URL with no indication why that link is useful in the context will be immediately recycled so those poor electrons get another shot at a meaningful existence.

If your suggestions are useful enough to make me write on a specific topic, I will do my best to give credit where credit is due.