Around the end of February, rumors started floating around about a new reflector called memcached that could be used to fuel distributed denial-of-service (DDoS) attacks. A day or so later, defenders were prepping for huge attacks, but threat actors were even quicker to capitalize on their newest toy.
On Feb. 28, exposed memcached servers were used to fuel a 1.3-TB-per-second attack, the largest recorded to date. Since the incident, security professionals have made great strides toward limiting the number of exposed memcached systems. While the speed at which these reflection attacks unfolded was much faster than usual, memcached shows a pattern of actions that’s becoming familiar in the DDoS world.
Memcached: A Brief History
Memcached isn’t a new service by any means. Designed to take the load off of memory usage, memcached is a daemon that enables remote storage and quick retrieval of data with little overhead. The vulnerability that makes it so dangerous as a reflector was first highlighted at Black Hat 2010. Like many of the cyberthreats we face, the problem with memcached is that it has been exposed to the internet when it’s only meant to be used internally. The original research indicated that an attacker could replace the contents of the key stores used by the service if the Transmission Control Protocol (TCP) port 11211 was exposed and authentication was not enabled for the service.
At that point, the memcached vulnerability was concerning, but not terribly useful for most attackers. Fast forward to the end of this past February, and the situation changed significantly. The first part of the problem was that several different distributions of Linux had changed their default configurations. The memcached service was configured in an open state on all interfaces by default, rather than needing to be explicitly opened by administrators. The second part of the problem came when researchers realized that memcached was open on User Datagram Protocol (UDP) port 11211 and wasn’t requiring authentication on tens of thousands of systems around the globe.
Since they are connectionless, UDP protocols are inherently less secure than TCP. These protocols allow attackers to spoof the destination for traffic, which is the basis for most reflection attacks. The third leg of the problem comes in two forms. If the attacker just wants to use whatever data is in the memcached key stores, he or she will get a moderate amplification rate to use for DDoS purposes. But a threat actor can also take advantage of the lack of authentication with memcached and stuff as much information as the system can handle into the key stores.
Once the service is stuffed to the gills, all it takes is a 203-byte request to unleash an attack of up to 100 MB response per vulnerable server. Before memcached, the highest amplification factor an attacker could expect from a service such as Network Time Protocol (NTP) or Domain Name System (DNS) might be in the 1,000 to 2,000 range at best. Compare that to memcached, with an amplification factor somewhere between 10,000 and 50,000, and you can see why attackers were salivating when this attack vector was discovered.
Memcached Reflection Attacks Break DDoS Records
It’s hard to express just how quickly attackers moved to capitalize on memcached. In the security community, rumors of the vulnerability started circulating over the weekend, and by Monday, Feb. 27, several organizations had already seen attacks. On Feb. 28, Akamai protected a customer from what is possibly the largest DDoS attack ever recorded — a punishing 1.3 Tbps firehose of traffic from memcached servers across the globe.
Today, the issues caused by memcached have been largely mitigated, but not completely. The initial count of 50,000 vulnerable servers was quickly whittled down to 10,000 in the initial days of the attacks and has since dropped below a few thousand servers. Multiple internet service providers (ISPs) and cloud services have taken steps to prevent their networks from being the source of memcached attack traffic. Linux defaults have been updated to prevent the preconfigured exposure of the service to the internet, and administrators across the globe have updated their configurations to block port 11211 and require authentication.
A Wake-Up Call for Developers
Memcached is simply the latest example of a valid service that was developed and deployed with little or no thought to security. It was quite a wake-up call when Mirai burst onto the scene with a 623-Gbps attack against security writer Brian Krebs. This was the first big leap in attack traffic in several years and highlighted the fact that attackers were looking for new resources to fuel DDoS attacks. Memcached is proof that the bad guys haven’t stopped their search for new pools of vulnerable protocols and services to exploit.
It took compounded vulnerabilities and a change to default configurations to turn the memcached service into the threat it became earlier this year. These attacks serve as a reminder that security has to be a primary consideration from the development stage of a piece of software through to its end of life. The cold, hard truth is that eventually, every service will be exposed, even if it was built for entirely legitimate purposes.
Download the 2018 Gartner Magic Quadrant for Application Security Testing