November 27, 2017 By David Bryan
Dustin Heywood
4 min read

This is the second article in a series talking about our password cracking tool called the Cracken. Be sure to read part one for the full story.

The IBM X-Force Red team had a serious need for a powerful, dedicated password cracking system — something our whole global team could easily use. In my 10-year career as a penetration tester, I’ve learned that this is 100 percent necessary, since most of our projects are time-limited to a week or two. That means a pen tester might only have three to four days to crack a password before the project is over. An actual attacker has much more time.

Cracking From the Cloud

We started our password cracking adventures on a cloud computer. We needed to get something going that didn’t require months of lead time and would enable us to crack passwords for our projects. Our first machine only had four graphics processing units (GPUs). It worked, but it was pretty slow. Then along came an eight-GPU system in the cloud. This was better, but still not super awesome compared to some of the hash rates coming out of the regular video cards.

When we started, our cloud password cracking habit only cost, at most, a few hundred dollars a month. By the time we had enough data to justify our rig, we were spending $2,500 to $3,000 per month on our limited 200- to 300-hour cracking runs.

Lets do some quick math on these costs: A new, fully loaded system that would do what we needed retailed for about $22,000 to $24,000. If we kept paying $2,500 a month, that would end up costing us $30,000 a year, so we only ran jobs about 200 to 300 hours per month to keep costs down. To run jobs 24/7, we would be paying $105,000 a year for one system. We really needed two of them, so building our own system very quickly became a priority. That would save us $500,000 to $1 million over a five-year period.

The hardware we used consisted of the components listed below.

  • Lenovo M3650 Server:
    • 2x Intel Xeon CPU E5-2650 v4 — 12 cores per processor, two threads per core = 48 threads;
    • 128 GB of random-access memory (RAM) — 8 GB of RAM/GPU card is recommended; and
    • 1 TB of disk.
  • Cubix Rack Expander 5U with 8x PCI x16 Gen3 slots:
    • 8x Nvidia GTX 1080 Cards

Building Our Own Password Cracking Rig

We ordered our password cracking hardware in February 2017. Had the GTX Ti cards been released by the time we finalized our order, we would have gone with those cards, since they deliver a significant bump in hash rates over the plain GTX 1080. Our next rig will most likely use them.

We can easily justify the cost of this cracking rig just based on monthly costs. However, it’s worth breaking down hash cracking rates.

  • Cloud provider
    • 8x Nvidia Tesla K80s
    • SHA1 and Hashcat: 14.8 GH/s
    • Bcrypt and Hashcat: 15.2 kH/s
    • NTLM and Hashcat: 66.493 GH/s
  • X-Force Red’s Cracken
    • 8x Nvidia GTX 1080s Founders Edition
    • SHA1 and Hashcat: 66.1 GH/s
    • Bcrypt and Hashcat: 118.6 kH/s
    • NTLM and Hashcat: 334 GH/s

Building our own hardware gives us so much more horsepower. Keep in mind that these numbers are only from one node. When we built the Cracken, we built two nodes, meaning we have a total of 16 GPUs that we can toss crack jobs at. Based on those numbers, the GTX 1080 cards are significantly faster at cracking hashes at a fraction of the cost.

A few years ago, on another cracking cluster, I replaced the thermal grease on a batch of ATI 390Xs, and that actually helped reduce the temperature of the cards. We had been hitting the high end of the thermal shutdown quite often during our runs. One card failure would take out the entire PCI bus and server. To get the system back online, it required a physical power down of the system. After installing new thermal grease on the ATI cards, it dropped the cards’ high-end temperatures by at least 4 degrees Celsius, keeping them from hitting their thermal shutdown point.

Before installing our Nvidia GTX 1080 FE cards, I had the bright idea of swapping out thermal grease on one of the cards. I did a tear down, cleaned off the factory thermal paste and put on my own super high-end paste. I’m a rebel, so I voided the warranty on one of our 16 cards. Did this help? Not enough to justify the time it takes. It seems that the GTX 1080 cards have good thermal grease, and the fan on the Founders Edition does the job to move heat away from the cards out of the box. It also doesn’t hurt that the external cases have three very large fans to move air.

With the Nvidia GTX 1080 cards, thermal shutdown hasn’t been an issue. But with these cards, it really is much more about having proper airflow and strong, reliable power supplies. We have them in an external GPU case so we can prevent these cards from crashing and taking the whole cluster with it.

Key Takeaways

To summarize the key lessons I learned during this experience:

  • Make sure the room you put your cracking rig in has proper cooling.
  • Did you know that most building HVAC systems shut down overnight and during the weekends? You, too, could find out the hard way when you burn up your cluster of GPUs.
  • Properly size all your power requirements for the cards. Do not cheap out on this.
  • Make sure you have enough memory for each GPU; 8 GB/GPU is recommended for Hashcat.
  • Make sure there is proper cooling and air flow for all the cards.
  • Do not use cheap power cables or PCI cables. You will cause a fire or at least burn up some cables. It is best use to a backplane to handle the amps and bus frequency.

In our third and final installment in this series, we will talk about some tips and tricks to get the most out of Hashcat and some fun things we have done with it.

In the meantime, follow us on Twitter: @XForceRed, @Evil_Mog, @_videoman_ and listen to our recent podcast on ‘Cracken’ passwords.

More from X-Force

Stealthy WailingCrab Malware misuses MQTT Messaging Protocol

14 min read - This article was made possible thanks to the hard work of writer Charlotte Hammond and contributions from Ole Villadsen and Kat Metrick. IBM X-Force researchers have been tracking developments to the WailingCrab malware family, in particular, those relating to its C2 communication mechanisms, which include misusing the Internet-of-Things (IoT) messaging protocol MQTT. WailingCrab, also known as WikiLoader, is a sophisticated, multi-component malware delivered almost exclusively by an initial access broker that X-Force tracks as Hive0133, which overlaps with TA544. WailingCrab…

Empowering cybersecurity leadership: Strategies for effective Board engagement

4 min read - With the increased regulation surrounding cyberattacks, more and more executives are seeing these attacks for what they are - serious threats to business operations, profitability and business survivability. But what about the Board of Directors? Are they getting all the information they need? Are they aware of your organization’s cybersecurity initiatives? Do they understand why those initiatives matter? Maybe not. According to Harvard Business Review, only 47% of board members regularly engage with their CISO. There appears to be a…

GootBot – Gootloader’s new approach to post-exploitation

8 min read - IBM X-Force discovered a new variant of Gootloader — the "GootBot" implant — which facilitates stealthy lateral movement and makes detection and blocking of Gootloader campaigns more difficult within enterprise environments. X-Force observed these campaigns leveraging SEO poisoning, wagering on unsuspecting victims' search activity, which we analyze further in the blog. The Gootloader group’s introduction of their own custom bot into the late stages of their attack chain is an attempt to avoid detections when using off-the-shelf tools for C2…

Hive0051’s large scale malicious operations enabled by synchronized multi-channel DNS fluxing

12 min read - For the last year and a half, IBM X-Force has actively monitored the evolution of Hive0051’s malware capabilities. This Russian threat actor has accelerated its development efforts to support expanding operations since the onset of the Ukraine conflict. Recent analysis identified three key changes to capabilities: an improved multi-channel approach to DNS fluxing, obfuscated multi-stage scripts, and the use of fileless PowerShell variants of the Gamma malware. As of October 2023, IBM X-Force has also observed a significant increase in…

Topic updates

Get email updates and stay ahead of the latest threats to the security landscape, thought leadership and research.
Subscribe today