This is the second article in a series talking about our password cracking tool called the Cracken. Be sure to read part one for the full story.

The IBM X-Force Red team had a serious need for a powerful, dedicated password cracking system — something our whole global team could easily use. In my 10-year career as a penetration tester, I’ve learned that this is 100 percent necessary, since most of our projects are time-limited to a week or two. That means a pen tester might only have three to four days to crack a password before the project is over. An actual attacker has much more time.

Cracking From the Cloud

We started our password cracking adventures on a cloud computer. We needed to get something going that didn’t require months of lead time and would enable us to crack passwords for our projects. Our first machine only had four graphics processing units (GPUs). It worked, but it was pretty slow. Then along came an eight-GPU system in the cloud. This was better, but still not super awesome compared to some of the hash rates coming out of the regular video cards.

When we started, our cloud password cracking habit only cost, at most, a few hundred dollars a month. By the time we had enough data to justify our rig, we were spending $2,500 to $3,000 per month on our limited 200- to 300-hour cracking runs.

Lets do some quick math on these costs: A new, fully loaded system that would do what we needed retailed for about $22,000 to $24,000. If we kept paying $2,500 a month, that would end up costing us $30,000 a year, so we only ran jobs about 200 to 300 hours per month to keep costs down. To run jobs 24/7, we would be paying $105,000 a year for one system. We really needed two of them, so building our own system very quickly became a priority. That would save us $500,000 to $1 million over a five-year period.

The hardware we used consisted of the components listed below.

• Lenovo M3650 Server:
  • 2x Intel Xeon CPU E5-2650 v4 — 12 cores per processor, two threads per core = 48 threads;
  • 128 GB of random-access memory (RAM) — 8 GB of RAM/GPU card is recommended; and
  • 1 TB of disk.

• Cubix Rack Expander 5U with 8x PCI x16 Gen3 slots:
  • 8x Nvidia GTX 1080 Cards

Building Our Own Password Cracking Rig

We ordered our password cracking hardware in February 2017. Had the GTX Ti cards been released by the time we finalized our order, we would have gone with those cards, since they deliver a significant bump in hash rates over the plain GTX 1080. Our next rig will most likely use them.

We can easily justify the cost of this cracking rig just based on monthly costs. However, it’s worth breaking down hash cracking rates.

• Cloud provider
  • 8x Nvidia Tesla K80s
  • SHA1 and Hashcat: 14.8 GH/s
  • Bcrypt and Hashcat: 15.2 kH/s
  • NTLM and Hashcat: 66.493 GH/s

• X-Force Red’s Cracken
  • 8x Nvidia GTX 1080s Founders Edition
  • SHA1 and Hashcat: 66.1 GH/s
  • Bcrypt and Hashcat: 118.6 kH/s
  • NTLM and Hashcat: 334 GH/s

Building our own hardware gives us so much more horsepower. Keep in mind that these numbers are only from one node. When we built the Cracken, we built two nodes, meaning we have a total of 16 GPUs that we can toss crack jobs at. Based on those numbers, the GTX 1080 cards are significantly faster at cracking hashes at a fraction of the cost.

A few years ago, on another cracking cluster, I replaced the thermal grease on a batch of ATI 390Xs, and that actually helped reduce the temperature of the cards. We had been hitting the high end of the thermal shutdown quite often during our runs. One card failure would take out the entire PCI bus and server. To get the system back online, it required a physical power down of the system. After installing new thermal grease on the ATI cards, it dropped the cards’ high-end temperatures by at least 4 degrees Celsius, keeping them from hitting their thermal shutdown point.

Before installing our Nvidia GTX 1080 FE cards, I had the bright idea of swapping out thermal grease on one of the cards. I did a tear down, cleaned off the factory thermal paste and put on my own super high-end paste. I’m a rebel, so I voided the warranty on one of our 16 cards. Did this help? Not enough to justify the time it takes. It seems that the GTX 1080 cards have good thermal grease, and the fan on the Founders Edition does the job to move heat away from the cards out of the box. It also doesn’t hurt that the external cases have three very large fans to move air.

With the Nvidia GTX 1080 cards, thermal shutdown hasn’t been an issue. But with these cards, it really is much more about having proper airflow and strong, reliable power supplies. We have them in an external GPU case so we can prevent these cards from crashing and taking the whole cluster with it.

Key Takeaways

To summarize the key lessons I learned during this experience:

• Make sure the room you put your cracking rig in has proper cooling.
• Did you know that most building HVAC systems shut down overnight and during the weekends? You, too, could find out the hard way when you burn up your cluster of GPUs.
• Properly size all your power requirements for the cards. Do not cheap out on this.
• Make sure you have enough memory for each GPU; 8 GB/GPU is recommended for Hashcat.
• Make sure there is proper cooling and air flow for all the cards.
• Do not use cheap power cables or PCI cables. You will cause a fire or at least burn up some cables. It is best use to a backplane to handle the amps and bus frequency.

In our third and final installment in this series, we will talk about some tips and tricks to get the most out of Hashcat and some fun things we have done with it.

In the meantime, follow us on Twitter: @XForceRed, @Evil_Mog, @_videoman_ and listen to our recent podcast on ‘Cracken’ passwords.