Tuesday, August 29, 2017

320 Million Hashes Exposed


Earlier this month (August 2017) Troy Hunt founder of the website Have I been pwned? [0] released over 319 million plaintext passwords [1] compiled from various non-hashed data breaches, in the form of SHA-1 hashes. Making this data public might allow future passwords to be cross-checked in a secure manner in the hopes of preventing password re-use, especially of those from compromised breaches which were in unhashed plaintext.

Our group (in collaboration with @m33x and @tychotithonus) made an attempt to crack/recover as many of the hashes as possible, both for research purposes and of course to satisfy our curiosity while using this opportunity as a challenge. Although each of the pwned password packs released at the time (3 in total at this writing) were labeled as 40-character ASCII-HEX SHA-1 hashes, we worked under the assumption that “No hash list larger than a few hundred thousand entries, contains only one kind of hash!” - and these lists were no exception.

Nested Hashes
Although the majority of the passwords recovered were plaintext, as expected, we also noticed there were a number of plaintexts themselves being hashes or some form of non-plaintext. This suggested that we were dealing with more than just SHA-1.

Out of the roughly 320 million hashes, we were able to recover all but 116 of the SHA-1 hashes, a roughly 99.9999% success rate. In addition, we attempted to take it a step further and resolve as many “nested” hashes (hashes within hashes) as possible to their ultimate plaintext forms. Through the use of MDXfind [2] we were able to identify over 15 different algorithms in use across the pwned-passwords-1.0.txt and the successive update-1 and update-2 packages following that. We also added support for SHA1SHA512x01 to Hashcat [3].

Taking a deeper dive into the found “plaintexts,” we realized there were hashes-within-hashes, hashes of seemingly garbage data, what appears to be “seeded” hashes, and more. Here is a list of the hash types we found:

There are other hashes we have not completely resolved yet - some of which may be seeded hashes. For example, we see:

sha1(md5(md5($salt).md5($pass)))
sha1(md5($salt).md5($pass)))
sha1(md5(md5($salt1).md5($pass)).$salt2)
sha1(md5($salt1).md5($pass).$salt2)

… and much more.

Personal Identifiable Information
We also saw unusual strings from incorrect import/export that was already present in the original leak. This links the hash to the owner of the password, which was clearly not intended by Troy. We found more than 2.5m email addresses and about 230k email:password combinations.
<firstname.lastname@tld><:.,;| /><password>
<truncated-firstname.lastname@tld><:.,;| /><password>
<@tld><:.,;| /><password>
<username><:.,;| /><password>
<firstname.lastname@tld><:.,;| /><some-hash>

Trash / Other Non-Passwords
Furthermore, there were obviously other strings that were not passwords, but rather fragments of files.  For example:

005a97e5323dac9a43c06bb5fe0a75973ee5e23f:<div><embed src="http://apps.rockyou.com/fxtext.swf?ID=31478642&nopanel=true&stage=true" quality="high" scale="noscale" width="405.37" height="116.475" wmode="transparent" name="rockyou" type="application/x-shockwave-flash" pluginspage="http://www.macrom


006bb7e8893618b02f979dd425e689b4ae64df10:honeyDo you realize who is in this image: http://thecoolpics.com/who.jpg . Just think for a moment and tell me o you realize who is in this image: http://thecoolpics.com/who.jpg . Just think for a moment and tell me soon ;))

Bad Line Parsing
We observed a number of passwords which appeared as they were truncated at length 40 but contained data following the linefeed terminator of the input lists.

n.doe@gmail.com:password:123456jane.doe@

We assumed this was either caused by a parsing error or some anomaly. To recover these strange processed plaintexts, some utilities were coded [4] to emulate the particular behavior of concatenating successive lines while restricting them to 40 characters.

john.doe@gmail.com:password:123456jane.d
ohn.doe@gmail.com:password:123456jane.do
hn.doe@gmail.com:password:123456jane.doe
n.doe@gmail.com:password:123456jane.doe@

Furthermore, to find the correct position where the initial parsing error occurred, we searched our dictionaries from the right to the left (see [4]) concatenating characters like this:

123456jane.doe@ho
o
ho
@ho
e@ho
...
123456jane.doe@ho


 An example of a bad/invalid email imported into the haveibeenpwned.com website

Hashcat’s Hexception
During hash processing, we also caught a glimpse into Troy’s methodology.  We believe that he processed some “cracked” passwords as well, suggested by the presence of $HEX[] plaintexts. This also revealed a bug in Hashcat’s $HEX[] encoding.

For example, consider the following hash:

0b20b6ad0b6c7fd3655e8734cb48c001567983eb:$HEX[244845585b623436653635373737393666373236625d]

Initially, when this was found with Hashcat, it appeared as:

0b20b6ad0b6c7fd3655e8734cb48c001567983eb:$HEX[b46e6577796f726b]

The hash could not be verified as the solution since:

sha1(binary[b46e6577796f726b]):[9def6b97e0095ac93331bc2780cc35a21d9cc752]

We discovered that Hashcat fails to correctly encode a literal string with $HEX[], if the literal string starts with $HEX[.  This means that if you take the output of Hashcat, say from hashcat.pot and try to re-crack it using the passwords in the hashcat.pot file - you will end up with “unsolvable” hashes.  As part of our work involves building dictionaries that we can reuse, we consider this a significant bug.

Some tools [5] were put together to properly re-encode the output from Hashcat, into the proper string:

$HEX[244845585b623436653635373737393666373236625d]

This then works properly as a reusable password with Hashcat and MDXfind, as it decodes into the literal string:

$HEX[b46e6577796f726b]

This issue has been resolved in a beta version of Hashcat [6].

We also uncovered a second bug in Hashcat, which was later corrected in a beta version. When using certain rules, we found that the solutions that Hashcat was offering also did not hash back to the correct value.  We ended up with hundreds of  “solutions” that really were not solutions at all. This is one of the reasons that we always try to double-check our work, to ensure that we have accurate hashes and plaintexts.

As a final check, we took just the SHA1x01 passwords we found and re-ran them through both Hashcat (Beta v3.6.0-351-gec874c1) and MDXfind. The results were quite illuminating. The test system used was a 4 core Intel Core i7-6700K system, with 4x GTX1080 cards and 64GB of memory. Using Hashcat, we found that loading more than about 250,000,000 hashes at a time was not possible [7] and as a result, the list was broken up into chunks of 225m hashes.


Program
Time to Complete
Hashes Found
Hashcat
55 minutes
318,932,512
MDXfind (all hashes)
9 minutes
318,933,582
MDXfind (225m chunks)
9 minutes
318,933,582

From our usage patterns, it is evident that both applications have their strengths and caveats. MDXfind shows its strength when the hashlist is too large to fit into GPU memory, when many algorithms need to be checked in parallel and when very long password strings need to be tested. Hashcat, on the other hand, shines when parallel compute is needed; such as running large rule sets and large keyspaces. Using the tools in tandem gives us the best of both worlds since we can feed the left list of each successive attack into either program to achieve optimal efficiency and coverage.

To further illustrate the problem with password reuse (and the importance of validation), the hashes were re-run using just the found password of Hashcat (Beta v3.6.0-351-gec874c1).  This resulted in 86,954 hashes not being recovered. These are primarily due to the $HEX encoding error that Hashcat makes.

Distributed Tasks
Once the hashlist was small enough where the size of the hashlist had negligible effects on search speed, distributed brute-force and mask attacks were conducted via Hashtopussy [8] a Hashcat wrapper.  Combining our hardware, we were able to achieve peak speeds of over 180GH/s on SHA-1, to put things into perspective that's roughly the speed of 25x GTX1080s. We were able to cover ?a length 1-8, ?l?d length 9-10 and ?b length 1-6 effortlessly.

Statistical Properties
In order to speed up the analysis of such a large volume of plaintexts, a custom tool was coded “Panal” (will be released at a later time) to quickly and accurately analyse our large dataset of over 320 million passwords. The longest password we found was 400 characters, while the shortest was only 3 characters long. About 0.06% of passwords were 50 characters or longer with 96.67% of passwords being 16 characters or less.  Roughly 87.3% of passwords fall into the character set of LowerNum 47.5%, LowerCase 24.75%, Num 8.15%, and MixedNum 6.89% respectively. In addition we saw UTF-8 encoded passwords along with passes containing control characters. See [9] for full Panal output.

Length.png 
Charset.png

Summary
Blocking common passwords during account creation has positive effects on the overall password security of a website [10]. While blacklisting 320m leaked passwords might sound like a good idea to further improve password security, it can have unforeseeable consequences on usability (i.e, the level of user frustration). Conventional blacklist approaches typically include the 10k most common passwords to limit online password guessing attack consequences. Until now, there has been no evidence to support which blacklist size provides an optimal balance. 

Post written in collaboration with @m33x and @tychotithonus

Resources
[0] 2017-08-03: Have I been pwned? by Troy Hunt
https://haveibeenpwned.com
[1] 2017-08-03: Introducing 306 Million Freely Downloadable Pwned Passwords 
https://www.troyhunt.com/introducing-306-million-freely-downloadable-pwned-passwords
[2] 2017-08-03: MDXfind v1.93
https://hashes.org/mdxfind.php
[3] 2017-08-28: Hashcat sha1(sha512($pass)) patch
https://gist.github.com/hops/9beda82cf3d21ab99a2971bf8d00dbb4 
[4] 2017-08-27: Some tools we developed to deal with incorrectly parsed strings
https://gist.github.com/m33x/3e0ab19a53384c036db29f996cb60733
[6] 2017-08-20: Hashcat Issue “hexify also all password of format $HEX[]”
https://github.com/hashcat/hashcat/issues/1340
[7] 2017-08-18: Hashcat Issue Potential Silent Cracking Failures at Certain Hash-Count
https://github.com/hashcat/hashcat/issues/1336
[8] 2017-08-03: Hashtopussy by s3inlc
https://github.com/s3inlc/hashtopussy
[9] 2017-8-29: Panal (Password Analysis) 320m HIBP Passwords
https://gist.github.com/m33x/03031e764ae5de179315270973c5871f
[10] 2017-08-03: Password Creation in the Presence of Blacklists
https://www.internetsociety.org/doc/password-creation-presence-blacklists



73 comments:

  1. Will you publish the cracked list? I've got some tools I'd like to run against it. Royce sent me an initial list, so if I could just get a diff, it could help with bandwidth.

    ReplyDelete
  2. Would be cool to publish this list! <3

    ReplyDelete
  3. In the statistical section could you PLEASE define your terms. While I/we can guess, it would be nice if at least you provided a link to the definitions character set terms used.

    PLUS, you used 2 different sets of abbreviations and VALUES. One set in the text and a similar but different set in the chart ie LowerCase 24.75 in text and Lcase 26% on the chart. I assume/guess they are referring to the same thing ...

    Thanks.

    ReplyDelete
  4. I'd like to see some review on how passwords are built using Automata theory. That is how are most passwords selected using types of chars. Such as, x% of passwords start with a capital letter, followed by 4 undercase, followed by a number and special character. This information could be used to better configure hashcat's brute force options to better crack most passwords.

    ReplyDelete
  5. Your blog is very useful for me.I really like you post.Thanks for sharing.

    ดูหนัง

    ReplyDelete
  6. This comment has been removed by a blog administrator.

    ReplyDelete
  7. You cracked this:

    006bb7e8893618b02f979dd425e689b4ae64df10:honeyDo you realize who is in this image: http://thecoolpics.com/who.jpg . Just think for a moment and tell me o you realize who is in this image: http://thecoolpics.com/who.jpg . Just think for a moment and tell me soon ;))

    out of a sha1 hash? That doesn't seem possible. There is too much entropy. Was this value already in a wordlist that you have?

    ReplyDelete
  8. I was very fortunate to be able to visit your page.Thanks

    ReplyDelete
  9. Thanks for explaining CynoSure Prime. Thank you so much. Just what I was looking for !!!!
    clipping path
    clipping path
    clipping path
    clipping path
    car photo editing

    ReplyDelete
  10. You have good taste and sense of decorating what I see also on the graphics of your blog - elegant, like the colors are not lacking but they are chosen and toned down, without variegation and dense. Best regards -
    remove background from image photoshop

    ReplyDelete
  11. Thank you so much dear for sharing such an amazing blog, this is really valuable information for me. Visit our page for the Best Website Designing Company in Delhi, India.
    Website Designing Company in Delhi

    ReplyDelete
  12. Nice blog.
    https://www.graphic-aid.com/clipping-path
    https://www.graphic-aid.com/
    https://www.graphic-aid.com/neck-joint-service
    https://www.graphic-aid.com/photo-color-correction

    ReplyDelete
  13. Having a website for any kind of business is a must in today’s competitive scenario but what about having an attractive design of the website that directly appeals the senses of the user. Get in touch with Jeewangarg – The Best Website Designing Company in Delhi to get the Appealing Website Designs.

    ReplyDelete
  14. Thanks for sharing!!!
    It’s clear that sex dolls are changing the way we view sex. Even the experts believe that by 2050, people will not only be doing sexual intercourse with robots, but they will fall in love with them. Thus, it’s a good idea to buy a high-quality programmable artificial intelligence sex doll from the reputed store and enjoy the real fuck in the comfort of your home.

    ReplyDelete
  15. Thanks for posting this blog, this is really very informative. Here we are presenting netcreativemind.com
    SEO/SEM Placement Consultant, Information security Placement Consultants
    SEO SEM Specialist
    Hire Information Security Specialists
    Data Analyst Recruitment Agency

    ReplyDelete
  16. For Best private ambulance call at Almas ambulance service in Delhi. We help patients to get best medical care on right time by our ambulance service.
    http://www.almasambulance.com
    International air ambulance
    Air ambulance Kolkata
    Air ambulance guwahati
    Air ambulance patna
    Air ambulance ranchi

    ReplyDelete
  17. Nice Blog, keep it up for more updates about this type of blog.Carolina Classics is the manufacturer of best F-100 Classic Ford Truck Parts| Buy Ford F100 truck parts online at Carolina Classics.Classic Ford Truck Parts
    F-100 Ford Truck Parts
    Classic Ford Truck Body Parts

    ReplyDelete
  18. Thanks For sharing this informative post. Here we are presenting Let's Life Breathe.
    5 Best Tips to Beat Pollen Allergies - BreatheFresh

    ReplyDelete
  19. Great post! Lovely to read it and looking forward for another post like this.
    thank you for sharing.
    Best Website designing company

    ReplyDelete
  20. That was such an awesome content to read and going through it.Thanks for such a good information.our product related for servo voltage stabilizer and transformer manufecturer company in Delhi Our company is also Step Down Transformer Manufecturer in Delhi.

    Servo Stabilizer Manufacturer in india
    what is Step Down Transformer
    Distribution Transformers Manufacturer in india
    Step Down Transformer

    ReplyDelete
  21. Corpac is India’s No.1 Professional PP Corrugated Sheets Manufacturers, suppliers and Currently being exported to countries like Qatar, New Zealand, Germany, Philippines, UK and Spain, this product is achieving new growth with over 10 years Experience. Corrugated plastic sheets are very versatile material that can be used in a variety of applications around the homes, companies and also for craft projects, call 9899362119." We manufacture Polypropylene Corrugated Sheets and fabricated products under one roof. We market our products not only in India but export to various countries across the world.

    ReplyDelete
  22. Nice blog, Mutual Fund Advisor qualified experts who can comprehend the reason and subtleties of a store for the fund.
    Mutual Fund Advisor

    ReplyDelete
  23. Thanks for sharing the useful information. This is a nice blog.

    ReplyDelete
  24. Rozana Spokesman: Latest Punjabi News on Politics, Sports, Business, Fashion, Hindi Films, Punjabi Films, National and International News reports. News from Northern India, Punjab, Delhi, Haryana, Chandigarh, Mumbai.
    No1 Punjabi Newspaper 2019
    Lates Punjabi Newspaper 2019
    Best Punjabi Newspaper 2019

    ReplyDelete
  25. Mari Bergabung Bersama Kami di hoyajackpot
    Website Kartu Online Dengan Tingkat Kemenangan Tertinggi
    Kartu Hoki Yang Kami Bagikan setiap saat!!
    Memiliki Bonus New Member 200% dan Spesial Promo Menyambut ramadhan untuk info Lebih Lengkap Hubungi
    HoyaJackpot
    Bandar Judi Terpecaya
    SobatPoker

    ReplyDelete
  26. Nice blog, Get you website designing responsive and creative and also digital marketing services at ogen info system Delhi, India.
    Web Development Company in Delhi

    ReplyDelete
  27. If you want to get the best high-quality real estate photo editing service at a cheap price, So click
    real estate photo editing service. We provide high-quality real estate photo editing service at a cheap price from clipping expert Asia.

    ReplyDelete
  28. This comment has been removed by the author.

    ReplyDelete
  29. This comment has been removed by the author.

    ReplyDelete
  30. This comment has been removed by the author.

    ReplyDelete
  31. This comment has been removed by the author.

    ReplyDelete
  32. Thanks for sharing the useful information.

    Cutsncamera corporate video production company in Delhi, NCR serving corporate film production services as a leading corporate film makers in Delhi, India.

    Corporate Video Production in Delhi NCR
    Documentary Film Production Delhi NCR
    Video Editing Services Delhi NCR
    Business Promotional Films Delhi NCR

    Need high quality corporate video production in Delhi NCR? We are the leading producer of corporate films, testimonials and explainer videos in Delhi NCR.

    Get in touch with us now:
    Mail 📧: info@cutsncamera.in
    Call📞: +91 7042 111 33 5
    Visit our official site: http://www.cutsncamera.com

    ReplyDelete
  33. A realistic mini sex doll brings the ultimate sexual pleasure, even more than a human partner. Their flexibility and submissiveness allow you to literally try any sexual stunt without necessarily cheating or risking deadly Sexually Transmitted Infections.

    ReplyDelete
  34. Hi Author!

    Your ability to express your thoughts and ideas is clearly impressive. Your blog helped me to get more good writing skills. Thank you so much for this helpful post. Keep sharing such wonderful posts. I will always follow your blog. Moreover Choose Leading web design companies in new jersey, USA to get best attractive Website for getting more leads.

    ReplyDelete
  35. Agar aap apne husband se pareshan hai or uss se door rhana cahati hai toh aap Talaq lene ki dua ko kijiye aap pati aapko khud ba khud talaq de dega

    ReplyDelete
  36. This is Very very nice article. Everyone should read. Thanks for sharing. Don't miss WORLD'S BEST BikeRacingGame

    ReplyDelete
  37. This is Very very nice article. Everyone should read. Thanks for sharing and I found it very helpful. Don't miss WORLD'S BEST CarGames

    ReplyDelete
  38. Thanks For Sharing With us !!!!

    Are you in problem from back pain, neck pain, shoulder pain or other one pain and the cause of this pain you are not enjoying your lifestyle then you have visited your nearest chiropractic in Singapore.

    ReplyDelete
  39. This is very nice article. Thanks for sharing and I found it very helpful. Read about world's best realistic sex doll .

    ReplyDelete
  40. Thanks for this post.
    Want to make your own website!!!
    Contact ACIL the best web design company in Delhi.
    WEB DESIGN COMPANY IN DELHI

    ReplyDelete
  41. Awesome Blog, Get the best Investment Advice and information about the Best Mutual Funds Company in india.
    Mutual Fund Advisor

    ReplyDelete
  42. Thanks for sharing nice information with us. i like your post and all you share with us is update and quite informative, i would like to bookmark the page so i can come here again to read you, as you have done a wonderful job. With the top Web Designing Agency in New Jersey USA name Globtier, you’ll get all of these components and services at best cost.

    top web development sites in New Jersey USA

    web design and development agency in New Jersey usa

    ReplyDelete
  43. thanks for amazing blog @http://www.airoshotblast.net/

    ReplyDelete
  44. very nice
    http://www.eskisehircilingir.biz/

    ReplyDelete
  45. The website for 10th class result 2019 has been launched students can go to this website 10classresult.com to check their 10th class result.

    ReplyDelete