Potential Security Leak

What Happened?

On March 29th 2013 we discovered that one of the MusicBrainz database dumps contained password hashes for a large portion of MusicBrainz accounts. While we don’t believe that these password hashes are either useful or widely distributed, we are requiring all users change their passwords.

What Data Was Leaked?

bcrypt password hashes, with a cost parameter of 8, for all accounts as of March 25th 2013.

Why Did This Happen?

We’ve recently began work on a long standing ticket against MusicBrainz server – MBS-357, “don’t store passwords in clear text”. We’re going to be moving away from clear text passwords, and we’ve decided to use one of the current industry standards for hashing passwords – bcrypt. Using bcrypt means that MusicBrainz will store only the hashes of passwords, which in laymans terms is a “fingerprint” of the password. Hashing means that we never store the actual password, but only the hash. There are many hashing functions available, and bcrypt is designed to be an expensive hash to compute with an adjustable “cost” – this makes it very hard to find out what the original password was via brute force attacks.

While this does mean that it’s hard to extract passwords from the hashes, the initial round of hashing passwords to move away from clear text is time consuming. As such, we built a small program that would gradually hash passwords over the course of a few days in order to make the switch from clear text passwords to secure password hashes done with as little downtime as possible.

This script hashed the password into the bcrypt_password column for all editors, and would also be notified when users changed their password in order to update the hash. Unfortunately, our database dump scripts sanitize this data by excluding data after-the-fact, rather than declaring what data to dump before running the script. As such, it dumped the entire editor table with the new column, as we forgot to add a rule to exclude this column.

Our Response

The database dumps that contain this data were promptly deleted, and have been replaced with correctly sanitized database dumps. Unfortunately logs from this server do show that this database dump was downloaded, and as we have no real indication of where this data now is, we’re treating this seriously. We have adjusted our database dumping scripts to be very specific about exactly which data they should export, so that in the future we will not leak private data by making the same mistake again.

We’re extremely sorry about this mistake, and while we don’t believe this data should allow attackers to retrieve user passwords, we can’t be 100% certain. As such, we require that all users change their password as soon as possible.

52 thoughts on “Potential Security Leak

  1. jesus2099

    Is it possible to know whether my hashed password was in there or not ?
    Could tell (privately maybe) each user in this « large portion of MusicBrainz accounts »  ?

  2. jesus2099

    Never mind my question it’s « all users created before 2013.3.25. »
    But maybe people would like to understand « with a cost parameter of 8 »

  3. jesus2099

    Sorry again, just for the record it’s very high level stuff, and it’s explained in today’s IRC log if you want to know. 🙂
    ALL ACCOUNTS BEFORE CREATED BEFORE 2013.3.25. (almost all of you) got their password published in an encrypted form (not plain text) but still it’s advised to change it right away, especially if you are using same passwords on many websites then change it everywhere as soon as possible… that kind of stuff…

  4. argon18

    I noticed that I didn’t get an email about this issue. Are there plans to email members and ask them to change their passwords? If not all users, certainly email auto-editors since a compromised auto-editor account could cause us much grief.

    It’s unfortunate that this happened, but I appreciate the professionalism of how the disclosure was handled. This is exactly the type of response that I prefer when this kind of problem arises.

  5. reosarevok

    People will get a prompt to change their password when they try to log in, but it might make sense to email indeed…

  6. InvisibleMan78

    Thank you for the information about this “bad thing”.

    One question:
    If I change my password here on MB, do I have to change the (same) password on AcoustID (to submit Fingerprints) too?

  7. Ben Ockmore

    When you change your password on here, you’ll need to use the updated password when you log into AcoustID, as it queries MB with the login details you use there.

    You’ll also need to update the Account Information in the General options of MusicBrainz Picard if you use that, and possibly other taggers too.

  8. Matt

    Stuff like this happens. Thanks for making this public, it’s the professional thing to do – even if the probability of abuse is rather low compared to leaks at other sites.

  9. Lukas Lalinsky

    InvisibleMan78:
    No, AcoustID doesn’t need to know anything about your MB password. The website does ask you to use the password to log in, but beyond that it’s not saved nor used anywhere. Your identification on the AcoustID site is your user API key.

  10. rimi ghose

    Im a musician. My content habitually gets used a
    ltered by those that ate not myself. I created an and cant remember my user name or password

  11. Gord

    Sigh. I don’t know what my old password is, and I need it to be able to change to the new one. Oh well, not that I’ve missed it…

  12. Brody Hess

    It needs to be said that I extremely appreciate the email I received this email. It was forthright, professional, and sincere in it’s intent to look out for the MB user-base. Thank you for doing what you do. Your service has helped me immensely, and I definitely don’t plan on discontinuing use.

  13. Robert Muil

    Another vote of confidence here. Appreciate your prompt and open disclosure of the mistake.

  14. Moi

    You were using Hash without any Salt? That would help anyone with a bcrypt dictionary to perform much faster attacks.

  15. Steve

    Wow, you were using a proper key-derivation function instead of just salted sha1? As an infosec professional, I think this is the best I’ve ever felt about my password hash being leaked; thanks for doing it right!

  16. Keith

    Just to add a fuel to this fire. The same username/password combination that I used on MusicBrainz was used to breach an account I had for trading bitcoins.

    The combination was previously leaked from another site before the MusicBrainz leak, but as far as I can tell no other users from that event have been breached.

    I would recommend all users of MusicBrainz, in addition to changing the PW here to change their passwords at any location where this combination was used.

    Check out Steve Gibson’s site to choose a good password: http://www.grc.com/haystack.htm

  17. Edson

    Dears, Morning!

    My old password is not validade? I think don’t forget but, is possible. Your can help-me? I don’t see options to receive my old password for my e-mail.

    I need create new account? Please, thank you very much?

  18. tee el cee

    Does sharing the “cost parameter” in the blog article AID a brute force attacker?

  19. ocharles

    No, the cost parameter would have been in the database dump anyway (it’s part of the hash).

  20. ocharles

    Not sure I understand your question, but we were using a random salt for every password hash.

  21. IH

    I am not sure I understand. Why can’t you tell who downloaded the dump? Isn’t this a “private” server? I would hope that only an authorized person could download the file.

  22. Silver Knight

    Am a little late to the party here, but I also would like to add my vote of confidence regarding the handling of this situation and my appreciation for your honesty and forthrightness about what was involved. This is how these sitations SHOULD be handled. Professionally, quickly, and honestly (not necessarily in any particular order there). 😉

    Thanks much for a fantastic service, and thank you for bringing your password system properly up to date with modern security standards/levels. Muchly appreciated!

  23. Yonatan

    Submitting a password longer than 64 characters in the reset password form results in an ugly error message, including a lovely trace.
    Better make sure you sanitize all user strings.

  24. Joel

    I agree with Steve above. I’m gladdened by the fact that you were using bcrypt. I’m more used to announcements of leaks from websites using something stupid like unsalted sha-1. It is a little disturbing though that passwords were stored in plaintext at any time.

  25. Jon

    Please tell me the password hashes were salted. If they were nit lease start salting them in the future.

  26. reosarevok

    Jon: “we were using a random salt for every password hash” in the comments above.

  27. Jeremy

    Since it has been stated that there is random salt for each user, the next question is: were the salt values also in the dump?

  28. Charlie

    I don’t even think I actually have an account with this site, but I received an email from them with my username saying their site had been hacked.

    I’ve never even heard of this page.

  29. reosarevok

    Charlie: some accounts are 10 years old, it’s perfectly possible for you to have forgotten ever hearing about us by now 🙂

  30. strat666

    Security breaches can happen, no matter how hard we try to minimise the possibility. When they do occur, this is how to handle them.

    Well done, MusicBrainz.

  31. rctgamer3

    “People will get a prompt to change their password when they try to log in”

    This is pure bullshit.
    I tried logging in just now and it just worked. No message, no change-password error; NOTHING.
    I didn’t even get the email people are talking about in the comments above. I’ve changed my password anyway, but still. You should send out another email telling users about the hack, because aside from some automatic “Note added” emails in/over the last couple of months i never received the news about the hack, and only found out because the newspost was posted to HN (Hacker News) ~4 hours ago.

  32. Mike

    The fact that you do not consider this a major security fault shows how little you know about computer security. Programs exist which can test hundreds of thousands of password hashes a second on GPUs and these are easily cracked, particularly since it seems you were not salting your password hashes. The fact that you were literally just giving hackers the ability to crack thousands of accounts offline is ridiculous and demonstrates massive technological incompetence.

  33. gaspode

    😉 dont worry, it happens to every one. I m now used to these messages.

    I still have several questions :

    – Why did the website asked me my previous password if it has leaked ? shouldn’t you assume that hackers have it ?
    To be honest, I felt really hesitant to enter it, I thought it was a scam.

    – you discovered it like 2 weeks ago, I just received the email right now.
    That’s quite some delay.

    – shouldn’t you also warn people to change passwords on other website where the same password & username was used ?
    Had I been a hacker, after sweating sufficiently at the website hacking & brute forcing successfully a hash, I wouldn’t let it go & I’d rather make as much use of password as I can, hoping the same password got used for other websites 😉

  34. k

    This is the most positive report of security breach I’ve read in aeons!
    Seeing all those embarrassing breaches every day, it’s really encouraging to see somebody finally doing things right.
    Out of curiosity though, I wonder how many login attempts per second you can afford with cost of 8 and what fraction of this is actual expected rate.

  35. Ubot

    @Mike, read the post again and don’t make unfounded speculation about lack of salts.

    > What Data Was Leaked?
    > bcrypt password hashes, with a cost parameter of 8, for all accounts as of March 25th 2013.

    “bcrypt” implies there is a salt. The work factor part is also relevant, because it means the dictionary attacks you mention are impractical for all but the lowest-entropy passwords.

  36. Richard

    I got your email warning me my details had been leaked, but I have never even heard of you or your website, so how did I end up with an account here?

  37. reosarevok

    Richard: Either you created it a long time ago and just forgot, or someone else has access to your email and registered. I’m guessing it’s the former!

  38. Mika

    While I certainly appreciate the heads up via email, took you guys kind of long. While this blog post went up in reasonable amount of time, what took you guys nearly two weeks to get the emails out?

  39. Fred

    That’s pretty horrible that you used to store plain text passwords! I can’t imagine the consequences of such a leak if bcrypt hadn’t take place yet. Anyway, thankfully enough you moved to bcrypt and not md5 or equivalent useless hash algorithm.

  40. cedmars

    My email account has been hacked just after your mail and it was the same password. 😦
    Problem solved… I hope

  41. Thomas

    Hi! I can’t change my password. When trying to login, the change password screen appears, but it does not save my new password. When I try to login with the new credentials, I get a wrong password message. With the old password I am taken to the change passwort page, again.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s