Schema change release, 2015-05-18 (including upgrade instructions)

Our previously mentioned schema change release is finished! Below will be upgrade instructions, including configuration updates for replication access tokens.

This release does not include UI for several of the schema change patches, which will (hopefully) happen for next release on June 1. The incomplete patches are MBS-7489 (credits for artists in relationships), MBS-4145 (tag upvote/downvote), and MBS-8004 (collections for additional entity types). These patches have had their schema change components finished, but the UI was incomplete or needed more work.

Schema Change Upgrade Instructions

These are largely as previous upgrade instructions, using the tag v-2015-05-18-schema-change. The primary difference is the inclusion of configuring an access token for replication.

  1. Make sure your REPLICATION_TYPE setting is RT_SLAVE and your DB_SCHEMA_SEQUENCE is set to 21 in lib/DBDefs.pm. If you’re running a standalone server, you can run the upgrade, but it may be easier to just import a new data dump!
  2. Ensure you’ve replicated up to the most recent replication packet available with the old schema. (if you’re not sure, run ./admin/replication/LoadReplicationChanges and see what it tells you; if you’re ready to update, it should say “Mismatched schema sequence, 21 (database) vs 22 (replication packet)”).
  3. Take down the web server running MusicBrainz, if you’re running a web server.
  4. Turn off cron jobs if you are automatically updating the database via cron jobs.
  5. Switch to the new code with git fetch origin followed by git checkout v-2015-05-18-schema-change
  6. Run ./upgrade.sh (or carton exec -Ilib -- ./upgrade.sh if you’re using carton, with very old setups).
  7. Run cpanm --installdeps --notest . to ensure your perl-based dependencies are up to date. This release adds a dependency on LWP::Protocol::https, for fetching replication packets from the new server; many systems may already have this installed, but it should be verified.
  8. Set DB_SCHEMA_SEQUENCE to 22 in lib/DBDefs.pm as instructed by the output of ./upgrade.sh
  9. Assuming you have been updating your server with replication, it will now be necessary to configure an access token:
    1. Go to https://metabrainz.org/supporters/account-type and choose your account type as applicable. If you’re an individual, non-commercial user of the data, choose “non-commercial”; if not, choose an applicable tier in the “commercial” section. If you’re not sure of the appropriate tier, make your best guess; it can be adjusted if necessary.
    2. Then, from https://metabrainz.org/profile, create an access token, which should be a 40-character random alphanumeric string provided by the site.
    3. Finally, add this token to lib/DBDefs.pm under the REPLICATION_ACCESS_TOKEN configuration option. The final configuration section should look something like sub REPLICATION_ACCESS_TOKEN { "ck3UpgwgOXhWC6SpFcd99rZOTjzfrei3gQlgZZ9z" }.
    4. Don’t reveal your access token! If you do, inadvertently, you can use the MetaBrainz site to generate a new token, invalidating the old one. (The one in the example above is one I created for myself and then invalidated — don’t get any ideas, it won’t work!)
  10. Turn cron jobs back on, if applicable.
  11. Restart the MusicBrainz web server, if applicable. It’s also recommended you restart memcached.

Finally, the list of bugs closed this release:

Bug

  • [MBS-4436] – Medium titles cannot be longer than 255 charaters

Improvement

  • [MBS-1347] – Implement aliases for release groups, releases and recordings
  • [MBS-7906] – maybe don’t show “”≠null diff. in edit pages
  • [MBS-8279] – Remove empty_artists etc. database functions

New Feature

  • [MBS-8302] – Add Live Data Feed access token support

Task

  • [MBS-8266] – Make medium titles VARCHAR NOT NULL
  • [MBS-8278] – Update DB_SCHEMA_SEQUENCE in DbDefs.pm.sample
  • [MBS-8283] – Remove DB constraint that disallows empty event names

Not included in this list but also relevant is MBS-8349, which while fixed for a previous release, in this release is also applied to old slave servers, which may help performance for some queries.

20 thoughts on “Schema change release, 2015-05-18 (including upgrade instructions)

  1. InvisibleMan78

    Point #7 doesn’t work in your latest VM:
    bash: cpanm: command not found

    What exactly do you mean with ‘ REPLICATION_ACCESS_TOKEN configuration option’?
    There is no such option or line in my DBDefs.pm (again in VM).
    Do I have to add a complete new line like:
    sub REPLICATION_ACCESS_TOKEN { “ck3UpgwgOXhWC6SpFcd99rZOTjzfrei3gQlgZZ9z” }
    Are you sure, that I have to enclose my token with ” ” not with single quotes ‘ ‘ ?

  2. theguardian

    I’m getting an error when I run ./upgrade.sh. It tries to drop a few indexes that don’t exist than ends with ERROR: relation “track” does not exist.

  3. Tim

    How long does “Vacucuming DB” take? Been going for over 20 minutes now…

  4. Tim Arner

    I was able to upgrade but it broke my web UI.
    Is there going to be a new VM image release ?

  5. Stefan

    Afraid that I am having the same issue @InvisibleMan78 had, could you please release a new version of the VM?

  6. ianmcorvidae Post author

    @InvisibleMan78: It will need to be added new, most likely. The particular type of quotes doesn’t matter; either will work. cpanm may not work if you’re using a very old setup, where you may instead need to use something like ‘carton install LWP::Protocol::https’, and similarly for any other modules it might complain about. Plain ‘carton install’ should also work, but we haven’t supported that for some time.

    @theguardian: Dropping indexes that don’t exist isn’t particularly an issue, since it intentionally drops things by several names that existed historically, so no individual server will have indexes under all the previous names. However, it sounds like your database is in a problematic state of some sort; the ‘track’ relation is rather important. If you can determine why you don’t have that table it may be fixable, but the easiest solution may be to import a new, fresh database once there’s a new data dump.

    @Tim Arner: If you have more details of exactly what you mean by breaking it, I may be able to help — what’s broken exactly? The old carton setup may need more modules installed, if you’re using that (as mentioned in response to InvisibleMan78 above), and/or you may need to update node.js dependencies with npm and recompile javascript and CSS files to use, or it could be something else entirely.

  7. ianmcorvidae Post author

    @Tim: sorry, missed your comment earlier. Vacuuming can take a while, depending on your hardware, but if you’re in a hurry, once it’s to that stage you can bring back everything else (though it’ll be at somewhat reduced capacity until vacuuming is finished).

  8. Andrew Hill (@AndyHill)

    I’m also seeing some of the things mentioned above, using your latest VM (just downloaded a couple days ago). cpanm isn’t installed – I was able to install it, though. As far as the broken website on port 5000, what I see is plain text, as if no style is applied, and no images are loaded. Firefox console reports 404 errors on :5000/static/build/

  9. ianmcorvidae Post author

    @AndyHill: yeah, the “current” VM is very out of date 😦 hopefully a new one can be put together soon. If you change things to cpanm, the way you run the server may also need to be amended to use a different install location for perl dependencies, but try it out first. The broken website styling sounds like you need to compile the styles (and presumably also scripts). To do so, you’ll need nodejs and its package manager npm; then, run ‘npm install’ to install dependencies, and run ‘./script/compile_resources.pl’ to build the static resources. This is documented somewhat in INSTALL.md, most specifically https://github.com/metabrainz/musicbrainz-server/blob/master/INSTALL.md#installing-nodejs-dependencies (there’s also an earlier section with install instructions for nodejs/npm on ubuntu systems)

  10. Tim

    Thanks @ianmcorvidae, I came right. I also had to add a new line for the sub REPLICATION_ACCESS_TOKEN parameter but all working now.

  11. reosarevok

    Have you followed the step no. 9 from this post? I suspect that is what you’re missing.

  12. Niels Lange (@komatzu)

    @reosarevok: thanks for the hint! I was using the MySQL setup with mbzdb. I guess this won’t work any longer 😦 The code is two years old… Looks like I have to setup a server from scratch…

  13. reosarevok

    Tommy: if you’re asking about what last.fm does with our data, that’s their decision not ours, so if they have stopped adding tracklists there’s nothing we can do about it from our side. Please bring it up with them!

  14. Herman Booysen

    Brilliant! This is my first release to install, and the program is wonderful. Thanks a million!

  15. Jesse

    My database is able to update itself via replication, but the web api is refusing connections. I wasn’t able to run the cpanm or the carton commands, both said command not found. I’m guessing my issue is related to that, any other suggestions? This is a version of your year old image currently on your site that has been upgraded to the current state.

  16. Brimur

    A new VM would be great. I have the current one upgraded to 21 but these steps and the issues I see in the comments seem like there is more going on than I might be able to deal with technically 😦

  17. brimur69

    A new VM would be great. This upgrade is a lot more technical than the last few. My VM is up to #21 but with all the issues people are having moving to #22 I think it might be too technical for a lot of people.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s