Schema change release, 2016-05-23 (with upgrade instructions)

Starting with this release, PostgreSQL 9.5 is now our minimum supported version. In order to import any future data sets, you will need to upgrade your installation to version 9.5.

Due to unforeseen problems with the Live Data Feed (AKA replication), users with slave databases will be required to first import a fresh data dump into their new 9.5 installation. We apologize that this is the case, but even had this stream not been broken, doing a clean import is faster and easier than doing the migration. For details on what happened during this rather lengthy schema change release, stay tuned for a post mortem blog post that covers the details.

If you have a non-replicated standalone database, you can use pg_upgrade and run ./upgrade.sh directly, but for simplicity we strongly recommend importing the latest data dump. Thus, we will only provide instructions for a clean import:

  1. Make sure you have PostgreSQL 9.5 installed, and your database settings in lib/DBDefs.pm are updated to point to the 9.5 installation if you currently have an older version of postgres running. If you already have postgres 9.5 and want to replace the existing database there, you’ll need to drop it first (using dropdb or from within psql). Be careful that you’re not dropping any important data if this is a standalone database that you’ve made changes to.
  2. Take down the web server running MusicBrainz, if you’re running a web server.
  3. Turn off cron jobs if you are automatically updating the database via cron jobs.
  4. Switch to the new code with git fetch origin followed by git checkout v-2016-05-23-schema-change-v2
  5. Run cpanm --installdeps --notest . to ensure your perl-based dependencies are up to date. Note the dot at the end.
  6. Set DB_SCHEMA_SEQUENCE to 23 in lib/DBDefs.pm
  7. Download the latest data dumps. If you don’t need historical edit data, excluding the edit dump will speed up your import significantly.
  8. Initialize a new database from the data dumps downloaded in step 7. Detailed instructions for doing this are located in INSTALL.md in the musicbrainz-server repository; if your data dumps are in /tmp, the command should simply be something like ./admin/InitDb.pl --createdb --import /tmp/mbdump*.tar.bz2.
  9. After the import has finished, turn cron jobs back on, if applicable.
  10. Restart the MusicBrainz web server, as well as memcached, if applicable.

We would like to thank bitmap, Gentlecat, zas, chirlu, reosarevok, gcilou for contributing directly to the release and we’d also like to thank all of the people who helped test, debug or otherwise offer support in this quite difficult release. Thank you!

And finally, here’s the list of changes you can expect in the upgrade:

Bug

  • [MBS-6406] – Admins can’t change email addresses
  • [MBS-8288] – Missing indexes for inverse lookup on *_gid_redirect tables
  • [MBS-8669] – Primary key for place table missing on old slaves
  • [MBS-8906] – Release pages ISE if CB doesn’t return JSON from its API for whatever reason
  • [MBS-8928] – If you submit the release editor without being logged in, it displays “[object Object]” as an error mesage
  • [MBS-8943] – Some pages do not respect DB_READ_ONLY setting

Improvement

  • [MBS-1873] – Fix vote tallies for edits
  • [MBS-3887] – Duplicate artist and label names not being checked against alias
  • [MBS-8287] – Log deleted entities that were in a subscribed collection
  • [MBS-8433] – Work attributes don’t have a uuid
  • [MBS-8716] – Store the edit data in a JSONB column
  • [MBS-8717] – Move the edit data to a separate table
  • [MBS-8838] – Add gids to all *_type* tables
  • [MBS-8873] – Convert and unify artist credit editors to React
  • [MBS-8909] – Add logos to IMDb and VGMdb links in the sidebar
  • [MBS-8939] – Update the Instagram logo used in the sidebar
  • [MBS-8940] – Let banner message editors dismiss the banner only temporarily

Task

  • [MBS-8656] – Bring edit table indexes back into sync
  • [MBS-8719] – Stop materializing of edit and vote counts
  • [MBS-8720] – Add a materialized view of edit note recipients
  • [MBS-8727] – Prevent duplicate votes
  • [MBS-8800] – Create the earthdistance extension and add a geodetic index for place coordinates
  • [MBS-8804] – Add BRIN indexes for timestamp columns
  • [MBS-8897] – add new entity icons
  • [MBS-8938] – Schema changes to support alternative tracklists

11 thoughts on “Schema change release, 2016-05-23 (with upgrade instructions)”

  1. We appreciate all your hard work and congratulate the team.
    Is there an update to the search server also available at this time? After upgrading the database to the new schema, the index building code crashes in building the recording index.

  2. The search server is supposed to still work, actually (though its output won’t contain the new MBIDs for types etc.). It does on the MeB systems.

    There was only one issue (though not affecting the recording index, I think?) that is fixed in the newest commits on the master branch.

  3. After a clean build of the search server index building proceeds to place index where it hangs. I have opened a ticket (SEARCH-429).

  4. I managed to upgrade the current VM by trial and error. See


    # With the following commands I upgraded the current VM (2015-08-06) so it would import the current db dumps (Schema 23).
    # Please follow the script step by step. It is not an automated script but rather a sequence of commands and instructions.
    # You certainly have to adjust the postgres install section to your distro, the git tag to check out, and the import section at the end.
    # https://github.com/metabrainz/musicbrainz-server/blob/master/INSTALL.md
    cd /home/musicbrainz/musicbrainz-server
    ## update postgresql
    sudo apt-get remove –purge postgresql-9.3 postgresql-client-9.3 postgresql-contrib-9.3 postgresql-musicbrainz-collate postgresql-musicbrainz-unaccent
    # https://www.postgresql.org/download/linux/ubuntu/
    echo "deb http://apt.postgresql.org/pub/repos/apt/ trusty-pgdg main" | sudo tee -a /etc/apt/sources.list.d/pgdg.list
    wget –quiet -O – https://www.postgresql.org/media/keys/ACCC4CF8.asc | \
    sudo apt-key add –
    sudo apt-get update
    sudo apt-get install postgresql-9.5 postgresql-server-dev-9.5 postgresql-contrib-9.5
    ## update mb-server
    # https://blog.musicbrainz.org/2016/05/25/schema-change-release-2016-05-23-with-upgrade-instructions/
    sudo git fetch origin
    tag=v-2016-08-01
    sudo git checkout ${tag}
    ## install postgresql extensions
    cd ..
    sudo git clone https://github.com/metabrainz/postgresql-musicbrainz-collate.git
    cd postgresql-musicbrainz-collate
    sudo make install
    cd ..
    sudo git clone https://github.com/metabrainz/postgresql-musicbrainz-unaccent.git
    cd postgresql-musicbrainz-unaccent
    sudo make install
    cd musicbrainz-server
    ## set db schema sequence to 23, replication type to RT_STANDALONE
    sudo vim lib/DBDefs.pm
    ## set postgresql permission
    # insert "local all all trust"
    sudo vim /etc/postgresql/9.5/main/pg_hba.conf
    ## update db
    sudo dropdb -U musicbrainz musicbrainz
    base=/media/sf_Downloads
    sudo ./admin/InitDb.pl –createdb –import ${base}/mbdump.tar.bz2 ${base}/mbdump-derived.tar.bz2 –echo

    for instructions

  5. I think your link got eaten by wordpress? I’m just seeing two linebreaks between “See” and “for”…

  6. You are right. gist.github.com/notEvil/be975e41693b88e0e5587fbf1dbe2ce3
    alternatively look for “mb.vm.upgrade.sh” at gist (gist.github.com)

  7. Hi, thanks for providing the steps. I’ve hit a snag at the last step. When attempting to dropdb I get the following message:

    Peer authentication failed for user “musicbrainz”

  8. google tells me that it’s an postgres auth issue. So did you insert “local all all trust” without ” (full access to all users) at the very top? And did you restart postgresql or the entire VM before dropping the db?

  9. Hey @notEvil thanks for the excellent guide!

    @Steve I ran into the same snag. I’m not that knowledgeable but my guess is it has sth to do with how the Postgresql remove+install of 9.5 is done. Somehow doing “./admin/InitDb.pl –createdb –clean” (when in /home/musicbrainz/musicbrainz-server) and then continuing with the dropdb worked for me. Cheers.

Leave a comment

This site uses Akismet to reduce spam. Learn how your comment data is processed.