Schema change release, 2017-05-15 (including upgrade instructions)

We’re happy to announce the release of our May 2017 schema change today! Thanks to all who were patient during today’s downtime as we released everything to our production servers.

This is a fairly minor release as far as schema changes go, but please do report any issues that you come across.

Currently, the only visible change for editors is the ability to add multiple lyrics languages to works. We’ve also modified the schema to support dynamic attributes for entities other than works, but the UI for that won’t be complete for another release or two.

Now, on to the instructions.

Schema Change Upgrade Instructions

Note: Importing the latest data dump is always a valid alternative to running ./ on an existing database, if you’d prefer to also get new data in one go. Just follow the relevant instructions in The rest of the instructions here assume an in-place upgrade.

  1. Make sure DB_SCHEMA_SEQUENCE is set to 23 in lib/
  2. If you’re using the live data feed (your REPLICATION_TYPE is set to RT_SLAVE), ensure you’ve replicated up to the most recent replication packet available with the old schema. If you’re not sure, run ./admin/replication/LoadReplicationChanges and see what it tells you; if you’re ready to upgrade, it should say “This replication packet matches schema sequence #24, but the database is currently at #23.”
  3. Take down the web server running MusicBrainz, if you’re running a web server.
  4. Turn off cron jobs if you’re automatically updating the database via cron jobs.
  5. Switch to the new code with git fetch origin followed by git checkout v-2017-05-15-schema-change.
  6. Run cpanm --installdeps --notest . (note the dot at the end) to ensure your perl-based dependencies are up to date.
  7. Downgrade DBD::Pg by running cpanm TURNSTEP/DBD-Pg-3.5.3.tar.gz (version 3.6.0 breaks things currently).
  8. Run ./ (it may take a while to vacuum at the end).
  9. Set DB_SCHEMA_SEQUENCE to 24 in lib/ as instructed by the output of ./
  10. Turn cron jobs back on, if applicable.
  11. Restart the MusicBrainz web server, if applicable. It’s also recommended you restart redis. If you’re accessing your MusicBrainz server in a web browser, run npm install followed by ./script/

For those curious, here’s the list of resolved tickets (excluding MBS-8393):


New Feature

  • [MBS-9271] – Prevent usernames from being reused


  • [MBS-9273] – Fix the a_ins_edit_note function in older setups to not populate edit_note_recipient for own notes
  • [MBS-9274] – Fix the edit_note_idx_post_time_edit index in older setups to handle NULL post_time


  • [MBS-5452] – Support multiple lyric language values for works

37 thoughts on “Schema change release, 2017-05-15 (including upgrade instructions)

  1. InvisibleMan78

    Thanks for the above instructions.
    Are they valid for your VM too?

  2. mwiencek Post author

    @InvisibleMan78 AFAIK, there’s nothing special about the VM that would make these steps not work, assuming you know how to take down the web server there, turn off cron, etc. The commands listed should apply to any MBS install, VM or not. But, I also haven’t tested any of this inside the VM.

  3. rembo10

    I had to run `cpanm –installdeps –notest .` before `./`
    `npm install` before running `./script/` in order to get all the dependencies

  4. mwiencek Post author

    @rembo10, thanks, I’ve updated the steps in the blog post too in case it helps others. I’m curious what modules you had missing that needed.

  5. theguardian

    I’m getting errors when running plackup and the web server fails to start.

    raven@1.2.0 alert: no DSN provided, error reporting disabled
    raven@1.2.0 alert: uncaughtException:

    Looks like this bug was fixed with raven@2.0.0, but upgrading to that version causes other problems. (ref:

  6. InvisibleMan78

    How can we apply the latest fix from @mwiencek?
    Could please someone instruct us with the necessary commands?

  7. InvisibleMan78

    Where should we report errors like
    “undef error – malformed JSON string, neither tag, array, object, number, string or atom, at character offset 0 (before “(end of string)”) at lib/MusicBrainz/Server/ line 42. ”
    calling the local webserver in a browser?

  8. mwiencek Post author

    @InvisibleMan78 At please!

    I’ve noticed the same error in production, but only 4 times (so far) out of thousands and thousands of requests, so I’m not sure how to reproduce it yet. If you can consistently reproduce it, that would be great information to provide in the ticket.

  9. boehmi

    Were anybody actually able to upgrade the VM? After “cpanm –installdeps –notest .” I needed to install libjson-perl and libmoose-perl, but then I I get these errors when running “./”

    vagrant@musicbrainz:~/musicbrainz-server$ ./
    ~/musicbrainz-server/admin ~/musicbrainz-server
    Use of uninitialized value $ENV{“DB_ENV_POSTGRES_USER”} in string at lib/ line 51.
    Use of uninitialized value $ENV{“DB_ENV_POSTGRES_PASSWORD”} in string at lib/ line 51.
    Use of uninitialized value $ENV{“DB_PORT_5432_TCP_ADDR”} in string at lib/ line 51.
    Use of uninitialized value $ENV{“DB_PORT_5432_TCP_PORT”} in string at lib/ line 51.
    Use of uninitialized value $ENV{“DB_ENV_POSTGRES_USER”} in string at lib/ line 51.
    Use of uninitialized value $ENV{“DB_ENV_POSTGRES_PASSWORD”} in string at lib/ line 51.
    Use of uninitialized value $ENV{“DB_PORT_5432_TCP_ADDR”} in string at lib/ line 51.
    Use of uninitialized value $ENV{“DB_PORT_5432_TCP_PORT”} in string at lib/ line 51.
    Use of uninitialized value $ENV{“DB_ENV_POSTGRES_USER”} in string at lib/ line 51.
    Use of uninitialized value $ENV{“DB_ENV_POSTGRES_PASSWORD”} in string at lib/ line 51.
    Use of uninitialized value $ENV{“DB_PORT_5432_TCP_ADDR”} in string at lib/ line 51.
    Use of uninitialized value $ENV{“DB_PORT_5432_TCP_PORT”} in string at lib/ line 51.
    Attribute (port) does not pass the type constraint because: Validation failed for ‘Int’ with value “” at constructor MusicBrainz::Server::Database::new (defined at lib/MusicBrainz/Server/ line 53) line 52.
    MusicBrainz::Server::Database::new(‘MusicBrainz::Server::Database’, ‘HASH(0xe1b300)’) called at lib/MusicBrainz/Server/ line 19
    MusicBrainz::Server::DatabaseConnectionFactory::register_databases(‘MusicBrainz::Server::DatabaseConnectionFactory’, ‘READWRITE’, ‘HASH(0x1df8dc8)’, ‘TEST’, ‘HASH(0xe1af70)’, ‘READONLY’, ‘HASH(0xe1b120)’, ‘SYSTEM’, ‘HASH(0xe1b300)’) called at lib/ line 51
    require called at -e line 1
    main::BEGIN() called at lib/ line 0
    eval {…} called at lib/ line 0
    Compilation failed in require at -e line 1.
    BEGIN failed–compilation aborted at -e line 1.

  10. SB

    I had the same issue as boehmi updating the VM. Any tips or suggestions welcome.

  11. Alvaro

    Same here… Unable to update schema on VM with dockers.
    Same error as boehmi

  12. Jeff

    boehmi, try making these changes In the lib/ file,

    $ENV{“DB_PORT_5432_TCP_ADDR”} to “db”
    $ENV{“DB_PORT_5432_TCP_PORT”} to “5432”
    $ENV{DB_ENV_POSTGRES_USER} to “musicbrainz”
    $ENV{DB_ENV_POSTGRES_PASSWORD} to “musicbrainz”

  13. boehmi

    Wouldn’t it be better to define the variables somewhere? Anyway, I made the changes and ran into the next problem(s). First psql was not installed. Easily solved by installing postgresql-client-9.3. But then the update script complained
    psql: could not translate host name “db” to address: Name or service not known
    I changed “db” to the actual hostname “musicbrainz”, but it still couldn’t connect to the DB.

    I guess putting everything in docker containers makes everything more complicated. Did anybody actually upgrade the VM successfully?

  14. InvisibleMan78

    -1 for me, still waiting that one of the MB-Developers upgrade the official MB-VM to a working, downloadable, actual version.

  15. mey

    Running into the same error. Please provide an updated vm 🙂 would be great.

  16. boehmi

    Ok, I got a little step further. The docker containers create virtual network devices and I figured out that the database runs on After running ./ again I get
    Error: Schema sequence must be 22 when you run this script

  17. mwiencek Post author

    @boehmi Regarding the “Schema sequence must be 22” error, you don’t have the latest code, because otherwise it would say 23, not 22.

  18. Matthew Griffin

    Got to where the schema should update, however, I get the same error as above. Edited the lib/DBDefs file and canhed the host to and set port to 5432, this allows connection but still failing on username and password. I have tried using musicbrainz for both but that is rejected.

    Further assistance would be helpful to get this resolved.

  19. Scott B.

    I made it further, on my zillionth attempt — everything I know about docker, I learned attempting to perform this update. I’m not out of the woods yet, but I’m closer. Maybe everybody else’s experience is different, but I have found that I need to manually do:

    screen docker-compose -f /home/vagrant/musicbrainz/musicbrainz-docker/docker-compose.yml up

    in order to bring everything up, and ^C in this context to gracefully shut it all down. This becomes relevant at the end.

    Relative to the original upgrade instructions posted at the top of this thread:

    1. Make sure you have a snapshot of your VM so you can back off and try again if something goes wrong!
    2. Use “docker exec -it musicbrainzdocker_musicbrainz_1 bash” to establish an interactive session inside the musicbrainz docker image; this is where you should be doing steps 5-9.
    3. You can restart replication at this point, but it really thrashes your I/O so I suggest waiting until you have finished the remaining work and are ready to allow your system to churn overnight.
    4. The final step (11) was a nightmare for me. You need to run it inside the docker image too. Generally, each time (there were many) I got an error running script/, starting with complaint about “is-integer”, I did “npm install is-integer” (substitute the module inferred from the error message) and then retry the script. When that (finally) runs without failing, stop and restart everything (reattach to screen, ^C to shut it down, then restart with the docker-compose command above). When you see MORE error messages in the startup log, reconnect to the image (docker exec) and “npm install ” too, followed by running the script again, followed by restarting the environment. Eventually the script AND the restart will both run apparently clean. (There are a lot of messages I see now that I saw before the schema change, so I’m tuning them out.)
    5. However, the web interface is broken. Just trying to connect to the landing page (http://myhost:5000) fails with “502 Bad Gateway”.

    As I write this, I have a totally current database but no way to query it. 😉 I’m hoping somebody with more understanding of the web mechanics can fill in the missing pieces for us.

  20. Alvaro

    I am waiting to finish VACUMing the DB. Ig I succed on update the schema on VM-MB I will post here and upload by torrent. The VACUM process is very intensive so it could take a while….
    (1 row)

    INSERT 0 24050
    UPDATE 0
    Tue May 30 19:57:54 UTC 2017 : Going to schema sequence 24
    UPDATE 1
    Tue May 30 19:57:55 UTC 2017 : Vacuuming DB.

  21. Jeff Sturgis

    To fix the 502 error you will need to do the following:
    1) Connect to the running musicbrainz container with docker exec -it musicbrainzdocker_musicbrainz_1 bash

    2) Copy this script into the root directory (/)

    3) Replace the / script with this script

    4) Restart the container.

  22. Scott B.

    @Jeff Sturgis. I performed (and now have rechecked) the remediation steps above (including the two addenda) and I’m still not back to normal operation. Still getting “502 Bad Gateway” in the browser, and /var/log/nginx/001-musicbrainz.error.log [in container] still logs entries like this:

    2017/06/06 13:20:03 [error] 16#16: *13 upstream prematurely closed connection while reading response header from upstream, client: w.x.y.z, server: , request: “GET / HTTP/1.1”, upstream: “”, host: “musicbrainz.a.b.c:5000”, referrer: “http://h.a.b.c/”

    I think at least all of the daemons are running now; here’s the output from ps -ef inside musicbrainzdocker_musicbrainz_1:

    root 1 0 0 12:44 ? 00:00:00 /bin/sh /
    root 9 1 0 12:44 ? 00:00:00 cron -f
    root 11 1 0 12:44 ? 00:00:01 redis-server *:6379
    root 15 1 0 12:44 ? 00:00:00 nginx: master process nginx
    www-data 16 15 0 12:44 ? 00:00:00 nginx: worker process
    root 21 1 0 12:44 ? 00:00:01 node /musicbrainz-server/script/../root/server-compat.js –socket /tmp/musicbrainz-template-renderer.socket
    root 22 1 0 12:44 ? 00:00:00 /usr/bin/perl /usr/local/bin/start_server –port=55901 — plackup -I lib -s Starlet -E deployment –nproc 10 –pid
    root 27 22 0 12:44 ? 00:00:06 /usr/local/bin/plackup
    root 28 21 0 12:44 ? 00:00:00 /usr/bin/nodejs /musicbrainz-server/root/server-compat.js –socket /tmp/musicbrainz-template-renderer.socket
    root 33 27 0 12:44 ? 00:00:00 /usr/local/bin/plackup
    root 34 27 0 12:44 ? 00:00:00 /usr/local/bin/plackup
    root 35 27 0 12:44 ? 00:00:00 /usr/local/bin/plackup
    root 36 27 0 12:44 ? 00:00:00 /usr/local/bin/plackup
    root 37 27 0 12:44 ? 00:00:00 /usr/local/bin/plackup
    root 38 27 0 12:44 ? 00:00:00 /usr/local/bin/plackup
    root 39 27 0 12:44 ? 00:00:00 /usr/local/bin/plackup
    root 40 27 0 12:44 ? 00:00:00 /usr/local/bin/plackup
    root 41 27 0 12:44 ? 00:00:00 /usr/local/bin/plackup
    root 42 27 0 12:44 ? 00:00:00 /usr/local/bin/plackup
    root 46 0 0 12:51 ? 00:00:00 bash
    root 121 46 0 13:25 ? 00:00:00 ps -ef
    root 122 46 0 13:25 ? 00:00:00 cat

    Output from the docker … up command looks the same as before, minus the final error:

    musicbrainz_1 | Argument “Moose::Meta::Method=HASH(0x33a10d8)” isn’t numeric in numeric eq (==) at /usr/local/share/perl/5.22.1/MooseX/ABC/Trait/ line 61.
    musicbrainz_1 | Argument “Moose::Meta::Method=HASH(0x33a10d8)” isn’t numeric in numeric eq (==) at /usr/local/share/perl/5.22.1/MooseX/ABC/Trait/ line 61.
    musicbrainz_1 | Argument “Moose::Meta::Method=HASH(0x33a10d8)” isn’t numeric in numeric eq (==) at /usr/local/share/perl/5.22.1/MooseX/ABC/Trait/ line 61.
    musicbrainz_1 | Argument “Moose::Meta::Method=HASH(0x33a10d8)” isn’t numeric in numeric eq (==) at /usr/local/share/perl/5.22.1/MooseX/ABC/Trait/ line 61.

    Is there any other diagnostic information I can provide?


  23. Jeff Sturgis

    @Scott B. the output from ps looks good to me, can you try making a request from inside the musicbrainz container to the plack server with “wget” and see if it works. Also take a look at the output from “docker logs musicbrainzdocker_musicbrainz_1” after making the wget request and see if you have any errors.

  24. Scott B.

    @Jeff Sturgis. Here’s the result (using curl, with tracing) — and basically, there’s no result. 😉

    root@d01c6166ee03:/musicbrainz-server# curl –trace rsb curl: (52) Empty reply from server
    root@d01c6166ee03:/musicbrainz-server# more rsb
    == Info: Trying…
    == Info: Connected to ( port 55901 (#0)
    => Send header, 79 bytes (0x4f)
    0000: 47 45 54 20 2f 20 48 54 54 50 2f 31 2e 31 0d 0a GET / HTTP/1.1..
    0010: 48 6f 73 74 3a 20 31 32 37 2e 30 2e 30 2e 31 3a Host:
    0020: 35 35 39 30 31 0d 0a 55 73 65 72 2d 41 67 65 6e 55901..User-Agen
    0030: 74 3a 20 63 75 72 6c 2f 37 2e 34 37 2e 30 0d 0a t: curl/7.47.0..
    0040: 41 63 63 65 70 74 3a 20 2a 2f 2a 0d 0a 0d 0a Accept: */*….
    == Info: Empty reply from server
    == Info: Connection #0 to host left intact

    This resulted in a screenful of repeats of this message:

    [error] Caught exception in engine “Can’t locate object method “MEMCACHED_NAMESPACE” via package “DBDefs” at lib/ line 331.”

    I sense this will lead to an “ah-ha” moment on your part, but I can’t put the pieces together myself…

  25. Scott B.

    @Jeff Sturgis. I think I’m losing my mind. 😉 I’d done all of the git stuff as documented above, but subsequently had played with git a bit more and thought I might have screwed up something. I re-ran all of the steps from git fetch through scripts/ (except for running and everything ran clean, but I still don’t get any change in behavior from the VM. Except there are no messages in the docker logs any more. I don’t know if I was imagining them, or if they were old, or I fixed that but something else is still wrong.

    As a measure of my desperation, here is output from running strace against one of the plackup processes and pounding my browser’s refresh button until the connection went to the process I was tracing:

    root@vagrant-ubuntu-trusty-64:~# strace -p 15750 -f -r
    Process 15750 attached
    0.000000 accept(4, {sa_family=AF_INET, sin_port=htons(60088), sin_addr=inet_addr(“”)}, [16]) = 5
    16.673477 ioctl(5, SNDCTL_TMR_TIMEBASE or SNDRV_TIMER_IOCTL_NEXT_DEVICE or TCGETS, 0x7fff7ba68570) = -1 ENOTTY (Inappropriate ioctl for device)
    0.000137 lseek(5, 0, SEEK_CUR) = -1 ESPIPE (Illegal seek)
    0.000073 ioctl(5, SNDCTL_TMR_TIMEBASE or SNDRV_TIMER_IOCTL_NEXT_DEVICE or TCGETS, 0x7fff7ba68570) = -1 ENOTTY (Inappropriate ioctl for device)
    0.000065 lseek(5, 0, SEEK_CUR) = -1 ESPIPE (Illegal seek)
    0.000044 fcntl(5, F_SETFD, FD_CLOEXEC) = 0
    0.000084 fcntl(5, F_GETFL) = 0x2 (flags O_RDWR)
    0.000034 fcntl(5, F_SETFL, O_RDWR|O_NONBLOCK) = 0
    0.000044 setsockopt(5, SOL_TCP, TCP_NODELAY, [1], 4) = 0
    0.000099 read(5, “GET / HTTP/1.0\r\nHost: musicbrain”…, 131072) = 429
    0.001152 close(5) = 0
    0.000362 accept(4, ^CProcess 15750 detached

    Does this suggest anything to you? It seems to be reading the connect request (at the end) but then immediately closes the connection without doing anything. :-p [ “-r” is showing relative times between the calls, which might or might not be useful…]

    Still stumped,

  26. Scott B.


    I mused over @Jeff Sturgis’ comment about currency and did some more digging. I am up to date (at least with respect to these instructions):

    root@d01c6166ee03:/musicbrainz-server# git status
    HEAD detached at v-2017-05-15-schema-change
    Changes not staged for commit:
    (use “git add …” to update what will be committed)
    (use “git checkout — …” to discard changes in working directory)

    modified: postgresql-musicbrainz-unaccent (new commits)

    no changes added to commit (use “git add” and/or “git commit -a”)

    But I went and found which I believe is what we’re discussing, and it was *NOT* applied to my lib/ file.

    Therefore, I went and made the change manually:

    — lib/ 2017-06-06 12:38:43.488823776 +0000
    +++ lib/ 2017-06-10 19:50:42.706130240 +0000
    @@ -328,7 +328,7 @@
    my $self = shift;
    return {
    – prefix => $self->MEMCACHED_NAMESPACE(),
    + prefix => ‘MB:’,
    database => 0,
    test_database => 1,
    redis_new_args => {

    …and restarted everything. This resulted in two changes in behavior:

    1. It takes the browser a lot longer to fail, and sometimes it returns 504 Timeout instead of 502 Bad Gateway.
    2. When a connect comes in, the docker log spews zillions of this message:

    Use of uninitialized value in concatenation (.) or string at /usr/local/share/perl/5.22.1/ line 615.

    That would appear to be this line, for what it’s worth:

    615: $self->{sock} = $self->{builder}->($self)

    Oh well. Back to Google, but I thought I’d share my progress (or lack of same).

  27. Scott B.

    strace pointed me to the fact that, buried amidst those “Use of uninitialized value in …” messages is this one, that comes out first for each connection:

    [error] Caught exception in engine “Could not connect to Redis server at : Invalid argument at lib/MusicBrainz/ line 24.

    That might be more useful. Or not. I wish I read perl above preschool level, or knew anything about any of the building blocks in use here. :-p

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s