Schema change release, 2017-05-15 (including upgrade instructions)

44 thoughts on “Schema change release, 2017-05-15 (including upgrade instructions)”

1. Thanks for the above instructions.
   Are they valid for your VM too?

2. @InvisibleMan78 AFAIK, there’s nothing special about the VM that would make these steps not work, assuming you know how to take down the web server there, turn off cron, etc. The commands listed should apply to any MBS install, VM or not. But, I also haven’t tested any of this inside the VM.

3. I had to run `cpanm –installdeps –notest .` before `./upgrade.sh`
   and
   `npm install` before running `./script/compile_resources.sh` in order to get all the dependencies

4. @rembo10, thanks, I’ve updated the steps in the blog post too in case it helps others. I’m curious what modules you had missing that upgrade.sh needed.

5. I’m getting errors when running plackup and the web server fails to start.

   raven@1.2.0 alert: no DSN provided, error reporting disabled
   raven@1.2.0 alert: uncaughtException:

   Looks like this bug was fixed with raven@2.0.0, but upgrading to that version causes other problems. (ref: https://github.com/getsentry/raven-node/issues/301)

6. @theguardian, I opened a pull request earlier today which might help: https://github.com/metabrainz/musicbrainz-server/pull/510

   It updates node-raven like you mentioned, and also fixes some incompatibilities with node < 5.10. Let me know if those changes help and we'll get it merged quickly.

7. How can we apply the latest fix from @mwiencek?
   Could please someone instruct us with the necessary commands?

8. Commands to apply use this fix currently under review:
   git remote add mwiencek https://github.com/mwiencek/musicbrainz-server.git
   git fetch mwiencek && git checkout node-fixes

   (from https://chatlogs.metabrainz.org/brainzbot/metabrainz/msg/3910251/ )

9. Where should we report errors like
   “undef error – malformed JSON string, neither tag, array, object, number, string or atom, at character offset 0 (before “(end of string)”) at lib/MusicBrainz/Server/Renderer.pm line 42. ”
   calling the local webserver in a browser?

10. @InvisibleMan78 At http://tickets.musicbrainz.org please!

    I’ve noticed the same error in production, but only 4 times (so far) out of thousands and thousands of requests, so I’m not sure how to reproduce it yet. If you can consistently reproduce it, that would be great information to provide in the ticket.

11. Should we expect an mbslave update?

12. Were anybody actually able to upgrade the VM? After “cpanm –installdeps –notest .” I needed to install libjson-perl and libmoose-perl, but then I I get these errors when running “./upgrade.sh”

    vagrant@musicbrainz:~/musicbrainz-server$ ./upgrade.sh
    ~/musicbrainz-server/admin ~/musicbrainz-server
    ~/musicbrainz-server
    Use of uninitialized value $ENV{“DB_ENV_POSTGRES_USER”} in string at lib/DBDefs.pm line 51.
    Use of uninitialized value $ENV{“DB_ENV_POSTGRES_PASSWORD”} in string at lib/DBDefs.pm line 51.
    Use of uninitialized value $ENV{“DB_PORT_5432_TCP_ADDR”} in string at lib/DBDefs.pm line 51.
    Use of uninitialized value $ENV{“DB_PORT_5432_TCP_PORT”} in string at lib/DBDefs.pm line 51.
    Use of uninitialized value $ENV{“DB_ENV_POSTGRES_USER”} in string at lib/DBDefs.pm line 51.
    Use of uninitialized value $ENV{“DB_ENV_POSTGRES_PASSWORD”} in string at lib/DBDefs.pm line 51.
    Use of uninitialized value $ENV{“DB_PORT_5432_TCP_ADDR”} in string at lib/DBDefs.pm line 51.
    Use of uninitialized value $ENV{“DB_PORT_5432_TCP_PORT”} in string at lib/DBDefs.pm line 51.
    Use of uninitialized value $ENV{“DB_ENV_POSTGRES_USER”} in string at lib/DBDefs.pm line 51.
    Use of uninitialized value $ENV{“DB_ENV_POSTGRES_PASSWORD”} in string at lib/DBDefs.pm line 51.
    Use of uninitialized value $ENV{“DB_PORT_5432_TCP_ADDR”} in string at lib/DBDefs.pm line 51.
    Use of uninitialized value $ENV{“DB_PORT_5432_TCP_PORT”} in string at lib/DBDefs.pm line 51.
    Attribute (port) does not pass the type constraint because: Validation failed for ‘Int’ with value “” at constructor MusicBrainz::Server::Database::new (defined at lib/MusicBrainz/Server/Database.pm line 53) line 52.
    MusicBrainz::Server::Database::new(‘MusicBrainz::Server::Database’, ‘HASH(0xe1b300)’) called at lib/MusicBrainz/Server/DatabaseConnectionFactory.pm line 19
    MusicBrainz::Server::DatabaseConnectionFactory::register_databases(‘MusicBrainz::Server::DatabaseConnectionFactory’, ‘READWRITE’, ‘HASH(0x1df8dc8)’, ‘TEST’, ‘HASH(0xe1af70)’, ‘READONLY’, ‘HASH(0xe1b120)’, ‘SYSTEM’, ‘HASH(0xe1b300)’) called at lib/DBDefs.pm line 51
    require DBDefs.pm called at -e line 1
    main::BEGIN() called at lib/DBDefs.pm line 0
    eval {…} called at lib/DBDefs.pm line 0
    Compilation failed in require at -e line 1.
    BEGIN failed–compilation aborted at -e line 1.

13. I had the same issue as boehmi updating the VM. Any tips or suggestions welcome.

14. Same here… Unable to update schema on VM with dockers.
    Same error as boehmi

15. boehmi, try making these changes In the lib/DBDefs.pm file,

    $ENV{“DB_PORT_5432_TCP_ADDR”} to “db”
    $ENV{“DB_PORT_5432_TCP_PORT”} to “5432”
    $ENV{DB_ENV_POSTGRES_USER} to “musicbrainz”
    $ENV{DB_ENV_POSTGRES_PASSWORD} to “musicbrainz”

16. Wouldn’t it be better to define the variables somewhere? Anyway, I made the changes and ran into the next problem(s). First psql was not installed. Easily solved by installing postgresql-client-9.3. But then the update script complained
    psql: could not translate host name “db” to address: Name or service not known
    I changed “db” to the actual hostname “musicbrainz”, but it still couldn’t connect to the DB.

    I guess putting everything in docker containers makes everything more complicated. Did anybody actually upgrade the VM successfully?

17. -1 for me, still waiting that one of the MB-Developers upgrade the official MB-VM to a working, downloadable, actual version.

18. Running into the same error. Please provide an updated vm 🙂 would be great.

19. Ok, I got a little step further. The docker containers create virtual network devices and I figured out that the database runs on 172.17.0.2. After running ./update.sh again I get
    Error: Schema sequence must be 22 when you run this script

20. @boehmi Regarding the “Schema sequence must be 22” error, you don’t have the latest code, because otherwise it would say 23, not 22.

21. Got to where the schema should update, however, I get the same error as above. Edited the lib/DBDefs file and canhed the host to 127.0.0.1 and set port to 5432, this allows connection but still failing on username and password. I have tried using musicbrainz for both but that is rejected.

    Further assistance would be helpful to get this resolved.

22. I made it further, on my zillionth attempt — everything I know about docker, I learned attempting to perform this update. I’m not out of the woods yet, but I’m closer. Maybe everybody else’s experience is different, but I have found that I need to manually do:

    screen docker-compose -f /home/vagrant/musicbrainz/musicbrainz-docker/docker-compose.yml up

    in order to bring everything up, and ^C in this context to gracefully shut it all down. This becomes relevant at the end.

    Relative to the original upgrade instructions posted at the top of this thread:

    1. Make sure you have a snapshot of your VM so you can back off and try again if something goes wrong!
    2. Use “docker exec -it musicbrainzdocker_musicbrainz_1 bash” to establish an interactive session inside the musicbrainz docker image; this is where you should be doing steps 5-9.
    3. You can restart replication at this point, but it really thrashes your I/O so I suggest waiting until you have finished the remaining work and are ready to allow your system to churn overnight.
    4. The final step (11) was a nightmare for me. You need to run it inside the docker image too. Generally, each time (there were many) I got an error running script/compile_resources.sh, starting with complaint about “is-integer”, I did “npm install is-integer” (substitute the module inferred from the error message) and then retry the script. When that (finally) runs without failing, stop and restart everything (reattach to screen, ^C to shut it down, then restart with the docker-compose command above). When you see MORE error messages in the startup log, reconnect to the image (docker exec) and “npm install ” too, followed by running the script again, followed by restarting the environment. Eventually the script AND the restart will both run apparently clean. (There are a lot of messages I see now that I saw before the schema change, so I’m tuning them out.)
    5. However, the web interface is broken. Just trying to connect to the landing page (http://myhost:5000) fails with “502 Bad Gateway”.

    As I write this, I have a totally current database but no way to query it. 😉 I’m hoping somebody with more understanding of the web mechanics can fill in the missing pieces for us.

23. I am waiting to finish VACUMing the DB. Ig I succed on update the schema on VM-MB I will post here and upload by torrent. The VACUM process is very intensive so it could take a while….
    ….
    ALTER TABLE
    ?column?
    ———————————-
    20170503-mbs-9329-event-meta.sql
    (1 row)

    CREATE FUNCTION
    TRUNCATE TABLE
    INSERT 0 24050
    UPDATE 0
    COMMIT
    Tue May 30 19:57:54 UTC 2017 : Going to schema sequence 24
    UPDATE 1
    Tue May 30 19:57:55 UTC 2017 : Vacuuming DB.
    SET

24. @Scott B. Maybe we will get an answer for the 502-problem here or in the ticket:
    https://tickets.metabrainz.org/projects/MBVM/issues/MBVM-32

25. arrive to same port 😦

    502 Bad Gateway
    nginx/1.10.0 (Ubuntu)

26. Will there be a new VM image made with this update included?

27. +1 for a new VM please.

28. To fix the 502 error you will need to do the following:
    1) Connect to the running musicbrainz container with docker exec -it musicbrainzdocker_musicbrainz_1 bash

    2) Copy this script into the root directory (/) https://github.com/jsturgis/musicbrainz-docker/blob/master/musicbrainz-dockerfile/scripts/start_mb_renderer.pl

    3) Replace the /start.sh script with this script https://github.com/jsturgis/musicbrainz-docker/blob/master/musicbrainz-dockerfile/scripts/start.sh

    4) Restart the container.

29. oh also make start_mb_renderer.pl executable with chmod +x /start_mb_renderer.pl before restarting the container.

30. Sorry one other thing, you will need to add “sub FORK_RENDERER { 0 }” to your DBDefs.pm file.

31. @Jeff Sturgis. I performed (and now have rechecked) the remediation steps above (including the two addenda) and I’m still not back to normal operation. Still getting “502 Bad Gateway” in the browser, and /var/log/nginx/001-musicbrainz.error.log [in container] still logs entries like this:

    2017/06/06 13:20:03 [error] 16#16: *13 upstream prematurely closed connection while reading response header from upstream, client: w.x.y.z, server: , request: “GET / HTTP/1.1”, upstream: “http://127.0.0.1:55901/”, host: “musicbrainz.a.b.c:5000”, referrer: “http://h.a.b.c/”

    I think at least all of the daemons are running now; here’s the output from ps -ef inside musicbrainzdocker_musicbrainz_1:

    UID PID PPID C STIME TTY TIME CMD
    root 1 0 0 12:44 ? 00:00:00 /bin/sh /start.sh
    root 9 1 0 12:44 ? 00:00:00 cron -f
    root 11 1 0 12:44 ? 00:00:01 redis-server *:6379
    root 15 1 0 12:44 ? 00:00:00 nginx: master process nginx
    www-data 16 15 0 12:44 ? 00:00:00 nginx: worker process
    root 21 1 0 12:44 ? 00:00:01 node /musicbrainz-server/script/../root/server-compat.js –socket /tmp/musicbrainz-template-renderer.socket
    root 22 1 0 12:44 ? 00:00:00 /usr/bin/perl /usr/local/bin/start_server –port=55901 — plackup -I lib -s Starlet -E deployment –nproc 10 –pid fcgi.pid
    root 27 22 0 12:44 ? 00:00:06 /usr/local/bin/plackup
    root 28 21 0 12:44 ? 00:00:00 /usr/bin/nodejs /musicbrainz-server/root/server-compat.js –socket /tmp/musicbrainz-template-renderer.socket
    root 33 27 0 12:44 ? 00:00:00 /usr/local/bin/plackup
    root 34 27 0 12:44 ? 00:00:00 /usr/local/bin/plackup
    root 35 27 0 12:44 ? 00:00:00 /usr/local/bin/plackup
    root 36 27 0 12:44 ? 00:00:00 /usr/local/bin/plackup
    root 37 27 0 12:44 ? 00:00:00 /usr/local/bin/plackup
    root 38 27 0 12:44 ? 00:00:00 /usr/local/bin/plackup
    root 39 27 0 12:44 ? 00:00:00 /usr/local/bin/plackup
    root 40 27 0 12:44 ? 00:00:00 /usr/local/bin/plackup
    root 41 27 0 12:44 ? 00:00:00 /usr/local/bin/plackup
    root 42 27 0 12:44 ? 00:00:00 /usr/local/bin/plackup
    root 46 0 0 12:51 ? 00:00:00 bash
    root 121 46 0 13:25 ? 00:00:00 ps -ef
    root 122 46 0 13:25 ? 00:00:00 cat

    Output from the docker … up command looks the same as before, minus the final error:

    musicbrainz_1 | Argument “Moose::Meta::Method=HASH(0x33a10d8)” isn’t numeric in numeric eq (==) at /usr/local/share/perl/5.22.1/MooseX/ABC/Trait/Class.pm line 61.
    musicbrainz_1 | Argument “Moose::Meta::Method=HASH(0x33a10d8)” isn’t numeric in numeric eq (==) at /usr/local/share/perl/5.22.1/MooseX/ABC/Trait/Class.pm line 61.
    musicbrainz_1 | Argument “Moose::Meta::Method=HASH(0x33a10d8)” isn’t numeric in numeric eq (==) at /usr/local/share/perl/5.22.1/MooseX/ABC/Trait/Class.pm line 61.
    musicbrainz_1 | Argument “Moose::Meta::Method=HASH(0x33a10d8)” isn’t numeric in numeric eq (==) at /usr/local/share/perl/5.22.1/MooseX/ABC/Trait/Class.pm line 61.

    Is there any other diagnostic information I can provide?

    Thanks,
    -Scott

32. @Scott B. the output from ps looks good to me, can you try making a request from inside the musicbrainz container to the plack server with “wget 127.0.0.1:55901” and see if it works. Also take a look at the output from “docker logs musicbrainzdocker_musicbrainz_1” after making the wget request and see if you have any errors.

33. @Jeff Sturgis. Here’s the result (using curl, with tracing) — and basically, there’s no result. 😉

    root@d01c6166ee03:/musicbrainz-server# curl –trace rsb http://127.0.0.1:55901/ curl: (52) Empty reply from server
    root@d01c6166ee03:/musicbrainz-server# more rsb
    == Info: Trying 127.0.0.1…
    == Info: Connected to 127.0.0.1 (127.0.0.1) port 55901 (#0)
    => Send header, 79 bytes (0x4f)
    0000: 47 45 54 20 2f 20 48 54 54 50 2f 31 2e 31 0d 0a GET / HTTP/1.1..
    0010: 48 6f 73 74 3a 20 31 32 37 2e 30 2e 30 2e 31 3a Host: 127.0.0.1:
    0020: 35 35 39 30 31 0d 0a 55 73 65 72 2d 41 67 65 6e 55901..User-Agen
    0030: 74 3a 20 63 75 72 6c 2f 37 2e 34 37 2e 30 0d 0a t: curl/7.47.0..
    0040: 41 63 63 65 70 74 3a 20 2a 2f 2a 0d 0a 0d 0a Accept: */*….
    == Info: Empty reply from server
    == Info: Connection #0 to host 127.0.0.1 left intact
    root@d01c6166ee03:/musicbrainz-server#

    This resulted in a screenful of repeats of this message:

    [error] Caught exception in engine “Can’t locate object method “MEMCACHED_NAMESPACE” via package “DBDefs” at lib/DBDefs.pm line 331.”

    I sense this will lead to an “ah-ha” moment on your part, but I can’t put the pieces together myself…

34. @Scott B. did you already run git fetch origin followed by git checkout v-2017-05-15-schema-change? it sounds like you have an old version of DBDefs before this change was made.

    https://bitbucket.org/mwiencek/musicbrainz-server/commits/ae59487204f717a051c7128d85faa3a10b7cac25?at=react-server

35. @Jeff Sturgis. I think I’m losing my mind. 😉 I’d done all of the git stuff as documented above, but subsequently had played with git a bit more and thought I might have screwed up something. I re-ran all of the steps from git fetch through scripts/compile_resources.sh (except for running upgrade.sh) and everything ran clean, but I still don’t get any change in behavior from the VM. Except there are no messages in the docker logs any more. I don’t know if I was imagining them, or if they were old, or I fixed that but something else is still wrong.

    As a measure of my desperation, here is output from running strace against one of the plackup processes and pounding my browser’s refresh button until the connection went to the process I was tracing:

    root@vagrant-ubuntu-trusty-64:~# strace -p 15750 -f -r
    Process 15750 attached
    0.000000 accept(4, {sa_family=AF_INET, sin_port=htons(60088), sin_addr=inet_addr(“127.0.0.1”)}, [16]) = 5
    16.673477 ioctl(5, SNDCTL_TMR_TIMEBASE or SNDRV_TIMER_IOCTL_NEXT_DEVICE or TCGETS, 0x7fff7ba68570) = -1 ENOTTY (Inappropriate ioctl for device)
    0.000137 lseek(5, 0, SEEK_CUR) = -1 ESPIPE (Illegal seek)
    0.000073 ioctl(5, SNDCTL_TMR_TIMEBASE or SNDRV_TIMER_IOCTL_NEXT_DEVICE or TCGETS, 0x7fff7ba68570) = -1 ENOTTY (Inappropriate ioctl for device)
    0.000065 lseek(5, 0, SEEK_CUR) = -1 ESPIPE (Illegal seek)
    0.000044 fcntl(5, F_SETFD, FD_CLOEXEC) = 0
    0.000084 fcntl(5, F_GETFL) = 0x2 (flags O_RDWR)
    0.000034 fcntl(5, F_SETFL, O_RDWR|O_NONBLOCK) = 0
    0.000044 setsockopt(5, SOL_TCP, TCP_NODELAY, [1], 4) = 0
    0.000099 read(5, “GET / HTTP/1.0\r\nHost: musicbrain”…, 131072) = 429
    0.001152 close(5) = 0
    0.000362 accept(4, ^CProcess 15750 detached

    Does this suggest anything to you? It seems to be reading the connect request (at the end) but then immediately closes the connection without doing anything. :-p [ “-r” is showing relative times between the calls, which might or might not be useful…]

    Still stumped,
    -Scott

36. Closer?

    I mused over @Jeff Sturgis’ comment about currency and did some more digging. I am up to date (at least with respect to these instructions):

    root@d01c6166ee03:/musicbrainz-server# git status
    HEAD detached at v-2017-05-15-schema-change
    Changes not staged for commit:
    (use “git add …” to update what will be committed)
    (use “git checkout — …” to discard changes in working directory)

    modified: postgresql-musicbrainz-unaccent (new commits)

    no changes added to commit (use “git add” and/or “git commit -a”)
    root@d01c6166ee03:/musicbrainz-server#

    But I went and found https://bitbucket.org/mwiencek/musicbrainz-server/commits/ae59487204f717a051c7128d85faa3a10b7cac25?at=react-server which I believe is what we’re discussing, and it was *NOT* applied to my lib/DBDefs.pm file.

    Therefore, I went and made the change manually:

    — lib/DBDefs.pm.orig 2017-06-06 12:38:43.488823776 +0000
    +++ lib/DBDefs.pm 2017-06-10 19:50:42.706130240 +0000
    @@ -328,7 +328,7 @@
    sub DATASTORE_REDIS_ARGS {
    my $self = shift;
    return {
    – prefix => $self->MEMCACHED_NAMESPACE(),
    + prefix => ‘MB:’,
    database => 0,
    test_database => 1,
    redis_new_args => {

    …and restarted everything. This resulted in two changes in behavior:

    1. It takes the browser a lot longer to fail, and sometimes it returns 504 Timeout instead of 502 Bad Gateway.
    2. When a connect comes in, the docker log spews zillions of this message:

    Use of uninitialized value in concatenation (.) or string at /usr/local/share/perl/5.22.1/Redis.pm line 615.

    That would appear to be this line, for what it’s worth:

    615: $self->{sock} = $self->{builder}->($self)

    Oh well. Back to Google, but I thought I’d share my progress (or lack of same).

37. strace pointed me to the fact that, buried amidst those “Use of uninitialized value in …” messages is this one, that comes out first for each connection:

    [error] Caught exception in engine “Could not connect to Redis server at : Invalid argument at lib/MusicBrainz/Redis.pm line 24.

    That might be more useful. Or not. I wish I read perl above preschool level, or knew anything about any of the building blocks in use here. :-p

38. @Scott B. (or anyone really)…any further progress? I’m in the *exact* same situation as you. This is infuriating considering how long it took to update from the 12-2016 VM build. Somebody please debug this. We’ll help in any way possible!

39. If anyone is using the musicbrainz provided VM. I gave up with it and ended up using this docker image: https://hub.docker.com/r/linuxserver/musicbrainz/

    It works, is simple to maintain and also has upgrade functionality. Good luck!

40. Hi James, if I already have the old VM running inside Virtualbox, can I just change over the docker image to this one, and can I use the already synced database? If so, what are the steps that you followed?

41. @James: Thank you for this helpful info. Does this linked docker image include the MB search server?

42. Yes, I believe so. Seems to work well.

43. Thanks for the pointer, @James. I’d about given up on this but now I’m back and running normally. A few hours spent watching it load and sync the database was much less work and delay than the manual hacking I’d tried with the old VM.

    @Marc, I don’t think you can easily drop this into your old VM since it’s structured as a single docker image rather that a set of them. The layout of the tree in the exposed database directory isn’t exactly identical either, and I decided that paying a few hours up front to perform an “unnecessary” resynchronization was worth it to simplify the configuration going forward and making sure I can apply future image updates easily without having to think about what magic would be required to reconcile the two streams.

    Of course, my host system is running Debian Linux so it wasn’t much effort for me to throw away the VM, install docker, and follow the (short) instructions to fire up the linuxserver/musicbrainz image. If I were running on Windows, I might consider starting with the musicbrainz VM, destroying all of the old (broken) docker images, and loading the new image there in replacement for all of them. I’d still just do a complete database reload — I have no idea if the PostgreSQL version in the respective images is the same or binary-compatible.

    FWIW. 😉

44. Scott, I agree entirely. It did only take a few hours to resync and it was so worth it as the configuration is so many times more straightforward. I previously spent quite a bit of time trying (and failing) to get the VM image to update and every time the process has been different.

    Good luck Marc 🙂

    Docker FTW!

Leave a Reply

Enter your comment here...

Fill in your details below or click an icon to log in:

Email (required) (Address never made public)

Name (required)

Website

You are commenting using your WordPress.com account. ( Log Out /  Change )

You are commenting using your Facebook account. ( Log Out /  Change )

Cancel

Connecting to %s

Notify me of new comments via email.

Notify me of new posts via email.

Δ

This site uses Akismet to reduce spam. Learn how your comment data is processed.