Server update, 2016-06-06

9 thoughts on “Server update, 2016-06-06”

1. Thanks for this update.
   Do you have any news when the next Virtual Machine (with latest Schema Change) will be released?

2. Yep waiting for the virtual machine also. Both the big public mirrors are currently not replicating.

3. Well after a few days of trial & error, I got it working with a fresh install of the 8/5/2015 VBox template. Here’s my steps in case someone doesn’t want to wait for an update:

   ### Applies to MusicBrainz Server Virtual Machine (Virtual Box) Version: 2015-08-06
   ### commands for updating the machine to import a recent data set (20160608)

   ssh 192.168.change.me -p 2222 -lvm

   ### update musicbrainz files from git repo
   sudo su – musicbrainz
   cd ~/musicbrainz-server/

   git checkout master
   git fetch origin
   git checkout v-2016-06-06

   ### update DB_SCHEMA_SEQUENCE from 22 to 23
   ### don’t forget to set your API key too
   nano lib/DBDefs.pm

   ### upgrade postgresql from 9.3 to 9.5
   exit
   cd
   wget https://alioth.debian.org/scm/loggerhead/pkg-postgresql/postgresql-common/trunk/download/head:/apt.postgresql.org.s-20130224224205-px3qyst90b3xp8zj-1/apt.postgresql.org.sh
   chmod a+x apt.postgresql.org.sh
   sed -i -e ‘s/$CODENAME-pgdg main/$CODENAME-pgdg main 9.5/g’ apt.postgresql.org.sh
   sudo ./apt.postgresql.org.sh
   rm apt.postgresql.org.sh
   sudo apt-get install postgresql-9.5 postgresql-client-9.5 postgresql-contrib-9.5 -y
   sudo pg_dropcluster 9.5 main –stop
   sudo pg_upgradecluster 9.3 main
   sudo pg_dropcluster 9.3 main

   ### resolve /usr/lib/postgresql/9.5/lib/pgxs/src/makefiles/pgxs.mk “No Such File or directory” (becomes a problem if you try to install the extensions in next step before doing this)
   sudo apt-get install postgresql-server-dev-all -y
   sudo apt-get install postgresql-common -y

   ### reinstall musicbrainz postgresql extensions for 9.5
   sudo su – musicbrainz
   cd ~/musicbrainz-server/postgresql-musicbrainz-collate/
   make
   exit
   cd /home/musicbrainz/musicbrainz-server/postgresql-musicbrainz-collate/
   sudo make install

   sudo su – musicbrainz
   cd ~/musicbrainz-server/postgresql-musicbrainz-unaccent/
   make
   exit
   cd /home/musicbrainz/musicbrainz-server/postgresql-musicbrainz-unaccent/
   sudo make install

   sudo service postgresql restart

   ### now to create a new musicbrainz db from dumps
   sudo su – musicbrainz

   ###### grab the dumps from FTP if you don’t have them already…
   ### mkdir ~/dumps/
   ### cd ~/dumps
   ### wget ftp://ftp.musicbrainz.org/pub/musicbrainz/data/fullexport/20160608-001719/mbdump-editor.tar.bz2
   ### wget ftp://ftp.musicbrainz.org/pub/musicbrainz/data/fullexport/20160608-001719/mbdump-derived.tar.bz2
   ### wget ftp://ftp.musicbrainz.org/pub/musicbrainz/data/fullexport/20160608-001719/mbdump.tar.bz2

   cd ~/musicbrainz-server/

   ### to quickly double-check that things “should” work, create a clean db
   ### ./admin/InitDb.pl –createdb –clean –echo
   ### dropdb musicbrainz

   ### import the dumps
   dropdb musicbrainz
   ./admin/InitDb.pl –createdb –clean –echo –import ~/dumps/mbdump*.tar.bz2

   ### start replicating!
   ./bin/replicate now
   ./bin/replicate start

4. Well after much trial and error, I finally got recent data dumps working with the 8/2015 VBox template. This was in a FreeNAS jail, so YMMV.

   ### Applies to MusicBrainz Server Virtual Machine (Virtual Box) Version: 2015-08-06
   ### commands for updating the machine to import a recent data set (20160608)

   ssh 192.168.change.me -p 2222 -lvm

   ### update musicbrainz files from git repo
   sudo su – musicbrainz
   cd ~/musicbrainz-server/

   git checkout master
   git fetch origin
   git checkout v-2016-06-06

   ### update DB_SCHEMA_SEQUENCE from 22 to 23
   ### don’t forget to set your API key too
   nano lib/DBDefs.pm

   ### upgrade postgresql from 9.3 to 9.5
   exit
   cd
   wget https://alioth.debian.org/scm/loggerhead/pkg-postgresql/postgresql-common/trunk/download/head:/apt.postgresql.org.s-20130224224205-px3qyst90b3xp8zj-1/apt.postgresql.org.sh
   chmod a+x apt.postgresql.org.sh
   sed -i -e ‘s/$CODENAME-pgdg main/$CODENAME-pgdg main 9.5/g’ apt.postgresql.org.sh
   sudo ./apt.postgresql.org.sh
   rm apt.postgresql.org.sh
   sudo apt-get install postgresql-9.5 postgresql-client-9.5 postgresql-contrib-9.5 -y
   sudo pg_dropcluster 9.5 main –stop
   sudo pg_upgradecluster 9.3 main
   sudo pg_dropcluster 9.3 main

   ### resolve /usr/lib/postgresql/9.5/lib/pgxs/src/makefiles/pgxs.mk “No Such File or directory” (becomes a problem if you try to install the extensions in next step before doing this)
   sudo apt-get install postgresql-server-dev-all -y
   sudo apt-get install postgresql-common -y

   ### reinstall musicbrainz postgresql extensions for 9.5
   sudo su – musicbrainz
   cd ~/musicbrainz-server/postgresql-musicbrainz-collate/
   make
   exit
   cd /home/musicbrainz/musicbrainz-server/postgresql-musicbrainz-collate/
   sudo make install

   sudo su – musicbrainz
   cd ~/musicbrainz-server/postgresql-musicbrainz-unaccent/
   make
   exit
   cd /home/musicbrainz/musicbrainz-server/postgresql-musicbrainz-unaccent/
   sudo make install

   sudo service postgresql restart

   ### now to create a new musicbrainz db from dumps
   sudo su – musicbrainz

   ###### grab the dumps from FTP if you don’t have them already…
   ### mkdir ~/dumps/
   ### cd ~/dumps
   ### wget ftp://ftp.musicbrainz.org/pub/musicbrainz/data/fullexport/20160608-001719/mbdump-editor.tar.bz2
   ### wget ftp://ftp.musicbrainz.org/pub/musicbrainz/data/fullexport/20160608-001719/mbdump-derived.tar.bz2
   ### wget ftp://ftp.musicbrainz.org/pub/musicbrainz/data/fullexport/20160608-001719/mbdump.tar.bz2

   cd ~/musicbrainz-server/

   ### to quickly double-check that things “should” work, create a clean db
   ### ./admin/InitDb.pl –createdb –clean –echo
   ### dropdb musicbrainz

   dropdb musicbrainz
   ./admin/InitDb.pl –createdb –clean –echo –import ~/dumps/mbdump*.tar.bz2

   ### start replicating!
   ./bin/replicate now
   ./bin/replicate start

5. my previous comment left out grabbing the postgresql extensions from git via a clone, here’s the revised block for that step:

   ### reinstall musicbrainz postgresql extensions for 9.5
   sudo su – musicbrainz
   cd ~/musicbrainz-server
   git clone https://github.com/metabrainz/postgresql-musicbrainz-collate.git
   cd postgresql-musicbrainz-collate/
   make
   exit
   cd /home/musicbrainz/musicbrainz-server/postgresql-musicbrainz-collate/
   sudo make install

   sudo su – musicbrainz
   cd ~/musicbrainz-server
   git clone https://github.com/metabrainz/postgresql-musicbrainz-unaccent.git
   cd postgresql-musicbrainz-unaccent/
   make
   exit
   cd /home/musicbrainz/musicbrainz-server/postgresql-musicbrainz-unaccent/
   sudo make install

   sudo service postgresql restart

6. Thanks a lot Chris, this works great!
   Could you please provide similar instructions for the included search server (https://blog.musicbrainz.org/2016/05/31/updated-search-jarwar-files/)
   Without changes to the search server, I get the
   ERROR: type “earth” does not exist
   during a bin/reindex command

7. @InvisibleMan78 I’m getting that same error too… took me a couple days worth of index building before it happened too 😦 I’m working through it & will post back on my results

8. @Chris: Thank you for your feedback!
   Just to let you know: You can reduce the test time if you rebuild only one or two indexes. You can shorten the line in the bin/reindex script from
   –indexes area,artist,releasegroup,release,recording,label,work,tag,annotation,cdstub
   to something like
   –indexes artist
   and see if this index works. Unfortunately, I can’t find a working solution…

9. @InvisibleMan78 I got bin/reindex to complete successfully (19.6 hours! I’ll take your advise on a smaller rebuild next time…) by replacing the indexer jar and am now stuck at the same place I believe you and Greg are: https://blog.musicbrainz.org/2016/05/31/updated-search-jarwar-files/#comment-130119

   ### update Musicbrainz VM with 5/31/2016 search-indexer jar to fix ERROR: type “earth” does not exist

   ssh 192.168.change.me -p 2222 -lvm

   sudo su – search
   cd jar/
   mv index.jar 20140525-index.jar
   wget http://ftp.musicbrainz.org/pub/musicbrainz/search/index/search-indexer-a947c76.jar
   mv search-indexer-a947c76.jar index.jar
   exit

   ### initiate reindex rebuild as VM user
   ./bin/reindex

   ### wait 19.6 hours – success!!

   ### take VM snapshot/run Ubuntu updates

   ### Seeing this on MB web interface
   ### WARNING: Could not load artist index, index is corrupted: Format version is not supported (resource: MMapIndexInput(path=”/home/search/indexdata/artist_index/segments.gen”)): -3 (needs to be between -2 and -2)
   # see index loading errors in /var/log/jetty/__DATESTAMP__.stderrout.log
   # Could not load [indexname] index, index is corrupted: Format version is not supported (resource: MMapIndexInput(path=”/home/search/indexdata/[indexname]_index/segments.gen”)): -3 (needs to be between -2 and -2)

   ### Test using 5/25/2016 war file for jetty servlet
   # note: you don’t want multiple versions of a war file in the /var/lib/jetty/webapps directory
   cd /var/lib/jetty/webapps/
   sudo su
   mv ROOT.war ../20150525-ROOT.war
   wget http://ftp.musicbrainz.org/pub/musicbrainz/search/servlet/search-server-e2c77b9.war
   mv search-server-e2c77b9.war ROOT.war
   /etc/init.d/jetty restart

   # does not work so switch back to 5/25/2015 war
   mv ROOT.war ../20160531-ROOT.war
   mv ../20150525-ROOT.war ROOT.war
   /etc/init.d/jetty restart

   exit

   ### fix rotatelogs error upon /etc/init.d/jetty restart “/etc/init.d/jetty: 274: /etc/init.d/jetty: /usr/sbin/rotatelogs: not found”
   sudo nano /etc/init.d/jetty
   # change ROTATELOGS=/usr/sbin/rotatelogs to ROTATELOGS=/usr/bin/rotatelogs in “# Define other required variables” section

   ###
   I’ve done quite a few things from pramach’s post on my MB VM setup: http://forums.musicbrainz.org/viewtopic.php?id=5261 (looks like you may have too!)
   –timezone corrected
   –get_last_replication
   –get_replication_gap
   –run_replication added to vm user’s.profile
   –get-ip-address|show-ip-address added to /etc/network/if-up.d

Leave a Reply

Enter your comment here...

Fill in your details below or click an icon to log in:

Email (required) (Address never made public)

Name (required)

Website

You are commenting using your WordPress.com account. ( Log Out /  Change )

You are commenting using your Facebook account. ( Log Out /  Change )

Cancel

Connecting to %s

Notify me of new comments via email.

Notify me of new posts via email.

Δ

This site uses Akismet to reduce spam. Learn how your comment data is processed.