Schema change server update, 2012-05-15

Nearly one year after we released NGS, we have another schema change update with lots of new features!

This release contains 9 new features and improvements that take advantage of the new schema. These are:

  • More social user profiles which can now have Gravatars, languages (and the users proficiency) age and country.
  • More expressive aliases for artists, labels and works. Aliases can now have types, sort names and multiple aliases may be used per a locale, along with the ability to mark one alias as ‘primary’ for that locale.
  • Release group types have been separated into primary and secondary types. A release group now has 1 primary type and may have multiple secondary types. This allows us to have ‘remix compilation albums’, for example
  • Works may have multiple ISWCs
  • Artists, labels and relationships may be marked as ‘ended’ to indicate that they have ended, but the exact date is not known
  • Vinyl style/free text track numbers are now supported.
  • Works may have a lyrics language associated with them
  • Artists and labels may have multiple IPIs
  • We have moved to use ISO 639-3 for our language table. While not all languages are exposed at the moment, this gives us a lot more flexibility going forward.

Many thanks to nikki for going way beyond our expectations for testing (and patience!); to Ian McEwen for his continued work on statistics; and to the MusicBrainz team for making this all happen.

If you have a replicated instance of MusicBrainz, please follow these instructions to get your server running on the new schema:

  1. Take down the web server running MusicBrainz, if you’re running a web server.
  2. Turn off cron jobs if you are automatically updating the database via cron jobs.
  3. Make sure your REPLICATION_TYPE setting is RT_SLAVE
  4. Switch to the new code with git fetch origin followed by git checkout v-2012-05-15-schema-change
  5. Run carton install --deployment. If you have not switched your installation to using carton, please read INSTALL.md on how to do this.
  6. Run carton exec -- ./upgrade.sh from the top of the source directory.
  7. Set DB_SCHEMA_SEQUENCE to 15 in lib/DBDefs.pm
  8. Turn cron jobs back on, if needed.
  9. Restart the MusicBrainz web server, if needed.

If you are running a mbslave mirror, check out the latest code and read the upgrade instructions in the README file.

Bug

  • [MBS-3189] – Remove unused ref_count column and related functions
  • [MBS-4616] – Add work language statistics
  • [MBS-4629] – /cover-art page shows no collections
  • [MBS-4637] – Timeline graph won’t graph anything without an entry in statistics/view.js
  • [MBS-4640] – Clicking cover art opens box with “����” (4 U+FFFD)
  • [MBS-4642] – Thickbox CSS interferes with MB CSS
  • [MBS-4647] – Cover art page allows submitting edit with no cover art when JS is off
  • [MBS-4648] – Changing cover art type from “other” to unset causes Internal Server Error
  • [MBS-4678] – upgrade.sh is not ready for testing
  • [MBS-4679] – Internal server error adding secondary types to a release

Improvement

  • [MBS-1485] – Alias types
  • [MBS-1798] – Lyrics language for works
  • [MBS-1799] – Add ISO 639-3 language codes to the database
  • [MBS-1981] – Add blog feed to the home page
  • [MBS-2240] – Aliases: certain locale can be used only once in the list of aliases
  • [MBS-2532] – Allow more than one IPI per artist
  • [MBS-2851] – Timeline graph events should be in the database
  • [MBS-2885] – Allow more than one ISWC per work
  • [MBS-3646] – Split release group attributes into two types
  • [MBS-3788] – Alias improvements
  • [MBS-4625] – Improve wording of cover art tab when cover art comes from relationships
  • [MBS-4676] – Do not allow people entering deprecated relationships

New Feature

  • [MBS-842] – Allow vinyl style track numbers and sides
  • [MBS-1385] – Support unknown end dates
  • [MBS-3704] – Allow adding sort names to artist aliases
  • [MBS-4337] – Make user profile more social: add (optional) fields avatar, gender, birth year, country

Server update, 2012-04-30

We’ve released another set of bug fixes and improvements for the server. Thanks to Joachim LeBlanc, Johannes Weißl and the rest of the MusicBrainz team for helping on this release!

Bug

  • [MBS-2553] – Change wording for no External Links in the sidebar
  • [MBS-4041] – Nonexistent elections cause Internal Server Error
  • [MBS-4544] – Predefined advanced searches on /search no longer work
  • [MBS-4547] – URL cleanup broken on URL edit page
  • [MBS-4563] – Plugin::Diff is broken
  • [MBS-4564] – robots.txt served as octet-stream on test.mb
  • [MBS-4566] – Advanced Search Syntax Link missing from Search
  • [MBS-4573] – Show a message when a label has no releases
  • [MBS-4597] – Merging can fail if entities have aliases with the same locale
  • [MBS-4609] – Comment element on release group in /ws/2 should be removed

Improvement

  • [MBS-1167] – "Read more" for annotation previews should load the full annotation in place, not take you to a different page
  • [MBS-1881] – Move relationship types/attributes/instruments lists
  • [MBS-2479] – Mark approvals differently from normal Yes votes
  • [MBS-2814] – Reports don’t highlight entities with pending edits
  • [MBS-3684] – Improve sorting of the "Releases with superfluous data tracks" report
  • [MBS-4038] – Let users edit their permissions on test servers
  • [MBS-4063] – Exclude stuff marked Single AND UK from the SeparateDiscs report
  • [MBS-4125] – Sort report for creative commons download relationships
  • [MBS-4476] – Provide indication of non-front cover art
  • [MBS-4493] – ISRCs with multiple recordings: Do something about the DEF05 ISRCs
  • [MBS-4530] – Remove the recordings with CC download relationships report
  • [MBS-4560] – "No votes" failed edit should not fail in silence
  • [MBS-4572] – Open up /artist/ in robots.txt

New Feature

  • [MBS-4551] – Display which git branch is active
  • [MBS-4567] – Use markdown for README and INSTALL
  • [MBS-4576] – Allow for google analytics support
  • [MBS-4590] – Use /doc/About instead of /doc/About_MusicBrainz
  • We’ve also improved cover art support slightly.

Sub-task

  • [MBS-4160] – SoundCloud relationship under the External links section

Server update, 2012-04-10

Sorry for being a week behind on this release, but we’ve just finished pushing out another set of changes. Many thanks to Lukáš Lalinský, Paul Taylor and the rest of the MusicBrainz team for making this release happen! Here’s what we’ve just released:

Bug

  • [MBS-3619] – Statistics page doesn’t validate
  • [MBS-3794] – A "no votes" edits got wrong status "failed vote"
  • [MBS-3834] – Hovering over artist names in "edit medium" tracklist changes does not show the artist sort name
  • [MBS-4082] – Titles including quote marks are truncated when adding from a CD Stub
  • [MBS-4131] – Edits show credited-as name when it’s the same as the artist name
  • [MBS-4183] – Release editor: enters new artist if no radio button selected on ‘add missing entities’ tab
  • [MBS-4314] – Add interface elements to reorder tracks on a tracklist without having to edit the tracknumbers.
  • [MBS-4367] – Current search URLs are broken on the test server
  • [MBS-4418] – beta.mb is unresponsive
  • [MBS-4467] – ws/2/label doesn’t include label comment
  • [MBS-4484] – Limited user can vote on edits
  • [MBS-4500] – ISE: ‘Can’t call method "is_auto_editor"’ when attaching a TOC
  • [MBS-4504] – Set default unknown value for medium.format_name

Improvement

  • [MBS-684] – TOC lookup displays too little release info
  • [MBS-834] – Weird behaviour of the search checkboxes
  • [MBS-1728] – Make ModBot’s edit notes grey again
  • [MBS-1764] – Inconsistent display for merge edits
  • [MBS-2242] – Disable editing of Medium title when there’s only one medium
  • [MBS-2412] – User tags should be sorted
  • [MBS-2867] – RE: "Add n track(s)" should reset to 1
  • [MBS-3171] – Overview and Recording pages need to be able to exclude featured guest spots from display
  • [MBS-3208] – Trim leading/trailing whitespace in release editor prior to Add Missing Entities check
  • [MBS-3398] – Detect "M" in front of track numbers
  • [MBS-3482] – "YouTube Relatiionship Type" needs autofix and doc
  • [MBS-3628] – Prevent people from adding broken Facebook URLs
  • [MBS-4234] – Move Basic search logic from mbserver to searchserver
  • [MBS-4390] – Display disambiguation comments better in tooltips
  • [MBS-4490] – "CD N" in Add disc from existing tracklist is confusing
  • [MBS-4523] – Statistics: Use commas and right-align numbers
  • [MBS-4535] – Make visited links more prominent

New Feature

  • [MBS-3160] – Add view to artist pages that shows release groups/recordings/etc credited to that artist only (solo releases only – exclude collaborations, feat., etc)
  • [MBS-3266] – Allow filtering by artist credit

The Git tag for this release is v-2012-04-10-ngs-bug-fixes.

Search server release: 2012-03-23

Earlier today we switched the latest release of our search server live. Paul says:

In this new release of the search server the basic search will now search more fields and return variations of what you searched such as spelling mistakes and the scoring has also been improved. Advanced search now lets you search for when a field doesn’t exist and you can do exact searches that take accent characters into account using the new accent fields.

We’ve also upgraded to the latest version of Lucene, added a way to see how scores are calculated for results and added integration tests. Hopefully this will all help with future stability.

UPDATE: This code lives in svn and can be found at r13480. I’ve also created this tag.

Thanks for your hard work on this release, Paul. And also thanks to Ollie for taking time to push the server portions of this live today. Here are our release notes:

Bug

  • [SEARCH-168] – Recording Search doesnt consider track duration only recoridng duration
  • [SEARCH-191] – Duplicate qdur values being added to document

Improvement

  • [SEARCH-152] – Update Code to use Lucene 3.5
  • [SEARCH-160] – Searching artist by name and initial(s) is impossible, the initials are useless
  • [SEARCH-163] – Search for exact string, with no accents
  • [SEARCH-166] – Artist search does not always offer the obvious (to a human!) results
  • [SEARCH-170] – Move Basic search logic from mbserver to searchserver
  • [SEARCH-171] – Basic Search should work more like Solr Dismax Parser
  • [SEARCH-173] – Index Catalogos without space
  • [SEARCH-174] – Provide a way to search for empty/non-empty fields
  • [SEARCH-177] – Support for initialize index remotely http://localhost:8080/?init=mmap
  • [SEARCH-188] – Add Integration Tests

New Feature

  • [SEARCH-172] – Make it possible to search for barcode:"[none]"
  • [SEARCH-194] – Add debugging option to be able to get lucene explain for results

Server update 2012-03-19

We’ve just finished pushing out a new server update, that addresses a few minor bugs. We initially planned to roll these changes in with the release of our Cover Art Archive, but sadly this project needs a bit longer. Many thanks to Johannes Weißl, Wieland Hoffman, Ian McEwan and the MusicBrainz team for their work on this release.

Bug

  • [MBS-3897] – Missing space in template for ARs w/disambig
  • [MBS-4291] – Default install looks for compiled js/css.
  • [MBS-4368] – Release editor: track parser is broken
  • [MBS-4383] – Statistics page is showing "Active Last Week" stats twice, leaving out "Edited Last Week"
  • [MBS-4391] – Track parser broken
  • [MBS-4399] – Medium titles not displayed when attaching a discid to a medium
  • [MBS-4403] – DBD::Pg::st execute failed: ERROR: new row for relation "recording" violates check constraint "recording_length_check" at lib/Sql.pm line 396
  • [MBS-4404] – /ws/1 uses the wrong language code
  • [MBS-4407] – Edit artist alias should fail if edited alias is deleted

Improvement

  • [MBS-1363] – Add threading headers to mod note emails
  • [MBS-1473] – Report: unlikely language-script combinations
  • [MBS-3313] – New/old edits wording in edit search is misleading
  • [MBS-4044] – Cleanup Discogs URLs
  • [MBS-4284] – Normalize URL encoding for specific sites

A Git tag will be pushed out later. The commit sha is 03343aa460d1.

Server update 2012-02-27

Sorry for being a whole week behind the scheduled update time! However, we’ve got a huge release here, which will hopefully make up for the delay!

Many thanks to Calvin Walton, Johannes Weißl, Ian McEwen, Michael Wiencek and the MusicBrainz developers for all their work on this release.

Bug

  • [MBS-1818] – UX issues with "Add Disc" feature
  • [MBS-2363] – Release editor: Release Duplicates tab doesn’t keep selection…
  • [MBS-2889] – Advanced tracklist edit: artist field is disabled by default for multi-disc VA releases
  • [MBS-2967] – Newly created artist not displayed in edit notes diff when changing release artist
  • [MBS-3027] – License attribute isn’t hidden when the relationship type is automatically changed
  • [MBS-3290] – User:Whatever links in /doc/ should point to wiki
  • [MBS-3659] – Can’t call method "statistic_names" on an undefined value
  • [MBS-3776] – YouTube Streaming autoselect doesn’t load the attribute
  • [MBS-4010] – Duplicate artist credits
  • [MBS-4043] – URL relationship type description not updating
  • [MBS-4133] – Internal server error editing recording
  • [MBS-4141] – "Add missing entities" tab breaks if you have to add a new label and have an existing artist
  • [MBS-4213] – Internal Server Error when adding a new release (Validation failed for ‘Maybe[Int]’ with value 22439.2)
  • [MBS-4269] – Wrong expansion for youtu.be URLs
  • [MBS-4280] – Translation.pm doesn’t check that $msgid is defined before regexing it
  • [MBS-4285] – Empty title returned instead of "0"
  • [MBS-4287] – Artist Country condition doesn’t load on edit search
  • [MBS-4288] – Recording tab in the RE no longer shows track ACs
  • [MBS-4290] – Regression on javascript inline search’s keyboard events
  • [MBS-4292] – License problem with excanvas.js
  • [MBS-4293] – "Get wiki versions" on transclusion table does nothing
  • [MBS-4295] – FreeDB importing chokes on double quotes
  • [MBS-4297] – Amazon referrer stuff not included in sidebar links
  • [MBS-4311] – Sort unlinked pseudo-releases report
  • [MBS-4333] – Release editor > Importing from CDStubs is broken
  • [MBS-4351] – URL Relations with ampersands in the URL aren’t being escaped in XML returned by the WebService API
  • [MBS-4352] – JS URL cleanup for recording licenses

Improvement

  • [MBS-942] – Link back to editor’s collections on the collection page
  • [MBS-1389] – Accept "X.XX" track times
  • [MBS-1435] – Show ISRC / ISWC in normal search
  • [MBS-1657] – "Change vote + add note" in edit page
  • [MBS-1961] – Display more information on "merge work" edits
  • [MBS-2086] – Aliases: separate localised aliases from other aliases
  • [MBS-2120] – Track parser needs to understand – (en dash) as an artist/title separator
  • [MBS-2392] – Track artist {credits} not shown on CD lookup
  • [MBS-2573] – Make add alias default to the current name
  • [MBS-2808] – Annotation wiki formatting syntax help is missing
  • [MBS-2957] – Hide duplicate labels and cat#s in recording / RG views
  • [MBS-3371] – Report: releases with non-consecutive medium numbers
  • [MBS-3398] – Detect "M" in front of track numbers
  • [MBS-3737] – Change the colour of the artist names on the advanced tracklist
  • [MBS-4044] – Cleanup Discogs URLs
  • [MBS-4135] – Display collection owner in release page’s collection tab
  • [MBS-4262] – gzipping of dynamic content could use extension to javascript and JSON
  • [MBS-4332] – "Add medium" edit display: Release and position fields should be at top rather than at bottom

New Feature

  • [MBS-3938] – Add a report to show artists that may have disambiguation strings in the artist name
  • [MBS-4081] – Report: pseudo-releases without transl*tion relationships
  • [MBS-4362] – Report: Recordings with Creative Commons Download relationships

Task

  • [MBS-2836] – Trim annotations in database
  • [MBS-3416] – Automatically delete release annotations that consist only of white space

A Git tag should be live later, but the commit is b5f45992a24bf.

Server Update, 2012-02-06

We just finished pushing out an update last night, and sadly it didn’t go particularly smoothly. As part of ongoing efforts to try and fix the release editor, we temporarily made everything worse and ended up losing quite a lot of session data. I’m profusely sorry that this didn’t go better! We think things are stable again now though, so if you do see any bugs that you haven’t encountered before, please do make sure to let us know over at the bug tracker. Other than that, there were quite a few internal changes to our production setup in this release – we moved to a more modern Plack based setup and used our recent Carton packaging work. Here are the other bugs that got fixed. Thanks to Ian McEwan and Michael Wiencek for helping make this release happen!

Bug

  • [MBS-2112] – Using space as join word between two artists in a credit ends up with no space between them
  • [MBS-3677] – Recording page doesn’t highlight releases with pending edits or show disambiguation comments
  • [MBS-3949] – Lookup box not working on work-URL form
  • [MBS-4248] – Timeline graph doesn’t work due to poorly-escaped strings in JS
  • [MBS-3680] – Error 502 Bad Gateway loading certain releases from the webservice

Improvement

  • [MBS-343] – Lookup by MBID in inline searches
  • [MBS-4273] – WS should use ETags in order to save clients bandwidth (and us too)

New Feature

  • [MBS-4210] – Use separate robots.txt for staging and production servers

Task

  • [MBS-4189] – Display language zxx ("No linguistic content") in release editor form