Category: Development

2012-10-15 migration on postgres 8.4 is broken

Our users have reported a bug where migrating a slave DB on postgres 8.4 results in this error:

ERROR: syntax error at or near “ORDER”
LINE 6: SELECT array_agg(artist_credit ORDER BY artist_credit AS…

If you are 8.4 please wait on your migration — we’re hoping to have a workaround ready today. If you need to get this fixed sooner, moving to postgres 9.x or importing the 2012-10-17 data dump would solve this problem.

Sorry for the troubles!

Updating MusicBrainz slave instances for 2012-10-15

If you have a replicated instance of MusicBrainz, please follow these instructions to get your server running on the new schema:

  1. Take down the web server running MusicBrainz, if you’re running a web server.
  2. Turn off cron jobs if you are automatically updating the database via cron jobs.
  3. Make sure your REPLICATION_TYPE setting is RT_SLAVE
  4. Switch to the new code with git fetch origin followed by git checkout v-2012-10-15-schema-change
  5. Run carton install --deployment to install any new perl modules.
  6. Run carton exec -- ./upgrade.sh from the top of the source directory.
  7. Set DB_SCHEMA_SEQUENCE to 16 in lib/DBDefs.pm
  8. Turn cron jobs back on, if needed.
  9. Restart the MusicBrainz web server, if needed.

This upgrade requires quite a bit of disk-space to execute; your slave may run into trouble if there is less than 10Gb of disk space free. If you’re on a disk space constrained machine, you may want to consider re-importing the data rather than upgrading in place. The next data dump should be available in about 14-16 hours from now.

Schema change release 2012-10-15

We just finished the latest schema change release. Please stay tuned for instructions on how to upgrade your MusicBrainz slave instances.

This release features work from Nikki, Navap, Bitmap, Freso and the MusicBrainz developers. Thanks for your work everyone! The following changes were included in this release:

Bug

  • [MBS-240] – Attached files not downloading properly in transcluded page
  • [MBS-3610] – Ratings : 502 Bad Gateway
  • [MBS-3853] – Annotation history page allows the same version to be selected for both Old and New
  • [MBS-3963] – Edit search "artist in my subscription" should hide the artist input field
  • [MBS-4033] – Offered and installed search plugin name mismatch
  • [MBS-4206] – Whitespace isn’t removed from the last join phrase
  • [MBS-4220] – Artists shown to last 0 days if begin year = end year
  • [MBS-4268] – "{age} ago" does not work for all languages
  • [MBS-4747] – Wrong diff or wrong edit allowed for an URL
  • [MBS-4812] – Empty "Remove release label" edit was entered
  • [MBS-4929] – Some fields on user’s "Open edits" page are duplicating while merging works
  • [MBS-4978] – Release merges with strategy "Merge mediums and recordings" don’t show up on the affected recordings’ open edits list.
  • [MBS-5047] – Changing secondary RG types is an autoedit for all users
  • [MBS-5051] – Empty artists with pending edits still get the « will be removed automatically » warning
  • [MBS-5090] – Some entities are marked as having pending edits when they don’t
  • [MBS-5177] – Editors can edit after removing their email address
  • [MBS-5253] – Table release_group_secondary_type_join misses a created field
  • [MBS-5264] – Track Parser results in edits changing durations on tracks in mediums with discids in some cases
  • [MBS-5306] – Internal server error when searching for edits by artist country and relationship type
  • [MBS-5321] – Some relationship editor tests are failing.
  • [MBS-5329] – Relationship editor breaks when using translations
  • [MBS-5330] – Medium formats untranslated on release pages and in release editor
  • [MBS-5331] – Edit note label is cut off when translated
  • [MBS-5332] – Last updated timestamp is missing on some details pages
  • [MBS-5333] – cover art types translate incorrectly
  • [MBS-5336] – Relationships get double-translated
  • [MBS-5342] – .pot files are generated with the ‘perl-format’ po flag when they are not being passed to sprintf
  • [MBS-5344] – Edits display page, for an add cover art edit, for a removed release, has invalid link
  • [MBS-5346] – TuneCore blocking blocks valid ISRCs
  • [MBS-5357] – Translations creeping into edit info
  • [MBS-5368] – Changing the link type in the relationship editor causes the dialog to jump
  • [MBS-5372] – AR editor: paste should detect entity type
  • [MBS-5377] – MediaWiki upgrade broke WikiDoc redirects
  • [MBS-5382] – AR editor: "part of" ARs are displayed backwards
  • [MBS-5384] – Relationship editor: track artists only shown when AC has different number of artists
  • [MBS-5385] – Relationship editor shouldn’t merge relationships with different dates
  • [MBS-5390] – Relationship editor isn’t loading related works for some recordings
  • [MBS-5391] – Relationship editor doesn’t support the "ended" checkbox
  • [MBS-5392] – Relationship editor should display dates consistent with the rest of the site
  • [MBS-5396] – relationship editor changes to MB.utility.keys cause it to die on null
  • [MBS-5406] – "Subscriptions" editor tab shows 3 empty <li> inside <ul> between « jesus2099 is subscribed to: » and p.pageselector
  • [MBS-5412] – Some statistics tabs have partially-untranslated headers
  • [MBS-5413] – e.g. "Artist-Artist" and "Artist-Artist relationships" not translating correctly on /statistics/relationships
  • [MBS-5414] – Javascript tests no longer run in older versions of Opera
  • [MBS-5415] – "Types:" (under cover art) should often be in singular
  • [MBS-5417] – WikiDocs: Prevent access to File and Image namespace
  • [MBS-5419] – Chrome frame wasn’t removed completely/properly
  • [MBS-5425] – Some ModBot edit notes end up stored translated

Improvement

  • [MBS-1226] – Address verification email should include the client’s IP address
  • [MBS-1530] – More space/sections on the add disc id page
  • [MBS-1789] – No way to compare discid track length against tracklist length
  • [MBS-2449] – Search page should say when it was last updated
  • [MBS-2492] – Don’t show quality levels in edit documentation where they don’t apply
  • [MBS-2543] – Make the dash at the end of /cdtoc/ and /cdstub/ URLs optional
  • [MBS-2650] – After adding a relationship, go to the overview tab
  • [MBS-2717] – User’s ratings page should be paginated
  • [MBS-3606] – Matched CD TOC page should have a link to the disc id
  • [MBS-3791] – Drop Description from URLs
  • [MBS-4104] – Show release merges under all track/recording artists
  • [MBS-4405] – Report: Artists that may be persons shoudn’t show obvious groups
  • [MBS-4690] – Include links to all image sizes
  • [MBS-5099] – Make release/work language clickable
  • [MBS-5236] – Add stats for work types
  • [MBS-5279] – Try to avoid duplicating links in the sidebar
  • [MBS-5285] – Add NOT NULL constraints to disambiguation & join phrase column
  • [MBS-5290] – CAA replication
  • [MBS-5350] – Add a way to check # of open edits and edits entered in the last day
  • [MBS-5352] – /doc/Edit_Types/38 and similar should link to the documentation.
  • [MBS-5362] – Add cleanup and autoselect for iTunes links
  • [MBS-5364] – Ask the user to confirm navigation away from the relationship editor (mainly for backspace)
  • [MBS-5365] – In the relationship editor, pressing enter while focused in the entity lookup should add the relationship
  • [MBS-5366] – Make it easier to reuse artists in the relationship editor
  • [MBS-5371] – Don’t show ModBot as a limited user
  • [MBS-5375] – Remove whitespace above doctype declaration
  • [MBS-5376] – Add artist credit description
  • [MBS-5404] – Improve the relationship editor’s multiselect plugin

New Feature

  • [MBS-5320] – All data dumps should be signed by MusicBrainz
  • [MBS-5324] – Relationship editor should allow using auto-editor privileges
  • [MBS-5363] – Allow adding/editing release group relationships in the relationship editor

Task

  • [MBS-5182] – Add final controlled_for_whitespace checks
  • [MBS-5289] – Finish/merge log statistic features
  • [MBS-5291] – Move existing timeline stats into its own schema
  • [MBS-5316] – Remove PGOPTIONS/search_path from ./admin/psql
  • [MBS-5354] – Remove drastically out-of-sequence cover art images
  • [MBS-5355] – Remove depedency on Function::Parameters
  • [MBS-5374] – Commit MB logo into repo

The Git commit SHA for this release is 5bd71d211dceb25a7a0d545a14cb3273e3e89f19, tag is v-2012-10-15-schema-change.

Server update, 2012-10-01

We’ve just finished pushing out an update today, which will be the last update before the schema change, which is now due in 2 weeks. Sadly, this release doesn’t feature the much anticipated relationship editor, which had to be reverted at the last minute as we still think it could do with more user testing. Hopefully it will make the next release!

This release features work from Aurélien Mino, Nicolás Tamargo, nikki, patate12, Pavan Chander, Ulrich Klauer and the MusicBrainz developers. Thanks for your work everyone!

Bug

  • [MBS-3825] – Release Editor is not displaying the release title on the tracklist tab
  • [MBS-3866] – Disambiguation comments can be the same
  • [MBS-3902] – Database allows artist type group with a gender set
  • [MBS-4614] – Country dropdown is sorted by English name
  • [MBS-4670] – "Relate to…" maybe should not add #relate_to hash to address bar URL
  • [MBS-4861] – Data loss: Changing release artist revert track numbers
  • [MBS-5019] – Release group with no name created
  • [MBS-5089] – Statistics: "unique" number of ISWCs is higher than "all" number of ISWCs
  • [MBS-5111] – Editor statistics data regularly falls out of cache, and cannot be recomputed
  • [MBS-5199] – Recording disambiguation comment is not removed when it should in RE’s recordings tab
  • [MBS-5240] – Adding an IPI code to an artist with IPI pending fails
  • [MBS-5241] – Add Medium/Add Release do not show up in recording edit histories
  • [MBS-5242] – Add Medium edits don’t always link recordings, even those that exist
  • [MBS-5250] – Non-specific dates create negative ages
  • [MBS-5264] – Track Parser results in edits changing durations on tracks in mediums with discids in some cases
  • [MBS-5295] – Webservice should distinguish between "" and NULL as a value for release->barcode.
  • [MBS-5298] – WikiDocs links are not scheme-independent
  • [MBS-5301] – gravatars do not get loaded over SSL when they should be
  • [MBS-5315] – ISE moving a discid
  • [MBS-5319] – QUnit testrunner is not compatible with current phantomjs versions.
  • [MBS-5335] – $entity is undefined in WebServiceStash.pm

Improvement

  • [MBS-1579] – Add Release editor: option to swap track title <> artist
  • [MBS-4066] – Simple RE parser speed boost
  • [MBS-4539] – Add h1 to release editor
  • [MBS-4831] – Exclude releases with pending art edits from the CAA report
  • [MBS-5152] – Allow removing a disc ID from the disc ID page
  • [MBS-5281] – Display relationships on release-group overview page
  • [MBS-5282] – language switcher should appear when number of languages > 1, not > 0
  • [MBS-5337] – Add more classes to entity pages and when displaying edits

Task

  • [MBS-2404] – Private mb dump files to be encrypted
  • [MBS-5169] – Add "Rock in China" to the "Other Databases" whitelist

The Git commit SHA for this release is ce2da4b67e39821348cd849ca1957b8e15c33f52, tag is v-2012-10-01.

Server update, 2012-09-17

We’ve just finishing pushing out another update to the MusicBrainz servers. This release is mostly a bug fix release, with a few improvements thrown in for good measure. Thanks to Alastair Porter, Aurélien Mino, Nicolás Tamargo, nikki, and the MusicBrainz developers for their hard work on this release! Here’s what has changed:

Bug

  • [MBS-3540] – Set track length edits can become too wide
  • [MBS-3757] – ‘releasegroup’ used as the annotation type when displaying search results
  • [MBS-3956] – Merging URLs can be done as auto-edit
  • [MBS-4085] – Users with open edits can delete their account
  • [MBS-4143] – Removing a tag from a well-tagged entity can give "undefined" as text
  • [MBS-4365] – "Edit Recording Associations" displays track artists for recordings
  • [MBS-4584] – Track parser doesn’t understand fullwidth full stops
  • [MBS-4599] – Artist not shown next to release name in edit release label edits
  • [MBS-4872] – Can’t directly add a release to a collection if no collections exist
  • [MBS-4946] – Deleting account does not unset new profile fields
  • [MBS-4950] – Internal server error when cancelling a recording merge instead of submitting
  • [MBS-5010] – Running a slave database without READONLY database settings produces confusing exceptions
  • [MBS-5029] – Cover art from downvoted ASIN still being displayed in the sidebar
  • [MBS-5043] – Edit search forgets "low" data quality selection
  • [MBS-5066] – IPI fields are not validated properly
  • [MBS-5086] – Degrade release editor gracefully if release editor session memcached is not available.
  • [MBS-5120] – Edits changing an empty cat number to a non-empty one can overwrite each other
  • [MBS-5143] – Non-translatable items on "http://musicbrainz.org/relationships"
  • [MBS-5164] – Non-translatable items on "http://musicbrainz.org/artist/<MBID>/edit_annotation"
  • [MBS-5194] – Set primary alias for locale is auto-edit
  • [MBS-5204] – & in an image comment shows up as &amp; in edits
  • [MBS-5205] – Text diff incorrectly highlights first word that didn’t change
  • [MBS-5209] – statistics/editors largely untranslatable
  • [MBS-5210] – statistics/timeline shows the menu and page footer in English
  • [MBS-5211] – (Part of) page title untranslatable for statistics/*
  • [MBS-5212] – search/edits is largely untranslatable
  • [MBS-5217] – Internal server error loading a report when reports have never been generated
  • [MBS-5224] – Historic edits are shown incorrectly when the server is translated
  • [MBS-5235] – Edit search is not correctly ordered when edits have the same open time
  • [MBS-5239] – Dropdowns on "Add/Edit Relationship" pages are untranslatable
  • [MBS-5248] – Subscriptions mails should not have In-Reply-To and References headers
  • [MBS-5252] – data dump MD5SUMS file includes temporary paths and private dump
  • [MBS-5255] – Replication trigger for release_group_secondary_type_join is misnamed and duplicates trigger for release_group_secondary_type
  • [MBS-5257] – Split artist is stuck and doesn’t get removed
  • [MBS-5258] – DropReplicationTriggers.sql not in sync with CreateReplicationTriggers.sql
  • [MBS-5259] – Track Parser "detect track durations" is broken on releases with DiscIDs, appending lengths to Artist
  • [MBS-5270] – scripts complain about LWP not being loaded
  • [MBS-5271] – gravatar hash should be made of lower cased smtp address

Improvement

  • [MBS-169] – Show number of subscriptions on subscriptions page
  • [MBS-3360] – Remove unused tags and garbage collect tags on retagging
  • [MBS-3878] – Number of votes to fail normal and low quality data should be the same
  • [MBS-4465] – Cover Art Archive > Ability to navigate through zoomed cover arts
  • [MBS-4638] – Show more information on the reorder cover art page
  • [MBS-4779] – http://musicbrainz.org/iswc/ISWC url should accept various ISWC formatting
  • [MBS-5251] – URL statistics
  • [MBS-5263] – Cover art from sidebar should not be included in the set of images when you’re navigating through zoomed cover arts
  • [MBS-5267] – Make fallback to language detection via browser headers optional
  • [MBS-5273] – Normalise mobile Wikipedia URLs

New Feature

  • [MBS-4825] – Normalize blogspot URLs

Task

  • [MBS-5008] – Remove voting differences for data quality
  • [MBS-5249] – Update replication triggers verbosity for the search server updater module

Sub-task

  • [MBS-3135] – Predefined edit URL should be redirect/converted to standard edit searches

The Git commit SHA for this release is 8060824492, git tag v-2012-09-17.

Schema change release: Oct 15th

On October 15th, we’re going to update our schema once again. This time we’re only making minor changes and some cleanup from the last schema change. To find out exactly what will be changing, please take a look at our milestone for this release.

At this point we’ve frozen the list of tickets we’re going to adress for October 15th. We’re not going to accept any more schema change tickets for this release.

Please let us know if you have any questions!

Server update, 2012-09-03

Another 2 weeks have flown by, so we have another release for you! This one is a combination of small bug fixes and improvements, and includes work from Joachim LeBlanc, Michael Wiencek, nikki, and the rest of the MusicBrainz team. Thanks to everyone for their hardwork. Here’s what’s new:

Bug

  • [MBS-2301] – Attach TOC to new release – TOC/DiscId is lost when based on existing release/tracklist
  • [MBS-3541] – /ws/1 artist lookup returns odd release-group-list and omits releases from release-list
  • [MBS-3613] – CD Stub "Overview" and "edit" tabs are dead links when adding a new CD Stub
  • [MBS-3678] – Blank annotations are shown in summary if they contained text prior
  • [MBS-3955] – Editing a user doesn’t allow removing auto-editor privileges
  • [MBS-4074] – MBImport.pl: use head -c to be more portable
  • [MBS-4238] – Edit history: missing history?
  • [MBS-4393] – Recordings search does not find (some?) Chinese titles
  • [MBS-4574] – [removed] displayed while adding a label to a release
  • [MBS-4585] – Artist not found in inline search
  • [MBS-4792] – regression: /ws/2/release queries with inc=recordings+recording-level-rels+work-rels no longer includes iswcs on works
  • [MBS-4797] – DropFunctions.sql cannot be executed
  • [MBS-4804] – Various artist release groups aren’t correctly grouped by release group secondary type
  • [MBS-4844] – /ws/2/release?inc=user-tags+recordings only returns tags for a single recording
  • [MBS-4854] – Querying /ws/2/release with ?inc=tags+recordings+recording-level-rels+work-rels does not return tags on works
  • [MBS-4942] – If blog.musicbrainz.org is inaccessible, an internal server error is printed on the musicbrainz.org front page.
  • [MBS-4955] – Editor search does not load Gravatars properly
  • [MBS-4972] – "Add new artist" on the add missing entities tab should have a label
  • [MBS-4992] – Work details tab is missing a details header
  • [MBS-5018] – Some translations break front page styling
  • [MBS-5090] – Some entities are marked as having pending edits when they don’t
  • [MBS-5110] – Inserts into artist_tag_raw are not checking for existence first
  • [MBS-5119] – work-recording vs. work-recordings add relationship pages inconsistent on labels around inputs
  • [MBS-5137] – Amazon artist URLs are linked to the wrong page
  • [MBS-5138] – Amazon referral links are not always correctly generated
  • [MBS-5141] – Subscribed editors page shows "found edits" when not subscribed to any editors
  • [MBS-5151] – Edit count is wrong on /edit/open
  • [MBS-5154] – Country, Label, Catalog Number fields aren’t populated on the DiscID move page.
  • [MBS-5163] – Release editor : label search can’t find what both indexed and direct searches can
  • [MBS-5183] – Internal server error when adding an entity with double spaces
  • [MBS-5184] – malformed UTF-8 character in JSON string
  • [MBS-5187] – Cannot add release
  • [MBS-5188] – InitDb fails with a postgres beta
  • [MBS-5189] – CAA changes to InitDB and MBImport don’t work
  • [MBS-5190] – Adding track times to a release triggers selecting new recordings for all tracks
  • [MBS-5191] – Typo in error when showing an edit and cannot display cover art
  • [MBS-5196] – CSS: Centreing an image doesn’t work
  • [MBS-5197] – ISE looking at certain edit history
  • [MBS-5215] – ModBot SetTrackLengths script is corrupting track numbers

Improvement

  • [MBS-1326] – Put limits to the Big Red Cell in edits
  • [MBS-1848] – Add guess case for adding/editing alias
  • [MBS-2597] – Show number of releases in each group on "merge release groups" edit page
  • [MBS-2803] – Show release group type when merging
  • [MBS-3119] – Display more information on the merge work page
  • [MBS-3476] – Add more string substitutions for "join phrase"
  • [MBS-4354] – Edits through the webservice should display application name + version
  • [MBS-4382] – Display total medium/release duration
  • [MBS-4533] – Group releases on recording page by status
  • [MBS-4659] – Auto-editor approval should be shown in edit list view.
  • [MBS-4885] – Go back to the "Instrument Tree" tab after creating an instrument
  • [MBS-4979] – inline search: results aren’t as useful as regular search.
  • [MBS-5098] – Create Artist edits should show Born/Founded/Began (and equivalents for end date) rather than the generic names
  • [MBS-5121] – TuneCore song IDs are permitted in ISRC fields
  • [MBS-5135] – Display more information on the release merge page
  • [MBS-5170] – in the edit summary, there is "begin date"/"end date" reglardless of whether it’s "person", "group" or "other"
  • [MBS-5178] – statistics-collected nagios check should show when stats were collected
  • [MBS-5208] – Wrong grammar on open edit button.

Task

  • [MBS-5122] – Use scheme relative links to support https
  • [MBS-5124] – Ensure search server queries correctly time out
  • [MBS-5140] – Purge unused update scripts
  • [MBS-5186] – Open up release groups and labels in robots.txt

Sub-task

  • [MBS-5149] – Code Coverage Review: MB::S::Validation
  • [MBS-5153] – Code Coverage Review: MB::S::Controller::Statistics
  • [MBS-5160] – Code Coverage Review: MB::S::Controller::Browse
  • [MBS-5161] – Code Coverage Review: MB::S::Controller::Role::Load
  • [MBS-5198] – Code Coverage Review: MB::S::Data::Statistics::ByDate/ByName

The Git commit for this release is bd949ec726fa9d; Git tag v-2012-09-03.