Search Server Update, 2011-09-28

On Wednesday we rolled out an upgrade of our indexed search servers. These search servers are used for search in both the web service and website. This release includes some fixes and improvements added to make it easier to handle some particular search cases, plus few other fixes. A fairly critical bug with respect to work searching was came up yesterday after this update, but we’ve released a hot-fix patch to solve that problem. Here are the release notes:

Bug

  • [SEARCH-82] – Relation Type in Work relation list is reported as {xxxx}{yyyy}zzz
  • [SEARCH-104] – Querying "code:XXXXX" in a label search returns no results.
  • [SEARCH-106] – target-type attribute missing in relation-list elements
  • [SEARCH-110] – Web service is returning ‘disambiguation’ element in release groups
  • [SEARCH-112] – Label codes are indexed as strings
  • [SEARCH-113] – Label aliases don’t produce the expected results
  • [SEARCH-116] – Freedb import chokes on entries with multiple ids
  • [SEARCH-121] – Recording search only searches on recording artists not track artists if different

Improvement

  • [SEARCH-114] – Index unset artist countries as unknown
  • [SEARCH-115] – Allow advanced search for releases with zero mediums

New Feature

  • [SEARCH-90] – Search for standalone recordings

NGS Bug fixes 2011-09-21 and the New "Split Artist" Feature

Apologies for being a few days late here, but we’ve just finished updating the servers with the latest set of bug fixes and new features. This release brings a new feature that people have been asking for since NGS – the ability to “split” an artist into artist credits.

Split Artist

With NGS, we had the ability to link multiple artists to recordings/releases/etc. While our migration script handled most cases, there are some cases when it was unclear how to automatically split artists, and we opted to leave that artist unchanged. While it was possible to manually edit references to this artist, it was painful. We now have a convenient way to make this process easier. In the sidebar of artists there is now a “split into separate artists” link:

The new "split into separate artists" feature
The new "split into separate artists" feature

Clicking this link will present you with a form where you can choose the new set of artists. For example, here’s me splitting the single “Alix Perez & SpetraSoul” artist into the correct separate artists:

The "split artists" form
The "split artists" form

As with the other types of editing in MusicBrainz, this edit has to be voted on by other editors before it is applied.

The split artists edit type can only be used on artists that have no relationships, other than “collaboration” relationships. If the artist does have collaboration relationships, edits to remove all collaboration relationships will be entered at the same time. Note also that this will not remove the now empty artist entirely – that will happen when ModBot does its daily cleanup.

Bugs

We’ve continued working through the bugs as well, and here are the tickets that have been closed in this release.

  • [MBS-1133] – Don’t try to delete relationship types that are still in use
  • [MBS-1343] – Add AR instrument search function has problems with accented letters
  • [MBS-1380] – An URL ending in closing parenthesis gets broken when listed in edit notes
  • [MBS-1396] – Allow for URLs longer than 255 chars
  • [MBS-1411] – Don’t lowercase relationship attributes
  • [MBS-1520] – Tag lookup page has odd search results
  • [MBS-1879] – Remove the non-functional /search/plugins page
  • [MBS-1934] – Inconsistency: “see other versions of this release” vs. “see release group”
  • [MBS-1936] – Historic MusicBrainz::Server::Edit::Historic::EditArtist incorrect: doesn’t include artist names, always use “[removed]”
  • [MBS-1950] – Artist Sortname is shown instead of Artist Name in historic edits
  • [MBS-2047] – Email sender addresses have regressed
  • [MBS-2195] – Don’t allow submitting relationships where the relationship type has no description
  • [MBS-2335] – Other Lookup ISWC search should strip spaces
  • [MBS-2495] – Auto-edit status in edit documentation is misleading
  • [MBS-2529] – Orphaned release group and recordings after rejecting or cancelling a release/medium
  • [MBS-2911] – URL decoding should not decode non-utf-8 strings
  • [MBS-2934] – “Change release group” edit doesn’t appear in artist’s edits
  • [MBS-2944] – Vote on all edits control should only be shown to auto-editors
  • [MBS-3226] – Multi-artist credits with no artist selected produce very confusing behavior
  • [MBS-3227] – Lookup TOC page incomplete for logged out users.
  • [MBS-3269] – Track #s get automatically added to track titles every time the tracklist tab is opened
  • [MBS-3274] – Editing track names using with the basic editor removes track artists and disassociates recordings
  • [MBS-3278] – Warning about using various artists as track artist flashes on screen
  • [MBS-3347] – When trying to add cover art from disallowed sites, no error is shown
  • [MBS-3351] – ws/2 stacktrace browsing release by label
  • [MBS-3358] – “Open Edits” and “All Edits” tabs not marked as open
  • [MBS-3373] – URL cleanup stuff doesn’t always run in time
  • [MBS-3381] – Orphaned release groups/recordings created by release editor should be automatically removed
  • [MBS-3391] – Don’t allow to enter “performs instrument” without instrument
  • [MBS-3399] – No edit links for relationships in the appearances section on the artist relationships tab
  • [MBS-3402] – Do not allow subscriptions to “Various Artists” or “Deleted Artist”
  • [MBS-3403] – “Change release group” edit doesn’t show up in edit history / open edits of target release group
  • [MBS-3405] – Remove leading / trailing / double spaces in titles
  • [MBS-3434] – High DQ Release allows Auto-edits + incorrect edit conditions for merge
  • [MBS-3436] – “Wiki Version” column on transclusion page shows ERROR
  • [MBS-3443] – Inconsistency in sorting of “Edits by subscribed editors” ≠ “Edits for subscribed entities”
  • [MBS-3445] – Cannot search edits by editor name
  • [MBS-3453] – “Related works” on recordings page shows same work and AR set twice if there are two work<->recording ARs for the same work/recording pair
  • [MBS-3464] – Do not redirect www.musicbrainz.org to musicbrainz.org for /ws requests

Task

  • [MBS-1914] – Run script to remove “later/earlier release of” ARs

Improvement

  • [MBS-2669] – Filter edits by language
  • [MBS-2670] – Filter edits by artist country

The Git tag for this release is v-2011-09-21-ngs-bug-fixes. Happy editing!

NGS Bug fixes, 2011-08-22

Here’s what we got fixed in the last week!

Bug

  • [MBS-1928] – Missing artist roles in works page
  • [MBS-2035] – Cancelling/rejecting a medium will leave release in an unkown state
  • [MBS-2600] – Spurious "artist credit field is required" errors with recording edits
  • [MBS-2756] – ‘Guess Case’ too aggressive on word ‘Trance’
  • [MBS-2881] – Editing CD Baby URLs behaves weirdly
  • [MBS-2973] – Edit Medium can re-associate recordings behind your back
  • [MBS-2976] – Edit tracklist edit doesn’t link to new recordings
  • [MBS-3002] – Edit search crashes if any arguments are removed
  • [MBS-3009] – View transclusion edit history link is broken
  • [MBS-3012] – Exception trying to submit a decimal rating
  • [MBS-3017] – Edit failed dependency by MotBot for no obvious reason
  • [MBS-3071] – Edit search crashes with no search conditions
  • [MBS-3141] – Can’t select criterias in edit search
  • [MBS-3199] – Changing an artist on a tracklist to an artist with the same (credited) name is an auto-edit
  • [MBS-3213] – When removing tracks from a release, artist credits are erroneously listed as part of the edit
  • [MBS-3214] – ModBot failed dependency because of a non-existing artist but the artist was not edited
  • [MBS-3231] – When entering a CD stub there are no checks for field length
  • [MBS-3244] – Errors submitting data via the web service to test.musicbrainz.org
  • [MBS-3245] – 500 Internal Server Error when submitting release ratings via the web service
  • [MBS-3247] – Show recording artist in Remove ISRC edits
  • [MBS-3252] – Internal Server Error in edit search (Artist is in my subscriptions / Status Open)
  • [MBS-3253] – Edit search displays the same edit multiple times
  • [MBS-3254] – Internal Server Error in edit search when "My vote" is unselected
  • [MBS-3256] – Unable to move a disc id between CDs in the same release
  • [MBS-3257] – Searching by TOC fails for mediums which have no format
  • [MBS-3258] – column width in opera too small
  • [MBS-3262] – ws/2 returning invalid xml
  • [MBS-3265] – Webservice not escaping "<<" properly

Improvement

  • [MBS-2232] – Auto-approve Add Medium and Add Release Label edits for new releases
  • [MBS-2839] – Artist work list > Better design: don’t split by "categories" (composer, lyricist, …) and indicates artist role for each artist related to work

Beta 2 release of libmusicbrainz4

Andy Hawkins has continued to make great work on the 4th version of libmusicbrainz, and has just announced the second beta release. Andy writes:

I am pleased to announce the second Beta release of libmusicbrainz4, a client library for retrieving data from the MusicBrainz service using the latest /ws/2 web service.

This is a significant rewrite of the code since Beta 1, but this should mean it is much easier to maintain going forward. Unfortunately it is not ABI compatible with Beta 1, which will mean all client applications will need to be recompiled (and are likely to require some code changes).

The main changes are in the areas of list handling, with a cleaner internal implementation of lists. Additionally, the C interface is now generated automatically from an XML description file, and there is correct handling for some XML schema changes that have been made since the first release. Finally, the ownership of items in the C interface has changed slightly, please check the documentation for each method in the C interface for full details.

Note that some include files have been removed. I would recommend that you clean out the ‘include/musicbrainz4’ directory in your install location before installing this newer version.

The library is available to download here:

ftp://ftp.musicbrainz.org/pub/musicbrainz/libmusicbrainz-4.0.0beta2.tar.gz

Documentation is generated from the source (make docs), an online copy is available here:

http://users.musicbrainz.org/~luks/docs/libmusicbrainz4/

Please also check the sample programs (under ‘examples’) for details on how to use the library.

If you find any issues, please report them using JIRA (under the project ‘libmusicbrainz’) here:

http://tickets.musicbrainz.org/

As ever, please feel free to ask any questions about the library either here or via the #musicbrainz-devel channel on IRC (when I’m around).

NGS bug fixes, 2011-08-08

Apologies for the day late release, but here’s the latest installment of bug fixes and improvements now live on the main website:

Bug

  • [MBS-1847] – Work aliases not included in XML
  • [MBS-1849] – Alias locales not included in the XML
  • [MBS-1977] – ModBot is unable to close some edits
  • [MBS-2140] – Recordings automatically changed when editing release with CD+DVD with same track names
  • [MBS-2193] – "Failed" total on a user page includes edits canceled after NGS
  • [MBS-2246] – Cannot use "Change track artists" checkbox to change all track artists to same as release artist
  • [MBS-2306] – ws2 implementation doesn’t return discids with discid queries
  • [MBS-2365] – Adding a release as Various Artist it dosen’t add ask for adding new artists
  • [MBS-2632] – Ratelimit should cover all of /ws
  • [MBS-2819] – Edit URL edits have incorrect links
  • [MBS-2886] – WebService Internal error on inc parameter
  • [MBS-3043] – Release editor allows "null" as recording id in tracklist edits
  • [MBS-3103] – Changing alias locales is an auto-edit for regular editors
  • [MBS-3115] – Server error during submission of CD Stub for which there is already a disc ID
  • [MBS-3129] – Some set track lengths edits have no lengths shown
  • [MBS-3133] – inc=release-groups+type with release lookup throws error
  • [MBS-3137] – Label country not shown on relationships page
  • [MBS-3147] – Cover art script not working properly
  • [MBS-3162] – Direct Search is unable to find artists, even when exactly matched
  • [MBS-3166] – Error messages (from RE in particular) could be more helpful for users and developers
  • [MBS-3172] – POSTs are getting cancelled for taking more than 30 seconds of time
  • [MBS-3190] – "Remove work" edit type doesn’t apply

Improvement

  • [MBS-3013] – Guess Case should capitalize some words that are usually lowercase if they are followed by punctuation.
  • [MBS-3015] – ModBot’s "Remove Work" edits should be autoedits.

New Feature

Sub-task

  • [MBS-2772] – Filter by vote cast
  • [MBS-3004] – Search for subscribed artists (has been currently dropped)

Schema change release, 2011-07-11

Today we released our first schema change update since NGS. This change is quite a radical one, as it merges both of our databases (“READWRITE” and “RAWDATA”) into a single database. For most users of the database, this probably won’t affect you, but you’re encouraged to run the upgrade process anyway. Here’s what you need to do:

  1. Take down the web server running MusicBrainz, if you’re running a web server.
  2. Turn off cron jobs if you are automatically updating the database via cron jobs.
  3. Set DB_SCHEMA_SEQUENCE to 13 in lib/DBDefs.pm
  4. Make sure your REPLICATION_TYPE setting is RT_SLAVE
  5. Switch to the new code with git fetch origin followed by git checkout v-20110711-schema-change
  6. Run ./upgrade.sh from the top of the source directory.
  7. Install the perl modules Algorithm::Merge and Algorithm::Diff
  8. Turn cron jobs back on, if needed.
  9. Restart the MusicBrainz web server, if needed.

This process may take a while, as it has to dump one database into another, and download a few extra changes to ensure slaves aren’t missing any data. The RAWDATA database should no longer be in use and you should be able to drop it, but waiting to see that everything is working well might be a good idea.

This schema change does not introduce any new data. For everyone else, here’s a list of what got fixed since the last release!

Bug

  • [MBS-1977] – ModBot is unable to close some edits
  • [MBS-1979] – Unable to edit a “later translated versions” relationship with change direction
  • [MBS-2026] – Subscribed artists open edits won’t load for editors with large amounts of subscriptions
  • [MBS-2442] – MB postgres unaccent extension overwrites unaccent.so library shipped with postgres-contrib
  • [MBS-2689] – Timeline isn’t working in Opera 10
  • [MBS-2698] – Some tables are not replicated through Live data feed
  • [MBS-2812] – Adding track times has caused a failed dependency for an edit which changed the track titles.
  • [MBS-2826] – Web service returns malformed XML (not escaped properly)
  • [MBS-2831] – “Edit medium” edit display: Artist credits changes are incorrect
  • [MBS-2974] – I don’t receive e-mail when someone votes no to my edit any more
  • [MBS-2981] – Changing the case of recording comments should be auto-edits
  • [MBS-2986] – tracklist_index was not populated during NGS
  • [MBS-2995] – Caught exception in MusicBrainz::Server::Controller::WS::2::ReleaseGroup->release_group_browse “Can’t call method “format”

Improvement

  • [MBS-1500] – Remove tracklist_index.tracks
  • [MBS-1707] – Advanced tracklist in RE: “Title” to “Disc title”
  • [MBS-2242] – Disable editing of Medium title when there’s only one medium
  • [MBS-2434] – Combine READWRITE and RAWDATA
  • [MBS-2462] – Other edit types that don’t highlight what has changed between old and new values
  • [MBS-2583] – “Edit Medium” should show you which medium is being edited with respect to the overall release
  • [MBS-2767] – Release-group XML result should include first release date

NGS bug fixes, 2011-07-05

The wheels of change are slowly starting back up again, now that we have all our employees back 🙂 Here’s what we’ve got done since the last release:

Bug

  • [MBS-1849] – Alias locales not included in the XML
  • [MBS-2223] – Length not shown in edit when adding a standalone recording
  • [MBS-2586] – Search results do not show standalone recordings
  • [MBS-2807] – Stack trace requesting /ws/2/collection when user has no collections
  • [MBS-2846] – Internal Server Error on lot of editing histories
  • [MBS-2880] – Error viewing release group/URL
  • [MBS-2882] – Merge recording edit display incorrectly once applied
  • [MBS-2898] – Search indices aren’t updating, haven’t updated in over 20 hours

Improvement

  • [MBS-777] – Make cancelling take effect sooner
  • [MBS-1508] – Should release browse request allow inc=recordings
  • [MBS-2767] – Release-group XML result should include first release date

Beta release of libmusicbrainz4 with NGS support

Andy Hawkins and Lukáš Lalinský have been doing some work on libmusicbrainz, and a beta release of version 4 is now available. Here’s what Andy has to say:

Hi all,

In conjunction with Lukas I’ve been working on a complete rewrite of libmusicbrainz to support the new Web Service v2 released alongside NGS. I’m happy to announce the first Beta release of this new library (libmusicbrainz4).

The library is written as a relatively simple parser for the results from the Web Service, with as little logic as possible. This should mean it is easy to maintain to keep in step with any changes in the schema for the XML Web Service.

The main interface is a C++ class ‘CQuery’, that returns a Metadata object containing a parsed version of the response. There is also a C interface that loosely wraps the C++ classes.

Using the library requires a good understanding of the queries and responses to and from the Web Service.

The library is available to download here:

ftp://ftp.musicbrainz.org/pub/musicbrainz/libmusicbrainz-4.0.0beta1.tar.gz

Documentation is generated from the source (make docs), an online copy is available here:

http://users.musicbrainz.org/~luks/docs/libmusicbrainz4/

If you find any issues, please report them using JIRA (under the project ‘libmusicbrainz’) here:

http://tickets.musicbrainz.org/

I will attempt to be available via this mailing list and the #musicbrainz-devel IRC channel on freenode in order to answer questions if at all possible.

NGS bug fixes for 2011-06-14

Another week, another set of bug fixes! Here’s what we’ve addressed since the last release:

Bug

  • [MBS-1231] – Age information for a person looks weird
  • [MBS-1924] – Allowed merge direction for "Merge mediums and recordings" strategy seems non-intuitive
  • [MBS-1995] – Time from TOC is calculated incorrectly
  • [MBS-2000] – CC URL relationship does not allow selection of license
  • [MBS-2049] – Docs not in the transclusion table seem to be perma-cached
  • [MBS-2062] – Menu link ‘About->Auto-editor elections’ is invalid.
  • [MBS-2169] – Label entered regardless of clearing text
  • [MBS-2261] – Release group is listed twice on an artist page
  • [MBS-2347] – Seed particular release after clicking on the recording tabs it comes back with Frequent 502 Bad Gateway
  • [MBS-2376] – Inline search results disappear when clicking on the little arrows two times
  • [MBS-2391] – You should not be able to add the same CDTOC multiple times on the same medium
  • [MBS-2399] – Problems with mediums having duplicate CD TOCs
  • [MBS-2408] – Guess Case deletes volume and part numbers
  • [MBS-2447] – "Charter Member" thing no longer shown
  • [MBS-2483] – Edit URL edit for a removed URL displays wrong
  • [MBS-2485] – no way to merge a various artist clone into the default "Various Artists" artist
  • [MBS-2529] – Orphaned release group and recordings after rejecting or cancelling a release/medium
  • [MBS-2550] – Re-add the "Subscribe to artists I add" option
  • [MBS-2552] – Allow removing a RG that has relationships
  • [MBS-2560] – Internal Server Error when removing medium from a release
  • [MBS-2644] – Internal server error while trying to add a disc ID
  • [MBS-2645] – Error in the message when trying to relate works to recordings
  • [MBS-2651] – Edit shows up in unrelated artists’ edit history
  • [MBS-2658] – Edit URL edits cannot close if the URL being edited has since been merged
  • [MBS-2659] – Edit relationship fails if the relationship has since been edited into the same relationship
  • [MBS-2705] – Adding a note to an edit which has an existing note w/o timestamp gets presented out of order
  • [MBS-2724] – IPIs don’t appear when added in Add artist edits
  • [MBS-2744] – Track removals not shown in tracklist diff

Improvement

  • [MBS-1395] – Put relationship attributes descriptions inside of an HTML "label" tag
  • [MBS-1449] – Inline Search: relate to recordings and/or works on releases
  • [MBS-1982] – Add twitter link to home page
  • [MBS-2007] – Lack of spacing between the top nav search and the dropdown to accompany it
  • [MBS-2269] – Release editor > When new disc is added, check if the last disc is empty and replace it
  • [MBS-2299] – Display one-line TOC on discId page
  • [MBS-2430] – Tagger button for standalone recordings
  • [MBS-2459] – Timeline graph needs copyable URLs
  • [MBS-2697] – Could use a link to add a VA release group
  • [MBS-2721] – Sort recordings by comment

New Feature

  • [MBS-2567] – Statistics about artist countries

Sub-task

  • [MBS-1916] – Edit release label edit doesn’t display properly