Upgrade instructions for mb_server release 20071014

If you have an existing installation of the MusicBrainz server, follow the instructions below to upgrade your installation to the latest and greatest. You can skip steps #0 and #7 if you’re not running a slave server.

Continue reading “Upgrade instructions for mb_server release 20071014”

Server updated

We just updated the main server with the latest and greatest features:

New Features

  • Lucene Search – New search functions that uses a Lucene text search engine. See the text
    search documentation
    for details on how to use this new search. However, indexes are currently only updated once a day, so the old search
    feature is still available for when you need to find something that may have changed in the last day.
  • XMLWebservice – This new XML based web service drastically improves on the old RDF based web service by being more standards compliant and
    having a much more granular control over what data is returned for each query. Read the documentation
    how to use this new service. Please note that this is in BETA and the service is still subject to change!
  • New option to “approve” edits for auto moderators (719)
  • New links “Create relationship with this artist/album/track” and “Relate to URL” to speed up the process of
    adding AdvancedRelationships (21)
  • Adding more complete release dates is now auto moderation. (759)
  • New option to automatically subscribe to artists created by you (995)
  • Numerous improvements and bugfixes to the guess case and related Javascript tools.
    (12)
  • Subscribing to an artist redirects to the artist page, rather than the (sometimes) large list of subscribed artists
    (605)
  • More “intelligent” defaults guessing of new track artists for album moves (1015)
  • For move, artist changes and some other edits: link every participating entity in the moderation pages, and add as much information as necessary
    for checking its correctness. (610, 623 and
    920)
  • Improved Disc Id move, target album is searchable and moves are allowed for VA albums (155)
  • Reports: 2 small reports added (albums with language but no script and the other way around (621),
    simplification of the MultipleTRMsPerAlbum report for faster creation.
  • ASIN ARs now update the relevant albummeta table data as well. (1039)

Fixed bugs

  • Fixed merging of albums/artists with AdvancedRelationships (916)
  • Album Moving in batch mode will now change track artists as well (972)
  • Fixed some issues in “best match” calculation for searches (985)
  • Fixed failed moderations counter in user profile (1018)
  • Fixed bugs in ASIN AR cover image URL extraction. (1061)
  • and many more we already forgot about — please see this
    report
    for all the bugs that were closed for this releases.

Thanks to everyone who helped in this release!

Server Updates

Voting “inline”, a faster voting infrastructure, send e-mails using your real e-mail address, and a small number of other changes.

Changes mainly of interest to MusicBrainz Users

Inline Moderation

(This feature by Robert Kaye)

In an effort to reduce the voting backlog, the system now shows you
one vote at a time whenever you’re on an artist, album or track page.
The moderation chosen is, in order of preference: for this artist; for one of
your subscribed artists; for an artist related to this artist; or anything
at random.  When you click one of the three voting buttons (yes, no,
abs) then the voting panel reloads to show something else to vote on.
The rest of the page remains unchanged.

Faster Voting Pages

The moderation database tables have been reorganised behind the
scenes.  The result of this is that
the voting pages (in particular)
should be much faster.  See “Moderation Table Restructuring” below
for more technical details of what’s actually changed.

Reveal My E-Mail Address

When sending an e-mail to another moderator, it’s now possible
(as long as you’ve told MusicBrainz your e-mail address) to choose
to reveal your e-mail address to the other party.  Doing so makes it
much easier for them to reply to you.  The default setting is
controlled by a new preference, which in turn
defaults to “off” (i.e. don’t reveal my address).
Read the help for more information.

Other Changes

  • When searching for moderations, you can now search for mods
    on artists related to some given artist.
  • Tied votes are now passed (previously there had to be a clear majority to
    win).
  • When sending an e-mail to another moderator, sometimes extra quotes were
    added to the subject line (so instead of the subject being Hi
    there
    for example, it would be “Hi there”).  These
    stray quotes have now been removed.
  • The HTML generated by a “Link with Icon” (for an artist, album or track
    link) now ensures that the icon will be on the same line as the text which
    follows.
  • For “Edit Artist Alias”, “Edit Artist Sortname”, “Edit Track Number” and
    “Merge Artist” moderations, if the record being changed is missing then close
    the moderation with a “Failed prerequisite” status instead of “Internal
    error”.
  • The “Show TRM” page now sorts the matched tracks by artist and album, and
    only shows each album once (even if more than one track on that album
    matches the TRM).
    Example.

New and Changed Documentation

Changes mainly of interest to MusicBrainz Programmers

If you download and import the MusicBrainz database (including the
moderation tables) from time to time, read “Moderation Table Restructuring”
(below).

Changes mainly of interest to MusicBrainz Server Programmers

Mason

(This feature by Duncan Findlay)

The MusicBrainz server now uses Mason
1.25 (previously we used 1.05).  Mason underwent something of a
redesign following 1.05, so making it past this “hurdle” should help us in
the future.

Moderation Table Restructuring

The moderation-related tables have been reorganised, so as to speed up the
voting pages.  Almost all queries against the moderation tables deal
with open moderations, but Postgres seemed to be having trouble optimising
those queries (despite an index on moderation.status).  Well, maybe we
missed something obvious, maybe not.  Anyway, what we’ve done is to
split the three tables (moderation, moderationnote and votes) into six
(moderation_open, moderation_note_open, vote_open;
and
moderation_closed, moderation_note_closed, vote_closed).  So most
queries can now just refer to moderation_open, which is a MUCH smaller table
than moderation used to be.

As a side-effect, the data for closed moderations is now dumped into a
separate file (mbdump-closedmoderation.tar.bz2), making the
mbdump-moderation.tar.bz2 file much smaller.  Hence, if you
want to download open moderation data but have no need for closed
moderations, life has just become a little easier for you.

Other Changes

The CreateTables-mysql.sql
script has been removed.  It was generally poorly maintained, and
hardly ever tested, and the costs of maintaining it outweighed the
benefits.  If you want to create a MusicBrainz database in MySQL,
start with the CreateTables.sql
script.


RDFDump.pl
has been fixed. RDF dumps now contain all artists
we have in the database.

Bugs and RFEs Closed

Dave Evans

Server Updates

Moderation Search; Add Album gets “Guess” buttons; a new “moderation suggestions” report.

Changes mainly of interest to MusicBrainz Users

Moderation Search

Previously it was only possible to list moderations using one of a number of
fixed means: for example, “New Mods”, “FreeDB Mods”, “Artist Mods”
etc.  Now, all of these have been replaced a rather flexible
Moderation Search facility.

Moderation Search allows you to find moderations according to any
combination of the following criteria: moderation status, automod flag,
moderation type, moderator, vote cast (by you or anyone else),
artist (including “my subscribed artists”), and moderation ID (range).
You can also query by album, or by track.
Finally you can choose whether to show them oldest to newest, or the other
way round.  The number of possible combinations is
simply astronomical.

The old, fixed search types (“New”, “FreeDB” etc) now simply pre-fill the
search form with the appropriate values.  So for example, you can run
one of the preset searches (say, “My Failed Mods”), and then click “Advanced
Search” to customise your current search in whatever way you please.
There’s a new page which lists
the presets.  If you like you can set a new
preference
to cause the “Moderate” link in the top navigation bar to
go to that page instead of running the “New Mods” query.

Some other points of note:

  • Many more queries than before are possible.  For example, show me
    moderations by (some moderator name) which I haven’t voted on yet; or, show
    me (some moderator name)’s automods only.
  • Many of the moderation URLs have now moved.  Hopefully all the links on
    the site have been updated, but if you do hit an old link (e.g. in an old
    “subscriptions” e-mail) you should get redirected.
  • The “subscriptions” e-mails now contain an extra link, which is
    effectively like “New Mods”, but for your subscribed artists only.
  • There’s now a page where you
    can search for moderators
    (so you can include them in your
    search).  Interestingly this is currently the only place in the whole
    site where you can effectively “browse moderators”.  No doubt a page
    dedicated to doing just that will be added some time in a future release.
  • Sometimes your moderation searches may “time out” – just like normal
    searches (for tracks, etc) the search is cancelled if it’s still running
    after 30 seconds.  If this happens, try narrowing down your search,
    maybe by using the “moderation ID” filter (e.g. only where the ID is at
    least 530000).

Add Album

The manual “Add Album” pages now have “Guess Case” and “Guess Sortname”
buttons.

Other Changes

Previously only two statistics were recorded for each moderator; a count of
“accepted moderations”, and one of “failed moderations”.
Now the “accepted” count has been split into two, one for auto-moderations,
and one for the rest; and the “failed” count has also been split into two,
one for “failed vote” moderations, and one for the other types of
failure.  For example, see your own profile page.

When moving an album to another artist, the original artist is no
longer shown in the search results (bug #576354).

Each moderation type has a name, and just to ensure that the name of each
type is unique, two of the types have been renamed.
Previously there were two types of “Add Track” (one used up to
17th August 2001, and one used thereafter); now the former is called
“Add Track (old version)”.
Additionally there are in fact two types both known as “Merge Albums” (one
for single-artist albums, and one for Various Artists).
The latter is now called “Merge Albums (Various Artists)”.

The “auto-notes” added by the Album Editor (e.g. “The first of a set
of 3 moderations”) were previously recorded against the moderator’s own
name; now, they are added by the ModBot.

If you use the “compact” album listing, then manually switch to a “full”
listing, the expand/collapse links now work, instead of switching you back
to a “compact” listing.

There is now a new moderation suggestions report, called “Superfluous Data
Tracks”.  Thanks to our script/report writer,
Matthias Friedrich, for
this.  This report lists albums which include a final “data track”,
but have no Disc ID.

Also in the are of suggestions reports: the “bad entries” report
now does HTML-escaping properly.  The “unknown” report also now does
this; it also runs much more efficiently, shows counts of how many tracks
and artists it found, and correctly includes the page footer.

Voting more than once on a moderation sometimes caused that moderation to
show up twice when it should not – this is now fixed (bug #829524).

New and Changed Documentation

Changes mainly of interest to MusicBrainz Server Programmers

Added help text to the INSTALL file and to MBImport.pl, stating that
the current live SQL scripts must be used to import a live dataset.

RDFDump – the RDF dump file (link) is now valid XML. Previously it tried to
put long lines of hyphens (“—–” etc) within a comment (bug #829181).

Bugs and RFEs Closed

Dave Evans

Server Updates

Various improvements to the tagger lookup pages, the voting system, data accuracy and all sorts of other things.

Changes mainly of interest to MusicBrainz Users

Album “Meta” Counts

The count of tracks, disc ids and TRMs held for each album (the “albummeta”
data) was often inaccurate.  Additionally, some albums didn’t have an
“albummeta” row, which actually caused them to disappear entirely from their
respective artists’ pages!  These bugs have now been fixed.

The “compact” display mode of the “show artist” page displays these counts; if
one of the counts is zero it now just leaves a blank space, instead of showing
a zero and the relevant icon.  This makes it easier to spot which albums
have no disc ids or no TRMs.

Like the “compact” display mode, these meta counts are now also shown when you
perform an album search.

Non-Album Tracks

The album name “Non-album tracks” is now changed to “[non-album tracks]”, in
keeping with the style of using “[…]” to signify special meaning within
MusicBrainz (e.g. [silence], [data track] etc).

As new N.A.T. albums are created, they (i.e. the album name itself) won’t be
added to the search indices (although the tracks within them will be).  The
existing N.A.T. album names are staying in the indices, for now, but may be
removed soon.

If two artists, both with N.A.T. albums, are merged, then the two N.A.T.
albums are merged together, keeping all the tracks.

Previously if the track numbers of non-album tracks for an artist were not
continuous (if there were “gaps”, e.g. 1, 2, 4, 5) then adding a N.A.T. would
fail.  This is now fixed.

N.A.T. albums are automatically deleted when their last track is removed.

The “show track” page no longer shows the (meaningless) track sequence number
for non-album tracks.

The Tagger

The %-relevance values shown in the lookup page now use the same more detailed
algorithm (weighted similarity calculation) that the Tagger itself
uses.  This should make the overall matching process more accurate.

When showing a list of matching tracks, the track length is highlighted
according to how well it matches the length of the track you’re trying to
tag.  Green indicates a good match, red represents a very bad match, and
blue is somewhere in the middle.  If insufficient information is
available, it’s black.

Also when showing match tracks, a “tag” link is included on that page, so you
don’t need to click through to the next page in order to tag.

The main lookup page (/taglookup.html) now moves the input focus into the form
when the page loads.

The sidebar search panel now works when used within the Tagger.

Voting

You can now vote more than once on each moderation.  Only your most recent
vote counts.  In other words, you can change your mind.  Because of this, you
must now wait until the moderation closes before you can see a list of all
votes for a moderation (on /showmod.html).  If it’s your moderation you can
always see the votes, of course.

Also on the “show moderation” page (/showmod.html) you can now see the date
and time at which each vote was cast.  If someone votes more than once (and
changes their mind), then the old, ignored votes are shown in grey.

It’s now also possible to have a “No Vote” button” – go to your preferences
page to enable it.  This makes it much easier to vote on a whole page of
moderations, but not vote on some of them (in which case, those moderations
will keep on showing up in your list of things to vote on).  This is
controlled by two new preferences: “Abstain by default” and “Show the ‘No
Vote’ option”.  Try out the four possible combinations to see how they work.

Miscellaneous

A “Search Google” link has been added to the “show album” and “show track”
pages.  These Google links now search for exact phrases, hopefully making it
easier to find what you want more quickly.  So for example if you click the
“search” link from this track, the search query used is “Alice in Chains” “Love, Hate, Love”.

Clicking on the album attributes link (which normally pops up the attributes
guide) now works without Javascript too.

“Remove Track” mods now show both the track name (with link) and the album
name (with link) – as long as the track still exists, anyway.

In the “batch operation” page (where you select what you want to do to the
albums you just tagged), if fewer than 2 albums are tagged, then we now show a
disabled “Select” button, instead of the message “Merge album not
available”.  This is more consistent with how such user interfaces
usually work.

“Guess Case” now knows to keep these abbreviations in upper case: DJ, MC, TV,
MTV.

The “top moderators” page now shows the top 25 moderators, not just the top
10.

There was a problem with anonymously adding albums; this is now believed to be
fixed.

Some moderators have proven that they can’t be trusted with the ability to
“automod” case-changes, because they make many case changes which go against
the style guide for no valid reason.  To handle this, a new “privilege” has
been created, dubbed the “Untrusted” bit.  Moderators given this bit cannot
automod anything, except for “Add TRMs”.

Incomplete home page URLs entered in moderators’ profiles are now “guessed”,
so for example http://www.example.com becomes http://www.example.com/

On the “create new login” page, a few paragraphs of help text have been added
about what we will and won’t do with your e-mail address.

You must be logged in to send a message to another moderator.

Changes mainly of interest to MusicBrainz Programmers

Some TRMs were associated with some tracks more than once.  This caused some
RDF queries to return duplicate results.  These duplicates have now been
removed, and the “insert” code has been modified to prevent them from
reappearing.

The “FindDistinctTRMID” RDF query was failing because returning TRMs as
results wasn’t properly handled.  This has now been fixed.

Changes mainly of interest to MusicBrainz Server Programmers

Mail is now generated using MIME::Lite instead of “hand-crafting” the headers
etc.  As well as making it much more likely that this is done correctly, it
also means that it will be much easier to use more advanced MIME capabilities
(e.g. HTML e-mail, attachments) in the future.

Unique database indices have been added to trmjoin (trm, track) and albumjoin
(album, track).  A primary key “id” column has been added to both
moderator_preference and moderator_subscribe_artist.

Tagger sessions are now automatically recognised by the web server, and an
“mbt=1” is written into the session data (even before the user has logged
in).  This is done by detecting the request to “/tagger/intro.html?mbt=1”,
which is the first page loaded by the Tagger, and which nothing else would
normally use.  This makes it possible to more reliably detect when the web
session is within the Tagger, and adjust accordingly (e.g. the sidebar search
panel).

Bugs and RFEs Closed

Dave Evans