Author: gavinclarkeuk

Server Updates

Server updates: Advanced Relationships, duplicate artists, artist person/group support, artist from/to date and a few minor tweaks.

(This change log includes all changes made to the server since the date
of the last release.  Some of these changes are new as of the
above date; some were introduced between the previous release and this one).

Changes mainly of interest to MusicBrainz Users

Advanced Relationships

The much awaited Advanced Relationships feature is finally complete. Advanced Relationships allows
the user to create a relationship between an Artist, Album, Track or URL. Each relationship may
have optional attributes (e.g. instrument performed, guest performance) and/or a date range
for which the links applies.

This feature is best explained with some examples:

  • Artist – Artist relationship: The Edge is a member of U2
  • Album – Artist relationship: Incendio has acoustic guitar performed by Jim Stubblefield
  • Artist – Track relationship: Symphony No. 5 was conduced by Herbert von Karajan
  • Artist – URL relationship: Morcheeba has a home page at: http://morcheeba.net/

To see a good example of what is possible with Advanced Relationships, take a look at the album Incendio.
For complete documentation on the AdvancedRelationships feature, please see the Advanced Relationships Documentation on the wiki.

Artist changes

It is now possible to enter multiple artists with the same name. If you are adding an artist with the same
(or very similar) name to an existing artist, you will be prompted to enter an artist comment. This short artist
comment is used to differentiate this artist from other similar artists and will be shown whenever
artists appear in a list and the user needs to chose the correct artist.

Artists can now also be differentiated as a Group or as a Person (via the add or edit artist pages). Artists
can also have formed/born or dissolved/died dates (for groups and persons, respectively).

Moderating and Voting now requires a verified email address

As per discussion on the mb-users mailing list, users must now provide a verified email address
in order to vote or moderate. Hopefully this will increase the communication between moderators
and reduce the number of random voters.

New Preference: “Show ‘Inline’ Moderations”

Due to problems with some new moderators randomly voting NO on moderations, we have
turned off showing random moderations in the inline moderation i-frame. The inline moderation feature
is now only available to moderators who have had their account for more than two weeks
and will be disabled by default.

If you do opt to be shown inline moderations, then they will no longer
include moderations unrelated to the page you’re on.  (Previously
the system would select a moderation for the artist you’re looking at,
or otherwise one of your subscribed artists, or otherwise a related artist,
or otherwise any artist.  Now that final part has been removed).

Miscellaneous

  • JMP3Rename and MP3Tag were added to the related projects page.
  • The completely out of date and improbable design for MusicBrainz II has been removed from the DB Structure pages. Please
    refer to the more up to date MusicBrainz database structure page.
  • The funky “A” graphic that used to indicate an automoderation has been replaced by an (automod) text in a smaller font on the left hand side.

Changes mainly of interest to MusicBrainz Administrators

InitDb.pl has been cleaned up a bit. 
You can now use it to create the database (and user, and language, etc)
but with no tables in it: ./admin/InitDb.pl --createdb --empty-database.

If you try to import the wrong tar files (where the schema sequence
of the tar file doesn’t match that of your database), then previously
MBImport.pl would decompress all of the tar files, then find the error
(and then delete all the decompressed files).  It now finds the
error much sooner, without wasting all that time decompressing the
tar files first.

The Pending/PendingData tables are now built in to the main SQL scripts;
all MusicBrainz databases (including replication masters, and replication
slaves, and ones that aren’t replicating at all) will always include these
tables.

Bugs and RFEs Closed

Dave Evans and Robert Kaye

TRM Database Pruned

It’s TRM pruning time again.

At about 1100hrs UTC on January 25th the TRM database
was “pruned” again.  Here are the pertinent facts and figures:

  • the criterion used was: select trm from trm where lookupcount > 0;
  • before the prune,
    • the TRM server contained 3540483 TRMs,
    • the MB database contained 2199812 TRMs, of which 206335 had never been looked up
  • after the prune,
    • the TRM server contained 1993316 TRMs
    • although the sigserver said “Read 1978201 total signatures from all indexes”

For pretty pictures illustrating this, check out our
MRTG pages.

See server news for details of previous pruning.

Server Updates

Server updates: quite a collection of medium to small changes, including making artist subscriptions public.

(This change log includes all changes made to the server since the date
of the last release.  Some of these changes are new as of the
above date; some were introduced between the previous release and this one).

Changes mainly of interest to MusicBrainz Users

Artist Subscriptions Made Public

So far your list of subscribed artists has always been completely
private – no-one could see which artists you’re subscribed to, and
no-one could see who is subscribed to any given artist. 
This is now changing.

A new preference has been added, called
“Allow other users to see my subscribed artists“. 
The default setting of this new preference is ON, the opposite of
the previous behaviour, meaning that your artist subscription list
will become public.  If you don’t want your subscription list to
become public, you have 30 days from the date of this release
to log in and change this preference to “off”.  (You can change the
setting at any time, but if you want to be sure that no-one ever sees
your subscription list, you’ll have to change it to “off” within the first
30 days).

From the date of this release, you can
see how many users are subscribed to any given artist, including
users who have are “privately” subscribed.  However you won’t
immediately be able to see a list of users subscribed to an artist, or
vice versa. 
From 30 days after this release,
you’ll be able to see other users’ subscription lists (if their preferences
let you), and you can see what users are subscribed to any given
artist (but only those users whose lists are “public”). 
During the first 30 days those pages will simply show an error message.

Please read these notes about this
feature if you are concerned about the privacy implications.

New Preference: “When I Vote, Mail Me Notes”

A new preference has been added: “When I vote on a moderation,
mail me all future notes for that moderation”.  Hopefully
the meaning of this is quite clear.  The setting is checked
as each note is added, so you can globally turn this
behaviour on or off for all moderations (i.e. you can’t have this
option “on” for some moderations and “off” for others).

If you are an “Automatic Moderator”

The search page will
now notify you whenever there are any elections in which you may
participate (but have not yet done so).

There is a new preference, “Enable auto-moderator privileges”. 
If you turn this setting off, you can temporarily revoke your automod
privilege from yourself, allowing you to put some moderation to the vote
where it would otherwise have been automatically applied.  To
regain your automod privilege, simply
return to your preferences page and re-enable
the option.  This setting is not saved to the database; if you log
out and in again, the option will always be reset to “on”.

When nominating another user for auto-moderator status, there is now a
“confirm” page, so it’s harder to accidentally nominate someone.

Miscellaneous

When editing a track, the allowed track number range is
shown and checked against.

The Albums with superfluous data tracks
report now only shows tracks if they are the last track on the album
in question.

The sidebar “Quick Search” now includes search for Editor
(aka user, aka moderator).

On the preferences page, most options now have HTML labels (so you can click
on the text next to the tick box, as well as the tick box itself).  The
voting buttons on the main voting pages have been re-arranged to be easier
to read.  (Changes by Eli Miller – thanks!)

The “login” page is now never shown to logged-in users.  This happened
from time to time, and was always very confusing when it did.  The HTML
for the login box has also been tidied up.  One side-effect of this is
that the odd “jumping” behaviour seen by some Mozilla / Firefox users seems
to have been fixed.

If you use the “Picard” Tagger, then the
artist “album list” page now includes “tagger” links on all the
albums.  Also, the problem whereby sometimes you would see the “Nothing
to see here, move along” page has been fixed.

If you use the “Use Javascript to move the input focus when the page loads”
preference (which is on by default), then the tagger “lookup” page now
doesn’t move the focus, but the “submit a new CD” page
does.

Each album now includes an “Edit releases” link at the bottom, next to where
the releases are shown.

Kim
Plowright has contributed search
plugins for Firefox.

The “Reveal my e-mail address” option has been removed from the moderation
note window.

The album “release editor” now checks for valid dates.

The main artist page no longer shows the “more…” link for related artists
when there are, in fact, no more related artists to show.  (patch by
Matthew Exon)

You can now search for moderations by date / time:
example one,
example two. 
Searching for moderations (when your search includes both open and closed
moderations) is faster now, after
this
work by Yary Hluchan.  Thanks Yary!

From a moderator’s “profile” page, you can now search for their
Deleted Moderations.

When editing your own profile, you can
re-send yourself a “confirm my e-mail address” message at any time,
without having to change your e-mail address to something else and
back again. 
P.S.: all user accounts which have a
stored e-mail address have now had that address validated at least once.

Various things in moderation notes are now automatically turned into
hyperlinks to moderations: e.g. “moderation #123”,
auto hyperlinks in mod notes to other mods: e.g. “moderation #123”,
“mod 245725” or “edit #36483” would all become links to their respective
pages.

New and Changed Documentation

The page about disc IDs contained a link to what
is now some rubbishy “link farm” web site; that link has therefore been
removed. 
ruaok’s bio has been updated. 
The home page contained some invalid HTML, which has now
been fixed. 

Changes mainly of interest to MusicBrainz Server Programmers

The INSTALL file has been updated; PostgreSQL 7.4 or later is required,
and the list of dependencies from CPAN has been updated. 
Also a link to the DebianServerSetup
wiki page has been added.

admin/RemoveOldSessions now also removes orphaned lock files.

admin/SetSequences.pl now correctly sets the next sequence values
on the three “*_open” tables.  (This bug would manifest itself as
duplicate key errors whenever the moderation was closed).

The URL rewriting (e.g. of /artist/GUID etc) is now implemented
via MusicBrainz::Server::Handlers, not mod_rewrite.  In doing so
it’s now possible to use the “/artist/…” URL space for other things, e.g.
HTML pages (note: this page currently gives an error).

The SQL scripts have been tidied up (sorted, layout standardised etc). 
DropViews.sql now drops the views, not creates them 🙂

Artists, users and countries are now cached using memcached. 
The country list is no longer stored in each user’s %session.

The MusicBrainz::Server::DateTime module has been added. 
The MusicBrainz::Server::DeferredUpdate module has been rewritten
somewhat, including the ability to handle deadlocks. 

You can now customise the cookie name used (the default is
AF_SID). 
We now check for, trap, and work around HTML::Mason‘s unhelpful
behaviour of turning repeated arguments into an array reference.

Extra logging has been added TRM requests and to the
TrackInfoFromTRMId query.  In the latter query,
results are limited to 100 tracks, and looking up the “silence” TRM is now
disallowed.

Bugs and RFEs Closed

Dave Evans

TRM Database Pruned

It’s TRM pruning time again.

At about 2300hrs UTC on January 25th the TRM database
was “pruned” again. 
As we did last time, we removed all TRMs apart from the ones attached to
MusicBrainz tracks, this time using the additional criterion that
the TRM had to have been looked up at least twice. 

Just before the prune we had about 3633572 TRMs; after the prune we had about 1898435. 
For pretty pictures illustrating this, check out our
MRTG pages.

See also prune 1,
prune 2
and prune 3.

TRM Database Pruned

The TRM database has been pruned again, making the system much faster and more reliable again.

At about 2200hrs UTC on December 22nd the TRM database
was “pruned” again.
As we did last time, we removed all TRMs apart from the ones attached to
MusicBrainz tracks, where the TRM had been looked up at least once. 

Just before the prune we had about 3632598 TRMs; after the prune we had about 1989632. 
For pretty pictures illustrating this, check out our
MRTG pages.

See also Prune 1 and
Prune 2.

Server Updates

“Add Disc ID” moderations, and Annotations.

Changes mainly of interest to MusicBrainz Users

“Add Disc ID” Moderations

Whenever a disc ID is added to an existing album, it is now tracked
via an “Add Disc ID” moderation.  This applies both to disc IDs added
via the “CD lookup” interface (in which case the moderation is credited to
whoever performed the lookup), and also to those added as a result of a
FreeDB lookup (which fall under the “FreeDB” moderator). 
“Add Disc ID” moderations are not used in the case where
an album and a disc ID are added at the same time.

Annotations

Annotations allow you to add notes to artists and albums. 
See How Annotations Work
and the Annotations FAQ. 
Thanks to Matthias Friedrich for building the foundations of this feature.

Bugs and RFEs Closed

Dave Evans

TRM Database Pruned

The TRM database has been pruned again, making the system much faster and more reliable again.

At about 1930hrs UTC on November 4th the TRM database
was “pruned” again (see the previous time
for more information about this). 
This time we removed all TRMs apart from the ones attached to
MusicBrainz tracks, where the TRM had been looked up at least once. 
This is a slightly more aggressive prune compared to the previous time. 

The TRM database is now about
40% of the size it was before, which (like last time) means that
it now fits easily into the server’s memory, so the server as a whole
runs quickly
and reliably.

Server Updates

Better Disc ID support, UTF-8 support for FreeDB, several tweaks to the automoderator election system and the usual miscellaneous bunch of bug fixes and other changes.

Changes mainly of interest to MusicBrainz Users

Revamped Support for Disc IDs

Duplicate disc IDs are now allowed.  As a pleasant
side-effect you can now
search for albums based on FreeDB ID
as well as by disc ID. 
You can also inspect the disc ID details
much more closely than before.

FreeDB

FreeDB Moderations, (the mechanism whereby MusicBrainz automatically
imports data from FreeDB with no human intervention),
has been turned off – no more “FreeDB mods”. 
You can still do a manual FreeDB import
if you like.

Also, MusicBrainz now uses “FreeDB protocol 6”, which means much better
Unicode support when importing albums.

Automoderator Elections

All the e-mails which the system sends to the mb-automods mailing list
(when a candidate is nominated, when voting opens, etc) are now “Cc”d to
the candidate (assuming the candidate has entered an e-mail address). 
Previously it was possible, indeed probable, that the candidate had no idea
they were being nominated, and indeed accepted, right up until they got the
“Welcome to the mb-automods mailing list” e-mail.

At the end of an election, if the nominee was accepted, the system can now
make the successful candidate into an Automod without needing manual
intervention from the server administrators.  It can’t yet subscribe
the new auto-moderator to the “mb-automods” mailing list, however.

While voting is open, the tally of votes cast so far is now hidden to all
apart from the proposer, the seconders and the candidate.

If you are not logged in, the automod voting page now displays a more
helpful message than before.

Special “system” users (ModBot etc) can no longer be nominated for
auto-moderator status.

A small typo was fixed in the e-mails sent by The Returning Officer.

TRM Statistics

Ever since TRMs were introduced to MusicBrainz, we’ve kept a count of
how many times each TRM has been looked up by the Tagger (or similar apps). 
The problem was, we had no idea how often each TRM was then used to tag
each associated track.  So when looking at a TRM joined to several
tracks, working out which was the “most used” track was a matter of guesswork.

This release introduces the ability to count uses of TRMs (i.e.
against a specific track), as well as lookups.  You can see this
on the track detail page,
although all the “use” counts will all start out as blank (i.e. zero). 
In fact we’ll be keeping month-by-month lookup counts and use counts for
each TRM, so in theory you’ll be able to see how the tagging “popularity”
for each song rises and falls over time (although all you can see on the web
is the running total).

Other Changes

A bug was causing the artist search index to become
corrupted.  The bug has been fixed and the index has been rebuilt.

When albums are imported from FreeDB, and the ModBot adds a note
giving the URL of the original FreeDB data, that note is no longer
mailed to the original moderator.

When adding an album, both album attributes now default
to “not known” instead of “Album, Official”.

You can now make a case-change edit on artist aliases
(previously it erroneously complained that there was a conflicting
alias).  The code which adds and renames aliases has been made more
robust.

The edit artist page now includes a “copy” button (copies the name
into the sortname field).

“Guess Case” has been tweaked again: it no longer adds a space
after “.” if the next character is “.” (because we don’t want “…” to
become “. ..”).  It doesn’t check for a sub-title split when
inside parentheses (fixes “Album Title (Disc 1: Disc Name) bug). 
It converts “reprise” to lower case when within parentheses. 
It strips spaces after “(” or “[“, and before “]” or “)”.

The MusicBrainz data dumps now include Amazon cover art URLs.

The tagger search page is no longer fooled by extra whitespace
around your search query.

New and Changed Documentation

Changes mainly of interest to MusicBrainz Server Programmers

The INSTALL file has been further updated,
describing the installation process more fully and more helpfully than ever
before 🙂

InitDb.pl now does a much better job
of creating the replication function.  You can use
--with-pending=FILE to tell it where to find pending.so.

Various scripts (MBImport.pl, ImportReplicationChanges,
LoadReplicationChanges) now check the status of completed sub-processes more
carefully.

FixLength.pl now runs every
night.  It has been made more robust too – it reports any errors it
encounters and keeps running, and it also shows the IDs of any albums it
can’t fix.

Only one instance of LoadReplicationChanges
is now allowed at a time.

ProcessReplicationChanges
now deletes “pending” data as it is processed, which means that if the
script is stopped (or crashes) we can safely restart where we left
off.  Several options have also been added to help with
troubleshooting.

Bugs and RFEs Closed

Dave Evans

TRM Database Pruned

The TRM database has been pruned, making the system much faster again.

At about 11pm (UK time) on August 9th the TRM database
was “pruned”, removing all TRMs apart from the ones attached to
MusicBrainz tracks.  This resulted in the database becoming about
one-third of the size it was before.

Because the TRM database is smaller, it now fits into the server’s
memory, reducing the need for disk activity (which means the rest of
MusicBrainz runs faster too).

Because TRM requests can now be served from memory instead of from disk,
TRM responses are now much quicker,
which in turn means that
we don’t have to refuse as many TRM requests
due to the server being too busy.

(Note that most of those linked graphs started in mid March 2004,
just before the TRM server’s memory was upgraded.  This explains why
several of the graphs start with a sudden change in behaviour.)

Server Updates

No-vote changes are accepted by default, better replication support, links back to FreeDB, and a bunch of miscellaneous bug fixes and other enhancements

Changes mainly of interest to MusicBrainz Users

Changes to How Votes Work

Previously if a moderation expired
(1 week after it was entered) and no-one had voted
on it (not counting abstentions), the moderation would simply stay open
until someone did vote on it.  We all know where that got us
– a rather huge backlog of moderations waiting for votes. 😦

Now, the behaviour is a little different: once a moderation has expired
and there are no non-abstaining votes, then the system checks to see if
there are any moderators who are subscribed to the moderation’s artist
(not counting the original moderator).  If there are none, then the
moderation is accepted.  If there are subscribers, then the
moderation may stay open for up to a further
1 week.  (If a vote is cast within that time,
the usual rules apply).  If after that extra time there are
still no votes, then the moderation is accepted.

This therefore differs from the old behaviour in two ways: (1) The default
state for a moderation with no votes is to be accepted (instead of
languishing in an “open” state).  (2) The length of time that the
system will wait before doing so depends on whether or not the moderation’s
artist has any subscribers (not counting the original moderator).

Thanks to Matthias Friedrich for implementing this!

Web-Based Automoderator Elections

After several rather disorganised and confusing Automoderator elections,
it was decided that something more formal, controlled and automated was
probably a good idea.  Hence, the AutoTransaction elections are now
held via this web site.  If you’re an
Automoderator, you can nominate someone for Automod status by clicking on
the “Nominate” link on their profile page.  Anyone may examine the
election information, but it is a private ballot and only Automods may
vote.  The system posts automated messages to the
mb-automods mailing list.

MusicBrainz Mirror Servers

Rob has done lots of work to ensure that the system is more “replication
aware”.  There are various things that MusicBrainz mirror servers can’t
do – for example, they do not store any moderator / moderation data. 
Hence, you can’t log in to a mirror server, and you can’t submit TRMs to one
either.  So when you’re on a mirror server, at various times you’ll see
messages to the effect that what you’re looking for is only available or
possible on the main server.

Other Changes

Moderator Names: previously the system accidentally allowed
non-unique moderator names.  Obviously this was not a good thing,
so moderator names must now be unique.  Any duplicate moderators
have been merged together.

FreeDB Imports: MusicBrainz now remembers the FreeDB category/id
and stores that information in the moderation data as the album is
added.  The ModBot will also a note citing the URL of the FreeDB source
data, like this: “Imported from
http://www.freedb.org/freedb_search_fmt.php?cat=rock&id=710aca08“.

Voting: There was a bug whereby if two votes were cast on the same
moderation at the same time, then the “yes/no” totals could become
wrong.  This has now been fixed.

Error Messages: Errors thrown up by the database were often
not reported to the user.  For example, you may have seen a page which
simply said “Error:” (without giving an error message).  This has now
been fixed.

Disc ID Detail: On the Album Info page, you can now choose to examine
the details of any Disc IDs attached to this album, including the length of
each track (according to each Disc ID), in seconds or sectors. 
Example 1;
Example 2;
Example 3.

Album Attributes: When adding albums, the release type/status now both
default to “I don’t know” (i.e. “None”).  Also, on the “Edit album attributes”
page, you can now remove album attributes (set them back to “None”).

Guess Case: “Guess Case” now recognises
[untitled], [unknown] and [no artist]; it also no longer adds spaces
to punctuated numbers (e.g. 1,234.56)  It more correctly capitalises
hyphenated words “Like-This”.  It no longer capitalises “disc 1” (and
so on), nor various common non-English articles of speech
(der, und, de, du, les, et, …). 
It knows to capitalise YMCA, NYC and R&B.  (all changes by Rob
Rowe).

Release Editor: If there are no releases, you now immediately get a
blank row so you can add a release.  On the “review changes” page which
follows, the album name is now shown.

Miscellany:
Adding a moderation note to “change track name”
now also adds that note to the “change track artist” mod. 
The “Add to Cart” (Amazon) button has been fixed (by Andy Grundman). 
There is now a link to the Style page at the top
of the “list of moderations” pages. 
You can look up a moderator name on the main Search page. 
The section headings in the bar on left (e.g. Quick Search) are now
hyperlinks. 

Changes mainly of interest to MusicBrainz Server Programmers

Replication

As part of the work to improve the support for mirror servers, the way that
database connections are configured has completely changed: see the section
entitled “The Database” in
DBDefs.pm, and see also
Database.pm. 
Additionally, InitDb.pl now checks your
Postgres locale settings, and can take care of more of the configuration for
you.  “InitDb.pl –clean” now runs the scripts in the right order.

The INSTALL file and the
Import HOWTO have
been updated to reflect the new way of doing things.  A couple of
errors in INSTALL have also been fixed.

LoadReplicationChanges
now shows a message if we’re continuing to process a packet
part-way through.  As soon as the downloaded replication packet
has been decompressed, it is deleted.  The sanity checks on
schema_sequence and replication_sequence have been fixed. 
In ProcessReplicationChanges,
the progress meter has been tweaked to be more useful. Also, as
each replication transaction is processed, the packet data is deleted,
so that we can safely stop and restart, and be sure that we restart exactly
where we left off.

Miscellany

UpdateWordCounts now only writes to
the database if the numbers are currently wrong, to avoid creating
unnecessarily large replication packets. 
daily.sh now only show the
output of RunReports if
there was an error. 
QuerySupport::AssociateCDFromAlbumId was unused, and has
been removed. 
Mason 1.26 compatibility has been achieved by not using newlines between
<& and the component name (fixed by Matthias).

CreateTables.sql has been tidied up. 
An error has been fixed in InsertDefaultRows.sql. 
$trm->LoadFull was unused, and has been removed. 
Many instances of $sql->Finish have been moved to the
correct place in the code, or added where missing. 
$discid->LoadFull now always returns a reference to
an array, instead of sometimes returning undef. 

Variables in MM.pm have been tidied up, and some TODO comments have been
added.  MM::CreateDumpRDF was unused, and has been
removed. 
The RDF parser module (Parser.pm) has been rewritten, and is much more
maintanable.  Additionally we can now handle multiple ordinals per
query (in case we should ever need to do that). 
In Mail.pm, error checking has been added to the code which actually sends
e-mails.  DEBUG_MAIL_SPOOL has been added (handy for
testing).

Bugs and RFEs Closed

Dave Evans