Server Updates

Better Disc ID support, UTF-8 support for FreeDB, several tweaks to the automoderator election system and the usual miscellaneous bunch of bug fixes and other changes.

Changes mainly of interest to MusicBrainz Users

Revamped Support for Disc IDs

Duplicate disc IDs are now allowed.  As a pleasant
side-effect you can now
search for albums based on FreeDB ID
as well as by disc ID
You can also inspect the disc ID details
much more closely than before.

FreeDB

FreeDB Moderations, (the mechanism whereby MusicBrainz automatically
imports data from FreeDB with no human intervention),
has been turned off – no more “FreeDB mods”. 
You can still do a manual FreeDB import
if you like.

Also, MusicBrainz now uses “FreeDB protocol 6”, which means much better
Unicode support when importing albums.

Automoderator Elections

All the e-mails which the system sends to the mb-automods mailing list
(when a candidate is nominated, when voting opens, etc) are now “Cc”d to
the candidate (assuming the candidate has entered an e-mail address). 
Previously it was possible, indeed probable, that the candidate had no idea
they were being nominated, and indeed accepted, right up until they got the
“Welcome to the mb-automods mailing list” e-mail.

At the end of an election, if the nominee was accepted, the system can now
make the successful candidate into an Automod without needing manual
intervention from the server administrators.  It can’t yet subscribe
the new auto-moderator to the “mb-automods” mailing list, however.

While voting is open, the tally of votes cast so far is now hidden to all
apart from the proposer, the seconders and the candidate.

If you are not logged in, the automod voting page now displays a more
helpful message than before.

Special “system” users (ModBot etc) can no longer be nominated for
auto-moderator status.

A small typo was fixed in the e-mails sent by The Returning Officer.

TRM Statistics

Ever since TRMs were introduced to MusicBrainz, we’ve kept a count of
how many times each TRM has been looked up by the Tagger (or similar apps). 
The problem was, we had no idea how often each TRM was then used to tag
each associated track.  So when looking at a TRM joined to several
tracks, working out which was the “most used” track was a matter of guesswork.

This release introduces the ability to count uses of TRMs (i.e.
against a specific track), as well as lookups.  You can see this
on the track detail page,
although all the “use” counts will all start out as blank (i.e. zero). 
In fact we’ll be keeping month-by-month lookup counts and use counts for
each TRM, so in theory you’ll be able to see how the tagging “popularity”
for each song rises and falls over time (although all you can see on the web
is the running total).

Other Changes

A bug was causing the artist search index to become
corrupted.  The bug has been fixed and the index has been rebuilt.

When albums are imported from FreeDB, and the ModBot adds a note
giving the URL of the original FreeDB data, that note is no longer
mailed to the original moderator.

When adding an album, both album attributes now default
to “not known” instead of “Album, Official”.

You can now make a case-change edit on artist aliases
(previously it erroneously complained that there was a conflicting
alias).  The code which adds and renames aliases has been made more
robust.

The edit artist page now includes a “copy” button (copies the name
into the sortname field).

“Guess Case” has been tweaked again: it no longer adds a space
after “.” if the next character is “.” (because we don’t want “…” to
become “. ..”).  It doesn’t check for a sub-title split when
inside parentheses (fixes “Album Title (Disc 1: Disc Name) bug). 
It converts “reprise” to lower case when within parentheses. 
It strips spaces after “(” or “[“, and before “]” or “)”.

The MusicBrainz data dumps now include Amazon cover art URLs.

The tagger search page is no longer fooled by extra whitespace
around your search query.

New and Changed Documentation

Changes mainly of interest to MusicBrainz Server Programmers

The INSTALL file has been further updated,
describing the installation process more fully and more helpfully than ever
before 🙂

InitDb.pl now does a much better job
of creating the replication function.  You can use
--with-pending=FILE to tell it where to find pending.so.

Various scripts (MBImport.pl, ImportReplicationChanges,
LoadReplicationChanges) now check the status of completed sub-processes more
carefully.

FixLength.pl now runs every
night.  It has been made more robust too – it reports any errors it
encounters and keeps running, and it also shows the IDs of any albums it
can’t fix.

Only one instance of LoadReplicationChanges
is now allowed at a time.

ProcessReplicationChanges
now deletes “pending” data as it is processed, which means that if the
script is stopped (or crashes) we can safely restart where we left
off.  Several options have also been added to help with
troubleshooting.

Bugs and RFEs Closed

Dave Evans

TRM Database Pruned

The TRM database has been pruned, making the system much faster again.

At about 11pm (UK time) on August 9th the TRM database
was “pruned”, removing all TRMs apart from the ones attached to
MusicBrainz tracks.  This resulted in the database becoming about
one-third of the size it was before.

Because the TRM database is smaller, it now fits into the server’s
memory, reducing the need for disk activity (which means the rest of
MusicBrainz runs faster too).

Because TRM requests can now be served from memory instead of from disk,
TRM responses are now much quicker,
which in turn means that
we don’t have to refuse as many TRM requests
due to the server being too busy.

(Note that most of those linked graphs started in mid March 2004,
just before the TRM server’s memory was upgraded.  This explains why
several of the graphs start with a sudden change in behaviour.)

Server Updates

No-vote changes are accepted by default, better replication support, links back to FreeDB, and a bunch of miscellaneous bug fixes and other enhancements

Changes mainly of interest to MusicBrainz Users

Changes to How Votes Work

Previously if a moderation expired
(1 week after it was entered) and no-one had voted
on it (not counting abstentions), the moderation would simply stay open
until someone did vote on it.  We all know where that got us
– a rather huge backlog of moderations waiting for votes. 😦

Now, the behaviour is a little different: once a moderation has expired
and there are no non-abstaining votes, then the system checks to see if
there are any moderators who are subscribed to the moderation’s artist
(not counting the original moderator).  If there are none, then the
moderation is accepted.  If there are subscribers, then the
moderation may stay open for up to a further
1 week.  (If a vote is cast within that time,
the usual rules apply).  If after that extra time there are
still no votes, then the moderation is accepted.

This therefore differs from the old behaviour in two ways: (1) The default
state for a moderation with no votes is to be accepted (instead of
languishing in an “open” state).  (2) The length of time that the
system will wait before doing so depends on whether or not the moderation’s
artist has any subscribers (not counting the original moderator).

Thanks to Matthias Friedrich for implementing this!

Web-Based Automoderator Elections

After several rather disorganised and confusing Automoderator elections,
it was decided that something more formal, controlled and automated was
probably a good idea.  Hence, the AutoTransaction elections are now
held via this web site.  If you’re an
Automoderator, you can nominate someone for Automod status by clicking on
the “Nominate” link on their profile page.  Anyone may examine the
election information, but it is a private ballot and only Automods may
vote.  The system posts automated messages to the
mb-automods mailing list.

MusicBrainz Mirror Servers

Rob has done lots of work to ensure that the system is more “replication
aware”.  There are various things that MusicBrainz mirror servers can’t
do – for example, they do not store any moderator / moderation data. 
Hence, you can’t log in to a mirror server, and you can’t submit TRMs to one
either.  So when you’re on a mirror server, at various times you’ll see
messages to the effect that what you’re looking for is only available or
possible on the main server.

Other Changes

Moderator Names: previously the system accidentally allowed
non-unique moderator names.  Obviously this was not a good thing,
so moderator names must now be unique.  Any duplicate moderators
have been merged together.

FreeDB Imports: MusicBrainz now remembers the FreeDB category/id
and stores that information in the moderation data as the album is
added.  The ModBot will also a note citing the URL of the FreeDB source
data, like this: “Imported from
http://www.freedb.org/freedb_search_fmt.php?cat=rock&id=710aca08“.

Voting: There was a bug whereby if two votes were cast on the same
moderation at the same time, then the “yes/no” totals could become
wrong.  This has now been fixed.

Error Messages: Errors thrown up by the database were often
not reported to the user.  For example, you may have seen a page which
simply said “Error:” (without giving an error message).  This has now
been fixed.

Disc ID Detail: On the Album Info page, you can now choose to examine
the details of any Disc IDs attached to this album, including the length of
each track (according to each Disc ID), in seconds or sectors. 
Example 1;
Example 2;
Example 3.

Album Attributes: When adding albums, the release type/status now both
default to “I don’t know” (i.e. “None”).  Also, on the “Edit album attributes”
page, you can now remove album attributes (set them back to “None”).

Guess Case: “Guess Case” now recognises
[untitled], [unknown] and [no artist]; it also no longer adds spaces
to punctuated numbers (e.g. 1,234.56)  It more correctly capitalises
hyphenated words “Like-This”.  It no longer capitalises “disc 1” (and
so on), nor various common non-English articles of speech
(der, und, de, du, les, et, …). 
It knows to capitalise YMCA, NYC and R&B.  (all changes by Rob
Rowe).

Release Editor: If there are no releases, you now immediately get a
blank row so you can add a release.  On the “review changes” page which
follows, the album name is now shown.

Miscellany:
Adding a moderation note to “change track name”
now also adds that note to the “change track artist” mod. 
The “Add to Cart” (Amazon) button has been fixed (by Andy Grundman). 
There is now a link to the Style page at the top
of the “list of moderations” pages. 
You can look up a moderator name on the main Search page
The section headings in the bar on left (e.g. Quick Search) are now
hyperlinks. 

Changes mainly of interest to MusicBrainz Server Programmers

Replication

As part of the work to improve the support for mirror servers, the way that
database connections are configured has completely changed: see the section
entitled “The Database” in
DBDefs.pm, and see also
Database.pm
Additionally, InitDb.pl now checks your
Postgres locale settings, and can take care of more of the configuration for
you.  “InitDb.pl –clean” now runs the scripts in the right order.

The INSTALL file and the
Import HOWTO have
been updated to reflect the new way of doing things.  A couple of
errors in INSTALL have also been fixed.

LoadReplicationChanges
now shows a message if we’re continuing to process a packet
part-way through.  As soon as the downloaded replication packet
has been decompressed, it is deleted.  The sanity checks on
schema_sequence and replication_sequence have been fixed. 
In ProcessReplicationChanges,
the progress meter has been tweaked to be more useful. Also, as
each replication transaction is processed, the packet data is deleted,
so that we can safely stop and restart, and be sure that we restart exactly
where we left off.

Miscellany

UpdateWordCounts now only writes to
the database if the numbers are currently wrong, to avoid creating
unnecessarily large replication packets. 
daily.sh now only show the
output of RunReports if
there was an error. 
QuerySupport::AssociateCDFromAlbumId was unused, and has
been removed. 
Mason 1.26 compatibility has been achieved by not using newlines between
<& and the component name (fixed by Matthias).

CreateTables.sql has been tidied up. 
An error has been fixed in InsertDefaultRows.sql. 
$trm->LoadFull was unused, and has been removed. 
Many instances of $sql->Finish have been moved to the
correct place in the code, or added where missing. 
$discid->LoadFull now always returns a reference to
an array, instead of sometimes returning undef. 

Variables in MM.pm have been tidied up, and some TODO comments have been
added.  MM::CreateDumpRDF was unused, and has been
removed. 
The RDF parser module (Parser.pm) has been rewritten, and is much more
maintanable.  Additionally we can now handle multiple ordinals per
query (in case we should ever need to do that). 
In Mail.pm, error checking has been added to the code which actually sends
e-mails.  DEBUG_MAIL_SPOOL has been added (handy for
testing).

Bugs and RFEs Closed

Dave Evans

Server Updates

This release adds support for data replication, makes others’ votes more visible, and allows moderation notes to be e-mailed to other voters, not just the moderator.

Changes mainly of interest to MusicBrainz Users

Votes on open moderations are no longer as “aggressively hidden” as they
used to be.  Now you can always see the voting on any mod, regardless
of status etc, by going to the Moderation Detail
page.  On the main voting pages
you’ll only not be able to see the vote counts as long as you haven’t voted;
as soon as you vote (or when the mod closes, or if it’s your mod anyway)
then the vote counts are shown.

All URLs in moderation notes are now hyperlinked, not just ones that link to
other MusicBrainz pages.  See
it
in
action
(Technically there are limitations to this – only http, https and ftp URLs
are detected, and finding where the URL ends is a bit of a fuzzy science –
but it gets it right most of the time).

From now on, whenever you add a note to a moderation, you’ll get mailed all
future notes added to that moderation.  If you don’t like this new
behaviour, you can disable it via a new preference.  In that case
you’ll only get mailed notes which were added to your own mods.

Each entry in the Style Guide is now “linkable”,
so you can refer style guide entries to other users just by posting a link.

For new Tagger users, it’s now much more obvious that you have to copy your
username / password into the relevant part of the “Options” window.

Previously it was possible to add artists, albums etc with trailing
whitespace in their names.  This caused problems with artists
especially, where the name is meant to be unique.  The system now tries
much harder to remove such whitespace before the data is written to the
database.  As a one-off exercise for this release, all
trailing-whitespace data should now have been removed.

Previously, strange side-effects could have occurred in various places if the
names of artists, etc. contained the “_” or “%” characters.  These
potential side-effects have now been fixed.

The Top Moderators page has been revamped; the
moderator names listed there are now links to their profile pages.

On the Browse Releases page, there is now
a “title” attribute (which in most browsers is shown as a “tooltip”) which
shows the track, disc id and TRM id counts for each release.

If there is an error when adding a track, previously the system just
reported “Error”.  Now it shows you the full error message, as
originally intended.

Changes mainly of interest to MusicBrainz Programmers

The TRM “gateway” on the server has now been reimplemented in
mod_perl.  For the most part this shouldn’t make much difference to
you; but it does mean for example that you can use any URL at
trm.musicbrainz.org,
and that we can get
better
statistics
on use of the signature server. 
There are also some new “special” TRMs
which can be returned to indicate various exceptional conditions.

New and Changed Documentation

The documentation for the client
and TunePimp have been
extended and reorganised.

Changes mainly of interest to MusicBrainz Server Programmers

Replication

The server now supports one-way data replication, for most (but not all)
tables.  (The tables excluded are the moderation and moderator tables
– nine tables in all).  The replication is based on the “dbmirror” code
contributed to the Postgres codebase.

Every hour, the ExportAllTables script
runs and exports the changed rows into a “replication packet” file (a
compressed “tar” file).  Those files are then made available on our
FTP site
Every so often (for example, once a week) a complete database export is
run, and this too is made available on our FTP site.  By initially
importing one of these full databases (minus the moderation tables),
then applying each replication packet, a slave database can keep up-to-date
with the master.

The RDF Dump

We no longer perform regular RDF exports (the “mbdump.rdf.bz2” file). 
The script which did the exports still exists, however.  If you want RDF
then one way to do it would be to set yourself up as a Postgres mirror, and
run the RDF export script against your database.

Installation

The "cron" scripts
have been rewritten; they are now configured via
config.sh, and the
INSTALL file now mentioned the existence of
these scripts.

Other Changes

Text::Unaccent version 1.05 couldn’t handle non-string data safely
(including “undef”).  We’re now running a patched version of this
module; the relevant patch is available on our FTP site.

There is a new “cleanup” script,
UnusedTracks;
some old scripts have been removed.

Various database write operations have been moved into more appropriate places
in the code, as have some read operations.  Lots of SQL has been
rewritten, and some unused methods have been removed from
Sql.pm.

Mail.pm
now complains and shows the error message in the event of failure. 
There were miscellaneous HTML fixes. 
Cache.pm
was changed to allow false values; “remove” was renamed to “delete”.

Bugs and RFEs Closed

Dave Evans

MusicBrainz Tagger 0.10.4

The 0.10.4 version of the tagger fixes a number of problems related to saving tracks.

This version contains the following bug fixes:

  • VBR MP3 files that have no ID3v2 tag, but have an ID3v1 tag may have had
    their Xing header corrupted, which could cause these MP3 files to show up with
    incorrect durations in audio players.
  • The tagger now reads the release year from the TYER ID3v2 tag, which means that
    if the ID3v2 tag has the right year, but MB does not, the release year from the ID3v2
    tag will be used when renaming the files.
  • Some files that were previously saved and then saved again, may have had an erroneous (1) appended to the filename.
  • Don Redman provided two improved bitmaps for the save and submit buttons, so that they look better when they are disabled.
  • Don also provided a new animation for the ‘page loading’ animation in the tagger. Thanks Don!

Robert Kaye