TRM Database Pruned

The TRM database has been pruned, making the system much faster again.

At about 11pm (UK time) on August 9th the TRM database
was “pruned”, removing all TRMs apart from the ones attached to
MusicBrainz tracks.  This resulted in the database becoming about
one-third of the size it was before.

Because the TRM database is smaller, it now fits into the server’s
memory, reducing the need for disk activity (which means the rest of
MusicBrainz runs faster too).

Because TRM requests can now be served from memory instead of from disk,
TRM responses are now much quicker,
which in turn means that
we don’t have to refuse as many TRM requests
due to the server being too busy.

(Note that most of those linked graphs started in mid March 2004,
just before the TRM server’s memory was upgraded.  This explains why
several of the graphs start with a sudden change in behaviour.)

Server Updates

No-vote changes are accepted by default, better replication support, links back to FreeDB, and a bunch of miscellaneous bug fixes and other enhancements

Changes mainly of interest to MusicBrainz Users

Changes to How Votes Work

Previously if a moderation expired
(1 week after it was entered) and no-one had voted
on it (not counting abstentions), the moderation would simply stay open
until someone did vote on it.  We all know where that got us
– a rather huge backlog of moderations waiting for votes. 😦

Now, the behaviour is a little different: once a moderation has expired
and there are no non-abstaining votes, then the system checks to see if
there are any moderators who are subscribed to the moderation’s artist
(not counting the original moderator).  If there are none, then the
moderation is accepted.  If there are subscribers, then the
moderation may stay open for up to a further
1 week.  (If a vote is cast within that time,
the usual rules apply).  If after that extra time there are
still no votes, then the moderation is accepted.

This therefore differs from the old behaviour in two ways: (1) The default
state for a moderation with no votes is to be accepted (instead of
languishing in an “open” state).  (2) The length of time that the
system will wait before doing so depends on whether or not the moderation’s
artist has any subscribers (not counting the original moderator).

Thanks to Matthias Friedrich for implementing this!

Web-Based Automoderator Elections

After several rather disorganised and confusing Automoderator elections,
it was decided that something more formal, controlled and automated was
probably a good idea.  Hence, the AutoTransaction elections are now
held via this web site.  If you’re an
Automoderator, you can nominate someone for Automod status by clicking on
the “Nominate” link on their profile page.  Anyone may examine the
election information, but it is a private ballot and only Automods may
vote.  The system posts automated messages to the
mb-automods mailing list.

MusicBrainz Mirror Servers

Rob has done lots of work to ensure that the system is more “replication
aware”.  There are various things that MusicBrainz mirror servers can’t
do – for example, they do not store any moderator / moderation data. 
Hence, you can’t log in to a mirror server, and you can’t submit TRMs to one
either.  So when you’re on a mirror server, at various times you’ll see
messages to the effect that what you’re looking for is only available or
possible on the main server.

Other Changes

Moderator Names: previously the system accidentally allowed
non-unique moderator names.  Obviously this was not a good thing,
so moderator names must now be unique.  Any duplicate moderators
have been merged together.

FreeDB Imports: MusicBrainz now remembers the FreeDB category/id
and stores that information in the moderation data as the album is
added.  The ModBot will also a note citing the URL of the FreeDB source
data, like this: “Imported from
http://www.freedb.org/freedb_search_fmt.php?cat=rock&id=710aca08“.

Voting: There was a bug whereby if two votes were cast on the same
moderation at the same time, then the “yes/no” totals could become
wrong.  This has now been fixed.

Error Messages: Errors thrown up by the database were often
not reported to the user.  For example, you may have seen a page which
simply said “Error:” (without giving an error message).  This has now
been fixed.

Disc ID Detail: On the Album Info page, you can now choose to examine
the details of any Disc IDs attached to this album, including the length of
each track (according to each Disc ID), in seconds or sectors. 
Example 1;
Example 2;
Example 3.

Album Attributes: When adding albums, the release type/status now both
default to “I don’t know” (i.e. “None”).  Also, on the “Edit album attributes”
page, you can now remove album attributes (set them back to “None”).

Guess Case: “Guess Case” now recognises
[untitled], [unknown] and [no artist]; it also no longer adds spaces
to punctuated numbers (e.g. 1,234.56)  It more correctly capitalises
hyphenated words “Like-This”.  It no longer capitalises “disc 1” (and
so on), nor various common non-English articles of speech
(der, und, de, du, les, et, …). 
It knows to capitalise YMCA, NYC and R&B.  (all changes by Rob
Rowe).

Release Editor: If there are no releases, you now immediately get a
blank row so you can add a release.  On the “review changes” page which
follows, the album name is now shown.

Miscellany:
Adding a moderation note to “change track name”
now also adds that note to the “change track artist” mod. 
The “Add to Cart” (Amazon) button has been fixed (by Andy Grundman). 
There is now a link to the Style page at the top
of the “list of moderations” pages. 
You can look up a moderator name on the main Search page
The section headings in the bar on left (e.g. Quick Search) are now
hyperlinks. 

Changes mainly of interest to MusicBrainz Server Programmers

Replication

As part of the work to improve the support for mirror servers, the way that
database connections are configured has completely changed: see the section
entitled “The Database” in
DBDefs.pm, and see also
Database.pm
Additionally, InitDb.pl now checks your
Postgres locale settings, and can take care of more of the configuration for
you.  “InitDb.pl –clean” now runs the scripts in the right order.

The INSTALL file and the
Import HOWTO have
been updated to reflect the new way of doing things.  A couple of
errors in INSTALL have also been fixed.

LoadReplicationChanges
now shows a message if we’re continuing to process a packet
part-way through.  As soon as the downloaded replication packet
has been decompressed, it is deleted.  The sanity checks on
schema_sequence and replication_sequence have been fixed. 
In ProcessReplicationChanges,
the progress meter has been tweaked to be more useful. Also, as
each replication transaction is processed, the packet data is deleted,
so that we can safely stop and restart, and be sure that we restart exactly
where we left off.

Miscellany

UpdateWordCounts now only writes to
the database if the numbers are currently wrong, to avoid creating
unnecessarily large replication packets. 
daily.sh now only show the
output of RunReports if
there was an error. 
QuerySupport::AssociateCDFromAlbumId was unused, and has
been removed. 
Mason 1.26 compatibility has been achieved by not using newlines between
<& and the component name (fixed by Matthias).

CreateTables.sql has been tidied up. 
An error has been fixed in InsertDefaultRows.sql. 
$trm->LoadFull was unused, and has been removed. 
Many instances of $sql->Finish have been moved to the
correct place in the code, or added where missing. 
$discid->LoadFull now always returns a reference to
an array, instead of sometimes returning undef. 

Variables in MM.pm have been tidied up, and some TODO comments have been
added.  MM::CreateDumpRDF was unused, and has been
removed. 
The RDF parser module (Parser.pm) has been rewritten, and is much more
maintanable.  Additionally we can now handle multiple ordinals per
query (in case we should ever need to do that). 
In Mail.pm, error checking has been added to the code which actually sends
e-mails.  DEBUG_MAIL_SPOOL has been added (handy for
testing).

Bugs and RFEs Closed

Dave Evans

Server Updates

This release adds support for data replication, makes others’ votes more visible, and allows moderation notes to be e-mailed to other voters, not just the moderator.

Changes mainly of interest to MusicBrainz Users

Votes on open moderations are no longer as “aggressively hidden” as they
used to be.  Now you can always see the voting on any mod, regardless
of status etc, by going to the Moderation Detail
page.  On the main voting pages
you’ll only not be able to see the vote counts as long as you haven’t voted;
as soon as you vote (or when the mod closes, or if it’s your mod anyway)
then the vote counts are shown.

All URLs in moderation notes are now hyperlinked, not just ones that link to
other MusicBrainz pages.  See
it
in
action
(Technically there are limitations to this – only http, https and ftp URLs
are detected, and finding where the URL ends is a bit of a fuzzy science –
but it gets it right most of the time).

From now on, whenever you add a note to a moderation, you’ll get mailed all
future notes added to that moderation.  If you don’t like this new
behaviour, you can disable it via a new preference.  In that case
you’ll only get mailed notes which were added to your own mods.

Each entry in the Style Guide is now “linkable”,
so you can refer style guide entries to other users just by posting a link.

For new Tagger users, it’s now much more obvious that you have to copy your
username / password into the relevant part of the “Options” window.

Previously it was possible to add artists, albums etc with trailing
whitespace in their names.  This caused problems with artists
especially, where the name is meant to be unique.  The system now tries
much harder to remove such whitespace before the data is written to the
database.  As a one-off exercise for this release, all
trailing-whitespace data should now have been removed.

Previously, strange side-effects could have occurred in various places if the
names of artists, etc. contained the “_” or “%” characters.  These
potential side-effects have now been fixed.

The Top Moderators page has been revamped; the
moderator names listed there are now links to their profile pages.

On the Browse Releases page, there is now
a “title” attribute (which in most browsers is shown as a “tooltip”) which
shows the track, disc id and TRM id counts for each release.

If there is an error when adding a track, previously the system just
reported “Error”.  Now it shows you the full error message, as
originally intended.

Changes mainly of interest to MusicBrainz Programmers

The TRM “gateway” on the server has now been reimplemented in
mod_perl.  For the most part this shouldn’t make much difference to
you; but it does mean for example that you can use any URL at
trm.musicbrainz.org,
and that we can get
better
statistics
on use of the signature server. 
There are also some new “special” TRMs
which can be returned to indicate various exceptional conditions.

New and Changed Documentation

The documentation for the client
and TunePimp have been
extended and reorganised.

Changes mainly of interest to MusicBrainz Server Programmers

Replication

The server now supports one-way data replication, for most (but not all)
tables.  (The tables excluded are the moderation and moderator tables
– nine tables in all).  The replication is based on the “dbmirror” code
contributed to the Postgres codebase.

Every hour, the ExportAllTables script
runs and exports the changed rows into a “replication packet” file (a
compressed “tar” file).  Those files are then made available on our
FTP site
Every so often (for example, once a week) a complete database export is
run, and this too is made available on our FTP site.  By initially
importing one of these full databases (minus the moderation tables),
then applying each replication packet, a slave database can keep up-to-date
with the master.

The RDF Dump

We no longer perform regular RDF exports (the “mbdump.rdf.bz2” file). 
The script which did the exports still exists, however.  If you want RDF
then one way to do it would be to set yourself up as a Postgres mirror, and
run the RDF export script against your database.

Installation

The "cron" scripts
have been rewritten; they are now configured via
config.sh, and the
INSTALL file now mentioned the existence of
these scripts.

Other Changes

Text::Unaccent version 1.05 couldn’t handle non-string data safely
(including “undef”).  We’re now running a patched version of this
module; the relevant patch is available on our FTP site.

There is a new “cleanup” script,
UnusedTracks;
some old scripts have been removed.

Various database write operations have been moved into more appropriate places
in the code, as have some read operations.  Lots of SQL has been
rewritten, and some unused methods have been removed from
Sql.pm.

Mail.pm
now complains and shows the error message in the event of failure. 
There were miscellaneous HTML fixes. 
Cache.pm
was changed to allow false values; “remove” was renamed to “delete”.

Bugs and RFEs Closed

Dave Evans

MusicBrainz Tagger 0.10.4

The 0.10.4 version of the tagger fixes a number of problems related to saving tracks.

This version contains the following bug fixes:

  • VBR MP3 files that have no ID3v2 tag, but have an ID3v1 tag may have had
    their Xing header corrupted, which could cause these MP3 files to show up with
    incorrect durations in audio players.
  • The tagger now reads the release year from the TYER ID3v2 tag, which means that
    if the ID3v2 tag has the right year, but MB does not, the release year from the ID3v2
    tag will be used when renaming the files.
  • Some files that were previously saved and then saved again, may have had an erroneous (1) appended to the filename.
  • Don Redman provided two improved bitmaps for the save and submit buttons, so that they look better when they are disabled.
  • Don also provided a new animation for the ‘page loading’ animation in the tagger. Thanks Don!

Robert Kaye

MusicBrainz Tagger 0.10.3

The 0.10.3 version of the tagger fixes a number of bugs and provides new options for configuring the file names.

This bug fix version fixes problems in the id3v2 tags generated by the
0.10.2 version. This version also supports logging, removing id3v2 tags
from ogg files, sports new toolbar icons, shows special too short
and silence trms.

With the increased load on the TRM signature server tagging clients may encounter
TRM server busy messages and the tagger now captures those and reminds the user
to donate money to MusicBrainz when the TRM server becomes busy. The tagger also
supports the new %type (album type), %status (album status), %year, $month, %date
(for first release date of the album) and %country (first release country)
variables for the saving tagged files. These values are now also written to
the metadata tags.

Bugs and RFEs Closed

Robert Kaye