Category: MusicBrainz

Posts related to the MusicBrainz encyclopedia in general: community, data, client, server, etc.

Picard 1.4.2 released

Official MusicBrainz cross-platform music tagger Picard 1.4.2 is now out.

This is mainly a bugfix release, the only notable improvement is related to TOPE/TOAL tags.

Users can get Windows and MacOSx packages from Picard website downloads section.

Bug

  • [PICARD-1053] – Picard does not stop analyzer while moving
  • [PICARD-1055] – picard hangs with: RuntimeError: maximum recursion depth exceeded in cmp
  • [PICARD-1070] – The “Convert Unicode punctuation characters to ASCII” function only works in certain tags
  • [PICARD-1077] – ID3v2.4 text encoding settings are not saved correctly

Improvement

  • [PICARD-969] – Search dialog webservices get queued behind matched album requests
  • [PICARD-1034] – Picard not seeing TOPE and TOAL

Picard 1.4.1 released

Official MusicBrainz cross-platform music tagger Picard 1.4.1 is now out, it contains few fixes and enhancements, plus few extra features that didn’t make it to 1.4.

As usual you can get Windows and MacOSx packages from Picard website downloads section.

Most notable changes are the fix of a regression concerning drag-n-drop on MacOSx and improved cover art box, with better visual feedback about changes in cover art images.Capture du 2017-04-01 14:40:58

Thanks to all contributors who helped on this release: Antonio Larrosa, Frederik “Freso” S. Olesen, Laurent Monin, Lukáš Lalinský, Sambhav Kothari, Sophist, tungol, Ville Skyttä, Wieland Hoffmann.

And to all of our translators!

To report any issue about this release, please use our bug tracker. As usual you can also get help on the forum or on IRC (freenode, #musicbrainz).

We’ll now focus on 2.0, which will be mostly a port of 1.4.x to Python 3 and Qt5.
It will open the way to UI improvements, and improved overall performance.

Bug

  • [PICARD-953] – Album shown matched even if extra unmatched files
  • [PICARD-972] – Removing album with saves pending does not remove pending saves
  • [PICARD-973] – Pending log messages not flushed to stderr on quit
  • [PICARD-988] – Drag & Drop not working
  • [PICARD-990] – Picard violating ID3 standard for TXXX frames
  • [PICARD-996] – Disabling the cover art box and enabling it again doesn’t bring it back
  • [PICARD-998] – Disabling the action toolbar sometimes doesn’t work
  • [PICARD-1005] – If a cluster is moved to the album side of the main window it gets moved to unmatched files
  • [PICARD-1006] – Drag and drop for cover arts doesnt work on OSX
  • [PICARD-1010] – Unsetting View/Cover Art doesn’t work permanently
  • [PICARD-1011] – Toolbar tab order incorrect after PICARD-908
  • [PICARD-1014] – Number of images in release info is calculated incorrectly
  • [PICARD-1015] – Artwork tab of the Track Info DIalog doesn’t show changes anymore
  • [PICARD-1018] – CoverArtBox doesn’t show new/removed images after unmatched files are added/removed to the album
  • [PICARD-1023] – Directory persistence for Add Directory needs tweaking
  • [PICARD-1029] – Fix ~artists_sort metadata variable
  • [PICARD-1042] – Missing import for PICARD_APP_NAME

New Feature

  • [PICARD-258] – Visual feedback for changes to artwork in before-after pane.
  • [PICARD-1000] – Implement artwork diff for albums

Task

  • [PICARD-943] – Remove monkey patching of file write methods in picard formats
  • [PICARD-1041] – Replace Ok button text by Make It So! in Options dialog

Improvement

  • [PICARD-223] – Remove should work when Unmatched Files is selected
  • [PICARD-666] – Drop MP3 tagging support
  • [PICARD-951] – Always use HTTPS for musicbrainz.org
  • [PICARD-952] – Use Cover Art Archive over HTTPS
  • [PICARD-961] – Mention AcoustID on Scan button too
  • [PICARD-980] – Drag&drop cover art doesn’t work for images from amazon/google images/https links
  • [PICARD-1012] – Buttons on the “User Interface” and “Scripting” pages are smaller than buttons in other places
  • [PICARD-1016] – Multiple images in related tracks confusing
  • [PICARD-1021] – Picard loads all pending files before quitting
  • [PICARD-1024] – Allow specifying a configuration file path
  • [PICARD-1030] – Allow to add/replace cover art images and keep existing cover art

Picard 1.4 released

Official MusicBrainz cross-platform music tagger Picard 1.4 is now out, containing many fixes and new features!
The last time we put out a stable release was more than 2 years ago, so a lot of changes have made it into this new release. If you’re in a hurry and just want to try it out, the downloads are available from the Picard website.

Picard 1.4

picard-1-4-screenshot
A brand new Picard 1.4 install, fresh from the oven. Get it while it’s hot!

2 years is a long time for software, and Picard has been the subject of a couple of Google Summer of Code projects and plenty of developments required by changes in dependencies and the MusicBrainz project itself. Overall, the changes would be too numerous to list. If you’re curious to read it all, a complete list of changes can be seen on the website. Here are at least a few of the highlights: Continue reading “Picard 1.4 released”

Google is awesome: How should we spend $5000?

The Google Open Source Programs office is amazing!

In the past few years Google has sponsored our summit where we gather a pile of MusicBrainzers into one room and talk shop (and a bit of play) for a whole weekend. But, due to extenuating circumstances and our move to NewHost, we opted for a much smaller and lower key developer gathering in Barcelona. As a result we didn’t spend much money and I opted to not pester Google to support our summit as they have in the years past.

Then yesterday the lovely Cat Allman from OSPO send us a PO for $5000 and a stern reminder to send an invoice soon. After explaining the situation to Cat, she still felt it was appropriate to collect the money and to “use the money as best serves the project.” 

Wow, thank you!

The question I have, for all of our readers, contributors, editors, hackers and advocates: How do you think we should spend $5000 to best serve MusicBrainz and its sister projects?

I’ll leave it wide open to everyone to chime in — feel free to put suggest anything reasonable and meaningful in the comments. Please do the skip the “send it to me!” type comments. 🙂

Finally, I want to thank Google for its continued support of our projects. Through its annual support of $40,000, Summer of Code, Code-In, the paid data license for its Knowledge Graph and the support of our summit, Google is the biggest supporter of the MetaBrainz Foundation. By far: Google has donated more than $366,000!

THANK YOU GOOGLE. Your support really helps us along!

Summer of Code ’16 with Picard

Hi! I’m Rahul Raturi, GSoC participant for Picard. This was my first GSoC, and it’s been a pretty awesome experience. Following is the overview of my project.

About the project

The outline of the project is to allow searching for albums, artists and tracks from within Picard. This avoids switching back and forth between web browser and Picard for searching, say release. If Picard fails to auto tag a file usual flow to tag the file with correct metadata is to first select the file, then click on “Lookup in Browser”, then search correct release, and load it into Picard by clicking the green “Tagger” button. In some systems, the “Tagger” button wouldn’t show, which was also a nuisance. With this patch, the entities can be searched and optionally loaded into Picard using built-in search dialogs, so no application switching.

Search dialogs

Picard already provides search options (through a web browser) for three entities; namely track, artist, and album. So I’ve built search dialogs for these three.

  1. Track Search Dialog — Searches for tracks and allows optionally loading corresponding album back into Picard. track_dialog
  2. Album Search Dialog — Searches albums and optionally allows loading the selected one into Picard. Screenshot from 2016-08-15 17-08-09
  3. Artist Search Dialog —  Displays basic information about the artists. To get more information about the selected artist, there’s an option to lookup him/her in browser. Screenshot from 2016-08-20 15-55-42

Searching similar tracks/releases

This is another important part of the project. Sometimes Picard fails to auto tag a file (or a cluster), or incorrectly tag it. These dialogs may prove useful here. To get expected data, right click on the file (should be in “Unmatched Files” cluster), and select “Search for similar tracks…”. The track search dialog would pop up, and expected release can be looked up there. Same procedure is for searching clusters.

Links to my work

Each PR is based on the previous one. A new dialog in each, plus some improvements to existing dialog. For trying the dialog, clone the artist search branch, until it gets merged into master. It has the most recent changes.

Note: To use these dialog for searching, an option in User Interface setting about built-in search needs to be enabled.

Conclusion

It was quite fun doing this project. Thanks to Michael Wiencek (mentor) for the guidance and leniency :). Also the Picard team for the reviews. I look forward to contribute more to Picard, now that I’ve a better understanding of the code. Also for another Summer of Code.

Schema change release: What happened?

Now that we’ve finally finished the schema change release, I wanted to give an account of what happened in this arduous process. Before I dive into the details, I want to offer a picture that best sums up our current situation and challenges:

personal-container-mngmnt3

The shipping container is MusicBrainz and the boat is our hosting infrastructure. This picture perfectly describes the sort of challenges we’ve faced over the past few days. 🙂

Here is what happened:

Because the site was recently running slow and our search servers kept crashing, Zas and I were not available to help Bitmap prepare for the schema change release. This long process was left to Bitmap and Gentlecat to take care of on their own. We quickly realized that we were not ready for the release when the due date came and thus we delayed one week.

Sunday 22 May

Finally we were ready to proceed with the Postgres 9.5 upgrade. Once we started the process, we kept running into small problems that we didn’t get in our test setups. We do not have access to enough infrastructure to have a complete clone of our production environment, so we can only do so much to prepare for all the things that might happen when we run upgrades on our production servers.

All the while we attempted to start the upgrade, our backup database server was running much slower than anticipated. In the end we figured out that a step for optimizing the database (analyzing it) wasn’t carried out. During this time the site was really slow/unusable, but by the time the problem became apparent we had started the upgrade and could not turn back.

Once the upgrade was done, optimizing the database took much much longer than usual: 3 hours! This process wasn’t started until about 1am local time, which made for a very long night before that process finished. And even then we hit snags and had to start over a couple of times. At about 4:30am we had the site running on Postgres 9.5 in read only mode. The plan was to rest and start the schema change release in the morning.

Monday 23 May

Of course we had spent all of our time working on the Postgres upgrade and site stability, so our document that we use to plan the schema change was not in place. We spent the day preparing this and other bits for the release. To get an appreciation for what this document looks like, have a look! Note that some steps could be instant, others might take hours to carry out. Others might involve a sub-step or 20 not included in the document.

In the evening we were ready to make the change. By this point our backup DB was performing much better, so the read-only site worked acceptably. Thus, we started the release. Overall, the actual release process was reasonably smooth – we hit a few snags and had to do a lot of waiting for our slow servers. At about 1am in the morning things were finally complete. We proceeded with our sanity checks to make sure things went smoothly and all of them passed.

We proceeded to put the site into read-write mode and immediately saw portions of Postgres crashing, which is really bad. With community feedback we quickly deduced that some write operations were causing Postgres back-end processes to crash. We went back to read-only mode on the site and things stabilized and we finally went to bed at 3am.

Tuesday 24 May

In the morning we quickly found the source of database trouble with the help from the Postgres people on IRC. Thanks for the swift help Johto! We found that the steps for installing the updated third party extensions into Postgres had not completed correctly. Repeating the steps by hand fixed this problem.

Sadly yesterday morning we got an email informing us that our Live Data Feed replication stream had become corrupted. 🙁 This was heartbreaking news to us, since it means a great inconvenience to all of our Live Data Feed users. We immediately split into two teams: Zas, chirlu and myself to fix the root cause of the issue and Bitmap to investigate fixing the stream.

I proceeded to setup a test environment was able to quickly reproduce the problem. Zas and chirlu were an amazing support team Googling issues as I came across them. Within fairly short time we fixed the problem and deployed the fix to our database server. The problem was caused by a bug in a piece of code that we’ve been using for 13 years! A change in Postgres caused this bug to actually become a problem and corrupt our replication feed. 🙁

Once the problems were fixed we needed to initiate a new data dump and check to make sure the replication stream is working correctly. Of course we found a problem that we fixed and re-started the process to dump the data. Loads of hurry-up-and-wait situations to try our patience!

When we were satisfied that things were working correctly we re-enabled the site as read-write at about 1am and allowed people to continue editing. Exhausted we stumbled into bed waiting for data dumps to sync out to the FTP site.

Wednesday 25 May

Today Bitmap was flying home and as soon as WiFi became available on his flight he started working and helping with putting the schema change to bed. We’ve verified that everything is working as expected. At last this saga comes to and end and we can all take a break and catch up on sleep!

Thank you for your patience through all of this.

Schema change release, 2016-05-23 (with upgrade instructions)

Starting with this release, PostgreSQL 9.5 is now our minimum supported version. In order to import any future data sets, you will need to upgrade your installation to version 9.5.

Due to unforeseen problems with the Live Data Feed (AKA replication), users with slave databases will be required to first import a fresh data dump into their new 9.5 installation. We apologize that this is the case, but even had this stream not been broken, doing a clean import is faster and easier than doing the migration. For details on what happened during this rather lengthy schema change release, stay tuned for a post mortem blog post that covers the details.

If you have a non-replicated standalone database, you can use pg_upgrade and run ./upgrade.sh directly, but for simplicity we strongly recommend importing the latest data dump. Thus, we will only provide instructions for a clean import:

  1. Make sure you have PostgreSQL 9.5 installed, and your database settings in lib/DBDefs.pm are updated to point to the 9.5 installation if you currently have an older version of postgres running. If you already have postgres 9.5 and want to replace the existing database there, you’ll need to drop it first (using dropdb or from within psql). Be careful that you’re not dropping any important data if this is a standalone database that you’ve made changes to.
  2. Take down the web server running MusicBrainz, if you’re running a web server.
  3. Turn off cron jobs if you are automatically updating the database via cron jobs.
  4. Switch to the new code with git fetch origin followed by git checkout v-2016-05-23-schema-change-v2
  5. Run cpanm --installdeps --notest . to ensure your perl-based dependencies are up to date. Note the dot at the end.
  6. Set DB_SCHEMA_SEQUENCE to 23 in lib/DBDefs.pm
  7. Download the latest data dumps. If you don’t need historical edit data, excluding the edit dump will speed up your import significantly.
  8. Initialize a new database from the data dumps downloaded in step 7. Detailed instructions for doing this are located in INSTALL.md in the musicbrainz-server repository; if your data dumps are in /tmp, the command should simply be something like ./admin/InitDb.pl --createdb --import /tmp/mbdump*.tar.bz2.
  9. After the import has finished, turn cron jobs back on, if applicable.
  10. Restart the MusicBrainz web server, as well as memcached, if applicable.

We would like to thank bitmap, Gentlecat, zas, chirlu, reosarevok, gcilou for contributing directly to the release and we’d also like to thank all of the people who helped test, debug or otherwise offer support in this quite difficult release. Thank you!

And finally, here’s the list of changes you can expect in the upgrade:

Bug

  • [MBS-6406] – Admins can’t change email addresses
  • [MBS-8288] – Missing indexes for inverse lookup on *_gid_redirect tables
  • [MBS-8669] – Primary key for place table missing on old slaves
  • [MBS-8906] – Release pages ISE if CB doesn’t return JSON from its API for whatever reason
  • [MBS-8928] – If you submit the release editor without being logged in, it displays “[object Object]” as an error mesage
  • [MBS-8943] – Some pages do not respect DB_READ_ONLY setting

Improvement

  • [MBS-1873] – Fix vote tallies for edits
  • [MBS-3887] – Duplicate artist and label names not being checked against alias
  • [MBS-8287] – Log deleted entities that were in a subscribed collection
  • [MBS-8433] – Work attributes don’t have a uuid
  • [MBS-8716] – Store the edit data in a JSONB column
  • [MBS-8717] – Move the edit data to a separate table
  • [MBS-8838] – Add gids to all *_type* tables
  • [MBS-8873] – Convert and unify artist credit editors to React
  • [MBS-8909] – Add logos to IMDb and VGMdb links in the sidebar
  • [MBS-8939] – Update the Instagram logo used in the sidebar
  • [MBS-8940] – Let banner message editors dismiss the banner only temporarily

Task

  • [MBS-8656] – Bring edit table indexes back into sync
  • [MBS-8719] – Stop materializing of edit and vote counts
  • [MBS-8720] – Add a materialized view of edit note recipients
  • [MBS-8727] – Prevent duplicate votes
  • [MBS-8800] – Create the earthdistance extension and add a geodetic index for place coordinates
  • [MBS-8804] – Add BRIN indexes for timestamp columns
  • [MBS-8897] – add new entity icons
  • [MBS-8938] – Schema changes to support alternative tracklists

State of the Onion: MetaBrainz

In the past few weeks we’ve been hit with several traffic increases to MusicBrainz which is putting considerably more strain on our aging infrastructure than we’re happy with. If it seems that we’re not doing anything about it, that is because we’ve been busy behind the scenes trying to keep things moving forward. This sometimes doesn’t leave us a lot of time to keep the public informed on our work. Hopefully this blog post will fix this in the short term:

In 2011 we started to make plans to move MusicBrainz hosting into the cloud, but then out of the blue we were donated a pile of machines. There were so many machines that I postponed the cloud plans and prepared the donated machines for service. That has carried us for 4+ years with almost no hardware cost, which was really great. The plan was to move to the cloud sometime around 2015, but then I spent most of 2014/2015 dealing with conflicts in the team, putting us seriously behind schedule while our hardware decayed.

On top of that, we’ve recently had some “bad luck”. We have had some disrespectful commercial customers hit us really hard and we had to find and block them. We have had unexpected traffic spikes and when trying to address these unexpected traffic spikes, we had two more machines fail on us. These were the donated machines that we kept in reserve for just this moment. The loss of two machines caught us short on capacity to handle the increased demands on our servers.

So, now we face the tough question: Do we buy expensive hardware that we might use for 6 months (~$5000) or do we try and save the money and tough it out? I’d rather not spend so much money on such short term use if we can avoid it. We’re going to try and move to a new hosting facility somewhere in the EU, since that is where most of our users are.

Moving to a new hosting facility has an incredible number of dependencies that Christina (our Biz Dev manager), Zas and I have been working through. It may not seem like we have a plan, but we do, and we’re incredibly busy trying to make the plan happen. To give you a taste of what we’re up against:

  1. We want to move our hosting to Europe and have a business presence in Europe in order to reduce the costs and inefficiencies of being a solely US based business. A lot of our traffic, customers and contractors are in the EU and it simply makes sense to have a presence here.
  2. To establish a presence in the EU I needed local help to help with the business matters as well as researching and establishing an EU organization. So I needed to find a Biz Dev manager and that person is Christina.
  3. Once Christina was on board she researched our options about what was suited for us. Getting that process moving involved getting certified documents from California, board approval for spending funds to establish the organization, EU labor law research, (and we needed to swap a board member, too!), hiring help to establish the org. and generally navigating the Spanish bureaucracy. (See this only slightly exaggerated short film for some clues of our ordeal.)
  4. Once the org. had been established we needed to convince the bank to open a bank account for us. The draconian US banking laws extend worldwide and the local bank had to ensure that they were not opening themselves up to thousands of $$$ in accounting hassles just to allow a tiny non-profit to open a bank account. We finally have a bank account and have started paying our contractors with it!
  5. At the same time we’re also working to set up an office for the growing team here in Barcelona. That required a byzantine process that barely started when you sign the lease. Getting power, internet and water set up has taken a frustratingly long time. Had I known how long, I would have stayed at my co-working space for a while longer while addressing hosting issues.
  6. While Christina has been focused on the hardcore paperwork, Zas is keeping the site running, which itself requires many heroics. Zas and I have started planning the move to the EU hosting provider. We’ve got a 5-page document that collects some of the open questions and requirements around this process: https://docs.google.com/document/d/16KNm4KksNwz29Opk1aILOMtCmPIeXFuxxUjMoPT3th0/edit#heading=h.dpfvoz1idcro. Right now Zas and Bitmap are here in Barcelona and we’re going to work on establishing a formal plan for moving to the new hosting company. We’re currently comparing hosting company offerings – see what we’ve collected so far if you care to follow along. The amount of work required to make this happen is making my head hurt. (A special shoutout to KodeStar, lead developer of FanArt.tv, for providing a lot of useful feedback about our various options.)
  7. While Christina, Zas and I have our hands full, Bitmap and Gentlecat continue to release new features and work on the schema change. Not to mention all the contributions from Freso and Reosarevok to keep the community happy and polite while we deal with less than optimal site conditions. That said, I am really happy and proud of my team, trying to keep things running in sub-optimal conditions.

This is just a snapshot of everything that is happening behind the scenes that will culminate with the goal of moving to a new hosting company and being set up in the EU. And mind you, we’re doing this with a minuscule budget trying to be careful of how we spent our money.

Deprecating MBIDs

This post is an April Fools joke. Rest assured, we have no intention of changing the MBID system that MusicBrainz currently uses.

But, like all good parody news items, there is an element of truth behind this post. The announcement of the Echo Nest API shutdown is real, and with this change you will no longer be able to use the Echo Nest IDs to look up information. This particularly hurts users of the Million Song Dataset, which maps each track to the Echo Nest ID. The new Spotify API isn’t even providing any compatibility api or ID mapping, leaving users to look up 1 million Spotify IDs in the remaining months that the the old Echo Nest api will remain available.

At MusicBrainz, we understand the importance of a stable identifier system. That’s why, 16 years ago, we picked these unwieldy-looking UUID identifiers – that have since proven to have stood the test of time, with room to continue growing. You can look up an MBID made 16 years ago, today – and it will still work another 16 years in the future.

Hello all,

Following Echo Nest’s bold announcement that Echo Nest ids are being replaced by Spotify IDs, we figured it was time to make our own ID change public as well – MBIDs were a fantastic idea 16 years ago, but let’s face it, they’re not the most beautiful thing around, so our MBIDs will now also be replaced by Spotify IDs to help with a proper mapping across tools. Anything without a Spotify mapping will simply get purged. This should greatly simplify the data we have and remove any doubt for some releases whether they exist or not – if they’re on Spotify, they clearly exist!

We would like to commend Echo Nest on their brave leadership in this, giving us the courage to move on from our ancient heritage and try new things. With the speed technologies evolve in this digital age, it can be hard to keep up with things and keep things fresh, but Echo Nest is showing the way forward, and we’re delighted to be able to follow so quickly in their path.

I hope you all will welcome this bold move by our team. We hope to have it ready by next schema change. We know we’re excited! 😀

PS. No, we will not provide a mapping between MBIDs and the new Spotify IDs. We trust our data users to be capable to set things up on their own. Happy hacking! 🙂