Hello Everyone!
I’m Junaid (fettuccinae), an undergraduate Computer Science student at MGIT in India. This summer, I returned to MetaBrainz for my second GSoC project, where I worked on modernizing the MusicBrainz search under the mentorship of @kartikohri13 and @bitmap.
Project Overview:
MusicBrainz uses Apache Solr for search queries.
The previous implementation serialized all the response data into a single _store field.
The response writers reads _store, unmarshal the XML into a MusicBrainz XML Metadata Format (MMD) object and then serialize it again as XML or JSON for the response.
Few problems with this design are:
1. The indexer must construct a complete XML representation for every document, even though much of the same information is already available in normal Solr fields.
2. Most of the response data is stored in an opaque _store XML blob.
This project focused on moving the fields from one _store XML blob into their own flat fields (and JSON strings for nested fields).
The main goals of this project were:
1. Upgrade the Solr schema version from 1.5 to 1.7
2. Add fields (in configsets and indexer) to store all the data to be returned
3. Create response writers to return data from fields
The proposal for this project can be found here.
Result:
This project replaced the _store blob with stored fields and JSON string fields across all 16 search entities.
I reindexed both the old and new configurations using the sample MusicBrainz database, then force-merged the indexes before measuring them.
I wrote a python script to measure their sizes using Solr’s status API and compared them.
The total core size went from 3.8 GiB to 2.8 GiB, saving around 1 GiB or 26%.
| Collection | Old core | New core | Difference% |
| annotation | 15.8 MiB | 14.8 MiB | 5.8% |
| area | 7.9 MiB | 4.5 MiB | 42.9% |
| artist | 131.3 MiB | 81.2 MiB | 38.2% |
| cdstub | 94.0 MiB | 56.0 MiB | 40.5% |
| editor | 5.6 MiB | 1.9 MiB | 65.6% |
| event | 2.3 MiB | 1.4 MiB | 39.7% |
| instrument | 542.8 KiB | 405.2 KiB | 25.4% |
| label | 10.7 MiB | 6.3 MiB | 40.6% |
| place | 6.2 MiB | 3.7 MiB | 40.1% |
| recording | 3.0 GiB | 2.2 GiB | 26.1% |
| release | 37.2 MiB | 25.9 MiB | 30.5% |
| release-group | 42.1 MiB | 31.7 MiB | 24.7% |
| series | 1.2 MiB | 624.5 KiB | 49.6% |
| tag | 3.6 MiB | 1.6 MiB | 56.0% |
| url | 65.7 MiB | 44.4 MiB | 32.3% |
| work | 363.0 MiB | 307.1 MiB | 15.4% |
| Total | 3.8 GiB | 2.8 GiB | 26.1% |
With BEST_COMPRESSION codec enabled, it reduces to 2 GiB with a size difference of 1.8 GiB or 47.9%. (The trade-off here is the search speed performance)
| Collection | Old | New | Difference% |
| annotation | 15.8 MiB | 12.1 MiB | 23.4% |
| area | 7.9 MiB | 4.0 MiB | 49.7% |
| artist | 131.3 MiB | 66.2 MiB | 49.6% |
| cdstub | 94.0 MiB | 49.8 MiB | 47.1% |
| editor | 5.6 MiB | 1.7 MiB | 68.9% |
| event | 2.3 MiB | 1.2 MiB | 47.5% |
| instrument | 542.8 KiB | 370.2 KiB | 31.8% |
| label | 10.7 MiB | 5.5 MiB | 48.5% |
| place | 6.2 MiB | 3.2 MiB | 48.7% |
| recording | 3.0 GiB | 1.5 GiB | 49.5% |
| release | 37.2 MiB | 19.6 MiB | 47.2% |
| release-group | 42.1 MiB | 20.2 MiB | 52.1% |
| series | 1.2 MiB | 570.6 KiB | 53.9% |
| tag | 3.6 MiB | 1.4 MiB | 61.2% |
| url | 65.7 MiB | 38.1 MiB | 42.0% |
| work | 363.0 MiB | 236.1 MiB | 34.9% |
| Total | 3.8 GiB | 2.0 GiB | 47.9% |
Implementation details:
Upgrading the schema:
The Solr schema change from 1.5 to 1.7 enables docValues by default for primitive field types.
Most primitive fields are already stored directly with stored="true". Since docValues are mainly used for faceting, sorting, and function queries, I added docValues="false" to primitive field types to avoid storing the same value twice.ref_count is used in boost functions in some cores , which needs docValues.
I created a new int_dv field type with docValues="true" and used it for ref_count and the other count fields used by boost functions.
Finally, I bumped the schema version from 1.5 to 1.7 in each core and verified that indexing and search continued to work.
Configset:
I changed fields that can be stored directly to stored="true" and added the remaining fields with indexed="false" and stored="true".
I then removed _store field from the schemas and request parameters.
I replaced ngram with edge-ngram, since we don’t require non-edge permutations of a query during search.
Indexer:
In the SearchEntity class, I added a preserve_og flag. When converting a result dictionary into a Solr document, this flag preserves the original field order instead of converting the values to a set.
I also added an objconverter method, which allows a specific field’s structure to be converted before it is sent to Solr.
The preserve_og flag is enabled for simple parallel fields, such as tags. These fields do not need to be stored as JSON, instead, they use two parallel multi-valued fields. Preserving their order is necessary so that each tag_count remains associated with its corresponding tag_name.
The objconverter method is used to convert nested fields into JSON strings.
In each Search<Entity>, I moved fields from extrapaths to their respective fields and added the required objconverter methods for nested fields.
The existing wscompat converter logic for nested objects was mirrored to create these objconverter methods.
The remaining logic for creating MMD objects is now handled by the response writers.
Response Writer:
I added MBFlatXMLWriter, a field based response writer that reads the stored fields and, by mirroring the logic in wscompat's converters, create a MMD object and write it to the response.
I added entity-specific builders to construct MMD object from its stored fields. MBFlatXMLWriter uses these builders as helper classes.
I made MBJSONWriter which is currently used by MusicBrainz website search, to extend MBFlatXMLWriter instead of MBXMLWriter and verified that search works.
I also updated the tests to validate the new flat field logic instead of the old _store logic.
Future
The next steps for this project are to:
1. Benchmark it in production and measure disk, RAM, and CPU usage to see whether the optimizations hold up in a real environment or introduce any regressions.
2. Use Solr’s default response writers instead of custom response writers, and build the MMD objects in musicbrainz-server itself.
Conclusion
This project was more complex than my previous one. I spent a good amount of time understanding and deep-diving into Apache Lucene and Solr internals.
The implementation was a lot of fun because I was constantly experimenting with different approaches and reporting my findings. Each time I triggered a reindex, I stared at the terminal, hoping that indexing would not throw a wall of errors and that the core size would at least remain the same as the baseline.
Overall, the project gave me a much better understanding of search engines, some of the distributed-system techniques used by Solr, and the complexity of MusicBrainz itself.
I’m thankful to my mentors, @kartikohri13 and @bitmap, for their support and guidance. It has been a pleasure working with you all. Thanks for an incredible summer!