Standardized metadata fields

[MainshterHarrison]

It would be good to establish what metadata fields should exist across the corpus, what form the entries should take, and what they refer to. This would enable easier searching of metadata in future, make it easier to implement new features, allow contributors to standardize metadata when submitting texts, and help users access and use this information. The metadata is a really important part of the usefulness of a corpus.

At present there is some variation in field names and entries; for example, there is “transcription”, “transcribed”, and "transcribed & translated". There’s sometimes a date with the name in these fields, sometimes not. There’s also "translator" and "translated".

Ideally the metadata would show what kind of transcription has been made: diplomatic, normalized, typos corrected etc. It should show who made it and when.

Would also be good to pin down what is an acceptable entry for “type”. For example. there is “song”, “bannag”, “Carval”, and “Traditionary Ballad”. I can’t see any usefulness for “traditionary ballad” (as there’s only one of them) or “bannag”.

Where there are notes it is useful to know to whom they are due. Some texts have a lot of linguistic notes without saying who has written them. (On that subject it’s my opinion that linguistic notes should not appear inline at all as they clutter the page and are outside of the scope of a searchable corpus.)

“Source” should have a standardised format.

I’m sure others will have more suggestions about how this aspect of the project can be improved.

[david-allison]

Notes

To be handled in david-allison/manx-corpus-search#132 (keep the [1], but hide the notes until clicked on). Thoughts welcome

Some of the notes were from before we had Kelly's dictionary integrated, or a 'corrected text' functionality. Long-term, I expect a number will go.

I'd support adding initials for the notes, I see this as lower-priority, but check with the group, and if it's important I'll get to it.

Metadata

Agreed, and briefly discussed in our meeting on 2023-01-25.

• We can block a text from being added until the metadata is in a standard format: pretty much any logic which you desire is possible
  • e.g. 'Date of creation' is required
  • e.g. 'Original' must be one of ['English', 'Manx', 'Unknown'] etc...
  • This will be checked automatically when a new text is proposed to be added to the corpus, it will be the responsibility of a maintainer on GitHub (currently me) to fix if the metadata is incorrect.
• We can re-map specific field values when they're presented on the Corpus site
  • If a field is defined in the metadata as 'traditionary ballad', it can appear as 'ballad' on the Corpus.
  • If a field is missing, it can still be displayed in the metadata on the site
• I currently process the 'source' field in the case of newspapers, this allows us to link to the newspaper image on the iMuseum site.
• I don't want to enforce a schema (blocking using a field which we didn't formally define) - some documents may have arbitrary data which is better to display first, and standardise later

Given the principles above, I'd welcome an initial formalisation of what's required. I'd recommend tackling this via incremental improvements as those typically take much less time for a group to come to a consensus on.
I've sent you en email with a link to the Discord chat where the research group is looking to migrate asynchronous discussion to. Feel free to direct the interested parties to this thread.

Tags
I have a longer-term desire to implement a form of 'collections'/'tags' which allow users to browse documents using dimensions other than the search query.
This is aimed more at casual users, but there will be linguistic value in listing all works by a given author. Standardised metadata is intended to feed into this.

Prototype: https://manxcorpus.com/Tags

Rob suggested these tags off the top of his head

Tags

• Prominent Authors
  • Ned Beg
  • JJ Kneen
  • D Far
  • Stowell
  • Wilson
  • Phillips
• Methodist Materials
• Carvals
• Sermons
• Poetry
  • Religious Poetry
  • Secular Poetry
• Political
• Biographical
• Folklore
• Scientific
• Argicultural
• Fishing
• Laws
• Newspaper
  • By Newspaper
  • Viginteetes
• Historical
• Architectural
• NSEW
• Legal

Thanks! Give me a ping when the group have come to a consensus and I'll get it done.