Off-ticket, @patrickbrophy asked:

Should this be for each service or just the director?

Here, we're looking specifically for XRootD crashes -- so we'll be recording the information whenever the cache or origin component is enabled.

[Are we] tracking if XRootD crashes or the entire pelican process?

Just XRootD. Specifically, we probably want to record this information here, when pelican handles the death of an XRootD process by signal.

Additional thoughts:

• This should be in a generic "counters" table (feel free to tinker with the name, that one is a bit blah) that is string key / double value. (I think you need to have the value be in the billions before floating point precision loss means x + 1 == x). The key should be primary, unique, and indexed.
• Since this is for caches and origins -- and a single service could be both -- we're probably talking two keys.
• We don't need to blindly auto-publish all rows in the table to prometheus; rather, we can assume only known keys are published (so we can iterate through a list of keys set at compile time -- for this PR, there will be a single one).
• It may make sense to record "time of last crash" (unix timestamp) than a counter. Looking at the Prometheus documentation, the changes() function will allow prometheus to determine the number of crashes within a certain time period. Recording "time of last crash" avoids floating point precision issues to boot!