Idea: Go back to supporting Prefixes in S3

[cassanovaw]

What is the problem you are trying to solve?

My S3 bucket needs to support prefixed values. In other words in my main bucket is named foo I essentially want the ability to specify a subdirectory bucket such as foo/mirmir-alertmanager, foo/mirmi-blocks, foo/mimir-ruler.

mimir:

common:
  storage:
    backend: s3
    s3:
      endpoint: s3.us-east-1.amazonaws.com
      region: us-east-1
      secret_access_key: "${AWS_SECRET_ACCESS_KEY}" # This is a secret injected via an environment variable
      access_key_id: "${AWS_ACCESS_KEY_ID}"
alertmanager_storage:
  s3:
    bucket_name: foo/mimir-alertmanager
    endpoint: s3.us-east-1.amazonaws.com
    region: us-east-1
    secret_access_key: "${AWS_SECRET_ACCESS_KEY}" # This is a secret injected via an environment variable
    access_key_id: "${AWS_ACCESS_KEY_ID}"
blocks_storage:
  s3:
    bucket_name: foo/mimir-blocks
    endpoint: s3.us-east-1.amazonaws.com
    region: us-east-1
    secret_access_key: "${AWS_SECRET_ACCESS_KEY}" # This is a secret injected via an environment variable
    access_key_id: "${AWS_ACCESS_KEY_ID}"
ruler_storage:
  s3:
    bucket_name: foo/mimir-ruler
    endpoint: s3.us-east-1.amazonaws.com
    region: us-east-1
    secret_access_key: "${AWS_SECRET_ACCESS_KEY}" # This is a secret injected via an environment variable
    access_key_id: "${AWS_ACCESS_KEY_ID}"

Which solution do you envision (roughly)?

Support a prefix bucket

Have you considered any alternatives?

An alternative in S3 is to create 3 new buckets at the base which I don't want to necessarily do.

Any additional context to share?

It seems like this used to work.

How long do you think this would take to be developed?

Not sure

What are the documentation dependencies?

No response

Proposer?

No response

[56quarters]

Have you tried using the storage_prefix setting for each storage block? It should do exactly what you're suggesting if I'm not mistaken. docs

[cassanovaw]

So I guess you are saying I can move the bucket_name up to common.storage.s3 and then in each of the sections for alertmanager_storage, blocks_storage, and ruler storage add a new configuration parameter called storage_prefix. It seems like it used to be just the word prefix back in the Cortex days?

…

On Fri, Jan 31, 2025 at 1:17 PM Nick Pillitteri ***@***.***> wrote: Have you tried using the storage_prefix setting for each storage block? It should do exactly what you're suggesting if I'm not mistaken. docs <https://grafana.com/docs/mimir/latest/configure/configuration-parameters/#ruler_storage> — Reply to this email directly, view it on GitHub <#10550 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABGGXXX5C2G3JT44Q4YWLTD2NO42PAVCNFSM6AAAAABWIAF6OGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMMRYGAZDCOBSGU> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[cassanovaw]

And a related question... Loki seems to create the necessary buckets structure automatically..... Does Mimir do the same or do I need to pre create the subdirectory buckets in S3? On Fri, Jan 31, 2025 at 1:30 PM William Cassanova ***@***.***> wrote:

…

So I guess you are saying I can move the bucket_name up to common.storage.s3 and then in each of the sections for alertmanager_storage, blocks_storage, and ruler storage add a new configuration parameter called storage_prefix. It seems like it used to be just the word prefix back in the Cortex days? On Fri, Jan 31, 2025 at 1:17 PM Nick Pillitteri ***@***.***> wrote: > Have you tried using the storage_prefix setting for each storage block? > It should do exactly what you're suggesting if I'm not mistaken. docs > <https://grafana.com/docs/mimir/latest/configure/configuration-parameters/#ruler_storage> > > — > Reply to this email directly, view it on GitHub > <#10550 (comment)>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/ABGGXXX5C2G3JT44Q4YWLTD2NO42PAVCNFSM6AAAAABWIAF6OGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMMRYGAZDCOBSGU> > . > You are receiving this because you authored the thread.Message ID: > ***@***.***> >

[56quarters]

So I guess you are saying I can move the bucket_name up to common.storage.s3 and then in each of the sections for alertmanager_storage, blocks_storage, and ruler storage add a new configuration parameter called storage_prefix.

Yeah, that's what I'm thinking.

It seems like it used to be just the word prefix back in the Cortex days?

Possibly, a bunch of config got moved around and renamed when Mimir was forked from Cortex.

Loki seems to create the necessary buckets structure automatically..... Does Mimir do the same or do I need to pre create the subdirectory buckets in S3?

Object storage doesn't really have directories, it just a prefix on the keys created. So Mimir will create keys with the appropriate prefix and it should just work.

[cassanovaw]

I will give it a try and report back.

…

On Fri, Jan 31, 2025 at 1:42 PM Nick Pillitteri ***@***.***> wrote: So I guess you are saying I can move the bucket_name up to common.storage.s3 and then in each of the sections for alertmanager_storage, blocks_storage, and ruler storage add a new configuration parameter called storage_prefix. Yeah, that's what I'm thinking. It seems like it used to be just the word prefix back in the Cortex days? Possibly, a bunch of config got moved around and renamed when Mimir was forked from Cortex. Loki seems to create the necessary buckets structure automatically..... Does Mimir do the same or do I need to pre create the subdirectory buckets in S3? Object storage doesn't really have directories, it just a prefix on the keys created. So Mimir will create keys with the appropriate prefix and it should just work. — Reply to this email directly, view it on GitHub <#10550 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABGGXXX5NPNIJ23JSRPJHZ32NO73FAVCNFSM6AAAAABWIAF6OGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMMRYGA3DCOBVG4> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[cassanovaw]

@56quarters

Thank you for your help.

Changing prefix to storage_prefix allowed the chart to deploy successfully.
Incidentally, the bucket names listed in https://grafana.com/docs/mimir/latest/configure/configure-object-storage-backend/ did not work...For example, alert-manager did NOT work alertmanager did. I wonder if this is because the example assumed that I would not be using a prefix.

31 common:
32 storage:
33 backend: s3
34 s3:
35 bucket_name:
36 endpoint: s3.us-east-1.amazonaws.com
37 region: us-east-1
38 secret_access_key: "${AWS_SECRET_ACCESS_KEY}" #
39 access_key_id: "${AWS_ACCESS_KEY_ID}"
40 alertmanager_storage:
41 storage_prefix: alertmanager
42 blocks_storage:
43 storage_prefix: mimirblocks
44 ruler_storage:
45 storage_prefix: mimirruler

The deployment itself worked....But I do not seen any new prefix buckets getting created in S3.
What I do see is a directory named __mimir_cluster under which is mimir_cluster_seed.json.
I also see a directory called mimirblocks ( that matches my above ) and in that directory another __mimir_cluster/mirmir_cluster_seed.json.

Does that imply that I need to manually create the folders for Alertmanager and mimirruler at the root of my S3 bucket?