Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Added Bucket-config as a configmap in a Block-sync init container. #838

Closed
wants to merge 761 commits into from
Closed

Added Bucket-config as a configmap in a Block-sync init container. #838

wants to merge 761 commits into from

Conversation

kushalShukla-web
Copy link
Contributor

  1. Following prometheus/test-infra#779 and prometheus/prometheus#15183, the placement of the bucket-config file in the Prometheus repository has created some inconsistencies, such as:

    • Different data for different PRs.
    • The bucket-config file is required in multiple parts of PromBench, such as the Load Generator to query downloaded block data. This leads to downloading the entire Prometheus repository, which is not an efficient solution.

  2. To address these issues, I have moved all bucket configuration into a ConfigMap.

dependabot bot and others added 30 commits October 26, 2023 07:36
@dependabot
Bump golang.org/x/net from 0.10.0 to 0.17.0
e61316e 
Bumps [golang.org/x/net](https://github.com/golang/net) from 0.10.0 to 0.17.0.
- [Commits](golang/net@v0.10.0...v0.17.0)

---
updated-dependencies:
- dependency-name: golang.org/x/net
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <[email protected]>
@dependabot
Bump pyyaml from 6.0 to 6.0.1 in /tools/load-generator
84f6ad7 
Bumps [pyyaml](https://github.com/yaml/pyyaml) from 6.0 to 6.0.1.
- [Changelog](https://github.com/yaml/pyyaml/blob/6.0.1/CHANGES)
- [Commits](yaml/pyyaml@6.0...6.0.1)

---
updated-dependencies:
- dependency-name: pyyaml
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
@SuperQ
Merge pull request prometheus#590 from prometheus/dependabot/pip/tool…
d49e025 
…s/load-generator/pyyaml-6.0.1

Bump pyyaml from 6.0 to 6.0.1 in /tools/load-generator
@SuperQ
Merge pull request prometheus#602 from prometheus/dependabot/go_modul…
0663e2e 
…es/golang.org/x/net-0.17.0

Bump golang.org/x/net from 0.10.0 to 0.17.0
@SuperQ
Merge pull request prometheus#601 from prometheus/dependabot/go_modul…
67e3ea1 
…es/google.golang.org/grpc-1.56.3

Bump google.golang.org/grpc from 1.55.0 to 1.56.3
@SuperQ
Merge pull request prometheus#600 from prometheus/dependabot/pip/tool…
6cc5720 
…s/load-generator/urllib3-1.26.18

Bump urllib3 from 1.26.15 to 1.26.18 in /tools/load-generator
@dependabot
Bump github.com/go-git/go-git/v5 from 5.4.2 to 5.10.0
3aa7c05 
Bumps [github.com/go-git/go-git/v5](https://github.com/go-git/go-git) from 5.4.2 to 5.10.0.
- [Release notes](https://github.com/go-git/go-git/releases)
- [Commits](go-git/go-git@v5.4.2...v5.10.0)

---
updated-dependencies:
- dependency-name: github.com/go-git/go-git/v5
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
@SuperQ
Merge pull request prometheus#605 from prometheus/dependabot/go_modul…
2fec1db 
…es/github.com/go-git/go-git/v5-5.10.0

Bump github.com/go-git/go-git/v5 from 5.4.2 to 5.10.0
@SuperQ
Merge pull request prometheus#583 from prometheus/repo_sync
e9f14e5 
Synchronize common files from prometheus/prometheus
@dependabot
Bump golang.org/x/oauth2 from 0.8.0 to 0.13.0
4616a4d 
Bumps [golang.org/x/oauth2](https://github.com/golang/oauth2) from 0.8.0 to 0.13.0.
- [Commits](golang/oauth2@v0.8.0...v0.13.0)

---
updated-dependencies:
- dependency-name: golang.org/x/oauth2
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
@dependabot
Bump prometheus-client from 0.17.0 to 0.17.1 in /tools/load-generator
33351d0 
Bumps [prometheus-client](https://github.com/prometheus/client_python) from 0.17.0 to 0.17.1.
- [Release notes](https://github.com/prometheus/client_python/releases)
- [Commits](prometheus/client_python@v0.17.0...v0.17.1)

---
updated-dependencies:
- dependency-name: prometheus-client
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
@dependabot
Bump github.com/aws/aws-sdk-go from 1.44.213 to 1.46.4
40426c0 
Bumps [github.com/aws/aws-sdk-go](https://github.com/aws/aws-sdk-go) from 1.44.213 to 1.46.4.
- [Release notes](https://github.com/aws/aws-sdk-go/releases)
- [Commits](aws/aws-sdk-go@v1.44.213...v1.46.4)

---
updated-dependencies:
- dependency-name: github.com/aws/aws-sdk-go
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
@SuperQ
Merge pull request prometheus#607 from prometheus/dependabot/go_modul…
ed319c5 
…es/github.com/aws/aws-sdk-go-1.46.4

Bump github.com/aws/aws-sdk-go from 1.44.213 to 1.46.4
@SuperQ
Merge pull request prometheus#606 from prometheus/dependabot/go_modul…
dda85bf 
…es/golang.org/x/oauth2-0.13.0

Bump golang.org/x/oauth2 from 0.8.0 to 0.13.0
@SuperQ
Merge pull request prometheus#592 from prometheus/dependabot/pip/tool…
3944ea1 
…s/load-generator/prometheus-client-0.17.1

Bump prometheus-client from 0.17.0 to 0.17.1 in /tools/load-generator
@dependabot
Bump requests from 2.28.2 to 2.31.0 in /tools/load-generator
af5410d 
Bumps [requests](https://github.com/psf/requests) from 2.28.2 to 2.31.0.
- [Release notes](https://github.com/psf/requests/releases)
- [Changelog](https://github.com/psf/requests/blob/main/HISTORY.md)
- [Commits](psf/requests@v2.28.2...v2.31.0)

---
updated-dependencies:
- dependency-name: requests
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <[email protected]>
@SuperQ
Merge pull request prometheus#541 from prometheus/dependabot/pip/tool…
3c7f8ae 
…s/load-generator/requests-2.31.0

Bump requests from 2.28.2 to 2.31.0 in /tools/load-generator
@prombot
Update common Prometheus files
647cba8 
Signed-off-by: prombot <[email protected]>
@jesusvazquez
Update test-infra go modules
0238d8c 
Signed-off-by: Jesus Vazquez <[email protected]>
@SuperQ
Merge pull request prometheus#608 from prometheus/repo_sync
1662ec8 
Synchronize common files from prometheus/prometheus
@bboreham
Merge pull request prometheus#609 from prometheus/jvp/update-dependen…
9214fb1 
…cies

Update test-infra go modules
@bwplotka
Fixed comment monitor msg for profiles. (prometheus#599)
2de6172 
Signed-off-by: bwplotka <[email protected]>
@prombot
Update common Prometheus files
2038a07 
Signed-off-by: prombot <[email protected]>
@roidelapluie
Implement temporary legacy auth fix for GKE in prombench
4566fb2 
Due to the removal of provider-specific authentication code from kubectl
in Kubernetes v1.26, Prombench experiences connectivity issues. This
commit temporarily reinstates the legacy GCP plugin to maintain
functionality. A transition to the new `gke-gcloud-auth-plugin` will be
planned for future releases to align with the updated GKE authentication
process.

Signed-off-by: Julien Pivotto <[email protected]>
@roidelapluie
Merge pull request prometheus#616 from roidelapluie/quickfix
5184ba6 
Implement temporary legacy auth fix for GKE in prombench
@bboreham
Improve Prombench query behaviour (prometheus#618)
29467c4 
* Prombench: scrape cAdvisor metrics

So that queries for container_memory_rss return some data.

Signed-off-by: Bryan Boreham <[email protected]>

* Prombench: adjust load-gen queries

In some cases reduce the amount of data returned; in others reduce the
frequency so there is a gap between one batch and the next.
node_cpu has been renamed to node_cpu_seconds_total.

Signed-off-by: Bryan Boreham <[email protected]>

---------

Signed-off-by: Bryan Boreham <[email protected]>
@dependabot
Bump idna from 3.4 to 3.6 in /tools/load-generator
a58d58a 
Bumps [idna](https://github.com/kjd/idna) from 3.4 to 3.6.
- [Changelog](https://github.com/kjd/idna/blob/master/HISTORY.rst)
- [Commits](kjd/idna@v3.4...v3.6)

---
updated-dependencies:
- dependency-name: idna
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
@dependabot
Bump cloud.google.com/go/container from 1.26.2 to 1.29.0
5d3cd78 
Bumps [cloud.google.com/go/container](https://github.com/googleapis/google-cloud-go) from 1.26.2 to 1.29.0.
- [Release notes](https://github.com/googleapis/google-cloud-go/releases)
- [Changelog](https://github.com/googleapis/google-cloud-go/blob/main/CHANGES.md)
- [Commits](googleapis/google-cloud-go@container/v1.26.2...pubsub/v1.29.0)

---
updated-dependencies:
- dependency-name: cloud.google.com/go/container
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
@bboreham
Parca: update to 0.20.0
430e7ad 
Signed-off-by: Bryan Boreham <[email protected]>
@bboreham
Merge pull request prometheus#626 from prometheus/dependabot/go_modul…
cc5e017 
…es/cloud.google.com/go/container-1.29.0

Bump cloud.google.com/go/container from 1.26.2 to 1.29.0
kushalShukla-web and others added 28 commits November 24, 2024 16:54
@kushalShukla-web
Block-sync update. (prometheus#790)
1b766dc 
* Do nothing if secret is empty.
* add newline at bottom of file.
* Renamed key.yml to object-config.yml file.
* Replaced key with path and other parts of the file.

Signed-off-by: Kushal Shukla <[email protected]>
@prombot
Update common Prometheus files (prometheus#788)
a729fa3 
Signed-off-by: prombot <[email protected]>
@kushalShukla-web
Changed Upload functionality to upload TSDB data only.
5c51bcc 
Signed-off-by: Kushal Shukla <[email protected]>
@kushalShukla-web
Add remote-write to Prombench (prometheus#787)
bfde616 
* Created  Service and Deployment configuration for Prometheus sink
* 1. Added cpu and memory usage to remote-write.
2. make clusterIp headless.

Signed-off-by: Kushal Shukla <[email protected]>
@bboreham
Prombench: tweak rw-sink resource usage (prometheus#794)
55c5ba8 
Signed-off-by: Bryan Boreham <[email protected]>
@bboreham
Prombench: slow scraping from 5s to 15s
8755fc5 
More realistic for end-user workloads; less expensive in cloud resources.

Signed-off-by: Bryan Boreham <[email protected]>
@bboreham
Prombench: add Kubernetes resource requests
1a04212 
Best to tell the system what is needed.

Specific levels based on observing at GKE.

Signed-off-by: Bryan Boreham <[email protected]>
@kakkoyun
Merge pull request prometheus#797 from prometheus/add-requests
ce99a81 
Prombench: add Kubernetes resource requests
@kakkoyun
Merge pull request prometheus#796 from prometheus/slow-down-scraping
36f7726 
Prombench: slow scraping from 5s to 15s
@dependabot
Bump sigs.k8s.io/kind from 0.24.0 to 0.25.0 (prometheus#804)
93c4b25 
Bumps [sigs.k8s.io/kind](https://github.com/kubernetes-sigs/kind) from 0.24.0 to 0.25.0.
- [Release notes](https://github.com/kubernetes-sigs/kind/releases)
- [Commits](kubernetes-sigs/kind@v0.24.0...v0.25.0)

---
updated-dependencies:
- dependency-name: sigs.k8s.io/kind
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
@dependabot
Bump golang.org/x/oauth2 from 0.23.0 to 0.24.0 (prometheus#802)
aac370f 
Bumps [golang.org/x/oauth2](https://github.com/golang/oauth2) from 0.23.0 to 0.24.0.
- [Commits](golang/oauth2@v0.23.0...v0.24.0)

---
updated-dependencies:
- dependency-name: golang.org/x/oauth2
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
@bboreham
Merge pull request prometheus#786 from kushalShukla-web/upload
55c790d 
Changed Upload functionality to upload TSDB data only
@bwplotka
commentMonitor: Refactor, cleanup and rename to comment-monitor. (pro…
45cbb5a 
…metheus#805)

Some examples on how old one works (vs the unit test I added): prometheus/prometheus#15487

Signed-off-by: bwplotka <[email protected]>
@dependabot
Bump cloud.google.com/go/container from 1.41.0 to 1.42.0 (prometheus#803
6c8913b 
)

Bumps [cloud.google.com/go/container](https://github.com/googleapis/google-cloud-go) from 1.41.0 to 1.42.0.
- [Release notes](https://github.com/googleapis/google-cloud-go/releases)
- [Changelog](https://github.com/googleapis/google-cloud-go/blob/main/CHANGES.md)
- [Commits](googleapis/google-cloud-go@pubsub/v1.41.0...pubsub/v1.42.0)

---
updated-dependencies:
- dependency-name: cloud.google.com/go/container
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
@dependabot
Bump google.golang.org/api from 0.205.0 to 0.211.0 (prometheus#807)
ce00388 
Bumps [google.golang.org/api](https://github.com/googleapis/google-api-go-client) from 0.205.0 to 0.211.0.
- [Release notes](https://github.com/googleapis/google-api-go-client/releases)
- [Changelog](https://github.com/googleapis/google-api-go-client/blob/main/CHANGES.md)
- [Commits](googleapis/google-api-go-client@v0.205.0...v0.211.0)

---
updated-dependencies:
- dependency-name: google.golang.org/api
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
@bwplotka
prombench: Added support for --bench.version and `--bench.directory…
d2b0bb7 
…` flags. (prometheus#812)

* comment-monitor: Added flag support; added built-in help command.

Implements first phase of prometheus/proposals#41

Signed-off-by: bwplotka <[email protected]>

* prombench: Added support for `--bench.version` and `--bench.directory` flags.

See prometheus/proposals#41 for rationale.

Prometheus job got updated with prometheus/prometheus#15682

Signed-off-by: bwplotka <[email protected]>

* extended comment to include version and directory used.

Signed-off-by: bwplotka <[email protected]>

* addressed comments.

Signed-off-by: bwplotka <[email protected]>

* Makefile fix.

Signed-off-by: bwplotka <[email protected]>

* rm unnecessary print.

Signed-off-by: bwplotka <[email protected]>

* Improved help.

Signed-off-by: bwplotka <[email protected]>

* Fixed help.

Signed-off-by: bwplotka <[email protected]>

* Added missed test file.

Signed-off-by: bwplotka <[email protected]>

---------

Signed-off-by: bwplotka <[email protected]>
@bwplotka
Small fix to prombench Makefile to adjust for alpine docker image. (p…
9d4514a 
…rometheus#813)

No openssh installed, and fingerprint ask, so switching to HTTPs for git.

alpine mktemp does not have -t apparently too.

Signed-off-by: bwplotka <[email protected]>
@dependabot
Bump google.golang.org/grpc from 1.67.1 to 1.69.0 (prometheus#806)
7a6c108 
Bumps [google.golang.org/grpc](https://github.com/grpc/grpc-go) from 1.67.1 to 1.69.0.
- [Release notes](https://github.com/grpc/grpc-go/releases)
- [Commits](grpc/grpc-go@v1.67.1...v1.69.0)

---
updated-dependencies:
- dependency-name: google.golang.org/grpc
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
@bwplotka
Improved sink debugging flow. (prometheus#821)
4371345 
* Add PRW 2.0 traffic, improved sink debugging flow.

Signed-off-by: bwplotka <[email protected]>

* Removed PRW 2.0 as it depends on expensive metadata-wal-records feature.

Signed-off-by: bwplotka <[email protected]>

---------

Signed-off-by: bwplotka <[email protected]>
@bwplotka
fixed prometheus-builder:master causing npm install freeze on promben…
7263fd9 
…ch runs (prometheus#833)

This custom image is locally build from prometheus/golang-builder#296
that we agree to pause until Prometheus release. However, Prometheus release
needs working prombench. This is a tmp solution until we have an official image.

Mitigates prometheus#832

Signed-off-by: bwplotka <[email protected]>
@prombot
Update common Prometheus files (prometheus#814)
0b37928 
Signed-off-by: prombot <[email protected]>
@dependabot
Bump sigs.k8s.io/kind from 0.25.0 to 0.26.0 (prometheus#819)
a4e9134 
Bumps [sigs.k8s.io/kind](https://github.com/kubernetes-sigs/kind) from 0.25.0 to 0.26.0.
- [Release notes](https://github.com/kubernetes-sigs/kind/releases)
- [Commits](kubernetes-sigs/kind@v0.25.0...v0.26.0)

---
updated-dependencies:
- dependency-name: sigs.k8s.io/kind
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
@dependabot
Bump golang.org/x/net from 0.32.0 to 0.33.0 (prometheus#834)
3e86ef8 
Bumps [golang.org/x/net](https://github.com/golang/net) from 0.32.0 to 0.33.0.
- [Commits](golang/net@v0.32.0...v0.33.0)

---
updated-dependencies:
- dependency-name: golang.org/x/net
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
@kushalShukla-web
Added yaml file changes and prometheus builder changes that was removed
e49e1f6 
from block-sync.
Signed-off-by: Kushal Shukla <[email protected]>

Signed-off-by: Kushal Shukla <[email protected]>
@kushalShukla-web
Added --storage.tsdb.retention.size to retain data for certain size.
a493696 
Signed-off-by: Kushal Shukla <[email protected]>

Signed-off-by: Kushal Shukla <[email protected]>
@kushalShukla-web
Adding Bucket-config in configMap instead on prometheus Repository.
4c02a4f 
Signed-off-by: Kushal Shukla <[email protected]>

Signed-off-by: Kushal Shukla <[email protected]>
@kushalShukla-web
written an instruct to create bucket-config.yml file on Readme.
277b92b 
Signed-off-by: Kushal Shukla <[email protected]>

Signed-off-by: Kushal Shukla <[email protected]>
@kushalShukla-web
some new changes
8ec01c3 
Signed-off-by: Kushal Shukla <[email protected]>

replaced key with blocksync-location-key
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.