Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Concurrent jobs seem to compete for CPU #66

Open
webb-ben opened this issue Dec 23, 2024 · 0 comments
Open

Concurrent jobs seem to compete for CPU #66

webb-ben opened this issue Dec 23, 2024 · 0 comments
Assignees

Comments

@webb-ben
Copy link
Member

webb-ben commented Dec 23, 2024

When running Dagster on GCP, the CPU load seems to be shared round robin instead of consistently committed to a job. As in two jobs will switch between 100% and 0% of CPU load instead of sharing the loads at 50%. Have been testing with some dagster configurations that better allocate the CPU/memory options of a spawned container.

    container_kwargs:
      mem_limit: "2g" 
      cpuset_cpus: "1-2"
      cpu_shares: 512

Note this is running with 8 concurrent runs, but with 48 CPU /64+cores

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants