Add Secure Secret Management Support (Vault, AWS, GCP)

[yohamta]

Overview
We want to enhance Dagu to securely manage secrets by retrieving them from Vault, AWS Secrets Manager, or GCP Secret Manager. Since these devices can be physically stolen, our goal is to ensure secrets are never stored in plain text on the device and are only loaded at execution time with minimal exposure.

Proposal

1. Secret Providers Configuration

   • Extend DSL to include a secretProviders section for Vault, AWS, and GCP.
   • Example snippet:

     secretProviders:
       vault:
         address: "${VAULT_ADDR}"
         token: "${VAULT_TOKEN}"
       aws:
         region: "${AWS_REGION}"
         accessKeyId: "${AWS_ACCESS_KEY_ID}"
         secretAccessKey: "${AWS_SECRET_ACCESS_KEY}"
       gcp:
         credentialsJSONPath: "${GCP_CREDENTIALS_JSON}"

2. DAG File secrets: Section

   • Introduce a secrets: block in each DAG to declare which provider to use, the secret’s path/ARN, and an internal reference name.
   • On execution, Dagu retrieves the requested secrets and injects them as temporary environment variables for the relevant steps.
   • Example snippet:

     secrets:
       - name: DB_PASSWORD
         source: vault
         path: "secret/data/db_credentials"
         key: "password"
       - name: AWS_KEY
         source: aws
         arn: "arn:aws:secretsmanager:us-east-1:123456789012:secret:myToken-xxxx"
     
     steps:
       - name: migrate
         command: ./migrate.sh --db-pass "$DB_PASSWORD"
       - name: upload
         command: aws s3 cp data.out s3://my-bucket/ --access-key "$AWS_KEY"
         depends: migrate

3. Security Considerations

   • No Plain Text Logging: Ensure logs/UI never expose secret values, potentially masking them.

Any feedbacks on the design, or thoughts are very welcomed.