fix ssec: missing base64 encoding and md5 hash

[gschei]

This PR is based on #224 and replaces PR #217 (which has some code duplication and is not complete).

It fixes these issues:

• the SSEC-CustomerKey is stored in the Velero Container as plain 32 byte key and must be base64 encoded before sending it to AWS.
• as shown in PR 217 MD5 hash is missing for head/get/put operations

I tested this change successfully with an encrypted backup and restore with Velero and S3 Bucket.

[tonobo]

I'm pretty new to velero, i tried this mr locally and noticed there are issues while accessing the logs and futher contents from the bucket, probably caused by not passing the sse-c headers. am i holding it wrong or is there still something missing.

[kaovilai]

Is this necessary? This will mean the log trace won't propagate to velero pod logs with stack trace

[gschei]

The point of the fix is that err is always nil in this situation and errors.Wrapf here always returns nil, so to velero looks like key was read successfully (but is ""). I encountered this situation during testing, having a key < 32 bytes and no errors showed up in velero (but overall encryption did not work).

With this change, now the error is shown in the Backup-Resource correctly (but no stacktrace though).

[kaovilai]

ahh.. got it. Make sense. Here's a revised suggestion.

Suggested change

	return "", fmt.Errorf("contents of %s (%s) are not exactly 32 bytes", customerKeyEncryptionFileKey, customerKeyEncryptionFile)
	return "", errors.Errorf("contents of %s (%s) are not exactly 32 bytes", customerKeyEncryptionFileKey, customerKeyEncryptionFile)

[gschei]

thanks..PR updated

[gschei]

I'm pretty new to velero, i tried this mr locally and noticed there are issues while accessing the logs and futher contents from the bucket, probably caused by not passing the sse-c headers. am i holding it wrong or is there still something missing.

To test it successfully, you must mount the encryption key into the velero container (eg. from a secret) and then reference the file containing the key in the config of the BackupStorageLocation. See description here: https://github.com/vmware-tanzu/velero-plugin-for-aws/blob/main/backupstoragelocation.md.

I see now that the description says "<your_b64_encoded_32byte_string>" which is wrong, the key must be plain and is then encoded to base64 before sending it.

[tonobo]

@gschei you're right that's not my concern. I get it to work after checking the actual code not the problem, but once i enable the encryption part i'm unable to call velero backup logs for example. I always get an common error message as it's not passing the credentials somewhere. Local and within the container.

[tonobo]

EDIT: Nevermind, handled totally differnent

besides that, the internal kopia data mover seems to have trouble as well.

 data path backup failed: Failed to run data path service for DataUpload
    daily-pvc-backup-20241124210813-lkshf: error to initialize data path: error to
    boost backup repository connection backups-apps-kopia: error to connect backup
    repo: error to connect repo with storage: error to connect to repository: repository
    not initialized in the provided storage'

Can you confirm csi data movement works with sse-c backup locations?

P.S by dropping the sse-c key at all, it just works.

[gschei]

@gschei you're right that's not my concern. I get it to work after checking the actual code not the problem, but once i enable the encryption part i'm unable to call velero backup logs for example. I always get an common error message as it's not passing the credentials somewhere. Local and within the container.

Yes I see velero backup logs creates a DownloadRequest Resources which does not contain the Encryption Headers. Looks like object_store.CreateSignedURL needs to be extended too with Encryption Parameters.

[gschei]

It looks like AWS getSignedUrl dies not support SSECustomerKey, see https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html#getSignedUrl-property

Note:

Not all operation parameters are supported when using pre-signed URLs. Certain parameters, such as SSECustomerKey, ACL, Expires, ContentLength, or Tagging must be provided as headers when sending a request. If you are using pre-signed URLs to upload from a browser and need to use these fields, see createPresignedPost().

[kaovilai]

getSignedUrl

is currently required until vmware-tanzu/velero#6167 gets implemented.

[gschei]

One additional remark regarding the encryption key: original implementation ac31cd9 (from feb/22) stated that the key should be a "32 byte string". Later (oct/23) a863977 a documentation update was done which apparently was wrong because it contradicts the original statement, saying that the key should now be a "base64 encoded 32 byte string" which is not correct, because within the code only the first 32 bytes are read anyway (and base64 endoding will make the string longer).

From my point of view current implementation is not working at the moment - once due to missing base64 (or alternatively correct parsing of base64 encoded key) and also due to missing md5 hash - a problem which appeared later.

velero backup logs statement is generally not working for encrypted backups and will also not work after merging this PR. But creation and restore of encrypted backups would work now.

@kaovilai @reasonerjt Could you please have a look at this again and merge if OK?

[kaovilai]

Please update readme and or backupstoragelocation.md to reflect this PR or the current state of using customerKey since you're probably most versed in how to use this PR as is.

[kaovilai]

velero-plugin-for-aws/backupstoragelocation.md

Lines 76 to 87 in 2dd6bcb

	# Specify the file that contains the SSE-C customer key to enable customer key encryption of the backups
	# stored in S3. The referenced file should contain a 32-byte string.
	#
	# The customerKeyEncryptionFile points to a mounted secret within the velero container.
	# Add the below values to the velero cloud-credentials secret:
	# customer-key: <your_b64_encoded_32byte_string>
	# The default value below points to the already mounted secret.
	#
	# Cannot be used in conjunction with kmsKeyId.
	#
	# Optional (defaults to "", which means SSE-C is disabled).
	customerKeyEncryptionFile: "/credentials/customer-key"

Is this accurate as is?

[gschei]

@kaovilai I clarified the description. The point is, when I base64 encode the key within the kubernetes secret, it will be automatically base64 decoded by kubernetes when mounted into the container.

[kaovilai]

This lgtm. Just some clarifications. Thanks for the PR!

[reasonerjt]

lgtm, thanks