Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

HDDS-11512. Create Grafana dashboard for tracking system wide deletion #7813

Open
wants to merge 2 commits into
base: master
Choose a base branch
from

Conversation

Tejaskriya
Copy link
Contributor

@Tejaskriya Tejaskriya commented Feb 5, 2025

What changes were proposed in this pull request?

A system-wise deletion dashboard to track deletion progress is created in this PR. The existing dashboard is updated to include the below metrics:
SCM:
Deletion commands:

  • No. of commands sent (numBlockDeletionCommandSent)
  • No. of commands successful (numBlockDeletionCommandSuccess)
  • No. of commands failed (numBlockDeletionCommandFailure)

Delete transactions:

  • No. of transactions created (numBlockDeletionTransactionCreated)
  • No. of transactions sent (numBlockDeletionTransactionSent)
  • No. of transactions completed (numBlockDeletionTransactionCompleted)
  • No. of transactions successful (numBlockDeletionTransactionSuccess)
  • No. of transactions failed (numBlockDeletionTransactionFailure)
  • No. of transactions skipped (numSkippedTransactions)
  • Success percentage of delete (numBlockDeletionTransactionSuccess/numBlockDeletionTransactionSent*100)

DN:

  • No. of transactions received (receivedTransactionCount)
  • No. of successful transactions (block_deleting_service_metrics_processed_transaction_success_count)
  • No. of bytes successfully deleted (successBytes)
  • No. of block deletion request received (receivedBlockCount)
  • No. of blocks chosen for deletion (totalBlockChosenCount)
  • No. of blocks successfully deleted (successCount)
  • No. of blocks pending for processing (totalPendingBlockCount)
  • Success percentage of delete (successCount/receivedBlockCount*100)

OM:
KeyDeletion service:

  • Deleting_service_metrics_num_keys_processed
  • Deleting_service_metrics_num_keys_sent_for_purge

DirectoryDeletingService:

  • Deleting_service_metrics_num_dirs_sent_for_purge
  • Deleting_service_metrics_num_sub_files_sent_for_purge
  • Deleting_service_metrics_num_sub_dirs_sent_for_purge

PurgeRequest:

  • Deleting_service_metrics_num_keys_purged
  • Deleting_service_metrics_num_dirs_purged
  • Deleting_service_metrics_num_sub_files_moved_to_deleted_table
  • Deleting_service_metrics_num_sub_dirs_moved_to_deleted_dir_table

OpenKeyDeletion:

  • Om_metrics_num_open_keys_deleted
  • Om_metrics_num_open_keys_cleaned
  • Om_metrics_num_open_keys_h_sync_cleaned
  • Om_metrics_num_open_key_delete_requests
  • Om_metrics_num_open_key_delete_request_fails

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-11512

How was this patch tested?

Grafana with docker:
Screenshot 2025-02-05 at 1 29 38 PM
Screenshot 2025-02-05 at 1 37 06 PM
Screenshot 2025-02-05 at 1 38 47 PM
Screenshot 2025-02-05 at 1 38 14 PM
Screenshot 2025-02-05 at 1 39 30 PM
Screenshot 2025-02-05 at 1 39 53 PM
Screenshot 2025-02-05 at 1 44 34 PM
Screenshot 2025-02-05 at 1 45 35 PM
Screenshot 2025-02-05 at 1 45 54 PM

Copy link
Contributor

@aryangupta1998 aryangupta1998 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the patch @Tejaskriya, LGTM!

@ashishkumar50
Copy link
Contributor

@Tejaskriya Thanks for the patch, pls find comments below:

  1. OpenKeyDeletion: KeysDeleted and KeysCleaned, do we need two dashboards here? what is the difference?
  2. DataNode graph, Y-axis is wrong in "No of bytes successfully deleted".
  3. SCM graph, Y-axis names are missing in "No of deletion transactions created" , "No of failed deletion transactions".

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants