Resume from checkpoint with distributed optimizer-in-backward repro

[ebsmothers]

This is just a repro for the bug described in #2360

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/torchtune/2359

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 New Failure, 2 Unrelated Failures

As of commit 2aed52c with merge base 9b38360 ():

NEW FAILURE - The following job has failed:

• GPU tests / gpu_test (3.11, stable) (gh)
  tests/recipes/test_full_finetune_distributed.py::TestFullFinetuneDistributedRecipe::test_training_state_on_resume[llama3/8B_full-llama3-tune-4-1-True]

BROKEN TRUNK - The following jobs failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

• GPU tests / gpu_test (3.10, stable) (gh) (trunk failure)
  ##[error]The operation was canceled.
• GPU tests / gpu_test (3.9, stable) (gh) (trunk failure)
  ##[error]The operation was canceled.

This comment was automatically generated by Dr. CI and updates every 15 minutes.