Unable to deleted support bundles in "Failed" state

[wfchandler]

On dogfood we have a large number of support bundles in Failed state:

$ oxide --profile=dogfood bundle list
[
  {
    "id": "62180262-5748-43fe-932f-cd90f3d2fa1f",
    "reason_for_creation": "Created by external API",
    "reason_for_failure": "Nexus managing this bundle no longer exists",
    "state": "failed",
    "time_created": "2025-02-28T18:45:18.306219Z"
  }, {
    "id": "acfdd1ac-9ccb-42a7-a4a8-e8c4dbea6ce2",
    "reason_for_creation": "Created by internal API",
    "reason_for_failure": "Nexus managing this bundle no longer exists",
    "state": "failed",
    "time_created": "2025-05-01T20:07:18.363585Z"
  }
...

Attempting to delete these bundles fails with the following:

$ oxide --debug--profile=dogfood bundle delete --bundle-id 62180262-5748-43fe-932f-cd90f3d2fa1f
{"timestamp":"2025-12-22T15:15:30.203547Z","level":"DEBUG","message":"request succeeded","url":"https://oxide.sys.rack2.eng.oxide.computer/experimental/v1/system/support-bundles/62180262-5748-43fe-932f-cd90f3d2fa1f","path":"/experimental/v1/system/support-bundles/62180262-5748-43fe-932f-cd90f3d2fa1f","operation_id":"support_bundle_delete","remote_addr":"172.20.26.9:443","http.request.method":"DELETE","http.response.content_length":169,"http.response.status_code":400,"start_time":"2025-12-22T15:15:29.355531Z","duration_ms":847,"oxide.request_id":"2fa9cdc2-f26b-40c0-baa6-5767180f784e","target":"oxide"}
error
Error Response: status: 400 Bad Request; headers: {"content-type": "application/json", "x-request-id": "2fa9cdc2-f26b-40c0-baa6-5767180f784e", "content-length": "169", "date": "Mon, 22 Dec 2025 15:15:30 GMT"}; value: Error { error_code: Some("InvalidRequest"), message: "Cannot update support bundle state from Failed to Destroying", request_id: "2fa9cdc2-f26b-40c0-baa6-5767180f784e" }

The chain of operations is:

• NexusExternalApi::support_bundle_delete
• Nexus::support_bundle_delete
• Datastore::support_bundle_update

This final function fails because it is attempting to update the bundle state to Destroying, but Failed is a terminal state. Datastore::support_bundle_delete would succeed for these bundles, but we never reach that point.

Based on the "Nexus managing this bundle no longer exists" message, these bundles should have transitioned to Failing state, so it's not immediately obvious why the SupportBundleCollector background task didn't remove them, as it checks for items in Destroying and Failing state.