[BUG] API deployment to UAT (next env) fails for new operations after upgrading to v6.0.1.6

[megamax34]

Release version

v6.0.1.6

Describe the bug

I attempted to publish an API from Dev to UAT that included several new operations. The API already existed in UAT with a few operations previously deployed. I had added new operations in Dev and wanted to push those changes through.

Originally, I published this API to UAT using version 5.1.4 of the extractor/publisher pipeline. To stay current, I upgraded the pipeline to v6.0.1.6. The extractor ran successfully, and the publisher worked fine when targeting Dev. However, when publishing to UAT, everything worked until it tried to deploy the new operations.

The existing operations deployed without issue, but the new ones failed with the following error:

info: publisher[0]
Putting policy policy for operation scrubbed-already-there in API scrubbed-v1...
info: publisher[0]
Putting policy policy for operation scrubbed-new in API scrubbed-v1...
crit: publisher[0]
Application failed.
System.Net.Http.HttpRequestException: HTTP request to URI https://management.azure.com/subscriptions/***/resourceGroups/scrubbed/providers/Microsoft.ApiManagement/service/apimuatscrubbed/apis/scrubbed-v1/operations/scrubbed-new/policies/policy?api-version=2023-09-01-preview failed with status code 400. Content is '{"error":{"code":"ValidationError","message":"Entity with specified identifier not found","details":null}}'.
at common.HttpPipelineExtensions.<>c__DisplayClass11_0.b__0(Response response) in /home/runner/work/apiops/apiops/tools/code/common/Http.cs:line 173
at LanguageExt.Either.Left2.Match[B](Func2 Left, Func2 Right) at LanguageExt.Either2.IfLeft(Func`2 leftMap)
at common.HttpPipelineExtensions.PutContent(HttpPipeline pipeline, Uri uri, BinaryData content, CancellationToken cancellationToken) in /home/runner/work/apiops/apiops/tools/code/common/Http.cs:line 173
at common.ApiOperationPolicyModule.PutDto(ApiOperationPolicyUri uri, ApiOperationPolicyDto dto, HttpPipeline pipeline, CancellationToken cancellationToken) in /home/runner/work/apiops/apiops/tools/code/common/ApiOperationPolicy.cs:line 132
at publisher.ApiOperationPolicyModule.<>c__DisplayClass11_0.<b__0>d.MoveNext() in /home/runner/work/apiops/apiops/tools/code/publisher/ApiOperationPolicy.cs:line 181

The error message I’m seeing might not reflect the actual root cause — I’m not sure why it would attempt to reach that URL when the operation hasn’t even been deployed there yet.

The spec file is relatively short, and the policies are simple. There are no validation errors, and all namedValues referenced exist in the target environment.

I reviewed issues #329 , #339 , #513 , and #553 and tried all suggested workarounds except manually exporting the API spec from Dev and importing it into UAT — which defeats the purpose of using DevOps automation.

I believe #513 and @daviian are on the right track, and the issue discussed there may still be unresolved.

As a test, I reverted the extractor and publisher pipelines back to v5.1.4, re-extracted the API and all related resources, and published again. This time, everything worked as expected.

For context, my API is a REST API with about six operations and fairly basic policies.

Expected behavior

The publisher should have published the new operations to the api without error.

Actual behavior

Originally, I published this API to UAT using version 5.1.4 of the extractor/publisher pipeline. To stay current, I upgraded the pipeline to v6.0.1.6. The extractor ran successfully, and the publisher worked fine when targeting Dev. However, when publishing to UAT, everything worked until it tried to deploy the new operations.

The existing operations deployed without issue, but the new ones failed with the following error:

info: publisher[0]
Putting policy policy for operation scrubbed-already-there in API scrubbed-v1...
info: publisher[0]
Putting policy policy for operation scrubbed-new in API scrubbed-v1...
crit: publisher[0]
Application failed.
System.Net.Http.HttpRequestException: HTTP request to URI https://management.azure.com/subscriptions/***/resourceGroups/scrubbed/providers/Microsoft.ApiManagement/service/apimuatscrubbed/apis/scrubbed-v1/operations/scrubbed-new/policies/policy?api-version=2023-09-01-preview failed with status code 400. Content is '{"error":{"code":"ValidationError","message":"Entity with specified identifier not found","details":null}}'.
at common.HttpPipelineExtensions.<>c__DisplayClass11_0.b__0(Response response) in /home/runner/work/apiops/apiops/tools/code/common/Http.cs:line 173
at LanguageExt.Either.Left2.Match[B](Func2 Left, Func2 Right) at LanguageExt.Either2.IfLeft(Func`2 leftMap)
at common.HttpPipelineExtensions.PutContent(HttpPipeline pipeline, Uri uri, BinaryData content, CancellationToken cancellationToken) in /home/runner/work/apiops/apiops/tools/code/common/Http.cs:line 173
at common.ApiOperationPolicyModule.PutDto(ApiOperationPolicyUri uri, ApiOperationPolicyDto dto, HttpPipeline pipeline, CancellationToken cancellationToken) in /home/runner/work/apiops/apiops/tools/code/common/ApiOperationPolicy.cs:line 132
at publisher.ApiOperationPolicyModule.<>c__DisplayClass11_0.<b__0>d.MoveNext() in /home/runner/work/apiops/apiops/tools/code/publisher/ApiOperationPolicy.cs:line 181

The error message I’m seeing might not reflect the actual root cause — I’m not sure why it would attempt to reach that URL when the operation hasn’t even been deployed there yet.

The spec file is relatively short, and the policies are simple. There are no validation errors, and all namedValues referenced exist in the target environment.

When reverting back to v5.1.4 everything worked fine.

Reproduction Steps

1. Create a basic REST api in lower env (dev)
2. Create a few operations in that api
3. Extract that api and anything that pertains to it using the pipeline (v6.0.1.6)
4. (may want to skip to step 8 but continuing here might trigger a failure too) Publish that artifact to the next env (uat)
5. This should fail but in my case I used an older version of the extractor and publisher (v5.1.4) the first time they went to uat so the api and 2 operations were already there.
6. If it didnt fail already add two new operations in dev for that api.
7. extract and publish again and it should fail only on the new operations
8. To set this up better you may want to manually extract the spec file and import it into uat instead of running the publisher in step 4. then start at step 6 once dev and uat line up.

[github-actions]

  Thank you for opening this issue! Please be patient while we will look into it and get back to you as this is an open source project. In the meantime make sure you take a look at the [closed issues](https://github.com/Azure/apiops/issues?q=is%3Aissue+is%3Aclosed) in case your question has already been answered. Don't forget to provide any additional information if needed (e.g. scrubbed logs, detailed feature requests,etc.).
  Whenever it's feasible, please don't hesitate to send a Pull Request (PR) our way. We'd greatly appreciate it, and we'll gladly assess and incorporate your changes.

[kiddynomite]

I am experiencing the exact same issue as @megamax34 - same error, same steps to reproduce, etc. The only solution I have found is to roll back to the previous version, v5.1.4. I see this hasn't been addressed in quite some time (Aug 22), has there been any movement on this?