[Question] Using configuration YAML in v7.0.0-alpha.1.0.2

[Remo-L5]

Release version

v7.0.0-alpha.1.0.2

Question Details

I have reviewed the wiki and it shows that the CONFIGURATION_YAML_PATH should be the absolute path for the configuration YAML file.

I have purposely configured all of my artifacts to point to reference app insights, keyvaults, and backends that do not exist in a 'dev' environment to ensure that the configuration yaml is being used to override the values and point to the 'test' environment resources that do exist. However, that is not the case.

Expected behavior

The publisher tool should be overriding the references pointing from my dev environment with the references to the test environment specified in the configuration YAML file.

Actual behavior

The publisher tool is making API calls to apim with references to the dev environment and is not pulling in the configuration yaml file. I added steps to my pipeline to print out the contents to confirm the absolute path works.

Reproduction Steps

Pipeline Config for Config File display task:

Pipeline Config for publish task:

[github-actions]

  Thank you for opening this issue! Please be patient while we will look into it and get back to you as this is an open source project. In the meantime make sure you take a look at the [closed issues](https://github.com/Azure/apiops/issues?q=is%3Aissue+is%3Aclosed) in case your question has already been answered. Don't forget to provide any additional information if needed (e.g. scrubbed logs, detailed feature requests,etc.).
  Whenever it's feasible, please don't hesitate to send a Pull Request (PR) our way. We'd greatly appreciate it, and we'll gladly assess and incorporate your changes.

[guythetechie]

@Remo-L5 - in #197 , users were setting the CONFIGURATION_YAML_PATH in their ADO variable group, and this was overriding what was defined in the pipeline. Can you make sure you don't have it defined in both?

[Remo-L5]

@Remo-L5 - in #197 , users were setting the CONFIGURATION_YAML_PATH in their ADO variable group, and this was overriding what was defined in the pipeline. Can you make sure you don't have it defined in both?

@guythetechie I am not using variable groups in my configuration. I am just passing the values in as pipeline parameters instead.

extends:
  template: .pipelines/cd-template.yaml@templates
  parameters:
    projects:
      - name: ApiOps
        projectType: api-ops
        projectSpecs:
          serviceConnectionName: 'sc-alz-mgmt-apply'
          environment: 'test'
          resourceGroupName: 'rg-arsnl-test-westus3-app-01'
          apimServiceName: 'rll-apim'
          apimServiceOutputFolderPath: 'artifacts'
          configurationFilePath: 'configuration/configuration.test.yaml'

EDIT*

When I actually pass in these parameters it looks like this and provides the absolute path.

  - task: PowerShell@2
    displayName: Run publisher for ${{ parameters.environment}} environment
    inputs:
      targetType: "inline"
      script: |
        Set-StrictMode -Version Latest
        $ErrorActionPreference = "Stop"
        $VerbosePreference = "Continue"
        $InformationPreference = "Continue"

        & "$(PUBLISHER_FILE_PATH)"
        if ($LASTEXITCODE -ne 0) { throw "Running publisher failed."}

        Write-Information "Execution complete."
      failOnStderr: true
      pwsh: true
    env:
      API_SPECIFICATION_FORMAT: ${{ parameters.configurationFileFormat }}
      AZURE_BEARER_TOKEN: $(AZURE_BEARER_TOKEN)
      AZURE_CLIENT_ID: $(AZURE_CLIENT_ID)
      AZURE_CLIENT_SECRET: $(AZURE_CLIENT_SECRET)
      AZURE_TENANT_ID: $(AZURE_TENANT_ID)
      AZURE_SUBSCRIPTION_ID: $(AZURE_SUBSCRIPTION_ID)
      AZURE_RESOURCE_GROUP_NAME: ${{ parameters.resourceGroupName }}
      API_MANAGEMENT_SERVICE_OUTPUT_FOLDER_PATH: $(Build.SourcesDirectory)/${{ parameters.apimServiceOutputFolderPath }}
      API_MANAGEMENT_SERVICE_NAME: ${{ parameters.apimServiceName }}
      CONFIGURATION_YAML_PATH: $(Build.SourcesDirectory)/${{ parameters.apimServiceOutputFolderPath }}/${{ parameters.configurationFilePath }}

[NikolaiKleppe]

We have encountered similar issues with configuration files not being picked up, I'm curious our if "workaround" applies here.

It might have something to do with the COMMIT_ID parameter in the PowerShell@2 task, which defaults to publish-artifacts-in-last-commit.

I mention this because it seems that the configuration files are not picked up unless that specific commit actually contains a change to the API/artifact you are trying to publish/promote - Is this the case here?

If you change something on the specific artifact the publisher should pick up the configuration file, as the last commit now contains a relevant change.
If you have published everything already and then try to make changes to the configuration files, nothing happens.

Alternatively try out the publish-all-artifacts-in-repo value, just be careful as it overwrites everything.

[egiordano-cf]

I think we may have just run into this today. Made a change only to the configuration files to update some named values and the publisher did nothing running against the last commit.

[Remo-L5]

Here is what I found:

• the artifacts folder needs to be "named values" and so does the key name in the configuration YAML file.
• For the diagnostics, I was using a custom named diagnostic artifact folder not realizing only 'azuremonitor' and 'applicationinsights' are supported so I had to update the folder to match those names.
• in the example config file, the attribute 'properties' is missing and I just copied that as a starter template and that is why my API properties were not overriding.
• To get better output for debugging and seeing what the actual override were doing I added the environment variable Logging__LogLevel__Default: "Trace"

named values artifacts:

named values configuration override key:

Here is my updated configuration YAML file:

apimServiceName: you-apim-service-name
named values:
  - name: contoso-hotel-function-key-v1
    properties:
      displayName: "your-app-details-key"
      keyVault:
        identityClientId: "your-managed-identity-client-id"
        secretIdentifier: "https://your-vault-name.vault.azure.net/secrets/your-secret-name"
      secret: true
  - name: logger-contoso-its
    properties:
      displayName: "logger-contoso-its"
      keyVault:
        identityClientId: "your-managed-identity-client-id"
        secretIdentifier: "https://your-vault-name.vault.azure.net/secrets/your-secret-name"
      secret: true
backends:
  - name: contoso-hotel-functionapp-v1
    properties:
      url: "https://your-app-details.azurewebsites.net/api"
      resourceId: "https://management.azure.com/your-subscription-id/resourceGroups/your-resource-group-name/providers/Microsoft.Web/sites/your-app-details"
      credentials:
        header:
          "x-functions-key": [
            "{{contoso-hotel-function-key-v1}}"
          ]
diagnostics:
  - name: contoso-its
    properties:
      verbosity: verbose
      alwaysLog: true
      loggerId: "/loggers/contoso-its"
      logClientIp: true
      sampling:
        percentage: 100
        samplingType: fixed
      httpCorrelationProtocol: Legacy
      frontend:
        request:
          headers:
            - "Accept"
            - "Accept-Encoding"
            - "User-Agent"
          body:
            bytes: 512
          dataMasking:
            queryParams:
            - mode: "Mask"
              value: "queryParamNameGoesHere"
loggers:
  - name: contoso-its
    properties:
      description: "Logger override for Contoso IT Service Application Insights"
      credentials:
        instrumentationKey: "{{logger-contoso-its}}"
      loggerType: "applicationInsights"
      isBuffered: true
      resourceId: "/subscriptions/your-subscription-id/resourceGroups/your-resource-group-name/providers/microsoft.insights/components/your-app-details"
apis:
  - name: contoso-hotel-v1
    properties:
      isCurrent: true
      serviceUrl: "https://your-app-details.azurewebsites.net"
      diagnostics:
        - name: contoso-its
          properties:
            verbosity: Error
            loggerId: "/loggers/contoso-its"

THANK YOU @guythetechie for the assistance!!

[Remo-L5]

I mention this because it seems that the configuration files are not picked up unless that specific commit actually contains a change to the API/artifact you are trying to publish/promote - Is this the case here?

@NikolaiKleppe I think that is the case from my understanding, the artifact needs to change but it may be best to get confirmation from someone more knowledgeable like @guythetechie. It would be nice if a change to an artifact in the config file trigger that resource's re-deployment since that could be the case for making an update for higher environments only.