credential behavior used by codedeploy-agent differed depending on Amazon Linux version

[plane11]

Summary

The presence or absence of a IAM credential file on the instance where the CodeDeploy Agent is installed produces different results.

Environment

Common

• CodeDeploy Agent : OFFICIAL_1.7.0-92_rpm

AMI
(There is no difference in the detailed version. It is the same even if you use the latest version.)

• AL2 : ami-01fccab91b456acc2 (al2023-ami-2023.5.20240708.0-kernel-6.1-x86_64)
• AL2023 : ami-0b72821e2f351e396 (amzn2-ami-kernel-5.10-hvm-2.0.20240709.1-x86_64-gp2)

Steps

1. install codedeploy-agent successfully with Instance Profile, Agent running successfully and Deployment success
2. stop agent
3. switch user(sudo su -) and set IAM credential with aws configure with dummy access info for AccessDenied)
4. start agent

Result

Amazon Linux 2
Agent running successfully with Instance Profile without any Exceptions

2024-07-22T11:22:41 INFO  [codedeploy-agent(3277)]: master 3277: Spawned child 1/1
2024-07-22T11:22:41 DEBUG [codedeploy-agent(3281)]: Registering Plugins: ["codedeploy"].
2024-07-22T11:22:41 DEBUG [codedeploy-agent(3281)]: Loading plugin codedeploy from /opt/codedeploy-agent/lib/instance_agent/plugins/codedeploy/register_plugin
2024-07-22T11:22:42 DEBUG [codedeploy-agent(3281)]: Registered Plugins: #<Set: {InstanceAgent::Plugins::CodeDeployPlugin::CommandPoller}>.
2024-07-22T11:22:42 INFO  [codedeploy-agent(3281)]: On Premises config file does not exist or not readable
2024-07-22T11:22:42 DEBUG [codedeploy-agent(3281)]: InstanceAgent::Plugins::CodeDeployPlugin::CommandPoller: Configuring deploy control client: Region="us-east-1"
2024-07-22T11:22:42 DEBUG [codedeploy-agent(3281)]: InstanceAgent::Plugins::CodeDeployPlugin::CommandPoller: Deploy control endpoint override=
2024-07-22T11:22:42 DEBUG [codedeploy-agent(3281)]: InstanceAgent::Plugins::CodeDeployPlugin::CommandPoller: Enable auth policy = false
2024-07-22T11:22:42 INFO  [codedeploy-agent(3281)]: Creating client url from IMDS region and domain
2024-07-22T11:22:42 INFO  [codedeploy-agent(3281)]: CodeDeploy endpoint: https://codedeploy-commands.us-east-1.amazonaws.com
2024-07-22T11:22:42 INFO  [codedeploy-agent(3281)]: InstanceAgent::Plugins::CodeDeployPlugin::CommandExecutor: Archives to retain is: 5}
2024-07-22T11:22:42 DEBUG [codedeploy-agent(3281)]: InstanceAgent::Plugins::CodeDeployPlugin::CommandPoller: Initializing Host Agent: Host Identifier = arn:aws:ec2:us-east-1:482009018293:instance/i-04b2a2497a9fe5409
2024-07-22T11:22:42 DEBUG [codedeploy-agent(3281)]: InstanceAgent::Plugins::CodeDeployPlugin::CommandPoller: Validating CodeDeploy Plugin Configuration
2024-07-22T11:22:42 INFO  [codedeploy-agent(3281)]: Creating client url from IMDS region and domain
2024-07-22T11:22:42 INFO  [codedeploy-agent(3281)]: CodeDeploy endpoint: https://codedeploy-commands.us-east-1.amazonaws.com
2024-07-22T11:22:42 INFO  [codedeploy-agent(3281)]: Creating client url from IMDS region and domain
2024-07-22T11:22:42 INFO  [codedeploy-agent(3281)]: CodeDeploy endpoint: https://codedeploy-commands.us-east-1.amazonaws.com
2024-07-22T11:22:42 DEBUG [codedeploy-agent(3281)]: Current deploy control endpoint: https://codedeploy-commands.us-east-1.amazonaws.com
2024-07-22T11:22:42 DEBUG [codedeploy-agent(3281)]: InstanceAgent::Plugins::CodeDeployPlugin::CommandPoller: CodeDeploy Plugin Configuration is valid
2024-07-22T11:22:42 DEBUG [codedeploy-agent(3281)]: InstanceAgent::Plugins::CodeDeployPlugin::CommandPoller: Calling PollHostCommand:
2024-07-22T11:22:42 INFO  [codedeploy-agent(3281)]: Version file found in /opt/codedeploy-agent/.version with agent version OFFICIAL_1.7.0-92_rpm.
2024-07-22T11:22:42 INFO  [codedeploy-agent(3277)]: Started master 3277 with 1 children
2024-07-22T11:23:28 DEBUG [codedeploy-agent(3281)]: InstanceAgent::Plugins::CodeDeployPlugin::CommandPoller: PollHostCommand: Host Command =  nil
2024-07-22T11:23:29 DEBUG [codedeploy-agent(3281)]: InstanceAgent::Plugins::CodeDeployPlugin::CommandPoller: Calling PollHostCommand:
2024-07-22T11:23:29 INFO  [codedeploy-agent(3281)]: Version file found in /opt/codedeploy-agent/.version with agent version OFFICIAL_1.7.0-92_rpm.

Amazon Linux 2023
Agent has AccessDenied

2024-07-22T10:52:57 INFO  [codedeploy-agent(26949)]: master 26949: Spawned child 1/1
2024-07-22T10:52:57 DEBUG [codedeploy-agent(26951)]: Registering Plugins: ["codedeploy"].
2024-07-22T10:52:57 DEBUG [codedeploy-agent(26951)]: Loading plugin codedeploy from /opt/codedeploy-agent/lib/instance_agent/plugins/codedeploy/register_plugin
2024-07-22T10:52:58 DEBUG [codedeploy-agent(26951)]: Registered Plugins: #<Set: {InstanceAgent::Plugins::CodeDeployPlugin::CommandPoller}>.
2024-07-22T10:52:58 INFO  [codedeploy-agent(26951)]: On Premises config file does not exist or not readable
2024-07-22T10:52:58 DEBUG [codedeploy-agent(26951)]: InstanceAgent::Plugins::CodeDeployPlugin::CommandPoller: Configuring deploy control client: Region="us-east-1"
2024-07-22T10:52:58 DEBUG [codedeploy-agent(26951)]: InstanceAgent::Plugins::CodeDeployPlugin::CommandPoller: Deploy control endpoint override=
2024-07-22T10:52:58 DEBUG [codedeploy-agent(26951)]: InstanceAgent::Plugins::CodeDeployPlugin::CommandPoller: Enable auth policy = false
2024-07-22T10:52:58 INFO  [codedeploy-agent(26951)]: Creating client url from IMDS region and domain
2024-07-22T10:52:58 INFO  [codedeploy-agent(26951)]: CodeDeploy endpoint: https://codedeploy-commands.us-east-1.amazonaws.com
2024-07-22T10:52:58 INFO  [codedeploy-agent(26951)]: InstanceAgent::Plugins::CodeDeployPlugin::CommandExecutor: Archives to retain is: 5}
2024-07-22T10:52:58 DEBUG [codedeploy-agent(26951)]: InstanceAgent::Plugins::CodeDeployPlugin::CommandPoller: Initializing Host Agent: Host Identifier = arn:aws:ec2:us-east-1:482009018293:instance/i-03b839d4f08f2691a
2024-07-22T10:52:58 DEBUG [codedeploy-agent(26951)]: InstanceAgent::Plugins::CodeDeployPlugin::CommandPoller: Validating CodeDeploy Plugin Configuration
2024-07-22T10:52:58 INFO  [codedeploy-agent(26951)]: Creating client url from IMDS region and domain
2024-07-22T10:52:58 INFO  [codedeploy-agent(26951)]: CodeDeploy endpoint: https://codedeploy-commands.us-east-1.amazonaws.com
2024-07-22T10:52:58 INFO  [codedeploy-agent(26951)]: Creating client url from IMDS region and domain
2024-07-22T10:52:58 INFO  [codedeploy-agent(26951)]: CodeDeploy endpoint: https://codedeploy-commands.us-east-1.amazonaws.com
2024-07-22T10:52:58 DEBUG [codedeploy-agent(26951)]: Current deploy control endpoint: https://codedeploy-commands.us-east-1.amazonaws.com
2024-07-22T10:52:58 DEBUG [codedeploy-agent(26951)]: InstanceAgent::Plugins::CodeDeployPlugin::CommandPoller: CodeDeploy Plugin Configuration is valid
2024-07-22T10:52:58 DEBUG [codedeploy-agent(26951)]: InstanceAgent::Plugins::CodeDeployPlugin::CommandPoller: Calling PollHostCommand:
2024-07-22T10:52:58 INFO  [codedeploy-agent(26951)]: Version file found in /opt/codedeploy-agent/.version with agent version OFFICIAL_1.7.0-92_rpm.
2024-07-22T10:52:58 ERROR [codedeploy-agent(26951)]: InstanceAgent::Plugins::CodeDeployPlugin::CommandPoller: Error polling for host commands: Aws::CodeDeployCommand::Errors::AccessDeniedException - Aws::CodeDeployCommand::Errors::AccessDeniedException - /opt/codedeploy-agent/vendor/gems/aws-sdk-core-3.121.1/lib/seahorse/client/plugins/raise_response_errors.rb:17:in `call'

Expectation

the way the agent accesses the credentials should be the same, regardless of the difference in the linux version.
According to the document, ~/.aws/credentials has a higher priority than the instance profile. Then, the AccessDenied that occurs in AL2023 is normal behavior, and the fact that no error occurs in AL2 is a malfunction that does not recognize the credentials file in AL2.

Additional found

• AL2023 : ruby v3 -> sdk v3 gem 'aws-sdk', '~> 3' (document)
• Amazon Linux 2 : ruby v2 -> sdk v2 gem 'aws-sdk', '~> 2' (document)
• CodeDeploy Agent OFFICIAL_1.7.0-92_rpm : spec.required_ruby_version = '>= 2.7.0', spec.add_dependency('aws-sdk-core', '~> 3') (document)

Reference

https://docs.aws.amazon.com/sdk-for-ruby/v3/api/
https://docs.aws.amazon.com/sdk-for-ruby/v2/api/