How to inject variable determined at runtime to nodeadm userdata?

[rishabhjain62]

Hi,

We need to assign instance life cycle as node label to support some functionality. Earlier with bootstrap.sh this method used to work - --kubelet-extra-args "--node-labels=lifecycle=$(curl -s http://169.254.169.254/latest/meta-data/instance-life-cycle),worker_type=application"

But with nodeadm, I could not find reference to make it work.

`MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="BOUNDARY"

--BOUNDARY
Content-Type: application/node.eks.aws

---

apiVersion: node.eks.aws/v1alpha1
kind: NodeConfig
spec:
cluster:
name:
apiServerEndpoint:
certificateAuthority:
cidr: 10.1.0.0/16

--BOUNDARY
Content-Type: application/node.eks.aws

---

apiVersion: node.eks.aws/v1alpha1
kind: NodeConfig
spec:
kubelet:
config:
shutdownGracePeriod: 30s
featureGates:
DisableKubeletCloudCredentialProviders: true
flags:
- --node-labels=lifecycle=$(curl -s http://169.254.169.254/latest/meta-data/instance-life-cycle),worker_type=application

--BOUNDARY--`

[benjimin]

It would be super-useful if the nodeadm docs (or readme) provided an example (suggested a pattern) for how the config can incorporate information gleaned from the running instance (and not known in advance when the userdata is provided).

Some example use-cases (in terms of AWS):

• checking the IMDS for the AMI version and labelling the node with the AMI ID, even if the launch template uses a dynamically resolved AMI reference (via the SSM parameter for Amazon's latest recommended EKS node AMI) that allows keeping the AMI current without re-deploying the launch template (nor updating the userdata).
• for checking (via IMDS or cloud APIs) and parsing the tags on the instance (which are inherited from the ASG where they are used by the k8s cluster-autoscaler to predict what labels and taints the generated nodes will possess, especially when it is deciding whether to scale an ASG up from zero) and transferring them into kubelet label/taint arguments. Note this avoids the potential hazard that if labels/taints are manually specified in two places independently (both the ASG tags and the kubelet args) then inconsistency can arise (which can cause cluster-autoscaler to behave differently than intended, potentially deadlocking the cluster).
• labelling nodes with their corresponding EC2 instance IDs.

Currently (with AL2023) the following seems to work: provide a shell script in the userdata, have that script write a new config file (via a heredoc, which can use shell expansion to populate with info gleaned at runtime), then have the script run nodeadm init --config-source ... (possibly after nodeadm config check --config-source ... for debugging). However, this seems a bit imperfect (because nodeadm init is also already run by systemd in AL2023, and it departs from the recommendation to instead put raw nodeadm config manifests into the cloudinit userdata directly).