Consider using metrics from kube-state-metrics for billing #160
Description
Motivation
These metrics capture all pods - including short lived pods.
We can't rely on the timestamps to get the duration of pods, instead we'll need to use specific metrics for start and completion time.
This brings me to another point, the out-of-the-box deployment of kube-state-metrics does not emit kube_pod_completion_time. I looked into it, and the deployment had an argument that puts this in the metric-deny-list.
We'll also need to test out behaviors when a pod successfully completes vs when it's deleted vs when it's stuck terminating and then check what the end time actually say.
This was on the ocp-prod
spec:
automountServiceAccountToken: true
containers:
- args:
- --host=127.0.0.1
- --port=8081
- --telemetry-host=127.0.0.1
- --telemetry-port=8082
- |
--metric-denylist=
^kube_secret_labels$,
^kube_.+_annotations$
^kube_customresource_.+_annotations_info$,
^kube_customresource_.+_labels_info$,
- --metric-labels-allowlist=pods=[*],nodes=[*],namespaces=[*],persistentvolumes=[*],persistentvolumeclaims=[*],poddisruptionbudgets=[*]
- |
--metric-denylist=
^kube_.+_created$,
^kube_.+_metadata_resource_version$,
^kube_replicaset_metadata_generation$,
^kube_replicaset_status_observed_generation$,
^kube_pod_restart_policy$,
^kube_pod_init_container_status_terminated$,
^kube_pod_init_container_status_running$,
^kube_pod_container_status_terminated$,
^kube_pod_container_status_running$,
^kube_pod_completion_time$,
Completion criteria
Decision made on if we will use kube-state-metrics for invoicing. And if we are, creating a requirements doc and developing.
Description
- Evaluate pros and cons
- Figure out if we can enable pod end/finish timestamp
- Compare to current invoicing
- Decide if this is worth pursuing
- Create requirements
- create GH issues to develop
Note
This Epic was created in GitHub. If we need to add Issues from other Organizations, we will need to convert this Issue to a ZenHub issue. Please let @joachimweyl know, and he will convert it.