Process reaping not working correctly causing many zombie curl processes #6
Description
While checking processes using top/ps I notice a lot of curl zombie processes in the Keycloak container:
$ ps xa |grep '<defunct>'
424 ? Z 0:00 [curl] <defunct>
441 ? Z 0:00 [curl] <defunct>
519 ? Z 0:00 [curl] <defunct>
647 ? Z 0:00 [curl] <defunct>
715 ? Z 0:00 [curl] <defunct>
760 ? Z 0:00 [curl] <defunct>
852 ? Z 0:00 [curl] <defunct>
949 ? Z 0:00 [curl] <defunct>
951 ? Z 0:00 [curl] <defunct>
996 ? Z 0:00 [curl] <defunct>
1201 ? Z 0:00 [curl] <defunct>
1353 ? Z 0:00 [curl] <defunct>
1524 ? Z 0:00 [curl] <defunct>
1633 ? Z 0:00 [curl] <defunct>
1740 ? Z 0:00 [curl] <defunct>
1854 ? Z 0:00 [curl] <defunct>
1949 ? Z 0:00 [curl] <defunct>
2282 ? Z 0:00 [curl] <defunct>
2377 ? Z 0:00 [curl] <defunct>
2476 ? Z 0:00 [curl] <defunct>
...
A Keycloak container running for 6 months has hundreds of these zombie processes:
$ ps xa |grep '<defunct>' | wc -l
542
The issue with zombie processes is that each zombie process keeps using memory. So if the number of zombie processes keeps increasing the system will run eventually out of memory.
If you enable the forrest view in top (type V) you can see they are started by java:
1441 root 20 0 719712 5596 3260 S 0,0 0,3 338:23.89 `- containerd-shim
1604 root 20 0 2408100 41764 860 S 0,5 2,2 408:30.04 `- java
24736 root 20 0 0 0 0 Z 0,0 0,0 0:00.00 `- curl
11785 root 20 0 0 0 0 Z 0,0 0,0 0:00.00 `- curl
30184 root 20 0 0 0 0 Z 0,0 0,0 0:00.00 `- curl
30290 root 20 0 0 0 0 Z 0,0 0,0 0:00.00 `- curl
23660 root 20 0 0 0 0 Z 0,0 0,0 0:00.00 `- curl
23771 root 20 0 0 0 0 Z 0,0 0,0 0:00.00 `- curl
32055 root 20 0 0 0 0 Z 0,0 0,0 0:00.00 `- curl
2487 root 20 0 0 0 0 Z 0,0 0,0 0:00.00 `- curl
12872 root 20 0 0 0 0 Z 0,0 0,0 0:00.00 `- curl
The process launching curl is Keycloak:
$ ps xa | grep 1604
1604 ? Ssl 408:30 java -Xms64m -Xmx512m -XX:MetaspaceSize=96M -XX:MaxMetaspaceSize=256m -Dfile.encoding=UTF-8 -Dsun.stdout.encoding=UTF-8 -Dsun.err.encoding=UTF-8 -Dstdout.encoding=UTF-8 -Dstderr.encoding=UTF-8 -XX:+ExitOnOutOfMemoryError -Djava.security.egd=file:/dev/urandom -XX:+UseParallelGC -XX:MinHeapFreeRatio=10 -XX:MaxHeapFreeRatio=20 -XX:GCTimeRatio=4 -XX:AdaptiveSizePolicyWeight=90 -XX:FlightRecorderOptions=stackdepth=512 --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.base/java.security=ALL-UNNAMED -Dkc.home.dir=/opt/keycloak/bin/.. -Djboss.server.config.dir=/opt/keycloak/bin/../conf -Djava.util.logging.manager=org.jboss.logmanager.LogManager -Dquarkus-log-max-startup-records=10000 -cp /opt/keycloak/bin/../lib/quarkus-run.jar io.quarkus.bootstrap.runner.QuarkusEntryPoint start --optimized --spi-initializer-issuer-base-uri= --spi-theme-login-default=openremote --spi-theme-account-theme=openremote --spi-theme-welcome-theme=keycloak --spi-theme-admin-theme=keycloak
To fix this the Docker entrypoint or command probably needs to be updated.