[JENKINS-73208] "Scan Repository" on a big one leads to huge amount of threads and kills our Jenkins

[jenkins-infra-bot]

One of our repos under Github Organization has 4100 branches and 1100 pull requests.
The scanning is performed nightly, once it's running the amount of threads goes through the roof.

This happens right before "Finished branch indexing":

[Tue May 21 21:31:06 UTC 2024] Starting branch indexing...
...
[Tue May 21 22:09:16 UTC 2024] Finished branch indexing. Indexing took 38 min
Evaluating orphaned items in Pipeline Builder » 

It finished scanning at 22:09, here is the threads graph for that timeframe

Another multi-branch repo had 4330 branches and 3600 PRs.
I ran a manual scan, here's the result:

[Wed May 22 14:48:57 UTC 2024] Finished branch indexing. Indexing took 16 min

And the threads graph:


The Jenkins restarted at 14:50.

From docker logs:

2024-05-22 14:49:13.931+0000 [id=2668668]	WARNING	hudson.slaves.SlaveComputer#doJenkinsAgentJnlp: Agent "............." is connecting with the "-jnlpUrl" argument, which is deprecated. Use "-url" and "-name" instead, potentially also passing in "-webSocket", "-tunnel", and/or work directory options as needed.
OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f8f69800000, 16384, 0) failed; error='Not enough space' (errno=12)
OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f8f69700000, 16384, 0) failed; error='Not enough space' (errno=12)
[thread 301033 also had an error]#

# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 16384 bytes for committing reserved memory.
[thread 301034 also had an error]
OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f8f69600000, 16384, 0) failed; error='Not enough space' (errno=12)
OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f8f69500000, 16384, 0) failed; error='Not enough space' (errno=12)

[lots of similar messages]

This got to more than 21000 threads:

CheckMK
APP  17:49
Service PROBLEM notification
Host: jenkins.mycompany.domain (IP: jenkins.mycompany.domain)
Service: Number of threads
State: CRITICAL
Additional Info
CRIT - Count: 21608 threads (warn/crit at 3500 threads/4500 threads)(!!), Usage: 3.85%
Please take a look: @​cmkadmin
Check_MK notification: Wed May 22 14:49:15 GMT 2024

Please let me know if any other info needed.

---

Originally reported by eplodn1, imported from: "Scan Repository" on a big one leads to huge amount of threads and kills our Jenkins

• status: Open
• priority: Major
• component(s): branch-api-plugin
• resolution: Unresolved
• votes: 0
• watchers: 2
• imported: 20251211-141027

Raw content of original issue

One of our repos under Github Organization has 4100 branches and 1100 pull requests.
The scanning is performed nightly, once it's running the amount of threads goes through the roof.
  

This happens right before "Finished branch indexing":

[Tue May 21 21:31:06 UTC 2024] Starting branch indexing...
...
[Tue May 21 22:09:16 UTC 2024] Finished branch indexing. Indexing took 38 min
Evaluating orphaned items in Pipeline Builder » 


It finished scanning at 22:09, here is the threads graph for that timeframe
  
Another multi-branch repo had 4330 branches and 3600 PRs. 
I ran a manual scan, here's the result:

[Wed May 22 14:48:57 UTC 2024] Finished branch indexing. Indexing took 16 min


And the threads graph:
  
  
The Jenkins restarted at 14:50.
From docker logs:

2024-05-22 14:49:13.931+0000 [id=2668668]	WARNING	hudson.slaves.SlaveComputer#doJenkinsAgentJnlp: Agent "............." is connecting with the "-jnlpUrl" argument, which is deprecated. Use "-url" and "-name" instead, potentially also passing in "-webSocket", "-tunnel", and/or work directory options as needed.
OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f8f69800000, 16384, 0) failed; error='Not enough space' (errno=12)
OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f8f69700000, 16384, 0) failed; error='Not enough space' (errno=12)
[thread 301033 also had an error]#
There is insufficient memory for the Java Runtime Environment to continue.
Native memory allocation (mmap) failed to map 16384 bytes for committing reserved memory.
[thread 301034 also had an error]

OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f8f69600000, 16384, 0) failed; error='Not enough space' (errno=12)

OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f8f69500000, 16384, 0) failed; error='Not enough space' (errno=12)
[lots of similar messages]



This got to more than 21000 threads:

CheckMK
APP  17:49
Service PROBLEM notification
Host: jenkins.mycompany.domain (IP: jenkins.mycompany.domain)
Service: Number of threads
State: CRITICAL
Additional Info
CRIT - Count: 21608 threads (warn/crit at 3500 threads/4500 threads)(!!), Usage: 3.85%
Please take a look: @cmkadmin
Check_MK notification: Wed May 22 14:49:15 GMT 2024


Please let me know if any other info needed.

environment

Jenkins: 2.440.1<br/>
OS: Linux - 5.15.0-106-generic<br/>
Java: 17.0.10 - Eclipse Adoptium (OpenJDK 64-Bit Server VM)<br/>
---<br/>
allure-jenkins-plugin:2.31.1<br/>
amazon-ecr:1.114.vfd22430621f5<br/>
analysis-model-api:12.3.3<br/>
ansicolor:1.0.4<br/>
antisamy-markup-formatter:162.v0e6ec0fcfcf6<br/>
apache-httpcomponents-client-4-api:4.5.14-208.v438351942757<br/>
apache-httpcomponents-client-5-api:5.3.1-1.0<br/>
artifact-manager-s3:845.848.ve968b_8576c1c<br/>
asm-api:9.7-33.v4d23ef79fcc8<br/>
authentication-tokens:1.53.v1c90fd9191a_b_<br/>
aws-bucket-credentials:1.0.0<br/>
aws-credentials:231.v08a_59f17d742<br/>
aws-global-configuration:130.v35b_7b_96f53c3<br/>
aws-java-sdk:1.12.696-451.v0651a_da_9ca_ec<br/>
aws-java-sdk-api-gateway:1.12.696-451.v0651a_da_9ca_ec<br/>
aws-java-sdk-autoscaling:1.12.696-451.v0651a_da_9ca_ec<br/>
aws-java-sdk-cloudformation:1.12.696-451.v0651a_da_9ca_ec<br/>
aws-java-sdk-cloudfront:1.12.696-451.v0651a_da_9ca_ec<br/>
aws-java-sdk-codebuild:1.12.696-451.v0651a_da_9ca_ec<br/>
aws-java-sdk-codedeploy:1.12.696-451.v0651a_da_9ca_ec<br/>
aws-java-sdk-ec2:1.12.696-451.v0651a_da_9ca_ec<br/>
aws-java-sdk-ecr:1.12.696-451.v0651a_da_9ca_ec<br/>
aws-java-sdk-ecs:1.12.696-451.v0651a_da_9ca_ec<br/>
aws-java-sdk-efs:1.12.696-451.v0651a_da_9ca_ec<br/>
aws-java-sdk-elasticbeanstalk:1.12.696-451.v0651a_da_9ca_ec<br/>
aws-java-sdk-elasticloadbalancingv2:1.12.696-451.v0651a_da_9ca_ec<br/>
aws-java-sdk-iam:1.12.696-451.v0651a_da_9ca_ec<br/>
aws-java-sdk-kinesis:1.12.696-451.v0651a_da_9ca_ec<br/>
aws-java-sdk-lambda:1.12.696-451.v0651a_da_9ca_ec<br/>
aws-java-sdk-logs:1.12.696-451.v0651a_da_9ca_ec<br/>
aws-java-sdk-minimal:1.12.696-451.v0651a_da_9ca_ec<br/>
aws-java-sdk-organizations:1.12.696-451.v0651a_da_9ca_ec<br/>
aws-java-sdk-secretsmanager:1.12.696-451.v0651a_da_9ca_ec<br/>
aws-java-sdk-sns:1.12.696-451.v0651a_da_9ca_ec<br/>
aws-java-sdk-sqs:1.12.696-451.v0651a_da_9ca_ec<br/>
aws-java-sdk-ssm:1.12.696-451.v0651a_da_9ca_ec<br/>
basic-branch-build-strategies:81.v05e333931c7d<br/>
blueocean:1.27.12<br/>
blueocean-autofavorite:1.2.5<br/>
blueocean-bitbucket-pipeline:1.27.12<br/>
blueocean-commons:1.27.12<br/>
blueocean-config:1.27.12<br/>
blueocean-core-js:1.27.12<br/>
blueocean-dashboard:1.27.12<br/>
blueocean-display-url:2.4.2<br/>
blueocean-events:1.27.12<br/>
blueocean-git-pipeline:1.27.12<br/>
blueocean-github-pipeline:1.27.12<br/>
blueocean-i18n:1.27.12<br/>
blueocean-jira:1.27.12<br/>
blueocean-jwt:1.27.12<br/>
blueocean-personalization:1.27.12<br/>
blueocean-pipeline-api-impl:1.27.12<br/>
blueocean-pipeline-editor:1.27.12<br/>
blueocean-pipeline-scm-api:1.27.12<br/>
blueocean-rest:1.27.12<br/>
blueocean-rest-impl:1.27.12<br/>
blueocean-web:1.27.12<br/>
bootstrap5-api:5.3.3-1<br/>
bouncycastle-api:2.30.1.77-225.v26ea_c9455fd9<br/>
branch-api:2.1169.va_f810c56e895<br/>
build-failure-analyzer:2.5.1<br/>
build-monitor-plugin:1.14-876.v2162064db_a_93<br/>
build-name-setter:2.4.2<br/>
build-time-blame:75.v44a_51cc97cdd-2.1.0<br/>
build-token-root:151.va_e52fe3215fc<br/>
build-token-trigger:1.0.0<br/>
buildtriggerbadge:251.vdf6ef853f3f5<br/>
built-on-column:1.4<br/>
caffeine-api:3.1.8-133.v17b_1ff2e0599<br/>
checks-api:2.2.0<br/>
cloud-stats:336.v788e4055508b_<br/>
cloudbees-bitbucket-branch-source:886.v44cf5e4ecec5<br/>
cloudbees-disk-usage-simple:203.v3f46a_7462b_1a_<br/>
cloudbees-folder:6.901.vb_4c7a_da_75da_3<br/>
cobertura:1.17<br/>
code-coverage-api:4.99.0<br/>
command-launcher:107.v773860566e2e<br/>
commons-httpclient3-api:3.1-3<br/>
commons-lang3-api:3.13.0-62.v7d18e55f51e2<br/>
commons-text-api:1.11.0-109.vfe16c66636eb_<br/>
compound-slaves:1.2<br/>
compress-artifacts:98.vb_20f3c77ddf7<br/>
conditional-buildstep:1.4.3<br/>
config-file-provider:973.vb_a_80ecb_9a_4d0<br/>
copyartifact:722.v0662a_9b_e22a_c<br/>
coverage:1.14.0<br/>
credentials:1337.v60b_d7b_c7b_c9f<br/>
credentials-binding:677.vdc9d38cb_254d<br/>
data-tables-api:2.0.7-1<br/>
description-setter:239.vd0a_6b_785f92d<br/>
display-url-api:2.204.vf6fddd8a_8b_e9<br/>
docker-commons:439.va_3cb_0a_6a_fb_29<br/>
docker-java-api:3.3.4-86.v39b_a_5ede342c<br/>
docker-plugin:1.6.1<br/>
docker-workflow:572.v950f58993843<br/>
durable-task:555.v6802fe0f0b_82<br/>
echarts-api:5.5.0-1<br/>
email-ext:1814.v404722f34263<br/>
envfile:43.v2644436e7d71<br/>
extended-choice-parameter:382.v5697b_32134e8<br/>
extended-read-permission:53.v6499940139e5<br/>
favorite:2.208.v91d65b_7792a_c<br/>
flaky-test-handler:1.2.3<br/>
font-awesome-api:6.5.2-1<br/>
forensics-api:2.4.0<br/>
generic-webhook-trigger:2.2.1<br/>
git:5.2.2<br/>
git-client:4.7.0<br/>
git-parameter:0.9.19<br/>
github:1.39.0<br/>
github-api:1.318-461.v7a_c09c9fa_d63<br/>
github-branch-source:1789.v5b_0c0cea_18c3<br/>
github-oauth:597.ve0c3480fcb_d0<br/>
github-scm-trait-notification-context:40.vfa_7f31a_b_d7f8<br/>
gradle:2.12<br/>
groovy:457.v99900cb_85593<br/>
gson-api:2.10.1-15.v0d99f670e0a_7<br/>
handy-uri-templates-2-api:2.1.8-30.v7e777411b_148<br/>
htmlpublisher:1.33<br/>
http_request:1.18<br/>
instance-identity:185.v303dc7c645f9<br/>
ionicons-api:74.v93d5eb_813d5f<br/>
jackson2-api:2.17.0-379.v02de8ec9f64c<br/>
jakarta-activation-api:2.1.3-1<br/>
jakarta-mail-api:2.1.3-1<br/>
javadoc:243.vb_b_503b_b_45537<br/>
javax-activation-api:1.2.0-6<br/>
javax-mail-api:1.6.2-9<br/>
jaxb:2.3.9-1<br/>
jdk-tool:73.vddf737284550<br/>
jenkins-design-language:1.27.12<br/>
jenkins-jira-issue-updater:1.18<br/>
jersey2-api:2.42-147.va_28a_44603b_d5<br/>
jira:3.13<br/>
jira-steps:2.0.165.v8846cf59f3db<br/>
jjwt-api:0.11.5-112.ve82dfb_224b_a_d<br/>
jnr-posix-api:3.1.19-2<br/>
job-dsl:1.87<br/>
job-restrictions:0.8<br/>
jobConfigHistory:1229.v3039470161a_d<br/>
joda-time-api:2.12.7-29.v5a_b_e3a_82269a_<br/>
jquery:1.12.4-1<br/>
jquery3-api:3.7.1-2<br/>
jsch:0.2.16-86.v42e010d9484b_<br/>
json-api:20240303-41.v94e11e6de726<br/>
json-path-api:2.9.0-58.v62e3e85b_a_655<br/>
junit:1265.v65b_14fa_f12f0<br/>
last-changes:2.7.11<br/>
ldap:725.v3cb_b_711b_1a_ef<br/>
locale:431.v3435fa_8f8445<br/>
lockable-resources:1255.vf48745da_35d0<br/>
log-parser:2.3.3<br/>
logstash:2.5.0218.v0a_ff8fefc12b_<br/>
mailer:472.vf7c289a_4b_420<br/>
mapdb-api:1.0.9-40.v58107308b_7a_7<br/>
material-theme:0.5.2-rc100.6121925fe229<br/>
matrix-auth:3.2.2<br/>
matrix-project:822.824.v14451b_c0fd42<br/>
maven-plugin:3.23<br/>
metrics:4.2.21-449.v6960d7c54c69<br/>
mina-sshd-api-common:2.12.1-101.v85b_e08b_780dd<br/>
mina-sshd-api-core:2.12.1-101.v85b_e08b_780dd<br/>
monitoring:1.98.0<br/>
next-build-number:1.8<br/>
okhttp-api:4.11.0-172.vda_da_1feeb_c6e<br/>
pam-auth:1.10<br/>
parameterized-scheduler:277.v61a_4b_a_49a_c5c<br/>
parameterized-trigger:806.vf6fff3e28c3e<br/>
performance:960.v701db_0d66e1b_<br/>
pipeline-aggregator-view:104.v94a_e5f6cdb_c3<br/>
pipeline-aws:1.45<br/>
pipeline-build-step:540.vb_e8849e1a_b_d8<br/>
pipeline-github:2.8-159.09e4403bc62f<br/>
pipeline-github-lib:42.v0739460cda_c4<br/>
pipeline-graph-analysis:216.vfd8b_ece330ca_<br/>
pipeline-graph-view:243.vc9e11fec486a_<br/>
pipeline-groovy-lib:710.v4b_94b_077a_808<br/>
pipeline-input-step:495.ve9c153f6067b_<br/>
pipeline-milestone-step:119.vdfdc43fc3b_9a_<br/>
pipeline-model-api:2.2198.v41dd8ef6dd56<br/>
pipeline-model-definition:2.2198.v41dd8ef6dd56<br/>
pipeline-model-extensions:2.2198.v41dd8ef6dd56<br/>
pipeline-rest-api:2.34<br/>
pipeline-stage-step:312.v8cd10304c27a_<br/>
pipeline-stage-tags-metadata:2.2198.v41dd8ef6dd56<br/>
pipeline-stage-view:2.34<br/>
pipeline-utility-steps:2.16.2<br/>
plain-credentials:182.v468b_97b_9dcb_8<br/>
plugin-util-api:4.1.0<br/>
postbuild-task:1.9<br/>
prism-api:1.29.0-15<br/>
project-description-setter:1.2<br/>
promoted-builds:945.v597f5c6a_d3fd<br/>
pubsub-light:1.18<br/>
read-only-configurations:1.10<br/>
rebuild:332.va_1ee476d8f6d<br/>
resource-disposer:0.23<br/>
role-strategy:717.v6a_69a_fe98974<br/>
run-condition:1.7<br/>
s3:466.vf5b_3db_8e3eb_2<br/>
schedule-build:572.v345ea_4a_a_d3fb_<br/>
scm-api:690.vfc8b_54395023<br/>
script-security:1336.vf33a_a_9863911<br/>
simple-theme-plugin:176.v39740c03a_a_f5<br/>
skip-notifications-trait:313.vd1337c8f8134<br/>
slack:715.v1cfed1b_9c63c<br/>
snakeyaml-api:2.2-111.vc6598e30cc65<br/>
sse-gateway:1.26<br/>
ssh:2.6.1<br/>
ssh-agent:367.vf9076cd4ee21<br/>
ssh-credentials:337.v395d2403ccd4<br/>
ssh-slaves:2.948.vb_8050d697fec<br/>
ssh-steps:2.0.68.va_d21a_12a_6476<br/>
sshd:3.322.v159e91f6a_550<br/>
structs:337.v1b_04ea_4df7c8<br/>
subversion:1256.vee91953217b_6<br/>
swarm:3.46<br/>
text-file-operations:1.3.2<br/>
text-finder:1.27<br/>
theme-manager:215.vc1ff18d67920<br/>
throttle-concurrents:2.14<br/>
timestamper:1.27<br/>
token-macro:400.v35420b_922dcb_<br/>
trilead-api:2.142.v748523a_76693<br/>
uno-choice:2.8.3<br/>
validating-string-parameter:183.v3748e79b_9737<br/>
variant:60.v7290fc0eb_b_cd<br/>
vdoo-vision:1.9<br/>
warnings-ng:11.3.0<br/>
workflow-aggregator:596.v8c21c963d92d<br/>
workflow-api:1311.v4250456a_e552<br/>
workflow-basic-steps:1058.vcb_fc1e3a_21a_9<br/>
workflow-cps:3894.3896.vca_2c931e7935<br/>
workflow-durable-task-step:1353.v1891a_b_01da_18<br/>
workflow-job:1400.v7fd111b_ec82f<br/>
workflow-multibranch:773.vc4fe1378f1d5<br/>
workflow-scm-step:427.v4ca_6512e7df1<br/>
workflow-step-api:657.v03b_e8115821b_<br/>
workflow-support:907.v6713a_ed8a_573<br/>
ws-cleanup:0.45<br/>

4 attachments

• image (10).png

  

• image (11).png

  

• image (9).png

  

• screenshot-1.png