[Bug]: Dropsonde (?) delays reload on openvox8

[jpartlow]

Is this a critical security issue?

• This is not a security issue.

Describe the Bug

When, running the test suite, particularly on debian systems, on openvox-server 8, I'm seeing systemctl reload puppetserver times of ~30s (as opposed to ~2s) with unhelpful debug logs like:

2025-06-18T18:58:18.233Z INFO  [async-dispatch-3] [p.t.s.s.scheduler-service] Shutting down Scheduler Service                                                                           
2025-06-18T18:58:18.234Z DEBUG [af8d8e8a-7a91-4b71-904d-5b67775f655a_QuartzSchedulerThread] [o.q.c.QuartzSchedulerThread] batch acquisition of 0 triggers                               
2025-06-18T18:58:18.234Z INFO  [clojure-agent-send-off-pool-1] [o.q.c.QuartzScheduler] Scheduler af8d8e8a-7a91-4b71-904d-5b67775f655a_$_NON_CLUSTERED shutting down.                    
2025-06-18T18:58:18.234Z INFO  [clojure-agent-send-off-pool-1] [o.q.c.QuartzScheduler] Scheduler af8d8e8a-7a91-4b71-904d-5b67775f655a_$_NON_CLUSTERED paused.                           
2025-06-18T18:58:18.234Z DEBUG [clojure-agent-send-off-pool-1] [o.q.s.SimpleThreadPool] Shutting down threadpool...                                                                     
2025-06-18T18:58:18.234Z DEBUG [clojure-agent-send-off-pool-1] [o.q.s.SimpleThreadPool] Waiting for thread af8d8e8a-7a91-4b71-904d-5b67775f655a_Worker-5 to shut down                   
2025-06-18T18:58:18.240Z DEBUG [af8d8e8a-7a91-4b71-904d-5b67775f655a_Worker-10] [o.q.s.SimpleThreadPool] WorkerThread is shut down.                                                     
2025-06-18T18:58:18.327Z DEBUG [af8d8e8a-7a91-4b71-904d-5b67775f655a_Worker-1] [o.q.s.SimpleThreadPool] WorkerThread is shut down.                                                      
2025-06-18T18:58:18.328Z DEBUG [af8d8e8a-7a91-4b71-904d-5b67775f655a_Worker-6] [o.q.s.SimpleThreadPool] WorkerThread is shut down.                                                      
2025-06-18T18:58:18.329Z DEBUG [af8d8e8a-7a91-4b71-904d-5b67775f655a_Worker-8] [o.q.s.SimpleThreadPool] WorkerThread is shut down.                                                      
2025-06-18T18:58:18.334Z DEBUG [af8d8e8a-7a91-4b71-904d-5b67775f655a_Worker-2] [o.q.s.SimpleThreadPool] WorkerThread is shut down.                                                      
2025-06-18T18:58:18.335Z DEBUG [af8d8e8a-7a91-4b71-904d-5b67775f655a_Worker-7] [o.q.s.SimpleThreadPool] WorkerThread is shut down.                                                      
2025-06-18T18:58:18.335Z DEBUG [af8d8e8a-7a91-4b71-904d-5b67775f655a_Worker-9] [o.q.s.SimpleThreadPool] WorkerThread is shut down.                                                      
2025-06-18T18:58:18.337Z DEBUG [af8d8e8a-7a91-4b71-904d-5b67775f655a_Worker-4] [o.q.s.SimpleThreadPool] WorkerThread is shut down.                                                      
2025-06-18T18:58:18.633Z DEBUG [af8d8e8a-7a91-4b71-904d-5b67775f655a_Worker-3] [o.q.s.SimpleThreadPool] WorkerThread is shut down.                                                      
2025-06-18T18:58:20.234Z DEBUG [clojure-agent-send-off-pool-1] [o.q.s.SimpleThreadPool] Waiting for thread af8d8e8a-7a91-4b71-904d-5b67775f655a_Worker-5 to shut down                   
2025-06-18T18:58:22.235Z DEBUG [clojure-agent-send-off-pool-1] [o.q.s.SimpleThreadPool] Waiting for thread af8d8e8a-7a91-4b71-904d-5b67775f655a_Worker-5 to shut down
2025-06-18T18:58:24.235Z DEBUG [clojure-agent-send-off-pool-1] [o.q.s.SimpleThreadPool] Waiting for thread af8d8e8a-7a91-4b71-904d-5b67775f655a_Worker-5 to shut down
2025-06-18T18:58:26.236Z DEBUG [clojure-agent-send-off-pool-1] [o.q.s.SimpleThreadPool] Waiting for thread af8d8e8a-7a91-4b71-904d-5b67775f655a_Worker-5 to shut down
2025-06-18T18:58:28.236Z DEBUG [clojure-agent-send-off-pool-1] [o.q.s.SimpleThreadPool] Waiting for thread af8d8e8a-7a91-4b71-904d-5b67775f655a_Worker-5 to shut down
2025-06-18T18:58:30.237Z DEBUG [clojure-agent-send-off-pool-1] [o.q.s.SimpleThreadPool] Waiting for thread af8d8e8a-7a91-4b71-904d-5b67775f655a_Worker-5 to shut down
2025-06-18T18:58:32.237Z DEBUG [clojure-agent-send-off-pool-1] [o.q.s.SimpleThreadPool] Waiting for thread af8d8e8a-7a91-4b71-904d-5b67775f655a_Worker-5 to shut down
2025-06-18T18:58:34.238Z DEBUG [clojure-agent-send-off-pool-1] [o.q.s.SimpleThreadPool] Waiting for thread af8d8e8a-7a91-4b71-904d-5b67775f655a_Worker-5 to shut down
2025-06-18T18:58:36.238Z DEBUG [clojure-agent-send-off-pool-1] [o.q.s.SimpleThreadPool] Waiting for thread af8d8e8a-7a91-4b71-904d-5b67775f655a_Worker-5 to shut down
2025-06-18T18:58:38.239Z DEBUG [clojure-agent-send-off-pool-1] [o.q.s.SimpleThreadPool] Waiting for thread af8d8e8a-7a91-4b71-904d-5b67775f655a_Worker-5 to shut down
2025-06-18T18:58:40.239Z DEBUG [clojure-agent-send-off-pool-1] [o.q.s.SimpleThreadPool] Waiting for thread af8d8e8a-7a91-4b71-904d-5b67775f655a_Worker-5 to shut down
2025-06-18T18:58:42.240Z DEBUG [clojure-agent-send-off-pool-1] [o.q.s.SimpleThreadPool] Waiting for thread af8d8e8a-7a91-4b71-904d-5b67775f655a_Worker-5 to shut down
2025-06-18T18:58:44.240Z DEBUG [clojure-agent-send-off-pool-1] [o.q.s.SimpleThreadPool] Waiting for thread af8d8e8a-7a91-4b71-904d-5b67775f655a_Worker-5 to shut down
2025-06-18T18:58:46.241Z DEBUG [clojure-agent-send-off-pool-1] [o.q.s.SimpleThreadPool] Waiting for thread af8d8e8a-7a91-4b71-904d-5b67775f655a_Worker-5 to shut down
2025-06-18T18:58:48.235Z DEBUG [clojure-agent-send-off-pool-1] [o.q.s.SimpleThreadPool] Waiting for thread af8d8e8a-7a91-4b71-904d-5b67775f655a_Worker-5 to shut down
2025-06-18T18:58:48.236Z INFO  [async-dispatch-3] [p.t.s.s.scheduler-core] Failed to shutdown schedule service in 30 seconds  

Indicating a delay shutting down the scheduler and thread pool.

Disabling droponsde via /etc/puppetlabs/puppetserver/conf.d/puppetserver.conf alleviates this, but I don't have any other specific logging relevant to the actual problem, unfortunately.

Expected Behavior

About a 2s reload time.

Steps to Reproduce

A debian12 system with openvox-server 8 installed.

systemctl reload puppetserver

Environment

Openvox8
Debian/Ubuntu

Additional Context

No response

Relevant log output

[binford2k]

This is likely caused by a combination of two things:

• on startup, dropsonde caches a list of all Forge modules and this can take some time on the first run
• when it runs a report it attempts to make a PuppetDB connection and that's likely either not available or maybe a race condition.

With #9 we'll be dropping the weird trapperkeeper based integration and moving dropsonde to a standard cron/systemd scheduled task and this will likely be resolved at that point.

Leaving this open in the meantime to ensure we follow up and validate the fix.