perf: double the round robin batch size, improve queue filtering

[Ziinc]

• benchmarks

• improve add to table logic by reducing queue iterations and leaning on pattern matching for faster filtering

• double round robin chunk size to reduce reductions by half

• filter out full queues so that we don't add more events to them. Add them to startup queue instead to let DynamicPipeline resolver know that we need another pipeline queue

• removes unnecessary telemetry from QueueJanitor

• traverse_queues moves towards using :ets.match instead, minor memory improvement

• add_to_table moves towards reducing tid lookups -> large speedup

• list_counts shifts towards using direct match with list_counts_with_tids -> moderate ips improvement

Notable speedups:

add_to_table

before:

##### With input 1000 events #####
# Name                      ips        average  deviation         median         99th %
# old (legacy)           4.54 K      220.07 μs     ±7.28%      219.25 μs      253.52 μs

after:

##### With input 1000 events #####
Name                      ips        average  deviation         median         99th %
new (optimized)        3.11 K      321.58 μs     ±6.50%      318.75 μs      376.42 μs

Memory usage statistics:

Name               Memory usage
new (optimized)        99.35 KB

**All measurements for memory usage were the same**

Reduction count statistics:

Name            Reduction count
new (optimized)          5.72 K

traverse_queues - memory improvements and 1% speedup in ips

##### With input 10 sources, 3 backends, 10 queues each #####
Name                   ips        average  deviation         median         99th %
match             123.34 K        8.11 μs    ±86.97%        7.92 μs        9.83 μs
match_object      121.86 K        8.21 μs    ±88.70%           8 μs          10 μs
select            121.75 K        8.21 μs    ±94.13%           8 μs          10 μs

Comparison: 
match             123.34 K
match_object      121.86 K - 1.01x slower +0.0985 μs
select            121.75 K - 1.01x slower +0.106 μs

Memory usage statistics:

Name            Memory usage
match                0.89 KB
match_object         1.45 KB - 1.63x memory usage +0.56 KB
select               1.47 KB - 1.65x memory usage +0.58 KB

**All measurements for memory usage were the same**

Reduction count statistics:

Name         Reduction count
match                    391
match_object             393 - 1.01x reduction count +2
select                   397 - 1.02x reduction count +6

list_counts

Before:

# ##### With input 25 sources, 3 backends, 30 queues each #####
# Name             ips        average  deviation         median         99th %
# legacy      15.2 K       65.72 μs     ±8.57%       64.54 μs          76 μs

After:

##### With input 25 sources, 3 backends, 30 queues each #####
Name                            ips        average  deviation         median         99th %
list_queues                 17.46 K       57.29 μs    ±10.44%       56.63 μs       68.17 μs
list_queues_with_tids       17.26 K       57.94 μs     ±9.71%       57.21 μs          70 μs
list_counts_with_tids       16.91 K       59.13 μs    ±12.89%       58.29 μs       72.14 μs
list_counts                 16.84 K       59.39 μs    ±20.62%       58.33 μs       74.88 μs

Comparison: 
list_queues                 17.46 K
list_queues_with_tids       17.26 K - 1.01x slower +0.65 μs
list_counts_with_tids       16.91 K - 1.03x slower +1.84 μs
list_counts                 16.84 K - 1.04x slower +2.10 μs

Memory usage statistics:

Name                     Memory usage
list_queues                   1.70 KB
list_queues_with_tids         2.10 KB - 1.23x memory usage +0.40 KB
list_counts_with_tids         3.57 KB - 2.10x memory usage +1.87 KB
list_counts                   3.98 KB - 2.34x memory usage +2.28 KB

**All measurements for memory usage were the same**

Reduction count statistics:

Name                  Reduction count
list_queues                    8.31 K
list_queues_with_tids          8.35 K - 1.00x reduction count +0.0330 K
list_counts_with_tids          8.49 K - 1.02x reduction count +0.178 K
list_counts                    8.48 K - 1.02x reduction count +0.164 K

[Ziinc]

Will be splitting this PR up as identifying the regression cause is quite tricky

[Ziinc]

closing as this has been broken down into smaller chunks parts.