How to do IO args for parallel cmdstan?

[SteveBronder]

Summary:

I think stan-dev/stan#3033 should hopefully go in next week or so and I'd like to figure out some of the command line arguments and processing those input args related to running multiple chains in a Stan program

We need to figure out

1. The actual arg to specify multiple parallel chains
2. How to handle args of initial values of parameters, inverse mass matrices, and output files

Docs also need updated to describe the behavior of the rng with multiple chains as well as any arguments

New args

There's a few ways to specify the number of chains. What I have in the prototype is

STAN_NUM_THREADS=20 examples/dawid/dawid sample num_samples=500 \
  num_warmup=500 data file=examples/dawid/caries_dump.R chains n_chains=8

But idt I like this because, what other args would go into chains? I think what might be better here is to have a parallel "meta arg" so users would write

STAN_NUM_THREADS=20 examples/dawid/dawid sample num_samples=500 \
  num_warmup=500 data file=examples/dawid/caries_dump.R parallel n_chains=8

We want to add this meta arg for specifying the number of parameters and gpu info anyway so I think this is a good opportunity to add it so users can write things like

examples/dawid/dawid sample num_samples=500 \
  num_warmup=500 data file=examples/dawid/caries_dump.R parallel n_chains=8 n_threads=20

Does anyone disagree with this?

Multiple IO

How to handle multiple init and output files is kind of tricky. The definition for init right now is

init=<string>
  Initialization method: "x" initializes randomly between [-x, x], "0" initializes to 0, anything else 
    identifies a file of values
  Valid values: All
  Defaults to "2"

So the only issue is the file of values. I think what we could do here is a simple rule like

init=<string>
  Initialization method: "x" initializes randomly between [-x, x], "0" initializes to 0, anything else
   identifies a file of values. If `n_chain` is greater than one the program will first search
   for "path/to/{init_filename}_{1:n_chain}.{file_ending}" and if not found will then search for
   a file "path/to/{init_filename}.{file_ending}"
  Valid values: All
  Defaults to "2"

So if the user specifies their init file as "my_init.json" we will first look for "my_init_1.json", "my_init_2.json", etc. and if that fails then we look for "my_init.json".

I think we can do the same thing for the initial mass matrix. For the output files we just take the output file they specify and make a new set out output files with a _1:n_chain before the file extension. If anyone has better ideas I'm all ears!

Current Version:

v2.26.1

[mitzimorris]

one way to figure this out is to write the docs first - here's the chapter in the CmdStan manual that goes over the set of arguments and the (expletive deleted) argument hierarchy: https://mc-stan.org/docs/2_26/cmdstan-guide/command-line-interface-overview.html

this is NUTS-HMC specific, correct? I'm happy to help with the CmdStan argument parser wrangling.

regarding proposal for naming of input files, this seems like a reasonable hack for the current CmdStan arg parser.

we also need to consider what happens for all of the outputs - this includes Stan CSV output files, diagnostics file, profile file.

note that the CmdStan process has a single stdout and stderr and progress messages ("refresh") are written to stdout. will this work as expected for parallel cmdstan?

[SteveBronder]

one way to figure this out is to write the docs first - here's the chapter in the CmdStan manual that goes over the set of arguments and the (expletive deleted) argument hierarchy: https://mc-stan.org/docs/2_26/cmdstan-guide/command-line-interface-overview.html

Could be good! I'll make a PR.

this is NUTS-HMC specific, correct? I'm happy to help with the CmdStan argument parser wrangling.

For now yes, but the scheme can be applied to all of our samplers / optimizers

we also need to consider what happens for all of the outputs - this includes Stan CSV output files, diagnostics file, profile file.

Oh good point. I think those should be per chain so I think my scheme for output should work

note that the CmdStan process has a single stdout and stderr and progress messages ("refresh") are written to stdout. will this work as expected for parallel cmdstan?

So std::cout and friends are by the standard threadsafe, but what can happen is that multiple threads write to the streams buffer one after another which can cause the output to be all jarbled and goofy looking. Like for example here

  void operator()(const std::string& message) {
    *output_ << comment_prefix_ << message << std::endl;
  }

Those multiple << are each functions which on separate threads can happen in an interleaved manner leaving a weird looking output.

I think a simple solution is in the stream writer to do something like in this stackoverflow post

void operator()(const std::string& message) {
  std::stringstream streamer; 
  streamer << comment_prefix_ << message << std::endl;
  *output_ << streamer;
}

By writing to a stringstream first those writes happen within the thread and then there's no interweaving when writing to std::cout etc. since there's only one << buffer write for each thread

[SteveBronder]

I think my plan is to work out the prototype a bit more to make sure all the stuff I'm saying about I/O makes sense, then update the docs with those things

[mitzimorris]

I think a simple solution is in the stream writer to do something like in this stackoverflow post

void operator()(const std::string& message) {
  std::stringstream streamer; 
  streamer << comment_prefix_ << message << std::endl;
  *output_ << streamer;
}

By writing to a stringstream first those writes happen within the thread and then there's no interweaving when writing to std::cout etc. since there's only one << buffer write for each thread

this sounds most reasonable.

[bbbales2]

parallel n_chains=8 n_threads=20

In cmdstanr these are specified like:

chains = 4,
parallel_chains = getOption("mc.cores", 1),
threads_per_chain = NULL,

Simplest thing is match that. If we skip the meta arg we avoid the extra parallel in parallel chains=4 parallel_chains=4.

---

I also like the rule that is search for filename X, if that isn't available, search for filename X1, X2, etc. where X1 and X2 two are rewrites of the name X. I'm not sure which direction is less confusing. We could add a check for ambiguity I guess.

[optional] I prefer .[chain_id] to _[chain_id].

So we're optionally broadcasting all the non-data inputs.

Rewriting the filenames on outputs also seems good.

---

That stringstream trick looks neat. Let's definitely do that.

[SteveBronder]

chains = 4,
parallel_chains = getOption("mc.cores", 1),
threads_per_chain = NULL,

So at this level I think it's a bit different where for cmdstanr chains=4 they run 4 separate processes, parallel_chains is for R running those programs in parallel, and threads per chain is the amount of threads each program can use. I think here we might want something like

parallel chains=4 threads=16

Because threads here dictates the total number of threads available to cmdstan (a program could have parallel chains and each one uses a reduce_sum())

@rok-cesnovar should I also move the device and platform id options for OpenCL under parallel as well?

[wds15]

Yes, I also think we need to specify the total number of threads and that’s it...but we should do a quick benchmarkbto see if the tbb is indeed clever enough to run this optimal. So we do not want 8 cores used for one chain at a time, but instead 2 cores per chain when running 4 chains total.

[SteveBronder]

@wds15 yeah that's a good idea. fyi I just put up a PR to set the number of threads by an argument. Once that's in I can add the threads argument

stan-dev/math#2455

[bbbales2]

Yeah yeah, so in cmdstanr it is:

chains <- number of chains to run (not necessarily all of these in parallel)
parallel_chains <- maximum number of chains to work on at any time
threads_per_chain <- threads to use for each chain

I think here we might want something like

There and here will quickly be the same thing :D

So we do not want 8 cores used for one chain at a time, but instead 2 cores per chain when running 4 chains total.

The advantage of specifying total threads is that if there's only 1 chain left then all the threads can go there? Or once we switch to this do we actually lose the ability to even specify threads_per_chain? I'm okay switching I just want to understand the switch. Also @rok-cesnovar so he sees this.

If we do just num_chains and total_num_threads, then we somehow need to say that tasks corresponding to running a chain have higher priority for the threadpool than tasks generated by reduce sum. That sound right?

[bbbales2]

And then if we switch to chains/threads, then there isn't anymore a need for a parallel_chains? I guess the only need there would be if memory usage was a problem with the autodiff.

[SteveBronder]

The advantage of specifying total threads is that if there's only 1 chain left then all the threads can go there? Or once we switch to this do we actually lose the ability to even specify threads_per_chain?

Tmk threads_per_chain just tells each cmdstan program how many threads are available right? So like with one program idt we can tell the tbb how many threads we have per chain can we?

If we do just num_chains and total_num_threads, then we somehow need to say that tasks corresponding to running a chain have higher priority for the threadpool than tasks generated by reduce sum. That sound right?

@wds15 do you know if the tbb already gives higher priority to the top processes in nested parallelism? I assume it would

And then if we switch to chains/threads, then there isn't anymore a need for a parallel_chains? I guess the only need there would be if memory usage was a problem with the autodiff.

I think the arguments would change behavior where parallel chains would tell cmdstan to run X chains instead of right now where it runs X processes with one chain each.

Idt there would be any memory issues with running multiple chains (it should use less memory)

[yizhang-yiz]

Depends on what are we going to do with parallel chains, all chains may be equal or there maybe a chain taking the lead. I think we should maintain flexibility by separating # of threads from # of chains.

As to args what I did for MPI cross chain is to have additional args under adapt meta-arg:
num_cross_chains, cross_chain_window, cross_chain_rhat, cross_chain_ess
and allow init arg to take regex file names, e.g
https://github.com/metrumresearchgroup/Torsten/blob/e50ca5e799ac29cb9d004e1791193072e4ad85d1/cmdstan/stan/src/stan/mcmc/cross_chain/mpi_cross_chain.hpp#L137

[wds15]

@wds15 do you know if the tbb already gives higher priority to the top processes in nested parallelism? I assume it would

I don't know it... that's why I push for a quick test.

From a quick search I found this:

https://stackoverflow.com/questions/30589361/intel-tbb-disable-nested-parallelism

Which means to me we should try it out. In case performance is not good, we can control the concurrency level per chain by running each within a task_arena.

[SteveBronder]

Depends on what are we going to do with parallel chains, all chains may be equal or there maybe a chain taking the lead. I think we should maintain flexibility by separating # of threads from # of chains.

Right now in the current PR I just want to do the embarrassingly parallel thing and run N chains in one program. But all the tooling around setting that up should make it easy to do auto warmup (and the code from here should be a good starting point for that).

As to args what I did for MPI cross chain is to have additional args under adapt meta-arg:
num_cross_chains, cross_chain_window, cross_chain_rhat, cross_chain_ess and allow init arg to take regex file names, e.g

Wrt the regex, how does the user know which init was used for which chain? I think I like the _i approach just so the user knows if something goofy happens with chain i they know it was caused by init_file_i

From a quick search I found this:

Oh yeah that is weird behavior. I'll try a benchmark today or tomorrow to see how this holds up

[yizhang-yiz]

Wrt the regex, how does the user know which init was used for which chain? I think I like the _i approach just so the user knows if something goofy happens with chain i they know it was caused by init_file_i

I trust that in the end user won't be using it for a complex regex match but some simple pattern, such as xxx_1_xxx.json, xxx_2_xxx.json, ... or xxx_a_xxx.json, xxx_b_xxx.json, ..., similar to _i approach but with more flexibility. More importantly user is compelled to think what init files to select.

EDIT: also the output csv would have init file pointing to the correct file for a certain chain, right? So user don't need keep in mind the init file-chain map.