Running multiple optimization runs in parallel

[Garren-H]

Hi all

I was just wondering if it would be possible to run multiple optimization runs in parallel using cmdstan? I have been using the multiprocessing.Pool module to open several instances of the same program, but the problem with this is the memory consumption for large problems. Using multiprocessing would create several instances of the same program and store both the data and transformed data multiple times in memory. Am I correct in saying that when we use the sampling or pathfinder methods (with multiple chains/paths) that the data and transformed data is only stored once in memory? And if so is there a way in which we can specify multiple chains for the optimization module?

[WardBrian]

This does not have support at the Stan level, so it’s not currently possible in cmdstanpy. It could be supported in Stan, but to my knowledge this is the first time someone has asked so I don’t think we had really considered it.

Can I ask why you’re interested in this? Is it that your model is multimodal enough that optimization regularly returns different values?

[Garren-H]

Yes, the specific model may be multi-modal

[Garren-H]

Continuing this thread (probably not completely relevant to cmdstanpy), on the cluster when I run a single optimization process the progress is automatically synced to the standard error file when having the default show_console=False. When I now do multiple optimizations using multiprocessing, this behavior is not maintained, show_console=False only syncs the the messages "cmdstan info: process started" or something along these lines. Is there a way in which I can keep this behavior? I have used show_console=True but this syncs the output to the standard output file, which is not desired as I print some other relevant information to the standard output file.

Also would it be advisable to set STAN_NUM_THREADS to the number of threads per optimization chain, or the total number of cores available? Following the thread I posted on the stan forum, I would think it would be better to specify this as the total number of cores, creating a pool with the total number of cores, which will redistribute cores when some chains finish before others?

[WardBrian]

Can you elaborate a bit more on your question about the progress output? I'm not sure I follow.

I believe the advice is when the different runs are in different processes, like for optimization, you should set it to be the per-chain number, but I'm not entirely sure. TBB may be able to balance across multiple processes. You can always try both and monitor the number of threads in something like htop to see if you're oversubscribed

[Garren-H]

I shall try this thanks Brian.

When I only do a single optimization run without specifying the show_console arg, hence defaulting to false, the progress from stan automatically syncs to the standard error file, with some lines 'cmdstan: Info - Process started' (or something along these line) being synced to the output file (along with Process done once optimization has terminated). I can then monitor the progress whilst the chains are running through a combination of ssh and cat commands to view the standard error file, *.ER.

When I now use multiprocessing (same args), the lines 'cmdstan: Info - Process started' is synced to the standard error file instead of the output file. The progress from stan is thus not synced to any file. When I now specify show_console=True the progress from stan syncs to the standard output file, *.OU. I would prefer that the stan progress be sync to the *.ER file instead of the *.OU file when using multiprocessing

[WardBrian]

Hm, I'm not sure -- I believe cmdstanpy is capturing both standard error and standard out internally, so the fact that either are appearing in the .ER or .OU files from your job scheduler is a bit surprising to me.

[Garren-H]

I think it has to do with the multiprocessing creating several child processes? So the .OU and .ER files for the parent and child processes are not the same? Not completely sure about this however