add Scheduler API to modify size (number of threads) of the scheduler

[Peter]

One use case is that we want to use n threads. We start a Scheduler with n-1 threads and in the main thread, we have pump creating jobs that are sent to the scheduler. When the job creation is completed we call Scheduler::wait which means that we are now only using n-1 threads. It would be useful to increase the scheduler before calling Scheduler::wait so the scheduler is using n threads.

An implementation would

• for increasing the size add a thread to the thread_group worker and make the new worker act on the queue as its sisters.

• for decreasing these size, we need to send a poison pill to one worker. This means that a poison pill cannot be recursive as it is now, but the job handler should be responsible for feeding the queue with enough pills to kill all workers.

The question is how to communicate the interface in Scheduler to the JobHandler object. It seems the obvious (only) way to communicate between Scheduler and JobHandler? is via the Queue<Job>, which is used for several types of communications, including passing on new jobs, telling the handler to wrap up things, and also for the Worker to tell the Handler that the job has finished. All jobs has a state that pays attention to and handle the jobs differently; it seems we could introduce a new state and/or a new type of job that tells the handler to modify the size of the thread_group.

[Peter]

Ticket #914 was marked related

[Peter]

In 3808:

Move the data member of the JobHandler? into its own data class,
JobHandlerData?, which is thread safe. This class is now owned by the
Scheduler and the JobHandler? has access via a pointer. This fixes #924
as the class is thread safe and since the sharing between Scheduler
and the JobHandler? class is now more complete also allow future
additions of the interface such as mentions in tickets:

refs #899, #910, #914 and #916.

[Peter]

In 3848:

add functionality to alter number of threads used by Scheduler. closes #910