What are the basic rules for sizing a thread pool which contains tasks that:
- are always network-bound;
- access external services with different throughput and latency.
My main concern is using the bandwith optimally ( i.e. don't process tasks serially, but don't open 1200 network connections either );
Is your network access synchronous or asynchronous?
If your network access and request processing is asynchronous, then the thread pool size can be => Number of available cores + 1.
UPDATE: By available cores, I mean the number of physical processors available on the system. A thread pool is required even for a server using async I/O to take advantage of multiple physical processors.
If your network access and request processing is synchronous, then there is no hard-and-fast rule for sizing the thread pool. It is always better to make the thread pool size configurable in this case. A guestimate for a default value, given that your request processing is not CPU bound, could be:
(Request Processing Latency/Network Latency) * (Number of Available cores + 1) with a maximum value of 4 * Number of Avaliable cores.