Scalability through Thread Economy: Asynchronous operations against the multithreaded producer / consumer queues on the thread pool?


Async programming is a way to achieve scalability in web servers through thread economy, so that very few non-blocking threads can handle many simulataneous requests. Node.js for instance, achieves scalability using only a single thread using async operations.

I am currently using the database MongoDb and it's official C# driver which does not support async operations yet. I am therefore considering using a simple producer/consumer queue to handle mongodb requests to reduce the number of blocking threads. This is done by letting thread pool threads insert db requests on a queue and then let them continue with other tasks. The queue has one more more dedicated threads doing the actual the db requests, and when the requests return with results, the result are handed over to a thread pool thread.

However, I am now wondering if using a queue is necessary when using the thread pool (through TPL and tasks from c# 4.0) because the thread pool has a max limit on the number of threads. When this limit is reached, requests are queued until thread pool threads become available. So is sound as if the thread pool provides queue functionality out of the box, and therefore nothing would be gained by using my own queue or what?

Another thing I am wondering about is the following comment from the excellent "C# 4.0 in a nutshell" book, page. 928: "There is an exception to the don't block rule. It's generally OK to block while calling a database server - if other threads are competing for the same server. This is because in a highly concurrent system, the database must be designed such that the majority of queries execute extremely quickly. If you end up with thousands of concurrent queries, it means the requests are hitting the database faster than it can process them. Thread economy is then the least of your worries."

I cannot see why it is OK to block on a db request, compared to blocking on other stuff, such as requests to other servers. Would it not be better NOT to block on database requests so the the thread is freed to serve other requests which may not need db access.

To sum up: Can thread economy be achieved by relying on the max number of thread pool threads, or would it be better to make a simple producer consumer queue, and why is it ok to block on call to database servers?

It is not okay to block TP threads on dbase queries. The quoted phrase stipulates that it is only okay if all of the TP threads are blocking on such queries. Can't argue with that but it seems rather artificial.

The threadpool manager's primary job is to ensure that it never runs more threads than there are available cores in the machine. Because that makes threading inefficient, context switching between threads is pretty expensive. That however won't work very well if an executing TP thread is blocking and not doing any real work. The TP manager isn't smart enough to know that a TP thread is blocking and cannot predict for how long it is going to block. Only the dbase engine would have a guess at it and it doesn't tell.

So the TP manager has a simple algorithm to deal with this, if none of the executing TP threads complete within a reasonable time then it allows another one to run. You now have more active threads than cpu cores. "Reasonable time" is half a second for the .NET TP manager.

This continues if necessary, additional threads are allowed to run as long as the existing ones are stuck in a rut.

Actually getting to the ThreadPool.GetMaxThreads() number of threads is incredibly unhealthy. It is a huge number, 250 times the number of cores in the machine. On a 4 core machine, it would take 999 executing threads that haven't made any progress for 499 seconds to get to the maximum. Those threads would consume a cool gigabyte of address space for their stacks. You are way beyond the "there's something wrong here" observation if it ever gets there.

There are some easy quantifiable numbers in this answer. Once an operation starts to take more than half a second then you need to start thinking about having it execute on a dedicated Thread. Burning half a second is only really possible by blocking so a Thread is much more appropriate than a TP thread. And yes, use a thread-safe queue to feed it with operation requests. It is also important that you put an upper limit on the number of pending requests so you don't flood the queue. Throttle the producer by blocking. And of course, don't forget what will happen a year from now. Databases never get faster when they age.