Web application background process, novice design question

advertisements

I'm building my first web application after many years of desktop application development (I'm using Django/Python but maybe this is a completely generic question, I'm not sure). So please beware - this may be an ultra-newbie question...

One of my user processes involves heavy processing in the server (i.e. user inputs something, server needs ~10 minutes to process it). On a desktop application, what I would do it throw the user input into a queue protected by a mutex, and have a dedicated background thread running in low priority blocking on the queue using that mutex.

However in the web application everything seems to be oriented towards synchronization with the HTTP requests.

Assuming I will use the database as my queue, what is best practice architecture for running a background process?


There are two schools of thought on this (at least).

  1. Throw the work on a queue and have something else outside your web-stack handle it.

  2. Throw the work on a queue and have something else in your web-stack handle it.

In either case, you create work units in a queue somewhere (e.g. a database table) and let some process take care of them.

I typically work with number 1 where I have a dedicated windows service that takes care of these things. You could also do this with SQL jobs or something similar.

The advantage to item 2 is that you can more easily keep all your code in one place--in the web tier. You'd still need something that triggers the execution (e.g. loading the web page that processes work units with a sufficiently high timeout), but that could be easily accomplished with various mechanisms.