Which framework, languages ​​or databases should I use for frequent updating of data in real time?


I'm working on a asset-tracking system that involves lots of packets of location information being updated every 5 seconds. We will be tracking about 30 gps units, which send information to a server that is controlled by the vendor. The vendor allows us to access the data through php requests. An example request looks like this:


The result of such a request is json formatted data. What I need to do is use that data to move markers on a web interface in real time. I plan on using jquery and ajax to get the job done. Here's the relevant code (I can't share the actual url, username, or password):

function requestLocation() {
        console.log('requesting data');
        reload = setInterval(loadLocData,1000);

//the following url request is for testing/devlopment and will change before final release
function loadLocData(){
    var now = new Date();
    var currentTime = now.getTime();
            type: "GET",
            url: //(the url goes here),
            username: //(my username goes here),
            password: //(my password goes here),
            data: {action: 'showposition',
                    operation: 'path',
                    format: 'json',
                    reqtype: 'dbid',
                    version: '2',
                    logvers: '3',
                    count: '5', //we will only use the five most recent points
                    starttime: ''+(currentTime-1500000),
                    endtime: ''+(currentTime+5000) //a five second buffer to ensure we're getting the most recent updates. may need to tweak this
            dataType: 'json',
            success: function(data){
                for(i=0;i<data.length;i++) {
                    //a function that will update the marker locations

The code returns the following error:

XMLHttpRequest cannot load https:///interface.php?action=showposition&operation=path&format=json&reqtype=dbid&version=2&logvers=3&count=5&starttime=1314037475842&endtime=1314038980842. Origin http://maraudergroup.com is not allowed by Access-Control-Allow-Origin.

After talking with our vendor's tech support, it seems I won't be able to import the data this way. I think what I need to do then, is use a script to grab the location data from their servers and write it to our own databases. Then instead of querying their databases, I can get the data from our own through ajax. However, I am unsure if this is the way to do things and I don't know which languages, frameworks, or database to use. Should I set up a cronjob to run the script every five seconds or so? Or is there a better way to do it? Or, even better, is there a workaround I can use to get the data with ajax? Whatever I use will need to handle frequent updates (every five seconds, or maybe even faster).

I have a little experience with php, mysql, and java. I'd also be willing to spend time to learn python, since I have a class on it this coming semester anyways.



Thanks for all your responses. As per Agustus's suggestion I was able to successfully implement a solution with php, although I came up with a solution before looking at the yahoo link. I'm not sure if what I've done could be considered a proxy, but it does work. I called the script getData.php and stored it on my server. Here's what it looks like (with sensitive data omitted):

$currentTime = time();
$r = new HttpRequest('https://(the domain name goes here)/interface.php', HttpRequest::METH_GET);
$r->addQueryData(array('username' => (my username here),
                       'password' => (my password here),
                       'action' => 'showposition',
                       'operation' => 'path',
                       'format' => 'json',
                       'reqtype' => 'dbid',
                       'version' => '2',
                       'logvers' => '3',
                       'starttime' => ''.($currentTime*1000-1500000),
                       'endtime' => ''.($currentTime*1000+5000)
try {
    echo $r->send()->getBody();
} catch (HttpException $ex) {
    echo $ex;

And (although you probably could've guessed what this looks like), here's the modified ajax request:

function loadLocData(){
            type: "GET",
            url: "php/getData.php",
            dataType: 'json',
            success: function(data){
                //a function that will update the marker locations

Thanks again!

The error message just means that it's not allowed to connect to another domain with XMLHTTPRequests (AJAX). So, write a little proxy to run on your own domain that just loads the data from the foreign server and passes it through to your JavaScript.

See Mozilla's documentation for more information on the matter.

Alternatively, you might request additional HTTP headers to loosen the browsers security policy. See Access-Control headers on the same page.
To get back to the question, your vendor's page would additionally need to send the following HTTP headers:

Access-Control-Allow-Origin: http://your-site.com
Access-Control-Allow-Credentials: true
Access-Control-Allow-Methods: GET

The first one grants permissions to your site to request data from the vendor's server via AJAX. The second header allows this even if credentials are in use (permits authentication with username and password). The third header allow only GET request since you don't need anything else you should go with that.

If the former part of the answer seemed too technical or you can't change the headers of your vendor's server/script your might find a proxy solution simpler. Yahoo has a not so technical introduction to AJAX and proxying which could be handy for you – even code samples are provided. Note that it does not matter much which language you choose for proxying.