How to use cURL to find subdomains containing specific strings?

I am trying to build a list of all the city pages on ghix.com, which does not have such a complete directory. To do this I am using their 'city id' which is unique for each city, but does not follow any particular order.

I am using cURL and PHP to loop through the possible domains to search for those that match actual cities. Simple enough. See the code bellow, which produces a 500 Internal Service Error.

If it worked, output should be a list of 'city ids' which do not match actual cities. If the URL matches an actual city it will not have (0) on the page, but if it does not match a city it will have (0) on the page.

I have looked over and corrected this several times, what is causing the error?

<html>
<?php
for ($i = 1; ; $i <= 1000000; $i++) {
  $url = "http://www.ghix.com/goto/dynamic/city?CityID=" . $i;
  $term="(0)";
  curl_setopt($ch, CURLOPT_URL, trim($url));
  $html = curl_exec($ch);
  if ($html !== FALSE && stristr($html, $term) !== FALSE) { // Found!
    echo $i;
    Echo "br/";
  }
}
?>
</html>

UPDATE a slightly different approach I tried, with the same effect...

<html>
<?php

for ($i = 1; $i <= 100; $i++) {
$url = "http://www.ghix.com/goto/dynamic/city?CityID=" . $i;
$term="(0)";
curl_setopt($ch, CURLOPT_URL, trim($url));
$html = curl_exec($ch);
if (strpos($ch,$term)) {
    echo $url;
    echo "<br>";
}
?>
</html>

In your first chunk of code, you have an extra ; in the for conditions. Next, you need to initialize cURL with $ch = curl_init(); around the beginning. That opens the handler $ch that you call on later. Finally, use 1= for the false condition for if instead of the exclamation and double equal. After these fixes, I'm not getting any 500 errors. After that, it's just a matter of collecting the data from the pages and putting it in the right places.

In the second chunk of code, you still need to initialize cURL. Then you need to put in another curly bracket at the end to close out the for loop. And then it's a matter of dealing with the output from cURL.

It seems you are getting more typo errors than anything. Watch your server logs and they will tell you more about what you are looking for. And for cURL, read up on the options that you can set in PHP on the PHP site. It's a good read. Good luck.

Related Articles