Robot access

Posts: 7
Joined: Wed Jul 15, 2009 11:50 pm

Re: Robot access

Post by roncli » Thu Jul 16, 2009 12:15 am

Miles wrote:WTB more ideas. If there is something data you want to get out of the site without parsing the whole html page into machine readable stuff again - reply here.
A nice addition for each participant would be to add up their stats (including pets) from the raid. Perhaps this would be a separate page that you would pass a raid ID or something on to. Example stats would include damage done (total, DPS, and damage time), healing done (total, HPS, and healing time), and overhealing%. I'm sure others can think of more stats that would be useful as well.

Something like this would allow me to import raid data into my website and compare the data from raid to raid, identify weak spots and see who's pulling most of the weight. :)

Site Admin
Posts: 1524
Joined: Mon Mar 30, 2009 7:12 pm

Re: Robot access

Post by Miles » Tue Jul 21, 2009 12:49 am

Yeah. We're still messing with the long term stats, but so far, without luck. v1 exported damage/healing done, group by spell, group by player, group by fight, but that created an insane amount of data, even if you put it together per fight in a blob and zip it before sending it to the database; postgresql reported ~100k per report, x150k = we have a problem.

Incidentally, that was the reason why the site broke; reverted the feature, but forgot that it was "tweaked" already on live and it broke on indenting issues. Yay dynamic languages, things don't break on compile but on runtime. NOT. Oh well.

Try 2 will be a clone of WMO's player rankings - if you can't design better, copy from the best; we'll probably make that data available per guild via a feed.

Posts: 157
Joined: Mon Apr 13, 2009 3:31 am

Re: Robot access

Post by gcbirzan » Wed Jul 29, 2009 1:29 pm

Regarding our conversation earlier and my request for that log, can I just run a robot (as the previous poster, one request every 5 seconds) to get the full log? (I'm only interested in that particular one, won't be a regular thing)
Edit: Thinking of grabbing 2-3 seconds at a time, hm. Maybe once every 10 seconds would be more appropriate, since there are a lot of seconds in that log. :-P

Site Admin
Posts: 1524
Joined: Mon Mar 30, 2009 7:12 pm

Re: Robot access

Post by Miles » Wed Jul 29, 2009 1:33 pm

Yeah, no problem. The limits are mostly for the "SELECT * from reports" via html people.

Posts: 4
Joined: Tue Apr 21, 2009 2:34 pm

Re: Robot access

Post by scrusi » Fri Aug 21, 2009 12:47 pm

I tried to implement a feed of my guild's raids into our website, getting xml (or json, doesn't matter) from your page via curl. For some reason I always get a 301 header back with no additional info. (HTTP/1.1 301 MOVED PERMANENTLY)
Since my php runs in safe mode I can't activate CURLOPT_FOLLOWLOCATION . Any idea how I can fix this?

I'm parsing various pages via curl (i.e. armory, wowhead) and yours is the only one throwing me a 301.

edit: if I leave out the optional parameter I get a 200 response but no content. Here's the curl_getinfo:

Code: Select all

    [url] =>
    [content_type] => 
    [http_code] => 200
    [header_size] => 0
    [request_size] => 211
    [filetime] => -1
    [ssl_verify_result] => 0
    [redirect_count] => 0
    [total_time] => 0.054272
    [namelookup_time] => 0.001369
    [connect_time] => 0.006873
    [pretransfer_time] => 0.006956
    [size_upload] => 0
    [size_download] => 0
    [speed_download] => 0
    [speed_upload] => 0
    [download_content_length] => 0
    [upload_content_length] => 0
    [starttransfer_time] => 0.054131
    [redirect_time] => 0
I've played around with various curl options to no avail.

edit 2: the much simpler file_get_contents works fine, so my issue is solved for now. I'll leave this in case you ever mean to make the whole thing work with curl.

Site Admin
Posts: 2741
Joined: Sun Apr 05, 2009 4:14 pm

Re: Robot access

Post by Maihem » Fri Aug 21, 2009 11:30 pm

Getting a 301 Moved Permanently usually means you forgot to add a trailing slash to the URL. The server will then redirect you to the corrected URL. If you're sure you added the trailing slash and still want to use cURL, please paste a piece of code and I'll have a look at it.

UPDATE: cURL works fine for me, though it does indeed give a 301 when not using a trailing slash. See code block below.

Code: Select all


function get_http_status($url) {
    $ch = curl_init($url);
    curl_setopt_array($ch, array(
            CURLOPT_HEADER          => true,
            CURLOPT_RETURNTRANSFER  => true));
    $rv = curl_exec($ch);
    $eol = strpos($rv, "\n");
    return substr($rv, 0, $eol);

$url = '';

printf("No slash\t: %s\n", get_http_status($url));
printf("With slash\t: %s\n", get_http_status("$url/"));


Code: Select all

No slash:       HTTP/1.1 301 MOVED PERMANENTLY
With slash:     HTTP/1.1 200 OK

Posts: 4
Joined: Tue Apr 21, 2009 2:34 pm

Re: Robot access

Post by scrusi » Sat Aug 22, 2009 12:47 am

Going back it seems that I didn't use the slash when running the query with the added t parameter. At least I do get a 200 with raids/?t=xml now while raids?t=xml returns a 301.

Are you sure you changed nothing though? I'm absolutely certain that I got empty strings back even on the 200s before - and now I get fine json and xml with the same code.
Well, I suppose I had some kind of error that I fixed without noticing it ;)

Site Admin
Posts: 1524
Joined: Mon Mar 30, 2009 7:12 pm

Re: Robot access

Post by Miles » Sat Aug 22, 2009 6:34 pm

Yea I guess, haven't touched the this part of the site for quite a while now, last time we modified was 2 weeks ago, fixing a null date -> 500 bug.

Posts: 1
Joined: Tue Aug 25, 2009 3:38 am

Re: Robot access

Post by Aurless » Tue Aug 25, 2009 3:41 am

Any chance we could get a name attr on the <Boss> tags to avoid having to do a lookup on the NPC id? Unless you have a suggestion for a easy lightweight way to look up the string name of the npc?

Posts: 4
Joined: Fri Aug 28, 2009 5:56 am

Re: Robot access

Post by Vorend » Fri Aug 28, 2009 5:59 am

Hello, I was thinking about implementing a raid attendance tracking module for our guild with that feed. However, when I try to access it with our guild id, I get an Internal Server Error.

Could you look at it please? ... s/?t=plain

The error is the following:

Error: "'NoneType' object has no attribute 'id'" in getZones (/var/python/worldoflogs/src/wol/core/views/

Thanks a lot! ;)