Robot access

Robot access

Postby Miles » Mon Jun 15, 2009 9:59 pm

Please note that programmatic access to this site is forbidden unless it's on an explicitly allowed url.


We're working with EQDKP-Plus to provide an xml / json feed for the raid status, other feeds can be added if there is demand; but never, ever just crawl the whole site with a for-loop, that will get you on an IP ban very quickly.

Someone tried to crawl /guilds/{1-20000}/ today at 10Mbit; although the only effect was a spike on the cpu/network graph, that's not very nice and resulted in some banhammer action.

On known feed urls, we can take steps to ensure we don't take a performance hit even with lots of access.


Note: Data is provided under the Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License. If you wish to use it commercially, please contact us for more information.

Image

For the "attribution" part of the license, just a link back with raid data provided by World of Logs and the license (options; Image is fine) in the footer is enough.


Short FAQ

Q: So, can I use the data here on my guild homepage?
A: Yes. Just include a link back in the footer.

Q: And what about web services / software that builds on this idea?
A: It depends. CC didn't define exactly what non commercial is, but if you're just building something in your spare time, it's okay. If you're building a free software package (*dkp, like eqdkp) and use feeds from here, the users who install it are using it for non commercial purpose. But if you're a (free or paid) guild hoster, work for a company who wants to builds web services around this data, contact us for a commercial license. You wouldn't like the Sharealike part of the license anyway.
Miles
Site Admin
 
Posts: 1526
Joined: Mon Mar 30, 2009 7:12 pm

Re: Robot access

Postby Miles » Tue Jun 16, 2009 2:54 am

"Last 10 raids uploaded" feed is up. Note: only publicly visible raids are exported.

Url: http://www.worldoflogs.com/feeds/guilds/{ guild id }/raids/
Optional parameter: t = one of (json, xml, plain), defaults to json
Example: http://www.worldoflogs.com/feeds/guilds ... s/?t=plain


You're allowed to fetch this as much as you wish for your own guild, integrate it in your guild site, etc etc. But if it's possible, please respect the max-age of 15 minutes with local caching.

Extensive crawling (read: 100+ different guilds) isn't allowed unless okayed by an admin, send me an pm or email if you want to build something on top this db.
Miles
Site Admin
 
Posts: 1526
Joined: Mon Mar 30, 2009 7:12 pm

Re: Robot access

Postby hoho » Tue Jun 16, 2009 9:13 am

I don't have any immediate plans but how much scanning would be allowed without pissing you off or requiring special permission? Say one request per second/5 seconds maybe?
hoho
 
Posts: 313
Joined: Tue May 05, 2009 7:34 pm

Re: Robot access

Postby Miles » Tue Jun 16, 2009 1:23 pm

Yeah, that sounds reasonable on feeds; those can take a few k hits per minutes without a problem. Just don't go load reports every few seconds, that takes quite a bit of resource to process - you'll have to wait a bit longer on those.
Miles
Site Admin
 
Posts: 1526
Joined: Mon Mar 30, 2009 7:12 pm

Re: Robot access

Postby Miles » Mon Jun 22, 2009 3:43 pm

Er, totally forgot to publish the fields in the guild raids feed. So here it is:

Code: Select all
RaidList = {
   rows: [ Raid1, Raid2, ... ]
}

Raid = {
   date: milis since unix epoch, UTC,
   dateString: date formatted in guild timezone,
   duration: duration in seconds
   
   zones: [ Zone1, Zone2, ... ]
   
   bossCount: int, // distinct bosses seen
   killCount: int,
   wipeCount: int,
   
   damageDone: int, // set tp zero for the moment - not done with extended stats yet
   damageTaken: int,
   healingDone: int
   
   // any more ideas?
   
   // note: any object can gain extra fields, but it's safe to ignore them if you don't
   // understand the fields / need the data.
}

Zone = {
   id: int, // can be used to link to wowhead or guild stats @ WoL later
   name: name,
   difficulty: N or H,
   playerLimit: 10/25
   typeId: int, // WoL internal type; that's the 2 / 4096 you see on /fights/ urls. Used to link to those pages.
}

(can be a bit outdated, not sure. Copy pasted it from mailbox.)

WTB more ideas. If there is something data you want to get out of the site without parsing the whole html page into machine readable stuff again - reply here.
Miles
Site Admin
 
Posts: 1526
Joined: Mon Mar 30, 2009 7:12 pm

Re: Robot access

Postby Darganis » Tue Jun 30, 2009 4:12 pm

Would you be able to add a field that list the names of those that attended? I can parse the dashboard page and grab the list of attendees but in the feed would make this MUCH better. We do a lot with attendance for our raids and making sure every gets credit by pinging a reliable source is a great help.

I already love grabbing the raids via json. Thanks a tone for that.

- Darganis

http://dasypygal.dyndns.org/
Darganis
 
Posts: 1
Joined: Tue Jun 30, 2009 4:04 pm

Re: Robot access

Postby Miles » Tue Jun 30, 2009 4:37 pm

Sure, it's only a few extra lines. It's available now under the raid.participants key, value is a list of names. xml: Raid/Participants/Participant
Miles
Site Admin
 
Posts: 1526
Joined: Mon Mar 30, 2009 7:12 pm

Re: Robot access

Postby Phaye » Wed Jul 01, 2009 7:10 am

Miles wrote:WTB more ideas. If there is something data you want to get out of the site without parsing the whole html page into machine readable stuff again - reply here.


Any chance of getting a simplified RSS feed?

- Raid Date
- Raid Instance
- Link to Report

Would be kind of neat to have a very simplified feed we can plug into our sites, or readers that'll just quickly link us the last 10 raids for our guilds.
Phaye
 
Posts: 4
Joined: Wed Jul 01, 2009 7:07 am

Re: Robot access

Postby Miles » Wed Jul 01, 2009 3:32 pm

Hmm, I'll have to read up on how RSS works again then, the last time I wrote something for it there were like 3 standards competing, RSS-old vs RSS-new vs Atom...
Miles
Site Admin
 
Posts: 1526
Joined: Mon Mar 30, 2009 7:12 pm

Re: Robot access

Postby Phaye » Wed Jul 01, 2009 8:47 pm

Basically it's still the same though generally RSS 2.0 seems to be sticking around and widely accepted, leaving it just at RSS vs Atom.

I'm preferential to RSS over Atom, but if you can get either of the simplified feeds working, i can adjust our site accordingly.

BTW, absolutely love the log parser, graphs, etc. Within 1hr or trying this all out, I immediately abandoned WWS & WMO. :D

Thanks for all your effort & work.
Phaye
 
Posts: 4
Joined: Wed Jul 01, 2009 7:07 am

Next

Return to General Discussion

Who is online

Users browsing this forum: Baidu [Spider] and 1 guest

cron