Mining data from BGG

[dyeyk2000]

Hi All,

I'm sure many of us here are very familiar with boardgamegeek.com aka BGG. It's certainly an indispensable tool for any board gamer.

BGG also has a very robust data bank of all the games, including ratings, bayesian ratings, standard dev of ratings, etc aka nerdox heaven haha. Now just like any nerdox, I wanna play with these numbers. But to do that, I need to mine the data. And no, manually opening each game page and taking down notes is not a very smart option. Haha.

So TTW faithful, is there a way to "mine" the data from BGG? I've heard about XML, but I'm just not that techy (yet ). In particular, I'm hoping to mine the past rating of a game, say from a previous month. You know.. nerdy stuff like that hehe.

Any ideas?

Thanks guys!

[maskmanjoe]

wow i didnt know you were that interested in the data collected by BGG? Out of curiosity, to what end would you use this? I mean most visitors and regulars of BGG are only interested in the qualitative, high level ones like reviews, comments, session reports and the like (plus the occasional boardgame girl or boardgame cat pic). Very few are interested in the numbers. Some people dont even give much value to the ranking system of BGG as they dont like how it ranks games based on the players' willingness to play a game again.

The few who do crunch the numbers wanted to show some statistically measurable hypothesis. Bout you, what are you interested in?

You can probably message Aldo and the people behind BGG. There may be a system already in place for folks interested in accessing their numerical data.

[dyeyk2000]

Disclaimer: I'm a excel whore beyond belief. I'll fiddle with all kinds of numbers. Lalo na if hobby haha.

With that out of the way. You are correct on all accounts MMJ, but I believe the BGG rating system is still robust enough for the purpose it serves. To many, it basically asks a very simple question -- rate a game from 1 to 10.

I'm mostly focused on what seems to be an immovable top 100 games on the website. Like you pointed out, there are flaws in the system and I would say this is one of them. The "geek rating" uses bayesian estimates that rewards games with numerous votes. Games like El Grande, released in 1995, benefits from the over 10,000 votes it has received ever since. New ratings on El Grande barely register in the standings anymore. On the other side of the coin, new games have trouble getting any upward traction in the rankings because of the low vote count that they register compared to the giants.

Basically, my theory is games like Caylus have long been receiving poor votes than are apparent. It's just a product of the times as the game design begins to pale in comparison to newer games. However, this doesn't register in the rankings due to the huge voting base it has accumulated. At the same time, a newer game that would probably better appeal to today's crowd fails to get any exposure in the rankings.

I'm doing this for people new to the hobby I guess. Research has shown that people will always tend to gravitate towards the popular. By virtue that popular well rated games are probably very good games. However, the top 100 data is flawed and does not actually show the popular well rated games by today's standards. I'm hoping this data can fix that.

Yeah I've already tried asking a couple of forumers, but not yet Aldie. I'll probably get to that soon.

[Adrian (the other one)]

@Jake:

Anak ka nga ng ME. Hahaha.  I haven't seen this numbers-fetish since my good friend from ME moved to Spain.  Have you asked the guys back at the department? Aly perhaps?

[dyeyk2000]

Adrian - Who's this ME friend of yours? Baka kilala ko haha.

Oooh does Aly play boargdames also? Hehe.

[Talismanic (Mark)]

Interestingly enough, Twilight Struggle is at numero uno with 6000+ votes (Take that PR!). Ha ha ha, the problem with BGG rankings is that its really a "popularity contest" and not necessarily a rating of "playing quality". Also, I see many of the more recent games have actually penetrated the list, Dominion, Agricola, Le Havre, etc. Its really a question of enough BGGers playing something and saying "this is worth playing".

That's why there are also thematic rankings, geeklists, and individual reviews one can look at. The problem of rankings is more apparent than real, as there are many many sources for information on a game, including the rules of the game itself which is now often released ahead of the game's release. Also, every year there are top ten lists from every reviewer and gamesite, as well as awards.

To be sure, however, actually playing the game is the only real way to know if you'll like it.   

[dyeyk2000]

To be sure, however, actually playing the game is the only real way to know if you'll like it.   

Chakit cha bulcha eh >,<

To your point Talismanic, it really is a popularity contest -- and in this case, the rich get richer -- generate enough buzz and you can assure a snug position in the rankings and tons of reviews. Sana there's a way to get those lesser known good games some exposure lang, and hopefully, gain some traction.

[Adrian (the other one)]

Adrian - Who's this ME friend of yours? Baka kilala ko haha.

Oooh does Aly play boargdames also? Hehe.

I'm sure you know him.  Ralph's a common friend of ours, I believe.  Though he should be back for vacation by Christmas-time and finished his Spain MBA by end of next term.  I've known ralph to be as number-curious just like you're showing here sa forums.

Re: Aly.  Hahaha. I don't think he plays boardgames -- but he might be QMIT-savvy and tech-savvy enough to answer your problem.

[Adrian (the other one)]

Interestingly enough, Twilight Struggle is at numero uno with 6000+ votes.

Boo-YAH!

[Adrian (the other one)]

To be sure, however, actually playing the game is the only real way to know if you'll like it.   

Chakit cha bulcha eh >,<

Actually, the easiest way I've found is just to go around the community, join the different meet-ups, and try out games other people own.

Initially, stepping out of your comfort zone, gaming with people you don't know, or even travelling far from your usual 'hunting grounds' can be difficult -- but it's really the best way to sample them all.

I've gotten to play games I'm interested in that afterwards, I don't feel like buying anymore.  Also, I've played games that I didn't know existed, and afterwards felt the urge to buy my very own copy.

Gaming with everyone else's collection feels like I've got 500 games, instead of just under a hundred.

[dyeyk2000]

I'm sure you know him.  Ralph's a common friend of ours, I believe.

Ah of course, Ralph. He's definitely the type haha.

Let's see, I haven't spoken to Aly in ages.

Interestingly enough, Twilight Struggle is at numero uno with 6000+ votes.

Boo-YAH!

This was the game Arnold and I ended up playing (a learning game, which we expectedly didn't finish). Definitely a very very interesting game. I was tempted to open my own still shrink-wrapped purchase last night just to rummage through it. Haha.

Actually, the easiest way I've found is just to go around the community, join the different meet-ups, and try out games other people own.

Oh yes. And that too. Haha. Well aware of this and definitely need to attend one of the bigger meets soon. Till then.. playing with numbers is fun! (harhar nerd)

[Talismanic (Mark)]

Interestingly enough, Twilight Struggle is at numero uno with 6000+ votes.

Boo-YAH!

Yeah, kaso as much as I like TS, the present controversy is really making me doubt the rating system sa BGG. Its become a ratings war with "10s" and "1s" being given arbitrarily, as the geeks scramble to prop up their favorites.

In any case, they should modify the voting system. I've never been a fan of giving each person a vote, you should earn the privilege to vote, or the right must be regulated at least.   

[Talismanic (Mark)]

It's really sad that the farmer and plantation-owner fans are so emotionally tied to the ranking of their favorite game (or game genre, i.e. euros) that they would actually take the time to enter bogus "1" ratings on TS just to keep it out of its rightful #1 spot. I'd love to see some of these individuals admit to their actions and explain themselves. Unfortunately, people who do this kind of thing usually avoid the light.

-Leo Zappa
(desertfox2004)

The internet drama about the rankings is COMEDY GOLD! So it turns out its a bunch of rabid Euro players trying to take out the wargame! Hilarious.

Check out:

http://www.boardgamegeek.com/thread/593728/30-of-all-1-ratings-given-in-the-last-5-weeks-/page/1

[Talismanic (Mark)]

Wow, I knew there were a few jackoffs rating things like ass-hats, but looking into it for a minute makes it clear that the ratings system is a complete joke. I just looked at the people rating TS a 1, and it includes a guy who has rated nearly 500 games, giving 360 of them a 1. I always wondered why every game seems to rank between 6.5 and 7.5, and now I know: Ass-hattery. My guess is that a small group of people makes a bunch of accounts, log into each of them and trash any game that they care to with a bunch of 1's, and give a bunch of 10's to other games.

This site should dump and disregard any rater whose ratings don't fall into something resembling a bell curve, or people who create an account, dump a bunch of 1's and 10's into the rating system, and never log in again.

I think it the site policies for BGG are often gutless, and render some site functions worthless, and this is just another case.

Dan Lokemoen
(someotherguy)

Ha ha ha ha. Funny stuff, but true as well.

[cramnhoj]

That's why I don't really check the statistics for games on BGG.  If I want to look into a game I just read reviews to see if it's something I like. 

As for the data mining, you'll need to make a program to scrape the data you need then just dump it onto a database or excel file and you can crunch it as much as you like. Dunno if there's a readily made program for this but doing it the hard way would incur accessing each game's page parsing the html for the part of the page you need, namely the statistics, which should be doable in most programming languages.