Analysis of NAF statistics

[AndeeT]

Sann; I think I get what you mean. Is it an artefact due to the way that some software 'sort' data?

Thanks dode for the clarification RE: home/away CAS. That makes much more sense! So it's more like "CAS inflicted" / "CAS sustained".... well I guess the differing values makes sense then. Still, I feel like there is some analysis to be had here ...

...talking of which; I have redone the graph with 95 % confidence intervals (+/- 1.96*SEM for a proportion), Ta da;

BB_W_CONF_INT.png

Now, that was already a busy graph, so I moved the race descriptions to the side. Even still, it can still be hard to compare confidence intervals, so, for mine, and everyone else's sake, I have made a very 1990's looking table (split in two) that shows when confidence intervals between races are overlapping. Look up the column of your race and compare to other races down the rows. If the comparison is green, confidence intervals between the two races are overlapping, i.e we don't have enough data to know if the difference in the races Win % are actually statistically significant. If the comparison is tan coloured, the races Win % are statistically significant.

For example, Ogre, Goblin and Halfling all overlap each others confidence intervals (this is actually an easy one to spot on the graph!)

Best Wishes

AndeeT

(Graphs to come)

[AndeeT]

First half of 95% confidence interval comparison table;

Confidence_Int_1.jpg

[AndeeT]

second half;

Confidence_Int_2.jpg

[AndeeT]

Hey everyone,

Another no frills scatter graph, this time comparing away CAS/game (or as I think of it, CAS suffered/game) to Win %, per race. As you can see there is a strong negative correlation (Pearson's Correlation Coefficient; -0.78).

AWAY_CAS_VS_WIN.png

Do the Home/away CAS count those CAS inflicted but later healed by apothecary I wonder? That may change things.

Source: http://fumbbldata.azurewebsites.net/stats2.html

Best Wishes,

AndeeT

[Moraiwe]

Taking more casualties makes it harder to win. I think we can file that last graph under "something everybody already knew."

(Sorry if I sound a bit negative here. I guess you do want to do the research in case a surprise turns up.)

[AndeeT]

Hi Moraiwe,

Not negative at all! I came to the same conclusion myself, as I guess others will. That said, as dode pointed out earlier, scatter graphs/correlations on their own can't imply that one thing causes the other, but in this case, if you take into account the design of the game (+anecdotal evidence) I think it is obvious as you say that less men = tougher time for you!

Interestingly, if you look at CAS difference (CAS inflicted - CAS suffered) there is almost no correlation. This may be due to the fact that different Low AV teams cope differently with CAS, i.e your high MA/high AG/low AV races (Skaven, Wood Elf/Elf) can pull off wins with very few men on the pitch while your low MA/low ag/Low AV teams (Halfling, Goblin, Ogre) struggle much more with less men.

Best Wishes,

AndeeT

[rolo]

Interesting chart!

What jumps out at me is how tightly grouped some of these teams are.

"Group A" is what most tournaments consider Tier 1, with Necromantic the lowest performing team in this group as befitting their reputation as "the worst tier 1 or the best tier 2" team. What's interesting to me is that team performance doesn't really seem related to team popularity within this group, although Undead and Wood Elves are sort of on a half-tier on their own as both the highest performing and most popular in this group.

"Group B" is all of the classic Tier 2 teams, and quite tightly packed - they're all significantly less popular than the "Tier 1" teams, although Pro Elves seem to do just fine in tournaments, the rest seem to perform visibly worse than Group A. They also seem to have a stronger relationship between performance and popularity, although that's just me eyeballing it - I don't want to start a 10 page flame war about statistics.

Stunties are both unpopular (although generally as popular as high end tier 2 teams) and terrible, but if you go to a tournament with stunties, you probably have other motivations than trying to maximize your win chances. I suspect Ogres get dragged down by the teams who load up on Piling On and lose games 0-8 or so, but there's no way to separate out "Ogre teams actually trying to win games" from "Ogre teams going for Most CAS" in your data set.

Then there's Orcs and Humans, who are outliers on this graph and look like they're more popular than they "Should" be given their performance. Both are clear tier 2 teams by win rate, but Humans are as popular as low end Tier 1 teams, while Orcs are the most popular team in the NAF (and also in most leagues!). Of course, those teams have been packaged in every boxed set since the 90s, are easy to find, and mostly straightforward to play. I think Orcs are also popular because they "Feel" safe; even if you're losing, you're probably beating up your opponent, and the Orcs perfectly embody the ideals of toughness, violence, and stupidity which are glorified in Blood Bowl lore.

Anyway. Thanks for making the charts!

[AndeeT]

Hey rolo,

Thanks! And thank you for your discussion of the graph. I couldn't agree more with your point about orcs and humans; there has to be some 'being in the box game' effect going on there. That's not to say that other races are played purely on their chances of winning; that may be a factor but as you say, some people like the challenge of e.g halflings. Then again, there could be myriad reasons for picking a particular team! I happen to like pointy hats and big noses so of course, Chaos Dwarves for me!

Nice point RE: Ogre CAS effect; not something I had considered.

I like your categorisation into groups and the eye does tend to group the teams in that way. However, it is definitely worthwhile looking at the 95% confidence intervals for the win% when trying to determine 'differences'. You will see that there is some overlap between groups and that they aren't as distinct as a first glance leads you to believe.

The tables that I posted earlier are great for quickly comparing groups. For example, high elf ('group b') share overlapping confidence intervals with necromantic ('group a') and orc ('orc' group). The last one is hard to see just on the graph as orcs as so far over the other side of the graph. The overlapping confidence intervals between high elf and necromantic and High elf and orc means that we can't conclude that high elf have a different win % to orc or necromantic. Does that makes sense?

Best Wishes,

AndeeT

[dode74]

AndeeT - is that graph based on pure win rates as opposed to (wins + draws/2)/GamesPlayed? The y-axis suggests as much.

As you rightly say, eyeballing graphs to find groupings is not sound methodology.

[AndeeT]

Dode - it's plain old win rate. I don't know enough about confidence intervals* to be able to construct confidence intervals around a measure like (win + draw)/2. I could do 1.96 * SEM for a proportion but I can't be certain that is technically correct.

Do you have an idea for how to calculate CI's for that measure dode? It's sort of a mean of two proportions no? But we don't have all contributing data points so we can't calculate sum of squared deviations from the mean. Treating it as a proportion (draws + wins) also makes sense but if we divide by two, how can we take that into account in the CI's

(Edit - confidence intervals of mathematics - this may be simple for someone with more maths skills than me - I only got as far as GCSE )


Best Wishes,

AndeeT

[dode74]

You can still use the proportion method when you don't have all the datapoints - all you're doing is counting a draw as half a win and half a loss.

[AndeeT]

My bad! I have been reading that measure wrong...woops.

I had taken it as;
((wins+draws)/2)

Which is why I started to think of it as a mean and then wondering how to calculate sum of squares from the mean without the data points.

Thank you for pointing out that it is 'counting a draw as half', which made me realise that it is;

((wins)+(draws/2))

That makes more sense! I don't think I got the logic behind it when CyberedElf asked for it at the start of the thread, but i do now. The formula they used was (wins + draws/2)/total, which, without the extra parenthesis, is a little open to interpretation (Sorry, I need things spelled out for me!).


I get that the CI formula for proportion doesn't need data points (handy that!).

My point around confidence interval construction is this;

Given that the formula for a 95 % confidence interval around a proportion is;

P +/- 1.96 * (SQRT(P(1-P)/n))

where P is the proportion and n is the sample size

And given that we are halving the magnitude of draws in the numerator of the standard error, it follows (my kind of ropey maths logic) that we should halve the magnitude of draws in the denominator of the standard error, i.e halve the sample size of draws. This would give us;

(WP+(DP/2)) +/- 1.96 * (SQRT((WP + (DP/2)) (1-WP-(DP-2))/(nWP+(nDP/2))

where WP is the win proportion, DP is the draw proportion, nWP is the win sample size and nDP is the draw sample size

Please let me know if that makes any sense at all or if I have missed something. As I say, I am not a mathematician by trade so that formula could very well be off.

Best Wishes,

AndeeT

[AndeeT]

I've been thinking a bit more about how to incorporate '(draws/2)' into the standard error denominator. I think my above attempt was a bit off as I forgot about the 'lose' sample size. So, a second attempt;



(WP+(DP/2)) +/- 1.96 * (SQRT(((WP+(DP/2))*(1-WP-(DP/2))/(nW+nL+(nD/2)))))

where WP is the win proportion, DP is the draw proportion, nW is the win sample size, nL is the lose sample size and nD is the draw sample size

Failing to take '(draws/2)' into account in the denominator of the standard error could result in confidence intervals that are smaller than they should be.

Best Wishes,

AndeeT

*EDIT to give Excel friendly version of formula and correct error

[AndeeT]

Doing some more thinking (!) and I've just realised a more logical way of summing up what I was trying to describe above. In other words, the problem with using (win+(draw/2)) and a standard confidence interval formula for a proportion, without taking into account how (draw/2) changes the relationship of the win/draw/loss proportions.

The problem being that the confidence interval formula for a proportion assumes 1 as the sum of all the 'parts' of the proportions (i.e win + lose + draw proportions = 1). When we calculate our measure as (win+(draw/2) we break that rule if we don't go back and recalculate the proportions based on the new total games played.

Take the Undead NAF total results as an example; Win/Draw/Loss; .445, .236, .319. Sum those and you get 1.

Recalculate as Win/(Draw/2)/Loss and you get .445/.118/.319 for a total of 0.882

If we are going down the route of (win+(draw/2) as our measure, it makes sense to re-calculate the Win/Lose/Draw proportions given that the total number of games are now equal to; (no. of games won) + (no. of games lost) + (no. of games drawn/2).

Doing this for undead we get the following Win/(Draw/2)/Loss proportions; 0.505/0.134/0.362. Which importantly (ignoring rounding error), sum to 1.

We should now be able to drop this into the 'standard' formula for a 95% confidence interval of a proportion;

P +/- 1.96 * (SQRT(P(1-P)/n))

i.e.

(WP+(DP/2)) +/- 1.96 * (SQRT((WP+(DP/2))*(1-(WP+(DP/2)))/n))

where n is the revised total number of games, WP is the revised win proportion and DP is the revised draw proportion

Please let me know what you think.

Best Wishes,

AndeeT

[Moraiwe]

I've always felt that Win rate being defined as (W+D/2)/G is a somewhat flawed measure of success given that many leagues/tourneys (if not the majority) value draws at less than half a win.

For the leagues/events I play in, (W+D/3)/G gives a more accurate representation of success.