Software needed to reproduce this blog post

This blogpost is written as a Jupyter notebook containing Python code, and is fully reproducible. The idea is to make Blood Bowl data analysis accessible to others. Using open source tooling reduces the barriers for others to build on other people’s work.

import random
import time

import requests # API library

import numpy as np
import pandas as pd

pd.set_option('display.max_rows', 500)
pd.set_option('display.max_columns', 500)

Blood Bowl online: FUMBBL

The FUMBBL website (https://fumbbl.com) contains a large amount of data. From coach pages, with their teams, to team rosters, with players, and match histories. It’s all there.

To obtain FUMBBL data, we need to fetch it match by match, team by team. To do so, the site creator Christer Kaivo-oja, from Sweden, has made an API that allows us to easily fetch data. What follows is a short demonstration how the API works, before we fetch the FUMBBL match and team data of the last 12 months.

Behold, the power of Requests

I use the Python Requests library to make the API call over HTTPS and obtain the response from the FUMBLL server. The response is in the JSON format, a light-weight data-interchange format which is both easy to read and write for humans, and easy to parse and generate by computers. So this makes it a natural choice for an API.

Here is an example of what is available at the coach level. The full documentation of the API can be found at (https://fumbbl.com/apidoc/).

response = requests.get("https://fumbbl.com/api/coach/teams/gsverhoeven")
# display the complete JSON object {}
response.json()

{'id': 255851,
 'name': 'gsverhoeven',
 'teams': [{'id': 1003452,
   'coachId': 255851,
   'name': 'Hillywood Hellraisers',
   'rosterId': 50,
   'race': 'Human',
   'teamValue': 1090000,
   'canLfg': 'Yes',
   'isLfg': 'Yes',
   'games': '3',
   'divisionId': 1,
   'division': 'Ranked',
   'leagueId': 0,
   'league': None,
   'status': 'Retired',
   'raceLogos': [{'size': 32, 'logo': 486290},
    {'size': 48, 'logo': 486291},
    {'size': 64, 'logo': 486292},
    {'size': 96, 'logo': 486293},
    {'size': 128, 'logo': 486294},
    {'size': 192, 'logo': 486295}]},
  {'id': 1035833,
   'coachId': 255851,
   'name': 'Pharaoh Munchers',
   'rosterId': 51,
   'race': 'Tomb Kings',
   'teamValue': 960000,
   'canLfg': 'Yes',
   'isLfg': 'No',
   'games': '0',
   'divisionId': 1,
   'division': 'Ranked',
   'leagueId': 0,
   'league': None,
   'status': 'Active',
   'raceLogos': [{'size': 32, 'logo': 486296},
    {'size': 48, 'logo': 486297},
    {'size': 64, 'logo': 486298},
    {'size': 96, 'logo': 486299},
    {'size': 128, 'logo': 486300},
    {'size': 192, 'logo': 486301}]},
  {'id': 1035835,
   'coachId': 255851,
   'name': 'Blackbox Bastards',
   'rosterId': 50,
   'race': 'Human',
   'teamValue': 840000,
   'canLfg': 'No',
   'isLfg': 'No',
   'games': '1',
   'divisionId': 10,
   'division': 'Blackbox',
   'leagueId': 0,
   'league': None,
   'status': 'Retired',
   'raceLogos': [{'size': 32, 'logo': 486290},
    {'size': 48, 'logo': 486291},
    {'size': 64, 'logo': 486292},
    {'size': 96, 'logo': 486293},
    {'size': 128, 'logo': 486294},
    {'size': 192, 'logo': 486295}]},
  {'id': 1036599,
   'coachId': 255851,
   'name': 'Seven cities of Gold',
   'rosterId': 52,
   'race': 'Lizardmen',
   'teamValue': 1000000,
   'canLfg': 'Yes',
   'isLfg': 'No',
   'games': '0',
   'divisionId': 1,
   'division': 'Ranked',
   'leagueId': 0,
   'league': None,
   'status': 'Active',
   'raceLogos': [{'size': 32, 'logo': 486302},
    {'size': 48, 'logo': 486303},
    {'size': 64, 'logo': 486304},
    {'size': 96, 'logo': 486305},
    {'size': 128, 'logo': 486306},
    {'size': 192, 'logo': 486307}]},
  {'id': 1038960,
   'coachId': 255851,
   'name': 'Hillywood Hellraisers 2.0',
   'rosterId': 50,
   'race': 'Human',
   'teamValue': 1110000,
   'canLfg': 'Yes',
   'isLfg': 'Yes',
   'games': '4',
   'divisionId': 1,
   'division': 'Ranked',
   'leagueId': 0,
   'league': None,
   'status': 'Active',
   'raceLogos': [{'size': 32, 'logo': 486290},
    {'size': 48, 'logo': 486291},
    {'size': 64, 'logo': 486292},
    {'size': 96, 'logo': 486293},
    {'size': 128, 'logo': 486294},
    {'size': 192, 'logo': 486295}]},
  {'id': 1050267,
   'coachId': 255851,
   'name': 'Gooische Heidebeukers',
   'rosterId': 4974,
   'race': 'Orc',
   'teamValue': 1330000,
   'canLfg': 'Yes',
   'isLfg': 'Yes',
   'games': '6',
   'divisionId': 5,
   'division': 'League',
   'leagueId': 14883,
   'league': 'Benelux Hate Bowl',
   'status': 'Post Match Sequence',
   'raceLogos': [{'size': 32, 'logo': 486332},
    {'size': 48, 'logo': 486333},
    {'size': 64, 'logo': 486334},
    {'size': 96, 'logo': 486335},
    {'size': 128, 'logo': 486336},
    {'size': 192, 'logo': 486337}]},
  {'id': 1052980,
   'coachId': 255851,
   'name': '[2020] Hillywood Hellraisers',
   'rosterId': 4964,
   'race': 'Human',
   'teamValue': 1000000,
   'canLfg': 'Yes',
   'isLfg': 'No',
   'games': '0',
   'divisionId': 2,
   'division': 'Competitive',
   'leagueId': 0,
   'league': None,
   'status': 'Active',
   'raceLogos': [{'size': 32, 'logo': 486290},
    {'size': 48, 'logo': 486291},
    {'size': 64, 'logo': 486292},
    {'size': 96, 'logo': 486293},
    {'size': 128, 'logo': 486294},
    {'size': 192, 'logo': 486295}]},
  {'id': 1053065,
   'coachId': 255851,
   'name': '[2020] Grrrl power',
   'rosterId': 5141,
   'race': 'Amazon',
   'teamValue': 960000,
   'canLfg': 'Yes',
   'isLfg': 'No',
   'games': '0',
   'divisionId': 2,
   'division': 'Competitive',
   'leagueId': 0,
   'league': None,
   'status': 'Active',
   'raceLogos': [{'size': 32, 'logo': 486194},
    {'size': 48, 'logo': 486192},
    {'size': 64, 'logo': 486195},
    {'size': 96, 'logo': 486191},
    {'size': 128, 'logo': 486193},
    {'size': 192, 'logo': 486190}]},
  {'id': 1060696,
   'coachId': 255851,
   'name': 'Hakflem support team',
   'rosterId': 4978,
   'race': 'Underworld Denizens',
   'teamValue': 730000,
   'canLfg': 'Yes',
   'isLfg': 'No',
   'games': '0',
   'divisionId': 2,
   'division': 'Competitive',
   'leagueId': 0,
   'league': None,
   'status': 'Active',
   'raceLogos': [{'size': 32, 'logo': 603383},
    {'size': 48, 'logo': 603381},
    {'size': 64, 'logo': 603378},
    {'size': 96, 'logo': 603382},
    {'size': 128, 'logo': 603380},
    {'size': 192, 'logo': 603379}]}]}

Parsing JSON data

Let’s have a closer look at the JSON data structure here. We have a list of key-value pairs. Using brackets [] we can choose keys and retrieve their values. Some keys contain simple values, such as name,

response.json()['name']

'gsverhoeven'

but some return as value a new list of key-value pairs, such as teams. Actually this is a “list of”lists of key-value pairs”, since we have a separate list for each team. Even the list of a single team contains new structure, for example under the key raceLogos.

response.json()['teams'][2]['raceLogos']

[{'size': 32, 'logo': 486290},
 {'size': 48, 'logo': 486291},
 {'size': 64, 'logo': 486292},
 {'size': 96, 'logo': 486293},
 {'size': 128, 'logo': 486294},
 {'size': 192, 'logo': 486295}]

response.json()['teams'][2]['name']

'Blackbox Bastards'

What data do we need? And in what shape?

Now we know how the data comes in, we need to think about which variables we want, and how to structure them. The most straightforward level to analyze race strength is to look at match outcomes. At its core, the data consists of matches played by teams, commanded by coaches. Furthermore, we expect race strength to change over time, as new strategies are discovered by the players, or new rules get introduced. So the time dimension is important as well.

So, let’s go with a flat data frame with rows for each match, and columns for the various variables associated with each match. These would include:

• Coach ids
• Team races
• Team ids
• Date of the match
• Outcome (Touchdowns of both teams)

With this basic structure, we can add as many match related variables in the future, keeping the basic structure (each row is a match) unchanged.

So lets get the match data!

Step 1: API scraping the match data: df_matches

So we are mostly interested in the current ruleset, this is BB2020. This ruleset became available for play on FUMBBL at september 1st 2021, and two months later, already some 5000 games have been played. We also want to compare with the previous ruleset, where we have much more data available. How far do we go back? Let’s go for roughly 12 months of BB2016 ruleset matches, and a few months of BB2020 matches.

The easiest way to collect match data over a particular period of time is to just loop over match_id. The most recent match at the time of writing was 4.347.800, and since roughly 100.000 matches are played each year, we can fiddle about and we find match 4.216.258 played on august 1st, 2020. So that means we need to collect some 130K matches.

VERY IMPORTANT: We do not want to overload the FUMBBL server, so we make only three API requests per second. In this way, the server load is hardly affected and it can continue functioning properly for all the Blood Bowl coaches playing their daily games!

To collect 130K matches, we will need 130000*0.333/3600 = 12 hours.

# estimated hours fetching data
(4347800-4216257)*0.333/3600

12.16773

df_matches = pd.DataFrame(columns=['match_id', 'match_date', 'match_time',  
    'team1_id', 'team1_coach_id', 'team1_roster_id', 'team1_race_name', 'team1_value',
    'team2_id', 'team2_coach_id', 'team2_roster_id', 'team2_race_name', 'team2_value',
    'team1_score', 'team2_score'])

target = 'data/df_matches_' + time.strftime("%Y%m%d_%H%M%S") + '.h5'
print(target)

end_match = 4347800
begin_match = 4216258
n_matches = end_match - begin_match
full_run = 0
print(n_matches)

if(full_run):
    for i in range(n_matches):
        api_string = "https://fumbbl.com/api/match/get/" + str(end_match - i)
        # wait 0.33 s on average between each API call
        wait_time = (random.uniform(0.5, 1) + 0.25)/3
        time.sleep(wait_time)
        match = requests.get(api_string)
        match = match.json()
        if match: # fix for matches that do not exist
            match_id = match['id']
            match_date = match['date']
            match_time = match['time']
            team1_id = match['team1']['id']
            team2_id = match['team2']['id']
            team1_score = match['team1']['score']
            team2_score = match['team2']['score']  
            team1_roster_id = match['team1']['roster']['id']
            team2_roster_id = match['team2']['roster']['id']            
            team1_coach_id = match['team1']['coach']['id']
            team2_coach_id = match['team2']['coach']['id']
            team1_race_name = match['team1']['roster']['name'] 
            team2_race_name = match['team2']['roster']['name'] 
            team1_value = match['team1']['teamValue']
            team2_value = match['team2']['teamValue']
            #print(match_id)     
            df_matches.loc[i] = [match_id, match_date, match_time, 
                team1_id, team1_coach_id, team1_roster_id, team1_race_name, team1_value,
                team2_id, team2_coach_id, team2_roster_id, team2_race_name, team2_value,
                team1_score, team2_score]
        else:
            # empty data for this match, create empty row
            match_id = int(end_match - i)
            df_matches.loc[i] = [np.NaN, np.NaN, np.NaN, 
            np.NaN,np.NaN,np.NaN,np.NaN,
            np.NaN,np.NaN,np.NaN,np.NaN,
            np.NaN,np.NaN, np.NaN, np.NaN] # try np.repeat([np.NaN], 13, axis=0) next time
            df_matches.loc[i]['match_id'] = int(match_id)
        if i % 100 == 0: 
            # write tmp data as hdf5 file
            print(i, end='')
            print(".", end='')
            df_matches.to_hdf(target, key='df_matches', mode='w')

    # write data as hdf5 file
    df_matches.to_hdf(target, key='df_matches', mode='w')
else:
    # read from hdf5 file
    df_matches = pd.read_hdf('data/df_matches_20211215_212935.h5')

df_matches.shape

data/df_matches_20211230_192705.h5
131542

(131542, 15)

Since we manually filled the pandas DataFrame, most of the columns are now of object datatype. We need to change this to be able to work properly with the data, as well as store it properly. Here I convert each column manually, however I later found out about DataFrame.infer_objects(), that can detect the proper dtype automatically. This I will try next time.

# convert object dtype columns to proper pandas dtypes datetime and numeric
df_matches['match_date'] = pd.to_datetime(df_matches.match_date) # Datetime object
df_matches['match_id'] = pd.to_numeric(df_matches.match_id) 
df_matches['team1_id'] = pd.to_numeric(df_matches.team1_id) 
df_matches['team1_coach_id'] = pd.to_numeric(df_matches.team1_coach_id) 
df_matches['team1_roster_id'] = pd.to_numeric(df_matches.team1_roster_id) 
df_matches['team2_id'] = pd.to_numeric(df_matches.team2_id) 
df_matches['team2_coach_id'] = pd.to_numeric(df_matches.team2_coach_id) 
df_matches['team2_roster_id'] = pd.to_numeric(df_matches.team2_roster_id) 
df_matches['team1_score'] = pd.to_numeric(df_matches.team1_score) 
df_matches['team2_score'] = pd.to_numeric(df_matches.team2_score) 

# calculate match score difference
df_matches['team1_win'] = np.sign(df_matches['team1_score'] - df_matches['team2_score'])
df_matches['team2_win'] = np.sign(df_matches['team2_score'] - df_matches['team1_score'])

# mirror match
df_matches['mirror_match'] = 0
df_matches.loc[df_matches['team1_race_name'] == df_matches['team2_race_name'], 'mirror_match'] = 1

# 5K mirror matches
df_matches.query('mirror_match == 1').shape

(5081, 18)

# Make BB2020 Khorne team name equal to BB2016 to compare them more easily
df_matches.loc[df_matches['team1_race_name'] == "Khorne", 'team1_race_name'] = 'Daemons of Khorne'
df_matches.loc[df_matches['team2_race_name'] == "Khorne", 'team2_race_name'] = 'Daemons of Khorne'

# convert team value 1100k to 1100 integer and and above / below median (= low / high TV)
df_matches['team1_value'] = df_matches['team1_value'].str.replace('k$', '')
df_matches['team1_value'] = df_matches['team1_value'].fillna(0).astype(np.int64)

df_matches['team2_value'] = df_matches['team2_value'].str.replace('k$', '')
df_matches['team2_value'] = df_matches['team2_value'].fillna(0).astype(np.int64)

df_matches['tv_diff'] = np.abs(df_matches['team2_value'] - df_matches['team1_value'])

df_matches['tv_match'] = df_matches[["team1_value", "team2_value"]].max(axis=1)

df_matches['tv_bin'] = pd.cut(df_matches['tv_match'], 
    bins = [0, 950, 1250,1550, 1850, float("inf")], 
    labels=['< 950', '1.1K', '1.4K', '1.7K', '> 1850']
)

df_matches = df_matches.dropna(subset=['match_date'])

# 131K matches
len(df_matches)

131509

df_matches['week_number'] = df_matches['match_date'].dt.isocalendar().week

# cannot serialize numpy int OR Int64 when writing HDF5 file, so we go for plain int as all NAs are gone now
df_matches['week_number'] = df_matches['week_number'].fillna(0).astype(int)

# add year based on match date (but want it based on match ISO week)
df_matches['year'] = pd.DatetimeIndex(df_matches['match_date']).year

# manual fix year for ISO week 2020-53 (2020 has 53 ISO weeks, including a few days in jan 2021)
df_matches.loc[(df_matches['year'] == 2021) & (df_matches['week_number'] == 53), 'year'] = 2020

df_matches['week_year'] = df_matches['year'].astype(str) + '-' + df_matches['week_number'].astype(str)

df_matches['week_date'] = pd.to_datetime(df_matches['week_year'].astype("string") + '-1', format = "%Y-%U-%w")

# manual fix of week date (grrrr)
df_matches.loc[(df_matches['week_date'] == '2021-01-04') & (df_matches['week_number'] == 53), 'week_date'] = pd.to_datetime('2020-12-31')

Step 2: HTML Scraping the inducements and coach rankings for each match

Next, we collect for all the matches in df_matches the inducements and coach rankings. This information is not available through the API, but is presented on a HTML page at https://fumbbl.com/FUMBBL.php?page=match&id=4350014 summarizing information for in this case match 4350014.

I highly recommend this tutorial for a great introduction to BeautifulSoup.

In addition, to clean up the scraped text, I used the re Python module (Regular expressions), part of the Python standard library to extract the actual inducements from the text string that contains them.

from bs4 import BeautifulSoup
import re

df_inducements = pd.DataFrame(columns=['match_id', 'team1_inducements', 'team2_inducements', 'coach1_ranking', 'coach2_ranking'])

target = 'data/df_inducements_' + time.strftime("%Y%m%d_%H%M%S") + '.h5'
print(target)

end_match = 4347800  
begin_match = 4216257

n_matches = end_match - begin_match

full_run = 0

print(n_matches)

if(full_run):
    for i in range(n_matches):
        match_id = end_match - i
        api_string = "https://fumbbl.com/FUMBBL.php?page=match&id=" + str(match_id)
        # wait 0.33 s on average between each GET call
        wait_time = (random.uniform(0.5, 1) + 0.25)/3
        time.sleep(wait_time)
        response = requests.get(api_string)

        soup = BeautifulSoup(response.content, 'html.parser')

        if soup.find("div", {"class": "matchrecord"}) is not None:
            # match record is available
            inducements = soup.find_all("div", class_="inducements")

            pattern = re.compile(r'\s+Inducements: (.*)\n')

            match = re.match(pattern, inducements[0].get_text())
            if match:
                team1_inducements = match.group(1)
            else:
                team1_inducements = ''

            match = re.match(pattern, inducements[1].get_text())
            if match:
                team2_inducements = match.group(1)
            else:
                team2_inducements = ''

            coach_rankings = soup.find_all("div", class_="coach")

            coach1_ranking = coach_rankings[0].get_text()
            coach2_ranking = coach_rankings[1].get_text()

            df_inducements.loc[i] = [match_id, team1_inducements, team2_inducements, coach1_ranking, coach2_ranking]

        if i % 100 == 0: 
                    # write tmp data as hdf5 file
                    print(i, end='')
                    print(".", end='')
                    df_inducements.to_hdf(target, key='df_inducements', mode='w')

    # write data as hdf5 file
    df_inducements.to_hdf(target, key='df_inducements', mode='w')
else:
    # read from hdf5 file
    
    df_inducements = pd.read_hdf('data/df_inducements_20211215_112148.h5') 


df_inducements.info()

data/df_inducements_20211230_192709.h5
131543
<class 'pandas.core.frame.DataFrame'>
Int64Index: 131509 entries, 0 to 131508
Data columns (total 5 columns):
 #   Column             Non-Null Count   Dtype 
---  ------             --------------   ----- 
 0   match_id           131509 non-null  object
 1   team1_inducements  131509 non-null  object
 2   team2_inducements  131509 non-null  object
 3   coach1_ranking     131509 non-null  object
 4   coach2_ranking     131509 non-null  object
dtypes: object(5)
memory usage: 6.0+ MB

df_inducements.query("match_id == 4347799")

#pattern = re.compile(r'\s+Inducements: (.*)\n')
pattern = re.compile(r'.*CR (.*)\s\(.*')

match = re.match(pattern, df_inducements.loc[0, 'coach2_ranking'])

if match is not None:
    print(match.group(1)) # group(0) is the whole string
else:
    print("match is none")

df_inducements.loc[0, 'coach2_ranking']

144.51

'LoxodonP Veteran CR 144.51 (-0.76)'

# Dataprep fix match_id
df_inducements['match_id'] = pd.to_numeric(df_inducements.match_id) 

# Dataprep: add the coach rankings as separate cols
df_inducements['coach1_CR'] = df_inducements['coach1_ranking'].str.extract(r'.*CR (.*)\s\(.*')
df_inducements['coach2_CR'] = df_inducements['coach2_ranking'].str.extract(r'.*CR (.*)\s\(.*')

df_inducements['coach1_CR'] = pd.to_numeric(df_inducements['coach1_CR'])
df_inducements['coach2_CR'] = pd.to_numeric(df_inducements['coach2_CR'])

df_inducements['CR_diff'] = np.abs(df_inducements['coach1_CR'] - df_inducements['coach2_CR'])
df_inducements['CR_diff'] = df_inducements['CR_diff'].astype(float)

df_inducements['cr_diff2'] = df_inducements['coach1_CR'] - df_inducements['coach2_CR']

df_inducements['cr_bin'] = pd.cut(df_inducements['cr_diff2'], bins = [-1*float("inf"), -30, -20, -10, -5, 5, 10, 20, 30, float("inf")], 
 labels=['{-Inf,-30]', '[-30,-20]', '[-20,-10]', '[-10,-5]', '[-5,5]', '[5,10]', '[10,20]', '[20,30]', '[30,Inf]']) 

df_inducements.dtypes

match_id                int64
team1_inducements      object
team2_inducements      object
coach1_ranking         object
coach2_ranking         object
coach1_CR             float64
coach2_CR             float64
CR_diff               float64
cr_diff2              float64
cr_bin               category
dtype: object

team1_inducements = df_inducements[['match_id', 'team1_inducements']]
team2_inducements = df_inducements[['match_id', 'team2_inducements']]

# make column names equal
team1_inducements.columns = team2_inducements.columns = ['match_id', 'inducements']

# row bind the two dataframes
inducements = pd.concat([team1_inducements, team2_inducements], ignore_index = True)

# convert comma separated string to list
inducements['inducements'] = inducements['inducements'].str.split(',')

# make each element of the list a separate row
inducements = inducements.explode('inducements')

# strip leading and trailing whitespaces
inducements['inducements'] = inducements['inducements'].str.strip()

# create "star player" label
inducements['star_player'] = 0
inducements.loc[inducements['inducements'].str.contains("Star player"), 'star_player'] = 1

# create "card" label
inducements['special_card'] = 0
inducements.loc[inducements['inducements'].str.contains("Card"), 'special_card'] = 1

df_matches = pd.merge(df_matches, df_inducements, on='match_id', how='left')

df_sp = (inducements
            .groupby("match_id")
            .agg(has_sp = ("star_player", "max"))
            .reset_index()
)


df_matches = pd.merge(df_matches, df_sp, on = "match_id", how = "left")

df_matches['match_id'] = pd.to_numeric(df_matches.match_id) 

Step 3: Create matches by team DataFrame

When analyzing the data, we also like to have a dataframe df_mbt (df_matches_by_team) that contains, for each match, a separate row for each team participating in that match. This structure is nicely visualized at the Nufflytics blog. Such a dataset is suitable for analysis by team, e.g. win rates. We can extend this further by adding, at the match level, data that is specific for each team - coach pair, such as team value, coach rating etc. For example, we can imagine adding more team level data, such as casualties caused during the match, or team composition at the start of the match etc.

# make two copies, one for each team in the match
team1_data = df_matches[['match_id', 'match_date', 'week_number',   'year', 'week_year', 'week_date', 'team1_id',
    'team1_coach_id', 'team1_race_name', 'team1_value', 'team1_win', 'tv_diff', 'tv_match',
    'tv_bin', 'mirror_match', 'coach1_CR', 'CR_diff', 'has_sp']].copy()

team2_data = df_matches[['match_id', 'match_date', 'week_number',   'year', 'week_year', 'week_date', 'team2_id', 
    'team2_coach_id', 'team2_race_name', 'team2_value', 'team2_win', 'tv_diff', 'tv_match', 
    'tv_bin','mirror_match', 'coach2_CR', 'CR_diff', 'has_sp']].copy()

team1_data.columns = team2_data.columns = ['match_id', 'match_date', 'week_number', 'year', 'week_year', 'week_date', 'team_id', 
    'coach_id', 'race_name', 'team_value', 'wins', 'tv_diff', 'tv_match', 
    'tv_bin', 'mirror_match', 'coach_CR', 'CR_diff', 'has_sp']

# combine both dataframes
df_mbt = pd.concat([team1_data, team2_data])

df_mbt.loc[df_mbt['wins'] == 0, 'wins'] = 0.5
df_mbt.loc[df_mbt['wins'] == -1, 'wins'] = 0

# convert to float
df_mbt['wins'] = df_mbt['wins'].astype(float)

df_mbt.query("coach_id == 255851").sort_values('match_date')

Step 4: Fetch data on team division and ruleset

Let grab for all teams in df_mbt the team division and ruleset.

A limitation of the FUMBBL API is that it shows only the latest version of the teams and leagues data. This hides the fact that leagues have changed their rules since they were first created. For example, the NAF used BB2016 rules up until summer of 2021, and thereafter switched to the new BB2020 ruleset for their latest online tournament. So we have to use our “domain knowledge” here to interpret the data properly.

# make list of all teams that need to be fetched
team_ids = list(df_mbt['team_id'].dropna())

# get unique values by converting to a Python set and back to list
team_ids = list(set(team_ids))

len(team_ids)

46831

So we have to fetch data for 47K different teams.

We use the same approach as above, looping over all team_id ’s and making a separate API call for each team.

IMPORTANT: here too, we limit ourselves to a maximum of 3 API calls per second to avoid overloading the FUMBBL server

df_teams = pd.DataFrame(columns=['team_id', 'division_id', 'division_name',  'league' ,
    'ruleset', 'roster_id', 'race_name',  'games_played'])

target = 'data/df_teams_' + time.strftime("%Y%m%d_%H%M%S") + '.h5'
print(target)

fullrun = 0

if fullrun:
    print('fetching team data for ', len(team_ids), ' teams')
    for t in range(len(team_ids)):    
        api_string = "https://fumbbl.com/api/team/get/" + str(int(team_ids[t]))
        wait_time = (random.uniform(0.5, 1) + 0.25)/3
        time.sleep(wait_time)
        team = requests.get(api_string)
        team = team.json()
        # grab fields
        team_id = team['id']
        division_id = team['divisionId']
        division_name = team['division']
        ruleset = team['ruleset']
        league = team['league']
        roster_id = team['roster']['id']
        race_name = team['roster']['name']
        games_played = team['record']['games']
        # add to dataframe
        df_teams.loc[t] = [team_id, division_id, division_name, league, ruleset, roster_id, race_name, games_played]
        if t % 100 == 0: 
            # write tmp data as hdf5 file
            print(t, end='')
            print(".", end='')
            df_teams.to_hdf(target, key='df_teams', mode='w')
    
    df_teams.to_hdf(target, key='df_teams', mode='w')
else:
    # read from hdf5 file
    df_teams = pd.read_hdf('data/df_teams_20211215_211746.h5')


df_teams['roster_name'] = df_teams['roster_id'].astype(str) + '_' + df_teams['race_name']

df_teams.shape

    

data/df_teams_20211230_192715.h5

(46831, 9)

# add ruleset_version and division_name from xlsx
ruleset_division_names = pd.read_excel('data/ruleset_division_names.xlsx',  engine='openpyxl')

df_teams = pd.merge(df_teams, ruleset_division_names, on= ['league', 'ruleset', 'division_id'], how='left')

df_teams['division_name'] = df_teams['new_division_name']

df_teams = df_teams.drop('new_division_name', 1)

df_teams['division_id'] = pd.to_numeric(df_teams.division_id) 
df_teams['roster_id'] = pd.to_numeric(df_teams.roster_id) 
df_teams['team_id'] = pd.to_numeric(df_teams.team_id) 
df_teams['games_played'] = pd.to_numeric(df_teams.games_played) 

df_teams['league'] = pd.to_numeric(df_teams.league) 
df_teams['ruleset'] = pd.to_numeric(df_teams.ruleset) 

(df_teams
    .groupby(['ruleset', 'league', 'division_id', 'division_name',  'ruleset_version'], dropna=False)
    .agg( n_teams = ('ruleset', 'count')
    )
    .sort_values('n_teams', ascending = False)
    .query('n_teams > 150')['n_teams']
    .reset_index()
)

df_mbt = pd.merge(df_mbt, df_teams.drop('race_name', 1), on='team_id', how='left')

df_matches = pd.merge(df_matches, df_teams.drop(['race_name', 'roster_id', 'roster_name', 'games_played'], 1), left_on='team1_id', right_on = 'team_id', how='left')

df_matches['team1_id'] = pd.to_numeric(df_matches.team1_id) 
df_matches = df_matches.drop('team_id', 1)

Step 5: adding team tiers

According to this article from the NAF from 2017, already since 2010 efforts were made to balance things out a bit between the different team strengths. For example, the weaker teams get more gold to spend on players, or get more so-called “Star player points” to spend on skilling players up. According to the NAF, traditionally team tiering consists of three groups, with Tier 1 being the strongest teams, and tier 3 the weakest teams. The GW BB2020 rule book also contains three tier groups, that are similar to the NAF tiers: except for Humans and Old World Alliance. And in november 2021, Games Workshop published an update of the three tier groups, now with High Elves moving from tier 2 to tier 1, and Old World Alliance moving back to tier 2.

This is most naturally added to the df_mbt dataframe, as it can differ for each team in a match.

race_tiers = pd.read_excel('data/race_tiers_mapping.xlsx',  engine='openpyxl')
race_tiers = race_tiers[ ['race_name', 'bb2020_tier', 'naf_tier', 'bb2020_nov21_tier']]
race_tiers = race_tiers.dropna()

# add bb2020 tiers
df_mbt = pd.merge(df_mbt, race_tiers, on='race_name', how='left')

Save all prepped datasets as HDF5 files

target = 'data/df_inducements_final.h5'
df_inducements.to_hdf(target, key='df_inducements', mode='w', format = 't', complevel = 9)

target = 'data/inducements_final.h5'
inducements.to_hdf(target, key='inducements', mode='w', format = 't',  complevel = 9)

target = 'data/df_matches_final.h5'

df_matches.to_hdf(target, key='df_matches', mode='w', format = 't',  complevel = 9)

target = 'data/df_mbt_final.h5'

df_mbt.to_hdf(target, key='df_mbt', mode='w', format = 't',  complevel = 9)

Choosing a license for the public dataset

An important part of making data publicly available is being explicit about what is allowed if people want to use the dataset. However, before we do so, we have to check if we are actually allowed to publish the data. This is explained nicely in a blogpost by Elizabeth Wickes.

Since our data will come from the FUMBBL.com website, we check the Privacy policy where all users, including myself have agreed on when signing up. It contains this part which is specific to the unauthenticated API, which we use to fetch the data, as well as additional public match data, such as which inducements are used in a match, and the Coach rankings of the playing coaches that were current when the match was played.

Content you provide through the website
All the information you provide through the website is processed by FUMBBL. This includes things such as forum posts, private message posts, blog entries, team and player names and biographies and news comments. Data provided this way is visible by other people on the website and in most cases public even to individuals without accounts (not including private messages), and as such are considered of public interest. If direct personal information is posted in public view, you can contact moderators to resolve this. Match records are also considered content in this context, and is also considered of public interest. This data is collected as the primary purpose of the website and it is of course entirely up to you how much of this is provided to FUMBBL. 

Third party sharing
Some of the public data is available through a public (*i.e. unauthenticated*) API, which shares some of the information provided by FUMBBL users in a way suitable for third-party websites and services to process.

The data available through the unauthenticated API is considered non-personal as it only reflects information that is public by its nature on the website. The authenticated API will only show information connected to the authenticated account.

I conclude that since the match data is already considered public content, there is no harm in collecting this public data in a structured dataset and placing this data in a public repository. The final step is then to decide what others are allowed to do with this data. In practice, this means choosing a license under which to release the dataset. I decided to choose a CC0 license: this places the data in the public domain, and people can use the dataset as they wish. Citing or mentioning the source of the data would still be appreciated of course.