gateway-gamers.net/sites/40k-rankings/statcheck-elo-scraper/README.md

# Stat Check Elo Scraper

This is a pure-Python scraper for the public Excel workbook embedded at:

`https://www.stat-check.com/elo`

It does not need Selenium, Playwright, Chromium, pandas, or openpyxl. It uses only the Python standard library.

## How It Works

The Stat Check page embeds a public OneDrive Excel workbook through a `1drv.ms` URL. Microsoft's web viewer does not expose a stable direct `.xlsx` URL in the page HTML, but it does use a public OneDrive metadata API behind the scenes.

The scraper reproduces the useful part of that flow:

0. Read the current `1drv.ms` embed link from the Stat Check page's iframe. Stat Check replaces this link occasionally, so it is looked up on every run; `EMBED_URL` in `pure_python_elo_scrape.py` is only a fallback for when the page can't be read or its link returns 404.
1. Request an anonymous Microsoft "Badger" token from:
   `https://api-badgerp.svc.ms/v1.0/token`
2. Convert the public OneDrive embed URL into Microsoft Graph's `shares/u!...` id format.
3. Call:
   `https://my.microsoftpersonalcontent.com/_api/v2.0/shares/{share_id}/driveItem?action=EmbedView`
   with `Authorization: Badger <token>`.
4. Read `@content.downloadUrl` from the metadata response.
5. Download the real `.xlsx`.
6. Convert the first worksheet to CSV by reading the XLSX ZIP/XML structure directly.

The Microsoft endpoints used here are undocumented and could change, but this currently works without browser automation.

Requests that fail with HTTP 429, a 5xx status, or a network error are retried up to three times (after 5, 15 and 45 seconds).

## Usage

```bash
python3 pure_python_elo_scrape.py
```

Default outputs:

- `elo-pure.xlsx`
- `elo-pure.csv`

You can also pass output paths:

```bash
python3 pure_python_elo_scrape.py statcheck-elo.xlsx statcheck-elo.csv
```

## Files

- `pure_python_elo_scrape.py` - gets the Badger token, resolves the workbook download URL, downloads the XLSX, writes CSV.
- `download_and_parse.py` - helper functions for downloading and converting XLSX to CSV with stdlib only.
- `sample-elo.csv` - sample output captured during testing.

## Verification From Testing

On June 28, 2026, this downloaded a `3,465,320` byte workbook and produced a `4,145,399` byte CSV with `40,945` rows. The CSV header was:

```csv
Rank,Delta,Name,W,L,D,G,WR%,Elo,Delta,Last Event,Country
```