ECG Arrhythmia Classification
A controlled comparison of evaluation protocols for ECG beat classification, using 100,705 annotated beats from the MIT-BIH Arrhythmia Database. Holding the model and the features constant and varying only the train/test partition moves macro F1 from 0.904 to 0.446.
I built this as practice in scraping and analysing public data, so it should be read as a skills exercise. The findings summarise what is already present in the source dataset.
Question
Published accuracy figures for arrhythmia classification frequently exceed 98%, a level that would imply the problem is effectively solved. Those figures are usually produced by partitioning beats at random, which allows beats from the same patient to appear in both the training and the test set.
I therefore trained one random forest twice on identical data and identical features, varying only the partition, so that any difference in reported performance is attributable to the evaluation protocol alone.
Results
| Partition | Accuracy | Macro F1 |
|---|---|---|
| Random, with beats shuffled and patients on both sides | 98.78% | 0.904 |
| Inter-patient, where DS1 trains and DS2 tests | 90.39% | 0.446 |
- Under a random partition the same patient appears on both sides of the split, so the classifier can identify a recording after seeing thousands of beats from it, without ever having to learn what a ventricular beat looks like in general.
- Accuracy falls only eight points between the two protocols while macro F1 halves, so the aggregate figure absorbs almost none of the loss.
- The fall is steeper class by class, since supraventricular ectopic beats drop from F1 0.918 to 0.087 and fusion beats from 0.736 to 0.002.
- Normal beats make up 89.5% of the dataset, so a majority-class predictor already reaches roughly 89% accuracy while detecting nothing of clinical interest.
Interpretation
- The score obtained under a random partition estimates within-patient recall and not the generalisation a clinical deployment would require, because the model is rewarded for identifying which recording a beat came from, a different task from identifying the rhythm.
- Aggregate accuracy is insensitive to exactly the failure that matters here, since the class distribution is severe enough that a trivial predictor scores close to the reported headline, which is why macro F1 is carried alongside accuracy throughout.
- The collapse concentrates in the supraventricular ectopic and fusion classes, so a model reported at 98% may detect almost none of the events it would be deployed to catch.
- Comparison across published work is therefore only meaningful where the partitioning protocol is stated, and the inter-patient DS1/DS2 figure is the one that should be quoted.
Figures
Method and limitations
Beats are windowed 90 samples before and 110 after each annotated R-peak, normalised per beat, and paired with two RR-interval features so that the model can observe whether a beat arrived early. Labels follow the five AAMI classes, and the four paced records (102, 104, 107 and 217) are excluded under AAMI recommended practice. The DS1/DS2 division follows de Chazal et al. (2004) so that results remain comparable across papers. A single random forest with fixed hyperparameters is used throughout, so the absolute figures set no ceiling on achievable performance, and the comparison is between evaluation protocols, saying nothing about which model performs best.