Political provocation classifiers and the Toxicity Index (TI7)
Eight SetFit classifiers for tactics of political provocation in short social media posts, released with O'Neill, Cabanillas, Kepner, Goggin, and Brooks (2026), The Toxicity Index: measuring political provocation in the United States Congress (manuscript in preparation). Three of them, together with four general-purpose classifiers, produce the seven-component Toxicity Index (TI7) used in the paper.
Classifiers in this repository
| Folder | Construct | Positive class | Role in the paper |
|---|---|---|---|
setfit_ragebait |
Ragebait | Content framed to provoke outrage beyond its informational content | TI7 component |
setfit_tribal_signal |
Tribal signaling | Us-versus-them framing marking in-group loyalty or out-group hostility | TI7 component |
setfit_performative_outrage |
Performative outrage | Theatrical indignation staged for an audience | TI7 component |
setfit_political_advocacy |
Policy advocacy | Substantive communication of policy positions or legislative action | Reported beside TI7 |
setfit_genuine_anger |
Genuine anger | Text coded as angry rather than staged (see the caution below) | Excluded from TI7 |
setfit_epistemic_manipulation |
Epistemic manipulation | Cherry-picking, straw-manning, false equivalence | Released, not analyzed |
setfit_engagement_bait |
Engagement baiting | Polls, calls to action, rhetorical bait | Released, not analyzed |
setfit_agency_language |
Agency language | Active, agentic framing (versus passive or victim framing) | Released, not analyzed |
Each classifier turns a post into a numerical representation with a fine-tuned sentence-embedding model (sentence-transformers all-mpnet-base-v2) and applies a logistic-regression head (SetFit). Load with SetFitModel.from_pretrained("jimnoneill/xbox-compulsion-classifier", subfolder="setfit_ragebait") and use predict_proba for the positive-class probability.
Training data
Labels were produced in two steps. A language model assigned initial scores from 0 to 1 under written definitions with examples from both parties, and one author then reviewed every score and corrected it where needed.
| Pool | Posts | Source | Dates | Classifiers |
|---|---|---|---|---|
| A | 1,972 | One senator's personal account (X API archive) | 25 Jul 2022 to 22 Feb 2026 | Ragebait and epistemic manipulation (pool A only); the four below with pool B |
| B | 1,949 | 13 senators' accounts, from the Hugging Face dataset m-newhauser/senator-tweets | 20 Jan to 30 Dec 2021 | Tribal signaling, performative outrage, engagement baiting, agency language (with pool A) |
| B | 200 | Donald Trump's account, from the Trump Twitter Archive (the one noncongressional account) | 2010 to 2017 | As above |
| C | 1,000 | 562 congressional accounts in the congresstweets archive | 2017 to 2023 | Policy advocacy, genuine anger |
The 14 accounts of pool B are balanced by party: seven Democratic (counting Senator Sanders) and seven Republican.
Reviewed scores were converted to present and absent labels with construct-specific cutoffs; posts between the cutoffs were left out.
| Classifiers | Present at or above | Absent at or below |
|---|---|---|
| Ragebait, epistemic manipulation | 0.65 | 0.35 |
| Tribal signaling, performative outrage, engagement baiting, agency language, policy advocacy | 0.65 | 0.30 |
| Genuine anger (clearly angry posts were rare) | 0.40 | 0.10 |
Performance
Five-fold cross-validation against the reviewed labels. Posts were grouped by account where account identifiers existed and by exact text for pool A, so that the same account or text never contributes to both sides of a fold. Each fold retrains with the original hyperparameters and training-set size.
| Classifier | n | AUC (SD) | F1 at 0.5 (SD) | Precision | Recall |
|---|---|---|---|---|---|
| Ragebait | 1,851 | 0.899 (0.020) | 0.795 (0.017) | 0.741 | 0.860 |
| Tribal signaling | 3,336 | 0.905 (0.027) | 0.737 (0.047) | 0.659 | 0.848 |
| Performative outrage | 3,836 | 0.907 (0.023) | 0.751 (0.038) | 0.679 | 0.847 |
| Epistemic manipulation | 1,420 | 0.931 (0.022) | 0.823 (0.031) | 0.786 | 0.864 |
| Engagement baiting | 4,092 | 0.883 (0.036) | 0.648 (0.056) | 0.550 | 0.791 |
| Agency language | 3,151 | 0.883 (0.034) | 0.748 (0.038) | 0.702 | 0.807 |
| Policy advocacy | 995 | 0.931 (0.013) | 0.905 (0.021) | 0.924 | 0.888 |
| Genuine anger | 901 | 0.970 (0.021) | 0.766 (0.082) | 0.684 | 0.887 |
Cross-validation measures agreement with the training-label scheme. It is not independent validation and does not show that the probabilities are calibrated.
Human-coding pilot: 150 congressional posts were drawn from four strata chosen to probe the anger question (60 that the general-purpose emotion model labeled anger with high confidence, 30 labeled joy, 30 with high advocacy and low hostility scores, and 30 random posts, balanced by party). The first coder affirmed a sheet prefilled with model scores; the second coder independently completed a blank sheet. Krippendorff's alpha was about 0.83 (genuine anger), 0.80 (advocacy) and 0.81 (performative outrage). Against the second coder's labels, the performative-outrage classifier reached AUC 0.93 and the policy-advocacy classifier 0.75. The pilot is a diagnostic check of three constructs, not a population performance estimate.
Caution on the genuine-anger classifier
Against the second coder, the genuine-anger classifier ranked human-coded anger well (AUC 0.87), but only ten of the 150 pilot posts were anger-positive. In an every-third-day sample of the congresstweets archive it flagged 29 percent of posts, and its score correlated strongly with performative outrage (Spearman rho 0.80). It behaves as a hostility and partisan-attack detector and is excluded from TI7. Use it only with that interpretation.
A related finding concerns a general-purpose model: the anger score of the TweetNLP emotion model (cardiffnlp/twitter-roberta-base-emotion) ranked the second coder's anger labels inversely (AUC 0.13), and 63 of the 96 pilot posts it labeled anger were coded as policy advocacy. We do not recommend it for measuring anger in political communication.
Toxicity Index (TI7)
TI7 for a post is the mean of seven positive-class probabilities: offensive language, toxicity, negative sentiment and irony (general-purpose), and ragebait, tribal signaling and performative outrage (this repository). It ranges from 0 to 1. A member's or account's score is the mean over non-repost posts. Policy advocacy is reported as a separate score.
| General-purpose classifier | Model |
|---|---|
| Sentiment | cardiffnlp/twitter-roberta-base-sentiment-latest |
| Emotion | cardiffnlp/twitter-roberta-base-emotion |
| Offensive | cardiffnlp/twitter-roberta-base-offensive |
| Irony | cardiffnlp/twitter-roberta-base-irony |
| Hate | cardiffnlp/twitter-roberta-base-hate-multiclass-latest |
| Toxicity | s-nlp/roberta_toxicity_classifier |
Code and archives
- Analysis code: https://github.com/jimnoneill/xbox-toxicity-index
- Training labels (with post text) and annotation: https://huggingface.co/datasets/jimnoneill/xbox-political-compulsion-dataset
- Software archive: https://doi.org/10.5281/zenodo.24095873 (version 2026.11.09)
- Derived results and annotation metadata, including the numeric training-label projections and per-classifier cutoffs: https://doi.org/10.5281/zenodo.24095876 (version 2026.11.09)
The xbox/ package in this repository contains the scoring code and the analysis modules (stream hygiene, rhythm descriptors, roll-call location model, point-process models, binned count models and the forecasting experiments).
Superseded files
bayesian_model_results.json, calibrated_model_v2.json, cohort_signatures.csv, cohort_v2_results.csv, and the modules xbox/compulsion_signatures.py, xbox/disorder_baseline.py and xbox/fusion.py belong to an earlier version of this project that estimated a compulsion probability from posting rhythm. That model is not used in the paper and its earlier validation figures should not be relied on.
Intended use and limits
Research on political communication at the population level. The scores describe text; they are not designed to support inferences about the mental health or character of any individual. Training labels were reviewed by a single author, and only performative outrage among the TI7 components was checked in the human-coding pilot. TI7 is an operational composite that has not been independently validated as a scale.
Citation
O'Neill, J., Cabanillas, J., Kepner, W., Goggin, S. N., & Brooks, J. (2026). The Toxicity Index: measuring political provocation in the United States Congress. Manuscript in preparation.