Skip to content
Annotation Sensitivity Explorer

Methods, Data and Limitations

This page summarizes the data and definitions behind the site. Full details are in the two papers: Kern et al. (2023), Annotation Sensitivity: Training Data Collection Methods Affect Model Performance (Findings of EMNLP), for the human ratings, and Reiter et al. (2026), Reliable but Sensitive: Evaluating LLM Annotation Beyond Performance (under review), for the LLM labels.

Code and source data

Data

LLM labels. Seven models (GPT-4o-mini, GPT-5.4, Mistral Large 3, Mistral Medium 3.5, Llama 3.1 8B, Llama 3.1 70B, Llama 4) labeled the 3,000 tweets under 12 prompt recipes, three runs each, at temperature 1: 755,999 usable labels per outcome. One Llama 3.1 8B response (joint, OL-first design) could not be parsed and is excluded. A temporal re-collection of GPT-4o-mini and GPT-5.4-mini are not included.

Human ratings. The exact subset used in Kern et al. (2023): 44,900 ratings from 917 annotators on the same 3,000 tweets, five questionnaire versions, three ratings per tweet per version (100 ratings are missing in versions C and D). The public Hugging Face file also contains 44,250 ratings on 50 additional tweets; those are excluded.

Tweet text is the text distributed with the Kern et al. data, with @-mentions replaced by @######. The 3,000 tweets are a stratified sample of the Davidson et al. (2017) corpus.

Words used on this site

Setup
Any one way of producing labels: a human questionnaire version, or one model under one prompt recipe. There are 90. In the data files and the CSV downloads this is the instrument column.
Prompt recipe
One of the 12 LLM task designs: how the two questions are asked (joint OL-first, joint HS-first, or separate calls) × one or six tweets per prompt × with or without a confidence score.
Questionnaire version
One of the five human instruments, A–E.
Reliability vs. sensitivity
Reliability is agreement within one setup, across its three runs or three annotators. Sensitivity is agreement between two setups. They are different questions, and a setup can score high on the first and low on the second. That contrast is the point of the comparison page.

Definitions

Eligible labels
In the separate-call designs only one construct is asked per call. The construct that was not asked is excluded, never counted as a negative label.
Label-level prevalence
Share of positive labels among all eligible labels of a setup (LLM: every run; humans: every rating). Shown on the site as “share labeled yes, every individual label”.
Item-level label
Shown on the site as “one label per tweet”. LLM: the modal label over three runs (a cell with one missing run and a 1–1 tie counts as negative, as in the paper's R code). Humans: the majority of the three ratings within a version; a 1–1 tie between two valid ratings counts as positive. The paper's human reference is the majority of all 15 ratings.
Reliability
Fleiss' κ across the three runs (LLM) or three annotators (humans) of one instrument, over tweets with three valid labels. Human cards also show Krippendorff's α.
Sensitivity
Cohen's κ and raw agreement between the item-level labels of two instruments, over tweets with a label on both sides. Raw agreement is always shown because κ depends on prevalence.
Label flips
Tweets whose item-level label differs between the two instruments, by direction.

Comparing humans and LLMs

Human version A (both constructs on one screen, HS first) is a close structural match for the LLM design “Joint, HS first · Base”. Versions B–E share features with separate LLM calls, but separate calls keep no order between constructs, and a block of 50 sequential human judgments is not the same as six tweets in one prompt. The five human versions were answered by different annotator panels, so differences between them include panel sampling variation.

Data files

Known limitations of this build