Same tweets, different labels
Does the way you ask for a label change the label you get? We took 3,000 tweets from a widely used hate speech dataset and had them labeled for offensive language (OL) and hate speech (HS), by people and by LLMs, and changed how the questions were asked.
917 people were randomly assigned to one of five versions of the labeling task. Seven LLMs labeled the same tweets under twelve prompt recipes. Below, you can see how much these choices move the labels. Diagrams of each setup are at the bottom of the page.
A setup is one way of producing labels: a human questionnaire version, or one model run under one prompt recipe. There are 90 of them here.
Does the setup change the labels?
Each dot shows the share of tweets that one setup labeled as offensive language or hate speech. Run the same LLM with the same prompt again and the share barely moves, so the labels look consistent. Change the prompt or the model and the share can move a lot. The five human versions differ too.
Loading…
Want the same question across all 90 setups at once? See what moves the label most.
Compare two setups tweet by tweet
Each link opens a side-by-side comparison of the same 3,000 tweets.
How people labeled the tweets
Each person labeled up to 50 tweets and stayed in one version for the whole task. The definitions of offensive language and hate speech were the same in every version; only the layout and order changed. Every tweet was labeled by three people in each version.
How LLMs labeled the tweets
The LLMs got the same definitions people did. We changed three things about the prompt and tried every combination. We call one combination a prompt recipe.
If you collect or evaluate labels
Three things follow from the numbers above.
- Repeatability is not stability. An LLM that gives you the same label three runs in a row can still give a very different answer under a prompt you would have considered equally reasonable. Reporting only repeat-run agreement overstates how solid a label set is.
- The setup is part of the result. Prevalence here moves by tens of points across choices nobody would call wrong. A prevalence figure without its prompt recipe, or a questionnaire without its layout, is not reproducible.
- Which choice matters is model-specific. Some models barely notice the prompt; others move as much from a prompt change as from being swapped for a different model. You cannot assume the sensitivity you measured on one model carries to another.
The ranking of what moves the label most is where these come from; Methods & Data has the definitions and the limitations.