MIT study shows AI models behave like swayable voters during elections

A groundbreaking study from MIT and Stanford researchers tracked 11 major AI language models—including GPT-4, Claude, and Gemini—throughout the 2024 presidential campaign, revealing that these systems behaved more like swayable voters than neutral information sources. The findings expose how AI models can shift their responses based on real-world events, demographic prompts, and public narratives, raising significant concerns about their reliability and potential influence on democratic processes.

What you should know: The study conducted over 12,000 structured queries between July and November 2024, marking the first rigorous examination of how AI models behave during a live democratic event.

• Models demonstrated measurable shifts in behavior when prompted with demographic identifiers like “I am a Black Republican” or “I am a Hispanic Democrat.”
• “Models are ‘swayable’,” said Sarah H. Cen, lead author of the study and Stanford researcher, highlighting their reactive and inconsistent nature.

Key findings from Biden’s withdrawal: When President Biden dropped out on July 21 and endorsed Kamala Harris, models didn’t respond proportionately to the political shift.

• “Trump’s relative association with ‘American’, ‘competent’, ‘decisive’, ‘effective’, ‘ethical’, ‘honorable’, ‘intelligent’, ‘qualified’, ‘tenacious’, and ‘trustworthy’ rose more than for Harris,” Cen explained.
• Even traits like “honorable” and “intelligent,” which might logically shift toward the new Democratic nominee, saw sharper gains for Trump instead.

Election prediction inconsistencies: Models proved unreliable as forecasting tools, frequently returning conflicting outcomes when simulating exit polls.

• “When primed to think about different topics or issues, models return exit poll predictions that correspond to different election outcomes,” sometimes favoring Harris, sometimes Trump.
• “This suggests that models are not necessarily reliable forecasters,” according to the research team.

Memory limitations and implications: The study couldn’t test chatbots with conversational memory due to cost constraints, but researchers believe the effects would be amplified.

• “We would have loved to study [stateful] chatbots because that’s the way that the vast majority of people interface with LLMs,” Cen said.
• Models with memory could become increasingly biased by recalling past interactions, “reinforcing earlier impressions and compounding the effects of how prompts are written.”

Model refusal patterns: Different AI systems showed varying willingness to answer politically sensitive questions.

• OpenAI’s GPT-4 frequently refused to predict election outcomes, which researchers viewed as “a successful form of ‘guardrailing.'”
• Perplexity’s models and Gemini were more willing to offer direct answers, while GPT series often hedged responses.
• Refusal rates spiked around specific adjectives like “trustworthy” or “corrupt,” suggesting built-in sensitivity filters.

Internal inconsistency problems: Models demonstrated a lack of self-consistency when researchers reverse-engineered their exit poll responses.

• “These models are not self-consistent,” said Cen, “sometimes ‘predicting’ a Harris win and sometimes ‘predicting’ a Trump win depending on the exit poll question being asked.”
• This creates feedback loops where polarized training environments get reinforced rather than balanced.

Broader implications: The research challenges the narrative that AI models are neutral tools and has applications beyond U.S. politics.

• “LLMs both reflect and shape public sentiment,” Cen noted, highlighting their dual role as mirrors and influencers of public opinion.
• The team envisions future research in multi-party democracies like India, Germany, or Israel, where political dynamics are more complex.
• The complete dataset is publicly available on HuggingFace, a platform for sharing AI research, for further study into forecasting accuracy, guardrail robustness, and AI self-awareness.