OpenAI Bans Russian and Iranian Accounts for Using ChatGPT in Influence Campaigns

OpenAI investigated the impact of the different campaigns and found that those focused on placing false narratives in established media outlets appeared to have reached a wider audience than campaigns that focused on using fake social media accounts to spread their message. The Russian operations targeted countries in Latin America and appeared aimed at undermining support for Ukraine and influencing local politics, OpenAI said. The accounts appeared to be mostly using ChatGPT to write reports detailing their work to unspecified higher-ups.

OpenAI said one of these reports appeared to describe the widest-reaching influence operation it has come across so far, because it led to a public response from a political figure. In that campaign, the Russian operatives said they impersonated a regional educational authority in Peru and tricked some schools into organizing activities about Ukraine that featured Stepan Bandera. Bandera is admired by nationalists for leading a drive for Ukrainian independence before and during World War II. He is reviled by others, including in Poland, because his movement sided with the Nazis and killed Poles and Jews. Peruvian outlet Extra reported that one such event did take place at a Peruvian school.

A Polish news report about the school event prompted a far-right Polish member of the European Parliament to suggest banning the entry of people who professed support for Bandera. OpenAI wrote in its report, "The Russian fake thus both fed on, and fed into, historical tensions between Ukrainians and Poles."

Darren Linvill, a co-director of Clemson University's Watt Family Innovation Center Media Forensics Hub who studies online influence operations, said Russians are constantly engaging in these sorts of active measures. He described trying to engage in narrative laundering, creating fake stories and fake personas to either create or spread propaganda that aligns with Russia's worldview and Russia's political agenda.

The Russian internal memos created with ChatGPT also claimed that the operatives ran a fake think tank in Latin America and recruited unwitting researchers to interview experts and compose reports. OpenAI said that effort was similar to a 2020 Russian operation posing as an independent news outlet linked to the late oligarch Yevgeny Prigozhin's Internet Research Agency, which was involved in efforts to influence the 2016 U.S. election.

In other instances, OpenAI said, the Russian operatives took credit for events they were not responsible for. The Russians did use AI to generate fake material, OpenAI said, but it was a minority of the workload. Linvill said, "These Russian underlings were trying to use AI to create a shortcut and do less work in the same way that a lot of us use AI, but in this case it helped get them caught."

While the Russian operatives primarily used AI to update their superiors, Iranian operatives used AI to help them push out content in Persian and English. The campaign mainly focused on the U.S.-Iran war and, to a lesser degree, American politics. OpenAI said the operation's activity "appeared consistent with a commercial actor running a for-hire influence campaign" but that it was unable to identify who that was.

According to OpenAI, Iranian accounts made up seven fake authors to pitch and polish long articles to small media outlets around the world. They published almost 100 articles in this way. The fake authors published articles in outlets including the progressive U.S. news site Daily Kos and the U.K.-based Middle East Monitor. According to NBC News, the Middle East Monitor published at least 21 articles by four fake authors. The outlet, which did not respond to a request for comment, has since taken down the author pages.

The Daily Kos wrote in response to NPR's request for comment that its community section, where one of the fake personas posted, is separate from staff reports. Anyone can sign up to post in the community section, and posts are "not subject to review by Daily Kos staff prior to publication," according to a disclaimer on the site. The community section does have rules for posting, including prohibiting "Pretending to be someone you're not." The posts by the persona are still up.

Linvill said AI was valuable to the operatives above and beyond just being a shortcut, adding that it helped them create language that looked and felt authentic.