🤥 Faked Up #25

I spent 24 hours on X's "Election Integrity Community" so you didn't have to, Dublin's hoax Halloween was an SEO success, and HeyGen let me create a false video about Valencia's floods

I am several hours late because I spent way too much time reviewing tweets about alleged voter fraud in the US presidential election. Thanks for your patience!

This newsletter is a ~6 minute read and includes 69 links.

---

HEADLINES

ChatGPT’s Search hallucinates citations. Pro-Trump AI bots on X argued with themselves. The viral TikTok of Vini Jr. saying the Ballon d’Or was “blatantly stolen” was AI-generated. Before the US vote, 286 YouTube videos containing election misinformation received 47 million views and X’s Community Notes failed to deliver. Meta ran tens of thousands of election-themed scam ads. A fake academic journal is trying to capitalize on AI hype. 1 in 20 new Wikipedia pages may be partially AI-generated.

---

TOP STORIES

X’S FLIMSY FACT FINDERS

X’s Election Integrity Community was set up by Elon Musk’s political action committee to crowdsource evidence of voting irregularities.

The 65,000-strong group failed to do so. Instead, it served as a dumping ground for superficial speculation.

Over the weekend, anonymous vigilantes posted a grainy video of a voter dropping off their ballot in Lincoln, Nebraska, baselessly labeling him was a “ballot mule.” The video racked up more than 1.5 million views before getting called out as a “malicious conspiracy theorist” by the local election official who confirmed the “mule” was a first-time voter. A fact-checking quote tweet by the local Republican Party reached a fraction of those who saw the original post, and about as many as those of another post spreading this lie posted to the Election Integrity Community.

To get a sense of what other rumors were circulating in the community, I grabbed every tweet posted in four 10-minute intervals throughout Election Day and once again at 8am this morning.

My resulting sample was 491 tweets.¹ 490 of them were posted by a pro-Trump account.

Overall, I saw no obviously AI-generated content in the mix and little in the form of a coordinated effort to use the group to seed new conspiracy theories.

I also came across only one tweet with a Community Note appended, despite X’s promise to help them appear at “lightning speed.”

Instead of generating net new content, the Election Integrity Community primarily served as a place for armchair conspiracy theorists to get their fix of fraud narratives. Most tweets were third-person accounts, typically quote tweets of more popular posts (often quote tweets themselves) accompanied by suggestive emojis or short statements about the vote bring rigged.

A plurality of claims focused on temporary issues with tabulating machines and voting locations in swing states (most notably in Cambria, Pennsylvania; Mohave, Nevada; and Northville, Michigan — as well as the amply reported recount in Milwaukee). Some posters reprised attacks on Dominion, which won a $787 million settlement from Fox News in 2020 around false claims about its voting machines.

Others went further afield, amplifying false claims that California’s electric utility PG&E targeted conservative counties with planned power shutoffs (these started after voting was over).

Harder to verify individual claims about getting turned away due to registration issues or wearing MAGA gear were sprinkled among references to content by fan favorites James O’Keefe, LibsofTikTok and fake news site RealRawNews.

With no hint of irony, one post amplified a made-up claim attributed to Maryland Democrat Jamie Raskin saying Democrats would refuse to certify a Trump victory.

As the night went on, and results started looking promising for Donald Trump, the community slowed down. Whereas there were more than 14 tweets posted each minute in the 10 am and 4 pm windows, by 9 pm the rate had slowed down to 8 tweets per minute.

Many of these seemed like half-hearted attempts to recreate denialist hits from 2020 around rigged media projections and late night shifts in leading candidates. But with the race being called relatively early, interest in uncovering voter fraud sagged.

This morning the pace was slower still, with fewer than a post a minute in the 8 am slot. Members branched out from evidence of voter fraud to congratulatory tweets, AI slop, and engagement bait.

My 24-hour journey down the X voter fraud rabbit hole reaffirmed my belief that misinformation is as much expressive as it is injunctive. Posters were not seeking to convince anyone in this fully bought-in community that the election was rigged against Trump. They were, for the most part, manifesting their fears for an outcome that didn’t align with their worldview. Once reality aligned with their preferences, rumoring became less urgent.

That said, I don’t think that this group is fully done. Besides raising questions about Senate races that didn’t align with the state’s presidential result, there seems to be a rising revisionist effort to prove the 2020 election totals were fraudulent based on the total Democratic votes in 2016 and 2024. Besides displaying at best a first grade logic, this narrative ignores the fact that millions of votes are still being counted in California.

---

This week Faked Up comes without a paywall.

If you enjoy Faked Up and are on the fence about upgrading, please consider doing so if you (or your employer!) can afford to.

The average weekly post takes ~12 hours of my time to research, write, and edit; paid subscribers are essential to continue investing that time into this endeavor.

---

FAKE BOMBS

Fake bomb threats were reported in at least 50 polling places across the United States.

This includes locations in Arizona and Pennsylvania, but the majority appear to have targeted Georgia’s Fulton County (a Democratic stronghold). The local police chief told media that they had investigated 32 of these incidents, five of which required the voting location to be temporarily closed for a sweep (voting hours were extended to compensate).

Though the FBI has said that “many” of the bomb threats “appear to originate from Russian email domains,” they have not been attributed to a state-backed operation yet.

In a separate event, a Georgia poll worker appears to have tried to frame a voter who complained about him during early voting by sending a threatening letter to the Jones County Elections Office. The bizarre letter, received on Oct. 22, included the postscript “PS. boom toy in early vote place. cigar burning. be safe.” The worker has been arrested.

DANA’S DISINFO

Over 200 people were killed in Eastern Spain last week following torrential rain caused by an isolated low-pressure area (“DANA” in Spanish). Maldita published a roundup of 61 hoaxes that followed the natural disaster, which includes false claims about the causes of the event and the relief effort. Unsurprisingly, one of the hoaxes evokes the HAARP conspiracy theory. The fact-checkers also detected a bot network seemingly based in India trying to capitalize on interest in the tragedy.

HEYGEN’S MODERATION

Deepfake video creator HeyGen has been in my feeds a bunch lately, including through this deranged video generated by AI prompter-in-chief Ethan Mollick.

HeyGen prohibits misinformation, defined somewhat handwavingly as “false claims, challenging the facts, misstating social events like elections, health info, voter suppression, harming civic integrity and election content.”

Of course, I had to check for myself. Anecdotally, the tool’s protections are pretty good. When I tried to get the AI avatars to makes false claims about voter fraud in the US election or NASA causing Hurriane Helene, my videos were rejected.

Still, I was able to create one video in which “Borjan” reads out, word-for-word, a viral fake about the Valencia rains debunked by Maldita. He’s not very realistic, tbh, but I’m assuming a paid version would get me a higher quality result.

TRICK, THEN RETREAT

Many of you asked me to write about the fake parade that drew thousands of people to Dublin’s O’Connell Street on Halloween night.

The event originated on an SEO-chasing clickbait website that posted about a parade in the Irish capital that would include “spectacular floats” and “thrilling street performances.” It appears to have reached audiences primarily by ranking well on Google and being picked up by unwitting TikTok creators.

In a frankly hilarious interview with Wired, myspirithalloween[.]com creator (and SEO marketer) Nazir Ali said he was “highly depressed” and apologized to his “Irish brothers” for the unintentional scam. Ali claimed that this specific website has been live only 3 months but that his sites have an otherwise spotless record when it comes to Irish events. “People even congratulated us” about a listing for St Paddy’s Day, he claims. “They thanked us for listing events about St Patrick's Day.”

Given the formulaic structure of the post and the blending of unrelated events, my bet is myspirithalloween.com fell victim to a hallucination from whichever LLM they used to generate the blog posts. Asked to create summaries for Halloween-related festivities in countries across the world, the LLM probably started with real events such as the Macnas parade and then just kept going.

Of course, disappointing Dublin revelers was not the primary goal here. The purpose of Ali’s operation is to drive traffic to websites covered in Google Ads. Suitably, when I visited myspirithalloween this morning, one of them was for Google’s own AI chatbot.

SEARCH AND YE SHALL FIND

Researchers at Princeton and Microsoft studied two large samples of Bing results to try and understand the extent to which the Microsoft search engine returned unreliable news sites.

(The study was published on Science Advances, a highly reputable peer-reviewed journal, but it is worth noting the conflict of interest of the Microsoft co-authors, who assert the company did not have pre-publication approval.)

Across the two samples, the researchers collected a total of almost 14 billion search result pages (SERPs) that included at least one of the 8,000 domains whose reliability has been rated by NewsGuard. The researchers argue their largest sample, dating back to June-August 2022, provides “a representative sampling of heavily searched queries.”

Overall, the study finds that unreliable sites were returned in just ~1% of the SERPs, far less frequently than reliable sites (27% to 41% depending on the sample).

More important still, the likelihood of being exposed to an unreliable site was 20 times higher for navigational queries, i.e. those that included the website’s name.

This is a significant distinction to make because it teases out the role a search engine has in discovery versus retrieval of low quality information.

Think of it as the difference between getting to infowars[.]com from the query [infowars] versus the query [sandy hook].

---

NOTED

1. David Clements: The Evangelist of Election Refusal (Lawfare)

2. In the podcast election, top shows cast doubt on integrity of 2024 vote (WaPo)

3. Inoculation and accuracy prompting increase accuracy discernment in combination but not alone (Nature Human Behavior)

4. Community Notes and potential vote brigading (Conspirador Norteño)

5. NYU Researchers Develop New Real-Time Deepfake Detection Method (NYU)

6. On Telegram, a Violent Preview of What May Unfold on Election Day and After (NYT)

7. Delegitimizing the Messenger: the Assault on Fact-Checkers (Council on Foreign Relations)

8. Blocking of X in Brazil fuels conspiracy theories in the U.S. that Harris will promote censorship (Aos Fatos)

1. If anyone wants access to the spreadsheet, reach out