🤥 Faked Up #23

A network of Catholic pages uses AI slop to grow its reach, Alexa flips the meaning of fact checks, and the Pentagon shops for deepfakes.

This newsletter is a ~7 minute read and includes 47 links.

---

I’M ALL EARS!

🙏 Thanks to everyone took the FU survey last week! I’d like to get ~100 responses, so if you have 2 minutes please smash the blue button:

I take survey responses seriously! For instance, one of you told me that FU sometimes feels a little “far from Europe.” So I tallied the regional focus of my top stories over the past three months and found that while exactly half of the items (41) were not tied to any geography, fully 20 of the rest were about the United States, versus 9 in Europe, 8 in Asia-Pacific, 4 in Latin America and only 1 in the Middle East and North Africa. While the US does produce a lot of deepfake news, especially with the election coming up, I’ll be looking to even out these distributions more moving forward.

---

HEADLINES

YouTube is implementing a “captured with camera” label that leverages the C2PA standard. Two models are not renewing their contract with Synthesia after their AI avatars were used by dictatorships. A deepfake romance scam in Hong Kong cost victims more than $46 million. More experts are worried about the Pixel 9’s AI features. Meta is testing facial recognition to fight celeb-bait scams. A sloppy Chinese influence operation painted U.S. Senator Marco Rubio as insufficiently pro-Trump. Singapore blocked 10 websites falsely posing as local news sources.

---

TOP STORIES

AI SLOP FOR PROSELYTISM

A network of 14 purportedly religious-themed Italian Facebook pages has been pumping out unrelated AI clickbait to grow its reach and promote a pair of catechists.

The pages have names like “The light of Jesus and Mary” or, more simply, “Jesus.” They all post several AI-generated pictures a day, typically of couples celebrating anniversaries, farmers showing off their produce or kids celebrating their birthday.

“If the farmer doesn’t work, we starve to death. Let’s thank the world’s farmers,” reads one post published last week featuring a woman holding a basket of peaches. The buttons on her blouse are chaotically implausible and at least one peach is hanging upside down, but users don’t appear to mind. “Thank you farmers, I wish you health and joy,” replies one. “Every time I taste a meal, I will think of your efforts” writes another.

The pages have a collective reach of over 130,000 followers and behave similarly. Over the past few months, they surreptitiously edited popular AI clickbait posts like the one above by replacing the image with a fire-and-brimstone graphic about six catastrophic biblical prophesies. The edited post invites users to connect to a Facebook Messenger or WhatsApp account “for your safety and that of your family.”

I followed those links across the 14 pages. First, they sent me to one of three smaller Facebook groups about religion. After some automated messages, those groups connected me to one of two allegedly Rome-based catechists. These two women have little online presence and have spent the past few weeks sending me occasional blessings via Messenger. Whereas I initially thought this whole thing was a pathway to a scam, I think at least one of them is legitimately trying to convert me.

That doesn’t mean the pages at the top of the funnel are legit. They each have 10+ page admins, many of which are not based in Italy. Their AI slop has no religious purpose. I suspect (but cannot prove) that they are offering their bait-and-switched callouts to smaller pages in exchange for a fee.

Either way, this is yet another episode in what the folks at 404 Media have called the zombification of Facebook. The AI slop sloshing about in these pages is made possible by algorithmic choices indifferent to post quality, a semi-automated apparatus of content-agnostic creators, and a digitally naive cluster of core users.

You can read a longer version of my findings in Italian on Facta.

ALEXA, WE GOTTA TALK

Amazon’s smart assistant Alexa¹ attributed at least five false claims to British fact-checking website Full Fact. These include allegations that the Northern Lights were evidence of geo-engineering and that Prime Minister Keir Starmer announced a boycott of Israel.

Not only did the fact-checkers not make the claims Alexa parroted — they explicitly debunked them! As Full Fact’s Head of AI Andrew Dudfield wrote on LinkedIn:

Full Fact spends considerable time and effort in writing high quality and well researched content. It is incumbent on anyone that passes on our content to others to be equally mindful of this complexity and ensure all users have access to high quality good information, set in the context required and with suitable caveats applied. We use the commonly followed ClaimReview web standard to ensure that this is as easy for other people to do as possible and I would recommend that Amazon utilise this.

The ClaimReview that Andy is so very politely referring to is a type of schema that structures a fact check’s claim and rating. Highly machine-readable stuff that Amazon could ingest!²

And this isn’t the end of Alexa’s misinformation problem. While in my tests it generally avoided answering questions about elected officials and sensitive topics like health, it confidently pushed three debunked claims about Taylor Swift’s movie in Israel, Justin Bieber’s AI-generated song about Diddy and the Kansas City Chiefs refusing to support Pride.

Even though all three have relevant fact checks annotated with ClaimReview, Alexa chose instead to point to reliable news sources like cupstograms[.]net or econsor-shops[.]de which are either offline or redirecting to scammy dating websites: