Recently I found myself having to disprove something that Google’s AI made up.
A reader of my Cord Cutter Weekly newsletter wanted to know why they couldn’t use the Apple TV app on their Roku after signing up for the service through the Roku Channel. This is in fact a notable downside with Roku’s subscription marketplace—you have to use Roku’s own apps and menu system to access the content—but Google’s AI overview was claiming otherwise.
Turns out Google was basing its answer on a discussion thread from a random coding website I’d never heard of, and when I replicated the query myself, the source link didn’t even work. Google also cited a separate CNet article that didn’t back up the reader’s claim at all.
Stories like this are why I never think of tools like ChatGPT or Gemini as trusted advisors, even if that’s how AI companies want you to see them. In my mind, they’re more like unreliable witnesses or interrogation suspects. They’re capable of provide useful information, but you might need to grill them to get it.
This story first appeared in Advisorator, Jared’s weekly tech advice newsletter. Sign up to get free tech tips every Tuesday.
Ask for sources (and check them)
Every major AI chatbot has a feature where you can define how it should respond to you. In ChatGPT, for instance, you’ll find this in Settings > Personalization under the Customization heading.
My instructions always include a line like this: “When making factual claims, cite your sources with links.” If an AI chatbot nonetheless fails to cite sources, I treat this as a red flag and immediately ask for proof.
But as my Roku example shows, the mere existence of citations doesn’t always amount to accuracy. You also need to make sure the link actually works, and that it supports the core claim that AI is making.
Doesn’t this defeat the purpose of using AI in the first place? Not necessarily. When searching the web for answers, I’ve found that AI is most useful for defining and refining what I’m looking for with natural language. Even if I still have to check the links myself, it can surface results that a traditional web search might miss.
Be persistent
Sometimes AI can veer off course even when it’s working with verifiable source materials.
Last fall, for instance, I asked ChatGPT to help explain an aspect of the FCC’s proposal to adopt a new standard for over-the-air TV. I provided a link to a PDF file on the FCC’s website and asked the AI to summarize what the document said about the the proposed rules’ impact on antenna coverage.
In response, ChatGPT fabricated an entire set of claims about how the new standard would reduce antenna coverage in a way the FCC still found acceptable. When I asked ChatGPT to quote directly from the document, it invented two paragraphs out of thin air. When I pointed out that those quotes didn’t exist, ChatGPT claimed to be just paraphrasing. Only when I finally uploaded a copy of the PDF did ChatGPT acknowledge that its core claims were entirely made up.
While I was annoyed that ChatGPT had wasted my time, it reinforced the idea that you need to treat these conversations more like interrogations. Otherwise you’ll never know that you’re being lied to.
Google’s expanding AI answers
I’m burying the lede here, but the reason I’m thinking about all this right now is because of Google Search’s increasingly conversational AI features.
Last year, Google launched an AI Mode that gives you answers in a chat window, much like with ChatGPT or Google’s own Gemini assistant. Until last week, this mode was entirely separate from the AI Overviews that Google sticks atop its regular search results.
Now, Google is adding a text box to the bottom of those responses. This allows you to ask follow-up questions, in turn funneling you into AI Mode to continue the conversation.
While I have misgivings about anything that moves links to other websites further down the page, the new text box is at least another opportunity to push back. If Google links to a source that turns out to be a dead end, presents a random Reddit comment as an authoritative citation, or otherwise gives you unverifiable information, you can call the AI out and demand a better response.
AI answers are appealing because they remove the friction from looking up information online. Instead of having to click through to websites full of ads, pop-ups, and filler text, you can just get a direct answer and walk away satisfied.
But if you’ve ever impulse-bought an unnecessary Amazon item, got caught in an endless Instagram scroll, or one-tap reordered something unhealthy on DoorDash, you know that technology’s seamlessness isn’t always in your best interest. Sometimes you need to add a little friction back.
This story first appeared in Advisorator, Jared’s weekly tech advice newsletter. Sign up to get free tech tips every Tuesday.