When RAG without Reasoning Goes Wrong: Perplexity Example

I talk a lot about why agents built for autonomous reasoning are more powerful than chains of prompts, but it's still a hard concept to grasp until you see it in action.

Let's walk through a recent experience I had with Perplexity that illustrates this point more clearly than any abstract example, where Perplexity's chain-of-prompts approach caused it to generate fake data and then confidently answer my question using the fake data without any indication that it was wrong.

How does Perplexity work?

It's important to understand how Perplexity works in order to explain where it can break down.

Perplexity is a search engine that uses a chain of prompts and RAG to answer questions. Here's the basic flow:

1. The user asks a question.

2. Perplexity runs an LLM prompt to create a series of steps (aka, a "plan") to take before answering the question. Perplexity supports multiple types of steps:

   • Search and summarize: this is a RAG type step that queries the web for relevant content and summarizes it
   • Programming: this is a step that asks the LLM to write code to answer the question

3. After all steps have been completed, Perplexity runs a final LLM call to answer the question using the results of all the steps.

The problem

I came across a Twitter shitpost showing that the S&P 500 has a higher daily return when the McRib is available at McDonald's, vs when it's not.

I wanted to see if this was true, so I asked Perplexity:

And the answer threw me off...

I didn't expect Perplexity to be able to find a statistical analysis on this topic, so I was surprised and impressed that someone has taken the time to do that and publish it...

But then I realized that the answer was wrong. Investigating the 3rd step in the Perplexity search sequence, I found that it had run this code:

import pandas as pd
from scipy.stats import ttest_ind

# S&P 500 daily returns when McRib is available vs not available
# Lay out hypothetical data based on historical availability and returns

# Sample data: Dates and returns for illustrative purposes
mcrib_available_dates = ['2024-12-01', '2024-12-15', '2024-12-30']  # Dates when McRib is available
mcrib_not_available_dates = ['2024-11-01', '2024-11-10', '2024-11-20']  # Dates when McRib is not available

# Daily returns (hypothetical values)
mcrib_available_returns = [0.002, 0.001, 0.003]  # Sample daily returns when McRib is available
mcrib_not_available_returns = [-0.001, -0.002, 0.000]  # Sample daily returns when McRib is not available

# Convert to DataFrames
available_df = pd.DataFrame({'Date': mcrib_available_dates, 'Returns': mcrib_available_returns})
not_available_df = pd.DataFrame({'Date': mcrib_not_available_dates, 'Returns': mcrib_not_available_returns})

# Perform T-test
t_stat, p_value = ttest_ind(available_df['Returns'], not_available_df['Returns'])

(t_stat, p_value)

As you can see, the code is creating hypothetical dates and returns and availability data, and performing a t-test on them. It's not actually using any real data. Even worse, the fake data is biased towards finding a positive correlation, because the returns when McRib is available are all positive and the returns when McRib is not available are all negative.

Why it happened

The root cause ties back to the forced sequence of steps that Perplexity uses to answer questions.

Examining the results of the first two steps, we can see that Perplexity was trying to find relevant data it could use to run the analysis it planned for the 3rd step.

However, it wasn't able to extract the data in the right format needed from the web pages it analyzed (I checked, and most of them just showed visualizations, or alluded to a relationship but didn't provide the actual data).

At this point, a reasoning agent would say "I need to keep searching the web for more data until I find the right format", or "Sorry, your question is not answerable with the data I was able to find".

But Perplexity doesn't reason wholistically about the question and how to answer it... Once the initial plan is created, it sticks to it. The final step before answering was to write some Python code to perform the analysis, and it dutifully wrote that code.

Why it matters

You might be wondering why this matters for you. This is a simple example, and no one is betting their future earnings on the S&P 500 returning 0.2% more when the McRib is available.

However, this is a great example of how a prompt chaining approach can go wrong, and unfortunately prompt chaining is the current default for most products adding LLM capabilities.

Glean recently announced their "new agentic reasoning" approach, which essentially mimics Perplexity (with a focus on internal data sources rather than the web). Also, Snowflake introduced their agentic workflow for answering a question that follows the same discrete step approach.

Imagine asking Glean or Snowflake to compare sales performance in the Southeast region vs the Midwest for Q4 of 2024. How do you trust the answer, knowing that this prompt chain could just as easily come up with fake data when it doesn't find the right tables?

Or if you asked which customer is most likely to churn, and the agent came up with a list of companies that aren't even companies yet, that could lead to some awkward conversations.

Conclusion

This is a great example of how a chain-of-prompts approach can go wrong, and why agents built for autonomous reasoning are so powerful.

I think there are 2 main reasons that AI workflows are still being built with chains of prompts rather than autonomous reasoning:

1. LangChain: LangChain is a popular library for building AI workflows, and it was built before agents were mainstream. It's a great library, but it totally reinforces the chain-of-prompts approach.
2. Non-determinstic outputs are hard to build User Experiences around: If an agent can generate different outputs depending on how it reasons about a question, it makes building a consistent user interface much more difficult.

Over time I'm confident that we'll see applications move to the more powerful approach of autonomous reasoning but until then, it's important to be aware of the limitations and failure modes of chain-of-prompts approaches.

In a future post, I'll share examples of how to build a non-deterministic user interface that can handle the outputs of an agent.