In Part One of our series, AI for Business Development in Golf, we explored how various Large Language Models (LLMs) handled the task of providing a list of daily fee and semi-private courses in Rhode Island. We learned that some models are better than others (Claude won that one), but none really delivered satisfactory coverage.
In Part Two, we’re going to explore another common misconception about LLMs: consistency. If you ask the same model the same question multiple times, should you expect the same answer?
The Challenge
This time, the prompt we used was similar: “Give me a list of every truly private (not semi-private) golf club in Rhode Island.”
But here’s the catch. We kept giving the same question to the same model. We hit Claude with this question seven times in separate chats. And we ran it in a way where Claude couldn’t learn from its previous attempts. Seven independent queries into the same model with the same prompt.
Based on Part One, we already knew not to expect perfect coverage. Claude identified only 27 of the 37 daily fee and semi-private facilities in Rhode Island. Missing a few private clubs would not have been surprising.
What we wanted to understand was whether the model would be consistent.
The Results
What happened during this experiment ended up being an excellent teaching exercise in how LLMs work, because several surprising and undesirable behaviors emerged. For that reason, we’re going to walk through the results run by run.
Oh wait, one more important thing. The correct answer to how many private golf clubs there are in Rhode Island, excluding semi-private clubs, is 19. We spend a lot of time researching and categorizing golf facilities, so this is exactly the kind of question we care about getting right.
Run 1: 14 clubs
Claude says they are going to search the internet for the information. It analyzes aggregator sites and online directories, including the Rhode Island Golf Association. It returns 14 clubs.
Run 2: 13 clubs
Claude once again goes out to search the web. This time, it only returns 13 clubs. It found two new clubs, Lincoln Country Club and Shelter Harbor Golf Club, but missed three clubs that it found in the first run. It also incorrectly included a semi-private club that offers tee times and threw in a course in Massachusetts because it “was close to the border”.
Run 3: 15 clubs
This was the best run of them all. It found 15 of the 19 clubs. It identified Alpine Country Club, one that was missed by the previous two runs. But even then, Claude failed to identify Quidnessett Country Club, which it had correctly identified in both of the previous runs.
Run 4: 11 clubs
Here’s a disclaimer: I did not expect this to happen. But explaining what happened requires a quick overview of how these LLMs work.
One of the things tools like Claude do is they make a judgment about whether they know enough information in their training (previously collected) data to answer the question without going to the internet. So if you say “Who was the First President of the United States” – it just knows the answer. It doesn’t have to search the internet to find the answer, it just spits out George Washington.
In our first three runs, Claude said things like “I should search for current information on this, since golf club memberships and statuses can change.” It then goes and compiles information from the web.
In this fourth run, it didn’t do that. It went by memory. And it only gave 11 facilities. And that’s not even the scary part.
The scary part is that Claude returned Carnegie Abbey Club, a name that has not existed since the club rebranded as The Aquidneck Club in 2019. In other words, Claude wasn’t merely incomplete. It was relying on information that was over seven years out of date.
Runs 5 through 7: 10, 11, 13
We continued running the experiment until Claude finally reproduced a previous answer exactly. That didn’t happen until the eighth attempt.
During these final runs, we had yet another instance where Claude chose to answer “by memory”, and once again gave us the club that was renamed in 2019. Across all seven attempts, Claude produced seven distinct lists of golf clubs. And we also had our all-time low, an instance where Claude barely returned half of the private clubs in Rhode Island.
Conclusions
So what did we learn from this experiment. For my perspective, there were three main takeaways:
1. Lack of Consistency
Claude (and LLMs in general) cannot be relied upon to consistently give you the same answer. It’s an incredible tool with incredible capabilities. Claude is not designed to produce identical answers every time. Small differences in how the model reasons through a task can lead to materially different outputs, even when the prompt remains unchanged.
If you had seven sales reps all querying Claude for lists like this one, our research shows it’s likely each and every one of them will be using a different list.
2. Risk of getting outdated data
You run the risk of getting outdated data, and in some cases, severely outdated data. Not only is Claude scouring the web, which is littered with inaccurate and out-of-date information, but it sometimes decides to answer without doing any research.
3. Repeated Prompting Cannot Fill Every Gap
Repeated prompting cannot solve every data problem. While running the same query multiple times helped uncover some clubs that were missed in earlier attempts, two private clubs, Weekapaug Golf Club and Glocester Country Club, never appeared across any of the seven runs. This suggests the issue wasn’t inconsistency, but rather a lack of underlying knowledge. For business development professionals, this is an important distinction. Running the same prompt over and over may improve coverage, but it cannot reliably uncover organizations that the model has little or no awareness of in the first place.
The lesson isn’t that AI is bad. In fact, Claude remains one of the most impressive tools I’ve ever used. The lesson is that understanding its limitations is just as important as understanding its strengths. If your business development strategy depends on complete, accurate, and current market intelligence, you need to know where AI can help and where it can quietly lead you astray.
If you’re tired of relying on incomplete, outdated, or inaccurate facility data in your business development efforts, request a free trial of the most comprehensive golf facility intelligence platform in the industry.