Best AI for Research Revealed: Claude vs ChatGPT vs Gemini vs Perplexity

Key Concepts

• Large Language Models (LLMs) for academic research
• PDF interrogation/extraction
• Reference accuracy/hallucination
• Chat GPT (Pro version)
• Gemini (Pro version)
• Claude (Pro version)
• Perplexity (Pro version)
• Accuracy rate
• Misleading questions/prompts
• Benchmarking AI tools

PDF Interrogation

• Objective: To determine the accuracy of paid LLMs in extracting information from uploaded PDFs.
• Methodology:
  • Uploaded PDFs to each LLM.
  • Asked simple questions (e.g., "Is this in the paper?") and questions requiring expansion on concepts within the paper.
  • Included "misleading questions" – fabricated information presented as plausible to test the AI's ability to discern truth.
• Results:
  • Claude demonstrated the highest accuracy, with the lowest error rate.
  • Perplexity performed second best.
  • Gemini followed Perplexity.
  • Chat GPT performed the worst, even worse than its free version in previous tests.
• Conclusion: Claude is the best paid option for interrogating PDFs due to its high accuracy and reliability.

Citation Accuracy

• Objective: To assess the LLMs' ability to retrieve accurate academic references from the literature.
• Methodology:
  • Asked LLMs to act as experts and find references on specific topics (e.g., water-based organic photovoltaics).
  • Included misleading questions (e.g., "Explain the Stapleton theory of photovoltaics," which doesn't exist).
• Results:
  • Chat GPT was the most reliable, with over 80% accuracy in providing real references.
  • Claude performed the worst in this task.
  • Gemini and Perplexity fell between Chat GPT and Claude.
• Argument: While LLMs can provide references, specialized tools like SciSpace, Elicit, and Consensus are better suited for academic source retrieval due to their use of real databases. LLMs are prediction models and can hallucinate plausible but non-existent references.
• Conclusion: Chat GPT is the best option among the tested LLMs for reference retrieval, but users must verify the accuracy of the provided references.

Overall Performance and Take-Home Messages

• Scatter Plot Analysis: A scatter plot comparing content accuracy (PDF interrogation) and reference accuracy revealed no clear winner across all tasks.
• Key Finding: The best LLM depends on the specific academic task.
• Claude Pro: Best for accurate interrogation of PDFs, with the lowest error rate and most reliable document interpretation/summarization.
• Chat GPT Pro: Best for accurate reference retrieval, achieving 82.35% accuracy, outperforming other models.
• Gemini and Perplexity: Offer a balance, performing strongly on content but weaker on references.
• "If you want something to interrogate PDFs, Claude is the only one at the moment that really provides the best responses if you're putting in PDF. So that's worth the money, isn't it? Not to be lied to."
• "Chat GPT pro best for accurate uh references achieve the highest accurate uh reference accuracy at 82.35% outperforming everything else which Claude is down at 40%. So if you are using Claude to inter interrogate PDFs do not use it to go to get references."
• Recommendation: Choose between Claude and Chat GPT based on the specific academic task. Always verify the information provided by any LLM.

Conclusion

The video provides a comparative analysis of paid LLMs (Chat GPT, Gemini, Claude, and Perplexity) for academic research tasks. It highlights the strengths and weaknesses of each tool in PDF interrogation and reference retrieval. The key takeaway is that no single LLM is universally superior; the optimal choice depends on the specific task at hand. Claude excels at PDF interrogation, while Chat GPT is more reliable for reference retrieval. The video emphasizes the importance of verifying information from LLMs and suggests exploring specialized tools for specific research needs.