DeepSeek, the Chinese AI chatbot currently leading the App Store in downloads, has received poor accuracy ratings in a recent evaluation, raising concerns about its reliability. The chatbot scored a concerning 83% failure rate in accuracy tests and ranked 10th out of 11 in comparison to its Western counterparts.
Key Findings:
- 30% of DeepSeek’s responses contained false information.
- 53% failed to answer questions effectively, often providing irrelevant or vague responses.
- Only 17% of responses managed to correct or debunk false claims.
Despite being highly popular, DeepSeek has shown significant technical limitations. One of the chatbot’s key shortcomings is its knowledge cutoff in October 2023, leaving it unable to provide accurate information about current events. It also inserts government-friendly narratives into its answers, even when not prompted by questions related to China. For example, when asked about Syria, it echoed a Chinese government stance on non-interference in foreign affairs, which had no relevance to the query.
Misinformation Concerns
The audit revealed that DeepSeek was particularly susceptible to spreading misinformation, especially when responding to prompts intended to manipulate the AI model into sharing false claims. This vulnerability highlights the potential for bad actors to weaponize DeepSeek to spread misleading or harmful content.
Industry Implications
This assessment comes amid growing competition in the AI space, with China and the United States vying for technological leadership. DeepSeek’s Terms of Use place the responsibility for verifying the chatbot’s accuracy on the users, rather than the developers. NewsGuard criticized this as a “hands-off” approach that shifts the burden of responsibility to the end user.
With its high fail rate and susceptibility to manipulation, DeepSeek’s reliability is in question. Users are urged to cross-check information from this and similar chatbots with credible sources before fully trusting their outputs.
