Two incidents involving artificial intelligence have recently captured public attention, highlighting the hidden dangers of its authoritative tone.
The first took place in Chuzhou, Anhui province, where 67-year-old farmer Uncle Wu consulted an AI assistant for weed-killing methods for his sesame field. The AI promptly delivered a detailed plan, including the herbicide name, dosage, and spraying procedure. Uncle Wu applied it across his entire field, but the formula contained a soybean-specific herbicide that is strictly forbidden for use on sesame crops. As a result, all 150 mu of sesame plants withered away, and his investment of over one hundred thousand yuan was lost entirely.
The second incident occurred in Hangzhou, Zhejiang province, where a man who had quarreled with his wife decided to go for a walk to clear his head. Without a destination, he asked AI for a suggestion. After thoroughly inquiring about his personality, height, and physical condition, the AI recommended an outdoor hiking route. The man set off without hesitation, only to find himself in an uninhabited development zone at an altitude of over 1,300 meters, surrounded by dense forests and rugged terrain, where getting lost or suffering from hypothermia at night was highly likely. He slipped and fell, becoming trapped, and it took rescue teams four to five hours to bring him to safety.
In both cases, the individuals were of different ages and faced different problems, but the common thread was that AI supplied incorrect information, leading to real-world losses. Uncle Wu, a lifelong farmer with abundant agricultural experience who had been using AI for over a year, trusted the AI because its answer was packed with technical jargon and delivered with unwavering conviction. But when he later questioned it, the AI immediately backtracked, admitting that the herbicide it recommended was indeed responsible. The platform explained that AI lacks an independent knowledge base and that its responses are generated by aggregating publicly available online information. Only upon closer inspection did Uncle Wu notice a small, faint line at the bottom of the dialogue box: "AI-generated content may contain errors. Please verify."
Many people have had similar experiences. No matter how complex the question or how obscure the field, AI almost always responds in a confident and knowledgeable manner, sometimes even citing sources that make it appear highly professional. For those lacking sufficient knowledge and judgment in a particular domain, it is all too easy to be swayed and not question the answer at first. It is often only when users detect an error and confront the AI that they realize its confidence is merely a facade—once caught, it does a complete about-face. Yet it is precisely this "confident" and "professional" posture that lends an air of authority to AI-generated responses, fostering dependence among users, especially those without specialized knowledge, and potentially causing them real harm from false or misleading information.
Where does this AI confidence come from? Many current large language models are essentially probabilistic language prediction and generation systems. They excel at imitating linguistic styles but lack the ability to verify facts. Moreover, the internet-based data used to train these models is a mixed bag of credible and unreliable sources. The more flawed information AI absorbs, the more likely it is to produce inaccurate or biased outputs. Research has shown that during the training process, human annotators tend to give higher scores to responses delivered with less hesitation. AI, picking up on this preference, subsequently optimizes itself to encode a confident declarative tone as a more probable output pathway, reducing the frequency of hedging words like "probably," "maybe," "perhaps," or "I am not sure."
As AI models continue to evolve, their applications have expanded far beyond casual conversation into real-world domains such as medical consultations, agricultural guidance, and government services, reaching vast numbers of people. This means that AI's tendency to appear confident while remaining prone to "hallucinations" could lead to consequences that are far more severe and unpredictable. Such risks demand heightened vigilance. In high-stakes areas involving personal safety, health, or significant financial interests, current AI risk warnings are simply inadequate. In Uncle Wu's case, when the AI provided a pesticide formula that could destroy an entire season's harvest, the only warning was a small, gray line in the corner of the interface: "Please verify with caution." In the Hangzhou hiker's case, the AI recommended a difficult trek without any advice on essential professional gear.
To attract users, some platforms overhype AI as being "omnipotent" in their marketing, yet when problems arise, they deflect responsibility by emphasizing AI's role as an aggregator of information. In response, users must clearly recognize AI's limitations, maintaining a sense of caution during interactions. Especially when it comes to professional decisions involving health, safety, investment, or the law, we must never rely solely on AI, nor be fooled by its contrived self-assurance. The platforms offering these services must also step up and take real responsibility, establishing more robust tiered risk-warning mechanisms that clearly communicate AI's capabilities and known flaws. If any losses occur, platforms must not use disclaimers like "for reference only" or "please verify" as a shield; they must bear full legal accountability.