The main problem I see is that Google just shouldn't include AI results. And they definitely shouldn't put their unreliable LLM front and center on the results page. When you google something, you want accurate information, which the LLM might have, but only if that data was readily available to begin with. So the stuff it can help with is stuff the search would put first already.
For anything requiring critical thought or research, the LLM will often hallucinate or misrepresent. The danger is that people do not always apply critical thinking. Defaulting to showing an LLM response is extremely dangerous, and it's basically pointless.
The main problem I see is that Google just shouldn't include AI results. And they definitely shouldn't put their unreliable LLM front and center on the results page. When you google something, you want accurate information, which the LLM might have, but only if that data was readily available to begin with. So the stuff it can help with is stuff the search would put first already.
For anything requiring critical thought or research, the LLM will often hallucinate or misrepresent. The danger is that people do not always apply critical thinking. Defaulting to showing an LLM response is extremely dangerous, and it's basically pointless.