I was reading an article about the “artificial intelligence” of antagonists in the first The Last of Us, and there is something I believe applies to 90% of what the industry calls artificial intelligence:
“[…] to make the Infected feel grounded, entertaining, and believable.
Botta, Mark. Infected AI in The Last of Us, Game AI Pro 2 – Collected Wisdom of Game AI – edited by Rabin, Steven.
The best way to achieve these goals is to make our characters not stupid before making them smart. Characters give the illusion of intelligence when they are placed in well-thought-out setups, are responsive to the player, play convincing animations and sounds, and behave in interesting ways.
Swap infected and characters for artificial intelligence or the name of an AI product and it applies as well: a lot of what is called artificial intelligence is just automation that acts seemingly less stupid.
It is not an issue for a person or company to spend money on this, even when the benefits are nebulous, uncertain or even non-existent. But when it starts to be uncritically applied in the public civil sphere, including judicial decisions that may be based on fiction, brakes must be put on. Mystification and stupidity, even if well disguised, are acceptable in the private sphere. In public policy and the justice system, it is not.