In his latest progress report for Frayed Knights, Jay Barnson discusses the implementation of artificial intelligence and how difficult it can be to keep the challenge high but the frustration factor low.
Face it, if the enemies were really smart, the players would be dead in ANY dungeon delve. All you need is for the enemies to coordinate their defense, focus on the healers first (who effectively make up the (logistics) arm of the party, if you were to use a military analog), then burn down the other spellcasters (the (artillery) in a military analog) and finish off the rest of the party at their leisure. There are some variants (maybe neutralizing the artillery-casters first, if the healers aren't potent enough to keep up), but ultimately if I were playing the AI with a real desire to win and really, really hurt the party, I'd focus fire on killing or at least neutralizing party members one by one.
Knights of the Chalice still one of my favorite indie RPGs had some pretty evil, potent AI. It could get frustrating, but it was fun too. They weren't exactly 100% optimized, but anybody who has played it knows that the game was all about protecting your wizard(s). One advantage to the AI in a class-based game is that character roles are very nicely defined, making strategy easier to implement.
In a more skill-based game, or a hybrid like Frayed Knights, it's a little trickier to define. And even so. does it really make sense to make the AI very smart and ruthless? One of the main tricks of a DM in a dice-and-paper game is to have the enemies back off a little on (finishing off) a heavily wounded PC to give the players time to avoid a TPK (Total Party Kill). Nearly wiping out the entire party is fun for all involved. Actually doing so not so much.