I too have 20'd out on some checks, so I feel your pain. I've not found it to be a bad rule/mechanic, only poor dice rolling on my part. Just remember it goes the other way too when your opponent craps out on the dice. Also, a critical, involving the unit staying in the fight, and being able to act is far from stupid. I've seen my whole strategy change when I realized the power of a unit that is unaffected by Panic, and seen the value in having individuals with Commanding Presence to rally panicked units. It's a huge part of the game.
Gettin' to use a beat up warband that hangs in there on a panic check right away is balanced by the fact that you would either lose them outright, on a 20, keep'em on a 1, or any other fail is gonna require a rally. It really gets interesting when you put your Dwarves against the Stygian or Devout. Southpaw? You gotta plan for these things even more. Cheers.