Depends on the area you're in. Try going to the Ashlands and you will get bored of them soon enough. But people are still overreacting about them in my opinion.

It's not that they're 'difficult', it's that they're obnoxious to deal with, and relentless. You kill one (or 3), walk 30', and the battle music starts anew--they just grated on your nerves after a while. My policy was always to kill 'em with fire. Good way to train your distruction. Or, levitate so high they couldn't see you. That was another favorite tactic.
Interesting that we also find the other common ingredient (trama root) in the same areas where the cliff racers are most abundant. I have a theory that cliff racers eat trama root, but it may be the other way around -- those plants look pretty fierce. 