My Nord is lv 57 and hasn't begun the civil war questline or been to High Hrothgar. If the bandits continue to scale to my character, then it should be a fun battle.
I'm level 44 and just went to see the Grey Beards.
I don't have a problem killing bandits though. Stealth shots with a bow for the win.