Banking level ups will actually make the game easier. You will be facing stale enemies, while your skill levels continue to rise. Leveling up as soon as you are able, will increase the level of the enemies as well as the level of the Dragons.
Granted you will not be using perks, but I bank those anyway. So leveling up but not spending perk points will give the better challenge.