I don't find the challenge of enemy's fun but the adventure of doing it more fun
You actually have the correct attitude for this TES games. It is not about the combat, but immersing yourself in the world and exploring. Of course there is combat in the world, and you usually have to deal with it at some level. Some people look forward to the combat, some people think it is an obstacle to exploration. However, the character progression is such, that with certain choices, you can make an uber character that is not challenged at all by combat, which leaves you free to explore the world without worrying about combat.
Of course, if you want to be more involved with combat, you build a different kind of character for that.