You have just learned a valuable lesson:
"Just because you can do something doesn't mean you should".
This is roleplaying:
"I am an Orc and a Warrior therefore I have no interest in the College of Winterhold".
So don't.
You can look forwards to "re-play value" on subsequent playthroughs that allow you to explore more character types.
To be fair thanks to the media we are pre-disposed to an "I want it all and I want it now" mentality but try to resist!

Az
This.
I'm currently running two characters, and I RP very heavily with every character I start.
My first is a Dunmer Vampire Assassin. I've just completed the DB quests, and am in the process of eliminating my 'predatory competition' from the game (i.e., I'm a Vampire eliminating other Vampires). This character has zero interest in the College of Winterhold, the Companions, and only partial interest in the Thieves Guild just to use their network of Fences.
This character is all about the Dark Brotherhood, and has the secret goal of turning the city of Markarth into his own personal Necropolis.
My second character is a Dunmer Vampire Mage who is an Assassin, but this character is only interested in the Dark Brotherhood and the College of Winterhold. He ignores all other Factions, and explores dungeons and crypts only as a means of funding.
Bethesda keeps including all of the different variations of content to give you as much to do as possible. this way you
actually have the choice to live another life, in another world, exactly as you want to live it.
You don't
have to do anything if you don't want to. That's one of the beauties of this game series.