You said the main quest requires you to visit the college. It doesn't, it suggests you could look there for information about the scroll, but you don't have to if you explore more of the map. Both my warrior and my thief have never set foot into the college (the marker isn't even on the map) or spoken to any of the members but completed the main quest.
I suppose it depends on your choices along the way.
I spoke with Pathurnaax, he told me to talk to either the Greybeards or the Blades. Arngeir told me to go to the College of Winterhold to inquire about the Elder Scroll.
On the other hand, I have a friend who randomly encountered Septimus Signus' cave by exploring and picked up the quest from him without ever visiting Winterhold or Pathurnaax.
In any case, I'm not really referring to the question of whether the Main Quest requires you to talk to certain NPCs or visit the Guilds. I enjoyed the way that various quests can intersect and I think this is a good thing.
In the way that the OP mentions you can "fail" Brynolf's quest, I was commenting that it would be nice if there were a few more dialog options when speaking with NPCs related to the guilds which are more appropriate for player characters who have no intention of joining a particular guild. I imagine there will be some mods that address this.