Refering to UESPwiki, there is a hidden entrance to the College of Winterhold via the Midden(: UESP wiki says: "It can be accessed[...] through an easy-to-miss cave in the northern face of the rocky outcropping the College rests upon, the only way to access it without joining the College").

So, where is this cave exactly? Any clues? Screenshots?

I really dont want to join the College of Winterhold, not for buying spellbooks or what ever reason (it is a rp-thing...).

Thanx for your help! Edit: If you persuade Faralda or show to her that you are the Dragoborn, can you go in WITHOUT actually joining it? I guess it shows up somewhere in the menu if you belong to the guild or not.

That back entrance isn't an entrance, it's an exit. You can force yourself up it if you're pacient enough, but it's clearly not intended to be used to access the College.

And if you do, everyone acts as if you had joined the college Except Faralda that still stands at the bridge. You can even use the room