Yes. You absolutely can break the game by exploring. I entered Alftand while exploring. Tried to go through the door to the Animonculory and fell through the worldspace and landed on the catwalk outside the dungeon.
That has nothing to do with you entering the dungeon early. However, you are not supposed to go into that dungeon until the game unlocks the door, so if you glitch yourself in, who knows what you are going to break. Then again, they locked the door for a reason.
Out of the 26 or so dungeons that are permanently tied to a quest, only about 5 or 6 of them should be avoided until the NPC tells you to go there. They even patched out most of those issues. The rest of the quest dungeons, the quest starts when you show up. The list linked above can tell you which of the few dungeons you want to avoid. The balance of the other 150 dungeons are free and clear to explore when ever you want.
There were a couple of quests that got messed up by finding the items early, but that has been addressed in patch 1.3 and 1.4. Quest items that can not be dropped is still to be addressed, but that is another issue and does not break the quests.