Just about every quest giver makes you think you have to finish his quest before the world ends if you don't. My question is are there any quests with an actual time limit? Or can you just let them linger on your roster for ever and finish them on a whim?

Take your time.