I finished talking to karliah after mercer is revealed to be the bad guy (saw this coming a mile away). She says i should go to winterhold to decipher gals's diary to prove mercer is a [censored] bag, but am i allowed to go back to the thieves guild during this quest or will i be branded a traitor or something?

no you can go back in after all the thieves guild quests are done.