I've experienced it too, it's one of the few places that gives me consistent issues. One of the worst locations is up by the guard tower, looking in most directions seems to lower my fps to 15-25. Most areas seem to be fine but there seems to be issues with optimization when looking in certain directions.
Running on mostly ultra settings with 8x FXAA. AMD 965 @ 3.4ghz Radeon HD 5850