I seem to lag alot when im inside a house, cave, or somewhere not outside. I have figured out that my fps is dropped dramatically when im looking at candles or any light source. (Magic doesn't lag me)
If I am outside on the world map or in a city and see some candles or torches on a wall I don't lag at all. Its only when im inside places. It only lags when im looking at the candles. If i turn and look away the lag is gone.

Is there any way to fix this with a mod or the construction set? Its kind of annoying, because I expect the game to lag less when im inside a cave or indoors, BUT I lag more!

This has been discussed to death, it's likely a shadowing glitch. Try setting your shadows to Medium if not already, they're rendered on the CPU so let us know your system specs. Also consider an FPS cap (I cap mine to 50).