As of 6/12/12 I have found the following bugs/problems/possible improvements I've found with Mounted Combat:

• Cannot talk with any characters while on horse. Theives will not confront you, and will simply run away when trying to talk to them afterwords.
  
• Character puts weapon away before getting off horse. This is simply an annoyance, since sometimes we get off the horse to fight something bigger, like a bear.
  
• Maybe there could be a quick-unmount for when you have your weapon drawn, since your character tends to get off quite slowly.

These are more for my notes for Bethesda rater than for the community, just thought I'd post them. If there are anymore anyone else finds, feel free to post them on here.