I then put some back into archery but left the Ranger perk empty, continued playing Skyrim for a bit, came in contact with a bad guy that I felt like filling up with arrows, pulled out the bow and sure enough my character was still moving fast with the bow drawn just like he would if he still had the Ranger perk.
It's not a horrible bug I mean after all I now get a free perk but just thought I let people know about it.