Maybe I'm not understanding the Fame concept correctly, or perhaps this is a glitch like I think it is...but if someone knows why this is happening it would be very helpful..

I've done a ton of quests for the NCR. Mostly wearing plain armor (Non Faction Related). I finally got some good armor and its the NCR Bandoleer Armor (its the 2nd highest in my inventory) so I have changed into that. Now when I take that specific armor off, my NCR fame goes from Idolized (from all the quests I've done from them) right back down to Neutral. If I put another piece of NCR armor on (for example, the NCR Ranger Patrol Armor) my fame is still at Neutral. Same goes for if I put non NCR armor on again. Is this a glitch or is my fame directly tied to this one piece of armor I have in my inventory?

it is most likely a glitch, that or your fame with them was never above neutral, and you were simply disguised as that faction due to the armor.