Being an Atlanta native I was all set to jump on this thread and provide my opinion, but it seems several people have already highlighted the amazing store that is Better Baseball. Seriously, no place better, just stop looking. Two locations, I have only been to the one in Marietta, which is EXACTLY like a candy store for adults, but with a baseball theme.
http://www.betterbaseball.com/
132 Carruth Dr SE, Marietta, GA 30060
3220 Cumberland Dr, Chamblee, GA 30341