Looks like you have some scratches on those rims, i would use 2000 grit sandpaper on them. It doesn't sound like they have any clear coat on them since its turning your rag black.
I used 2000-3000grit sandpaper combo on my rims (although they only have a metal/chrome lip) with some elbow grease they look much better.
Also had some other metal parts clear powder coated but that didnt really help and cleaning that is a pain, so i learned my lesson not to clear coat bare polished metal (metal sealant is ok

).
Take the rim off and experiment on the back of it for best results.
