First, figure out what's causing the swirls. Do you have a lit of bonded contaminates that your pushing with your towels or are your towels causing them, themselves.
Cheater: Black tinted detailer of your choice, or...
Get a waterless wash and clean the worst area. Using the ziploc baggie trick, feel for contaminates but clay anyways. Then, with a mild polish, using straight-line technique, HAND polish that area and buff clean.
If that cleans up for you and takes care of the problem, spray it with detailer and dry it off with you mf towel a couple times in your typical fashion. If you catch even a single new scratch, it's your towels. If not, it's probably the bonded contaminates being relocated while detailing.
As always, YMMV...
Sent from my HTC DNA