There is no doubt that rolling is associated with the culture so when you are around it; unfortunately it's something that's going to come up for a number of reasons some of which Dave listed.
There is also a new wave of people hitting the scene (at least out here in ny) who don't have any idea of what to expect but the stereotypes so that's their way of trying to come to their own conclusions about the scene.
I would say, while it is frustrating it makes no sense really to get angry. A polite no will suffice and the next time they see you, they won't have to ask and maybe will ask m