I don't want to try to be friends anymore.

I think that, when someone consistently makes ill-guided rude comments about you sort of out-of-the-blue that make you feel sad, maybe you should not talk to that person anymore. No matter how nice and funny and kind-of-sad-in-the-same-way-that-you-are-sad and vulnerable they are.

Because it makes you feel sad. And what's the point in that?