Separating the two is possible, but let's face it. It's incredibly easy to form negative associations between things even with minimal interaction.
I've had it happen, though that possibility is why I generally keep most fandoms at arm's length, so to speak. The recycled opinions, toxic...