Grew up with his music and enjoyed it, still do. Heal the world still brings me to tears whenever I hear it. As far as music goes, he was a talented guy.
I think it's important to separate between the person and the artist. What someone does as a person doesn't change their music, but also someone's music doesn't change who they are as a person. Just because you're talented and successful, it doesn't make automatically make you a saint (nor does it automatically make you a villain, either). Those are two very separate issues. Sure, you may choose to boycott a product as a result of someone's behaviour as a person, as is your right as a person and a consumer, but ultimately whether you liked the product in the first place will be about the product, not who made it.
None of us know what actually did and didn't happen back then (apart from what was documented, like holding the baby over the balcony, climbing the tree in that documentary, and so forth) so no amount of arguing back and forth will settle that. But I think it's dangerous to make assumptions on the guilt/innocense of a person you don't know personally and a trial you didn't witness. If you say he's guilty and he wasn't, then you're persecuting an innocent person - if you say he's innocent and he wasn't, then you're disbelieving a victim. It's grim either way you look at it.
Most importantly, just because someone thinks something else of something you love, doesn't mean they are attacking you. I agree that giving you grief about it is wrong, but giving people grief about not agreeing with you is just as bad.