Dad hanging out with my ex

My dad totally despised my sons bio dad when him and I were dating. Things were okay at first and then he became really abusive. He refused to work and help support my son and eventually decided he wasn't ready for a serious relationship so he called the cops and told them to put me and my son out. I freely left b4 they even got there and slowly but surely moved on with my life. That was 3 years ago. I'm now 24 married and happy, and after his recent release from prison after serving a yr, my ex has made his presence known by going to my fathers housewith his very flirtatious mother whose always admitted her liking for my dad (he has money and she is a gold digger). My ex lives extremely close (2 houses away) to my mom and about 10 minutes from me, but never attempts to approach me or my mom directly knowing his wrongs in the past. When he was first released I tried to set up a child support/visitation plan with him like a civil adult-he never did it. I didn't worry about it because I don't need the support and I actually plan to have his rights removed I've just been trying to give him the chance to do the right thing. But he was more worried about getting in my pants-not a chance. I spoke with my dad and he sees nothing wrong with hanging out with my ex, saying 'he's my sons bio dad BUT HE NEVER SUPPORTED HIM OR GAVE A CRAP ABOUT HIM. And my son doesn't even know him. My dad always asks to let my son stay at his house and I know its because my ex comes over. I don't want him around my son but I don't want to punish my dad, but he's disrespecting me as a woman coming out of a abusive relationship and a parent.  He's even started popping up on all my social networking pages requesting me as a friend, and sending me messages. Advice.