Is it wrong of me to not want my son to be a part of his fathers family?

my ex was abusive & his family enabled his behavior. they saw it right in front of there own face & tried to lie & get me arrested instead & said it was my fault because i make him mad. his parents sell weed, and would sell and smoke weed with him even though they knew i didnt want that in our lives. they went for months at a time without ever calling or seeing me or my son or finding out how we are. his father abused me in front of him, never wanted to change his diaper or feed him, and was overall a pretty bad father unless he was in a great mood. i really dont want my son to be anything like them, i dont want him around any of them, & to be honest, i will tell him when he gets older how bad they were to me & him & hope that he doesnt want to be a part of there family because they dont deserve him. am i wrong for that?
btw my son is 2 right now.