Am I doing something wrong?

My BF is pushing me further and further away. I just moved in with him in a different state and am trying to make friends. This one girl that he used to talk to has been sweet to me and she has two kids. We went to hang out tonight and take the kids to Mcdonalds to play and he got mad. He says it like him hanging with my ex (who was abusive). I told him it wasn't because she and him were not even really together. He is nervous I will go party with her and he based this off of one time he got me drunk. It was my first time drinking and I drank too much and acted stupid. Everyone I meet he seems to find something wrong with. I have hung out with her before and he hasn't gotten mad. He got mad this time cause he heard me talking about drinking. Not me drinking but she was talking about drinks. He gets mad so easily, that I don't know if this is a valid reason. What do you think?