I was on similar situation, but on other side. My situation is more complicated as I’m single parent and non residential parent on the same time. In past I try to get into relationship, mother of my boys (they live with me) don’t have any issues with my personal life, so this was OK. But mother of my daughter (she don’t live with me) became mad, abusive etc.
I don’t want to blame anyone, but have you think why he doing like this? always blame is on both sides. sometimes doing something we are guilty, in other case when we not doing what we should, we are also guilty. It’s easy always blame to others, but we never look at yourself.