Its not acceptable what he’s doing. If I were you I’d put.my foot down and say that it isn’t happening any more. He is welcome to come and collect the kids but he wont be coming in. He cant accuse you of stopping him seeing the kids, because you aren’t. At the moment it sounds like he’s more interested in controlling and bullying you than having a relationship with his children.
I expect he’ll throw his toys out the pram and kick off, but you know what, tough! Thats his problem not yours.