I have the same problem even with a court order in place. My ex is also abusive and the court have told him that he is not allowed to phone or text me unless there is an emergency with the kids and he is not allowed to my house we do drop offs and pick ups in a supermarket carpark.
Our current agreement is he has them two set nights in the week and every other weekend and we alternate christmas so we either have them christmas eve and christmas day until 4pm or 4pm christmas day and boxing day regardless of who’s days it falls on. We are supposed to communicate by email to arrange school holidays but he often doesn’t reply to me. I am considering going back to court to get the school holidays set in stone to stop this, he has also moved and is refusing to tell me his new address.
My ex is also controlling, it took me 4 years to get a divorce and 5 years to get our house sold (i moved out with the kids and he remained in it) and the kids are the only way he can interfere and try to control me now.