The most important is that your son is happy, that he has a clear routine and that access fits with school (obviously not at the moment). How old is he? If 10 or more, what would he prefer?
Don’t feel threatened. You have offered mediation and your ex has refused, so let him take you to court. He may have refused it because it stops him bullying you or because of the cost. Either way, the court will decide access and then he can’t control you anymore.