I had a problem like this one in the past and it ended up causing loads of stupid arguments so I hope it doesn’t come to that for you.
My ex-partner objected to anyone else doing the child care but was totally unreliable himself and wouldn’t see his own children even when he was ordered to let alone when it would suit me, in fact he deliberately let me down many times just to be vindictive and to control/stop me going out.
Best thing is to make your own arrangements and just don’t tell him, it’s not his business who you ask when it’s your time with the children so long as they are suitable. At the end of the day what can he do about it (other than rant at you). What people have said to you is right, it’s your responsibility not his and why should you not go out, letting him be the sole babysitter gives him control of your life and that’s a no