Hi Sazzle, the most obvious thing here is that how you parent your little boy at your house in your time is YOUR business and what your ex thinks is not relevant.
secondly, your son is little. He needs loving and nurturing and being cuddled. Your ex seems to treat life like the army. It’s not surprising your son doesn’t want to go there. What is the new partner like?
I think I’d keep a detailed diary of everything your ds says, and if you are still concerned, consult your GP. Make sure your son knows you will believe him.