Firstly why did you agree if your ex’s parents don’t live with her? The law states that a child under the age of 18 can go between home unless there is reason to believe the children’s health is at risk or if there is vulnerable people in a home
Now that you have agreed the whole way through lockdown I think the safest option would be to continue to stick with the agreement until further guided by the government to reduce the spread of the virus between two homes. You just don’t know if you, your ex or the children could be carriers of the virus.
However, I don’t think there is anywhere to go with this other than seek the advice of a solicitor.
This is only my opinion and hope it helps.