You are right to refuse ,as a 7 year old shouldnt be travelling from 9 pm until 3 am in the morning . Nothing will happen on the drive in the car as roads will be a lot quieter than during day though. Surely it would be better if he picked up your son from school when they re-open ( 8th March)and then travel as he would arrive around 9/10pm . Apart from this friday the next pick up your son would be at school and extremely tired if this arrangement was a regular thing.
I am surprised he just doesnt stay local as well as must cost a lot in petrol going back and forwards anyhow. He could book somewhere for 2 nights and i guess have his son from friday 3/4pm until sunday 3/4pm . If he waited a little longer there will be more places open as well from April 12th.
You are within your rights to refuse and say he is not thinking of his son and his needs.