You should definitely take your son’s wishes in account, also find out why he wants to be with you more, as great as that is, is there something more to it? Especially if he is verbally abusive towards you he might be saying stuff that your son has picked up on.
I share care my boys, 10 and 4, we take 3 nights a week and take turns with a Friday night each week. This way over a fortnight it really is 50/50, in the overnight sense. I hope that helps.