How old are your children? If they are missing you, then why would you not want to see them?
As for going back and forth between your mum’s house and your gf’s house, as an adult, you are supposed to choose one household and stay there, Why not stay permanently at your mum’s and have your children every other weekend as before?
That would allow your children to see you, and protect everyone.
I can see why your ex might not be happy. Your kids are probably asking for their daddy, and you are prioritising someone else’s child.
But if your mum doesn’t want them to stay, then you don’t have much choice.
None of this is easy. Can you take your kids out for their daily exercise so at least they get to see you regularly?