I think that you need to focus on security for you and your children. To me that would be buying alone. There’s nothing to say in the future he couldn’t move in permanently and rent out some of his home.
The home was lovely to be offered, but do you really want your finances entwined with a boyfriend and his sibling?! That has potential to be a worse nightmare than an ex husband!