Is this fair, or am I being selfish?

That is unfair to you. He is not being a good dad at all. That is unfair you AND your children. Maybe knock some sense into him by saying "When the kids grow up they aren't going to really have fond memories of their dad." Have him imagine being a grandparent and then his kids say "Oh well my dad never really was around much and didn't care for me and my mom did all the work. Or at school being jealous of kids with cool dads who take them fishing and stuff. My friend had a huge fallout with his dad and my friends in his teens. His dad feels reLly guilty because there's nothing he can do to patch up the relationship with his son. The dad buys stuff for my friend but my friend has told me he's a real terrible dad who's never been there and he is pissed about it. HELLO THIS IS YOUR HUSBANDS KIDS! Ask him how he would feel if the kids got killed or committed suicide or ran away. Have the kids tell him they love him. Try and get away as a family. You are not alone, trust me..I know