Jkrup, like most things in life, pool ownership is about priorities. People don't think twice about spending that on a car, for example, or on cumulative family vacations over 10 years, for example. As a pool owner, I'd personally rather have the pool than a new car (I buy used) and I've travelled enough in life that I wouldn't mind forgoing trips if push came to shove

in season, I don't even want to travel
With that said, in terms of the impact of a pool on a family budget, there are a few considerations. First of all, its great to be the "place of gathering" - which happens pretty quickly when you have a pool. If you're the type of hosts who are inclined to provide guests with food and drink, you will likely spend more on steak and ale than you could ever spend on liquid chlorine and muriatic acid in a season
Utility costs to operate are in my area pretty low - eg $20-$40 mo. for a regular pump depending on run time, less on a variable speed pump. Heating a pool -- necessary in my particular location due to shade, overnight temps, and a desire to swim in warm water at dawn, CAN be expensive but also can be dramatically reduced by using a solar cover. Due to a tight footprint, I've not historically used a solar cover and we keep the pool at 88. Our average gas bills for constant heat run about the same as our winter house heating costs - generally $200-$300, a bit higher in early May or Oct. But any people run pools without heat and ergo have no associated expense. We could too...we just don't want to
Lastly, your sunk cost on construction isn't the "final" cost to a amortise over the life of the pool. If vinyl liner, for example, you will need to replace it at some point, and would be best to plan on so-doing in approx 10 years. Current costs for vinyl replacement run in the neighborhood of $4,000. Heaters are good for an average of 7 years. I'm on year 15 of my pump and sand filter, but my equipment is enclosed in a heated pool house that seems to extend the life of the equipment.
There's not a lot else to go wrong if you follow TFP pool maintenance. Depending on size, you might spend between 40-80/ month on supplies such as liquid chlorine, muriatic acid, etc. -- more if you're on well water and use sequestrant. If you use a Salt Water Generator, most people get between 5-7 years or more out of a $400 cell.
My only child is now 25, and I've had this house and pool for 5 years. My only regret was that I didn't have this house and pool when he was young

But with that said, time flies. Don't get a pool if its strictly for the children -- the pool will outlast their youth. Be sure that YOU will want and enjoy the pool long after they've flown the coop! I do know people with pools who "got it for the kids" and aren't themselves daily swimmers. I think the happiest pool owners are ones who love to swim, obviously. So make sure you and your partner fall into that category before pulling the trigger.