We always have had pools, but they were for "us" - the kids - as much as anything else. My dad, though, swam until his last year and loved swimming, so the pool was also for him. My mom, on the other hand, waited until my brothers were off in college and she was an empty-nester, and BOOM had her pool dug up/removed, and the area converted to more space for her gardens. She NEVER swam, so once her last kids were out of the house, she changed that part of the yard to what she most enjoys - gardens for her flowering plants and wildlife. My older sisters and my one brother who all have kids have gone the pool route. The sister whose daughter is now grown, though, didn't bother to go the pool route again when she moved after my niece was out of the nest.
So, I definitely think things like pools, basketball courts, tennis courts, etc, all make less and less sense once they're just something extra you have to maintain, but aren't enjoying very often or ever. There's a cost to remove stuff, though, so you also have to balance it versus maintenance and future resale. Absolutely great when you have kids and are using them, not so much later.