The FC guidelines here are designed to keep algae in check. The lower limit really is the absolute minimum you should never get under so you don't risk algae growth. You should always aim to have a safety buffer to account for testing errors, low circulation patches in the pool, or just unexpected events (work, kids sick, etc) where you can't top up with bleach when you planned for.
If your pool really doesn't have any algae, then going occasionally down to or even slightly below the minimum won't trigger an instant algae bloom. But if you already have algae in the water then going lower might let them thrive. That's where brushing is important.
If you are not that thorough with your brushing then staying towards the upper end of the recommended range would be a good idea.
But even at the TFP minimum you would still have some safety margin to keep the water safe. Most pathogens are killed very quickly even with low levels of chlorine. There are some exceptions, but that's another story.
As long as only your family is using the pool, person to person transmission is usually not a problem, because your are exposed to each other's germs anyway, you just need to make sure that germs get killed reasonably fast. But when having other people in the pool then you really want very fast bacteria kill times to avoid person to person transmission in the water. And you want to make sure that e.g. bird droppings get sanitized quickly.
But as mentioned by others already, just because the water is clear, doesn't mean it's necessarily safe. Bacteria grow much faster than algae, should the FC be too low.