It's not that it won't work, but that is isn't necessary. By the way, Natural Chemistry Phos Free has a clarifier in it to help prevent cloudiness from the lanthanum phosphate precipitate.
I had over 3000 ppb phosphates in my pool and there's a pool owner in Hawaii on this forum with over 30,000 ppb phosphates and yet these and other high phosphate pools are kept algae free with chlorine alone. Phosphate removers need to be looked at in the same vein as algaecides. They are insurance you can buy for extra money if you don't think you are able or willing to properly maintain an FC level appropriate for your CYA level.
There is no question that a pool high in phosphates is more "reactive" if you let the chlorine get too low. Algae will then be limited in their growth rate by sunlight and temperature, rather than by nutrient level, but they are still limited in their growth to roughly doubling every 3-8 hours under ideal conditions. With the appropriate FC/CYA level, the algae is killed before it is able to reproduce in even one generation. It is essentially killed when spores get blown into the pool.
You are right that phosphates can get introduced in may different ways. They are in my fill water at 400 ppb since it is often used as a metal corrosion inhibitor by municipal water districts. It is in fertilized soil. It is released from HEDP-based metal sequestrants when they break down from chlorine.
If you have had cloudy water when you weren't using a phosphate remover, then you weren't properly following the practices of maintaining a proper chlorine level relative to the CYA level and didn't close and open the pool in a way to prevent or easily remove most algae growth.
To each his own. It's most important that you and others understand what goes into your pool, what it does, what it costs, what alternatives are available, etc. Then you can make your own informed decision as you have done. That's the true spirit of this forum.