Mine were at least that high for a long time (measured in the lab I manage) and they didn't really seem to affect anything, Re: algae growth so long as I kept the chlorine at the proper levels. At the time, I was using bleach and travelled a lot so I often came back to a green pool and had to SLAM. At some point, I decided to go ahead and get rid of the phosphates in the hopes that maybe I could stop the algae blooms from happening so fast and/or shorten the SLAM duration.
First off, don't bother with PhosFree- That stuff is completely useless. Get the P-10,000 from Amazon. For my smallish 14K gallon pool, I only used about 8 ounces, mixed it with a bucket of pool water, and spread it around the pool. Once it was mixed well, I shut off the pump and just let it settle out. Next morning, it looked like I had popped a DE grid at the bottom of the pool. I vacuumed all of that to waste and let the pump circulate the rest of the day. Next morning, phosphates were pretty much at zero. They've since climbed up a little from the rains so I might repeat this process soon.
Has it helped the algae blooms? Can't say. Right after that, Houston had the deep freeze and I popped some of my PVC plumbing. At that point I upgraded to salt so I can now leave town for a week and not have any issues.