Phosphates as high as 3000, and possibly even much higher, are not going to have any significant impact on the pool. Keeping phosphates below 300 can reduce chlorine consumption a little, but the effect is small. There is no significant variation in chlorine demand as you vary phosphates between say 500 and 3000 (and probably much higher as well though that is not well established).
Various soaps contain phosphates as part of a more complex substance. The chlorine will attack the soap complex and break it up into smaller pieces, consuming chlorine. One of the smaller pieces is phosphate. Adding soap will result in extra chlorine demand, which will go away when all of the soap is broken down. The phosphates will remain, not causing problems, unless you use a product like PhosFree.
I suppose that the PhosFree may have bound to some of the soap that had not broken down yet and cleared that out, which could be why chlorine demand went down when you added PhosFree.