This is very true. I develop subdivisions and this is standard practice.
Having worked in countless housing developments doing the site development, the only part of the lot if it was filled, that needs to be compacted is the building envelope. That is the place where the house will be built. The size of the envelope is determined by the size of the house to be built. It's usually big enough where the builder can shift around where the house will be built. The areas outside of the envelope can be filled and not compacted. I've seen where those areas are filled with top soil.
What could be going on here is the PB dug down and didn't hit virgin ground or the material used to fill is unsuitable to build on. What he probably wants to do is dig down some more and bring it back up to grade " the bottom of the pool " with the crush and run, and put it in lifts and compact each lift to get something suitable to build on.
At any rate you can get a soils engineer to take a look at the site and he can determine if the fill is suitable to build on. I would do this if you don't trust what the PB is telling you.
Just my 0.02. ��