You MUST use either washed in sand or the smallest gravel available. The gently washed in sand and gravel will settle the least, ensuring the walls are provided the support they need. Fill the pool and back fill at the same time, never letting there be more than a foot of difference in height between the two. To do otherwise will result in warped walls, cracks in the gel, or worse.
The use of spoils (dug out dirt) as back fill will not prevent warped walls and will settle unevenly, resulting in lost support under the concrete apron around the pool.
That the installer is even suggesting that tells me he needs to be watched like a hawk.
Ensure the shell is FULLY supported on the ground and level to within 1/4" all the way around before adding water.
Scott