In concept there is nothing wrong with building on fill. People build houses, roads, bridges, and buildings on fill all of the time. The thing is, that fill is engineered. Soils will compact to a maximum density only at a specific moisture content. Too wet or too dry and you will never reach maximum compaction. If that happens, and then later on the fill reaches that ideal water content, it will settle.
Joe Pool (or Joe Homeowner) who removes some soil from the high side, and then "packs it real good" with a plate compactor that they rent from the local big box store - or worse a jumping jack, or even worse a hand tamper - is just asking for trouble down the road.
ABG are also very finicky. They are some of the most unstable things ever. It is not like building a shed or garage that is mostly rigid, braced, and has a solid floor that can bridge the occasional small soft spot. An ABG is a garbage bag filled with water with a hula hoop jammed in the opening for support. That is about it. If something starts to settle, the whole mess is going sideways quickly.
So, short answer is you can build an ABG on fill, but is HIGHLY recommended that you do not, because you REALLY need to know what you are doing (I have not even got into measuring soil density, moisture content, calculating maximum dry density, or acceptable methods of ground modification). The outcome of having a 27ft pool fail, is roughly 17,000 gallons of water going wherever it want to.
The very fact that the fill is described as "red clay" is worrisome. Clay in and of itself is a difficult fill to work with, especially when it comes to compaction (vibratory compaction is next to useless). Clayey sands on the other hand are excellent. You also fail to note if this is an expansive clay or not (hopefully not). Now if your contractor says they are going to use select fill, can provide the soils data on it, and is going to test it as they compact it, that is a different story.
Also, you are building over an old leach field. How supportive is that ?