I built my pool myself, and my complete excavation cost $1800, I can't imagine that it would cost anywhere near what they are quoting you. Further, if it wasn't covered in your contract, they should have had you agree to the additional work prior to actually doing it, or they run the risk of footing the bill themselves. You can offer them a "bone", in order to settle the amount.. something like 1500 I would say, which should be plenty. Otherwise, you can tell them to "pound sand" if it is not in your contract about the overruns, and they did not ask for your permission and discuss it prior to digging, its their own problem. Heck, if it was rock and they had to use a jack hammer, the costs probably would not have been that exorbitant.
If you can't agree, I am sure they are licensed with the Contractor's Board in your state, and a phone call to the board might help. They will typically send someone out to evaluate the entire situation, including the up-charge. No company wants the Contractor's Board to start getting involved.