All of your concerns are legit. The problem is, this is what masquerades as construction these days. Where consumers must become inspectors and general contractors, just to make sure they get a mediocre job!
A construction project is now a constant negotiation. You're at their mercy, and they know it. Stand your ground. Fight the battles you can win.
But you do have some leverage, if you don't pay him for more than he has produced. That's the key. Many (I wanna say most) contractors are living day to day, robbing Peter to pay Paul. He cannot afford to have a payment delayed, and often times will ask for one before its time. If he does that, that's the clue, that you now have the upper hand. Or at least some hand.
Lastly, if you have a dispute about somethings that is merely "how you would do it" vs "how he would do it," there is not much you can do about that. He does have some discretion about how he builds your pool, as long as the end result satisfies the plans and the building codes. But...
There is a thing known as "Construction Standards" that might apply (it's known by various names). A contractor cannot just build something any which way, or cut corners, but do sub-par work. He is legally liable for some degree of quality. I cannot define it, but a local attorney should be able to. Again, you can't rely on city inspectors for it. And maybe not even some other contractor, who might have his own inaccurate ideas about what it means. But it is something you'll hear in a court case, and then a judge decides what it means (usually based on some sort of expert witness testimony). Anywho, you get the gist.
You have his money, and you have a right, a legal right, to expect a certain level of quality. That's your leverage.