With a small residential construction job like this, your real hold over the builder is the last check at completion. It's usually a decent chunk of his profit. Unless you're really convinced that it's an irremediable catastrophe, let him do his job. Keep track of your concerns, and at the end go over the work with a fine tooth comb and give him a punchlist. Don't give him the final check until you're satisfied he's addressed your concerns.
I had somewhat similar problems with my builder - I bugged him about the compaction and the way the rebar lay, I complained about backfilling with sand, I argued about pipe diameter, etc. I paid for extra gravel, and I still have $1,000 of his money until he demonstrates that the pump is working correctly when we open the pool this spring. There are still problems - shallow end too small, wrong jets, wrong light, jets not on separate valves, barely functional curtain drain - but I can live with them.
Good luck!