Sorry about your frustration and the slow progress. Not to minimize your woes, but as TK points out, we've seen a lot worse here. Before you start down litigation road, let's take stock. You mention that there are too many issues to list. Fair enough, but what you've described so far does not necessarily require litigation, and not at this time. I'm sure every consumer would like their construction project to go perfectly smoothly, with no defects or problems, and end on time. That is the goal, and certainly a reasonable expectation, but unfortunately not always the reality, and not always the general contractor's fault. This is never more true than with pool construction projects. There are a lot of sub's involved, a lot of materials involved, difficult conditions to deal with, and a lot can go wrong. And that's in times when pool contractors are not overwhelmed with work, and handicapped by supply chain issues!
That said, there are plenty of shoddy contractors, so let's sort this out. Problems and delays are part of the process, but it's what the general does about them that determines next steps. And you say you've paid him "enough money to cover his costs" but he is entitled to profit during the course of the job. He's gotta eat, too, right? You can't expect him to work for months for you and only get paid enough to buy materials. That's not reasonable.
The damage you describe is unfortunate, but that, too, can be part of a construction project. He's literally moving heaven and earth for you, well earth anyway, and that requires using big equipment in a small space that is already filled with structures and plants and all sort of obstacles. What did he say he was going to do about the damage? And did he say when it was to be repaired? It's reasonable to sort this out amicably. The tile is another example. You ordered tile together, it ended up being defective. He's replacing it. That happens. Scratch that off the list. Scratch "driving 50 miles" off the list, too (that's a sign you're letting emotional components of this situation get the best of you, sorry to be so blunt). You caught the plumbing problem, and that too, got fixed. Scratch that off the list. Yah, that was bad, and should have been caught by the general, but we're all working together here, right? To get the best pool possible?
Again, I'm not siding with the contractor, but I'm not yet hearing that he's an ogre. Do you have a contract? Does it have a draw schedule? As long as he's not asking to violate it, and you are retaining enough money to complete the job, than him asking for a scheduled draw is not "audacity."
If your goal is to sue somebody, then by all means bring your blood to a boil and go for it. But if you want your pool finished for swim season, and you want a quality job, then I suggest you reevaluate your position and instead find better ways to work with this guy. Follow Herman's good advice, and start documenting the process. Communicate in writing when possible. Follow up on-site meetings and conversations with emails that detail all the talking points. You should be doing that even on a job that has no problems. But I would suggest, after only four months, you rethink litigating anything at this point.
When I get stuck like this, I give it the ol' Judge Judy test. I use this test in all sorts of confrontation situations. I picture myself describing the situation to Judge Judy, and try to imagine her response. She'll either chew me out, or she'll tear into the other guy. If you told JJ this story, and said after four months of heavy construction, during the winter months when you can't even swim, that the contractor has allowed some problems to occur but he's been fixing them all at his expense... I'm not so sure she'd be chewing out the contractor. She'd ask him what's up, and maybe demand a timetable, but she'd then ask you what, exactly, are your damages, and what are you doing in court?
Sorry, probably not the support you were looking for, but for now, I'd rather see you figure out how to work with this guy, and save the talk about litigation for when you have no other option. That's my take, at least until you describe the other issues that might paint an uglier picture.