Absolutely unbelievable. I'm so sorry to hear this news. Contractors. Amazing what they try to pull. Let us know if you are seeking thoughts about next steps, but suffice to say, the good news is that this happened in California, and he will get racked over the coals eventually, as this state is very much pro-consumer and anti-contractor. I'm in the throws of a bad pool contractor situation myself, so I have nothing but sympathy for you. My situation will resolve itself in my favor, as will yours.
What you were asking for, while justified, is actually a complete non-issue. This guy is off the rails and his actions are indefensible, even if you were a complete jerk about the way you were proceeding (not saying you were, just saying that doesn't matter in the least). Walking off a job in CA is a HUGE no-no. He's already buried himself. If you don't mind me asking: Have you given him any money? If so, do you think you are ahead or behind in what is owed? Does your contract have a three-day cancellation clause? That can be key in your situation.
Hopefully, he's a hothead who was having a bad day, and who will come to his senses, and this will resolve itself with an apology and a handshake and everything will get back on track. That said, if that doesn't occur in a day or two, I'd recommend you go here and get started. Let this guy know in no uncertain terms that you mean business and have no qualms about taking immediate action if he doesn't get back to work:
http://www.cslb.ca.gov/Consumers/Filing_A_Complaint/
I wish I could convince you not to stress as this will resolve in your favor at some point, but I know from my own experience that is much easier said than done. You'll get great support from TFP, and rest assured, if you can, that this is a mere bump in the road to an amazing backyard, that WILL come to fruition, with this contractor or another.
And if it's of any help: while my situation caused me considerable effort, stress and grief, now that it's almost behind me, I have come to see that getting rid of the offender, and getting the repairs done (to resolve the damage my guy caused) has resulted in a significantly better pool than when I started, the majority of which is going to be financed by my contractor, thank you very much. Point being, in the grand scheme, this guy did me a favor, and he'll get what's coming to him.
Maybe your guy will finish your pool, and maybe he wasn't meant to, and if not, maybe the next guy will do a better job.
Hang in there... things will be better in the morning...