If you make a verbal or written contract, you can usually get out of it as long as the other party has not substantially completed any part of the contract.
If the other party has relied on the agreement and has incurred costs, you will usually be responsible for the costs incurred.
Since the contract is for more than $500.00, the concept called the Statute of Frauds probably applies, which requires a written contract.
The statute of frauds (SOF) is a legal concept that requires certain types of contracts to be executed in writing. The statute covers contracts for the sale of land, agreements involving goods worth over $500, and contracts lasting one year or more.
If the contractor can prove that they incurred costs related to the agreement, they might be entitled have those costs covered.
For example, if they were holding a date for you and they subsequently declined a new request for work due to the date being held, that might constitute a cost.
Ordering a pool might be a cost depending on exactly what was done.
Did the manufacturer begin constructing a custom pool?
Was anything pulled from inventory and shipped or prepared for shipping?
Did the contractor spend time designing or planning the project?
If they can't prove costs incurred, they should return the money.
Just my opinion, not legal or professional advice.