If you get to the point where you don’t like someone and you don’t trust them and you don’t want to deal with them anymore, you can fire them and you have to deal with the consequences, which can mean extra costs depending on why you are terminating the contract.
If you don’t have a justifiable cause for terminating the contract, then you can be liable for all costs up to the time you fire them.
The costs can include lost opportunity if the builder can show that they turned away work due their commitment to your job.
The builder might be entitled to some profit on work up to the point where you fire them, but that’s included in the costs for work and materials completed at retail price.
It’s never ideal to have to fire a contractor, but if you get to the point where you can’t deal with them anymore, it is what it is; you fire them and deal with the consequences.