You still haven't said where you live, so I'll answer as if you're in California, which is all I know about anyway.
No, he can't tear it out, and you don't have to pay him yet. You are technically in the middle of what's called a "Good Faith Dispute." And until it's resolved, he can't tear it out, and you don't have to pay him. If it turns out he was in the right, you'll then need to pay him and you might be on the hook for interest, late fees etc. If you're in the right, he must fix it before he gets paid. He technically can't even demand payment, let alone collect it, until the resolution is complete.
If you think this is turning south, you should start your paper trail, stop talking to him, and communicate only in writing. But cooler heads might prevail, so no harm in a sit down with him and the manufacturer and see if you all can come to a reasonable resolution before you start firing cannons. It'd be prudent for you to have someone in your corner, literally, who can "witness" the conversation and help you keep track of what was said (and be able to testify to that should it go down that nasty road).
Good luck, be tough, be careful, be fair.
PS. That's all assuming your location has similar laws. Which is not guaranteed. CA is pro consumer in these types of matters...