This actually isn't too far fetched and you can buy units specially made for use with swimming pools. It's pretty close to a geo-thermal heat-pump system. The water handling tubing is "upgraded" to deal with the pool water.
The place where I work actually has one. The pool is in the shade...