It's virtually impossible to change a bromine pool to a chlorine pool. Without getting overly technical, the active form of chlorine, as soon as you add it, will find the inactive form of bromine, and make it active, becoming inactive itself in the process. So basically, every time you add chlorine, you are effectively adding bromine. (I have a bromine spa, and when my bromine levels drop, I add chlorine, with no bromine content, to increase them.)
The only way I know of is to drain the pool. Perhaps someone with more knowledge knows of some other way, but while it's easy to convert chlorine to bromine, the opposite is very hard.
If you do try to drain the pool, make sure you read up on how to do this. It's not a simple matter at all, and draining a pool can frequently cause irreversible damage (draining a small amount off the top won't, but draining it completely can). A vinyl liner may produce a big bubble on the bottom as water flows in, permanently screwing up your pool's shape. Fiberglass pools have been known to pop out.
In general, you're probably right that you need to switch. Bromine is generally not recommended for outdoor pools, especially in sunny areas.