I'll say you SHOULD brush your pool frequently.
However, I'll also say I've brushed mine once, maybe twice a year over the past 5 years (maybe a few more times the first few years we were here) because I'm busy, lazy, and dislike brushing the pool. I'm careful about the chlorine levels, always well above the minimum, and have never had an algae issue. Not saying it's a good idea, just a data point
We had a Polaris 280 (which would climb the walls but didn't brush, and now have a Dolphin Explorer E30 that climbs and brushes the walls so I don't have to feel guilty about not doing so.