Having grass that close to a pool is almost as bad as putting sand by the ocean. (I've tried to convince the Big Guy he should replace all that, but so far he hasn't responded!) My now-removed lawn was a good 5' away from my pool, separated by deck and coping, but my pool would get trashed with grass every Tuesday (mowing day). There are lots of reasons why almost every pool you've ever seen in a magazine or website, or used yourself, is surrounded by 3' or more of some sort of hard surface: concrete or pavers or decking, etc. Landscaping and grass right next to a pool looks good on paper, not so good in reality. With a decent amount of pool traffic, I'd be surprised if grass would do well. Pool water is saltwater.
To your question, I wouldn't want gravel so close to either my skimmer or my auto-filler, because gravel will wind up in both. Smaller chunks could make it through the baskets and into your impeller and filter. Gravel in the leveler, if it's anything like mine, would end up in the equalizer tube and would be tough to remove.
If I can't talk you into surrounding your pool with concrete, at least surround your skimmer and leveler with it. You could use colored and/or textured concrete, or aggregate, to soften its appearance.