Now as i understand it, you do not have a suction type pool cleaner installed that normally connects to the "dedicated line" ( or vac port as you call it ) on the pool wall, at one point there most likely was a pool-vac, barracuda, kreepy, etc that came with the pool when new, dedicated lines are the best way to get maximum performance from a suction style pool cleaner, when compared to there unpredictable and inefficient operation when the cleaner is forced to share the suction resources from the skimmer, which makes both cleaner and skimmer very inefficient and problematic.
If you have children, you should cover the vac port by threading in whats known as a "vac-loc" or be certain the vac port is off, if there is suction being pulled from an exposed open vac port line and a child's hand gets grabbed by the suction it is capable of, ouch.
Because you have the dedicated line installed, it would benefit you greatly to purchase a suction style pool cleaner this will eliminate the manual cleaning required now, it is uncommon to see a pool today that does not have some type of automatic cleaner, many ,many pool service company's will not service a pool without one.
There are two situations here,
the first is your current one, and it is regarding the diverter valve, your not pulling as much suction from the main drains as you think you are, look inside the skimmer, i am sure you will notice water movement indicating the skimmer has the majority of the suction from the pump, any suction resources left for the main drains via the diverter valve are minimal, and further weakened by the fact that two main drains tie into the one pipe, there function is highly overated, the only true way to have optimal drain pull from the bottom is to have the drain separate and on its own dedicated valve by the pump.
Ask yourself will you benifit more from surface skimming? if so, 86 the diverter valve, the precious resources you give up for that, vs the minimal effectiveness applied to drains with valve in place will make more sense if you have leaf, debris demand on surface, etc.
However, if you grasp the diverter and hold it in such a way as to keep the float (the 4'' diameter part that rattles inside diverter when holding it in your hand while shaking it) steady with your fingers spread around the floats perimeter as you carefully lower inside skimmer to cover the two pipes, as it falls into place over the pipes in skimmer you should notice a strong grab from the suction pipe and notice the water movement inside skimmer is calm indicating the float has now been stuck in position by suction and it will maximize the pull from the drain, this takes some practice to do, and will only stay like that as long as pump does not shut off. this is the only way to improve the suction to the drain over just placing the valve in skimmer.
This might be helpful when brushing dust etc towards drains.
situation #2
if and when you get a cleaner for the dedicated line, the diverter valve is no longer needed, the pool vac moving along the floor is sucking things up as it goes, removing the diverter provides a stronger skimmer pull now that it doesn't have to share any of its, albeit, minimal resources with drain funtion, and can pull surface debris better as a result, get it.
I'm done.