Well, (and not trying to be a wise guy) there really is no "bigger" problem. We all use chlorine in our pools to keep them sanitary and to keep them algae free. That said, the issue is not as black and white as we all want it to be. Sometimes the guidelines we suggest just don't seem to keep your pool as crystal clear as others but NEVER lose sight of the fact that it is chlorine that kills algae and then filtration that clears our pool water of any visible debris.
You have green that brushes away so, simply put, you do not have enough chlorine (at least in that area) to keep it from reforming. What's the fix? Apply more chlorine to that area and brush it more frequently to expose it to the increased chlorine level. Regardless of your test results and maintenance, just remember what you have done is not enough. There is no alternative and there is really no other "bigger problem". It is simply a matter of adequate chlorine. If your pool requires more than our guidelines, just give it some more. I think you will find that you will "catch up" and eliminate that problem just by being persistent.
The cloudiness or dullness you see may not be completely related......it sounds like more of a filtration issue. What is your clean psi? Is your gauge showing an increase in psi over a few days? When do you backwash? Have you ever decided to run the pump 24/7 to combat the cloudiness?