You will not be able to use a traditional vacuum with the pump/filter that came with your pool, sorry to say! It is just too wimpy.
Best advice I can give is to stir the water with your skimmer, making a whirlpool in the middle. As the junk from the bottom flies around the whirlpool, you can get it out with the skimmer. This action also suspends smaller particles so the filter can get them. After you get the big junk, you might try putting pantyhose over the skimmer to get a finer mesh, and get some of the smaller stuff.
ETA you can try putting some cotton or aquarium filter material (the stuff that looks like clouds) in the bag for your venturi vac... that will sometimes help catch the smaller stuff, too.