If your skimmer has two holes in the bottom of it, it (the pieces) could be entering through the main drain. Some systems pull main drain water un-screened through the skimmer, therefore, it could be something (introduced) into the pool. Either from the surface or the plumbing (anything down stream of the filter). If you have SCUBA equipment, or can hold your breath, take the main drain cover off to see if that sheds any light.
Can you make marks with the pieces, as though its charcoal like? Some rubbers get stiff and brittle when they are age-ed or subject to long exposures to water and chemicals. Although the silver or grey color would disclaim rubber. Maybe its my eyes, but all the pieces look to be somewhat dished, or curved. Is this pool colored? Could that be slurry from the plaster that fell down into the main drain line and had just been sitting there for years until perhaps something happened chemically that caused it to begin releasing from the plumbing?
Note: did not see page two at time of post...