Most thermometers are heavier than water so weigh the skimmer down. At least my chrome one does for my skimmer basket. Normally the skimmer basket doesn't float as much as you are describing (i.e. not enough to pop off the skimmer cover).
As for rocks, they are probably OK as they are already weathered and unlikely to dissolve. You can test them by adding some acid to them. If they fizz, then they have too much calcium carbonate in them -- not that this is terrible, but you might as well use something that won't tend to either dissolve nor form scale. If they change significantly in color when acid is added, then don't use them as they probably have too much metal content.