I choose stones that are roundish, sort of egg-shaped with a flattish side. I lay the stone with the flat side flat in my sling pouch, not on its side as it would seem more natural to do. I have found the projectile to fly more predictably when loaded in this manner. Too flat of a stone will take all manner of unpredictable paths. I have seen flat stones take a 90 degree turn in mid-flight. I have also experimented with sun hardened clay.
I've used ammo ranging from spherical to eggish, smooth to rough. I only avoid jagged ends that might cut my pouch from the force. I much, much prefer spherical ammo, I find it's much more accurate. If there is a flat side, I'll put that flat side against the pouch. But I would much rather shoot spheres and if I have to shoot one with flats, I'll try to find one with as many flats as possible on it. I also like to put several small stones in the pouch and shoot them all at once for the shotgun effect. This is particularly good with birds and small game, but does need a little larger pouch than my usual one. I've not experienced the erratic flight paths you describe, but this may be due to my avoiding if at all possible ammo with flat sides. The release from the pouch imparts a spin of sorts to the stone, and this may account for your erratic flights. This spin isn't much, but with a flat side that could rotate towards the path of flight, that might make it plane off. Oh, and for size, I like them to be about 1" diameter, but no more than 1 1/2" diameter or less than 1/2" diameter, unless I'm "shotgunning." I think the smallest I've used, even when shotgunning, is 1/8".