Depends on if it is indoors, or outdoors.
Real ballast is crushed stone, the more angular the better. Haven't got the spec with me, but around 70-100mm is about right. Scaled down, that's around 4-6mm or so.
There is a grading system called All Pass, that means when sieved, everything below a certain size passes through. AP 6 is a nice looking size to go for. But, here's the catch, that will represent 'dirty' ballast, looks nice, but will wash away over time.... all the fines sink into the earth, leaving you will a severe shortage of ballast.
Outdoors, for the past 22 years, I have used 10mm angular chip. Well oversize, but it does what ballast is supposed to do, that is hold the track in line, support it, and allow the rain to drain away.
Do NOT use round pebbles, pretty nice to look at, but pretty useless because the roundy bits roll over each other, and will not hold any shape.
View attachment 297931
Random shot from my railway... but not my train, that was taken at a group running day that I hosted.