Eight years ago, when I built my railroad, I used gravel/granite type chippings that I purchased from a DIY store. I did encounter the issue with the electrics that Charles, in Colorado, mentioned. I changed to buying gravel chippings from a builders merchant ever since. The colour, incidentally, is. more or less. the same as the ballast used on must British railroad track formations.
I buy it in 25kg. bags: both 10mm. and 20mm. Certain areas are suited to each size. In some areas I mix it.
A base of 3" thick timbers and concrete blocks are what I used initially in the track beds construction. None of it has ever moved and of course weeds do not come up though it.
I do, on a regular basis, prick out seedlings which have germinated in the ballast: that will happen when your track bed is a ground level. But that is all part of the garden railway maintenance schedule.

Every couple of years I replace the ballast; it is surprising how much loam is beneath the ballast generated from dust and decaying foliage etc. This is sifted and laid on the gravel areas - driveway and greenhouse surrounds, so it is not wasted. Working on a Heritage railway did give me a good idea how railways were built and of course built to last.