I fix all my ballast with PVA, but it must be the waterproof variety, mixed neat in a small plastic pot with the ballast of your choice (I use alpine grit). Just enough PVA to give the ballast an even coating is enough. I then apply this to the track with an old desert spoon (the tool of my choice !), working it between the sleepers and building-up a shoulder as required. Any 'tweeks' can then be made to the track to cross-level, apply cant to curves etc before the glue sets.
Yes the ballast does go white several times after rain, but this soon stops and will weather nicely.
Another advantage is that should you require to lift track then the glue can be disolved by pouring hot (not boiling) water over it and then simply picking away the ballast.
My track is laid on a concrete bed, then ballasted as above. Early sections have been down some 5 years now, the only problems are in areas where I experimented with non-waterproof PVA or tried to make it go farther by diluting with water. Waterproof un-diluted PVA works for me.