Hi @zach, the random sort refreshes only at publish, because publishing updates the CDN cache.
The sort from CDN cache is updated every 12 hours otherwise or if using some custom script solution (at the moment).
I would look into creating a new post about how to create this using custom code until the desired behavior is updated to the feature. It certainly sounds like this is a good thing to make possible 
Stay tuned for more updates in this area.