You can try just hiding the the empty items with the current column layout and se if that produce the result you want.
If it is not refined enough, you will want to change the collection wrapper to display flex and set the collection item empty state to hidden. I don’t know if you used flexbox much, but see how you go and advise if you get stuck.