I would use scroll interaction only.
Set “While page is scrolling” to the “Page trigger”. And move the layers based on percentage of the page scrolled. You can simulate the move of layers to Fullpage.js by bringing the percentage of layer movement closer to each other.
I created a demo project so you can see the settings.