Scroll into view interaction issue

Hello guys!

So, I am working on this scroll into view interaction. Although the interaction works as it is supposed to when I scroll down but it acts weirdly when I scroll back up. I tried changing things and offset but still, it is causing issues. It would be great if I could get some help from you guys.

Thank you
Sagar Arora

Here is my public share link: