For what it’s worth, here is my experience with trying to find where the issues are when it comes to PageSpeed Insights.
Overall this site is pretty optimised - all images as compressed as possible, other than GTM, nothing but stock webflow js, animations etc. Relatively few and simple animations, overall pretty simple layouts. Hosted with Webflow.
As-is currently it’s typically around 62 - not completely horrible, but far from ideal
Removing every single animation, that pushes up the score by a couple points to 65 - so those have little impact
If I remove every single image from the page, it reaches 70-73 - good but not a massive improvement and not one that is acceptable for real world
On the same page without images, animations etc, if I remove the font that is being loaded from Google Web Fonts (DM Sans, only 2 weights), that pushes the score into the upper 85-88
At this point I’m effectively out of all options for what can be optimised on the page. There is just one thing left - Google Tag Manager. On this site it’s not overly heavily used, is mostly there for GA4, Adsense conversion linker, and Google Optimize.
Now if I entirely remove GTM from this page (after all other changes above), it takes me from mid-high 80s, to 97. Keeping GTM but excluding the Optimize tag typically yields only a 1 point difference for me)
Taking the same page that was at the start scoring 62 at the start and only removing GTM from it, takes it to 84 without any other changes.
So effectively using Google’s own product (GTM) alone, produces as big of a hit on performance on Google’s own performance test (which does contribute to search rankings even if slightly), as virtually every other change that a user can make to optimise their site.