Ok I know now caused the loading delay.
I removed the Google Optimize container tag and the page speed turned to normal.
Anyone have an idea why the Google Optimize tag is causing such a performance problem?
No issues with GTM (Maybe you use the same code twice, load google-analytics and so on by GTM and manually by copy-paste code). The opposite is true (GTM optimize site speed from code integration side).