I did an embed as you suggested and copy-pasted the exact code that Webflow had written.
I discovered that the webm video format is not compatible with IE11 so I removed that source and it worked!
Then I tried adding it back in but listing it second (after mp4) and that worked too!