This happens because the fields are bound to the collection list, not the other way around. But if you feel this should be added as a feature, I would recommend checking out our Wishlist. First to see if it or something already exists, if not create a new item and then post the link back here.
Yes, I understand why it’s like this I often reuse modules like e.g. contact cards that is linked to a collection with employees. Now I have to make a copy of the collection element and manually unlink all the connections to the collection list, making it a static element before pasting it in to an new collection list/page. It would be a nice feature to get an option to automatically unlink all connections to the previous collection source when getting this error message: