I think you might be right, the value might need to be a slug for the import to work.
If you can’t change the values to slugs, you can use PowerImporter instead. It will allow you to upload reference fields by any value, including name (as long as it uniquely identifies a single collection item.
Hey Marc — I am having the same issue (and have tried both slug and name), and it picks up for some, but not others. There is also no consistency to which ones are recognized and not.
I also tried PowerImporter, but I’m getting this error — Upload file error: undefined method gsub’ for nil:NilClass`