Category Listing - only display categories with associated content

I have a blog, with a reference field for the categories. I want to list the blog categories in the sidebar, and a navigation menu for the blog content. I don’t yet have blog posts in all of the categories in the category set. I would like a way to show only the categories in the sidebar, that have a blog post assigned to them.