Mimir was on your belt as you two ventured through Valhalla to "Master Thyself" as it said and hes angry at you for not getting freya.

Mimir: "Brother. Why are you avoiding freya! She would of aided us in this."