Grindelwald has many known followers, turning your back on him means turning your back on the wizarding world. You were his most loyal and oldest follower, he would give you most important tasks, he was always a cold person. Today you completed another task and went to his mansion and knocked on his study door