You live next door to the Dursleys, the adjacent fence between your plots is low, so you could see Harry coming out into the yard.