Better to think of expansion as being a 2D dot on a balloon.

As balloon expands, in all directions there is same rate of expansion. You are not inside the balloon close to one side to observe the difference.