Behavioural studies have suggested that number manipulation involves shifting attention along a left-to-right oriented continuum. However, most of these studies do not inform us about the time course of attention shifts during number processing. In this study, we used an eye-tracker with high spatio-temporal resolution to study horizontal and vertical gaze patterns in two experiments that combined the free exploration of visual scenes with a verbal numerical task. The first experiment required participants to compare numbers (from 20 to 70) to a reference (45). Results showed that eye position deviated relatively to the magnitude of numbers: a rightward and upward shift was observed when the number was larger than the reference compared to smaller, starting after hearing the number decade and lasting until response production. The second experiment investigated the mental solving of addition (43+4) and subtraction problems (53-6). A first difference in eye position was observed just after hearing the operator: the plus sign shifted the eye rightward compared to the minus sign. A second difference was observed later between problem offset and response onset: addition shifted the eye rightward compared to subtraction, but only in slow participants. These findings provide unique evidence about the time course of attention shifts during number comparison and support a two-stage model of mental arithmetic where the operator first triggers the setting of spatial frame to help situating the response relative to the first operand, and the calculation then proceeds with a re-calibration of the frame reflecting the computational constraints.