During the first slope (positive slope) the "sampling" time is a fixed value and therefore the duration gives no clue about what the input voltage is. The peak value attained contains the only clue but that is unknown to this type of ADC.
During the 2nd slope (negative slope) the input voltage is disconnected and the counter begins. The voltage on the capacitor is discharged by a constant current source and when that capacitor voltage has reached zero, the counter is inhibited thus the count attained is proportional to the original input voltage.
If the original input voltage was bigger the fixed discharging current would take longer to cause the voltage to reach zero thus count is proportional to input voltage.
Maybe think of it like this - you sample the input voltage onto a capacitor then you disconnect the input voltage and the capacitor remains "storing" the input voltage (sample and hold). Then you start a counter and simultaneously start discharging the capacitor with a fixed amount of current. The time taken to discharge that cap to zero volts is proportional to the original sampled voltage.