You must add a pull down resistor to your button.
The GPIO pins are sensitive to voltages. When you press the button, the voltage (vs GND) at the GPIO is set to 5V, and the MCU will see this. But when you release the button, what is the voltage at the GPIO? 0V? probabliy not, what forces that voltage to be 0V?
When left open, or floating, the value of the GPIO is undefined. It could be 1, or 0. Or could switch quickly (oscillate). On AT Tiny, when left open under some condition the input stage starts to consume a lot of current... (which is bad for a low power MCU...)
Thus, you could solve this by adding a resistor (called a "pull down") between your GPIO and GND (10k is fine).
Or you may connect your button to GND instead of 5V and enable the internal pull up resistor of the MCU GPIO (I don't know if this particular model offers that feature).