Setting current limit for A4988 driver using a 5V 28byj-48 stepper
Before you can even begin to select the current limit for your stepper motor you need to understand its configuration and how you can connect it to an A4988 driver.
The 28byj-48 electrical configuration looks like this:

The Stepper is typically driven by a ULN2002, and typically at 5-12V supply voltage. This might help you.
Notice the Red wire!!!! The center tap of each winding is connected together and brought out of the stepper. Only one half of the center tapped windings are ever activated at once. I'll leave it to you to research the ULN2002 drive mechanism.
You want to drive the stepper with an H-Bridge that expects two completely isolated windings. Most common stepper motors look like this:

Most common is the Bipolar Stepper shown above with two completely isolated phase windings.
To run the 28byj-48 you need to open the motor and break the connection between the coil center points. Here is a step by step. If you attempt to run the motor with the Red wire simply disconnected, then there WILL be cross currents between the windings which will impact the accuracy of the steps you achieve (especially if you use micro stepping).
Once you have isolated the center wires you will notice that you are using the two half windings in series (look at the uni-polar diagram to see what is meant). Since each half winding on the 28byj-48 is 50 Ohms, you now have a 100 Ohm winding to drive. If you are driving the stepper motor at Full step, and want to achieve the rated current of 100 mA, then you need at least 10V drive capability. So operation at even the low value for the A4988 (8V) is not going to work.
Since the A4988 controls the current in the stepper motor by switching it on and off rapidly (PWM), you are going to need at least 12-15V to drive the stepper and be able to set the current to 100 mA (or above).
Next you can study the datasheet for the A4988.
In particular read the section on "Internal PWM Current Control" on page 9.
Critical in the calculation is the value of Rs1 and Rs2. Since you have not provided these values it's impossible to tell you (or calculate) the Vref required to set this current.
If you are willing to use a manual procedure with a multimeter to set the current you can try this:
- Set the A4988 to Full step (MSL pins).
- Set a multimeter on current mode in series with one of the bipolar coils.
- Set the Vref pot to minimum current.
- Turn on the unit.
- Increase the Vref pot till you read 100 mA in the coil. Not that here you are setting the average/RMS current, not the peak current which is the purpose of the setting. If you can't reach 100 mA then you can try increasing the supply voltage. Certainly if you decrease the supply voltage you will see the current drop off drastically.
- Turn off and disconnect your multimeter, then set your microstep (MSL) value if you are using it.
- You should be good to go.
You showed the Stepper current as 260 mA in your question, but I am sure that this value is wrong. The 5V 28byj-48 stepper has a 50 Ohm half coil. So this implies a 100 mA current at 5V. You can certainly overcurrent the stepper, but it will get hotter