So if I am not mistaken, you are only asking for the bit of plastic that you push and nothing else, none of the contacts or anything?
If that is the case, I've done some digging and I have found that they are called 'caps', push button caps to be more precise, found a whole load on RS: Take a look here.
But I am sure that there are plenty of other options if you look around, I typed in 'tactile switch accessories' to come up with this among many other things before I actually found what I think you are looking for!
Would probably be easier just to buy the push button, actuator, tactile switch, whatever you want to call it, rather than make one but each to there own! Hope this helps.