What you are talking about is called a "front panel" for the computer and, starting in the 1960's anyway, would often would look something like this:
Most computers from that era (from IBM such as the IBM 1130, for example, and from DEC like the PDP/8 -- both of which I used) included a front panel as a necessary interface. The above is from one of a variety of PDP-11's that I programmed, for a time when working on the Unix v6 kernel in 1978.
Behind the scenes, there is of course some hardware. You can see some of what that looked like below, taken from an IMSAI 8080 (circa late 1970's):
The LEDs would be driven (indirectly, so as to lighten the load) from various bus lines that were accessible. In modern terms, this would be something like being able to tap into and observe the 288-pin DDR4 DIMM lines on a typical PC motherboard. The idea is similar, anyway. The LEDs might also tap into the CPU, too, being able to examine ALU status bits (like carry) and also the register values. For earlier computers, it wasn't hard since the CPU itself was a bunch of circuitry and not an integrated circuit. So the signals were available and wires could be brought over to pick off the values.
The switches would actually operate the memory bus. So you could set up the switches to an address you wanted to examine or change. Then you'd click a switch to "force" the address into a place where it could operate the bus. Then you'd use the switches again to set up a data value and click another switch to "deposit" the switch settings into the address that had earlier been set up. A shortcut was also allowed where you didn't have to set up an address every time, because the front panel would modify the previous address for you (automatically, or by using an INCREMENT switch and/or a DECREMENT switch) so you could often just "go to" an address once, then add a series of data values one at a time through the front panel.
You could write entire entire programs this way. And I've done that more times than I'd like to admit. But usually, there was a very simple program which wasn't very large that could then automatically read from some other source of data. This was called a "bootstrap" program. It would then, much more quickly, bring in a larger piece of code. It's possible that the short, simple bootstrap was too stupid to read in the entire system so it would instead do only the basics and get something somewhat larger than itself so that this next program could do even more work. And so on until the final program or operating system was finally loaded up and could run. If there were several, then each step would be another "bootstrap" of the "bootstrapping process."
Front panels were also wonderful in another way. You could stop the program at any time and examine memory, registers, status values and so on. And you could single-step through a program, as well, watching these details change. This is a lot like using a debugger program, today, in single-step mode. A difference is that today the operating systems and the programs running under them are more complex and we now have multi-cpu and multi-core systems with several levels of cache memory, branch prediction logic, multiple functional units per core, internal RISC instructions in a re-order buffer, and a retire unit that causes out-of-order execution to appear to be in-order. Not to mention chipset in-bound and out-bound queues and a host of other "complexities." A useful front panel would be ... expensive to make. Even a simple front panel would cost a lot as those switches alone would probably set you back $15 each, or more. Mechanicals cost a lot, these days.
This is similar to the front panel of a computer I soldered together back in early 1975 (it's not exactly the same):
Note the metal bat-handle switches which were used on the above and on the Altair 8800 I soldered up. Unlike the kind-to-finger versions of the PDP-8/e, PDP-11/20, PDP-11/45, PDP-11/70, and so on, they didn't even bother adding the plastics. And as a consequence, you could honestly get finger calluses from using them (like a guitar player might.) It could be a little painful if you were doing a lot of work on the front panel of an Altair 8800.
The Altair 8800 (and the later IMSAI 8080) were designed using the Intel 8080A chip, which is an IC. There were no wires into the CPU. So the front panel had to use "tricks" in order to get that information. It did this by forcing instructions onto the bus and then single-stepped the 8080A in order to get the information it wanted. This made the front panel a little more complex, perhaps, than earlier computers required. But it worked.
