I've met many projects in which an AVR microcontroller uses with a bootloader (such as the Arduino), but I don't understand the concept very well.
How can I make a bootloader (for any microcontroller)?
After writing my bootloader, how it is programmed to the microcontroller (like any .hex program burnt on the flash rom of the AVR, or some other method)?
A bootloader is a program that runs in the microcontroller to be programmed. It receives new program information externally via some communication means and writes that information to the program memory of the processor.
This is in contrast with the normal way of getting the program into the microcontroller, which is via special hardware built into the micro for that purpose. On PICs, this is a SPI-like interface. If I remember right, AVRs use Jtag, or at least some of them do. Either way, this requires some external hardware that wiggles the programming pins just right to write the information into the program memory. The HEX file describing the program memory contents originates on a general purpose computer, so this hardware connects to the computer on one side and the special programming pins of the micro on the other. My company makes PIC programmers among other things as a sideline, so I am quite familiar with this process on PICs.
The important point of external programming via specialized hardware is that it works regardless of the existing contents of program memory. Microcontrollers start out with program memory erased or in a unknown state, so external programming is the only means to get the first program into a micro.
If you are sure about the program you want to load into your product and your volumes are high enough, you can have the manufacturer or a distributor program chips for you. The chip gets soldered to the board like any other chip, and the unit is ready to go. This can be appropriate for something like a toy, for example. Once the firmware is done, it's pretty much done, and it will be produced in large volumes.
If your volumes are lower, or more importantly, you expect ongoing firmware development and bug fixes, you don't want to buy pre-programmed chips. In this case blank chips are mounted on the board and the firmware has to get loaded onto the chip as part of the production process. In that case the hardware programming lines have to be made available somehow. This can be via a explicit connector, or pogo pin pads if you're willing to create a production test fixture. Often such products have to be tested and maybe calibrated anyway, so the additional cost of writing the program to the processor is usually minimal. Sometimes when small processors are used a special production test firmware is first loaded into the processor. This is used to facilitate testing and calibrating the unit, then the real firmware is loaded after the hardware is known to be good. In this case there are some circuit design considerations to allow access to the programming lines sufficiently for the programming process to work but to also not inconvenience the circuit too much. For more details on this, see my in-circuit programming writeup.
So far so good, and no bootloader is needed. However, consider a product with relatively complex firmware that you want field upgradable or even allow the end customer to upgrade. You can't expect the end customer to have a programmer gadget, or know how to use one properly even if you provided one. Actually one of my customers does this. If you buy their special field customizing option, you get one of my programmers with the product.
However, in most cases you just want the customer to run a program on a PC and have the firmware magically updated. This is where a bootloader comes in, especially if your product already has a communications port that can easily interface with a PC, like USB, RS-232, or ethernet. The customer runs a PC program which talks to the bootloader already in the micro. This sends the new binary to the bootloader, which writes it to program memory and then causes the new code to be run.
Sounds simple, but it's not, at least not if you want this process to be robust. What if a communication error happens and the new firmware is corrupt by the time it arrives at the bootloader? What if power gets interrupted during the boot process? What if the bootloader has a bug and craps on itself?
A simplistic scenario is that the bootloader always runs from reset. It tries to communicate with the host. If the host responds, then it either tells the bootloader it has nothing new, or sends it new code. As the new code arrives, the old code is overwritten. You always include a checksum with uploaded code, so the bootloader can tell if the new app is intact. If not, it stays in the bootloader constantly requesting a upload until something with a valid checksum gets loaded into memory. This might be acceptable for a device that is always connected and possibly where a background task is run on the host that responds to bootloader requests. This scheme is no good for units that are largely autonomous and only occasionally connect to a host computer.
Usually the simple bootloader as described above is not acceptable since there is no fail safe. If a new app image is not received intact, you want the device to continue on running the old image, not to be dead until a successful upload is performed. For this reason, usually there are actually two special modules in the firmware, a uploader and a bootloader. The uploader is part of the main app. As part of regular communications with the host, a new app image can be uploaded. This requires separate memory from the main app image, like a external EEPROM or use a larger processor so half the program memory space can be allocated to storing the new app image. The uploader just writes the received new app image somewhere, but does not run it. When the processor is reset, which could happen on command from the host after a upload, the bootloader runs. This is now a totally self-contained program that does not need external communication capability. It compares the current and uploaded app versions, checks their checksums, and copies the new image onto the app area if the versions differ and the new image checksum checks. If the new image is corrupt, it simply runs the old app as before.
I've done a lot of bootloaders, and no two are the same. There is no general purpose bootloader, despite what some of the microcontroller companies want you to believe. Every device has its own requirements and special circumstances in dealing with the host. Here are just some of the bootloader and sometimes uploader configurations I've used:
Since the bootloader was itself a complicated piece of code containing a full TCP network stack, it had to be field upgradeable too. They way we did that was to have the upload server feed it a special app whose only purpose was to overwrite the bootloader once it got executed, then reset the machine so that the new bootloader would run, which would cause the upload server to send the latest main app image. Technically a power glitch during the few milliseconds it took the special app to copy a new image over the bootloader would be a unrecoverable failure. In practise this never happened. We were OK with the very unlikely chance of that since these devices were parts of large installations where there already were people who would do maintainance on the system, which occasionally meant replacing the embedded devices for other reasons anyway.
Hopefully you can see that there are a number of other possibilities, each with its own tradeoffs of risk, speed, cost, ease of use, downtime, etc.
What is the concept of the bootloader?
Picture this scenario: You have a fair amount of storage on your microcontroller - enough to store more than 2-3 programs or applications that are independent of each other. Suppose that when you boot your device, you may want to be able to choose which one to run. So what would you need to support this? You would need a starting program which then allows you to choose between the others at boot-time.
How it works?
A bootloader is that program - it is the first thing to run and can load other applications into specific places in memory (either persistent like FLASH, or volatile like RAM) and then jumps to that desired program where it will then take over execution from there.
How to make an avr bootloader (or for any microcontroller)?
I have never made a bootloader but this is how I think I would go about doing it: start writing a firmware program just as you normally would - but make sure it is placed in an area such that it is always the first thing to run when the device boots. Off the top of my head, some of the functionality I would want out of this small program: ability to upload a new program to an available spot in memory, erase a previously uploaded program, choose which program to run (if there is more than one), and have some kind of storage data structure (grow-able jump table?) to be able to remember where the other programs are and jump to them. Interaction could be done over UART where it can present you a very simple terminal menu and ability to upload firmware over that same channel.
How it is programmed to the microcontroller (like any .hex program burnt on the flash rom of the avr, or some other method)?
If it is a completely blank chip with no existing bootloader that can update itself, then you would need to burn to FLASH much like you described using whatever technique is required to accomplish that (ICSP in the case of AVR).
This is by no means comprehensive of what "bootloaders" are. Depending on what you want out of one or the system they're designed for, you can design one to upload stuff to a specified location in RAM instead of FLASH, and begin execution at any arbitrary memory location. Or maybe you want one that is capable of choosing which operating system to load when your PC boots (see grub for example). Bootloaders for 8-bit microcontrollers tend to be very simple.
A note about Arduino: This bootloader only manages one program AFAIK, it also takes over the serial port to manage firmware uploading and other stuff.
The concept of a "boot" loader is similar to the concept of "priming" a pump. In other words, you need "something" that loads a program to a given address and then start executing the program at that given address. That something, is the boot loader. In the simplest case, the boot loader "appears" at the CPU's designated starting address (zero, most likely), loads the program into the required memory segment, transfers control to it, and "disappears." The appearance and disappearance are controlled by "external" hardware. One possible implementation would be by using a ROM that is activated via a "hardware" reset, and deactivated with a "software" reset. The loader in the ROM can be as simple or as complex as required, and needs to be written in the binary form that the particular CPU understands. If the address space used by the ROM is not needed, ROM deactivation would not be required. Obviously, EEPROM, ePROM, flash PROM, etc. could be used instead of the ROM.
