Compiling code to run from external RAM

Question

I am considering designs for a minimalist game system based on a PIC18F85J5. Part of my design is that games can be loaded from an SD card without reprogramming the chip or flashing the program memory. I chose that chip because it has an external memory interface that will allow me to run code from an external SRAM.

The basic idea is that the internal program memory will contain an interface for browsing the sd card, and once the user selects a program it will copy a hex file from the sd card to the external ram, and then jump execution into the external ram space.

The internal program memory will also have various libraries for graphics, controller input and other various utilities.

I am fairly confident I know how to make the internal firmware parts work fine. The problem is creating programs to run from the external RAM. It doesn't feel the same as targeting a regular pic, and it needs to be aware of the library functions that are available in the internal memory, but not recompile them, only link to them. It also needs to start using addresses just after the 32k of internal flash, not at zero. Is there a good way to compile a program using these types of constraints?

I am using the MPLab IDE, but I am not super familiar with it, or how to do this kind of customization.

Answer 1

You have two separate issues:

1. Building the code for the external RAM address range.

   This is actually very easy. All you have to do is make a linker file that contains only the address ranges you want the code to occupy. Note that you not only need to reserve a particular program memory address range for these external apps, but also some RAM space. This RAM space, like the program memory addresses, needs to be fixed and known. Simply make only those fixed and known address ranges available for use in the linker file. Don't forget to make them NOT available in the base code linker file.

   After the base code loads a new app into external memory, it has to know how to execute it. The easiest thing is probably to have execution start at the first external RAM location. This means your code will need one CODE section at that absolute start address. This contains a GOTO to the right label in the rest of the code, which will all be relocatable.

2. Linking apps to library routines in the base code.

   There is no immediately simple way to do this with the existing Microchip tools, but it's not that bad either.

   A much bigger issue is how you want to deal with base code changes. The simplistic strategy is to build your base code, run a program over the resulting map file to harvest global addresses, then have it write a import file with EQU statements for all the globally defined symbols. This import file would then be included in all app code. There is nothing to link, since the app source code essentially contains the fixed address references to the base code entry points.

   That is easy to do and will work, but consider what happens when you change the base code. Even a minor bug fix could cause all the addresses to move around, and then all existing app code would be no good and have to be rebuilt. If you never plan to provide base code updates without updating all apps, then maybe you can get away with this, but I think it is a bad idea.

   A better way is to have a defined interface area at a chosen fixed known address in the base code. There would be one GOTO for every subroutine that app code can call. These GOTOs would be placed at fixed known addresses, and the external apps would only call to those locations, which would then jump to wherever the subroutine actually ended up in that build of the base code. This costs 2 program memory words per exported subroutine and two extra cycles at run time, but I think is well worth it.

   To do this right you need to automate the process of generating the GOTOs and the resulting export file that external apps will import to get the subroutine (actually GOTO redirector) addresses. You might be able to do with with some clever usage of MPASM macros, but if I were doing this I would definitely use my preprocessor since it can write to a external file at preprocessing time. You can write a preprocessor macro so that each redirector can be defined by a single line of source code. The macro does all the nasty stuff under the hood, which is to generate the GOTO, the external reference to the actual target routine, and add the appropriate line to the export file with the known constant address of that routine, all with appropriate names. Perhaps the macro just makes a bunch of preprocessor variables with regular names (kindof like a run-time expandable array), and then the export file is written once after all the macro calls. One of the many things my preprocessor can do that MPASM macros can't is to do string manipulation to build new symbol names from other names.

   My preprocessor and a bunch of other related stuff is available for free at www.embedinc.com/pic/dload.htm.

Answer 2

Option 1: Interpreted Languages

This doesn't directly answer the question (which is an excellent question, BTW, and I hope to learn from an answer which does address it directly), but it's very common when doing projects which can load external programs to write the external programs in an interpreted language. If resources are tight (which they will be on this processor, have you thought about using a PIC32 or small ARM processor for this?), it's common to restrict the language to a subset of the full specification. Even further down the chain are domain-specific languages that only do a few things.

For example, the elua project is an example of a low-resource (64 kB RAM) interpreted language. You can squeeze this down to 32k of RAM if you remove some features (Note: It won't work on your current processor, which is an 8-bit architecture. Using external RAM will probably be too slow for graphics). It provides a fast, flexible language in which new users could easily program games if you provide a minimal API. There is plenty of documentation available for the language online. There are other languages (like Forth and Basic) which you could use in a similar manner, but I think that Lua is the best option at the moment.

In a similar vein, you could create your own domain-specific language. You would have to provide a more full-fledged API and external documentation, but if the games were all similar then this wouldn't be too difficult.

In any case, the PIC18 is probably not the processor I'd use for something which involves custom programming/scripting and graphics. You may be familiar with this class of processors, but I'd suggest that this would be a good time to use something with a display driver and more memory.

Option 2: Just reprogram the whole thing

If, however, you're already planning on programming all the games yourself in C, then don't bother with loading just the game logic from the SD card. You have just 32kB of Flash to reprogram, and could easily get a 4 GB microSD card for this. (Note: larger cards are often SDHC, which is harder to interface with). Assuming that you use every last byte of your 32 kB, that leaves room on the SD card for 131,072 copies of your firmware with whatever game logic you need.

There are plenty of appnotes for writing bootloaders for PICs, like AN851. You'd need to design your bootloader to occupy a specific region of memory (probably the top of the memory region, you would specify this in the linker), and specify that the full firmware projects do not reach this region. The appnote spells this out in more detail. Just replace "Boot section of the PIC18F452" with "Boot section I specify in the linker" and it will all make sense.

Then, your bootloader just needs to allow the user to select a program to run from the SD card, and copy the whole thing over. A UI could be that the user has to hold down a push button to enter the selection mode. Ordinarily, the bootloader would just check the status of this button on reset, and, if it's not being held down, boot into the game. If it's held down, it would need to allow the user to choose a file on the SD card, copy the program over, and continue to boot into the [new] game.

This is my current recommendation.

Option 3: Deep magic involving storing only part of the hex file

The trouble with your envisioned mechanism is that the processor doesn't deal with APIs and function calls, it deals with numbers - addresses to which the instruction pointer can jump and expect there to be code which executes a function call according to an API spec. If you try to compile just a part of the program, the linker won't know what to do when you call check_button_status() or toggle_led(). You may know that those functions exist in the hex file on the processor, but it needs to know precisely which address they reside at.

The linker already breaks your code into multiple sections; you could theoretically break this into additional sections with some -section and #pragma incantations. I have never done this, and don't know how. Until the above two methods fail me (or someone posts an awesome answer here), I probably won't learn this mechanism, and so I can't teach it to you.