Unstable access to external MRAM when using Nucleo STM32 H7 board

Question

I have an asynchronous MRAM device, AS3004316 4Mb 35ns (https://pt.mouser.com/datasheet/2/1122/1Mb_32Mb_Parallel_x16_MRAM_2-1949428.pdf), connected to an STM32h743 nucleo board through the Flexible Memory Controller (FMC).

Clock speed of the board is set to 480MHz, with the FMC clock being set to 240MHz. My base timing settings are {Address Setup Time: 1, Data Setup Time: 9, Bus turn around time: 3}. MRAM is mapped to address 0xc0000000.

I ran 2 tests. The first writes the same 16-bit value to all even addresses in the MRAM memory region and then reads those addresses back, and checks that the 16-bit value was correctly stored. No errors are found in this test. The second test chooses random mram addresses, and random values. It writes each of the random values to the random addresses, and then checks to see if the value was correctly stored. Out of sequences of 1000 write operations, between 0 and 10 addresses return the wrong values. Increasing address setup, data setup, and bus turnaround times does not seem to help. I recorded some of the address/value pairs which resulted in error. Setting just those addresses to those values which previously were not correctly stored, outside of the test which executes multiple writes, shows no errors.

I also ran some tests with hashmaps implemented over the MRAM, where changing the hash function, or increasing the timing settings (address setup, data setup, ...) can be the difference between the hashmap working correctly or not. When running on STM32 memory, they always work correctly. I have tried two different MRAM boards, with similar problems.

What could cause this erratic behaviour? Could it be, for example, lack of decoupling capacitors? The MRAM interfaces with the Nucleo board through a PCB I have designed, which has one 100nF decoupling capacitor next to each MRAM Vcc Pin. Do I need more? Or does the problem come from somewhere else?

EDIT 1: Added PCB images and note (deleted, not relevant to the question)

EDIT 2: Added test code

int main(void)
{
  HAL_Init();

  /* Configure the system clock */
  SystemClock_Config();

  /* Initialize all configured peripherals */
  MX_GPIO_Init();
  MX_ETH_Init();
  MX_FMC_Init();
  MX_CRC_Init();


  HAL_Delay(1000);
  srand(23786);
  while (1)
  {
    //SEQUENTIAL TEST
    //test_memory_consistency();
    
    //RANDOM TEST
    test_memory_consistency_random();
    
    //Test one of the adress/value pairs which show as error in the test
    uint32_t address = 0x6000c360;
    uint16_t value = 32079;

    *(uint16_t *) address = value;
    
    //Both values match when tested here
    printf("Value is: %u Stored is: %u\n", value, *(uint16_t *) address);

    HAL_Delay(30*1000);
  }
}

//RANDOM TEST
void test_memory_consistency_random(){
    uint16_t random_array_size = 1000;
    uint16_t random_16bits[random_array_size];
    uint32_t random_mram_addresses [random_array_size];
    uint32_t errors = 0;

    generate_16bit_randoms(random_16bits,random_array_size);
    generate_random_mram_addresses(random_mram_addresses, random_array_size);
    for(int i = 0; i < random_array_size; i++){
        * (uint16_t *) random_mram_addresses[i] = random_16bits[i];
    }

    for(int i = 0; i < random_array_size; i++){
        if(* (uint16_t *) random_mram_addresses[i] != random_16bits[i]){
            errors++;
            printf("Memory test error with value %d, at address: %lx\n Value stored: %u\n ", random_16bits[i], random_mram_addresses[i], * (uint16_t *) random_mram_addresses[i]);
            if(random_mram_addresses[i] < MRAM_BANK_ADDR || random_mram_addresses[i] > MRAM_BANK_ADDR_LIMIT_4Mb){
                printf("Address was invalid\n");

            } else{
                //Addresses are confirmed to be valid
                printf("Address was valid\n");
            }
            if(* (uint16_t *) random_mram_addresses[i] != random_16bits[i]){
                //Second test shows same result
                printf("Second test also came out wrong\n");
            }
        }
    }
    //Each run shows around 1-3 errors, even when slowing down system clock.
    printf("Random-write test errors: %ld\n",errors);

}

//SEQUENTIAL TEST. No errors.
int test_memory_consistency (){
    int memory_consistent = 1;
    uint16_t constant;
    uint16_t max = ~0; //Max int for 16 bits
    uint32_t errors = 0;
    uint8_t any_error = 0;

    //NOTE: Tested up to number 26800
    //Constant test
    for(constant = 0;constant != max ; constant++){
        errors = test_write_read_consistency_constant_immediate((void *) & constant, 16);
        if(errors > 0){
            printf("Memory test errors with constant %d, #errors: %ld\n", constant, errors);
            any_error = 1;
        }
        if((constant % 100) == 0){
            printf("Tested %d different constants\n",constant);
        }
    }

    //Test last one
    errors = test_write_read_consistency_constant_immediate((void *) & constant, 16);
    if(errors > 0){
        printf("Memory test errors with constant %d, #errors: %ld\n", constant, errors);
        any_error = 1;
    }


    if(!any_error){
        printf("Memory showing no errors\n");
        memory_consistent = -1;
    }

    return memory_consistent;
}

//SEQUENTIAL TEST
//Tested with every number up to 26800, with 16-bit sized operations
//No errors detected
uint32_t test_write_read_consistency_constant_immediate(void * constant, uint8_t bit_size){
    uint8_t byte_size = bit_size / 8;
    uint32_t errors = 0;

    //Write, read and verify
    for(uint32_t addr = MRAM_BANK_ADDR; addr < MRAM_BANK_ADDR_LIMIT_4Mb; addr+=byte_size){
        switch(bit_size){
            case 8:
                * (uint8_t *) addr = * (uint8_t *) constant;
                if ((* (uint8_t *) addr) != * (uint8_t *) constant){
                    errors++;
                }
                break;
            case 16:
                * (uint16_t *) addr = * (uint16_t *) constant;
                if((* (uint16_t *) addr) != (* (uint16_t *) constant)){
                    printf("Error at address %lu, value at MRAM: %u, constant expected: %u\n", addr, (* (uint16_t *) addr), * (uint16_t *) constant);
                    errors++;
                }
                break;
            case 32:
                * (uint32_t *) addr = * (uint32_t *) constant;
                if((* (uint32_t *) addr) != * (uint32_t *) constant){
                    errors++;
                }
                break;
            case 64:
                * (uint64_t *) addr = * (uint64_t *) constant;
                if((* (uint64_t *) addr) != * (uint64_t *) constant){
                    errors++;
                }
                break;
            default:
                printf("Unsupported bit_size for write");
                return -1;
        }
    }
    return errors;

}

/* FMC initialization function , generated by STM32 IDE*/
static void MX_FMC_Init(void)
{

  FMC_NORSRAM_TimingTypeDef Timing = {0};
  FMC_NORSRAM_TimingTypeDef ExtTiming = {0};

  /** Perform the SRAM1 memory initialization sequence
  */
  hsram1.Instance = FMC_NORSRAM_DEVICE;
  hsram1.Extended = FMC_NORSRAM_EXTENDED_DEVICE;
  /* hsram1.Init */
  hsram1.Init.NSBank = FMC_NORSRAM_BANK1;
  hsram1.Init.DataAddressMux = FMC_DATA_ADDRESS_MUX_DISABLE;
  hsram1.Init.MemoryType = FMC_MEMORY_TYPE_SRAM;
  hsram1.Init.MemoryDataWidth = FMC_NORSRAM_MEM_BUS_WIDTH_16;
  hsram1.Init.BurstAccessMode = FMC_BURST_ACCESS_MODE_DISABLE;
  hsram1.Init.WaitSignalPolarity = FMC_WAIT_SIGNAL_POLARITY_LOW;
  hsram1.Init.WaitSignalActive = FMC_WAIT_TIMING_BEFORE_WS;
  hsram1.Init.WriteOperation = FMC_WRITE_OPERATION_ENABLE;
  hsram1.Init.WaitSignal = FMC_WAIT_SIGNAL_DISABLE;
  hsram1.Init.ExtendedMode = FMC_EXTENDED_MODE_ENABLE;
  hsram1.Init.AsynchronousWait = FMC_ASYNCHRONOUS_WAIT_DISABLE;
  hsram1.Init.WriteBurst = FMC_WRITE_BURST_DISABLE;
  hsram1.Init.ContinuousClock = FMC_CONTINUOUS_CLOCK_SYNC_ONLY;
  hsram1.Init.WriteFifo = FMC_WRITE_FIFO_DISABLE;
  hsram1.Init.PageSize = FMC_PAGE_SIZE_NONE;
  /* Timing */
  Timing.AddressSetupTime = 1;
  Timing.AddressHoldTime = 15;
  Timing.DataSetupTime = 10;
  Timing.BusTurnAroundDuration = 4;
  Timing.CLKDivision = 16;
  Timing.DataLatency = 17;
  Timing.AccessMode = FMC_ACCESS_MODE_A;
  /* ExtTiming */
  ExtTiming.AddressSetupTime = 1;
  ExtTiming.AddressHoldTime = 15;
  ExtTiming.DataSetupTime = 10;
  ExtTiming.BusTurnAroundDuration = 4;
  ExtTiming.CLKDivision = 16;
  ExtTiming.DataLatency = 17;
  ExtTiming.AccessMode = FMC_ACCESS_MODE_A;

  if (HAL_SRAM_Init(&hsram1, &Timing, &ExtTiming) != HAL_OK)
  {
    Error_Handler( );
  }
}

EDIT 3: Added function that generates random MRAM addresses

void generate_random_mram_addresses(uint32_t * random_mram_addresses, uint16_t nr_elements){
    //srand(time(NULL));
    uint32_t random;
    uint32_t range = (MRAM_BANK_ADDR_LIMIT_4Mb - 2) - MRAM_BANK_ADDR;

    for(int i = 0; i < nr_elements;i++){
        random = rand();
        random_mram_addresses[i] = MRAM_BANK_ADDR + (random % range);
        random_mram_addresses[i] -= (random_mram_addresses[i] % 2);
    
        if(random_mram_addresses[i] < MRAM_BANK_ADDR || random_mram_addresses[i] > (MRAM_BANK_ADDR_LIMIT_4Mb - 2)){
            printf("Generated address that does not sit on MRAM\n");
            HAL_Delay(30*1000);
        }
    }
}
```

Answer 1

A read or write operation can take 30mA of current. That is going to be difficult to source through 10nH of inductance that the pins and board create, so you will need caps. You need a decoupling cap, and probably more than just a 0.1uf. I would do a 0.1uf and 1uf (or 10uf) in parallel on VCC. You also need a continuous ground plane. This can be checked also by placing a scope probe on ground and VCC of the part and measuring the voltage. The voltage ripple should not dip below ~10% of the value of VCC.

Check the rise times and transitions on the signals for read and write operations.

Get a logic analyzer and make sure you are following the following timing. The logic analyzer will need to be at least 500MHz. Make sure you put the ground near the ground of the MRAM.

Source: https://pt.mouser.com/datasheet/2/1122/1Mb_32Mb_Parallel_x16_MRAM_2-1949428.pdf

Answer 2

Embarassing to admit, but the problem, turns out, was in the "generate_random_mram_addresses" function which did not account for the possibility of duplicate addresses. Hence, the same address sometimes got assigned two different values. Later, when checking the value stored in each of the generated addresses, only the most recent write would be observable, and thus the other values assigned to the same address would show up as errors.

To solve the problem, the following lines were added:

if(hasDuplicate(random_mram_addresses, random_mram_addresses[i] ,i - 1)){
    i--;
}

So, the hardware design was not the problem in this case, but rather the testing function implementation.

For the case of the hashmap implementations I expect it to be a separate issue, which I deem unrelated to the hardware design for now. If it turns out that the hardware was the problem in that case, I will update my answer.