Suppose I want to make a static library (.a file), which contains an ISR. That library should be used by many AVR MCUs, but each device has different ISR vector name, so it cannot be defined in .c file. Is it correct according to C conventions to define the ISR in the header file?
Your vector table will always be project-specific. The correct approach is to define the ISR inside the .c file of the relevant driver handling the hardware that the ISR belongs to.
If you need to expose the ISR name to the file containing the vector table, you should add a function declaration of the ISR in the .h file of the driver.
For example, if you are writing code that uses a UART peripheral of a specific MCU - lets call it "AVR123", then you should create a driver named something like "avr123_uart.h" + "avr123_uart.c". The ISR will be located in avr123_uart.c and all code communicating with the ISR will be there too.
If you need portable or platform-independent code, you can create a hardware abstraction layer (HAL) on top of the driver. Meaning you'll have a file "uart.h" which declares some functions uart_init, uart_read, uart_write and so on. These functions are then implemented in avr123_uart.c. We may say that "avr123_uart" inherits "uart.h" and that "uart.h" is an abstract base class.
The application call only includes uart.h but links the MCU-specific driver. That way you don't have to change a thing in the application code if changing hardware. Simply link the relevant driver.
Under no circumstances should you "splatter" ISR implementation all over the code, or place them in files that have nothing to do with the given hardware. No other code than the driver where the ISR resides should communicate with it.
My answer doesn't answer the actual question but proposes a different solution.
I wouldn't try to include the link to the actual ISR into the library.
Instead write code which is supposed to be called by the actual ISR and let the people using your library handle the implementation of the ISR.
This could also mean, that your library is no longer device dependent, as you offload those parts to the outside of the library.
You must document the interface to your library very well, so people know what your function requires to work correctly - if you have any part where you think "Oh that's just logical" - write it down. You're so involved with the library that you don't realize how different it might be for a "simple" user.