Prompt 35 'restart' and you get a warning that component DUT is not bound. That's legal in VHDL.
Your instantiated component name doesn't match the name of the entity you compiled.
Modelsim has verror that returns an expanded definition of what causes the error:
vsim Message # 3473: The specified component has not been explicitly bound and no default binding has been found for it. This means that your VHDL design does not include a configuration specification or component configuration for the specified component instance in order to indicate the entity/architecture to use for that component. Additionally, no entity of the same name as the component, having ports and generics matching those of the component declaration, was found when searching all visible libraries for a default binding. Simulation can occur, but no VHDL code will be executed for the unbound component instance: it is essentially a null instance.
To cure it you can change name of mux or configure your testbench to use mux for Mwidth_by_by_Ninputs_MUX (DUT). The latter is done with a configuration specification:
signal outp: std_logic_vector(8 - 1 downto 0);
-- configuration specification added to map mux to Mwidth_by_Ninputs_MUX
for DUT: Mwidth_by_Ninputs_MUX use entity work.mux; -- ADDED
begin
--connect DUT
DUT: Mwidth_by_Ninputs_MUX port map (...
Prompt 36 and you ran the simulation without the DUT. The missing component can't drive your outputs. You non-specifically note the Y output isn't correct (outp is all 'U's, the default initial value). Your waveform format translates those to 'X's.
Fix those and something else will pop up. The default initial value for SLT is INTEGER'LEFT (maximum negative integer value), out of bounds of a in mux.
The declarations for SEL and SLT are at fault and the error is a run time error in mux for the assignment:
y <= a(sel);
a has a declared range of 0 to 2 ** N -1.
You can constraintconstrain the declaration of SEL:
sel : in integer range 0 to 2 ** n - 1;
And SLT to match:
signal slt: integer range 0 to 2 ** 3 - 1;
and now your simulation will complete:
The configuration specification bound mux for DUT (a Mwidth_by_Ninputs_MUX virtual component).
Fixing the range constraints for the SEL and SLT to remain in bounds allows the simulation to run without an a bounds error.