The compile error is telling you that Verilog does not allow you to declare a signal using a variable number of bits. You are trying to declare W with a variable number of bits (k is a variable), which is illegal Verilog syntax.
There is nothing wrong with your function. You just can not declare a signal that way. This compiles (but is untested):
module test(input [7:0] inp,
output reg [1:0] out
);
always @(*) begin
out[0] = is9(inp, 0);
out[1] = is9(inp, 1);
end
function reg is9 (input [7:0] a, input sel);
reg [3:0] nib;
nib = (sel) ? a[7:4] : a[3:0];
is9 = (nib == 4'd9);
endfunction
endmodule
Verilog makes it trivial to do what you want without the function:
module test(input [7:0] inp,
output reg [1:0] out
);
always @(*) begin
out[0] = inp[3:0] == 4'd9;
out[1] = inp[7:4] == 4'd9;
end
endmodule