Say I have a count signal in a counter VHDL file and want to display this in my simulation output, what would I have to do to my testbench to output such data?
This isn't a VHDL question but a tool question. The details may vary from one simulator to another but typically there's a tree view showing the testbench and all components in it - and components inside them, recursively.
Select the component you're interested in, and its internal signals will appear in the signal list view. Select some or all of those signals and there will be some means (right click?) to add them to the Wave window.
Now re-start and re-run the simulation.
(If that's not detailed enough, look at manuals, tutorials, or tell us which simulator you're using. GHDL- which is command-line based - has a command line option to collect all signals, then you select the ones you want in GTKWave)