If the buffer is stable with a cable attached (including any worst-case load, short or open), then it could be done without series resistor, yes.
Note that this assumes the buffer is "0 ohms" at all frequencies of interest. Which is rarely the case, and perhaps you will find equalization or termination necessary after all.
An unterminated source requires a terminated load, and this is called a singly-terminated transmission line. Without a termination, signals reflect back and forth along the line, drawing large currents from the buffer (or developing large voltages at a CC buffer), and perhaps causing unexpected damage, or emissions.
The most common alternative is a doubly terminated line, where a CV source has added series resistance, or a CC source has shunt resistance, or the source (by nature, or through clever design) happens to have some internal resistance inbetween, in combination with external resistors as needed. In any case, the source resistance is made to match the line's characteristic impedance, and this terminates the line regardless if it's open or short, or anywhere inbetween, at the far end. The major downside is, the voltage or current gain is halved.
On another topic, a series resistor also provides a tiny amount of protection against direct ESD strike or cable discharge events (CDEs). Bandwidth allowing, this resistor can be used in combination with ESD clamping diodes or other protective devices to greatly enhance the buffer's robustness.
(As example: the most dramatic case I've seen in equipment was probably gas discharge tubes (GDTs) used with 22Ω carbon composition resistors, to protect three channels of a ~100MHz video amplifier from potential arc discharge of a cathode-ray tube (CRT). Being that the tube is powered by ~30kV, and has some ~nF behind that (2nd anode capacitance to aquadag), a direct strike can be incredibly destructive. It's rare enough you see 30kV ESD (HBM (human body model) or similar) ratings, but this is more like 30kV MM (machine model), far more severe. Carbon comp resistors are noted for high pulse tolerance, hence their choice here.)
Regarding trace impedance: note that it only needs to be matched, within some tolerance, over some length given by the maximum frequency of interest, and maximum tolerable impedance error or SWR. If frequencies are low relative to the trace or pin lengths, it doesn't much matter. If frequencies are very high, even pin and via impedances can matter (e.g., the latter particularly in the 10s of ps range).
Thus, if you place the buffer and connector adjacent, trace length is minimized, and the sensitivity to its impedance is minimized, for a given bandwidth and tolerance. Conversely, if they must be some distance apart, more attention will need to be paid to the impedance (and other aspects, like losses, or dispersion, if applicable).
