The 'invert' function causes the protocol analyzer to interpret a logic '0' as logic '1' and vice versa. This may be necessary to decode a waveform which is 'inverted' relative to the 'normal' logic levels for that protocol.
RS232 specifies +3~15V for the Start bit and '0' data bits, and -3~15V for '1' data bits and the Stop bit(s). However when translated to and from TTL/CMOS logical levels the signal is usually inverted. In 'TTL serial' the Start bit is low and Stop bit is high, and data bits match positive TTL logic levels (ie. high level = '1', low level = '0').
The analyzer detects the start of a word by looking for the transition between Stop/idle and Start bit. The detected start point will vary depending on the signal polarity and data present, so an 'inverted' signal will show decoded words in different positions and there won't be any obvious relationship between the detected bits.
To transform the signal in your own custom program, all you have to do is interpret high level as logic '0' and low level as logic '1'. You will have to decode the signal yourself, by looking for the beginning of a Start bit (transition from '1' to '0') skipping 1.5 bit periods to read the level in the middle of each data bit, and finally checking for the Stop bit.