0
\$\begingroup\$

The binary raw data from the SAR sensor is processed in fpga and outputted as hex. Now, this data we currently store as ASCII in .txt files. However, it takes lots of space in storing.

Is there an efficient way to store these hex data so that it consumes less space and we can efficiently retrieve it for processing using a python application.

\$\endgroup\$
  • \$\begingroup\$ How much space savings do you need to achieve? 2-3x? 10x? \$\endgroup\$ – anrieff Nov 12 '18 at 10:38
  • \$\begingroup\$ By the way, I doubt your FPGA "outputs in Hex". It probably just sends you raw data, right? The question whether you interpret that in hexadecimal, binary, decimal, to base 254 or not at all is up to you..,. \$\endgroup\$ – Marcus Müller Nov 12 '18 at 10:42
  • \$\begingroup\$ Yes we receive raw data from the fpga and convert it to hex to store it in a text file which is further used for processing . Is there a better way to handle and store it? Even if we get a space saving of 50% , it would be a good achievement... \$\endgroup\$ – Neethu Johnson Nov 13 '18 at 4:54
  • \$\begingroup\$ Write the raw data in a binary file (you will have at least a 16x reduction in size), then use this answer to read it in python (assuming it's floats): stackoverflow.com/questions/6286033/… \$\endgroup\$ – Olivier Sohn Nov 13 '18 at 11:53
3
\$\begingroup\$

Simply don't convert numbers to text.

It's really as simple as that.

Don't use text files.

Simply store the data as-is. You've got the samples in RAM as consecutive 16-bit signed integers? Store that buffer. As 64bit double-precision floating point? Store that buffer.

It's really simple. Every programming language has a fwrite, write, whatever, call to save raw data from RAM into a file. Just don't convert the data to text in between.

Simply don't use a text file to store anything that isn't text. By the way, saving floating point numbers as decimal numbers in text files is just setting yourself up for rounding errors, where you get different results when you read the data from a intermediate save file and when you directly process them.

For an example where this is used, see the GNU Radio FAQ. GNU Radio is a software defined radio framework, and hence, we've often got to deal with laaaaaarge numbers of samples. It also gives an example of how to read the files from Python (it's really just numpy.fromfile, nothing magical).

\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.