1
\$\begingroup\$

I would like to know how to do the profiling of an image acquisition and storing pipeline on an FPGA based system, capturing images from a CMOS image sensor through LVDS interface, do some basic image processing and then storing the data onto two SSDs.

I'm talking about 100 fps, 11.94 MegaPixels, 16 bit per pixel, so the bandwidth is quite huge: it's not about the classic cameras we've used to.

\$\endgroup\$
  • 1
    \$\begingroup\$ What do you mean by "profiling"? \$\endgroup\$ – Tom Carpenter May 30 '16 at 23:43
  • \$\begingroup\$ Understand how much resources I need for a certain task, so I can know which FPGA SoC to choose. \$\endgroup\$ – user6401990 May 31 '16 at 0:28
  • 3
    \$\begingroup\$ I see where you are coming from, though profiling isn't really the right term here. FPGAs aren't programmed in the same sense as a computer is. You describe what hardware is required using code, then synthesize it using the toolchain from the specific vendor whose FPGA you are targeting, and that will tell you the resource utilisation of the design. FPGA designs don't change dynamically, so you don't really profile it. \$\endgroup\$ – Tom Carpenter May 31 '16 at 0:52
  • 2
    \$\begingroup\$ "some basic image processing" is a remarkably opaque phrase. \$\endgroup\$ – WhatRoughBeast May 31 '16 at 2:49
  • 1
    \$\begingroup\$ I'm sorry. I'm talking about dark frame subtraction, FPN subtraction, and maybe a basic noise processing. \$\endgroup\$ – user6401990 May 31 '16 at 12:30
3
\$\begingroup\$

The data bandwidth you mentioned is certainly part of the calculation, but only the beginning: the FPGA and the camera module need a compatible interface that can reach the required speed.

Whether your processing pipeline can be realized depends very much on your definition of "basic image processing". Ideally your algorithm is parallelizable so you can create multiple instances, and optimized for FPGA resource usage to avoid running out of limited resources like multipliers.

Resource usage on FPGAs is not always linear, so five copies of the same logic may use ten times as many LUTs as four copies, simply because you've run out of some "special" blocks and the fifth instance tries to emulate it, or because an instance needs to be wrapped around a special block.

Speaking in the abstract: you can always compile and simulate your design with the FPGA vendor's toolchain, even without the actual hardware. If the constraints are properly specified and synthesis succeeds without showing timing errors, then it should also run -- there is no dynamic reallocation of resources that would make behaviour unpredictable unless you explicitly add that (which may be necessary, e.g. to share multiplier units).

Interpreting errors from FPGA compilations and optimizing designs are rather complex topics though, which entire books have been written about. The compiler reporting a lack of resources could mean that you either need a larger FPGA with more tables, or you need to rewrite the algorithm, or both.

\$\endgroup\$
0
\$\begingroup\$

I see where you are coming from, though profiling isn't really the right term here. FPGAs aren't programmed in the same sense as a computer is. You describe what hardware is required using code, then synthesize it using the toolchain from the specific vendor whose FPGA you are targeting. You then get a synthesis report that will tell you the resource utilisation of the design. FPGA hardware doesn't change dynamically - that is to say, you know the exact resource usage at compile time and so you don't really "profile" it.

Furthermore, do you mean FPGA, or FPGA SoC - the two are different. Some FPGAs have an embedded hard-processor (SoC). Others don't. If you are planning on trying to do this in software on an embedded (soft or hard) processor in the FPGA vs. dedicated logic, the answer is very different.

To add to Simons answer, it is virtually impossible to say what resources any given design will use simply because it depends not only on the design, but also many other factors such as the specific family if FPGA you are using, what portion of the FPGA you are using, and the clock speeds you are running at. In fact you can change one line of code which is seemingly inconsequential, and when you then recompile the entire arrangement of the design and resource utilisation can change dramatically due in part to synthesis optimisations.

For FPGA designs, generally what you do is start designing the logic and algorithms that you plan to use first. Pick a vendor (e.g. Altera, Xilinx, etc.) and start doing test compiles of parts of your design. To begin with you can pick any family of FPGA, it's just to get an idea of how much in the way of resources are required on various different FPGA families (you can try more than one).

Once you have some idea of what is required, you can then look at special function requirements. You will need LVDS SERDES blocks and PLLs for your camera interface, so look at what speeds each family supports. Also if you are doing lots of multiplication and DSP stuff which it seems like you are, you want to look at families which have an abundance of DSP resources (hardware multipliers, etc.).

Once you have an idea of which family you are going to go for, start designing your code further around that family. Do more test compiles with larger and larger chunks of your design. This will give you a rough estimate of the resource requirements you will need in your final design, and you can then settle on a specific FPGA - the larger they are, the more expensive the are, so you don't really want to be going with one which has hundreds of times more resources than you require, but equally you don't want to go for one too small for the design to fit.

Ideally you want to aim for an FPGA that your design will use about 70% of or less. The resource usage goes up as you use more and more of the design due to logic duplication to meet timing and also routing congestion (too much stuff trying to fit into too small of a space). Why 70%? well that's just a rough guess - sometimes even if you are only using a tiny fraction of the resources, you can run into problems if all of those resources are trying to be in the same place in the design (due to timing requirements, etc.) resulting in much higher peak utilisation.

\$\endgroup\$
  • \$\begingroup\$ So I have to start writing the code using the IDE and then watch the resources needed? \$\endgroup\$ – user6401990 May 31 '16 at 12:36
  • \$\begingroup\$ @user6401990 you don't so much watch the resources needed, you compile the design and it will tell you what is required. Think of it like when you build a program for a microcontroller - the compiler will tell you the flash usage and static memory requirements. The synthesis tool will tell you the LUT usage, DSP block usage (if any), RAM block usage (if any), pin usage, transceiver usage, etc. \$\endgroup\$ – Tom Carpenter May 31 '16 at 12:41
  • \$\begingroup\$ that makes sense for me, thank you! However since I'm starting right now with FPGAs I must choose a Vendor and a Family to invest money and time in it (let's say: ALTERA, Cyclone V SoC) so I can buy a dev board and start coding withe the vendor's specific IDE. \$\endgroup\$ – user6401990 May 31 '16 at 12:45
  • \$\begingroup\$ @user6401990 the beauty of FPGAs is you don't need to by hardware immediately. You can do an awful lot of the design with simulation before you even need to consider what FPGA family you are going to use. For the Cyclone series for example Quartus (Altera toolchain) is free. So you can set that up and start developing code. Once you have some idea of what resources you will need, you can then look at actual hardware. \$\endgroup\$ – Tom Carpenter May 31 '16 at 12:50
0
\$\begingroup\$

I worked in a company who wanted to design a 4K camera at 300fps using a low cost FPGA!

The project was unrealistic because of many issues that they did not count into their product planning.

The difficulty of designing a camera, specially a 'high-end' is mainly the memory bandwith and the amount of memory you have acess to.

If you want to capture a 12MP image sensor at 100fps and 16bits, you will need 12e6x100x16= 1.92e10 bits/sec for the interface toward the camera.

The problem starts after that, you have to choise to either capture the data and process it as it is, or convert it to RGB which will tripple the amount of data you have plus it needs a large number of math operation plus you must keep at least 3 lines of data in a memory to be able to handle it.

The next step depends on what you want to do with the data, if you want to perform Dark Current filter or any other filtering, which needs a significant number of memory and math operations.

The BIG problem is when you want to store the data. You can chose to compress the image, or save it in RAW format. Both of these have benefits and drawbacks, resource-bandwidth is always a difficult choice one have to make when designing such high performance product.

I don't this it is easy to answer your question, but it is possible to design such a product using a high-end FPGA, but the price will be pretty high (IMO)

\$\endgroup\$
  • \$\begingroup\$ I think that the mentioned company intended to compress the data, and I don't think they wanted 16bit or even 12bit or 10bit data. I'm talking about 16bit undemosaiced data directly stored onto two SSDs. My main problem is the bandwith (from camera to FPGA and from it to storage). I want to keep everything as simple as possible in order to keep costs down as much as I can. Could you give me an idea of your experience? The requirement was 4k 300fps data in? 8bit undemosaiced in? LVDS interface? How about compression and storage? Which family of FPGA wasn't enough? \$\endgroup\$ – user6401990 May 31 '16 at 12:39
  • \$\begingroup\$ Well, most of the high end FPGAs are using 300 or 600MBps LVDS lines (some support higher speed, but I am not familiar with them). any modern Xilinx FPGA family can handle the 12Mp/100fps data. For SSD, are you talking about SATA 3.2? Becasue with SATA 3, the maximum theoretical throuput is 1.2e10bps for 2 disk! How can you save 1.92e10bps data into those 2 disks without compression? \$\endgroup\$ – FarhadA May 31 '16 at 13:55
  • \$\begingroup\$ oh... the last time I did calculation was for 16bit, 3.69 MP, 100 fps... I must reconsider how to solve it, maybe increasing SSDs (I don't care if the recording time is divided by half) or trying to implement wavelet compression \$\endgroup\$ – user6401990 May 31 '16 at 18:20
  • \$\begingroup\$ Ah, ok, that is a WHOLE different story. it is a HUGE difference between 3.69Mp and 11.94Mp! You have to remember that you need to have one SATA controller for each SATA device, and you need to have 6Gbps transceivers for the SATA channeles, that will be one of the major facts in chosing your FPGA. What that spec, you can easily do it with a Kintex 7 device and use the Kintex 7 evalution board: xilinx.com/products/boards-and-kits/ek-k7-kc705-g.html then you have to look for an HMC board with multiple SATA connectors on it. \$\endgroup\$ – FarhadA May 31 '16 at 19:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.