I am exploring a range of techniques to implement TMR clock trees as part of a global TMR design (all resources including i/o pins, clock trees, reset trees, logic and registers are implemented with triple redundancy). As I am not interested in being locked into any vendor's automated GTMR tools, I'm looking to do this by hand. My understanding is that GTMR is required in FPGA because a single-event-upset SEU in the CRAM bits of a SRAM based FPGA could disconnect the clock tree from a large portion of the logic that it was driving...

I observe the difficulties with the following 3 approaches on an Altera Cyclone V SE:

1) Altera's Cyclone ALTPLL can generate up to 6 clock outputs from one clock source. It is technically possible to request 3 output clocks, each output clock driven at the same target frequency, duty cycle and phase offset. Unfortunately the redundancy is "optimised away" by the fitter tool, resulting in only a single global clock tree being driven. Can someone propose how to prevent this optimisation? --> Yes, it is probably better to have three ALTPLL.. please see approach 2. However, approach 1 would be interesting to explore jitter between driving registers from 3 different clock networks using this approach.

2) In true GTMR fashion, lets assume we use three input clock pins, each input clock pin driving its own pair of {ALTPLL, ALTCLKCTRL} modules. With a rather generous application of guidance to the tools (e.g. syn_keep/syn_noprune/dont_touch like controls), it is possible to drive three global clock tree networks with 3 different PLL operating at the same speeds. I have setup a very simple timing harness (1 data input pin drives a shift register of 10 bits. Those 10 input bins are xored onto the state of 2x 10 bit shift register driving, each shift register driving one data output pin. The three shift registers are all driven by their own clock pin (pin_m?_clock)). With this simple test scheme in place, TimeQuest Timing Analyzer "setup summary" complains that: pin_m0_clock slack: 0.650 End Point TNS: 0.000 pin_m1_clock slack: -0.877 End Point TNS: -7.575 (flagged as an error) pin_m2_clock slack: -0.855 End Point TNS: -7.9641 (flagged as an error) I'm not sure what is required to address these errors in a safe way.

3) If we use three clock pins (either driven by a single clock source, or three frequency and phase synchronised clock sources), and three clock trees, in which each clock pin directly drives its own ALTCLKCTRL module, it is possible to drive three global clock tree networks through the FPGA. Using the same test harness described in (2) above, TimeQuest Timing Analyzer does not make any complaints. There is around -0.339 (m0_clock->m1_clock) to -0.583 (m0_clock->m2_clock) clock skew. I note that clock control for M0 and M1 are located physically close together on the middle-left-edge of the chip, where the third clock control for M2 is located on the middle-bottom edge of the chip and may explain some of the difference in skew (339 vs 583). [9 Nov 2014: I was able to reduce the skew from 583 back down to ~300 by clustering the altclktrl modules physically close together, and driving each of the 3 clock pins from a near-by a dedicated positive-edge clock input pin].

As an related item, I implemented a single-clock version of the same test harness to check for clock-skew between registers on the same clock.. and it was down to -0.071. I'm hoping there is some way to significantly reduce that >300 skew down to something much lower. (My understanding is at least one of the primary goals of reducing the jitter between the TMR clock-trees is to prevent metastability problems on the feedback loop of TMR finite state machines [ loopback -> voter -> FSM logic -> D-FF -> loopback ].)

I'd be interested to hear advice on how to improve this (3)rd approach. I am not sure exactly what the pro's/con's of driving the global clock networks directly from the pins are, but it seems like approach (2) above would be better if the errors reported in (2) could be overcome.

I appreciate all input, guidance and advice on how to correctly do / optimise any of the above three manual approaches. Please feel free to propose a even better manual approach for on-chip global TMR.


The Happy Techy

  • 1
    \$\begingroup\$ You might be fighting the wrong battle. Unless the parts and the tools are designed to support your application's requirements, you might be on thin ice. Perhaps you should talk with the factory first - it could be helpful. \$\endgroup\$ Mar 9, 2015 at 20:24

1 Answer 1


That's a doozy of a question. What's the research for?

I know the approach I'm going to suggest won't resolve the problem with the tools complaining, but it might minimize the skew in an actual implementation. I'm not familiar with Altera FPGA's, I've worked mostly with Xilinx S3 and S6 parts; I know this approach can be made to work in an S3, but not an S6, so it might be possible in your case.

In the S3, you can make small adjustments to the phase of the managed clocks (see pg4,5); in a research paper that I've read (I'm still looking for it, will update with a link if/when I find it), the researchers used this capability and a feedback loop to fine-tune the synchronization of a comms link.

If your FPGA has similar capabilities then you might be able to do something similar: use feedback loops to synchronize your three PLL's. It might also be able to help with the problem that three external clock sources and three PLL's will never have identical frequencies. They may be close - to within a few ppm - but never identical.

  • \$\begingroup\$ Q: What's the research for? A: Prototyping a fault tolerant real-time multi-core computer architecture. Re: "Three PLL's will never having identical frequencies". Is it correct to say: given there is a common clock, while each PLL may experience jitter with regard to the other 2 PLL, none of the 3 PLL will run completely out-of-phase to the other 2 PLL (assuming no SEU hits on the PLL's)? It seems to address SEU in one PLL requires something like link. Might cross-linking the feedback between PLL introduce a single point of failure? \$\endgroup\$ Nov 9, 2014 at 11:15
  • \$\begingroup\$ I'm not sure whether three PLL's driven by a common clock would settle at identical frequencies; my instinct is that they would, at least on average, although as you note their jitter would not be synchronized. I can see the controller/s in the feedback loop/s as single points of failure, but that could be mitigated by having them fail into a state that will force a reset of the controller and/or PLL. Using a common clock is a SPoF anyway, depending on where you've placed your system boundaries. \$\endgroup\$
    – markt
    Nov 9, 2014 at 11:30
  • \$\begingroup\$ I agree a single common clock (external to the FPGA) is a SPoF. Ideally global TMR would employ three independent but synchronised external clock sources. My initial prototyping will probably have a single external clock (limitation of COTS development boards). --- Guidance on how to correctly implement full GTMR down to the external clocks would be well received :-) I have not been able to find any books/documents/papers/.. disclosing all the low-level implementation details. \$\endgroup\$ Nov 9, 2014 at 11:39
  • \$\begingroup\$ I haven't explored TMR within an FPGA; I'm currently doing postgrad research on a distributed NMR implementation that happens to use FPGA's. That patent you linked to appears to be owned by Xilinx, so they've evidently supported research in the area; maybe change your FPGA choice ;) and see if they'll disclose any more to you. \$\endgroup\$
    – markt
    Nov 9, 2014 at 11:44
  • \$\begingroup\$ Re: "Distributed NMR" - sounds fun. Re FPGA choice: There are FPGA vendor neutral, automated TMR including Mentor Graphics and Synopsys. This is not a FPGA vendor issue. This is a "how to do it manually by hand" issue :-). Xilinx offer TMRTool for their FPGA which uses that patent. \$\endgroup\$ Nov 9, 2014 at 11:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.