Fault Tolerance (FT)

The primary and secondary controllers execute identical instruction streams on a cycle-by-cycle basis, strictly synchronized by a common clock signal. This is not simple standby redundancy; both systems process the same inputs simultaneously.

• Cycle-accurate logic: Outputs are compared at the end of each scan cycle.
• Immediate fault detection: Any divergence in state triggers an instantaneous alarm.
• Hardware-enforced: Often relies on specialized lockstep cores within a system-on-chip.

The mechanism by which control authority switches from a failed primary to a healthy secondary without any perturbation in the physical process. The output signal must not glitch, spike, or drop to a default state during the transition.

• Output smoothing: Analog values are maintained at the last known good value during the switchover.
• Sub-millisecond takeover: The failover is completed within a single control cycle to prevent watchdog timeouts.
• Critical for motion control: Prevents servo motors from experiencing a momentary loss of torque command.

A redundancy model where the backup system is fully powered, running, and maintains a real-time, bitwise-identical copy of the primary's dynamic memory. This includes the process image, timers, counters, and internal relays.

• Continuous synchronization: Data is transferred over a dedicated high-speed fiber link.
• No re-initialization: The standby does not need to reboot or reload the program upon takeover.
• Contrasts with warm standby: Warm standby only syncs at program boundaries, risking data loss.

The system must identify a failure within a strictly bounded time interval, typically a single maximum cycle time. This is achieved through a combination of hardware watchdogs and software heartbeat monitoring.

• Hardware watchdog timers: Independent circuits that must be toggled by the controller within a defined window.
• Heartbeat packets: Periodic UDP frames sent between redundant units to confirm liveness.
• Failure modes covered: Power loss, CPU halt, memory corruption, and I/O bus failure.

An advanced fault tolerance technique that protects against common-cause failures by using dissimilar hardware or software in the redundant channels. A bug in a specific CPU stepping or compiler version will not affect both channels simultaneously.

• N-version programming: Independently developed software executing the same specification.
• Dissimilar hardware: Using processors from different vendors for the primary and secondary.
• Defeats systematic faults: Prevents a single design flaw from bringing down the entire redundant pair.

A fault masking architecture where three identical systems process the same data, and a majority voter selects the correct output. A single faulty channel is automatically outvoted, providing uninterrupted operation without any switchover delay.

• Instant fault masking: No detection or reconfiguration time is required.
• Common in aerospace and safety-critical process industries.
• 2-out-of-3 (2oo3) voting: The system tolerates one fault while maintaining safe operation.

The fundamental mechanism of fault tolerance where a secondary controller executes the identical instruction stream in strict cycle-by-cycle synchronization with the primary controller. Both systems process the same inputs, execute the same logic, and produce the same outputs simultaneously. The secondary's outputs are typically suppressed until a fault is detected. This requires cycle-accurate determinism—any divergence in clock timing, interrupt handling, or memory state between the two systems will break lockstep and trigger a false failover. Modern virtualized PLCs achieve this through hardware-assisted virtualization and Precision Time Protocol (PTP) synchronization.

The transition of control authority from a failed primary to a healthy secondary without any observable disruption to the controlled process. Key characteristics include:

• Zero output discontinuity: Analog outputs do not spike, dip, or glitch during switchover
• State preservation: All timers, counters, and sequential function chart steps retain their exact values
• No scan cycle interruption: The process scan continues without missing a single execution cycle Achieving this requires the secondary to maintain a hot shadow copy of the primary's entire data memory, updated every scan. In IEC 61131-3 environments, this includes retaining persistent variables and retain data across the failover boundary.

A continuous health verification protocol where the primary and secondary controllers exchange periodic keep-alive signals over a dedicated, isolated communication path. If the secondary fails to receive a heartbeat within a configured watchdog timeout—typically measured in microseconds for high-speed processes—it immediately assumes the primary has failed and initiates takeover. Advanced implementations use multi-path heartbeats across redundant network interfaces to prevent false failovers caused by a single link failure. The heartbeat payload often includes a sequence number and CRC checksum of the current process state to detect silent data corruption before it propagates.

An extension of dual fault tolerance that employs three identical controllers executing in lockstep with a majority voting output mechanism. If one controller produces a divergent result, the two remaining controllers outvote it, and the faulty unit is isolated. This architecture tolerates not only fail-stop failures but also Byzantine faults where a controller produces incorrect but non-halting outputs. TMR is common in Safety Integrity Level 3 (SIL 3) applications such as turbine control and emergency shutdown systems. The voting logic itself must be implemented in a fault-tolerant manner, often using radiation-hardened or diverse-redundancy voter circuits.

The total elapsed time from the instant a primary controller fault is detected to the moment the secondary assumes full control authority. This metric is the critical performance indicator for any fault-tolerant system. Components include:

• Fault detection time: Heartbeat timeout plus diagnostic self-check duration
• Synchronization gap: Time since the last state mirroring update
• Output switchover time: Physical relay or network path reconfiguration delay In virtualized industrial control systems, failover latency can be reduced to sub-millisecond ranges using SR-IOV for direct NIC access and CPU pinning to eliminate hypervisor scheduling jitter during the transition.

The operational dashboard and API layer that provides visibility and control over the fault-tolerant pair. Functions include:

• Role designation: Forcing which controller is primary vs. secondary
• Synchronization status: Monitoring lockstep health and drift metrics
• Manual failover trigger: Initiating planned switchover for maintenance
• Audit logging: Recording every failover event with timestamps and root cause data In modern software-defined systems, this interface exposes REST APIs and OPC UA nodes that allow higher-level orchestration systems to programmatically manage redundancy policies across entire fleets of virtualized controllers.