The tremendous growth in the number of computing devices and physical busses connecting such devices has made distributed systems a logical solution for many applications in design, control and test.
While virtually all distributed systems require communication between the elements of the system, a subset of these systems need deterministic communication. These distributed applications may include distributed motion control, process control and monitoring, hardware-in-the-loop simulation, and synchronised remote data acquisition. Although engineers have used various busses to achieve deterministic communication to solve these applications, a deterministic Ethernet solution is appealing due to its widespread adoption and pervasive infrastructure. This article explores the need for deterministic communication, reviews the technical limitations of standard Ethernet networks, and investigates the techniques that overcome these challenges to implement deterministic communication across Ethernet.
Latencies limit distributed applications
Distributed systems can deliver flexibility and efficiency to machines or processes, but distribution can also imply latencies. Because nodes are physically separate, engineers must take into account the time it takes to transfer data between nodes. Moreover, when the latency is indeterminable, nodes on the network must be prepared to wait on data to be sent or received. For example, in a typical Ethernet-based network the period of waiting can vary greatly because it depends on a host of factors such as protocol used, bandwidth available, aggregate network traffic, and number of Ethernet devices. In many applications, engineers can work around this issue by using techniques such as buffering to compensate for these limitations. Consider, for example, a network application in which a server streams a video across a network connection, the client application buffers several seconds worth of video frames to make up for the inconsistent transfer of data.
While these buffering methods are suitable for a Web-based multimedia application, they are not appropriate for advanced distributed control applications. For example, a fly-by-wire aircraft simulation may require a flight control simulator to communicate a new angle parameter to the rudder control simulator. The rudder must receive the latest angle parameter within a fixed amount of time, otherwise the response of the rudder will be late and the flight control system will fail. Buffering data on the network does not alleviate the issue, because buffered data is useless in a dynamic system, such as an aircraft. The fly-by-wire simulation example falls on the extreme side of the network response spectrum, shown in Figure 1, but it demonstrates some of the challenges associated with distributed systems. Industrial control systems have similar requirements when I/O and processing is distributed among nodes. To meet the requirements of distributed control applications, the network itself must have a realtime (deterministic) response.
Inherent non-determinism of standard Ethernet
The unknown latency characteristic of Ethernet networks is due to the inherent design of the IEEE 802.3 Ethernet standard. For the transmission of data by many devices on a single network, Ethernet uses the carrier sense multiple access, with collision detection (CSMA/CD) mechanism. This standard establishes that a network device listens until no other devices transmit, then it begins a transmission. However, if another device begins simultaneously transmitting, a collision on the network may occur. Per the standard, the devices should detect that a collision has occurred and then wait for a random period of time before attempting to retransmit.
High-level Ethernet protocols, such as the commonly used transmission flow protocol (TCP) introduce additional handshaking to ensure that data sent across Ethernet has been received. Senders of TCP data wait until the receiver sends a positive acknowledgement (ACK). If the ACK is not received within a timeout period, the sender retransmits the data. The CSMA/CD access mechanism and flow control methods used by transmission protocols introduce a timing uncertainty to Ethernet networks that make them non-deterministic.
In fact, because standard Ethernet networks have not offered an adequate solution for deterministic communication, engineers solve applications with this requirement with proven deterministic busses such as the controller area network (CAN) bus and MIL-STD-1553 for automotive and military applications, respectively. In industrial applications, widely adopted fieldbusses, such as DeviceNet and Profibus, are used by control engineers. Other solutions include expensive reflective memory networks with fixed traffic patterns on token ring networks. The network infrastructures of these deterministic busses are generally vertical in use and design. For example, the CAN bus is designed for automotive networks and DeviceNET is designed for industrial sensor networks. This limits the scope of these applications to specific areas.
Why Ethernet?
Despite existing solutions for deterministic data transfer, and the technical challenges of achieving realtime Ethernet networks, there is a growing trend toward using Ethernet to solve these applications. There are many factors that drive this trend. Firstly, Ethernet ports and networks are ubiquitous. As more devices are 'wired', there is greater potential to connect downstream activities, such as a widget sorting machine, with upstream activities, such as enterprise-side inventory management applications. This can increase the productivity and reduce cost of large systems.
"If everything has an IP address, then you know the status of any piece of machinery or a system across a manufacturing enterprise," says Bob Parker, an analyst with IDC who tracks IT trends in manufacturing.
Secondly, the low cost of Ethernet makes it highly desirable in comparison to specific busses designed for vertical application areas such as CAN and MIL-STD-1553. Moreover, existing Ethernet infrastructure can support multiple protocols resulting in network re-use, cost reduction, and increased flexibility. Thirdly, many existing solutions have bandwidth limitations solved by Gigabit Ethernet. For example, the maximum throughput on a Gigabit Ethernet is close to 10 times that of DeviceNet.
For these reasons, the market has an influx of realtime Ethernet solutions. These range from soft realtime protocols such as Ethernet/IP, to SERCOS III, a hard realtime Ethernet protocol for distributed motion control. Additionally, the emergence of the IEEE 1588 precision time protocol (PTP), has made it feasible to use Ethernet for synchronised distributed applications. IEEE 1588 provides a standard method to synchronise devices on a network with submicrosecond precision.
Making Ethernet deterministic
How can Ethernet overcome its inherent non-determinism? Simply put, strict timing rules must be put in place by the system governing the network to make it deterministic. These 'rules' can be implemented in different ways to achieve the desired outcome. However, they usually have two common elements. First, all nodes on the network synchronise their time to a master clock. The master clock is established by one node designated as the master of the system. The IEEE 1588 protocol, for example, can establish the clock synchronisation for a deterministic Ethernet network. Second, system engineers configure a schedule that determines when each node on the network transmits data during each network cycle. To create this schedule, the system must know the amount of data transferred to reserve the appropriate time needed to send the data across the network.
One approach to implementing deterministic Ethernet is incorporated in the latest version of the National Instruments LabVIEW Real-Time Module, the graphical programming environment for developing deterministic applications. This software development tool introduces a new technology for deterministic data transfer across Ethernet called the time-triggered network. This technology is an example of how to achieve deterministic transfer of data across Ethernet. With the time-triggered network, two or more realtime PXI or PC targets can transfer data deterministically across a private Ethernet network using off-shelf-network interfaces. As shown in Figure 2, each PC is also connected to a public network for normal network traffic and communication with non realtime nodes. This two wire topology provides a level of redundancy to communications.
Other deterministic Ethernet protocols, such as Ethernet PowerLink, split the network cycle time so that regular Internet traffic, such as TCP/IP, can occur after deterministically scheduled packets are sent and received. This method requires less wiring between realtime nodes, but typically requires a gateway to schedule traffic coming from outside the realtime network.
Transferring data with shared memory blocks
One method of transferring data across a deterministic Ethernet network is through a shared memory scheme. All nodes on the network allocate a block of memory for every node on the network including itself. Shared memory blocks take all the data from one node and send it to all other nodes as one packet. Figure 3 represents a shared memory network of three nodes. Node A can only write to the A memory block, Node B can only write to the B memory block, and so on. In every network cycle, the data written from one node is sent or 'reflected' to all other nodes on the network.
For the shared memory network to be deterministic, the system must schedule the reflection of each shared memory block so they do not overlap. These reflections could be scheduled to transfer data between several nodes making up a flight simulation machine, for example. To configure the network, one node is designated as the master node while all other nodes are slaves. At the start of each cycle, the master node sends a cycle start packet to the slave nodes. Once the start cycle packet ends, nodes can begin to send shared memory blocks. Using a shared memory block to transfer multiple data items reduces the overhead per transfer.
The system can use the synchronised network clock to derive secondary timing sources based on when data is sent. For instance, the flight simulation machine may have a node that executes a software model of a flight control computer. The simulated flight control computer can be programmed to respond to the reflection of a shared memory block. The timing source can work in conjunction with a realtime task to process data as soon as it has been reflected to the network. The system can use a shared memory scheme to distribute processing between many nodes on an Ethernet network, thus increasing the processing power of a system.
Closing control loops across Ethernet
An alternate way of transferring data is using a dedicated slot method. When using this method, a user explicitly schedules when each data item will be transferred during the network cycle. The advantage is that a loop can be closed during a single network cycle. Figure 4 shows an example of a distributed motion controller.
In a practical implementation, the system can perform acquisition and actuation when the network is busy. In the example above, the deterministic network requires time during the network cycle to maintain clock synchronisation. During this period one node can perform the I/O. The I/O node then sends the feedback across Ethernet to the controller, which processes and transmits an output back across the network before the network cycle completes. Engineers can use this method for closing any type of control loop across the network, allowing a decentralised system for control that reduces overall wiring and complexity.
Deterministic network challenges
Developing and implementing a deterministic network presents several challenges. While standard Ethernet is inherently non-deterministic, collision detection and high-level flow control ensure data is retransmitted in cases where it has been lost due to network noise or collisions. Deterministic Ethernet networks eliminate collisions, but still face a challenge when data is lost due to noise, because retransmission can spoil the predefined network schedule. Users can incorporate redundant data transmission into the schedule, this however, reduces the overall network throughput. Consequently, deterministic Ethernet protocols must include robust error checking to alert nodes of possible data loss, and users must decide how to handle data loss situations.
Another challenge is incorporating deterministic Ethernet into applications distributed across a large geographical area, such as across multiple subnets. Generally speaking, deterministic Ethernet networks cannot span more than one subnet because routing and switching elements introduce large delays into network data transfers. With the IEEE 1588 protocol, users can synchronise clocks across subnets and therefore synchronise nodes spanning a larger network. However, when a system must deterministically transfer data, nodes typically reside on the same subnet.
Deterministic Ethernet meeting advanced application requirements
As can be seen in this article, emerging Ethernet technologies can overcome previous technical limitations and make a strong case for consideration in demanding realtime and industrial applications. By leveraging the existing Ethernet infrastructure and providing performance gains compared to conventional busses, deterministic Ethernet technologies are paving the future of distributed solutions for a wide range of applications.
Source:
1. http://www.techworld.com/networking/features/index.cfm?featureid=1439&Page=1&pagePos=20
For more information contact Michael Hutton, National Instruments, 0800 203 199, [email protected]
© Technews Publishing (Pty) Ltd | All Rights Reserved
printer friendly version