Core Facts
Provided with Core
Documentation	User Guide, Design Guide
Design File Formats	Encrypted HDL
Instantiation Templates	VHDL
Reference Designs & Application Notes	Vivado Project, See Reference Design Manual
Additional Items	Demo on ZCU102, ZCU106, KCU105
Support
Support Provided by Design Gateway Co., Ltd.

Design Gateway Co.,Ltd

E-mail: ip-sales@design-gateway.com

URL: www.design-gateway.com

Features

· TCP/IP stack implementation

· Support IPv4 protocol

· Support one session by one TOE40G IP (Multisession could be implemented by using multiple TOE40G IPs)

· Support both Server and Client mode (Passive/Active open and close)

· Support Jumbo frame

· Packet size for transmit must be aligned to 256-bit because Transmit data bus size is 256-bit

· Received data bus size is 256-bit, so total receive size must be aligned to 256-bit

· Transmit/Receive buffer size, adjustable for optimizing resource and performance

· Simple data interface by standard FIFO interface

· Simple control interface by single port RAM interface

· 256-bit FIFO interface with Ethernet MAC

· At least 200 MHz is recommended for clock input to TOE40G IP

· Reference design available on ZCU102, ZCU106 and KCU105 evaluation board

· Not support data fragmentation feature

· Customized service for following features

· Unaligned 256-bit data transferring

· Buffer size extension by using Windows Scaling feature

· Network parameter assignment by other methods

Table 1: Example Implementation Statistics for Ultrascale device

Family

Example Device

Fmax

(MHz)

CLB Regs

CLB LUTs

CLB¹

IOB

BRAMTile²

Design Tools

Zynq-Ultrascale+

XCZU9EG-FFVB1156-2

300

5268

5271

996

34.5

Vivado2017.4

Kintex-Ultrascale

KCKU040FFVA1156-2E

300

5268

5321

959

34.5

Vivado2017.4

Notes:

1) Actual logic resource dependent on percentage of unrelated logic

2) Block memory resources are based on 64kB Tx data buffer size and 64kB Rx data buffer size.

Applications

Figure 1: TOE40G IP Application

40 Gb Ethernet is the communication channel which can transfer data at ultra high speed with remote controlling system. By using TCP/IP protocol for transferring via 40 Gb Ethernet, the system can transfer very big data at very high speed rate with reliability. TOE40G IP is the IP which is integrated to the system for transfering data via 40 Gb Ethernet without using CPU and external memory. So, the IP can fit with the application which needs to send or receive data at high-speed rate by using FPGA solution such as video data streaming and sensor monitoring system.

Figure 1 shows the example application of sensor monitoring system. The data from sensor is stored to the FIFO and forwarded to remote system via 40 Gb Ethernet by TOE40G IP. TOE40G IP is desigend to support full-duplex transfer in the same session, so Remote system can send the paramter for controlling the sensor monitoring system via 40 Gb Ethernet. In our website, we also have FTP server demo by using TOEXG IP. Please contact us for more details about FTP server demo.

TOE40G IP is designed for transferring data at the highest speed, so the latency time is much from internal pipeline register and buffer. For low-latency application such as FinTech, it is recommended to use our low-latency IP instead. Please see more details of low-latency IP from our website.

https://dgway.com/Lowlatency-IP_X_E.html

General Description

Figure 2: TOE40G IP Block Diagram

TOE40G IP core implements TCP/IP stack by hardwire logic and connects with EMAC and PCS/PMA (BASE-R) module through Adapter logic as the lower layer hardware. User interface of TOE40G IP consists of two interfaces, i.e. Register interface for control signals and FIFO interface for data signals.

Register interface has 4-bit address for accessing up to 16 registers, consisting of network parameters, command register and system parameters. The IP supports only one session, so the network parameters set by the user are fixed for assigning to TOE40G IP and the target device. After that, start IP initialization by de-asserting reset signal. Also, the reset process is necessary when some network parameters must be changed. The initialization process has two modes to get MAC address of the target device. After finishing the initialization process, the IP is ready for transferring data with the target device.

Following TCP/IP standard, the connection must be created by opening the port as the first step. The IP supports for both active open (the port opened by the IP) or passive open (the port opened by the target device). After that, data can be transferred from both sides. To send the data, the user sets total transfer size and packet size to the IP and then transfers the data via TxFIFO interface which is 256-bit data size. When the data is received from the target, the user reads the received data from the IP via RxFIFO interface. After finishing data transferring, the connection can be destroyed by using active close (the port closed by the IP) or passive close (the port closed by the target).

To meet the user system requirement which may be sensitive on the memory resource or the performance, the buffer size inside the IP can be assigned by the user. Tx data buffer and Rx data buffer can be adjusted for Tx path and Rx path respectively. Using the bigger buffer size may increase the transfer performance in each direction. More details of the hardware inside the IP are described in the next topic.

Functional Description

TOE40G IP core can be divided into three parts, i.e. control block, transmit block and receive block.

Control Block

· Reg

All parameters of the IP are set via register interface which has 4-bit address signals and 32-bit data signals. Timing diagram of register interface is similar to single-port RAM interface. The address is shared for both write and read directions. The description of each register is defined as shown in Table 2.

Table 2: Register map Definition

RegAddr [3:0]	Reg Name	Dir	Bit	Description
0000b	RST	Wr /Rd	[0]	Reset IP. ‘0’: No reset, ‘1’: Reset. Default value is ‘1’. After all parameters are assigned, the user sets ‘0’ to this register for loading parameters and starting system initialization. User must set this register to ‘1’ and ‘0’ respectively when some network paramters are changed. The network parameters controlled by RST register are SML, SMH, DIP, SIP, DPN, SPN and SRV register.
0001b	CMD	Wr	[1:0]	User command. “00”: Send data, “10”: Open connection (active), “11”: Close connection (active), “01”: Undefined. The command operation begins after the user sets CMD register. Before setting this register to start new operation, user needs to confirm that the system is in Idle status by checking busy signal de-asserted to ‘0’. Busy signal is read by bit[0] of CMD register.
		Rd	[0]	System busy flag. ‘0’: Idle, ‘1’: IP is busy.
		Rd	[3:1]	Current operation. “000”: Send data, “001”: Idle, “010”: Active open, “011”: Active close, “100”: Receive data, “101”: Initialization, “110”: Passive open, “111”: Passive close.
0010b	SML	Wr /Rd	[31:0]	Define 32-bit lower MAC address (bit [31:0]) for this IP. To update this value, the IP must be reset by RST register.
0011b	SMH	Wr /Rd	[15:0]	Define 16-bit upper MAC address (bit [47:32]) for this IP. To update this value, the IP must be reset by RST register.
0100b	DIP	Wr /Rd	[31:0]	Define 32-bit target IP address. To update this value, the IP must be reset by RST register.
0101b	SIP	Wr /Rd	[31:0]	Define 32-bit IP address for this IP. To update this value, the IP must be reset by RST register.
0110b	DPN	Wr /Rd	[15:0]	Define 16-bit target port number. Unused when the port is opened in passive mode. To update this value, the IP must be reset by RST register.
0111b	SPN	Wr /Rd	[15:0]	Define 16-bit port number for this IP. To update this value, the IP must be reset by RST register.
1000b	TDL	Wr	[31:0]	Total Tx data length in byte unit, but the length must be aligned to 32-byte (data bus size). Valid range is 32-0xFFFFFFE0 (Bit[4:0] is ignored by the IP). User needs to set this register before setting CMD register = Send data (00b). The IP loads TDL register when CMD register is set. After the IP runs Send data command (Busy=’1’), the user can set the new value of TDL register for the next command. The user does not need to set TDL register again when the next command uses the same total data length.
1000b	TDL	Rd	[31:0]	Remaining transfer length in byte unit which does not transmit.

RegAddr [3:0]	Reg Name	Dir	Bit	Description
1001b	TMO	Wr	[31:0]	Define timeout value for waiting Rx packet returned from the target. The counter is run under Clk input, so timer unit is equal to 1/Clk (5 ns when Clk is 200 MHz). TimerInt is asserted to ‘1’ when the packet is not received in time. Please see more details of TimerInt from Read value of TMO[7:0] register. This value should be more than latency time in the network to generate timeout when packet is lost, not normal condition.
1001b	TMO	Rd	[31:0]	The details of timeout interrupt are shown in TMO[7:0]. Other bits are read for IP monitoring. [0]-Timeout from not receiving ARP reply packet After timeout, the IP resends ARP request until ARP reply is received. [1]-Timeout from not receiving SYN and ACK flag during active open operation After timeout, the IP resends SYN packet for 16 times and then sends FIN packet to close connection. [2]-Timeout from not receiving ACK flag during passive open operation After timeout, the IP resends SYN/ACK packet for 16 times and then sends FIN packet to close connection. [3]-Timeout from not receiving FIN and ACK flag during active close operation After the 1^st timeout, the IP sends RST packet to close connection. [4]-Timeout from not receiving ACK flag during passive close operation After timeout, the IP resends FIN/ACK packet for 16 times and then sends RST packet to close connection. [5]-Timeout from not receiving ACK flag during data transmit operation After timeout, the IP resends the previous data packet. [6]-Timeout from Rx packet lost, Rx data FIFO full or wrong sequence number The IP generates duplicate ACK to request data retransmission. [7]-Timeout from too small receive window size when running Send data command and setting PSH[2] to ‘1’. After timeout, the IP retransmits data packet, similar to TMO[5] recovery process. [21]-Lost flag when the sequence number of the received ACK packet is skipped. As a result, TimerInt is asserted and TMO[6] is equal to ‘1’. [22]-FIN flag is detected during sending operation. [23]-Rx packet is ignored due to Rx data buffer full (fatal error). [27]-Rx packet lost detected [30]-RST flag is detected in Rx packet [31],[29:28],[26:24]-Internal test status
1010b	PKL	Wr /Rd	[15:0]	TCP data length of one Tx packet in byte unit, but the length must be aligned to 32-byte. Valid from 32-16000. Default value is 1440 byte which is the maximum size of non-jumbo frame that is aligned to 32-byte. Bit[4:0] of this register is ignored by the IP. During running Send data command (Busy=’1’), the user must not set this register. Similar to TDL register, the user does not need to set PKL register again when the next command uses the same packet length.
1011b	PSH	Wr /Rd	[2:0]	Sending mode when running Send data command. [0]-Disable to retransmit packet. ‘0’: Generate the duplicated data packet for the last data packet in Send data command (default). ‘1’: Disable the duplicated data packet.

RegAddr [3:0]	Reg Name	Dir	Bit	Description
1011b	PSH	Wr /Rd	[2:0]	[1]-PSH flag in the transmit packet. ‘0’: PSH flag in TCP header of all transmitted packet is ‘0’ (default). ‘1’: PSH flag in TCP header of all transmitted packet is ‘1’. [2]-Enable to retransmit data packet when Send data command is paused until timeout, caused by the receive window size smaller than the packet size. This flag is designed to solve the system hang problem from the window update packet lost. Data retransmission can activate the target device to regenerate the lost ACK packet. All following conditions must be met to start data retransmission. (1) PSH[2] is set to ‘1’. (2) The current command is Send data and all data are not completely sent. (3) The receive window size is smaller than the packet size. (4) Timer set by TMO register is overflowed. ‘0’: Disable the feature (default) ‘1’: Enable the feature.
1100b	WIN	Wr /Rd	[5:0]	Threshold value in 1Kbyte unit for sending window update packet. Default value is 0 (Not enable window update feature). The IP transmits the window update packet when the free space of the receive buffer is increased from the value in the latest transmitted packet more than the threshold value. For example, the user sets WIN=”000001b” (1 Kbyte) and the the window size of the latest transmitted packet is equal to 2 Kbyte. After that, the user reads 1 Kbyte data from the IP. Free space of the receive buffer is updated from 2 Kbyte to be 3 Kbyte. The IP detects that the increased window size is more than 1 Kbyte (3K – 2K) which is the threshold value. As a result, the IP sends the window update packet to update the receive buffer size.
1101b	ETL	Wr	[31:0]	Extended total Tx data length in byte unit. The size must be aligned to 32 byte. Bit[4:0] is ignored by the IP. User sets this register during Send data command operating (Busy=‘1’) for extending total Tx data length. So, the data can be transmitted continuously without re-sending the new command to IP. The caution points to use this feature are as follows. 1) ETL register must be programmed when read value of TDL is not less than 128 Kbyte to be the safe gap that Busy is not de-asserted to ‘0’ before setting ETL register. 2) The set value of ETL must be less than (0xFFFFFFE0 – read value of TDL). For example, the user sets TDL = 3.5 Gbyte in the first send data command. After the IP completes 2 Gbyte data (remaining size = 1.5 Gbyte), the user sets ETL register = 1.5 Gbyte.The total length is equal to 5 Gbyte (3.5 Gbyte of TDL + 1.5 Gbyte of ETL).
1110b	SRV	Wr/ Rd	[0]	‘0’: Client mode (default). After RST register changes from ‘1’ to ‘0’, the IP sends ARP request to get Target MAC address from the ARP reply returned by the target device. IP busy is deasserted to ‘0’ after receiving ARP reply. ‘1’: Server mode. After RST register changes from ‘1’ to ‘0’, the IP waits ARP request from the Target to get Target MAC address. After receiving ARP request, the IP generates ARP reply and then de-asserts IP busy to ‘0’. Note: In Server mode, when RST register changes from ‘1’ to ‘0’, the target device needs to resend ARP request for TOE40G IP completing the IP initialization.
1111b	VER	Rd	[31:0]	IP version

· TCP Stack

TCP stack is the main controller of the IP for controlling the other modules in every process. The IP operation has two phases, i.e. IP initialization phase and data transferring phase.

After RST register changes from ‘1’ to ‘0’, the initialization phase begins. There are two modes for running the initialization phase, set by SRV[0] register, i.e. Client mode or Server mode. The parameters from Reg module is read by TCP Stack and then set to Transmit block and Receive block for transferring the packet to complete the initialization process, following the mode. After that, the IP changes to data transferring phase.

To transfer data between TOE40G IP and the target device, it consists of three processes, i.e. opening the port, transferring data and closing the port. The IP supports to start the port opening or closing by sending SYN or FIN packet when the user sets CMD register = “10” (port opening) or “11” (port closing). Also, the port can be closed or opened by the target device as passive mode when TCP Stack detects SYN or FIN packet from Receive block. During port opening or closing process, TCP Stack asserts Busy flag to ‘1’. After finishing transferring all packets and the port is completely opened or closed, Busy flag is de-asserted to ‘0’. ConnOn signal can be monitored to confirm the port status that is completely opened or closed. The data can be transferred when ConnOn is asserted to ‘1’ (the port is opened completely).

To send the data, the data from the user is stored in Tx data buffer and Tx packet buffer. After the network parameters are read to build TCP header by Packet Builder, Transmit block sends TCP packet including the data from the user to the target device via Ethernet MAC. When the target receives the data correctly, ACK packet is returned to Receive block. TCP Stack monitors the status of Transmit block and Receive block to confirm that the data is transferred successfully. If the data is lost, TCP Stack pauses the current data transmission and then start data retransmission process in Transmit block.

When the data is received by Receive block, TCP Stack checks the order of received data. If the data is in the correct order, normal ACK packet is generated by Transmit block. Otherwise, TCP Stack starts the data lost recovery process by controlling the Transmit block for generating duplicate ACKs to the target device.

Table 3: TxBuf/RxBufBitWidth Parameter description

Value of BitWidth	Buffer Size	TxBufBitWidth	RxBufBitWidth
9	16kByte	Valid	Valid
10	32kByte	Valid	Valid
11	64kByte	Valid	Valid

Transmit Block

Tx data buffer size can be adjusted by parameter assignment. Using bigger size may increase the transmit performance. The minimum size of Tx data buffer is limited by the transmit packet size, set by PKL register. Data from Tx data buffer is split to be one packet size stored in Tx packet buffer. TCP header is prepared and then combined with TCP data stored in Tx packet buffer to build complete TCP packet. The data in Tx data buffer can be flushed after the target device returns ACK packet to confirm that the data is completely received. After finishing the send data command, the user can change the packet size and total data size for the new send data command by updating PKL and TDL register respectively.

· Tx Data Buffer

This buffer size is set by “TxBufBitWidth” parameter of the IP. The valid value is 9-11 which is equal to the address size of 256-bit buffer, as shown in Table 3. The buffer size should be more than or equal to two times of Tx packet size, set by PKL register. This buffer stores the data from the user for preparing the transmit packet sent to the target device. Data is removed from the buffer when the target device confirms that the data is completely received. Consequently, when the buffer size is large, the IP can send many data to the target device without waiting ACK packet returned from the target to clear the buffer. If the ACK packet to clear the buffer is received and the user can fill the new data to the IP in time, the IP can send the new data to the target continuously without waiting the new data from the user. As a result, the system can achieve the best transmit performance on 40Gb Ethernet connection. Nevertheless, when the carrier and the networking interface have much latency time, all data in the Tx data buffer will be completely transferred before the ACK packet to flush the buffer is received and the new data is filled by the user. As a result, the transmit performance is reduced from the latency time.

If total data from user is more than the value of TDL register, the data still remains in the buffer for the next command. All data in the buffer is flushed when the connection is closed or the IP is reset. Please note that the IP cannot send the packet if the data stored in the buffer is less than transmit size. The IP must wait until the data from user is enough for creating one packet.

· Tx Packet Buffer

This buffer size is 16 Kbyte to store at least one transmit packet. Data in Tx packet buffer is forwarded to EMAC.

· Packet Builder

TCP packet consists of the header and the data. Packet builder receives network parameters, set in Reg module, and then prepares TCP header. Also, IP and TCP checksum are calculated to be TCP header. After that, all TCP header is built, the header combining with the data from Tx packet buffer is transmitted to EMAC.

Receive Block

In the receive block, Rx data buffer is included to store the received data from the target device. The data is stored in the buffer when the header in the packet is matched to the expected value, set by the network parameters inside Reg module. Also, the IP and TCP checksum in the packet must be correct. Otherwise, the received packet is rejected. Using bigger size of Rx data buffer may increase the receive performance. Besides, TOE40G IP can support the packet re-ordering when only one packet is swapped. For example, the receive order is packet#1, #3, #2 and #4 (packet #2 is swapped with packet#3). If the packet order is switched more than one packet such as packet#1, #3, #4, and #2 (packet #3 and #4 are received before packet#2), TOE40G IP cannot reorder the data and detect as data lost condition. After that, the data recovery process is run by generating duplicated ACK packet.

· Rx Buffer

This is temporary buffer to store all Rx packets from EMAC when the previous packet is not completely processed.

· Packet Filtering

The header in Rx packet are verifed by this module to validate the packet. The packet is valid when the following conditions are met.

(1) Network parameters are matched to the value in Reg module, i.e. MAC address, IP address and Port number.

(2) The packet is ARP packet or TCP/IPv4 packet without data fragment flag.

(3) IP header length and TCP header length are valid (IP header length is equal to 20 bytes and TCP header length is equal to 20 – 60 bytes).

(4) IP checksum and TCP checksum are correct.

(5) The data pointer decoded by the sequence number is in valid range.

(6) The acknowledge number is in valid range.

· Packet Splitter

This module is designed to remove the packet header and split only TCP data to store to Rx data buffer.

· Rx Data Buffer

This buffer size is set by “RxBufBitWidth” parameter of the IP. The valid value is 9-11 for 16Kbyte – 64Kbyte buffer size. Rx data buffer size is applied to be the window size of the transmit packet. When Rx data buffer is big enough, the target device can send many data to TOE40G IP without waiting ACK packet returned by the IP which may be delayed from the networking system. As a result, the bigger size of Rx data buffer may increase the receive performance.

The data is stored in the buffer until the user reads it. If the user does not read data out from the buffer for long time, the buffer is full and then the target device cannot send more data to the IP. So, it is recommended for the user logic to read the data from the IP when the data is ready. If the Rx data buffer is not full, the receive performance will not be dropped by the full window size.

User Block

The user module can be designed by using state machine to set the command and the parameters via register interface. Also, the status can be monitored to confirm the operation is finished without any error. The data path can connect with the FIFO for sending or receiving data with the IP.

40G/50G Ethernet MAC + BASE-R

The reference design uses 40/50 Gb Ethernet Subsystem which consists of EMAC and PCS/PMA for BASE-R, provided by Xilinx. More details of the IP are provided in following website.

https://www.xilinx.com/products/intellectual-property/ef-di-50gemac.html

Core I/O Signals

Descriptions of all parameters and I/O signals are provided in Table 4 and Table 5. The EMAC interface of the IP uses 256-bit FIFO standard as FWFT mode.

Table 4: Core Parameters

Name	Value	Description
TxBufBitWidth	9-11	Setting Tx Data buffer size. The value is the address bus size of this buffer.
RxBufBitWidth	9-11	Setting Rx Data buffer size. The value is the address bus size of this buffer.

Table 5: Core I/O Signals

Signal	Dir	Description
Common Interface Signal
RstB	In	Reset IP core. Active Low.
Clk	In	Clock input. Clock frequency should be at least 200 MHz to achieve the best performance.
User Interface
RegAddr[3:0]	In	Register address bus. In Write access, RegAddr is valid when RegWrEn=’1’.
RegWrData[31:0]	In	Register write data bus. Valid when RegWrEn=’1’.
RegWrEn	In	Register write enable. Valid at the same clock as RegAddr and RegWrData.
RegRdData[31:0]	Out	Register read data bus. Valid in the next clock after RegAddr is valid.
ConnOn	Out	Connection Status. ‘1’: connection is opened, ‘0’: connection is closed.
TimerInt	Out	Timer interrupt. Assert to high for 1 clock cycle when timeout is detected. More details of Interrupt status are monitored from TMO[7:0] register.
RegDataA1[31:0]	Out	32-bit read value of CMD register (RegAddr=0001b). Bit[0] is busy flag of TOE40G-IP.
RegDataA8[31:0]	Out	32-bit read value of TDL register (RegAddr=1000b).
RegDataA9[31:0]	Out	32-bit read value of TMO register (RegAddr=1001b).
Tx Data Buffer Interface
TCPTxFfFlush	Out	Tx data buffer within the IP is reset. Assert to ‘1’ when the connection is closed or the IP is reset.
TCPTxFfFull	Out	Asserted to ‘1’ when Tx data buffer is full. User needs to stop writing data within 4 clock cycles after this flag is asserted to ‘1’.
TCPTxFfWrEn	In	Write enable to Tx data buffer. Asserted to ‘1’ to write data to Tx data buffer.
TCPTxFfWrData[255:0]	In	Write data to Tx data buffer. Valid when TCPTxFfWrEn=’1’.
Rx Data Buffer Interface
TCPRxFfFlush	Out	Rx data buffer within the IP is reset. Assert to ‘1’ when the new connection is opened.
TCPRxFfRdCnt[10:0]	Out	Data counter of Rx data buffer to show the number of received data in 256-bit unit.
TCPRxFfLastRdCnt[4:0]	Out	Remaining byte of the last data in Rx data buffer when total received data in the buffer is not aligned to 32-byte unit. User cannot read the data until all 32-byte data is received.
TCPRxFfRdEmpty	Out	Asserted to ‘1’ when Rx data buffer is empty. User needs to stop reading data immediately when this signal is asserted to ‘1’.
TCPRxFfRdEn	In	Assert to ‘1’ to read data from Rx data buffer.
TCPRxFfRdData[255:0]	Out	Data output from Rx data buffer. Valid in the next clock cycle after TCPRxFfRdEn is asserted to ‘1’.

Signal	Dir	Description
Mac FIFO Interface (FWFT mode)
MacTxFfWrCnt[15:0]	In	Write data counter of MacTxFIFO in 256-bit. This signal is used to check full condition. If FIFO counter is less than 16-bit, please fill ‘1’ to upper bit.
MacTxFfWrData[255:0]	Out	256-bit write data to MacTxFIFO. Valid at the same clock as MacTxFfWrEn=’1’.
MacTxFfWrEn	Out	Assert to ‘1’ to write data to MacTxFIFO. This signal is asserted to ‘1’ continuously to transmit one Tx packet.
MacTxLastByteEn[31:0]	Out	Data byte enable of the last data in each Tx packet. This signal is valid when MacTxEnd=’1’ and MacTxFfWrEn=’1’. In IP core, this signal could be equal to two values, i.e. 0x0FFF_FFFF and 0x003F_FFFF.
MacTxSizeData[15:0]	Out	Total size of Tx packet in 256-bit unit. This value is stable during MacTxFfWrEn=’1’. Bit[15:11] is always equal to 0.
MacTxEnd	Out	Assert to ‘1’ to indicate the final data of each Tx packet.
MacRxFfEmpty	In	Empty flag of MacRxFIFO. Assert to ‘1’ when there is no data within MacRxFIFO.
MacRxFfRdEn	Out	Assert to ‘1’ to read data from MacRxFIFO. This signal could be asserted to ‘1’ when MacRxFfEmpty=’0’.
MacRxFfRdData[257:0]	In	Read data output from MacRxFIFO. This signal is valid at the same clock as MacRxFfRdEn=’1’. So, MacRxFIFO must be designed to run as FWFT mode. The definition of each received data is as follows. [255:0]: 256-bit received data from EMAC. [256]: Assert to ‘1’ to indicate the final data in each Rx packet from EMAC. [257]: ‘0’-normal packet, ‘1’-error packet. This bit must be asserted to ‘1’ at the same time as bit[256]=’1’ which indicates final data.

Timing Diagram

IP Initialization

The initialization process begins after user changes RST register from ‘1’ to ‘0’. TOE40G IP can run in two modes, set by SRV[0] register, i.e. Client mode (SRV[0]=’0’) and Server mode (SRV[0]=’1’). The details of each mode are shown in the following timing diagram.

Figure 3: IP Initialization in Client mode

As shown in Figure 3, in Client mode TOE40G IP sends ARP request and waits ARP reply returned from the target device. Target MAC address is extracted from ARP reply packet. After finishing, Busy signal (bit0 of RegDataA1) is de-asserted to ‘0’.

Figure 4: IP Initialization in Server mode

As shown in Figure 4, after reset process in Server mode, TOE40G IP waits ARP request sent by the target device. After that, TOE40G IP returns ARP reply to the target. Target MAC address is extracted from ARP request packet. Finally, Busy signal is de-asserted to ‘0’.

Register Interface

All control signals and the network parameters for the operation are set and monitored via Register interface. Timing diagaram of Register interface is similar to Single-port RAM which shares the address bus for write and read access. Read latency time of the read data from the address is one clock cycle. Register map is defined in Table 2.

As shown in Figure 5, to write the register, the user sets RegWrEn=’1’ with the valid value of RegAddr and RegWrData. To read the register, the user sets only RegAddr and then RegRdData is valid in the next clock cycle.

Figure 5: Register interface timing diagram

As shown in Figure 6, before the user sets CMD register to start the new command operation, Busy flag must be equal to ‘0’ to confirm that IP is in Idle status. After CMD register is set, Busy flag is asserted to ‘1’. Busy is de-asserted to ‘0’ when the command is completed.

Figure 6: CMD register timing diagram

Tx FIFO Interface

To send the data to IP core via Tx FIFO interface, Full flag is monitored to be flow control signal. The write signals are similar to write interface of general FIFO by using write data and write enable as shown in Figure 7.

Figure 7: Tx FIFO interface timing diagram

(1) Before sending data, user needs to confirm that full flag (TCPTxFfFull) is not asserted to ‘1’ and ConnOn must be equal to ‘1’. After that, TCPTxFfWrEn can be asserted to ’1’ with valid value of TCPTxFfWrData.

(2) TCPTxFfWrEn must be de-asserted to ‘0’ within 4 clock cycles to pause data sending after TCPTxFfFull is asserted to ‘1’.

(3) After finishing transferring all data, the port can be closed by TOE40G IP (active) or the target device (passive). After the port is closed, the following situations are found.

a) ConnOn changes from ‘1’ to ‘0’.

b) TCPTxFfFlush is asserted to ‘1’ to flush all data inside TxFIFO.

c) TCPTxFfFull is asserted to ‘1’ to pause data sent by the user during closing the connection.

Rx FIFO Interface

After the received data is stored in Rx data buffer, the user can read the data from Rx data buffer by using Rx FIFO interface. Empty flag is monitored to check data available status and then asserts read enable signal to read the data, similar to read interface of general FIFO, as shown in Figure 8.

Figure 8: Rx FIFO interface timing diagram by Empty flag

(1) TCPRxFfEmpty is monitored to check data available status. When data is ready (TCPRxFfEmpty=’0’), TCPRxFfRdEn can be asserted to ‘1’ to read data from Rx data buffer.

(2) TCPRxFfRdData is valid in next clock cycle.

(3) Reading data must be immediately paused by de-asserting TCPRRxFfRdEn=’0’ when TCPRxFfEmpty=‘1’.

(4) User must read all data from Rx data buffer before the connection is new created. All data in Rx data buffer is flushed and TCPRxFfFlush is asserted to ’1’ when the new connection is created. After finishing new connection created, ConnOn changes from ‘0’ to ‘1’.

(5) After finishing Flush operation, TCPRxFfEmpty is asserted to ‘1’.

Figure 9: Rx FIFO interface timing diagram by using read counter

If user logic reads data as burst mode, TOE40G IP has read counter signal to show the total data stored in Rx FIFO interface as 256-bit unit. For example, in Figure 9 there are five data available in Rx data buffer. So, user can assert TCPRxFfRdEn to ‘1’ for 5 clock cycles to read all data from Rx data buffer. The latency time from read enable (TCPRxFfRdEn) to read counter (TCPRxFfRdCnt) is 2 clock cycles.

MAC FIFO Interface

EMAC interface of TOE40G IP is designed to be FIFO in FWFT mode interface. To integrate TOE40G IP with 40G EMAC, adapter logic including FWFT FIFO is recommended as shown in Figure 10.

Figure 10: TOE40G IP and EMAC connection

Before transmitting packet, TOE40G IP monitors MacTxFfWrCnt to check if free space inside MacTxFIFO is enough to store one packet. After that, MacTxFfWrEn is asserted to ‘1’ continuously until end of packet. MacTxFfWrData is valid at the same clock as MacTxFfWrEn=’1’. At the end of packet, MacTxEnd is asserted to ‘1’ with the valid MacTxLastByteEn. MacTxSizeData is stable during transmitting each packet to show total data of the packet in 256-bit unit. The example of timing diagram when TOE40G IP sends packet and packet size = n is shown in Figure 11.

Figure 11: MacTxFIFO interface timing diagram

(1) TOE40G IP starts transmitting the packet by asserting MacTxFfWrEn to ‘1’ with the valid data on MacTxFfWrData when free space of MacTxFf is enough for storing the packet (MacTxFfWrCnt <65535-n). The packet is transmitted continuously until the end of packet.

(2) During transmitting a packet, transmit packet size is valid on MacTxSizeData.

(3) MacTxEnd is asserted to ‘1’ with the valid byte enable of the last data on MacTxLastByteEn signal when transmitting the last data of the packet.

(4)

Figure 12 shows the example of MacRxFIFO interface when Rx packet size is equal to n. MacRxFfEmpty is applied to monitor data status in MacRxFIFO. The interface uses FWFT FIFO characteristic, so MacRxFfRdData is valid at the same clock as MacRxFfRdEn=’1’.

Figure 12: MacRxFIFO interface timing diagram

(1) TOE40G IP starts reading the received packet when MacRxFfEmpty is de-asserted to ‘0’. MacRxFfRdEn is asserted to ‘1’ to read the data and MacRxFfRdData is valid in the same clock cycle which is FWFT FIFO characteristic. 256-bit received data is placed on bit[255:0].

(2) During reading the packet, if data is not ready to read (MacRxFfEmpty=’1’), MacRxFfRdEn is de-asserted to ‘0’ to pause data reading in the same clock.

(3) Bit256 of MacRxFfRdData is asserted to ‘1’ when the last data of the packet is transferred. Also, bit257 of MacRxFfRdData is valid to check error status in this clock. MacRxFfRdEn is de-asserted to ‘0’ after receiving the last data for 8 clock cycles which is the time for IP post-processing the packet.

Example usage

Client mode (SRV[0]=’0’)

The example step to set register for transferring data in Client mode is shown as follows.

1) Set RST register=’1’ to reset the IP.

2) Set SML/SMH for MAC address, DIP/SIP for IP address and DPN/SPN for port number.

Note: DPN is optional setting when the port is opened by IP (Active open).

3) Set RST register=’0’ to start the IP initialization process by sending ARP request packet to get Target MAC address from ARP reply packet. Busy signal is de-asserted to ‘0’ after finishing the initialization process.

4) The new connection can be created by two modes.

a. Active open: Write CMD register = “Open connection” to create the connection (SYN packet is firstly sent from TOE40G IP). After that, wait until Busy flag is de-asserted to ‘0’.

b. Passive open: Wait until “ConnOn” signal = ‘1’ (the target device sends SYN packet to TOE40G IP firstly).

5) a. For data transmission, set TDL register (total transmit length) and PKL register (packet size). Next, set CMD register = “Send Data” to start data transmission. The user sends the data to TOE40G IP via TxFIFO interface before or after setting CMD register. When the command is finished, busy flag is de-asserted to ’0’. The user can set the new value to TDL/PKL register and then set CMD register = “Send Data” to start the next transmission.

b. For data reception, user monitors RxFIFO status and reads data until RxFIFO is empty.

6) Similar to creating the connection, the connection can be destroyed by two modes.

a. Active close: Set CMD register = “Close connection” to close the connection (FIN packet is firstly sent by TOE40G IP). After that, wait until Busy flag is de-asserted to ‘0’.

b. Passive close: Wait until “ConnOn” signal = ‘0’ (FIN packet is sent from the target to TOE40G IP firstly).

Server mode (SRV[0]=’1’)

Comparing to Client mode which MAC address is decoded from ARP reply packet after TOE40G IP sends ARP request packet, Server mode decodes MAC address from ARP request packet. The process for transferring data is the same as Client mode. The example step of Server mode is shown as follows.

1) Set RST register=’1’ to reset the IP.

2) Set SML/SMH for MAC address, DIP/SIP for IP address and DPN/SPN for port number.

3) Set RST register=’0’ to start the IP initialization process by waiting ARP request packet to get Target MAC address. Next, the IP creates ARP reply packet returned to the target device. After finishing the initialization, busy signal is de-asserted to ‘0’.

4) Remaining steps are similar to step 4 – 6 of Client mode

Verification Methods

The TOE40G-IP Core functionality was verified by simulation and also proved on real board design by using ZCU102/ZCU106/KCU105 evaluation board.

Recommended Design Experience

User must be familiar with HDL design methodology to integrate this IP into their design.

Ordering Information

This product is available directly from Design Gateway Co., Ltd. Please contact Design Gateway Co., Ltd. For pricing and additional information about this product using the contact information on the front page of this datasheet.

Revision History

Revision	Date	Description
1.0	2-Oct-2018	New release
1.1	26-Nov-2018	Support KCU105 FPGA development board
1.2	2-Oct-2020	Update company info