This document shows the example to run TOE10GLL-IP (Cut-through mode) demo by using two test environments. First is run by using one FPGA board transferring TCP data via 10Gb Ethernet with TestPC which runs test application for transferring TCP data. Test performance in this environment is limited by the resource on TestPC. Second is run by using two FPGA boards for transferring 10Gb Ethernet data to each other. This environment achieves the best performance for transferring TCP data via 10Gb Ethernet by using TOE10GLL-IP.

In the document, topic 2 shows the example to set up 10Gb Ethernet card on TestPC to get the good performance for transferring data via 10Gb Ethernet when running the test by using the first test environment, FPGA and Test PC. Topic 3 shows the example console and test result when running under the first test environment. Finally, topic 4 shows the example console when running the second test environment, FPGA and FPGA. More details of each topic are described as follows.

Note: To set up FPGA test environment, please follow the instruction described in FPGA set up for TOE/UDP10G-IP document (“dg_toeudp10gip_fpgasetup_xxx”).

2 PC Setup

Before running demo, please check the network setting on PC. The example of setting 10Gb Ethernet card is described as follows.

2.1 IP Setting

Figure 2‑1 Setting IP address for PC

1) Open Local Area Connection Properties of 10-Gb connection, as shown in the left window of Figure 2‑1.

2) Select “TCP/IPv4” and then click Properties.

3) Set IP address = 192.168.7.25 and Subnet mask = 255.255.255.0, as shown in the right window of Figure 2‑1.

2.2 Speed Setting

Figure 2‑2 Set Link Speed = 10 Gbps

1) On Local Area Connection Properties window, click “Configure” as shown in Figure 2‑2.

2) On Advanced Tab, select “Speed and Duplex”. Set the value to “10 Gbps Full Duplex” for running 10-Gigabit transfer test, as shown in Figure 2‑2

3) On “Interrupt Moderation” window, select “Disabled” to disable interrupt moderation which would minimize the latency during transferring data, as shown in Figure 2‑3.

Figure 2‑3 Interrupt Moderation

4) On “Interrupt Moderation Rate” window, set value to “OFF”, as shown in Figure 2‑4.

Figure 2‑4 Interrupt Moderation Rate

5) Click “OK” button to save and exit all setting windows.

2.3 Power Option Setting

1) Open Control Panel and select Power Options as shown in the left window of Figure 2‑5.

2) Change setting to High Performance as shown in the right window of Figure 2‑5.

Figure 2‑5 Power options

3 Test result when using FPGA and TestPC

3.1 Display TCPIP parameters

Select ‘0’ to check current parameters in the demo. There are six parameters in Client/Server mode or seven parameters in Fixed MAC mode displayed on the console. To run with TestPC, it is recommended to initialize TOE10GLL-IP in Client mode.

Figure 3‑1 Display current parameters

1) Mode: Set mode to TOE10GLL-IP to initialize in Server, Client, or Fixed MAC. To run with PC, please input ‘0’ to initialize the IP in client mode.

2) FPGA MAC address: 12 digits of hex value to be MAC address of FPGA. Default value is 0x000102030405.

3) FPGA IP: IP address of FPGA. Default value is 192.168.7.42.

Note: This value is used to be server IP address parameter for test application on PC.

4) FPGA port number: Port number of FPGA. Default value is 60000.

Note: This value is used to be server port for test application on PC.

5) Target MAC address (displayed when running Fixed MAC mode only): 12 digits of hex value to show MAC address of the destination device. Default value is 0x554433221100.

6) Target IP: IP address of destination device (10 Gb Ethernet on PC). Default value is 192.168.7.25.

7) Target port number: Port number of the destination device to transfer 10 Gb Ethernet data. Default value is 60001.

To change some parameters, user can set by using menu [1].

3.2 Reset TCPIP parameters

Select ‘1’ to reset the IP and change IP parameters.

This menu is run to reset the IP with or without updating parameters. After user selects this menu, the current parameters are displayed on the console. User enters ‘x’ to use the current value or enters other keys to change some parameters. After the parameters are updated, TOE10GLL-IP is reset and starts the initialization process.

There are six or seven parameters to set in this menu. Each parameter is verified by CPU. The parameter is updated to TOE10GLL-IP when the input is valid. If the input is not valid, the parameter does not change. The description of each parameter is shown in topic 3.1(Display TCPIP parameter) and the range of each parameter is described as follows.

1) Mode: Input ‘0’ to initialize the IP as client mode.

2) FPGA MAC address: Input 12 digits of hex value. Add “0x” as a prefix to input as hex value.

3) FPGA IP address: A set of four decimal digits is separated by “.”. The valid range of each decimal digit is 0-255.

4) FPGA port number: Valid range is 0-65535.

5) Target MAC address (displayed when running Fixed MAC mode only): Input 12 digits of hex value. Add “0x” as a prefix to input as hex value.

6) Target IP address: A set of four decimal digits, similar to FPGA IP address. This value is IP address of Test PC.

7) Target port number: Valid range is 0-65535.

After finishing parameter assignment, new parameter set is displayed on the console. Next, reset signal is sent to the IP to use new parameter set. Finally, “IP initialization complete” is shown after IP completes initialization process, as shown in Figure 3‑2.

Figure 3‑2 Change IP parameter

3.3 Send Data Test

To transfer data from FPGA to PC, select ‘2’ to run send data test on FPGA and run “tcpdatatest.exe” on PC to receive data. User inputs test parameters for sending data on the console. On PC, user inputs test parameters of “tcpdatatest” to receive data via Command prompt. The sequence to run the test is shown as follows.

1) On FPGA console, input four parameters under send data test menu.

a) Input transfer size: Unit of transfer size is byte. Valid value is 1 - 0xFFFF_FFFF_FFFF. The input is decimal unit when input only digit number. User adds “0x” to be a prefix for hexadecimal unit.

b) Input packet size: Unit of packet size is byte. Valid value is 1 – 1460. The input is decimal unit when input only digit number. User adds “0x” to be a prefix for hexadecimal unit.

c) Enable PSH: TCP flag to inform that the destination device should return acknowledgement packet for the latest transmitted data immediately. Input ‘0’ to disable PSH flag or ‘1’ to enable PSH flag.

Note: When packet size is very small, enabling PSH flag can accelerate PC to return ACK packet.

d) Input Mode: Connection mode of FPGA. Input ‘1’ to set server mode.

2) If all inputs are valid, the recommended parameters to run test application on PC is displayed. Next, “Wait Open connection …” is displayed to wait until the application on PC is run.

3) On Command prompt, input test parameters following the recommended value. There are six parameters for “tcpdatatest”.

>> tcpdatatest <mode> <dir> <server IP> <server port> <bytelen> <pattern>

a) Mode: Input ‘c’ to set connection mode of Test PC to be a client.

b) Dir: Input ‘r’ to run Test PC for receiving test data from FPGA

c) Server IP: Input the same value as IP address of FPGA

d) Server port: Input the same value as port number of FPGA

e) Bytelen: Input the same value as “Input transfer size” of step 1a)

f) Pattern: Input ‘1’ to verify data from FPGA or ‘0’ to not verify data

4) After running the test application, the port is created. Current amount of transferred data is displayed on the console (transmitted data) and Command prompt (received data) every second. “Send data complete” is displayed on the console after all data are sent.

5) FPGA closes the connection. Finally, total transfer size, performance, and latency time are displayed on the console (transmit test results) as well as total transfer size and performance are displayed on the Command prompt (receive test results).

Figure 3‑3 shows the example of send data test. The left window is FPGA console operating as server while the right window is Command prompt on PC operating as client.

Figure 3‑3 Send data test (Cut-through mode)

If some inputs are invalid, “Out-of-range input” or “Invalid input” is displayed. After that, the operation is cancelled, as shown in Figure 3‑4 - Figure 3‑7.

Figure 3‑4 Error from invalid transfer size

Figure 3‑5 Error from invalid packet size

Figure 3‑6 Error from invalid input to enable/disable PSH flag

Figure 3‑7 Error from invalid mode

3.4 Receive Data Test

To transfer data from PC to FPGA, select ‘3’ to run receive data test on FPGA and run “tcpdatatest.exe” on PC to send data. User inputs test parameters on FPGA for receiving data on FPGA console. On PC, user inputs test parameters of “tcpdatatest” to send data on Command prompt. The sequence to run the test is shown as below.

1) On FPGA console, input three parameters in receive data test.

b) Input data verification mode: Set ‘0’ to disable data verification or ‘1’ to enable data verification sent from PC.

c) Input Mode: Connection mode of FPGA. Input ‘1’ to set server mode.

2) If all inputs are valid, the recommended parameters to run test application on PC are displayed. Next, “Wait Open connection …” is displayed to wait until the application on PC is run.

3) On Command prompt, input test parameters following the recommended value. There are six parameters for “tcpdatatest”.

>> tcpdatatest <mode> <dir> <server IP> <server port> <bytelen> <pattern>

a) Mode: Input ‘c’ to set connection mode of Test PC to be a client.

b) Dir: Input ‘t’ to run Test PC for sending test data to FPGA

c) Server IP: Input the same value as IP address of FPGA

d) Server port: Input the same value as port number of FPGA

e) Bytelen: Input the same value as “Input transfer size” of step 1a)

f) Pattern: Input the same value as “Input data verification mode” of step 1b). Select ‘0’ to send dummy data or ‘1’ to send incremental data.

Note: When PC sends the real data, performance will be less than dummy data. So, test performance result when disabling data verification and using dummy data is better than enabling data verification and using real data.

4) After running the test application, the port is created. Current amount of transferred data is displayed on FPGA console (received data) and Command prompt (transmitted data) every second.

5) “Connection closed” and “Receive data completed” are displayed on FPGA console after PC finishes sending all data and closing the connection. Finally, total transfer size, performance and latency time are displayed on the console (transmit results) as well as total transfer size and performance are displayed on the Command prompt (receive results).

Figure 3‑8 shows the example of receive data test when data verification mode on FPGA is disabled and dummy data is sent by PC. The left window is test result on FPGA console while the right window is test result on Command prompt.

Figure 3‑9 shows the example of receive data test when data verification mode on FPGA is enabled and incremental data is sent by PC.

Figure 3‑10 shows the example of error when data verification is failed. In the example, the error is caused from mismatch verification mode value. FPGA enables data verification while “tcpdatatest” sends dummy data. The error message is displayed on FPGA console.

Figure 3‑8 Receive data test without data verification

Figure 3‑9 Receive data test when enabling data verification

Figure 3‑10 Receive data test when data verification is failed

3.5 Full duplex Test

Select ‘4’ to run full duplex test to transfer data between FPGA and PC in both directions at the same time. User inputs test parameters on FPGA console and on PC Command prompt. “tcp_client_txrx_40G” application is called on PC to send and receive data by using the same port number. The sequence to run the test is shown as below.

1) On FPGA console, input five parameters in full duplex test.

b) Input packet size: Unit of packet size is byte. Valid value is 1 – 1460. The input is decimal unit when input only digit number. User adds “0x” to be a prefix for hexadecimal unit.

Note: When packet size is very small, enabling PSH flag can accelerate PC to return ACK packet.

d) Input data verification mode: Set ‘0’ to disable data verification or ‘1’ to enable data verification sent from PC.

e) Input Mode: Connection mode of FPGA. Input ‘1’ to set server mode.

2) If inputs are valid, the recommended parameters to run test application on PC are displayed. Next, “Wait Open connection …” is displayed to wait until the application on PC is run.

3) On Command prompt, input test parameters following the recommended value. There are four parameters for “tcp_client_txrx_40G”.

>> tcp_client_txrx_40G <server IP> <server port> <bytelen> <pattern>

a) Server IP: Input the same value as IP address of FPGA

b) Server port: Input the same value as port number of FPGA

c) ByteLen: Total transfer size in byte unit. Input the same value as “Input transfer size” of step 1a).

d) Pattern: Input the same value as “Input data verification mode” of step 1b).

Select ‘0’ to send dummy data and not verify received data.

Select ‘1’ to send incremental data and verify received data.

4) After running the test application, the port is created. During transferring data, current amount of transferred data is displayed on FPGA console and Command prompt every second.

5) “Send data complete” is displayed on FPGA console after finishing sending data, receiving data, and closing the connection. Finally, total transfer size, performance, and latency time are displayed on the console as well as total transfer size and performance are displayed on the Command prompt.

Repeat Step 4) – 5) as forever loop until the user cancels the operation by pressing input “Ctrl+C” on Command prompt and then enters any keys on FPGA console.

As shown in Figure 3‑11- Figure 3‑12, since running the application without data verification takes less PC resource, transfer performance of running full duplex without data verification shows better performance than the test with data verification. The left window is the test result on FPGA console while the right window is the test result on Command prompt.

Figure 3‑11 Full duplex test without data verification

Figure 3‑12 Full duplex test with data verification

3.6 Ping Test

For Ping command test, use can type “ping <FPGA IP address>” command on Command prompt to start the test. ICMP Echo request is generated from PC and then FPGA returns ICMP Echo reply to PC. Finally, the result to show round-trip time is displayed on Command prompt, as shown in Figure 3‑13.

Figure 3‑13 Ping command result on PC

4 Test result when using two FPGAs

4.1 Display TCPIP parameter

Select ‘0’ to check current parameters in the demo. There are six parameters in Client/Server mode or seven parameters in Fixed MAC mode displayed on the console.

Figure 4‑1 Display current parameters

1) Mode: Set mode to TOE10GLL-IP to initialize in Server, Client, or Fixed MAC. Input ‘0’ for Client, ‘1’ for Server, or ‘2’ for Fixed MAC.

2) FPGA MAC address: 12 digits of hex value to be MAC address of FPGA. Default value is 0x000102030405 (Client and Fixed MAC mode) or 0x001122334455 (Server mode).

3) FPGA IP: IP address of FPGA. Default value is 192.168.7.42 (Client and Fixed MAC mode) or 192.168.7.25 (Server mode).

4) FPGA port number: Port number of FPGA. Default value is 60000 (Client and Fixed MAC mode) or 60001 (Server mode).

5) Target MAC address (displayed when running Fixed MAC mode only): 12 digits of hex value to show MAC address of the destination device.

6) Target IP: IP address of destination device to transfer 10 Gb Ethernet data. Default value is 192.168.7.25 (Client and Fixed MAC mode) or 192.168.7.42 (Server mode).

7) Target port number: Port number of the destination device to transfer 10 Gb Ethernet data. Default value is 60001 (Client and Fixed MAC mode) or 60000 (Server mode).

To change some parameters, the user runs menu [1] (Reset TCPIP parameters).

4.2 Reset TCPIP parameters

Select ‘1’ to reset the IP and change IP parameters.

This menu is run to reset the IP with or without updating parameters. After user selects this menu, the current parameters are displayed on the console. User enters ‘x’ to use the same parameters while other keys are entered for changing some parameters. After the parameters are updated, TOE10GLL-IP is reset and starts the initialization process.

There are six or seven parameters to set in this menu. Each parameter is verified by CPU. The parameter is updated to TOE10GLL -IP when the input is valid. If which input is not valid, the parameter does not change. After user inputs all parameters, the IP is reset. The description of each parameter is shown in topic 4.1(Display TCPIP parameter) and the range of each parameter is described as follows.

Note:

1. When user needs to reset parameters on the server FPGA, the client FPGA must be also reset. The server must be reset before the client to wait until ARP request sent from the client.

2. Parameter of client and server must be matched, i.e.

a. Target IP of server = FPGA IP of client

b. FPGA IP of server = Target IP of client

c. Target port number of server = FPGA port number of client

d. FPGA port number of server = Target port number of client

1) Mode: Input ‘0’ (Client), ‘1’ (Server), or ‘2’ (Fixed MAC) to determine FPGA initialization mode. The conditions of mode setting between two FPGA boards are as follows.

a. If the first board is set to Server mode, another board must be set to Client mode.

b. If the first board is set to Fixed MAC mode, another board can be set to Fixed MAC mode or Client mode.

2) FPGA MAC address: Input 12 digits of hex value. Add “0x” as a prefix for hex input.

3) FPGA IP address: A set of four decimal digits is separated by “.”. The valid range of each decimal digit is 0-255.

4) FPGA port number: Valid range is 0-65535.

5) Target MAC address (displayed when running Fixed MAC mode only): Input 12 digits of hex value. Add “0x” as a prefix to input as hex value.

6) Target IP address: A set of four decimal digits, similar to FPGA IP address.

7) Target port number: Valid range is 0-65535.

After finishing all parameter assignment, the new parameters are displayed on the console. Next, reset signal is sent to the IP and then the IP runs initialization by using new parameters. Finally, “IP initialization complete” is shown after IP completes initialization process, as shown in Figure 4‑2 and Figure 4‑3.

Figure 4‑2 Change IP parameters when running Server and Client mode

Figure 4‑3 Change IP parameters when running Fixed MAC mode

4.3 Send Data Test (Server to Client)

To transfer the data, the user needs to set different connection mode on two FPGA. The first board is connected by Server mode while another board is connected by Client mode. This topic shows how to set the parameters for transferring data from Server mode to Client mode.

Note: Connection mode (Server or Client) is not related to the initialization mode (Server, Client, or Fixed MAC).

Select ‘2’ to run send data test on the server FPGA and select ‘3’ to run receive data test on the client FPGA. User inputs test parameters on FPGA console. The sequence to run the test is shown as below.

1) On Server console, input four parameters in send data test.

b) Input packet size: Unit of packet size is byte. Valid value is 1 – 1460. The input is decimal unit when input only digit number. User adds “0x” to be a prefix for hexadecimal unit.

d) Input Mode: Connection mode of FPGA. Input ‘1’ to set Server mode.

2) If all inputs are valid, “Wait Open connection …” is displayed.

3) On Client console, input three test parameters in receive data test.

b) Input data verification mode: Set ‘0’ to disable data verification or ‘1’ to enable data verification to verify data from FPGA running as Server. It is recommended to set ‘1’ to verify data from Server.

c) Input Mode: Connection mode of FPGA. Input ‘0’ to set Client mode.

After all inputs are valid, the operation begins.

4) After Client is run, current amount of transferred data is displayed on both consoles every second. “Send data complete” is displayed on Server console after finishing sending all data.

5) Server closes the connection. Finally, total transfer size, performance, and latency time are displayed on both consoles.

Figure 3‑4 shows the example of send data test. The left window is Server console and the right window is Client console.

If some inputs are invalid, “Out-of-range input” or ”Invalid input” is displayed and then the operation is cancelled, as shown in Figure 3‑4- Figure 3‑7 (similar to FPGA<->PC test).

Figure 4‑4 Send data test (Cut-through mode)

4.4 Receive Data Test (Client to Server)

To transfer data from Client (connection mode) to Server (connection mode), select ‘3’ to run receive data test on the server FPGA and select ‘2’ to run send data test on the client FPGA. User inputs test parameters on FPGA console. The sequence to run the test is shown as below.

Note: Connection mode (Server or Client) is not related to the initialization mode (Server, Client, or Fixed MAC).

1) On Server console, input three parameters in receive data test.

b) Input data verification mode: Set ‘0’ to disable data verification or ‘1’ to enable data verification to verify data from FPGA running as client. It is recommended to set ‘1’ to verify data from client.

c) Input Mode: Connection mode of FPGA. Input ‘1’ to set Server mode.

2) If all inputs are valid, “Wait Open connection …” is displayed.

3) On Client console, input four test parameters in send data test.

b) Input packet size: Unit of packet size is byte. Valid value is 1 – 1460. The input is decimal unit when input only digit number. User adds “0x” to be a prefix for hexadecimal unit.

d) Input Mode: Connection mode of FPGA. Input ‘0’ to set Client mode.

After all inputs are valid, the operation begins.

4) After Client is run, current amount of transferred data is displayed on both consoles every second.

5) “Connection closed” and “Receive data completed” are displayed on Server console after Client finishes sending all data and closing the connection. Finally, total transfer size, performance, and latency time are displayed on both consoles.

Figure 4‑5 Receive data test with data verification

4.5 Full duplex Test

Select ‘4’ to run full duplex test on Server FPGA and Client FPGA to transfer data in both directions at the same time. User inputs test parameters on FPGA console. The sequence to run the test is shown as below.

1) On Server console, input five parameters in full duplex test.

a) Input transfer size: Unit of transfer size is byte. Valid value is 1 - 0xFFFF_FFFF_FFFF. The input must match with transfer size, set in FPGA run as Client. The input is decimal unit when input only digit number. User adds “0x” to be a prefix for hexadecimal unit.

b) Input packet size: Unit of packet size is byte. Valid value is 1 – 1460. The input is decimal unit when input only digit number. User adds “0x” to be a prefix for hexadecimal unit.

d) Input data verification mode: Set ‘0’ to disable data verification or ‘1’ to enable data verification to verify data from FPGA run as Client. It is recommended to set ‘1’ to verify data from Client.

e) Input Mode: Connection mode of FPGA. Input ‘1’ to set Server mode.

2) If all inputs are valid, “Wait open connection…” is displayed.

3) On Client console, input five test parameters in full duplex test.

a) Input transfer size: Unit of transfer size is byte. Valid value is 1 - 0xFFFF_FFFF_FFFF. The input must match with transfer size in FPGA running as Server. The input is decimal unit when input only digit number. User adds “0x” to be a prefix for hexadecimal unit.

b) Input packet size: Unit of packet size is byte. Valid value is 1 – 1460. The input is decimal unit when input only digit number. User adds “0x” to be a prefix for hexadecimal unit.

d) Input data verification mode: Set ‘0’ to disable data verification or ‘1’ to enable data verification to verify data from FPGA running as server. It is recommended to set ‘1’ to verify data from server.

e) Input Mode: Connection mode of FPGA. Input ‘0’ to set Client mode.

After all inputs are valid, the operation begins.

4) After Client is run, current amount of transferred data is displayed on both consoles every second.

5) “Send data complete” is displayed on Client console after finishing sending all data, receiving all data, and closing connection. Finally, total transfer size, performance, and latency time are displayed on both consoles.

Repeat step 4) – 5) as forever loop until the user cancels the operation by pressing any keys on Server console and Client console.

Figure 4‑6 shows full duplex test. The left window is console of the server FPGA and the right window is console of the client FPGA. When running by two FPGAs, the performance with and without data verification are similar.

Figure 4‑6 Full duplex test with data verification

5 Revision History

Revision	Date	Description
1.1	4-Apr-22	Update latency time and add Ping test
1.0	20-Nov-20	Initial version release