FPGA setup for TOE100G-IP on Silicom NIC

Rev1.0　7-Jul-23

 

1      Overview. 1

2      Silicom NIC with TOE100G-IP setup. 3

3      Test environment setup when the target uses 100G standard NIC  9

4      Test environment setup when the target uses TOE100G-IP on FPGA. 10

5      Revision History. 13

 

 

1       Overview

 

This document describes how to setup Silicom fb2CGhh@KU15P NIC that is plugged-in to the host system and how to prepare the test environment for running TOE100G-IP with DMA feature demo on the Silicom NIC. PacketMover FPGA Acceleration platform by Silicom is used as a based design of this demo. TOE100G-IP by Design Gateway is implemented in the user logic area of this platform. With our TOE100G-IP integration on the platform, it allows the standard NIC feature and also the high-speed TCP/IP on 100Gb connection through our IP.

 

 

Figure 1‑1 Two test environments for running the demo

 

 

Figure 1‑1 shows the test environment to transfer the data on 100G Ethernet of this demo system. The first side of 100G Ethernet is the host system with Silicom NIC that includes TOE100G-IP. While another side of 100G Ethernet is the target system which may be Test PC with standard NIC or TOE100G-IP on the FPGA platform. When the target system uses the standard NIC, the test performance is not good and less than peak bandwidth of 100G Ethernet connection. While using TOE100G-IP on FPGA card to be the target system, the test performance can show the peak performance of 100G Ethernet connection.

 

For the first test environment, the example software applications to run on the target system that uses standard NIC are also provided by Design Gateway on both Windows OS and Ubuntu OS, i.e., “tcpdatatest” for half-duplex test and “tcp_client_txrx_single” for full-duplex test. For the second test environment, there are many reference designs of TOE100G-IP that are run on many FPGA cards provided on our website. Please see more details from following links.

 

TOE100G-IP Standard demo on Xilinx standalone board

https://dgway.com/products/IP/TOE100G-IP/dg_toe100gip_cpu_refdesign_xilinx.pdf

 

TOE100G-IP Multi-session demo on Xilinx board

https://dgway.com/products/IP/TOE100G-IP/dg_toe100gip_cpu_4ss_refdesign_xilinx.pdf

 

In the document, topic 2 shows the procedure to setup the system with Silicom NIC plugged in and to run the demo on the card. Topic 3 shows the hardware setup on the target system by using the 100G standard NIC. Finally, topic 4 shows the hardware setup on the target system by using TOE100G-IP demo on FPGA card. More details of each topic are described as follows.

 

 

2       Silicom NIC with TOE100G-IP setup

 

Before getting started, please prepare the release stuff of “Silicom PacketMover platform” version 1.4.0. More details about the platform can be checked from Silicom website by the following link.

https://www.silicom.dk/product-details/packetmover-fpga-acceleration/

 

Follow the instruction that is described in “readme.txt” in the Silicom release stuff to build the driver, install the driver, and verify the standard NIC function of the Silicom card.

 

Note: After finishing Silicom card system preparation, please confirm that the logical name of two Ethernet connections on Silicom NIC are “feth0” and “feth1”, as described in topic “Driver Load and Verification” in “readme.txt”. The demo software uses this logical name for running the test.

 

Design Gateway provides an archive file which is needed for running the demo. Extract the file and move “download” folder to directory path, “/1_4_0/sw/PacketMover_SW_1_4_0/”. In “download” directory, it consists of two files – “multcp_dma_silicom_exe” (test application of TOE100G-IP demo on Silicom NIC) and “tivoli_wolfsburg_2x100g_dgTOEIP.bit” (FPGA configuration file for downloading to Silicom NIC).

 

The system with Silicom NIC consists of two hardwares, as shown in Figure 2‑1.

 

 

Figure 2‑1 Silicom fb2CGhh@KU15P installation to the host

 

 

The steps to download configuration file to Silicom NIC are described as follows.

 

 

Figure 2‑2 Program the bit file to FPGA

 

 

1)    Plug-in fb2CGhh@KU15P card to the host system and power on. After that, program configuration file of the demo to the Silicom card by using following step. This step is run once and can be skipped for the next run time.

 

                     i.       Change current directory to Silicom driver directory. The example extracts the stuff to home directory. Therefore, use following command to change directory.

>> cd 1_4_0/sw/PacketMover_SW_1_4_0/driver

                   ii.        Load Silicom PacketMover driver by using the Silicom script file.

>> sudo ./load_driver.sh

                  iii.       After loading the driver successfully, the complete message is displayed.

                  iv.       Change the driver directory to the bin directory by the following command.

>> cd ../bin

                   v.       Use “fbupdate” tool to update FPGA configuration file that is stored in

“1_4_0/sw/PacketMover_SW_1_4_0/download/” directory.

>> ./fbupdate --flash ../download/tivoli_wolfsburg_2x100g_dgTOEIP.bit

                  vi.       Wait until the configuration fie is updated successfully. Confirm that there is no error message displayed.

                vii.        Power off the system by using the following command.

>> sudo poweroff

 

After finishing this step, the configuration file is updated. Therefore, the updated configuration file will be loaded to FPGA when the system is powered-on next time.

 

2)    Insert QSFP28 Transceiver of 100G Ethernet cable  to QSFP0 connector on fb2CGhh card.

Note: QSFP#0 is the Ethernet channel that supports standard NIC feature and two high-speed TCP sessions by TOE100G-IP. While QSFP#1 is the Ethernet channel that supports only standard NIC feature.

 

 

Figure 2‑3 100G Ethernet connection on Silicom fb2CGhh card

 

 

3)    Insert another side of 100G Ethernet cable to the target system. After that, Ethernet hardware connection is ready for use.

4)    Power on the host system and then prepare the environment of Silicom NIC with TOE100G-IP by following step.

 

                     i.       Change current directory to Silicom driver directory by using the following command.

>> cd 1_4_0/sw/PacketMover_SW_1_4_0/driver

                   ii.        Load Silicom PacketMover driver using the Silicom script file.

>> sudo ./load_driver.sh

                  iii.       After loading the driver successfully, the completed message is displayed.

                  iv.       Display all network interfaces including Silicom NIC interfaces by using the following command.

>> ifconfig -a

                   v.       Check the Silicom network interfaces status. Two interfaces - “feth0” and “feth1” are found and the status is disabled.

                  vi.       Set the IP address to Silicom network interface#0 (“feth0”) and enable it by using the following command (IP address = “192.168.100.42” and a network mask = “255.255.255.0”).

>> sudo ifconfig 192.168.100.42/24

                vii.        After that, use “ifconfig” to check the network interfaces.

>> ifconfig

               viii.       Verify that Silicom network interface#0 status (“feth0”) is “UP” and “RUNNING”. Also, its IP address must be matched with the set value in step vi.

                  ix.       Allow ARP table to be updated by Silicom network interface#0 (“feth0”) by using the following command.

>> sudo sysctl -w net.ipv4.conf.feth0.arp_accept=1

                   x.       Verify that the enable flag is updated to 1.

 

 

Figure 2‑4 Prepare Silicom NIC with TOE100G-IP environment

 

 

5)    Run the demo application “multcp_dma_silicom_exe” on the Silicom NIC Host, as shown in Figure 2‑5.

 

                     i.       Change the current directory from the driver directory to download directory that stores the demo application.

>> cd ../download/

                   ii.        Run the demo application as a root by using following command.

>> sudo ./multcp_dma_silicom_exe

                  iii.       The welcome message and the default parameters of the demo application is displayed after the application is run.

 

Note: “multcp_dma_silicom_exe” is pre-built application that is stored in the “download” directory, provided by Design Gateway.

 

 

Figure 2‑5 Run the demo application on Silicom NIC Host

 

 

6)    Wait until the target system completes to prepare the 100G Ethernet connection. After that, enter ‘x’ to skip parameter setting for using default parameters to begin TOE100G-IP initialization, as shown in Figure 2‑6. If the other keys are entered, the menu for changing parameter is displayed, similar to “Reset TCPIP parameters” menu which is described in the demo instruction document.

 

 

Figure 2‑6 Initialization complete

 

 

7)    To exit the demo application, enter “CTRL+C” while there is no test run. After that, “Exiting the application!” is displayed and the application is terminated, as shown in Figure 2‑7.

 

Note: If “CTRL+C” is entered while the test operation is running, user must re-program the FPGA by using Silicom script file “reload_fw.sh” in driver directory.

>> ./reload_fw.sh

 

 

Figure 2‑7 Close the software application

 

 

3       Test environment setup when the target uses 100G standard NIC

 

Please prepare the following environment.

Note: The provided test applications are proved for running on Windows 10 OS and Ubuntu 20.04 LTS OS.

 

Example hardware lists for running the demo are shown as follows.

1)    100G Network Adapter: MCX614106A-CCAT

https://docs.nvidia.com/networking/display/ConnectX6EN

2)    100G Ethernet cable

QSFP28 Transceiver: AMQ28-SR4-M1 https://www.sfpcables.com/100gb-s-qsfp28-sr4-optical-transceiver-module-1499

MPO to MPO cable: OM4-MPO-8MPO-1M https://www.sfpcables.com/mpo-to-mpo-multimode-om4-50-125-m-8-core-4381

3)    Target system: Turnkey accelerator system (TKAS-D2101) with Ubuntu 20.04 LTS Server OS

https://dgway.com/AcceleratorCards.html

 

 

Figure 3‑1 Demo setup when the target uses standard NIC

 

 

Note: Silicom NIC is configured to enable RS-FEC, so the network device that connects to Silicom NIC on the target system must support RS-FEC.

 

 

4       Test environment setup when the target uses TOE100G-IP on FPGA

 

Please prepare the following environment to set up the target system by TOE100G-IP on FPGA. The step to set up the TOE100G-IP on FPGA board are described in following document.

https://dgway.com/products/IP/TOE100G-IP/dg_toeudp100gip_fpgasetup_xilinx.pdf

 

Example hardware lists for running the demo are shown as follows.

1)    FPGA with TOE100G-IP demo: KCU116 board

https://www.xilinx.com/products/boards-and-kits/ek-u1-kcu116-g.html

2)    100G Ethernet cable: 4xSFP28 to QSFP28 connection

SFP28 Transceiver: AZS85-S28-M1

https://www.sfpcables.com/25gb-s-sfp28-sr-transceiver-850nm-up-to-100m-2866

QSFP28 Transceiver: AMQ28-SR4-M1 https://www.sfpcables.com/100gb-s-qsfp28-sr4-optical-transceiver-module-1499

MTP to 8xLC Fiber cable: OM4-MTP-8LC-1M

https://www.fs.com/products/68047.html

 

 

Figure 4‑1 Demo setup when the target is TOE100G-IP on FPGA

 

 

Note: Silicom NIC is configured to enable RS-FEC, so the network device that connects to Silicom NIC on the target system must support RS-FEC.

 

The step to setup test environment when the target system uses FPGA board with TOE100G-IP demo are described in more details as follows.

 

1)    Prepare Silicom NIC environment following step 1) – 5) of topic 2 (Silicom NIC with TOE100G-IP setup).

2)    Prepare target TOE100G-IP demo system following step 1) – 8) of topic 2 (Test environment setup when using FPGA and PC) in the FPGA setup document.

https://dgway.com/products/IP/TOE100G-IP/dg_toeudp100gip_fpgasetup_xilinx.pdf

3)    Start the initialization process. The example to initialize the target system by using TOE100G-IP demo system in Server mode is described as follows.

                   i.        Default parameters of the Silicom NIC Host are displayed after running the application.

                  ii.        Set ‘1’ on Terminal of the Target TOE100G-IP demo system for running Server mode.

                iii.       Default parameters of the Server are displayed, as shown in Figure 4‑2.

 

 

Figure 4‑2 Input mode for system initialization

 

 

4)    Input ‘x’ to use default parameters or other keys to change parameters. The parameters of Server mode must be set before the other side starts initializing. Please run the following step to complete the initialization process, as shown in Figure 4‑3.

                     i.       Set parameters on Server terminal (Target TOE100G-IP demo system).

                   ii.        Set parameters on Client terminal (Silicom NIC with TOE100G-IP) to start IP initialization by transferring ARP packet.

                  iii.       After finishing initialization process, “IP initialization complete” and main menu are displayed on Silicom NIC Host.

                  iv.       Check Server console that “IP initialization complete” and main menu are displayed. If not, open the new Terminal on Silicom NIC Host and then use “arping” command to send ARP request from the Silicom NIC Host by following command.

>> sudo arping -c 1 <target IP address>

 

After that, the MAC address of the target (TOE100G-IP on Server) is displayed on the new Terminal. Also, the “IP initialization complete” and main menu are displayed on the Server console.

Note: TOE100G-IP initialization in Server mode waits for ARP request from the Client. The application on Silicom does not send ARP request if ARP table has this IP address information. Therefore, ARP sending manually is required in this condition.

 

 

Figure 4‑3 Main menu of TOE100G-IP demo

 

 

5       Revision History

 

Revision

Date

Description

1.0

16-Dec-22

Initial version release