FPGA setup for TOE/UDP100G -IP with CPU Demo

Rev3.3 31-May-23

2 Test environment setup when using FPGA and PC. 4

3 Test environment setup when using two FPGAs. 21

1 Overview

This guide provides instructions on how to setup FPGA board and create a test environment for running the TOE100G-IP/UDP100G-IP demo or multi-session demo. The user has the option to create two test environments for transferring TCP/UDP payload data via a 100G Ethernet connection using either TOE100G-IP or UDP100G-IP. Figure 1‑1 illustrates these two options.

Figure 1‑1 Two test environments for running the demo

The first test environment requires one FPGA board and a PC with a 100G Ethernet card for data transfer. The PC runs a test application, such as “tcpdatatest” (half-duplex test for TOE100G-IP), “tcp_client_txrx_xg” (full-duplex test for TOE100G-IP), “tcp_client_txrx_single” (full-duplex test for multisession TOE100G-IP), or “udpdatatest” (test application for UDP100G-IP). The Serial console or JTAG Terminal is also run on the PC to act as the user interface console.

The second test environment involves two FPGA boards which may be different from each other. Both boards run the TOE100G-IP, multi-session of TOE100G-IP, or UDP100G-IP demo, with different initialization mode assigned (Client, Server, or Fixed-MAC).

The demo is implemented on multiple FPGA boards, each with different settings for RS-FEC features. For specific information of the RS-FEC settings one ach FPGA board, please refer to the table provided below.

Table 1‑1 The features of RS-FEC for TOE100G-IP/UDP100G-IP demo on each board

Board name	RS-FEC feature
TOE100G-IP demo
KCU116	Enabled
ZCU111	Enabled
VCK190	Enabled
UDP100G-IP demo
KCU116	Disabled

2 Test environment setup when using FPGA and PC

Before running the demo using an FPGA and PC, please prepare the following.

FPGA development boards: VCK190 board (with micro SD for SysCont), ZCU111, KCU116, Alveo U250 card, and FB2CGHH@KU15P card
PC with a 100 Gigabit Ethernet card installed
100G Ethernet cable:

a) KCU116 and ZCU111: 4xSFP28 transceiver (25GBASE-SR), QSFP28 transceiver (100GASE-SR), and MTP to 8xLC Fiber cable

b) Others: 2xQSFP28 transceiver (100GASE-SR) and MPO to MPO cable

USB cable for connecting the FPGA to the PC

a) VCK190: a USB type-C cable for programming FPGA and Serial console

b) KCU116: two micro USB cables for programming FPGA and Serial console

c) U250 card and ZCU111: a micro USB cable for programming FPGA and Serial console

d) FB2CGHH@KU15P card: a mini USB cable for programming FPGA and JTAGUART

For FB2CGHH@KU15P card, use AB18-PCIeX16 board provided by Design Gateway with ATX power supply to be power board for the card. More details of AB18 card are displayed on following website.

https://dgway.com/ABseries_E.html

Test application provided by Design Gateway for running on PC

a) TOE100G-IP: “tcpdatatest.exe” and “tcp_client_txrx_xg.exe”

b) Multi-session of TOE100G-IP: “tcpdatatest.exe” and “tcp_client_txrx_single.exe”

c) UDP100G-IP: “udpdatatest.exe”

Serial console software (except FB2CGHH@KU15P card) such as TeraTerm installed on PC. The console setting is Baudrate=115,200, Data=8-bit, Non-parity, and Stop=1-bit.

Note: FB2CGHH@KU15P card uses JTAG Terminal instead of Serial console.

Vivado tool for programming FPGA and JTAG Terminal (when using FB2CGHH@KU15P card), installed on PC

Note: The hardware listed below is an example for running the demo.

[1] 100G Network Adapter: Nvidia MCX614106A-CCAT

https://docs.nvidia.com/networking/display/ConnectX6EN

[2] 100G Ethernet cable

a) QSFP28 to QSFP28 connection (other than KCU116 and ZCU111)

QSFP28 Transceiver: AMQ28-SR4-M1

https://www.sfpcables.com/100G-s-qsfp28-sr4-optical-transceiver-module-1499

MPO to MPO cable: OM4-MPO-8MPO-1M

https://www.sfpcables.com/mpo-to-mpo-multimode-om4-50-125-m-8-core-4381

b) 4xSFP28 to QSFP28 connection (KCU116 and ZCU111)

SFP28 Transceiver: AZS85-S28-M1

https://www.sfpcables.com/25gb-s-sfp28-sr-transceiver-850nm-up-to-100m-2866

QSFP28 Transceiver: AMQ28-SR4-M1

https://www.sfpcables.com/100G-s-qsfp28-sr4-optical-transceiver-module-1499

MTP to 8xLC Fiber cable: OM4-MTP-8LC-1M

https://www.fs.com/products/68047.html

[3] Test PC:

Motherboard: Gigabyte Z590 AORUS MASTER (rev. 1.0)

CPU: Intel i7-11700K CPU 3.6 GHz

RAM: 64 GB DDR4

OS: 64-bit Windows10 OS

Figure 2‑1 Demo (FPGA <-> PC) on KCU116

Figure 2‑2 Demo (FPGA <-> PC) on FB2CGHH@KU15P

Figure 2‑3 Demo (FPGA <-> PC) on U250 card

Figure 2‑4 Demo (FPGA <-> PC) on ZCU111 board

Figure 2‑5 Demo (FPGA <-> PC) on VCK190 board

The steps for setting up a test environment using an FPGA board and a PC are described below.

1) Set up the FPGA board/card.

a) VCK190 board: Insert a micro SD card into the system controller’s SD card socket (J206). Then, set DIP switch (SW11) to be “ON OFF OFF OFF” to boot from SysCont SD.

Note: For more detailed instructions on setting up the SD card, refer to “Board Setup and Connection” and “Writing the image to micro SD card” topics on the following website.

https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/973078551/BEAM+Tool+for+VCK190+Evaluation+Kit#Board-Setup-and-Connection

Figure 2‑6 VCK190 board setting

b) ZCU111 board: Set DIP switch (SW6) to be “ON ON ON ON” to boot from JTAG.

Figure 2‑7 ZCU111 board setting

c) Other boards: No special requirement are needed.

2) Connect USB cables from the FPGA board to PC for JTAG programming and Serial console/JTAGUART.

a) KCU116 board: Connect two micro USB cables

b) VCK190 board: Connect a USB type-C cable

c) U250 card and ZCU111 board: Connect a micro USB cable

d) FB2CGHH@KU15P card: Connect a mini USB cable

3) Connect the power supply to the FPGA development board/FPGA accelerator card.

a) KCU116 board: Connect Xilinx power adapter 60W (12V)

b) ZCU111 board: Connect Xilinx power adapter 80W (12V)

c) VCK190 board: Connect Xilinx power adapter 180W (12V)

d) U250 card: Connect the card to Motherboard

e) FB2CGHH@KU15P card: Connect the card to PC or AB18-PCIeX16 board as below.

i) Confirm that two mini jumpers are inserted at J5 connector on AB18.

ii) Connect ATX power supply to AB board.

iii) Connect PCIe connector on FPGA board to Device Side (B-Side), as shown in Figure 2‑8.

Figure 2‑8 Set up AB18-PCIeX16 Connection for FB2CGHH@KU15P card

4) Connect the FPGA board to the PC via a 100G Ethernet cable.

a) KCU116 and ZCU111 boards: Insert four SFP28 transceivers and MTP to 8xLC fiber cable into the FPGA board. Check channel number of the four cables to make sure they match with Figure 2‑9.

Figure 2‑9 100G connection on KCU116 and ZCU111 board by QSFP28 to 4xSFP28 cable

b) Other boards: Use a 100G QSFP28 Transceiver and MPO to MPO cable. Connect them using the QSFP(1) connector, as shown in Figure 2‑10.

Figure 2‑10 100G connection on other boards by QSFP28 transceiver and MPO to MPO cable

5) Connect the other end of the MTP/MPO cable by inserting the QSFP28 transceiver to the 100G Ethernet card on PC.

6) Power on the FPGA board. For VCK190 board, please follow these steps.

i) Connect the VCK board to the PC using a USB cable. The PC should detect three USB Serial Ports. Select the third port to check the board boot-up message.

ii) Open a Serial console and connect to the third USB Serial port using the following parameters: Baud rate=115,200, Data=8-bit, Non-parity, and Stop bits=1-bit.

Figure 2‑11 Serial console for boot-up on VCK190

iii) Power on the board and wait until the boot message on the console is completed.

Figure 2‑12 Boot-up message of power-on sequence on VCK190

7) Open Serial console (except FB2CGHH@KU15P card which uses JTAG-Terminal). When connecting FPGA board to PC, multiple COM ports from FPGA connection are detected and displayed on Device Manager.

a) KCU116: Select standard COM port.

b) VCK190 and ZCU111: Select the first USB Serial port.

c) U250 card: Select the second USB Serial port.

Use following setting on the Serial console: Baud rate=115,200, Data=8-bit, Non-Parity, and Stop=1.

Figure 2‑13 Port for Serial console

8) To configure the programmable clock for the FPGA board, determine whether your board requires a programmable clock setup. If board has already used the desired clock frequency, you do not need to re-configure it.

a) KCU116 board: Set the programmable clock to 156.250 MHz (Default).

Figure 2‑14 Program the frequency of the reference clock for KCU116

b) ZCU111 board: Set the programmable clock to 156.250 MHz (Default).

Figure 2‑15 Program the frequency of the reference clock for ZCU111

c) VCK190 board: Set the programmable clock to 322.265625 MHz.

Figure 2‑16 Program the frequency of the reference clock for VCK190

d) Other boards: No action is required to set up the programmable clock.

9) Download configuration file and firmware to the FPGA board/Accelerator card using Vivado tool or Script file, depending on the board.

a) KCU116 board and U250 card: Use Vivado tool to program configuration file (bit file), as shown in Figure 2‑17.

Figure 2‑17 Program FPGA by Vivado

b) VCU190 and ZCU111 boards: Use the Vivado TCL shell to download the configuration file and the firmware. Browse to the download directory that includes bat file, pdi file, and elf file and type the command to run bat file.

Figure 2‑18 Download demo file on Vivado TCL shell by script file on VCK190/ZCU111

c) FB2CGHH@KU15P card: Use the vivado TCL shell to download the configuration file and firmware. Browse to the download directory that includes bat file, bit file, and elf file of and type the following command.

i) >>TOE100CPUTest_Silicom.bat/UDP100CPUTest_KU15P.bat

Note: This step is to download configuration file and firmware, as shown in Figure 2‑19.

Figure 2‑19 Command script to download demo file on Vivado TCL shell

ii) >> xsdb.bat

iii) >> connect -url tcp:127.0.0.1:3121

iv) >> targets -set -filter {name =~"*Debug*"}

v) >> jtagterminal -start

vi) >> con

Note: Upon all above steps are completed, JTAGUART module is connected and JTAG terminal is executed to be the user console, as shown in Figure 2‑20.

Figure 2‑20 Open JTAG Terminal

10) Upon opening the Serial console/JTAG Terminal, welcome message will be displayed.

i) Input ‘0’ to initiate the TOE100G-IP/UDP100G-IP in Client mode that asks for the PC MAC address through sending ARP request packet.

ii) The default parameter in Client mode will be shown on the console.

Figure 2‑21 Message after system boot-up

However, if there is an Ethernet connection problem and the status is linked down, an error message will be displayed instead of the welcome message, as shown in Figure 2‑22.

Figure 2‑22 Error message when ethernet connection link down

iii) If the user wishes to skip parameter setting and use default parameters to start the system initialization, input ‘x’ as shown in Figure 2‑23. If any other keys are entered, the menu for changing parameter will appear, similar to the “Reset TCPIP/UDPIP parameters” menu. The examples of running the main menu of TOE100G-IP, Multisession of TOE100G-IP, and UDP100G-IP are described in

“dg_toe100gip_cpu_instruction”, “dg_toe100gip_4ss_instruction”, and

“dg_udp100gip_instruction” documents, respectively.

Figure 2‑23 Initialization complete

Note: Transfer performance in the demo is limited by Test PC performance. The best performance can be achieved when the test is run using FPGA-to-FPGA connection.

3 Test environment setup when using two FPGAs

Before running the test, please prepare following test environment.

Two FPGA development boards, which can be either the same or different boards: VCK190 (with micro SD for SysCont) board, ZCU111 board, KCU116 board, Alveo U250 card, and FB2CGHH@KU15P card

100G Ethernet cable: 100G transceiver (BASE-SR) and fiber cable

a) KCU116 and ZCU111: Use 4xSFP28 transceiver (25GBASE-SR) with 8xLC Fiber cable

b) Others: Use QSFP28 transceiver (100GASE-SR) with MTP/MPO Fiber cable

USB cable for connecting from the FPGA to the PC

a) VCK190: a USB type-C cable for programming the FPGA and Serial console

b) KCU116: two micro USB cables for programming the FPGA and Serial console

c) U250 card and ZCU111: a micro USB cable for programming the FPGA and Serial console

d) FB2CGHH@KU15P card: a mini USB cable for programming the FPGA and JTAGAURT

Serial console software (except FB2CGHH@KU15P card) such as TeraTerm, installed on PC. The console setting is Baudrate=115,200, Data=8-bit, Non-parity, and Stop=1-bit.

Note: FB2CGHH@KU15P card uses JTAG Terminal instead of Serial console.

Vivado tool for programming the FPGA and JTAG Terminal (when using FB2CGHH@KU15P card), installed on PC

Figure 3‑1 TOE100G-IP/UDP100G-IP with CPU demo (FPGA<->FPGA)

The steps for setting up a test environment using two FPGAs are described below.

To get started with the demo, follow steps 1) – 9) of topic 2 (Test environment setup when using FPGA and PC) to set up the FPGA board and SFP28/QSFP28 connection. Once you have completed the configuration for two FPGA boards, a menu will be displayed on the Serial console/JTAG for selecting Client mode, Server mode, or Fixed MAC mode. Follow the detailed steps below to continue the demo.

1) Open the Serial console/JTAG Terminal for FPGA board#1 and FPGA board#2, which are set to initialize in Server/Client/Fixed-MAC mode. An example to initialize by Server-Client mode is below.

i) Set ‘1’ on the console of FPGA board#1 for running in Server mode.

ii) Set ‘0’ on the console of FPGA board#2 for running in Client mode.

iii) The default parameters for the selected mode will be displayed on the console, as shown in Figure 3‑2.

Note: The rules for setting the initialization mode are below.

· If the first board is initialized in Server mode, the other board must be initialized in Client mode.

· If the first board is initialized in Fixed MAC mode, the other board can be run in Client mode or Fixed MAC mode.

Figure 3‑2 Input mode

2) Input ‘x’ to use default parameters or use other keys to change parameters. The parameters of Server mode must be set before Client mode. The details are divided into two parts, running the TOE100G-IP demo and running the UDP100G-IP demo.

When running single-session or multi-session of TOE100G-IP,

i) Set parameters on the Server console (board#1 console).

ii) Set parameters on the Client console (board#2 console) to start IP initialization by transferring ARP packet.

iii) After finishing the initialization process, “IP initialization complete” and the main menu are displayed on the Server console and Client consoles.

Figure 3‑3 Main menu of TOE100G-IP (Single-session)

Figure 3‑4 Main menu of Multi-session TOE100G-IP

When running UDP100G-IP,

i) For Server mode (board#1 console), if user does not change the default parameters, input ‘x’ to skip parameter setting.

ii) For Client mode, the user must change Target port number (Target->FPGA) to use same value as Target port number (FPGA->Target).

iii) After finishing initialization process, “IP initialization complete” and the main menu will be displayed on the Server and Client consoles.

Figure 3‑5 Main menu of UDP100G-IP

4 Revision History

Revision	Date	Description
3.3	25-May-23	Add RS-FEC feature lists on each demo
3.2	17-May-23	Support ZCU111 board, add setting clock frequency, and update full-duplex test application from “tcp_client_txrx_40G” to “tcp_client_txrx_xg”.
3.1	12-Jul-22	Support VCK190 board
3.0	21-Mar-22	Add multi-session demo
2.0	4-Aug-21	Add UDP100G-IP
1.1	28-Apr-21	Support Silicom FB2CGHH@KU15P card
1.0	25-Feb-21	Initial version release