FPGA setup for TOE/UDP40G -IP with CPU Demo

2 Test environment setup when using FPGA and PC. 3

3 Test environment setup when using two FPGAs. 12

1 Overview

This guide provides instructions on how to setup FPGA board and create a test environment for running the TOE40G-IP/UDP40G-IP demo. The user has the option to create two test environments for transferring TCP/UDP payload data via a 40G Ethernet connection using either TOE40G-IP or UDP40G-IP. Figure 1‑1 illustrates these two options.

Figure 1‑1 Two test environments for running the demo

The first test environment requires one FPGA board and a PC with a 40G Ethernet card for data transfer. The PC runs a test application, such as “tcpdatatest” (half-duplex test for TOE40G-IP), “tcp_client_txrx_xg” (full-duplex test for TOE40G-IP), or “udpdatatest” (test application for UDP40G-IP). The Serial console is also run on the PC to act as the user interface console.

The second test environment involves two FPGA boards which may be different from each other. Both boards run the TOE40G-IP or UDP40G-IP demo, with different initialization mode assigned (Client, Server, or Fixed-MAC).

2 Test environment setup when using FPGA and PC

Before running the demo using an FPGA and PC, please prepare the following.

1) FPGA development boards: ZCU102, ZCU106, KCU105

2) PC with 40 Gigabit Ethernet card

3) 40G Ethernet cable: QSFP+ to 4xSFP+ breakout cable (QSFP+ for PC connection and four SFP+ for FPGA connection)

4) Four SFP+ connector on FPGA board: For ZCU106 and KCU105 boards, prepare an adapter board such as AB15-SFPFMC board (https://dgway.com/ABseries_E.html)

5) USB cable for connecting the FPGA and the PC: Two micro USB cables for programming FPGA and Serial console.

6) Test application provided by Design Gateway for running on PC.

a) TOE40G-IP : “tcpdatatest.exe” and “tcp_client_txrx_xg.exe”

b) UDP40G-IP : “udpdatatest.exe”

7) Serial console software such as TeraTerm instealled on PC. The console setting is Baudrate=115,200, Data=8-bit, Non-parity, and Stop=1-bit.

8) Vivado tool for programming FPGA, installed on PC.

Note: The hardware listed below is an example for running the demo.

i) Ethernet Adapter Cards: Nvidia MCX614106A-CCAT (10/25/40/50/100 GbE)

https://docs.nvidia.com/networking/display/connectx6en

ii) QSFP+ to four SFP+ breakout cable: FCBN510QE2C03

https://www.coherent.com/content/dam/coherent/site/en/resources/datasheet/networking/quadwire-40g-parallel-breakout-aoc-ds.pdf

iii) Test PC:

Motherboard : Gigabyte Z590 AORUS MASTER (rev. 1.0)

CPU : Intel i7-11700K CPU 3.6 GHz

RAM : 64 GB DDR4

OS : 64-bit Window10 OS

Figure 2‑1 UDP40G-IP demo (FPGA <-> PC) on ZCU102

Figure 2‑2 UDP40G-IP demo (FPGA <-> PC) on ZCU106

Figure 2‑3 UDP40G-IP demo (FPGA <-> PC) on KCU105

The steps for setting up a test environment using an FPGA board and a PC are described below

1) Power off system.

2) For ZCU102/ZCU106 board, set ARM Boot mode of ARM to be JTAG by setting SW6[4:1] to be ON, ON, ON, ON.

Figure 2‑4 Set ARM boot mode to JTAG for ZCU102/ZCU106

3) For ZCU102 board, insert jumper to J16, J17, J42, and J54 to set SFP_TX_DISABLE to 0b.

Figure 2‑5 Set SFP[0]-[3] TX DISABLE for ZCU102

4) Connect two micro USB cables from FPGA board to PC for JTAG programming and USB UART (Serial Console).

5) Connect power supply to FPGA development board.

6) Insert the QSFP+ cable into the PC and insert four SFP+ cables into the FPGA board.

a) For the ZCU102, insert the four SFP+ cables into the onboard SFP+ ports. Ensure that the channel number of the four cables match those shown in Figure 2‑6.

Figure 2‑6 Four SFP+ cables connecting to ZCU102 board

b) For the ZCU106/KCU105, insert the AB15-SFPFMC adapter into the FMC-HPC(0) slot on the FPGA board. Then, insert four SFP+ cables into the SFP+ ports on the AB15-SFPFMC adapter. Ensure that the channel number of the four cables match those shown in Figure 2‑7 for each board.

Figure 2‑7 Four SFP+ cables connecting to ZCU106/KCU105 board

7) Power on the FPGA board.

8) Open a Serial console. When connecting FPGA board to PC, multiple COM ports from FPGA connection are detected and displayed on Device Manager.

a) KCU105: Select standard COM port.

b) ZCU102 and ZCU106: Select the first USB Serial port.

Use following setting on the Serial console: Baud rate=115,200, Data=8-bit, Non-Parity, and Stop=1.

Figure 2‑8 Port number for Serial console

9) Download configuration file and firmware to the FPGA board using Vivado tool or script file, depending on the board.

a) KCU105 board: Use Vivado tool to program configuration file, as shown in Figure 2‑9.

Figure 2‑9 Configure bit file using Vivado tool

b) ZCU102 and ZCU106 boards: Use Vivado TCL shell to download the configuration file and the firmware. Browse to the download directory that includes bat file, bit file, and elf file and type the command to run bat file.

Figure 2‑10 Download demo file using script file on Vivado TCL shell

10) Upon opening the Serial console, welcome message will be displayed.

i) Input ‘0’ to initiate the TOE40G-IP/UDP40G-IP in Client mode that asks for the PC MAC address through sending ARP request packet.

ii) The default parameters in Client mode will be shown on the console.

Figure 2‑11 Message after system boot-up

However, if there is an Ethernet connection problem and its status is linked down, an error message will be displayed instead of the welcome message, as shown in Figure 2‑12.

Figure 2‑12 Error message when ethernet connection link down

iii) If the user selects to proceed with system initialization using default parameters, they can input ‘x’ to skip the parameter updating process, as shown in Figure 2‑13. Entering any other keys will prompt the parameter change menu, similar to the “Reset TCPIP/UDPIP parameters” menu. The examples of running the main menu of TOE40G-IP and UDP40G-IP are described in the “dg_toe40gip_cpu_instruction” and “dg_udp40gip_instruction” documents, respectively.

Figure 2‑13 Initialization complete

Note: Transfer performance in the demo is limited by the PC performance. The best performance can be achieved when the test is run using FPGA-to-FPGA connection.

3 Test environment setup when using two FPGAs

Before running the test, please prepare following test environment.

1) Two FPGA development boards, which can be either the same or different boards: ZCU102, ZCU106, and KCU105.

2) 40G Ethernet cable: Four SFP+ cables connecting between two FPGA boards.

3) Four SFP+ connector on FPGA board: For ZCU106 and KCU105 boards, prepare an adapter board such as AB15-SFPFMC board (https://dgway.com/ABseries_E.html)

4) USB cable for connecting the FPGA and the PC: Two micro USB cables per each FPGA board for programming FPGA and Serial console.

5) Serial console software such as TeraTerm instealled on PC. The console setting is Baudrate=115,200, Data=8-bit, Non-parity, and Stop=1-bit.

6) Vivado tool for programming FPGA, installed on PC.

Figure 3‑1 UDP40G-IP demo (FPGA<->FPGA) by ZCU102 and ZCU106

The steps for setting up a test environment using two FPGAs are described below.

To get started with the demo, follow steps 1) – 9) of topic 2 (Test environment setup when using FPGA and PC) to set up the FPGA board and SFP+ connection. Once you have completed the configuration for two FPGA boards, a menu will be displayed on the Serial console for selecting Client mode, Server mode, or Fixed MAC mode. Follow the detailed steps below to continue the demo.

1) Open the Serial console for FPGA board#1 and FPGA board#2, which are set to initialize in Server/Client/Fixed-MAC mode. An example to initialize by Server-Client mode is below.

i) Set ‘1’ on the console of FPGA board#1 for running in Server mode.

ii) Set ‘0’ on the console of FPGA board#2 for running in Client mode.

iii) The default parameters for the selected mode will be displayed on the console, as shown in Figure 3‑2.

Note: The rules for setting the initialization mode are below.

· If the first board is initialized in Server mode, the other board must be initialized in Client mode.

· If the first board is initialized in Fixed-MAC mode, the other board can be run in Client mode or Fixed-MAC mode.

Figure 3‑2 Input mode

2) Input ‘x’ to use default parameters or use other keys to change parameters. The parameters of Server mode must be set before Client mode. The details are divided into two parts, running the TOE40G-IP demo and running the UDP40G-IP demo.

When running TOE40G-IP,

i) Set parameters on the Server console (board#1 console).

ii) Set parameters on the Client console (board#2 console) to start IP initialization by transferring ARP packet.

iii) After finishing the initialization process, “IP initialization complete” and the main menu are displayed on the Server console and Client consoles

Figure 3‑3 Main menu of TOE40G-IP

When running UDP40G-IP,

i) For Server mode (board#1 console), if user does not change the default parameters, input ‘x’ to skip parameter setting.

ii) For Client mode, the user must change Target port number (Target->FPGA) to use same value as Target port number (FPGA->Target).

iii) After finishing initialization process, “IP initialization complete” and the main menu will be displayed on the Server and Client consoles.

Figure 3‑4 Main menu of UDP40G-IP

4 Revision History

Revision	Date	Description
1.00	12-Mar-24	Initial version release