FPGA setup on Alveo card Demo
Rev1.0 6-Jul-23
3 Test environment setup when the target uses 100G Ethernet card
4 Test environment setup when the target uses Alveo Accelerator card
1 Overview
This document describes how to setup Alveo accelerator Card within Accelerator system and prepare the test environment for running TOE100G-IP on Alveo card. The demo allows the data transmission between the host system (CPU) and the target test systems through Alveo accelerator Card, as shown in Figure 1‑1.
Figure 1‑1 Two test environments for running the demo
The demo is run on the Alveo accelerator card plugged into the system that supports to run the Accelerator card demo. Accelerator card has QSFP28 connector which supports up to 100Gb Ethernet connection. The demo uses a QSFP28 connector for transferring the TCP payload over 100Gb network environment. There are two test environments for this demo in order to transferring the 100Gb TCP payload.
The first test environment uses Test PC with 100Gb Ethernet card (Target#1) for transferring TCP payload data. Two test applications are designed for running on TestPC, i.e., “tcpdatatest” and “tcp_client_txrx(_40G)” for half-duplex test and full-duplex test, respectively.
The second test environment uses another Alveo accelerator card on another PC and runs TOE100G-IP on Alveo demo (Target#2) for transferring TCP payload data. This test environment is highly recommended to maximize the performance in 100G Ethernet.
In the document, topic 2 shows the procedure to setup the system with Alveo accelerator card plugged in and to run the demo on the card. Topic 3 shows the example console and test result when running under the first test environment, Accelerator system and 100G Ethernet card on Test PC. Finally, topic 4 shows the example console when running the second test environment, Accelerator system and Accelerator system. More details of each topic are described as follows.
2 Accelerator system setup
Before running this demo, please install the Alveo accelerator card into the system by following the steps from Xilinx installation document, provided as below.
https://docs.xilinx.com/r/en-US/ug1301-getting-started-guide-alveo-accelerator-cards
To run the demo, please prepare following environment.
- Alveo accelerator card: U250 or U50 card
- Accelerator system for Alveo card: TKAS-D2101 with Ubuntu 20.04 LTS Server OS
https://dgway.com/AcceleratorCards.html
Figure 2‑1 TOE100G-IP on Alveo card demo on Accelerator system
This topic shows the steps to prepare Accelerator system (TKAS-D2101 with U250) to run the demo.
1) Detect Accelerator card which connects on TKAS-D2101 to get the Card BDF ID by using following command.
>> sudo /opt/xilinx/xrt/bin/xbmgmt examine
Figure 2‑2 Examine Accelerator card
2) For U250 card, it needs to program the shell partition to Accelerator card once after system bootup. This step can be skipped for U50 card. More details are described in page 25 of UG1301 (V2.0).
i. Determine the partition file with full path by following command by using following command. The command generates the JSON output file which contains the path of partition file. The command parameter is “card_BDF” which is obtained by the previous command.
>> sudo /opt/xilinx/xrt/bin/xbmgmt examine --report platform --format json --output <output file.json> --device <card_BDF>
Figure 2‑3 Generate JSON report file of xbmgmt
ii. Program the shell partition to the target Alveo card by the following command along with the “card_BDF” and the path of partition file which are determined by step i) and step ii). The path to partition file in output JSON file needs to remove the backslash character (‘\’) before using in this command.
>> sudo /opt/xilinx/xrt/bin/xbmgmt program --device <card_BDF> --shell <path to partition file>
Figure 2‑4 Program the shell partition
3) Enable the host memory to reserve the memory for the target Alveo platform by using the following command with the “card_BDF” from the previous step.
>> sudo /opt/xilinx/xrt/bin/xbutil configure --host-mem --device <card_BDF> --size 1G enable
For more information about Host Memory Access of Xilinx Accelerator platform can be found on the following Xilinx site.
https://xilinx.github.io/XRT/master/html/hm.html
Figure 2‑5 Enable the host memory through XRT
3 Test environment setup when the target uses 100G Ethernet card
Please prepare the following environment to set up the target system by 100G Ethernet card.
· PC with 100 Gigabit Ethernet card
· 2xQSFP28 transceiver (100GBASE-SR) and MPO to MPO cable. One for connecting the Alveo accelerator card and another for plugging into 100 Gigabit Ethernet card of Test PC.
· Test application provided by Design Gateway for running on Test PC. “tcpdatatest” and “tcp_client_txrx(_40G)” for half-duplex test and full-duplex test, respectively.
Note: Example hardware for running the demo is listed as follows.
[1] 100G Network Adapter: MCX614106A-CCAT
https://docs.nvidia.com/networking/display/ConnectX6EN
[2] 100G Ethernet cable
QSFP28 Transceiver: AMQ28-SR4-M1 https://www.sfpcables.com/100gb-s-qsfp28-sr4-optical-transceiver-module-1499
MPO to MPO cable: OM4-MPO-8MPO-1M https://www.sfpcables.com/mpo-to-mpo-multimode-om4-50-125-m-8-core-4381
[3] Test PC:
Motherboard: Gigabyte Z590 AORUS MASTER (rev. 1.0)
CPU: Intel i7-11700K CPU 3.6 GHz
RAM: 64 GB DDR4
OS: 64-bit Windows10 OS
Figure 3‑1 Demo setup for Accelerator system and PC
The step to setup test environment by using Accelerator system and Test PC is described in more details as follows.
1) Connect 100Gb Ethernet cable between Xilinx Accelerator card and PC. Insert 100G QSFP28 Transceiver and MPO to MPO cable (Use QSFP1 connector for U250 card), as shown in Figure 3‑2.
Figure 3‑2 100Gb connection on Alveo card by QSFP28 transceiver and MPO to MPO cable
2) Another side of MTP/MPO cable, insert QSFP28 transceiver to connect with 100Gb Ethernet card on PC.
3) On Accelerator system, follow these steps to program the configuration file (xclbin) to Accelerator card, as shown in Figure 3‑3.
i) Open the terminal and change the directory where the xclbin file is located.
ii) Program the configuration file (xclbin) through PCIe by the following command along with the “card_BDF” and the xclbin file.
>> sudo /opt/xilinx/xrt/bin/xbutil program --device <card_BDF> -u <xclbin_file>
Note: The configuration file of this demo is “TOE100DMATest.xclbin” which is pre-built and ready to run in this system environment.
Figure 3‑3 Program the configuration file to the Accelerator card
4) Run the demo application which is “toe100g_dma_exe” on Accelerator system, as shown in Figure 3‑4.
i) On the same terminal after running the step 3, export the XRT (Xilinx) environment by using following command.
>> source /opt/xilinx/xrt/setup.sh
ii) Run the following command to launch the demo application.
>> ./bin/toe100g_dma_exe
Note: “toe100g_dma_exe” is pre-built application that is stored in the “bin” directory. The “bin” directory is located in the same directory as the configuration file (TOE100DMATest.xclbin).
iii) After opening the demo application, the welcome message is displayed.
iv) Input ‘0’ to start TOE100G-IP initialization in client mode (asking PC MAC address by sending ARP request).
v) The default parameters in client mode are displayed on the terminal.
Figure 3‑4 Run the demo application after programming configuration file
If Ethernet connection has the problem and the connection cannot be established, the error message is displayed on the console instead of welcome message, as shown in Figure 3‑5.
Figure 3‑5 Error message when the cable is unconnected
vi) Enter ‘x’ to skip parameter setting for using default parameters to begin TOE100G-IP initialization, as shown in Figure 3‑6. If the other keys are entered, the menu for changing parameter is displayed, similar to “Reset TCPIP parameters” menu. The example to run the main menu is described in “dg_toe100gip_dma_instruction” document.
Figure 3‑6 Initialization complete
Note: Transfer performance in the demo is limited by Test PC performance. The best performance can be achieved when the target is FPGA with TOE100G-IP system.
vii) To exit the demo application, enter “CTRL+C” to terminate the application. After that, “Exiting the application!” is displayed and the application is terminated. as shown in Figure 3‑7. Other methods to close the application are not recommended.
Note: Exit the application by “CTRL+C” is recommended when there is no test run. If the application is closed during the test (Send data test, Receive data test, or Full duplex test), user must re-program the configuration file (xclbin).
Figure 3‑7 Close the software application
4 Test environment setup when the target uses Alveo Accelerator card
Please prepare the following environment to set up the target system by Alveo accelerator card.
· Two Accelerator systems that integrates Alveo Accelerator card (U250 or U50 card)
· 100Gb Ethernet cable: 2xQSFP28 transceiver (100GBASE-SR) and MPO to MPO cable
Note: 100G Ethernet cable for running the demo is listed as follows.
· QSFP28 Transceiver: AMQ28-SR4-M1 https://www.sfpcables.com/100gb-s-qsfp28-sr4-optical-transceiver-module-1499
· MPO to MPO cable: OM4-MPO-8MPO-1M https://www.sfpcables.com/mpo-to-mpo-multimode-om4-50-125-m-8-core-4381
Figure 4‑1 Demo setup for two Accelerator systems
The step to setup test environment by using two Accelerator systems that run TOE100G-IP demo are described in more details as follows.
1) Download configuration file of TOE100G-IP on Alveo card demo to both Accelerator card, as described in step 1) – 3) of topic 3 (Test environment setup when the target uses ).
2) Follow step 4i) – 4iii) of topic 3 (Test environment setup when the target uses ) to run the demo application on Accelerator system. After that, the menu to select the initialization mode (Client mode, Server mode, or Fixed MAC mode) are displayed on the Terminal of Accelerator system.
3) Start the initialization process. The example to initialize by Server-Client mode is described as follows.
i) Set ‘1’ on Terminal of Accelerator system#1 for running Server mode.
ii) Set ‘0’ on Terminal of Accelerator system#2 for running Client mode.
iii) Default parameters for Server or Client are displayed, as shown in Figure 4‑2.
Note: The rules for setting the mode when using two TOE100G-IPs are described as follows.
If the first IP runs in Server mode, another IP must run in Client mode.
If the first IP runs in Fixed MAC mode, another IP can be run in Client mode or Fixed MAC mode.
Figure 4‑2 Input mode for system initialization
4) Input ‘x’ to use default parameters or other keys to change parameters. The parameters of Server mode must be set before Client mode. Please run the following step to complete the initialization process, as shown in Figure 4‑3.
i) Set parameters on Server terminal (Accelerator system#1).
ii) Set parameters on Client terminal (Accelerator system#2) to start IP initialization by transferring ARP packet.
iii) After finishing initialization process, “IP initialization complete” and main menu are displayed on Server terminal and Client terminal.
Figure 4‑3 Main menu of TOE100G-IP demo
5 Revision History
|
Revision |
Date |
Description |
|
1.0 |
25-Aug-22 |
Initial version release |