1 Introduction
This document provides a comprehensive guide for setting up an FPGA board to demonstrate the NVMeTCP25G-IP. It includes step-by-step instructions for connecting to a target NVMe SSD within a PC via a 25G Ethernet connection, as depicted in Figure 1.
Figure 1 Test Environment for the Demo
For user interactions, the FPGA employs a Serial console or JTAG UART interface. In the setup illustrated in Figure 1, a single PC serves dual roles: as the target system for NVMe/TCP operations and as the user interface via the Serial console.
Before initiating the test, ensure that the following hardware components are prepared and configured.
1) FPGA development boards: KCU116, VCK190, or FB2CGHH@KU15P card
2) PC setup
· Operating System: Install Linux with a kernel version 5.0 or later.
· Storage: Equip the IP with at least one NVMe SSD.
· Network: Install a 25G Ethernet card.
3) 25G Ethernet cable:
a) KCU116: 25G SFP28 Active Optical Cable (AOC) or 25G SFP28 transceivers (25G BASE-R) with optical cable (LC to LC, Multimode).
b) VCK190 and FB2CGHH@KU15P card: QSFP28 to 4xSFP28 cable.
4) Board power
a) KCU116 and VCK190: Board power adapter.
b) FB2CGHH@KU15P card: Power by PCIe connector. It is able to use AB18-PCIeX16 board provided by Design Gateway with ATX power supply. More details of AB18 card are displayed on following website.
https://dgway.com/ABseries_E.html
5) USB cable for connecting between FPGA and PC
a) KCU116: Two micro USB cables for programming FPGA and Serial console.
b) VCK190: A USB type-C cable is used for programming the FPGA and establishing the Serial console.
c) FB2CGHH@KU15P card: A mini USB cable for programming FPGA and JTAG UART.
6) User console
a) KCU116 and VCK190: Serial console software such as Putty installed on PC. The setting on the console is Baud rate=115,200, Data=8-bit, Non-parity, and Stop=1-bit.
b) FB2CGHH@KU15P card: JTAG Terminal
7) Vivado tool for programming the FPGA, installed on the PC.
Note: Additional Hardware References
· 25G Network Adapter: Nvidia MCX631102AC-ADAT
· Ethernet Cable Options:
a) KCU116: 25G SFP28 Active Optical Cable: SFP-25G-AO01
https://www.fs.com/sg/products/68335.html
b) VCK190 and FB2CGHH@KU15P card: SFP28 to QSFP28 connection
· SFP28 Transceiver: AZS85-S28-M1
https://www.sfpcables.com/25gb-s-sfp28-sr-transceiver-850nm-up-to-100m-2866
· QSFP28 Transceiver: AMQ28-SR4-M1
https://www.sfpcables.com/100gb-s-qsfp28-sr4-optical-transceiver-module-1499
· MTP to 4xLC Fiber cable: OM4-MTP-8LC-1M
https://www.fs.com/products/68047.html
· PC Specification: Gigabyte Z590 AORUS MASTER motherboard, equipped with 64 GB RAM, and LinuxOS kernel version 5.4.0-81.
· Target NVMe SSD: 1 TB WD 850SN.
Figure 2 NVMeTCP25G-IP Demo on KCU116
Figure 3 NVMeTCP25G-IP Demo on VCK190
Figure 4 NVMeTCP25G-IP Demo on FB2CGHH@KU15P
2 FPGA Setup
This section provides detailed instructions for preparing the FPGA board to run NVMeTCP25G-IP demo.
1) Connect USB cables between FPGA board and PC for JTAG programming and Serial console/JTAG UART.
· KCU116 board: Connect two micro USB cables.
· VCK190 board: Connect a USB type-C cable.
· FB2CGHH@KU15P card: Connect a mini USB cable.
2) Connect power supply to FPGA development board/FPGA accelerator card.
· KCU116 and VCK190 boards: Connect board power adapter.
· FB2CGHH@KU15P card: Connect the card to PC or AB18-PCIeX16 board by following steps.
i) Confirm that two mini jumpers are inserted at J5 connector on AB18.
ii) Connect ATX power supply to AB board.
iii) Connect PCIe connector on FPGA board to Device Side (B-Side), as shown in Figure 5.
Figure 5 Using AB18-PCIeX16 for Power on FB2CGHH@KU15P Card
3) Connect 25G Ethernet cable between FPGA board and PC.
a) KCU116: Insert 25G SFP28 AOC cable between FPGA board (on the left-most channel) and 25G Ethernet card on Test PC, as shown in Figure 6.
Figure 6 SFP28 AOC for Ethernet Connection on KCU116
b) VCK190 and FB2CGHH@KU15P: Insert QSFP28 to SFP28 cable using QSFP(1) connector on FPGA and plug SFP28 no.1 to 25G Ethernet card on Test PC, as shown in Figure 7
Figure 7 QSFP28 to 4xSFP28 Cable for Ethernet Connection
4) Power on the FPGA board.
For VCK190 board, ensure that the board boots from the SD card to launch the BEAM tool. Booting with a configuration other than the BEAM tool may cause issues with the clock programming functionality in the Board User Interface, which is required in the next step. Follow these steps to boot the VCK190 board to launch the BEAM tool.
i) Ensure the SD card contains the Linux Prebuilt images for the VCK190, officially released by AMD Xilinx, available at https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18842316/Linux+Prebuilt+Images
If the SD card does not have the correct image, update it by following the guidance in the section “Installing the System Controller Image” in UG1573: https://docs.amd.com/r/en-US/VCK190/VMK180-Board-Evaluation-and-Management-BEAM-Tool-User-Guide-UG1573/Installing-the-System-Controller-Image
ii) Insert the micro SD card into the system controller’s SD card socket (J206) and set the DIP switch (SW11) to ON OFF OFF OFF to boot from the SysCont SD, as shown in Figure 8.
Figure 8 SD Card Boot Setting on VCK190
iii) Connect the VCK190 board to the PC using a USB cable. The PC should recognize three USB Serial Ports. Use the third port to monitor the board boot-up message.
iv) Open a Serial console and connect to the third USB Serial port with following settings: Baud rate=115,200, Data=8-bit, Non-parity, and Stop bits=1-bit.
Figure 9 Serial Console Setting for VCK190
v) Power on the board and monitor the boot-up message on the console. Upon board boot-up completion, close this console to avoid port collision in future steps.
Figure 10 Boot-up Completion Message for VCK190
5) To configure the programmable clock for the FPGA board, determine whether your board requires a programmable clock setup. If the board has already used the desired clock frequency, you do not need to re-configure it.
a) KCU116 board: Use “KCU116 – Board User Interface” application to set the programmable clock to 322.265625 MHz, as shown in Figure 11.
Figure 11 Reference Clock Programming for KCU116
b) VCK190 board: Use “VCK190 – Board User Interface” application to set the programmable clock to 322.265625 MHz, as shown in Figure 12.
Figure 12 Reference Clock Programming for VCK190
c) FB2CGHH@KU15P card: No action is required to set up the programmable clock.
3 Demo Setup
This section outlines the configuration of the Serial console and FPGA programming tools on a PC necessary for running the demo. Instructions for both Windows OS and Linux OS are provided.
3.1 KCU116 and VCK190 Boards
3.1.1 Serial Consoles
When connecting the FPGA board to a PC, multiple COM ports (Windows OS) or USB Serial ports (Linux OS) associated with the FPGA connection will be detected. Follow the selection guide corresponding to the specific operation system.
Windows OS
1) Identify the correct COM port.
· For KCU116: Choose the standard COM port.
· For VCK190: Select the lowest number of the newly detected COM ports.
Figure 13 Used COM Port number for Serial Console on Windows OS
2) Configure your Serial console with the following settings (illustrated in Figure 14): Baud rate=115,200, Data=8-bit, Parity=None, and Stop bits= 1.
Figure 14 Serial Console Setting on Windows OS
Linux OS
1) Upon connecting the FPGA board, several USB Serial ports will appear. Use the following command to list the USB Serial ports.
>> dmesg | grep ttyUSB
Port selection for running the demo: Use the second port for both KCU116 and VCK190.
Figure 15 Scan USB Serial Ports on Linux OS
2) Configure the Serial console using the same Serial configuration as for Windows (Baud rate=115,200, Data=8-bit, Parity=None, and Stop bits= 1). Figure 16 shows example using the Putty application.
Figure 16 Serial Console Setting on Linux OS
3.1.2 FPGA Configuration
This section outlines the procedures to load the configuration file and the demo firmware onto the FPGA board. Depending on the FPGA board in use, the method for programming differs.
a) For KCU116 board: Utilize the Vivado tools for programming. Open the Vivado Hardware Manager and program the board with the required bit file as illustrated in Figure 17.
Figure 17 KCU116 Configuration
b) For VCK190 board: Use a script file via the TCL shell to load the bit file and the elf file.
Windows OS
Execute a batch file to load the configuration.
>> cd <directory containing the batch file>
>> nvmetcp25giptest_<buffsize>_<boardname>.bat
Figure 18 VCK190 Configuration on Windows OS
Linux OS
Use a TCL script for loading the configuration.
>> cd <directory containing the tcl file>
>> exec xsdb nvmetcp25giptest_<buffsize>_<boardname>.tcl
Figure 19 VCK190 Configuration on Linux OS
3.2 FB2CGHH@KU15P Card
Use Vivado TCL shell to download the configuration file and the firmware. Browse to the download directory that includes bat file, pdi file, and elf file. After that, run the test by typing following command.
1) >> nvmetcpiptest_Silicom.bat
This step is to download configuration file and firmware, as shown in Figure 20.
Figure 20 Download Demo File using Vivado TCL Shell
2) >> xsdb.bat
3) >> connect -url tcp:127.0.0.1:3121
4) >> targets -set -filter {name =~"*Debug*"}
5) >> jtagterminal -start
6) >> con
Above steps are to connect JTAG UART module and run JTAG terminal to be user console, as shown in Figure 21.
Figure 21 Open JTAG Terminal
4 Welcome message
Once the FPGA configuration is completed, a welcome message will be displayed on the console, as shown in Figure 22.
Figure 22 Welcome screen
5 Revision History
|
Revision |
Date (D-M-Y) |
Description |
|
1.01 |
20-Sep-24 |
Support VCK190 board |
|
1.00 |
25-Mar-21 |
Initial version release |