NVMeSW IP Core Demo Instruction
Rev1.1 17-Aug-23
This document describes the instruction to run NVMeSW-IP demo on FPGA development board by using the PCIe adapter board (AB18-PCIeX16 board or AB16-PCIeXOVR board). The demo is designed to write/verify data by connecting NVMe SSD through PCIe switch. User controls the test operation through NiosII command shell.
The sequence to run NVMeSW-IP with NVMe SSD directly (without PCIe switch) is almost same as NVMe-IP. Please see more details for running by using direct connection in “NVMe-IP demo instruction” document.
https://www.dgway.com/products/IP/NVMe-IP/dg_nvmeip_instruction_intel_en.pdf
1 Environment Requirement
To run the demo on FPGA development board, please prepare following environment.
1) Supported FPGA Development board: Arria10 GX development board
2) PC installing QuartusII programmer and NiosII command shell software.
3) The PCIe adapter board (AB18-PCIeX16 board/AB16-PCIeXOVR board) provided by Design Gateway
https://dgway.com/ABseries_E.html
4) Power adapter for FPGA board
5) ATX power supply for AB18
6) PCIe switch card with M.2 connector such as
a) Quattro 400 M.2 NVMe SSD adapter
https://www.aplicata.com/quattro-400/
b) Squid SKU-086-34 NVMe SSD Adapter
https://amfeltec.com/products/pci-express-gen-3-carrier-board-for-m-2-ssd/
7) 1-4 NVMe SSD(s) connecting on PCIe switch card
8) A micro USB cable for programming FPGA and JTAG UART, connecting between FPGA board and PC
Figure 1‑1 NVMeSW-IP demo environment setup on Arria10 GX
2 Demo setup
1) Power off system.
2) Connect 1-4 NVMe SSD(s) to PCIe switch card. Figure 2‑1 and Figure 2‑2 show the example to use four SSDs connected to PCIe switch card. The SSD channel number depends on the PCIe switch card characteristic.
Figure 2‑1 Connect NVMe SSD to PCIe Swtich#1
Figure 2‑2 Connect NVMe SSD to PCIe Swtich#2
3) As shown in Figure 2‑3, setup AB18 board by following step.
a) Confirm that two mini jumpers are inserted at J5 connector on AB18.
b) Connect ATX power supply to AB18.
c) Connect FPGA Side (A-side) on AB18 to PCIe connector on FPGA board
d) Connect PCIe switch to device side (B-Side) on AB18.
Warning: Please confirm that the PCIe switch is inserted in the correct side of AB18 (B-side, not A-side) before power on system.
Figure 2‑3 Connect AB18 to PCIe switch and FPGA board
e) Connect a micro USB cable for JTAG programming/UART.
Figure 2‑4 USB cable connection
f) Power on FPGA development board, ATX supply for AB18 board, and AB18 board, as shown in Figure 2‑5.
Figure 2‑5 Turn on power switch on FPGA and AB18 board
g) Use QuartusII Programmer to program “NVMeIPTest.sof” file, as shown in Figure 2‑6
Figure 2‑6 Programmed by QuartusII Programmer
h) Check LED status on FPGA board. The description of LED is as follows.
Figure 2‑7 Four LEDs to show the current status
Table 2‑1 LED Definition
|
GPIO LED |
ON |
OFF |
|
0 |
Normal operation |
Clock is not locked or reset button is pressed |
|
1 |
System is busy |
Idle status |
|
2 |
IP Error detect |
Normal operation |
|
3 |
Data verification fail |
Normal operation |
i) After finishing FPGA programming, LED[0] and LED[1] are ON during PCIe initialization process. LED[1] changes to OFF after PCIe initialization process is finished.
j) Open NiosII Command Shell and run “nios2-terminal”. On the console, the message is displayed to show current status as follows.
· “Waiting PCIe Linkup” is displayed after finishing configuration.
· After PCIe is linkup, “Waiting IP initialization” is displayed.
· After finishing NVMeSW-IP initialization, PCIe speed, number of PCIe lanes, and total detected devices are displayed. Finally, main menu to run six commands is shown on the console. The example message when connecting to four SSDs is shown in Figure 2‑8.
Figure 2‑8 Main menu after IP finishes initialization
3 Test Menu
3.1 Identify Command
Select ‘0’ to send Identify command to NVMe SSD.
Figure 3‑1 Test result when running Identify command
After that, user inputs to select the active device.
Input ‘9’ to run the command to all devices in the system or 0-3 to run only one device.
If the input is valid, the SSD information output from Identify command is displayed as shown in Figure 3‑1. The console shows three values for each SSD.
1) SSD model number: This value is decoded from Identify controller data.
2) SSD capacity: This value is signal output from IP.
3) Data size per LBA: This value is signal output from IP. Two values are supported, i.e. 512 byte and 4 Kbyte.
If user selects the unavailable device, error message is displayed as shown in Figure 3‑2.
Figure 3‑2 Error message when selecting the unavailable device
3.2 Write Command
Select ‘1’ to send Write command to NVMe SSD.
Figure 3‑3 Test result when running Write command
The first input from user is the device selection. User must input the device number for running Write command. If the input is valid, the selected device number is displayed on the console.
Next, user inputs three parameters as follows.
1) Start Address: Input start address to write SSD as 512-byte unit. The input is decimal unit when user enters only digit number. User can add “0x” to be prefix for hexadecimal unit. When LBA unit of SSD is 4 Kbyte, this input must be aligned to 8.
2) Transfer Length: Input total transfer size as 512-byte unit. The input is decimal unit when user enters only digit number. User can add “0x” to be prefix for hexadecimal unit. When LBA unit of SSD is 4 Kbyte, this input must be aligned to 8.
3) Test pattern: Select test data pattern for writing to SSD. There are five patterns, i.e. 32-bit incremental, 32-bit decremental, all-0, all-1, and 32-bit LFSR counter.
When all inputs are valid, the operation begins. During writing data, current transfer size is displayed on the console every second to show that system is still alive. Finally, total size, total time usage, and test speed are displayed on the console as test result.
Figure 3‑4 Example Test data of the 1st and 2nd 512-byte by using increment/LFSR pattern
Test data in SSD is split into 512-byte unit. For incremental, decremental, or LFSR pattern, each 512-byte data has unique 64-bit header consisting of 48-bit address (in 512-byte unit) and 16-bit zero value. The data after 64-bit header is the test pattern which is selected by user.
The left window of Figure 3‑4 shows the example when using 32-bit incremental pattern while the right window shows the example when using 32-bit LFSR pattern. The unique header is not included when running all-0 or all-1 pattern.
When user runs Write or Read command with 4-Kbyte LBA SSD, there is the message displaying on the console to show the input limitation which must be aligned to 8, as shown in Figure 3‑5. When the input does not align to 8, “Invalid input” is displayed and the operation is cancelled.
Also, Figure 3‑6 shows the example when the input is out of the recommended range for each parameter. The console displays “Invalid input” and then the operation is cancelled.
Figure 3‑5 Error message when the input is unaligned for SSD which has 4KB LBA unit
Figure 3‑6 Error message from the invalid input
3.3 Read Command
Select ‘2’ to send Read command to NVMe SSD.
Figure 3‑7 Test result when running Read command
The first input from user is the device selection. User must input the device number for running Read command. If the input is valid, the selected device number is displayed on the console.
Next, user inputs three parameters as follows.
1) Start Address: Input start address to read SSD as 512-byte unit. The input is decimal unit when user enters only digit number. User can add “0x” to be prefix for hexadecimal unit. When LBA unit of SSD is 4 Kbyte, this input must be aligned to 8.
2) Transfer Length: Input total transfer size as 512-byte unit. The input is decimal unit when user enters only digit number. User can add “0x” to be prefix for hexadecimal unit. If LBA unit of SSD is 4 Kbyte, this input must be aligned to 8.
3) Test pattern: Select test data pattern to verify data from SSD. Test pattern must be matched with the pattern using in Write Command menu. There are five patterns, i.e. 32-bit incremental, 32-bit decremental, all-0, all-1, and 32-bit LFSR counter.
Similar to Write command menu, test system reads data from SSD when all inputs are valid. During reading data, current transfer size is displayed on the console every second to show that system is still alive. Total size, total time usage, and test speed are displayed after finishing the operation.
“Invalid input” is displayed when some inputs are invalid or unaligned to 8 (when connecting to 4-KB LBA SSD).
Figure 3‑8 shows error message when data verification is failed. “Verify fail” is displayed with the information of the 1st failure data, i.e. the error byte address, the expected value, and the read value.
User can press any key(s) to cancel read operation or wait until finishing Read command. Similar to the normal condition, the output performance is displayed on the console when the user does not enter any key(s) to stop the operation.
When cancelling the operation, the read command still runs as the background process. It is recommended to power-off/on AB18/AB16, and then press “RESET” button to restart system.
Figure 3‑8 Data verification is failed
3.4 SMART Command
Select ‘3’ to send SMART command to NVMe SSD.
Figure 3‑9 Test result when running SMART command
After that, user inputs to select the active device.
Input ‘9’ to run the command to all devices in the system or input 0-3 to run only one device.
If the input is valid, the SSD SMART/health information (output from SMART command) us displayed as shown in Figure 3‑9. The console shows six parameters, described as follows.
1) Temperature in °C unit.
2) Total Data Read decoded as GB/TB unit. Additionally, raw data without decoding is displayed as 128-bit hexadecimal unit. The unit size of raw data is 512,000 byte.
3) Total Data Written decoded as GB/TB unit. Additionally, raw data without decoding is displayed as 128-bit hexadecimal unit. The unit size of raw data is 512,000 byte.
4) Power On Cycles: Display the number of power cycles.
5) Power On Hours: Display period of time in hours to show how long the SSD has been powered on.
6) Unsafe Shutdowns: Display the number of unsafe shutdowns of SSD
3.5 Flush Command
Select ‘4’ to send Flush command to NVMe SSD.
Figure 3‑10 Test result when running Flush command
After that, user inputs to select the active device.
Input ‘9’ to run the command to all devices in the system or 0-3 to run only one device.
When the input is valid, Flush command operation begins.
“Flush Command Complete” is displayed after finishing Flush operation.
3.6 Shutdown Command
Select ‘5’ to send Shutdown command to NVMe SSD.
Figure 3‑11 Test result when running Shutdown Command
After that, user inputs to select the active device.
Input ‘9’ to run the command to all devices in the system or 0-3 to run only one device.
Next, the confirmation message is displayed on the console. User inputs ‘y’ or ‘Y’ to start Shutdown operation or inputs other keys to cancel the operation.
After finishing Shutdown operation, “Shutdown Command Complete” is displayed on the console. As shown in the left side of Figure 3‑11, if all devices are selected, the last message is “Now NVMeSW-IP is inactive”. No main menu is displayed after finishing the operation. The user should shut down the system.
As shown in the right side of Figure 3‑11, if one device is selected, the updated device list is displayed. The device which has just finished Shutdown command is not available in the list. User needs to re-power the system to wake up the SSD.
4 Revision History
|
Revision |
Date |
Description |
|
1.0 |
30-Apr-19 |
Initial version release |
|
1.1 |
13-Feb-20 |
Support AB18 |