NVMeTCP10G-IP Demo Instruction
2.2 Ethernet Interface Setting on PC
2.3 NVMe/TCP Target setting on PC
2.4 Removing NVMe/TCP Target on PC
1 Overview
This document provides detailed instructions for running the NVMeTCP10G-IP demo, which facilitates access to an NVMe SSD within an NVMe/TCP Target. The target system setup of this demonstrates utilizes a PC running Ubuntu 20.04.1 OS. Operations performed during the demo involve writing to and reading from the NVMe SSD via a 10G Ethernet connection. These operations adhere to the NVMe/TCP sequence process, which users can manage through the FPGA console interface.
This guide is divided into two sections. Section 2 details the setup procedure for initializing the NVMe SSD to run as an NVMe/TCP target and the steps to safety remove the NVMe from the NVMe/TCP target configuration on a PC with Ubuntu OS. Section 3 explains the operations performed via the FPGA console, including detailed explanations of each test step and the presentation of test results.
2 PC Setup for NVMe/TCP
2.1 Application Installation
Prior to running the NVMeTCP10G-IP demo, it is necessary to install specific applications on the PC. These applications need to be installed only once and are not required to be reinstalled for subsequent uses.
Ethtool: This utility is used to tune network card performance. Install Ethtool by executing the following command in the terminal:
NVMe Command Line Interface (NVMe-CLI): This tool is used for managing NVMe SSDs under Linux OS. Install NVMe-CLI using the command:
2.2 Ethernet Interface Setting on PC
To optimize the performance of the 10G Ethernet network card before running the demo, follow these steps. Open a new terminal window to begin configuring the Ethernet interface.
1) To list the logical name of 10G Ethernet port on the Linux terminal, use following command:
Figure 2‑1 displays the output of this command. For example, “enp1s0f0” may be identified as the 10G Ethernet interface connected to the NVMe/TCP host.
Figure 2‑1 Display logical name of 10G Ethernet port
Figure 2‑2 IP address and Ethernet interface setting
2) Input “ifconfig <interface> <ipaddr_value> netmask <netmask_value>” to set the IP address and subnet mask for the desired Ethernet port:
The interface (‘enp1s0f0’) is set with an IP address of ‘192.168.10.100’ and a subnet mask of ‘255.255.255.0’.
3) Input “ifconfig <interface > mtu <mtu_value>” to enable jumbo frames by setting the maximum transfer unit (MTU) to 9000 using the following command:
4) Turn off Rx-Tx latency improvement algorithm to stabilize performance by “sudo ethtool -C <interface> adaptive-rx off adaptive-tx off”.
5) Set the highest rate of Rx interrupt to ensure the PC processes each received packet immediately by “sudo ethtool -C <interface> rx-usecs 0 rx-frames 1”.
2.3 NVMe/TCP Target setting on PC
This section provides instructions on setting up the PC to serve as NVMe/TCP target, following the configuration of the 10G Ethernet network.
Figure 2‑3 Target setting
1) Load the NVMe/TCP Target module using the following command.
2) Mount the kernel user configuration filesystem to manage NVMe/TCP target settings.
3) Create an NVMe target subsystem and set its NVMe Qualified Name (NQN). The NQN here should match the ‘TrgNQN’ setting to the NVMeTCP10G-IP demo. Assumed that the NQN is configured by “dgnvmettest”, use following commands for changing directory to the subsystem directory.
Note: While the ‘TrgNQN’ on the NVMeTCP10G-IP supports up to 256 characters, the demo system allows input up to 16 characters. Ensure the NQN length does not exceed this limit.
4) Set the attribute to allow any host to connect using following command. This is suitable for testing but should be restricted in production environments. The NVMeTCP10G-IP allows user to configure ‘Host NQN’ value.
5) Create a namespace within the subsystem and change directory using following command.
6) Assign a local NVMe SSD to the system and enable it using following command.
Note: Replace ‘nvme1n1’ with the correct device identifier for your SSD.
7) Create an NVMe target port to export the created subsystem and change directory using following command.
8) Configure Ethernet parameters for the NVMe target port, including
· IP address = 192.168.10.100
· Transport type = “tcp”
· Port number = 4420
· Address family = “ipv4”
Note: IP address value corresponds to configured value of Ethernet card port in section 2.2 (Ethernet Interface Setting on PC).
9) Create a symbolic link between the target port and the subsystem using following command.
Note: The Target NQN (dgnvmetest) must match the value configured in steps (step 3).
10) Check that the target is correctly configured by inspecting the system messages using following command.
11) Successful configuration messages will display the target IP address and port number.
Figure 2‑4 NVMe/TCP target setup success message
2.4 Removing NVMe/TCP Target on PC
This section provides step-by-step instructions on how to dismantle the NVMe/TCP target configuration on you PC after completing tests.
Figure 2‑5 Remove NVMe/TCP Target
1) Change the working directory to where NVMe/TCP configurations are stored.
2) Delete the directory corresponding to the target port to remove its connections.
3) Remove the subsystem linked to the NVMe SSD.
Once the target subsystem is successfully removed, the NVMe SSD should be accessible by the general NVMe device driver on the PC. Use “hexdump” or other tools to inspect the SSD and verify the written data.
By following these steps, you can ensure that the NVMe/TCP target configuration is removed, and the NVMe SSD is returned to its standard operational state for regular PC usage.
3 Test operation
Upon the completion of both PC setup and FPGA setup, the FPGA console displays a welcome screen, indicating readiness for operations. This is confirmed by the display of the IP name, IP version number, and the status of the Ethernet connection.
Figure 3‑1 Message after system boot-up
· IP Details: The name and version of the NVMeTCP10G-IP are shown. Additionally, the IP version of DG EMAC-IP is also displayed if it is utilized.
· Ethernet Status: The connection status of the FPGA’s Ethernet link is displayed. If the Ethernet link fails to establish, an error message prompting a check of the Ethernet cable appears.
Figure 3‑2 Error message when Ethernet connection is down
The main menu appears once the Ethernet link is established. This menu includes several options, each corresponding to different stages of the test operation. Initially, only the relevant menu items are made available to ensure the correct sequence of operations is followed. The first action after boot-up is Menu [0] for setting network parameters.
3.1 Set Network Parameter
Select ‘0’ from the main menu to configure IP parameters. This action initiates the process to set or update the network settings required for communication between the host and target. Initially, the current parameters are shown on the console. The user enters ‘x’ to continue with the existing setting, or enters any other key to begin altering the parameters. While setting each parameter, the parameter’s value is not updated if the user enters invalid value. Below are the lists of input parameters.
1) Target NVMe Qualifed Name (NQN): Identifies the target SSD. Accepts up to 16 characters and the default is “dgnvmettest”.
2) Host MAC Address: Requires a 12-digit hexadecimal value for the host by adding prefix with “0x”. Default is 0x000102030405.
3) Host IP Address: Sets four decimal digits separated by dots. Each segment ranges from 0-255. This value is assigned to be the IP address of the host. Default is 192.168.10.1.
4) Host Port Number: Inputs the Admin and IP ports for the host with valid inputs ranging from 0-65535. Defaults are 40000 (Admin) and 40001 (IO).
5) Target IP address: Same format as the Host IP. This value is assigned to be the IP address of the target. Default is 192.168.10.100.
After all parameters are set, they are displayed for user confirmation. If the user confirms the settings by entering ‘y’, the parameters are configured on the NVMeTCP10G-IP. A confirmation message, “IP parameters are set”, is shown upon successful configuration.
The main menu reappears with offering the ‘Connect’ command as the next step, as detailed in Figure 3‑3.
Figure 3‑3 Set Network Parameter result
3.2 Connect
Select ‘1’ from the main menu to establish the connection between the host and the target.
Figure 3‑4 Console result when Connect succeeded
The NVMeTCP10G-IP (the host) established a TCP/IP connection using the network parameters configured in Section 3.1 (Set Network Parameter). Following successful TCP/IP setup, the NVMe/TCP connection is established using the specified Target NQN. This ensures that the host is correctly linked to the designated target NVMe SSD.
Upon successfully connecting, the console displays “Connect target successfully” along with the capacity of the target NVMe SSD. The menu then updates to include options for Write, Read, and Disconnect commands, as illustrated in Figure 3‑4.
If the connecting processes takes longer than expected, the waiting time is updated and displayed every second. In cases where TCP/IP initialization fails or the host cannot connect to the target, the error status and detailed error information are shown. Additionally, the TestPin (Internal Test pin of the IP) is displayed.
If an error occurs, it is necessary to reset the system before attempting a new test. This requirement is shown in both Figure 3‑5 and Figure 3‑6.
Figure 3‑5 Console result when TCP/IP initialization fails
Figure 3‑6 Console result when the host fails to connect with the target
3.3 Write Command
Select ‘2’ from the main menu to perform a write operation to the target NVMe SSD. This command tests the host’s ability to send Write commands and pattern data across Ethernet to write data on the target NVMe SSD.
Figure 3‑7 Test result when running Write command
User inputs three parameters as follows.
1) Start Address: Input the start address where the write operation begins on the target NVMe SSD, specified in 512-byte units. Enter the address in decimal form or use the “0x” prefix for hexadecimal values. Ensure the address aligns to 4 KB (8 x 512 bytes) as this is the fixed data length for one Write command.
2) Transfer Length: Input the total size to be written in 512-byte units. Like the start address, this can be entered in decimal or hexadecimal (with “0x” prefix) and must align to 4 KB.
3) Test pattern: Choose a data pattern for writing. Available options include 32-bit incremental, 32-bit decremental, all 0, all 1, and 32-bit LFSR counter.
Note: SSDs may exhibit optimal performance with the all zeros pattern due to data compression algorithms inside the SSD controller.
Upon validating all inputs, the write process commences. The current amount of transferred data is displayed every second on the console to confirm ongoing operation. Upon completion, the console displays the total size transferred, total time used, and test speed.
For the incremental, decremental, and LFSR patterns, each 4 KB data block includes a unique 64-bit header consisting of a 48-bit address (in 512-byte units) followed by 6 bits set to zero. The remainder of the data block is filled with the chosen test pattern. Figure 3‑8 and Figure 3‑9 illustrated examples using the 32-bit incremental and LFSR pattern, respectively. The unique header is omitted when using all-zero or all-one patterns.
Figure 3‑8 Example Test data of the 1st and 2nd 4KB data block using incremental pattern
Figure 3‑9 Example Test data of the 1st and 2nd 4 KB data block using LFSR pattern
If invalid inputs are detected, the console displays an “Invalid input” message, and the operation is cancelled, as depicted in Figure 3‑10.
Figure 3‑10 Error message when the input is unaligned, out-of-range, or invalid
3.4 Read Command
Select ‘3’ to initiate a Read operation from the target NVMe SSD. This menu facilitates the testing of read operations by sending a Read command across Ethernet to the target. Following the command, the host awaits the data read from the target NVMe SSD.
Figure 3‑11 Test result when running Read command with 1-MB read buffer size
Figure 3‑12 Test result when running Read command with 32-KB read buffer size
The user inputs three parameters as follows.
1) Start Address: Input the start address where the read operation begins on the target NVMe SSD, specified in 512-byte units. Enter the address in decimal form or use the “0x” prefix for hexadecimal values. Ensure the address aligns to 4 KB (8 x 512 bytes) as this is the fixed data length for one Read command.
2) Transfer Length: Input the total size to be read in 512-byte units. Like the start address, this can be entered in decimal or hexadecimal (with “0x” prefix) and must align to 4 KB.
3) Test pattern: Choose a data pattern to verify the data received from the target. Ensure the test pattern matches the one used during the Write command. Options include 32-bit incremental, 32-bit decremental, all 0, all 1, and 32-bit LFSR counter.
Once all inputs are validated, the read process begins. Current transfer data amounts are displayed every second on the console to indicate ongoing operation. Upon completion, the console displays the total size transferred, total time used, and test speed.
According to the NVMeTCP10G-IP specification, read performance varies with the read buffer size (illustrated in Figure 3‑11 and Figure 3‑12). Larger read buffer sizes may enhance read performance.
If data verification fails, an error message is displayed, detailing the first failure data including error byte address, expected value, and the read value (shown in Figure 3‑13 and Figure 3‑14). If verification fails, users can press any key to cancel the operation.
Pressing any key during the read operation will cancel it, although the command may still run as a background process. This can result in an incomplete sequence, and it is recommended to reset the FPGA board afterward. Additionally, the port connection on the target may not close properly; thus, changing the Admin/IO port number before reconnecting is advised.
Figure 3‑13 Test result when data verification fails without cancellation
Figure 3‑14 Test result when data verification fails with cancellation
3.5 Disconnect Command
Select ‘4’ to disconnect the host from the target. This menu terminates the connection between the NVMeTCP10G-IP (host) and the target, previously established in Section 3.2 (Connect Command). Upon selecting Disconnect, a confirmation prompt appears on the console. Respond with ‘y’ to proceed with the disconnection or press any other key to cancel the operation.
Figure 3‑15 Console result when running Disconnect command
Following a successful disconnection, the message “Disconnect target successful” will be displayed on the console. This indicates that the TCP/IP and NVMe/TCP connections have been successfully terminated.
After disconnecting, the user has the option to
· Reset network parameters via the ‘Set Network Parameter’ menu for adjustments or setting up a new target.
· Re-establish the connection with the same target using the ‘Connect’ command.
4 Revision History
|
Revision |
Date |
Description |
|
1.02 |
29-Apr-24 |
Add table of content |
|
1.01 |
25-Mar-22 |
Add topic 2.4 NVMe/TCP target removing on TestPC |
|
1.00 |
3-Nov-21 |
Initial version release |