LL10GEMAC-IP with AAT Demo Instruction
Rev1.2 14-Jun-23
2.1 IP Address configuration for two ports of 10G Ethernet
2.2 Installation of “tcpreplay”
1 Overview
This document provides instructions for setting up the Alveo accelerator card and preparing the test environment to run the Accelerated Algorithmic Trading (AAT) demo. The default demo can be obtained from the following link provided by Xilinx.
To achieve lower latency time, the AAT demo has been modified to utilize the LL10GEMAC-IP from Design Gateway instead of the 10G/25G Ethernet subsystem. The LL10GEMAC-IP latency details can be found in the datasheet available at
https://dgway.com/products/IP/Lowlatency-IP/dg_ll10gemacip_data_sheet_xilinx_en.pdf
Running the AAT demo requires an Alveo accelerator card plugged into a compatible system. The accelerator card features a QSFP+ connector, which supports up to 4x10G Ethernet connections. For the demo, two of the 10G Ethernet connections are used: one for transferring example market data via UDP protocol and another for order transmission via FIX over TCP. Therefore, the second system needs to be prepared by integrating two channels of 10G Ethernet connection. In this document, the second system is prepared using a PC with a 10G Ethernet card. The “tcpreplay” tool must be run on the second system to send the example market data, and a TCP port must be opened to receive orders from the accelerator card via TCP. The test operation on the accelerator card is controlled via the “aat_shell_exe”.
Before running the test, please prepare the following test environment.
- Alveo accelerator card: U50 or U250 card
- Turnkey accelerator system, TKAS-D2101, compatible with the Alveo accelerator card. More information can be found at
https://dgway.com/AcceleratorCards.html
- 10G Ethernet cable: QSFP+ to four SFP+ cable. Example cable can be found at
https://www.sfpcables.com/5-meter-40g-qsfp-to-4-sfp-aoc-cable-om3-mmf-cisco-oem-compatible
· The PC for 10G Ethernet transferring with the Turnkey accelerator system, including
o Ubuntu 20.04 LTS Server OS
o Example market data for running AAT demo
o TCPreplay package for transmitting market data
o Two ports of 10G Ethernet connection such as Intel X710-DA2
Figure 1‑1 LL10GEMACIP with AAT demo (FPGA <-> PC) on Alveo U250/U50 card
2 Test PC setup
This topic provides instructions on preparing the PC for transferring market data and order packets with the Alveo accelerator card. The example demonstrates the setup on Ubuntu 20.04 LTS Server OS.
2.1 IP Address configuration for two ports of 10G Ethernet
First, determine the logical names of the Ethernet ports that connect to the SFP+#1 and SFP+#2 cables. The logical names may vary depending on your test environment. It is important to configure the correct IP address for the SFP+#1 and SFP+#2 connections.
1) Open a Linux terminal and enter the command “lshw -C network” to list the logical names of the 10G Ethernet ports. Figure 2‑1 illustrates an example result displaying the logical names for two 10G Ethernet connections. In this case, “enp1s0f0” represents the Ethernet port for SFP+#1, while “enp1s0f1” corresponds to the Ethernet port for SFP+#2.
Figure 2‑1 Display logical name of 10G Ethernet port
2) Configure the IP address of SFP+#1 (enp1s0f0) as “192.168.10.100” and SFP+#2 (enp1s0f1) as “192.168.20.100” using the “ifconfig” command, as shown in Figure 2‑2. Additionally, set the netmask to 24 using the same command.
Figure 2‑2 Configure IP address and netmask
3) Use the “ifconfig” command to confirm the IP address and netmask after completing the configuration.
Figure 2‑3 Verify IP address and netmask setting
2.2 Installation of “tcpreplay”
The Test PC needs to have “tcpreplay” installed in order to run the AAT demo. To install the package, enter the command “sudo apt-get install tcpreplay” in the terminal, as shown in Figure 2‑4.
Figure 2‑4 “tcpreplay” installation
3 Test environment setup
This topic outlines the steps to prepare the Turnkey acceleration system (TKAS-D2101 with U250/U50) for running the AAT demo.
Figure 3‑1 QSFP+ channel using on U250 board
1) Connect the QSFP+ to four SFP+ cable (4x10G Ethernet cable) between the Alveo accelerator card (U50/U250) and the Test PC. Use the SFP+ connectors no.1 and no.2.
i) Insert a QSFP+ transceiver into the QSFP+ connector on the Alveo accelerator card. If you are using the U250 card, which has two QSFP+ channels, make sure to insert it into the QSFP2 channel.
ii) Connect SFP+ no.1 (192.168.10.100) and SFP+ no.2 (192.168.20.100) to 10G Ethernet channel on Test PC.
Figure 3‑2 Xilinx environment setting on TKAS-D2101 for U250
2) Prepare the Xilinx environment and library on the TKAS-D2101, which connects the U250/U50 card through the PCIe connector.
i) Enter the command to set up the Xilinx environment and library in the terminal
>> source <install path>/Vivado/2022.1/settings64.sh
ii) Enter the command to open The Xilinx runtime
>> source /opt/xilinx/xrt/setup.sh
iii) Enter the command to set the environment for the Accelerator card (U50 or U250).
>> export PLATFORM_REPO_PATHS='/opt/xilinx/platforms'
For U250 card
>> export XILINX_PLATFORM='xilinx_u250_gen3x16_xdma_4_1_202210_1'
For U50 card
>> export XILINX_PLATFORM='xilinx_u50_gen3x16_xdma_5_202210_1'
>> export DEVICE=${PLATFORM_REPO_PATHS}/${XILINX_PLATFORM}/
${XILINX_PLATFORM}.xpfm
3) For the U250 card, you need to program the shell partition to the Accelerator card once after system bootup. Refer to page 26 of UG1301 (V2.0) for detailed instructions.
i) User the following command to detect the Accelerator card connected to TKAS-D2101 and obtain the Card BDF ID.
>> sudo /opt/xilinx/xrt/bin/xbmgmt examine
Figure 3‑3 Examine the accelerator card
ii) Use the following command, along with the “card_BDF” (obtained from step 3i), to generate a JSON output file containing the path of the partition file.
>> sudo /opt/xilinx/xrt/bin/xbmgmt examine --report platform --format json --output <output file.json> --device <card_BDF>
Figure 3‑4 Generate JSON report file of xbmgmt
iii) Program the shell partition to the target Alveo card using the following command, along with the “card_BDF” and the path of the partition file determined in step 3i and step 3ii. Remove the backslash character (‘\’) from the path to the partition file in the output JSON file before using in this command.
>> sudo /opt/xilinx/xrt/bin/xbmgmt program --device <card_BDF> --shell <path to partition file>
Figure 3‑5 Program the shell partition
4) Copy the “aat.u250_DGLL10GEMAC.xclbin” file provided by DesignGateway to the directory “../Accelerated_Algorithmic_Trading/build”. The default path of “aat_shell_exe” in the AAT demo design is “../Accelerated_Algorithmic_Trading/build/bin”.
Note: If the build directory does not contain “aat_shell_exe”, you needs to build it following the instructions in Chapter 7 (Building and Running the AAT) of the “UG1067 Accelerated Algorithmic Trading User Guide” document.
5) Navigate to the directory that contains “./bin/aat_dgllip_shell_exe” and the xclbin file of the demo. Then, run the demo using the following command.
>> cd <directory of xclbin file >
>> ./bin/aat_shell_exe
>> download aat.u250_DGLL10GEMAC.xclbin or aat.u50_DGLL10GEMAC.xclbin
Figure 3‑6 Download xclbin file
4 Run AAT Demo
To execute the demo, there are three steps for the user. Firstly, there is an initialization process to establish the connection and configure the parameters. Next, the market data transmission process begins, which involves sending market data from the Test PC. Lastly, the test status is received from the Accelerator card and displayed on the console of TKAS-D2101, providing a comprehensive result. Further information about each of these processes is provided below.
4.1 Initialization
To run the AAT demo, the user needs to enter a command on the Test PC console. This command enables the Test PC to listen on a specific port, which is designated for receiving order packets from the Accelerator card once it has completed processing the market data. Similarly, the Accelerator card requires configuration to establish parameters for receiving market data and sending the order packet. This configuration is done using “demo_setup_cfg” script file. Further information regarding the system initialization process is outlined below.
1) On the Test PC console, enter the command to listen on port 12345.
>> nc -l 192.168.20.100 12345 -v
Figure 4‑1 Listen TCP port on Test PC
2) You will see a confirmation message on the console “Listening on <Test PC name> 12345” indicating that the port is listening, as shown in Figure 4‑1.
3) Run the script file “demo_setup.cfg” on TKAS-D2101 to set up the parameters for processing market data. Use the following command.
>> run support/demo_setup.cfg
Note: The demo_setup.cfg script file can only be run once after downloading the xclbin file. If you need to rerun the script file, you must re-download the xclbin file.
Figure 4‑2 Run demo setup script
4) TKAS-D2101 will display messages indicating the setup process, as shown in Figure 4‑2.
5) If the parameter configuration is successful, the Test PC console will display a message indicating that the port has been opened successfully, such as “Connection received on 192.168.20.200 32768”, as shown in Figure 4‑3.
Figure 4‑3 Open connection successes
4.2 Market data transmission
To transmit the sample market data, follow these steps using the “tcpreplay” on Test PC. Additionally, you will need to open two consoles on Test PC: Test PC Console#1 and Test PC Console#2. Test PC Console#1 will display details of the order packet received, while Test PC Console#2 will be used to send the sample market data. Here are the detailed instructions.
1) Send Sample market data. Use the “tcpreplay” command to send the provided sample market data file in Xilinx AAT demo (cme_input_arb.pcap). Execute the following command with four parameters.
>> sudo tcpreplay –intf1=<eth I/F> --pps=<pac/sec> --stats=<stat period> <replay file>
i) <eth I/F> : Ethernet interface for sending market data (SFP+#1: enp1s0f0).
ii) <pac/sec> : Transfer speed, defined as the number of packets per second.
iii) <stat period> : Time period in seconds to display status on the console.
iv) <replay file> : File name of the data to transmit, which is cme_input_arb.pcap (the sample market data provided by Xilinx AAT demo).
Note: Contact Xilinx to obtain the sample market data and AAT demo.
Figure 4‑4 Send sample market data by “tcpreplay”
2) The console will display the status, showing the total number of transmitted packets every second.
3) On Test PC console#1, which is already connected to the listening port, the console will display the received data, which represents the sample order packet returned by the Accelerator card and serves as the result of the AAT demo.
Figure 4‑5 The sample data of order packet on SFP+#2 channel
4.3 AAT demo
This topic shows the example results of market data processing on the Accelerator card. The AAT demo design comprises several kernels responsible for processing the sample market data. Specifically, we will focus on four kernels: Ethernet kernel, Feed handler kernel, Order book kernel, and Order entry kernel. The following details provide insights into the sample results obtained from these kernels within the AAT demo design.
4.3.1 Ethernet Kernel
To check the status of the Ethernet kernel in the AAT demo system, follow these steps.
1) Enter the following command to display the status of Ethernet kernel.
>> ethernet getstatus
Figure 4‑6: Ethernet kernel status
2) The AAT demo system has four Ethernet channels: channel0 to channel3. In this example, channel#0 is used for receiving sample market data, and channel#1 is used for returning the sample order packet. Verify that the status of channel#0 and channel#1 is in good condition by checking the following parameters.
i) Rx Block Lock Status (Live) : LOCKED
ii) GT Power Good (Live) : true
4.3.2 Feed Handler Kernel
To check the status of the Feed Handler kernel in the AAT demo system, follow these steps.
1) Enter the following command to display the status of the Feed handler kernel.
>> feedhandler getstatus
2) The console will display the processed data count in various units, such as bytes, packets, and messages. Initially, before transmitting the sample market data, the processed data count will be zero. However, after transmitting the market data, the processed data count will no longer be zero.
Figure 4‑7 Feed handler kernel status
For example, in Figure 4‑7, the left window shows that the processed data count is initially zero. After transmitting the sample market data, the processed data count will increase, indicating that the Feed Handler kernel is processing the data.
In Figure 4‑4, it shows that there were 104 packets of sample market data sent by the Test PC. Out of these packets, 53 packets were deemed valid for Feed Handler processing, while the remaining packets were rejected by the Feed handler kernel.
4.3.3 OrderBook Kernel
To check the status of the OrderBook kernel in the AAT demo system, follow these steps.
1) Enter the following command to display the order book output from the Order Book kernel.
>> orderbook readdata
2) The console will show the current values of the order book. Initially, before transmitting the sample market data, the order book will be a clean state. However, after transferring all the sample market data, the order book will be updated by the OrderBook kernel.
Figure 4‑8 Updated OrderBook after finishing processing
For instance, in Figure 4‑8, the left window displays the clean status of the order book before transmitting the sample market data. The right window shows the updated order book after transferring all the sample market data. The bid/ask quantity and bid/ask price are updated by the OrderBook kernel, reflecting the changes in the market data.
4.3.4 Order Entry Kernel
To check the current status of the Order Entry kernel in the AAT demo system, follow these steps.
1) Enter the following command to display the current status of the Order Entry kernel.
>> orderentry status
2) Before starting the transmission of sample market data, you can check the status of Ethernet channel#1 from the Order Entry kernel status. Two key indicators to check the connection status are as follows.
i) Connection Established : true
ii) Connection Status : SUCCESS
Figure 4‑9 Order Entry kernel status
3) After transmitting the market data completely, the packet count of the Order Entry kernel will be updated from 0 to a new value. This value represents the total number of message/frames processed by the Order Entry kernel.
5 Revision History
|
Revision |
Date |
Description |
|
1.2 |
26-May-23 |
Add table of contents and correct the website of an example cables |
|
1.1 |
7-Sep-22 |
Support AAT2022Q1 released version |
|
1.0 |
11-Oct-21 |
Initial version release |