This document describes how to setup Alveo accelerator card and prepare the test environment for running AAT (Accelerated Algorithmic Trading) demo. The default demo is provided by Xilinx from following link.
The AAT demo is modified to use LL10GEMAC-IP, UDP10GRx-IP, and TOE10GLL-IP from Design Gateway instead of 10G/25G Ethernet subsystem, UDP/IP kernel, and TCP/IP kernel to achieve the lower latency time. More details of LL10GEMAC-IP, UDP10GRx-IP, and TOE10GLL-IP latency time are described in the datasheet.
https://dgway.com/products/IP/Lowlatency-IP/dg_ll10gemacip_data_sheet_xilinx_en.pdf
https://dgway.com/products/IP/Lowlatency-IP/dg_udp10grxip_data_sheet_xilinx_en.pdf
https://dgway.com/products/IP/Lowlatency-IP/dg_toe10gllip_data_sheet_xilinx_en.pdf
The AAT demo is run by using Alveo accelerator card plugged into the system that supports to run the accelerator card demo. The accelerator card has QSFP+ connector which supports up to 4x10G Ethernet connection. The demo uses at least two of 10G Ethernet connections, one for transferring example market data via UDP protocol and another for transferring the trade order via FIX over TCP. Therefore, the additional system with two of 10G Ethernet connections must be prepared.
In this document, Test PC is prepared integrating a 10G Ethernet card that supports two 10G Ethernet connections. “tcpreplay” is run to send the sample market data. While “TCP port” is opened to receive the trade order from the accelerator card via TCP. The accelerator card is plugged-in to Turnkey accelerator system and the user runs the test operation on the accelerator card by using “aat_dgllip_shell_exe”.
1 Test environment
Before running the test, please prepare following test environment.
- Alveo accelerator card: U250 card or U50 card
- Turnkey accelerator system, TKAS-D2101, for Alveo accelerator card
https://dgway.com/AcceleratorCards.html
- 10 Gb Ethernet cable: QSFP+ to four SFP+ cable
https://www.finisar.com/active-optical-cables/fcbn510qe2cxx
· Test PC for 10Gb Ethernet transferring with Turnkey accelerator system
o Ubuntu 20.04 LTS Server OS
o Sample market data for running AAT demo
o TCPreplay package for transmitting market data
o Two ports of 10G Ethernet connection such as Intel X710-DA2
Figure 1‑1 DG low-latency IPs with AAT demo (FPGA <-> PC) on Alveo U250/U50 card
2 Test PC setup
This topic shows how to prepare Test PC for transferring market data and the trade order with Alveo accelerator card. The example is the setting on Ubuntu 20.04 LTS Server OS.
2.1 IP Address setting for two ports of 10Gb Ethernet
Please confirm the logical name of Ethernet port that connects to SFP+#1 and SFP+#2 cable. The logical name depends on the test environment. It needs to configure the correct IP address for SFP+#1 and SFP+#2 connection.
1) To list the logical name of 10G Ethernet port on Linux terminal, type “lshw -C network”. Figure 2‑1 shows the example result of the logical name of two 10Gb Ethernet connections. In this figure, “enp1s0f0” is Ethernet port of SFP+#1 while “enp1s0f1” is Ethernet port of SFP+#2.
Figure 2‑1 Display logical name of 10G Ethernet port
2) Configure IP address of SFP+#1 (enp1s0f0) to “192.168.20.100” and SFP+#2 (enp1s0f1) to “192.168.10.100” respectively by using “ifconfig” command, as shown in Figure 2‑2. Besides, the netmask 24 is set by using the same command.
Figure 2‑2 Configure IP address and netmask
3) Use “ifconfig” command to confirm IP address and netmask after setting completely.
Figure 2‑3 Verify IP address and netmask setting
2.2 “tcpreplay” installation
Test PC needs to install tcpreplay for running AAT demo. To install the package, type the command “sudo apt-get install tcpreplay” on the terminal, as shown in Figure 2‑4.
Figure 2‑4 Install tcpreplay
3 Test environment setting
This topic shows the steps to prepare Turnkey acceleration system (TKAS-D2101 with U250/U50) to run AAT demo.
Figure 3‑1 QSFP+ channel using on U250 board
1) Connect QSFP+ to four SFP+ cable (4x10G Ethernet cable) between Alveo accelerator card (U250/U50) and Test PC. Two SFP+ connectors (SFP+ no.1 and no.2) are applied.
i. On U250 card, it has two QSFP+ channels, so insert QSFP+ cable to QSFP2 channel. While there is only one QSFP+ channel that can insert QSFP+ cable on U50 card.
ii. Connect SFP+ no.1 (192.168.20.100) and SFP+ no.2 (192.168.10.100) to 10Gb Ethernet channel on Test PC.
Figure 3‑2 Xilinx environment setting on TKAS-D2101 for U250
2) Prepare Xilinx environment and library on TKAS-D2101 which connects U250/U50 through PCIe connector.
i. Input command to setup Xilinx environment and library on terminal
>> source <install path>/Vivado/2022.1/settings64.sh
ii. Input command to open Xilinx run time
>> source /opt/xilinx/xrt/setup.sh
iii. Input command to set environment for the accelerator card.
>> export PLATFORM_REPO_PATHS='/opt/xilinx/platforms'
For U250 card
>> export XILINX_PLATFORM='xilinx_u250_gen3x16_xdma_4_1_202210_1'
For U50 card
>> export XILINX_PLATFORM='xilinx_u50_gen3x16_xdma_5_202210_1'
>> export DEVICE=${PLATFORM_REPO_PATHS}/${XILINX_PLATFORM}/
${XILINX_PLATFORM}.xpfm
3) For U250 card, it needs to program the shell partition to accelerator card once after system bootup. More details are described in page 26 of UG1301 (V2.0).
i. Detect the accelerator card which connects on TKAS-D2101 to get the Card BDF ID by using following command.
>> sudo /opt/xilinx/xrt/bin/xbmgmt examine
Figure 3‑3 Examine the accelerator card
ii. Determine the partition file with full path by using following command to generate the JSON output file which contains the path of partition file. “card_BDF” (output from step 3i) is applied to be a parameter for this command.
>> sudo /opt/xilinx/xrt/bin/xbmgmt examine --report platform --format json --output <output file.json> --device <card_BDF>
Figure 3‑4 Generate JSON report file of xbmgmt
iii. Program the shell partition to the target Alveo card by the following command along with the “card_BDF” and the path of partition file which are determined by step 3i and step 3ii. The path to partition file in output JSON file needs to remove the backslash character (‘\’) before using in this command.
>> sudo /opt/xilinx/xrt/bin/xbmgmt program --device <card_BDF> --shell <path to partition file>
Figure 3‑5 Program the shell partition
4) Copy “aat.U250/U50_DGLLIP.xclbin” file that is provided by DesignGateway to the directory “../Accelerated_Algorithmic_Trading/build”.
The default path of software application “aat_dgllip_shell_exe” in AAT demo design is
“../Accelerated_Algorithmic_Trading/build/bin”.
Note: Xilinx AAT reference design uses “aat_shell_exe” as a software name. Since the software is modified, it is re-named to “aat_dgllip_shell_exe”. If there is no “aat_dgllip_shell_exe” in bin directory, user needs to build it following the description in chapter 7 (Building and Running the AAT) of “UG1067 Accelerated Algorithmic Trading User Guide” document.
5) Browse to directory that includes “./bin/aat_dgllip_shell_exe” and xclbin file of the demo. After that, run the demo using following command.
>> cd <directory of xclbin file>
>> ./bin/aat_dgllip_shell_exe
>> download aat.U250_DGLLIP.xclbin or aat.U50_DGLLIP.xclbin
Figure 3‑6 Download xclbin file
4 Run AAT Demo
To run the demo, there are three steps for user running. The first step is the initialization process for preparing the connection and parameter configuration. Next is market data transmission process to start sending market data from Test PC. The last step is the test status monitoring that is returned by the accelerator card to show the result on the console of TKAS-D2101. More details of each step are described as follows.
4.1 Initialization
To run AAT demo, the user must input the command on Test PC console to listen the specific port that is applied to receive the order packet from accelerator card after finishing processing market data. Similarly, the accelerator card must be configured to set up the parameters for receiving market data and sending the order packet by using script file “demo_setup_cfg”. More details for system initialization are described as follows.
1) On Test PC console, type following command to listen the port no. 12345.
>> nc -l 192.168.20.100 12345 -v
Figure 4‑1 Listen TCP port on Test PC
2) After that, the confirmation message (“Listening on <Test PC name> 12345”) is displayed on the console to confirm port is listening, as shown in Figure 4‑1.
3) Run script file on TKAS-D2101 to setup the parameters for processing market data by using following command.
>> run support/demo_setup.cfg
Note: demo_setup.cfg can run only one time after downloading xclbin file. To re-run the script file, it needs to re-download xclbin file.
Figure 4‑2 Run demo setup script
4) After that, TKAS-D2101 displays the message from setup process, as shown in Figure 4‑2.
5) If the parameter is configured successfully, Test PC console displays the message that the port is opened successfully (“Connection received on 192.168.20.200 10201”), as shown in Figure 4‑3.
Figure 4‑3 Open connection is done
4.2 Market data transmission
This topic shows how to run “tcpreplay” on Test PC to send sample market data. Also, the result on Test PC is displayed on the listening port when the accelerator card returns the order packet. Therefore, the user needs to open two consoles on Test PC, Test PC Console#1 and Test PC Console#2. Test PC Console#1 shows the details of the order packet while Test PC Console#2 is applied to send the sample market data. More details to transmit sample market data are described as follows.
1) Send sample market data (cme_input_arb.pcap) by using “tcpreplay” command. Type following command with four parameters.
>> sudo tcpreplay –intf1=<eth I/F> --pps=<pac/sec> --stats=<stat period> <replay file>
i) <eth I/F> : Ethernet interface for sending market data (SFP+#2: enp1s0f1)
ii) <pac/sec> : Transfer speed by setting number of packets per second
iii) <stat period> : Set period time in second unit to display status on console
iv) <replay file> : File name to transmit the data, cme_input_arb.pcap
Note: cme_input_arb.pcap is the sample market data, provided by Xilinx AAT demo. Please contact Xilinx to request the sample market data and AAT demo.
Figure 4‑4 Send sample market data by “tcpreplay”
2) After that, the console displays the status to show total number of transmit packets every second.
3) On Test PC Console#1 which the port has already connected, the console displays the received data which is the sample order packet, returned by the accelerator card to be the AAT demo result.
Figure 4‑5 The sample data of order packet on SFP+#1 channel
4.3 AAT demo
This topic shows the example result of market data processing on the accelerator card. There are many kernels for processing the sample market data. This topic shows the result of four kernels in AAT demo design, i.e., Network kernel, Feed handler kernel, Order book kernel, and Order entry kernel. More details of the sample results are shown as follows.
4.3.1 Network Kernel
1) User inputs the following command to display the status of Network kernel.
>> network getstatus
Figure 4‑6: Network kernel status
2) There are four network channels (channel0 – channel3) on AAT demo system. The document shows the example of using channel#0 for returning sample order packet and channel#1 for receiving sample market data. Please confirm the status of channel#0 and channel#1 are in good status.
a) Kernel Reset (Main) : false
b) Linkup Status (channel0 and 1) : LOCKED
c) TOE-IP Session Active (channel0) : true
d) TOE-IP Session Number (channel0) : Number 0
e) TOE-IP Status and State (channel0) : Connected and STPACT
f) UDP-IP Session Active (channel1) : true
g) UDP-IP Session Number (channel1) : Number 0 and 1
Note: Network parameters are configured using script file as described in topic 4.1.
Figure 4‑7 Network kernel status after receiving market data
After transferring the market data, the trade order is generated via Channel#0. The data transmission to show the amount of completed transmitted data in byte unit is updated, as shown in Figure 4‑7.
4.3.2 Line Handler Kernel
1) User inputs the following command to display the status of Line handler kernel.
>> linehandler getstatus
2) Before sending the market data from Test PC, the Line Handler kernel must be set to map the UDP data from Network kernel to the specific split ID as follows.
a) Line Handler input port 0 maps the UDP session number 0 to Split ID 0
b) Line Handler input port 0 maps the UDP session number 1 to Split ID 0
Figure 4‑8 Line Handler status
4.3.3 Feed Handler Kernel
1) User inputs the following command to display the status of Feed handler kernel.
>> feedhandler getstatus
2) The console displays the processed data count in several units such as bytes, packets, and messages. As shown in Figure 4‑9, the left window shows the processed data count is equal to zero before starting transmitting sample market data. After finishing transmitting market data, the processed data count is not equal to zero.
Figure 4‑9 Feed handler kernel status
As shown in Figure 4‑4, there are 104 packets of sample market data sent by Test PC. However, there are 53 packets that are valid for Feed handler processing. Other packets are rejected by Line handler.
4.3.4 OrderBook Kernel
1) User inputs the following command to display the order book, the output from Order Book kernel.
>> orderbook readdata
2) The console displays the current value of order book. As shown in Figure 4‑10, the left window shows clean status of order book before the system transmits sample market data. While the right window shows the updated order book after transferring all sample market data. The bid/ask quantity and the bid/ask price are updated by OrderBook kernel.
Figure 4‑10 Updated OrderBook after finishing processing
4.3.5 Order Entry Kernel
1) User inputs the following command to display the current status of Order Entry kernel.
>> orderentry getstatus
2) After starting AAT demo and before starting sending the sample market data, the user can check the status of Network channel#0 (TCP/IP) from Order Entry kernel status.
i) Connection Established : true
ii) Current Connection Status : true
Figure 4‑11 Order Entry kernel status
3) After transmitting the sample market data completely, the packet count of Order Entry is updated from 0 to the new value to show total numbers of message/frames that is processed by Order Entry kernel.
5 Revision History
|
Revision |
Date |
Description |
|
1.1 |
7-Sep-22 |
Add TOE10GLL-IP |
|
1.0 |
4-Jan-22 |
Initial version release |