High-Speed File-Based Data Logging for 100G Ethernet Sustained Powered by NVMe-IP for Gen5 and exFAT2-IP

The Challenge: High-Speed Data Logging

As network and sensor data rates continue to increase, many data logger systems struggle to maintain sustained recording speed — especially when file system support is required. CPU-based architectures often become the bottleneck, leading to:

Dropped data and packet loss
Complex software stacks
Limited scalability

This FPGA-based data logging project demonstrates a standard, reusable data logger architecture, implemented with 100G Ethernet to clearly showcase high-performance capability.

FPGA-Based Reference Architecture

This project is built as an FPGA-based system, where all time-critical data handling is performed in hardware logic. Incoming data is buffered, processed, and recorded without relying on a CPU for the data path.

100G Ethernet Data Logger system architecture illustrating real-time traffic capture between network devices, DDR buffering, and sustained file-based recording to NVMe Gen5 SSD using NVMe-IP and exFAT2-IP for high-performance data center storage applications.

To demonstrate a demanding real-world scenario, the system is implemented as a 100G Ethernet data logger. This closed application allows customers to easily understand system behavior, performance limits, and design advantages — while still serving as a flexible base for other applications.

Watch the YouTube Demo Video

▶ Watch the demo on YouTube

Sustained Recording with DDR Buffering

One of the key challenges in high-speed logging is maintaining a stable, sustained write rate. In this system:

Incoming Ethernet data is temporarily buffered in DDR memory
DDR absorbs burst traffic and rate variations
Data is written to storage at a steady, sustained speed

This buffering mechanism ensures continuous recording without packet loss, even under full line-rate traffic.

High-Speed File-Based Recording (PCAPNG)

Unlike raw data dump solutions, this system records data directly into a standard file system format:

Packets are converted in FPGA logic into PCAPNG format
Files are written using exFAT, enabling large-capacity storage
Recorded files can be opened directly in Wireshark
No post-processing or conversion is required

This makes captured data immediately usable for analysis, debugging, and validation.

Built on Proven Storage IP Cores

The system is powered by two key IP cores that work together seamlessly:

IP Core	Key Features
NVMe-IP (PCIe Gen5)	High-speed NVMe SSD interface, optimized for sustained continuous write performance, designed for FPGA-based storage acceleration
exFAT2-IP	File system support directly in hardware, PC-compatible removable storage, simple data access without special tools

Key Features of NVMe-IP Gen5 and exFAT2-IP for High-Speed 100G File-Based Data Logging

Together, these IPs combine high-speed recording with easy data retrieval, creating a smooth workflow from capture to analysis.

Packet Filtering via Software Configuration

To improve usability, the system includes packet filtering control through a user application:

Users can configure filtering rules easily
No FPGA redesign is required for filter changes
Only selected packets are recorded, reducing storage load

This makes the system practical for real-world logging and monitoring scenarios.

Scalable on Agilex 7 FPGA Platform

The demo is implemented on an Intel Agilex 7 FPGA board, and the hardware resources can be adjusted to match final system requirements.

Examples of scalability include:

Component	Demo System	Board Capability
Ethernet	100G Ethernet	Up to 400G Ethernet
Memory	Single DDR component	Multiple DDR channels
Storage	Single NVMe SSD	Up to 8 NVMe SSDs

Scalable 100G Ethernet Data Logging Architecture – From Single NVMe to 400G and Multi-SSD Deployment

This flexibility allows customers to scale performance without changing the core architecture.