Requirements
Our client (which cannot be revealed due to NDA purposes) builds defence systems for homeland security. And their analytics division needed a workstation capable of high-volume data processing.
And they they approached us with a challenge:
“We have a drone surveillance system that generates 1 Giga Byte of data per second in the form of 1 KB packets over 10Gbps(bits) Network.
We need a system that can read this kind of throughput for 6 hours and store it in the disk.”
The requirement was clear: configure a system with at least 26 TB of SSD storage and the processing muscle to read 1 GBps from multiple network senders.
The significant constraint was an extremely tight budget. An enterprise-grade solution for this workload typically cost over 25 lakh INR; The client’s budget was less than half of that.
Original Plan
Our client had come prepared. They outlined a configuration built around Intel’s Ultra platform, stacked with seven high-speed 4TB NVMe drives and a dual 10-gigabit network interface.
Theoretically – you could fit all these parts together, but our testing team said it wouldn’t work.
The Challenge
Because consumer desktop platforms (Ultra & Ryzen) don’t give you enough PCIe lanes to run 7 NVMe drives and a high-speed Network card – because of its chipset limitations.
There is actually no consumer-grade motherboard that can fit 7 NVMEs – so 4 of these drives had to be installed on a NVMe expansion card – it looks like a Graphics Card, but it’s actually an enclosure which gives you 4 additional NVMe slots on top of the ones already present in the motherboard.
But to install this on your system – a special BIOS setting is required (4×4 PCIe Bifurcatiuon) which isn’t available on desktop grade platforms.
A quick demonstration of their workflow on one of our demo systems proved that this setup would not meet their requirements.
The speeds were slower, and the Expansion card was only detecting 3 SSDs. (6 in total)
Our Solution
We proposed moving to a workstation-grade AMD Threadripper platform:
- CPU: AMD Threadripper 3960X
- Motherboard: TRX40 Creator
- RAM: G.Skill Trident Z 32GB x2 @ 3600 MHz – ×2 (128GB total)
- GPU: Asus GT 710 (basic display, because their workload wasn’t GPU-bound)
- Storage:
- Sabrent Rocket Q4 4TB NVMe – ×6 in RAID 0 (primary high-speed array)
- Sabrent Rocket Q4 4TB NVMe – ×1 as a separate drive
- NVMe Expansion: MSI Aero Xpander Gen4
- PSU: 750W 80+ Gold Rated Power Supply
- Cabinet: 4U rackmount chassis
Because this (TRX40) platform has 88 PCIe lanes. More than enough for the NVMe expansion card and an optical NIC that can run at full speed in parallel without bottleneck.
Another benefit was HEDT platforms are factory-tested for longer and sustained performance throughput – which was preferred over theoretical peak numbers.
Yes, this configuration was more expensive on paper and would originally overshoot their original system price by a few lakhs.
But it was still well within their budget and the only cost effective solution without any compromises.
The Challenge: Returns 😛
4U racks aren’t made for desktop grade components – so everything that we chose was carefully measured to make sure that we don’t get something that doesn’t fit.
However – never does anything go exactly as planned. The Cooler didn’t fit.
while the tech specs of the Noctua NH-U14s exactly matched with the width of the rack. At the time of installation, we saw that the heatpipes protruded out of the cooler’s heatsink and went outside the limit.
Then, after pulling a few strings. a special enterprise-grade 10K RPM cooler (Dynatron A35 TR4) was procured to make our configuration work.
Testing and Validation
Before calling the build complete, we conducted a test to make sure our solution met the requirement of the client.
- Connected the workstation with a secondary test system with a 10Gbe Ethernet.
- Sent data to the test system in chunks of 1KB at the following rates of 5Gbps, 6Gbps, 7Gbps up to 10 Gbps.
- Stored the data on the workstation in SSD in Realtime without any data loss.
- Checked if this operation is sustaining for at least six hours continuously.
Outcome
The final configuration achieved a write speed of 15 GB per second and reliably recorded data for over 24 hours, far exceeding the six-hour requirement.
And the project was completed in a Budget of 8 lakhs (roughly 1/4th of similar enterprise grade solution.)
