Homelab

380 readers

9 users here now

Rules

Be Civil.
Post about your homelab, discussion of your homelab, questions you may have, or general discussion about transition your skill from the homelab to the workplace.
No memes or potato images.
We love detailed homelab builds, especially network diagrams!
Report any posts that you feel should be brought to our attention.
Please no shitposting or blogspam.
No Referral Linking.
Keep piracy discussion off of this community

founded 1 year ago

MODERATORS

[email protected]

Looking to build a 10Gbit network from scratch. (alien.top)

submitted 11 months ago by [email protected] to c/[email protected]

3 comments fedilink hide all child comments

I'm looking to upgrade my whole net to 10Gbit and i have two fairly inexpensive paths ahead of me:

X540-T2, which are fairly cheap in ebay but im not sure if they're chinese copycats or the real deal, i've heard about reliability issues, like not being recognized at all by motherboards.
Mellanox Cx3, a bit cheaper than X540-T2 but i don't know if they're legit or knock offs.

Both options are fairly old by 2023, almost 2024, both hit EoL already and i'm wondering if i should look for newer nics or just get this. Judging from what i've read i think i want to go with fiber instead DAC, since nics runs cooler (Lower temps would be really appreciated in my case), apparently they take less power to function and the price of the setup would be also lower for the Cx3 option. I'm a bit on a budget too so i don't want to go spending 100€+ per nic, i would need just 3 for now.

When it comes to switches, i think one QNAP QSW-2104-2T/S (Or a very similar one) will do the trick for now, i'll get something better once i can find multi 10Gbit port switches below 400€.

top 3 comments

sorted by: hot top controversial new old

[–] [email protected] 1 points 11 months ago (2 children)

I welcome a correction, but my understanding is that the X540 are needlessly toasty - even by 10Gbe NIC standards. Maybe take aim for the X550 instead?

Also, if you go SFP+ ; use DAC where ever you can. They produce almost no heat since there's no conversion of the signal.

Mellanox connectx-3's are wonderfull. Mine may be a counterfeit, I have no idea, but it works great and runs cool. Only downside is the cards limited support for sleep. The card is low power in it self, but it will keep the cpu from going in to lower levels of rest during inactivety. So keep that in mind if you plan to use it for something that idles a lot.

What's the goal? Purge gigabit, or keep some connections? What connections and why? How many machines, and what kind och PCIe slots ( lanes and version) do you intendent to use?

Good luck!

[–] [email protected] 1 points 11 months ago

I welcome a correction, but my understanding is that the X540 are needlessly toasty - even by 10Gbe NIC standards. Maybe take aim for the X550 instead?

I've read this a lot, that X540 runs really hot the the point the connection can end up throttling and killing the 10Gbit speed.

Mellanox connectx-3's are wonderfull. Mine may be a counterfeit, I have no idea, but it works great and runs cool. Only downside is the cards limited support for sleep. The card is low power in it self, but it will keep the cpu from going in to lower levels of rest during inactivety. So keep that in mind if you plan to use it for something that idles a lot.

I want my current server and the upcoming nas to go into idle as much as they can, every other equipment won't have to since they're personal computers, so it will be 2, maybe 3 machines max into sleep mode or high idle status.

What's the goal? Purge gigabit, or keep some connections? What connections and why? How many machines, and what kind och PCIe slots ( lanes and version) do you intendent to use?

I want to bring every room in my house to 10Gbit. I can't purge all 1Gbit devices because some can't be upgraded to 10Gbit (TV, brother's iMac, Raspberry-like devices, etc.). I got an offer from a local isp for 10Gbit internet speed for 25€/month or 1Gbit for 20€/month and want to lay the ground for every device that can be upgraded to support 10Gbit in case i take the offer.

For now it will be just my server (A Lenovo M83P SFF, according to Lenovo's data sheet both PCIe can draw up to 40w, one is 3.0 x16 and the other is 2.0 x4) and my personal computer (Mother board is a GA-B550M-DS3H, I'll use the bottom PCIe which is 3.0 x4). I plan to build a new NAS next year and possibly a network machine for pfSense, net-wide adblock, VPN, etc., i'm currently educating myself on the topic so i might integrate everything into the Lenovo or just buy a specific machine designed for networking.

[–] [email protected] 1 points 11 months ago

Check out https://forums.servethehome.com/index.php?threads/sfp-cards-with-aspm-support.36817/page-2 to find out how to enable ASPM on ConnectX-3s.

powertop shows the card reaches C6 in my Z620 running Proxmox.