Questions tagged [infiniband]

InfiniBand is a switched fabric computer network communications link used in high-performance computing and enterprise data centers. This tag should be used for questions about IB related hardware and software.

InfiniBand(IB) is a switched fabric computer network communications link used in high-performance computing and enterprise data centers.

Its features include high throughput, low latency, quality of service and failover, and it is designed to be scalable. The InfiniBand architecture specification defines a connection between processor nodes and high performance I/O nodes such as storage devices. InfiniBand host bus adapters and network switches are manufactured by Mellanox and Intel.

In addition to hardware manufactured by Mellanox and Intel there are a variety of software stacks available for the configuration and management of IB fabrics. Most major Linux distributions have some built-in support for Infiniband technologies, and the Open Fabrics Alliance(OFA) maintains the Open Fabrics Enterprise Distribution(OFED) which provides many software and firmware packages for the management, maintenance and setup of IB networks. In addition to the 'official' OFED maintained by the OFA, Mellanox and Intel both produce software stacks for their products that typically include their own customized versions of OFED.

116 questions
15
votes
4 answers

How can I determine which switch the Infiniband subnet manager is running on?

I recently inherited an Infiniband network containing multiple switches, and I know that one of these switches is running the subnet manager. The rest supposedly have that feature turned off, or were never enabled. The trouble is, I have no idea…
ajdecon
  • 1,311
14
votes
2 answers

what does `pipe ` mean in `ping` output?

While running a test to diagnose some alleged network problem, I got this output from ping: --- r01c01b11n02 ping statistics --- 30000 packets transmitted, 29995 received, 0% packet loss, time 6309ms rtt min/avg/max/mdev = 0.091/0.343/118.051/3.605…
9
votes
1 answer

Connect two infiniband cards to each other without a switch

Is it possible to connect two Mellanox ConnectX-2 cards on two seperate computers to each other without a switch in between? I am just trying to learn to use verbs programming without having to spend $ on an expensive switch. Do I need a special…
Ivan
  • 309
8
votes
1 answer

File-sharing over Infiniband on FreeBSD

I have a pair of servers, which I'd like to connect to each other directly -- rather than through the switch shared by the rest of the network. I expect to benefit from lower latency and higher bandwidth afforded by the direct connection (including…
Mikhail T.
  • 2,405
8
votes
1 answer

IPoIB (IP over InfiniBand) vs. RDMA performance

I have partly inherited a Linux HA cluster at the center of which currently serves a connection with DRBD 8 over IPoIB (IP over InfiniBand) between two Debian hosts. It ain't broken, so I won't fix it. I have also noticed that DRBD 9 supports RDMA,…
rookie09
  • 673
6
votes
2 answers

Compatibility and difference between QSFP+ and QSFP 40Gb/s infiniband connections

I am new to infiniband networks, but i need to extend an existing infiniband cluster with some nodes. During my exploratory web-research I came across some to me ambiguous descriptions of cables, connectors and switches. According to Wikipedia QSFP…
Hjan
  • 163
6
votes
4 answers

switchless Infiniband between two servers on RHEL 6.3

I have 2 servers running RHEL 6.3 which have 2 port Infiniband cards >lspci | grep -i infini 07:00.0 InfiniBand: QLogic Corp. IBA7322 QDR InfiniBand HCA (rev 02) I'm interested in connecting them directly to each other bypassing an Infiniband…
exfizik
  • 161
5
votes
1 answer

Passing through RDMA network devices to docker containers

I'm wanting to passthrough infiniband to a docker container so that I can run some high performance apps over ipoib and use rdma. Currently, I'm doing this with xen virtual machines. Now I'm looking into using CoreOS and docker as a much lighter…
hookenz
  • 14,848
5
votes
2 answers

Infiniband port status UP but can't open UMAD port ((null):0)

My system has 2 infiniband devices, one of which has both the ports up. $> ibstatus Infiniband device 'mlx4_0' port 1 status: default gid: fe80:0000:0000:0000:0002:c903:000f:0a9f base lid: 0x22 sm lid: …
Sidjana
  • 51
5
votes
2 answers

Virtual infiniband on kvm/qemu or another open source platform?

Is there any way to set up a virtual infiniband network within kvm/qemu? If not, is there a viable open source alternative which would allow virtual infiniband networking? (Note that I'm not looking to give VMs access to a physical IB network, I'm…
kdt
  • 1,450
5
votes
1 answer

Mellanox ConnectX-7 - Unable to get more than 250 Gbit/s

I have several machines with ConnectX-7 Infiniband cards and they're plugged into an Nvidia QM9700 switch. I've confirmed 400 Gbit NDR at both ends (ibstat on the host and in the console on the switch). The machines are running Ubuntu 22.04 and…
Evan
  • 497
4
votes
2 answers

Set up Windows 10 SRP initiator

How do can I setup an SRP Windows 10 initiator? I've seen several mentions of an initiator being included in WinOF but I haven't found any instructions on how to do it. The initiator will be Windows 10 Pro with a Connect-X2 DDR interface. I have…
4
votes
2 answers

Infiniband Verifying that RDMA is working

I have two identical computers with Mellanox cards connected to each other through a cable. No switch. Using opensm. I have run several tests, including ping_pong tests, ibping, etc. They all seem to work. However, when I run this test, it comes…
Ivan
  • 309
4
votes
1 answer

Infiniband + KVM

this question is a copy from stackoverflow, as I was told, my question would fit here better. Despite quite a long search for days, I could not find any good information about the following - though I think, I can not be the first to have this…
Daniel
  • 41
4
votes
1 answer

How do I enable SR-IOV for Mellanox ConnectX-3 Ubuntu 12.04.3?

I have a Supermicro 2027TR-HTFRF server that I'm using as a xen server. What I'm wanting to do is get SR-IOV operating for the Connectx-3 card so that I can expose Infiniband to virtual machines. Following this recipe I managed to get the I350…
hookenz
  • 14,848
1
2 3 4 5 6 7 8