Skip to main content

Infiniband

The infiniband node-type implements node communication over the Infiniband standard.

Prerequisites

This node-type requires libibverbs (>= 16.2) and librdmacm (>= 16.2).

Implementation

The source code of the node-type is available here: https://github.com/VILLASframework/node/blob/master/lib/nodes/infiniband.cpp

Configuration​

Every infiniband node can be configured to only read or write or to do both at the same time. The node configuration is divided into two sub-groups: in and out.

rdma_port_space
string
Default: "RC"
Enum: "RC" "UC" "UD"

This specifies the type of connection the node will set up.

  • RC provides reliable, connection-oriented, message based communication between the nodes. Packets are delivered in order. In this mode, one Queue Pair is connected to one other Queue Pair.
  • UC provides unreliable, connection-oriented, message based communication between the nodes. This service type is not officially supported by the RDMA communication manager and is implemented for scientific purposes in VILLASnode. The InfiniBand node-type source code provides information on how to enable this service type.
  • UD provides unreliable, connection-less, datagram communication between nodes. Both ordering and delivery are not guaranteed in this mode.

RC, UC, and UD are mapped to the Queue Pair types as RDMA_PS_TCP/IBV_QPT_RC, RDMA_PS_IPOIB/IBV_QPT_UC, and RDMA_PS_UDP/IBV_QPT_UD, respectively. If two nodes should be connected, both should be set to the same rdma_port_space.

More information on these two modes can be found on the manual page for rdma_create_id().

object (Input configuration (received by VILLASnode))
object (Output configuration (sent out by VILLASnode))
vectorize
integer
Default: 1

This setting allows to send multiple samples in a single message to the destination nodes.

The value of this setting determines how many samples will be combined into one packet.

Array of Hook Object (object) or Hook Name (string) (hook_list)
builtin
boolean (Builtin hook functions)
Default: true

By default, each node and paths has a couple of default hooks attached to them. With this setting the attachment of built-in hooks can be disabled.

{
  • "rdma_port_space": "RC",
  • "in": {
    },
  • "out": {
    },
  • "vectorize": 1,
  • "hooks": [
    ],
  • "builtin": true
}

Example​

node/etc/examples/nodes/infiniband.conf
nodes = {
    results = {
        type = "file"

        uri = "logs/ib_results-%Y%m%d_%H-%M-%S.log"
    }

    siggen = {
        type = "signal"

        signal = "mixed"
        values = 3
        frequency = 3.0
        rate = 100000.0
        limit = 100000
    }

    ib_node_source = {
        type = "infiniband"

        rdma_port_space = "RC"

        in = {
            address = "10.0.0.2:1337"

            max_wrs = 8192
            cq_size = 8192

            vectorize = 1

            buffer_subtraction = 128
        }

        out = {
            address = "10.0.0.1:1337"
            resolution_timeout = 1000

            max_wrs = 8192
            cq_size = 256

            vectorize = 1

            send_inline = true
            max_inline_data = 60

            use_fallback = true
        }
    }

    ib_node_target = {
        type = "infiniband"

        rdma_port_space = "RC"

        in = {
            address = "10.0.0.1:1337"

            max_wrs = 8192
            cq_size = 8192

            vectorize = 1

            buffer_subtraction = 128

            hooks = (
                { type = "stats", verbose = true }
            )
        }
    }
}

State diagram​