Man page - raw_ethernet_lat(1)

Packages contains this manual

Manual

perftest

NAME
DESCRIPTION
RUNNING TESTS
IMPORTANT NOTES
OPTIONS
RawEth only options
AUTHORS

NAME

ib_write_bw, ib_read_bw, ib_send_bw, ib_atomic_bw, ib_write_lat, ib_read_lat, ib_send_lat, ib_atomic_lat, raw_ethernet_bw, raw_ethernet_lat, raw_ethernet_burst_lat, raw_ethernet_fs_rate - benchmarks for various types of infinabnd performance

DESCRIPTION

Perftest is a package that includes various benchmarks that measures

different metrics & verbs performance which include many different options and modes.

RUNNING TESTS

Server:

./<test name> <options>

Client:

./<test name> <options> <server IP address>

Examples:

1- Running bidirectional bandwidth test using Write verb for 5 seconds with 8388608 as a message size and 3 qps:
Server: ./ib_write_bw -s 8388608 -b -D 5 -q 3
Client: ./ib_write_bw -s 8388608 -b -D 5 -q 3 1.1.1.2

2- Running latency test using Read verb for 5000 iterations with 32 as a message size:
Server: ./ib_read_lat -s 32 -n 5000
Client: ./ib_read_lat -s 32 -n 5000 192.168.0.1

IMPORTANT NOTES

1- The options that specific to modes in perftest must be the
same for both server and client.

2- Perftest applications may need to be ran with sudo when running from non root.
3- Perftest applications usually installed to the /usr/bin/.
4- Perftest may print some failures with syndroms to the stderr, perftest get those errors from rdma-core.

OPTIONS

-h, --help

Lists the available options to the screen.

-a, --all

Run sizes from 2 till 2ˆ23.
Not relevant for Atomic and RawEth.

-A, --atomic_type=<type>

Type of atomic operation from {CMP_AND_SWAP,FETCH_AND_ADD} (default FETCH_AND_ADD).
Relevant only for Atomic.

-b, --bidirectional

Measure bidirectional bandwidth (default unidirectional).
Relevant only for BW.

-c, --connection=<RC/XRC/UC/UD/DC/SRD>

Connection type RC/XRC/UC/UD/DC/SRD (default RC).
UD relevant only for Send verb.
SRD relevant only for Read, Write and Send verbs.
UC relevant only for Write and Send verbs.
Not relevant for RawEth.

--log_dci_streams=<log_num_dci_stream_channels> (default 0)

Run DC initiator as DCS instead of DCI with <log_num dci_stream_channels>.
Not relevant for RawEth.
System support required.

--log_active_dci_streams=<log_num_active_dci_stream_channels> (default
log_num_dci_stream_channels)

Not relevant for RawEth.
System support required.

--aes_xts

Runs traffic with AES_XTS feature (encryption).
Not relevant for RawEth and Write latency.
System support required.

--encrypt_on_tx

Runs traffic with encryption on tx (default decryption on tx).
Not relevant for RawEth and Write latency.
System support required.

--sig_before

Puts signature on data before encrypting it (default after).
Not relevant for RawEth and Write latency.
System support required.

--aes_block_size=<512,520,4048,4096,4160> (default 512)

Not relevant for RawEth and Write latency.
System support required.

--data_enc_keys_number=<number of data encryption keys> (default 1)

Not relevant for RawEth and Write latency.
System support required.

--kek_path <path to the key encryption key file>

Not relevant for RawEth and Write latency.
System support required.

--credentials_path <path to the credentials file>

Not relevant for RawEth and Write latency.
System support required.

--data_enc_key_app_path <path to the data encryption key app>

Not relevant for RawEth and Write latency.
System support required.

-C, --report-cycles

Report times in cpu cycle units (default microseconds).
Relevant only for latency.

-d, --ib-dev=<dev>

Use IB device <dev> (default first device found).

-D, --duration

Run test for a customized period of seconds.

-e, --events

Sleep on CQ events (default poll).
Not relevant for Write and RawEth.

-X, --vector=<completion vector>

Set <completion vector> used for events.
Not relevant for Write and RawEth.

-f, --margin

measure results within margins. (default=2sec).

-F, --CPU-freq

Do not show a warning even if cpufreq_ondemand module is loaded, and cpu-freq is not on max.

-g, --mcg

Send messages to multicast group with 1 QP attached to it.
When there is no multicast gid specified, a default IPv6 typed gid ’255:1:0:0:0:2:201:133:0:0:0:0:0:0:0:0’ will be used.
Relevant only for send non fsRate.

-H, --report-histogram

Print out all results (default print summary only).
Relevant only for latency and raw_ethernet_fs_rate.

-i, --ib-port=<port>

Use port <port> of IB device (default 1).

-I, --inline_size=<size>

Max size of message to be sent in inline.
Not relevant for Read and Atomic.

-l, --post_list=<list size>

Post list of send WQEs of <list size> size (instead of single post).
Relevant only for BW and raw_ethernet_burst_lat.

--recv_post_list=<list size>

Post list of receive WQEs of <list size> size (instead of single post).
Relevant only for BW and raw_ethernet_burst_lat.

-L, --hop_limit=<hop_limit>

Set hop limit value (ttl for IPv4 RawEth QP). Values 0-255 (default 64).
Relevant only for RawEth
Not relevant for raw_ethernet_fs_rate.

-m, --mtu=<mtu>

MTU size : 64 - 9600 (default port mtu) for RawEth else 256 - 4096.
Not relevant for raw_ethernet_fs_rate.

-M, --MGID=<multicast_gid>

In multicast, uses <multicast_gid> as the group MGID.
<multicast_gid> can be either decimal or hexadecimal, e.g. regarding the IPv4 224.0.0.30 :
Decimal: 0:0:0:0:0:0:0:0:0:0:255:255:224:0:0:30 , Hexadecimal: 0:0:0:0:0:0:0:0:0:0:0xff:0xff:0xe0:0:0:0x1e
Relevant only for send non fsRate.

-n, --iters=<iters>

Number of exchanges (at least 5, default for write 5000 else 1000 ).

-N, --noPeak

Cancel peak-bw calculation (default with peak up to iters=20000).
Relevant only for bandwidth.

-o, --outs=<num>

Relevant only for Read and Atomic.

-O, --dualport

Run test in dual-port mode.
Not relevant for RawEth.
Relevant only for bandwidth.
System support required.

-p, --port=<port>

Listen on/connect to port <port> (default 18515).

-q, --qp=<num of qp’s>

Num of qp’s(default 1).
Relevant only for bandwidth.

-Q, --cq-mod

Generate Cqe only after <--cq-mod> completion.
Relevant only for bandwidth.

-r, --rx-depth=<dep>

Rx queue size (default 512), if using srq, rx-depth controls max-wr size of the srq.
Relevant only for send non fsRate.

-R, --rdma_cm

Connect QPs with rdma_cm and run test on those QPs.
Not relevant for RawEth.

-s, --size=<size>

Size of message to exchange (default 65536 for bw, for lat 2).
Not relevant for Atomic.

-S, --sl=<sl>

SL (default 0).
Not relevant for raw_ethernet_fs_rate.

-t, --tx-depth=<dep>

Size of tx queue (default 128 for bw else 1).
Relevant only for bw and raw_ethernet_burst_lat.

-T, --tos=<tos value>

Set <tos_value> to RDMA-CM QPs. available only with -R flag. values 0-256 (default off).
Not relevant for RawEth

-u, --qp-timeout=<timeout>

QP timeout, timeout value is 4 usec * 2 ˆ(timeout), default 14.

-U, --report-unsorted

(implies -H) print out unsorted results (default sorted).
Relevant only for latency and raw_ethernet_burst_lat and raw_ethernet_fs_rate.

-V, --version

Display perftest version number.

-W, --report-counters=<list of counter names>

Report performance counter change (example: counters/port_xmit_data,hw_counters/out_of_buffer).

-x, --gid-index=<index>

Test uses GID with GID index.
Not relevant for RawEth.

-z, --comm_rdma_cm

Communicate with rdma_cm module to exchange data - use regular QPs.
Not relevant for RawEth.

--out_json

Save the report in a json file.

--out_json_file=<file>

Name of the report json file. (Default: "perftest_out.json" in the working directory).

--cpu_util

Show CPU Utilization in report, valid only in Duration mode.

--dlid

Set a Destination LID instead of getting it from the other side.
Not relevant for raw_ethernet_fs_rate.

--dont_xchg_versions

Do not exchange versions and MTU with other side.
Not relevant for RawEth.

--force-link=<value>

Force the link(s) to a specific type: IB or Ethernet.
Not relevant for raw_ethernet_fs_rate.

--use-srq

Use a Shared Receive Queue. --rx-depth controls max-wr size of the SRQ.
Relevant only for Send.

--ipv6

Use IPv6 GID. Default is IPv4.
Not relevant for RawEth.

--ipv6-addr=<IPv6>

Use IPv6 address for parameters negotiation. Default is IPv4.
Not relevant for RawEth.

--bind_source_ip

Source IP of the interface used for connection establishment. By default taken from routing table.
Not relevant for RawEth.

--latency_gap=<delay_time>

delay time between each post send.
Relevant only for latency.

--mmap=file

Use an mmap’d file as the buffer for testing P2P transfers.
Not relevant for RawEth.

--mmap-offset=<offset>

The mmap offset.
Not relevant for RawEth.

--mr_per_qp

Create memory region for each qp.
Relevant only for bandwidth.

--odp

Use On Demand Paging instead of Memory Registration.
System support required.

--output=<units>

Set verbosity output level: bandwidth , message_rate, latency.
Latency measurement is Average calculation.
bw (bandwidth / message_rate), latency (latency).

--payload_file_path=<payload_txt_file_path>

Set the payload by passing a txt file containing a pattern in the next form(little endian): ’0xaaaaaaaa, 0xbbbbbbbb, ...
Not relevant for RawEth and Write latency.

--use_old_post_send

Use old post send flow (ibv_post_send).

--perform_warm_up

Perform some iterations before start measuring in order to warming-up memory cache.
Not relevant for raw_ethernet_fs_rate.

--pkey_index=<pkey index>

PKey index to use for QP.
Not relevant for raw_ethernet_fs_rate.

--report-both

Report RX & TX results separately on Bidirectional BW tests.
Relevant only for bidirectional bandwidth.

--report_gbits

Report Max/Average BW of test in Gbit/sec (instead of MiB/sec).
Relevant only for bandwidth.

--report-per-port

Report BW data on both ports when running Dualport and Duration mode.
Not relevant for RawEth.
System support required.

--reversed

Reverse traffic direction - Server send to client.

--run_infinitely

Run test forever, print results every <duration> seconds.

--retry_count=<value>

Set retry count value in rdma_cm mode.
Relevant only for rdma_cm mode.
Not relevant for RawEth.

--tclass=<value>

Set the Traffic Class in GRH (if GRH is in use).
Not relevant for raw_ethernet_fs_rate.

--use-null-mr

Allocate a null memory region for the client with ibv_alloc_null_mr (3)

--use_cuda=<cuda device id>

Use CUDA specific device for GPUDirect RDMA testing.
Not relevant for raw_ethernet_fs_rate.
System support required.

--use_cuda_bus_id=<cuda full BUS id>

Use CUDA specific device, based on its full PCIe address, for GPUDirect RDMA testing.
Not relevant for raw_ethernet_fs_rate.
System support required.

--use_cuda_dmabuf

Use CUDA DMA-BUF for GPUDirect RDMA testing.
Not relevant for raw_ethernet_fs_rate.
System support required.

--use_hl=<hl device id>

Use HabanaLabs specific device for HW accelerator direct RDMA testing.
System support required.

--use_neuron=<logical neuron core id>

Use Neuron specific device for HW accelerator direct RDMA testing.
System support required.

--use_neuron_dmabuf

Use Neuron DMA-BUF for HW accelerator direct RDMA testing.
System support required.

--use_rocm=<rocm device id>

Use selected ROCm device for GPUDirect RDMA testing.
Not relevant for raw_ethernet_fs_rate.
System support required.

--use_hugepages

Use Hugepages instead of contig, memalign allocations.
Not relevant for raw_ethernet_fs_rate.

--wait_destroy=<seconds>

Wait <seconds> before destroying allocated resources (QP/CQ/PD/MR..).
Relevant only for bandwidth and raw_ethernet_burst_lat.

--disable_pcie_relaxed

Disable PCIe relaxed ordering.
Relevant only for bandwidth and raw_ethernet_burst_lat.
System support required.

--burst_size=<size>

Set the amount of messages to send in a burst when using rate limiter.
Relevant only for bandwidth and raw_ethernet_burst_lat.

--typical_pkt_size=<bytes>

Set the size of packet to send in a burst. Only supports PP rate limiter.
Relevant only for bandwidth and raw_ethernet_burst_lat.

--rate_limit=<rate>

Set the maximum rate of sent packages. default unit is [Gbps]. use --rate_units to change that.
Relevant only for bandwidth and raw_ethernet_burst_lat.

--rate_units=<units>

[Mgp] Set the units for rate limit to MiBps (M), Gbps (g) or pps (p). default is Gbps (g).
Relevant only for bandwidth and raw_ethernet_burst_lat.

--rate_limit_type=<type>

[HW/SW/PP] Limit the QP’s by HW, PP or by SW. Disabled by default. When rate_limit is not specified HW limit is Default.
Relevant only for bandwidth and raw_ethernet_burst_lat.

--use_ooo

Use out of order data placement.
System support required.

--write_with_imm

Use write-with-immediate verb instead of write.
Write tests only.

RawEth only options

-B, --source_mac

Source MAC address by this format XX:XX:XX:XX:XX:XX **MUST** be entered.

-E, --dest_mac

Destination MAC address by this format XX:XX:XX:XX:XX:XX **MUST** be entered.

-G, --use_rss

Use RSS on server side. need to open 2ˆx qps (using -q flag. default is -q 2). open 2ˆx clients that transmit to this server.

-J, --dest_ip

Destination ip address by this format X.X.X.X for IPv4 or X:X:X:X:X:X for IPv6 (using to send packets with IP header).
System support required for IPv6.

-j, --source_ip

Source ip address by this format X.X.X.X for IPv4 or X:X:X:X:X:X for IPv6 (using to send packets with IP header).
System support required for IPv6.

-K, --dest_port

Destination port number (using to send packets with UDP header as default, or you can use --tcp flag to send TCP Header).

-k, --source_port

Source port number (using to send packets with UDP header as default, or you can use --tcp flag to send TCP Header).

-Y, --ethertype

Ethertype value in the ethernet frame by this format 0xXXXX.

-Z, --server

Choose server side for the current machine (--server/--client must be selected ).

--vlan_en

Insert vlan tag in ethernet header.

--vlan_pcp

Specify vlan_pcp value for vlan tag, 0˜7. 8 means different vlan_pcp for each packet.

-P, --client

Choose client side for the current machine (--server/--client must be selected).
Not relevant for raw_ethernet_fs_rate.

-v, --mac_fwd

Run mac forwarding test.
Not relevant for raw_ethernet_fs_rate.

--flows

Set number of TCP/UDP flows, starting from <src_port, dst_port>.
Not relevant for raw_ethernet_fs_rate.

--flows_burst

Set number of burst size per TCP/UDP flow.
Not relevant for raw_ethernet_fs_rate.

--promiscuous

Run promiscuous mode.
Not relevant for raw_ethernet_fs_rate.

--reply_every

In latency test, receiver pong after number of received pings.
Not relevant for raw_ethernet_fs_rate.

--sniffer

Run sniffer mode.
Not relevant for raw_ethernet_fs_rate.
System support required.

--flow_label=<fl0, fl1,...>

IPv6 flow label.
Not relevant for raw_ethernet_fs_rate.

--tcp

Send TCP Packets. must include IP and Ports information.

--raw_ipv6

Send IPv6 Packets.
System support required.

--raw_mcast.

Relevant only for bandwidth.

AUTHORS

Hassan Khadour <hkhadour@nvidia.com>
Talat Batheesh <talatb@nvidia.com>