prov/rxm: add FI_PEER support to rxm #10510

aingerson · 2024-10-31T23:14:45Z

Add rxm support for FI_PEER (srx, cq, and cntr) and FI_AV_USER_ID to be used with the link provider

aingerson · 2024-10-31T23:15:04Z

(will move first 3 commits into separate PR, leaving here for testing)

j-xiong · 2024-11-15T18:07:31Z

prov/rxm/src/rxm_cq.c

-	ret = ofi_cq_write_error_trunc(rx_buf->ep->util_ep.rx_cq,
+	ret = ofi_peer_cq_write_error_trunc(rx_buf->ep->util_ep.rx_cq,
 				       rx_buf->recv_entry->context,
 				       rx_buf->recv_entry->comp_flags |
 				       rx_buf->pkt.hdr.flags,


The alignment is off after the change.

j-xiong · 2024-11-15T18:08:49Z

prov/rxm/src/rxm_cq.c

-		rxm_cq_write(rxm_ep->util_ep.tx_cq, app_context,
-			     comp_flags, 0, NULL, 0, 0);
+		(void) ofi_peer_cq_write(rxm_ep->util_ep.tx_cq, app_context,
+				comp_flags, 0, NULL, 0, 0, FI_ADDR_NOTAVAIL);
 	}
 }


j-xiong · 2024-11-15T18:11:27Z

prov/rxm/src/rxm_cq.c

+
+	if (ofi_peer_cq_write_error(rxm_ep->util_ep.rx_cq, &err_entry))
+		FI_WARN(&rxm_prov, FI_LOG_CQ,
+			"Unable to ofi_peer_cq_Wwrite_error\n");


Typo "Wwrite"

j-xiong · 2024-11-15T18:19:00Z

prov/rxm/src/rxm_ep.c

+				rxm_cq_write_rx_error(def_tx_entry->rxm_ep,
+					ofi_op_msg,
+					def_tx_entry->rndv_ack.rx_buf->
+					recv_entry->context, (int) ret);


Try break the line after ( so that all the arguments are aligned. That's the preferred way if lines become too long with the regular style. Same for a few more places following this.

j-xiong · 2024-11-15T18:19:40Z

prov/rxm/src/rxm_tagged.c

+		ret = ofi_peer_cq_write_error_peek(rxm_ep->util_ep.rx_cq, tag,
 					      context);


Check the alignment.

j-xiong · 2024-11-15T18:50:25Z

prov/rxm/src/rxm_msg.c

+	return util_srx_generic_recv(&rxm_ep->srx->ep_fid, msg->msg_iov,
+			msg->desc, msg->iov_count, msg->addr, msg->context,
+			flags | rxm_ep->util_ep.rx_msg_flags);


j-xiong · 2024-11-15T18:50:39Z

prov/rxm/src/rxm_msg.c

+	return util_srx_generic_recv(&rxm_ep->srx->ep_fid, &iov, &desc, 1,
+			src_addr, context, rxm_ep->util_ep.rx_op_flags);


j-xiong · 2024-11-15T18:50:53Z

prov/rxm/src/rxm_msg.c

+	return util_srx_generic_recv(&rxm_ep->srx->ep_fid, iov, desc, count,
+			src_addr, context, rxm_ep->util_ep.rx_op_flags);


j-xiong · 2024-11-15T18:52:44Z

prov/rxm/src/rxm_tagged.c

+	return util_srx_generic_trecv(&rxm_ep->srx->ep_fid, &iov, &desc, 1,
+				     src_addr, context, tag, ignore,
+				     rxm_ep->util_ep.rx_op_flags);


alignment off by 1 space.

j-xiong · 2024-11-15T19:02:58Z

prov/rxm/src/rxm_ep.c

+
+	if (attr->op_flags & FI_PEER) {
+		rxm_domain->srx = ((struct fi_peer_srx_context *)
+					(context))->srx;


Should we check if srx has already been set?

aingerson · 2024-11-20T23:18:53Z

@j-xiong Almost all of the alignment you pointed out are places where I did it on purpose to maintain the 80 character limit. Do you want me to change all of those to adding a new line after the open bracket? It looks weird to me and I don't think we do it most places in libfabric but I can change them all if you like.

j-xiong · 2024-11-21T01:53:13Z

Yes, I prefer making the changes if possible.

…

________________________________ From: Alexia Ingerson ***@***.***> Sent: Wednesday, November 20, 2024 3:19:29 PM To: ofiwg/libfabric ***@***.***> Cc: Xiong, Jianxin ***@***.***>; Mention ***@***.***> Subject: Re: [ofiwg/libfabric] prov/rxm: add FI_PEER support to rxm (PR #10510) @j-xiong<https://github.com/j-xiong> Almost all of the alignment you pointed out are places where I did it on purpose to maintain the 80 character limit. Do you want me to change all of those to adding a new line after the open bracket? It looks weird to me and I don't think we do it most places in libfabric but I can change them all if you like. — Reply to this email directly, view it on GitHub<#10510 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ACA7MLOVE5NFQA3W6WEXOID2BUKHLAVCNFSM6AAAAABQ7JINH2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIOBZG4ZTEMRVGM>. You are receiving this because you were mentioned.Message ID: ***@***.***>

aingerson · 2024-11-21T17:28:42Z

Update -
Fixed alignment and formatting issues
Rewrote bind path
Check for already existing srx on create and bind paths

j-xiong

Thanks.

j-xiong · 2024-11-22T23:06:13Z

@shijin-aws Is the AWS CI failure related?

j-xiong · 2024-11-23T17:46:43Z

bot:aws:retest

j-xiong · 2024-11-23T18:05:20Z

2 mpichtestsuite test cases failed with Intel CI over verbs_rxm:

not ok 417 - ./pt2pt/icprobe 4 # time=180.3842689991
APPLICATION TIMED OUT, TIMEOUT = 180s

not ok 418 - ./pt2pt/icprobe 4 # time=180.858914136887
APPLICATION TIMED OUT, TIMEOUT = 180s

The rxm SAR segment type enum was defined inside another struct. While techincally ok, this made it difficult for editors to find the type and reported compiler errors. This cleans it up to make it more readible and easier for editors to find the type Signed-off-by: Alexia Ingerson <[email protected]>

Add application side support for FI_AV_USER_ID which requires saving the fi_addr input as the internal fi_addr (for both the peer API srx use case and for reporting unique source address information). When supporting the capability for the application, remove it form the core provider information as it is only required on the top layer Signed-off-by: Alexia Ingerson <[email protected]>

Support using the peer APIs by default using the util peer helper functions. Instead of going through the rxm-specific functions to write to CQs and counters, use the ofi_peer_cq/cntr APIs which use the owner ops. In the default case where rxm is not being used as a peer these will go to the regular ofi_cq_write functions. Signed-off-by: Alexia Ingerson <[email protected]>

Remove rxm implementation of receive queues and leverage the util srx implementation which supports the peer srx API. This allows rxm to use the peer API calls to match receives. To do this, move the rxm protocol information from the receive entry into the rx_buf and allocate it dynamically as needed to track protocol information. This allows rxm to use the default peer_rx_entry instead of its own custom receive entry. With this last piece of the peer API implemented, rxm can also now advertise full support of the FI_PEER capability. Just like the FI_AV_USER_ID capability, rxm removes the bit from the core provider info as it is only a requirement from the application side and not from the message provider Signed-off-by: Alexia Ingerson <[email protected]>

aingerson force-pushed the rxm branch 3 times, most recently from c860a1e to 79f6e8b Compare November 1, 2024 19:20

zachdworkin mentioned this pull request Nov 1, 2024

contrib/intel/jenkins: Separate tcp & tcp;ofi_rxm testing #10508

Closed

aingerson force-pushed the rxm branch from 79f6e8b to 1a14eca Compare November 14, 2024 18:27

j-xiong reviewed Nov 15, 2024

View reviewed changes

aingerson added ⚠️ Do not merge evaluating labels Nov 18, 2024

aingerson force-pushed the rxm branch from 1a14eca to 7644cbf Compare November 21, 2024 17:26

aingerson removed ⚠️ Do not merge evaluating labels Nov 21, 2024

j-xiong approved these changes Nov 21, 2024

View reviewed changes

aingerson force-pushed the rxm branch from 7644cbf to 3812a22 Compare November 25, 2024 18:39

aingerson added 4 commits November 26, 2024 11:44

aingerson force-pushed the rxm branch from 3812a22 to e47be50 Compare November 26, 2024 19:44

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

prov/rxm: add FI_PEER support to rxm #10510

prov/rxm: add FI_PEER support to rxm #10510

aingerson commented Oct 31, 2024

aingerson commented Oct 31, 2024

j-xiong Nov 15, 2024 •

edited

Loading

j-xiong Nov 15, 2024

j-xiong Nov 15, 2024

j-xiong Nov 15, 2024

j-xiong Nov 15, 2024

j-xiong Nov 15, 2024

j-xiong Nov 15, 2024

j-xiong Nov 15, 2024

j-xiong Nov 15, 2024

j-xiong Nov 15, 2024

aingerson commented Nov 20, 2024

j-xiong commented Nov 21, 2024 via email

aingerson commented Nov 21, 2024

j-xiong left a comment

j-xiong commented Nov 22, 2024

j-xiong commented Nov 23, 2024

j-xiong commented Nov 23, 2024

		ret = ofi_peer_cq_write_error_peek(rxm_ep->util_ep.rx_cq, tag,
		context);

		return util_srx_generic_recv(&rxm_ep->srx->ep_fid, &iov, &desc, 1,
		src_addr, context, rxm_ep->util_ep.rx_op_flags);

prov/rxm: add FI_PEER support to rxm #10510

Are you sure you want to change the base?

prov/rxm: add FI_PEER support to rxm #10510

Conversation

aingerson commented Oct 31, 2024

aingerson commented Oct 31, 2024

j-xiong Nov 15, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aingerson commented Nov 20, 2024

j-xiong commented Nov 21, 2024 via email

aingerson commented Nov 21, 2024

j-xiong left a comment

Choose a reason for hiding this comment

j-xiong commented Nov 22, 2024

j-xiong commented Nov 23, 2024

j-xiong commented Nov 23, 2024

j-xiong Nov 15, 2024 •

edited

Loading