SQPoll mode multi fd echo sync mode only get max qps 300k/s

[marcoisajoke]

Hi
I am doing some testing on iouring in my lab
I use the SQPoll mode both in my server and my client
when I test async mode , mean 100 packet on fly in using 10 fd. Iouring run fantasic. It get 100w+ qps.
But when I use sync mode with large fd. I seems the qps can not go fast than 30w/s.
I am struggling in the problem and can not find the problem.

below is my testing result and repro
https://github.com/marcoisajoke/iouring_test/blob/main/pollingMode/perf.md

[isilence]

SQPOLL has enough downsides, like cache bouncing, possibly increased latency, elevated CPU consumption and so on, so the general recommendation is to first try normal / non-SQPOLL rings, and then experiment with SQPOLL and see if it benefits you somehow.

But when I use sync mode with large fd

What you mean by sync / async modes here? And let me clarify, by "large fd" do you mean a socket with a lot of traffic going through it?

[marcoisajoke]

Let me clarify.
sync mode client mean request/response like redis, next request must after last response
async mode client like MQ, next requst no need after last response.
large fd: mean lots of fd instance.

I use SQPOLL for its small latency. If I use normal / non-SQPOLL rings, then it will include system call which is not the best practice.

I am trying to build a echo server and client using SQPOLL
in below mode :
Sync mode (like redis)
Async mode (like MQ)

Sorry for my bad English.

I test Async mode client. It can touch 100w+ qps, which it is excellent.
But when I test Sync mode client. It onlg get 30w qps.

I can not find the bottom neck of sync mode

what I think these two mode's different:

1. fd count is different, async mode only have small number of socket connection while sync mode have large number socket connections.
2. while read from fd, async mode can do the batch read while sync mode only can fetch one packet

I doubt maybe the IOuring can not handle large number of fd which makes the performance can not be better.

[isilence]

Let me clarify. sync mode client mean request/response like redis, next request must after last response async mode client like MQ, next requst no need after last response

Is that a per connection rule? or to put it differently, if we assume that you have multiple connections, how many inflight io_uring requests there will be? I suspect you might have a latency as the limiting factor.

Ok, can you measure CPU util / idle /etc.? Separately for the CPU the app is on, and the CPU SQPOLL task occupies. I'd recommend to pin the app to a single CPU for that (ala taskset). A perf profile might be helpful as well:

perf record -g -- <./prog>
perf report --children > profile.txt

I use SQPOLL for its small latency. If I use normal / non-SQPOLL rings, then it will include system call which is not the best practice.

That's not necessarily "the best practise", and it'll make it more difficult understanding and tuning the basic performance of the system.

...

I test Async mode client. It can touch 100w+ qps, which it is excellent. But when I test Sync mode client. It onlg get 30w qps.

I assume "w" stands for thousands? 100.000 or 100k qps and so on?

I can not find the bottom neck of sync mode

what I think these two mode's different:

1. fd count is different, async mode only have small number of socket connection while sync mode have large number socket connections.

2. while read from fd, async mode can do the batch read while sync mode only can fetch one packet

It's more useful comparing apples to apples. E.g. try the same number of connections first.
Even if you try to keep the same amount of traffic being split in different number of connections, sockets do add more overhead. The more sockets you have, the less batching the kernel can apply on all different layers including io_uring. Btw, how many connections you have in both cases?

[isilence]

There is always overhead for using more objects, especially for simple APIs as sockets kernel need to do lots of additional locking and synchronisation, but I can't say much about your numbers without taking a look at stats and profiles.

And fwiw, you shouldn't have multiple recv requests inflight for the same socket as you can get results out of order of execution. I think there might be something wrong with your async mode if that's the case. Not to mention that each batch of data from net will have to wake up all requests, which is expensive. Take a look at multi shot recv requests, that might be what you want to use here.

[isilence]

Apparently the author removed the reply, and the link now points to nowhere.

[isilence]

You can reopen it, if you'll be willing to continue looking into it.