[QUERY] In what circumstances ReceiveMessagesAsync can return empty collection

[KlaudiuszBryjaRelativity]

Library name and version

Azure.Messaging.ServiceBus 7.17.5

Query/Question

Our application receives messages using the SDK, but we need to use a batching solution due to a dependency on a database that prefers batch inserts. We collect messages for up to 10 minutes or until a batch size is exceeded.

The ServiceBus is in the Standard Tier, and the region where the ServiceBus and application are located is the same (though they are in different resource groups).

To receive messages, we connect to a topic with a defined subscription. The traffic is quite low (around a few dozen messages per minute). The lock duration for the subscription is set to the default value of 1 minute.

What is our scenario?
An ASP.NET Core application creates a ServiceBus client using ASP.NET IoC. A client and receiver are created as singletons.

builder
    .AddServiceBusClientWithNamespace(productServiceBusOptions.FullyQualifiedNamespace)
    .WithName(productServiceBusOptions.ConnectionName);

and the receiver:

services.AddKeyedSingleton(ServiceBusConstants.ReceiverBatchingStrategyKey, (provider, _) =>
{
    var factory = provider.GetRequiredService<IAzureClientFactory<ServiceBusClient>>();
    var client = factory.CreateClient(productServiceBusOptions.ConnectionName);
    ServiceBusReceiverOptions options = new()
    {
        ReceiveMode = ServiceBusReceiveMode.PeekLock,
        PrefetchCount = 0 // explanation below
    };

    return client.CreateReceiver(productServiceBusOptions.Topic, productServiceBusOptions.Subscription, options);
});

Basic assumptions: PrefetchCount is set to 0 to avoid locking messages. This prevents messages from being moved to the DLQ. Messages are collected for 10 minutes, so we need to avoid locking messages if traffic is low.

const int _maxMessagesCount = 5000;
readonly TimeSpan _maxWaitTime = TimeSpan.FromSeconds(30);
...
var messages = await _receiver.ReceiveMessagesAsync(_maxMessagesCount, _maxWaitTime, token);

Over the course of 3 hours, 3-5 messages (depending on the region) were read, even though the queue had a few dozen messages. Surprisingly, most of the time ReceiveMessagesAsync() returned an empty collection. We noticed that the dead-letter queue received those messages, but ReceiveMessagesAsync() did not throw an exception (no exception was caught). I found that the ServiceBus metrics showed user errors during the test period (I can find anything in logs).
The empty collection is returned even when ReceiveMessagesAsync() is executed multiple times, repeatedly, one after another, without a break.

• What could be the reason that this method returns 0 messages (in my tests on a different ServiceBus, it returns 30-80 messages each time)?
• Is it possible to add logging to the ServiceBus receiver to get more detailed logs on why messages were moved to the DLQ?
• Additional question for the future: When prefetch is set to a value greater than zero, when exactly does the background process start prefetching messages? Is it right after the instance is created, after the first call to ReceiveMessagesAsync(), or is there a different solution for this? According to the documentation, this locks messages, so how can we prevent lock expiration when ReceiveMessagesAsync() is not executed immediately?

Environment

It will be hard to get because this is asp.net core app hosted on an AppService and I have no access to check details.
.Net 8.0
IDE - Rider or VS 17.13.1

[jsquire]

Hi @KlaudiuszBryjaRelativity. Thanks for reaching out and we regret that you're experiencing difficulties.

What could be the reason that this method returns 0 messages (in my tests on a different ServiceBus, it returns 30-80 messages each time)?

The short answer to your question is - any time a service operation returns no messages within the specified maxWaitTime without triggering an error.

The "why" could be due to a number of different factors including intermittent network slowness, limited resources on the application host causing intermittent contention that slows performance, or service-side issues. The client knows only that it made a service request and that the service did not stream down any messages before the wait time was exhausted.

To understand more, you would need to engage with the Service Bus service team and ask them to investigate why no messages were returned. Unfortunately, the Service Bus service team does not monitor our repository nor offer support based on issues here. If you choose to engage them, your best path forward for would be to open an Azure support request or inquire on the Microsoft Q&A site, which they do monitor.

Is it possible to add logging to the ServiceBus receiver to get more detailed logs on why messages were moved to the DLQ?

The client does not move messages to the DLQ unless you explicitly invoke DeadLetterMessageAsync for that message. The service will move messages to the DLQ automatically when the delivery count exceeds the configured threshold, the time-to-live on the message is exceeded or other reasons detailed in Overview of Service Bus dead-letter queues.

Here, again, the client has no insight nor influence over the service behavior. The dead-letter reason associated with the message should offer insight into why the service moved it.

When prefetch is set to a value greater than zero, when exactly does the background process start prefetching messages? Is it right after the instance is created, after the first call to ReceiveMessagesAsync(), or is there a different solution for this?

All Service Bus client types are lazy. The first time that you invoke a service operation, the connection is established, auth performed, and the associated AMQP link created. Prefetch is managed entirely by the AMQP transport and begins at link creation. Calling ReceiveMessagesAsync would trigger this process, as would calling any other service operation on the receiver.

According to the documentation, this locks messages, so how can we prevent lock expiration when ReceiveMessagesAsync() is not executed immediately?

If you create a receiver and don't invoke a service method immediately, nothing is prefetched. If the client or service idle timeout is triggered, the connection/link are closed and messages are not prefetched until you next invoke a service operation. If there's a transient failure that breaks the connection/link, messages are not prefetched until you next invoke a service operation.

As mentioned, prefetch is entirely managed by the AMQP transport which does not expose its state or contents to the client, nor can the client interact with prefetch. In short, this means that the client cannot tell you when something is in prefetch, can't see or renew its locks, and cannot tell prefetch to stop. The client can only influence prefetch behavior during link creation by enabling or disabling it and setting the number of desired messages.

[github-actions]

Hi @KlaudiuszBryjaRelativity. Thank you for opening this issue and giving us the opportunity to assist. We believe that this has been addressed. If you feel that further discussion is needed, please add a comment with the text "/unresolve" to remove the "issue-addressed" label and continue the conversation.