[BUG] Flaky Unit Tests

[askpt]

Description

We are currently experiencing some flaky unit tests that we should tackle. Example output:

Unhandled exception. System.Threading.Channels.ChannelClosedException: The channel has been closed.
   at OpenFeature.EventExecutor.ProcessFeatureProviderEventsAsync(Object providerRef) in /_/src/OpenFeature/EventExecutor.cs:line 238
   at System.Threading.Tasks.Task.<>c.<ThrowAsync>b__128_1(Object state)
   at System.Threading.ThreadPoolWorkQueue.Dispatch()
   at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart()
The active test run was aborted. Reason: Test host process crashed : Unhandled exception. System.Threading.Channels.ChannelClosedException: The channel has been closed.
   at OpenFeature.EventExecutor.ProcessFeatureProviderEventsAsync(Object providerRef) in /_/src/OpenFeature/EventExecutor.cs:line 238
   at System.Threading.Tasks.Task.<>c.<ThrowAsync>b__128_1(Object state)
   at System.Threading.ThreadPoolWorkQueue.Dispatch()
   at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart()


Test Run Aborted.
Total tests: Unknown
     Passed: 90
 Total time: 2.8570 Seconds
     6>_VSTestConsole:
         MSB4181: The "VSTestTask" task returned false but did not log an error.
     6>Done Building Project "/home/runner/work/dotnet-sdk/dotnet-sdk/test/OpenFeature.Tests/OpenFeature.Tests.csproj" (VSTest target(s)) -- FAILED.

As part of the debugging, we can try using the dotnet test—- blame to understand what is wrong.

Copilot

I also ran GH Copilot to try understand the error and this was the response.

Response

The failure appears to be related to a specific build issue within the workflow when setting up the .NET SDK versions 8.0.x and 9.0.x. Based on the log details and the workflow configuration, here are the steps to resolve the issue:

1. Ensure Compatibility: Verify that the project files are compatible with both .NET 8.x and 9.x versions.

2. Update Workflow: Modify the workflow to ensure the correct setup and testing commands are used.

Here is an updated version of the .github/workflows/code-coverage.yml file:

name: Code Coverage

on:
  push:
    branches: [main]
    paths-ignore:
      - "**.md"
  pull_request:
    branches: [main]
    paths-ignore:
      - "**.md"

jobs:
  build-test-report:
    strategy:
      matrix:
        os: [ubuntu-latest, windows-latest]

    runs-on: ${{ matrix.os }}

    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0

      - name: Setup .NET SDK
        uses: actions/setup-dotnet@v4
        env:
          NUGET_AUTH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
        with:
          dotnet-version: |
            8.0.x
            9.0.x
          source-url: https://nuget.pkg.github.com/open-feature/index.json

      - name: Install Dependencies
        run: dotnet restore

      - name: Run Test
        run: dotnet test --configuration Debug --no-build --verbosity normal /p:CollectCoverage=true /p:CoverletOutputFormat=opencover

      - uses: codecov/[email protected]
        with:
          name: Code Coverage for ${{ matrix.os }}
          fail_ci_if_error: true
          verbose: true
          token: ${{ secrets.CODECOV_UPLOAD_TOKEN }}

Make sure to:

1. Restore Dependencies: Add a step to restore dependencies using dotnet restore.
2. Run Tests: Ensure the dotnet test command includes the configuration and no-build flag to avoid unnecessary builds.

After updating the workflow, commit the changes and re-run the workflow to verify if the issue is resolved.

Notes

• https://learn.microsoft.com/en-us/dotnet/core/tools/dotnet-test
• https://github.com/open-feature/dotnet-sdk/actions/runs/13074052418/job/36481801840

[kinyoklion]

This looks to be a bug in the EventExecutor rather than just a flaky test. Writing to a channel will throw if the channel has been closed.

There is a purple note about it here: https://learn.microsoft.com/en-us/dotnet/core/extensions/channels#bounding-strategies

There isn't any synchronization between shutdown and the event processing loop.

            while (await reader.WaitToReadAsync().ConfigureAwait(false))
            {
                if (!reader.TryRead(out var item))
                    continue;

//!!!!!!!!!!!!!!! If you are executing any of the following, and ShutDownAsync is called or a providerr is registered, then you will get an exception.
                switch (item)
                {
                    case ProviderEventPayload eventPayload:
                        this.UpdateProviderStatus(typedProviderRef, eventPayload);
                        await this.EventChannel.Writer.WriteAsync(new Event { Provider = typedProviderRef, EventPayload = eventPayload }).ConfigureAwait(false);
                        break;
                }
            }

Theoretically this could happen in real situations and not jsut a test.

It seems the typical pattern is for the producer to close the channel after it has exhausted things it wants to produce.

So, typically I think that writes and completing the channel would be happening either on the same thread, or would be synchronized.

In our case though I think we could just handle the exception from the WriteAsync.