Most teams get this wrong once: they try to orchestrate long-running workflows from the UI, chaining REST calls and painting progress bars in JavaScript. It works in demos, then breaks in production. Lets take the opposite path: backend-owned orchestration with simple .NET building blocks (Channel, BackgroundService) and SignalR for live progress.

Meridian is a learning platform built on ASP.NET Core. Enrollment is a key workflow that touches multiple systems and can take minutes to complete. Its a perfect candidate for a robust orchestration pattern. Lets implement a reliable enrollment orchestrator that keeps the UI responsive and provides authoritative progress updates by designing a reliable Enrollment Orchestrator in ASP.NET Core using Channels and SignalR

Problem Statement

Enrollment is a long-running workflow with side effects, not a single request:

  1. Parse course source.
  2. Upsert learner.
  3. Upsert course.
  4. Create Jira epic.
  5. Persist enrollment.
  6. Create stories and quiz links.

Product expectation is still simple: submit once, watch progress, get a deterministic outcome. A common early approach is browser orchestration: call A, then B, then C from jQuery/Angular/React and render progress in the UI. This fails operationally:

  1. Tab close/refresh/network loss breaks orchestration ownership.
  2. Workflow correctness depends on client lifecycle.
  3. Retries can duplicate external side effects (Jira in this case).
  4. Progress UI is often estimated, not authoritative.
  5. Auditability and incident debugging become weak.

Principle: UI should observe workflow state, not own workflow execution.

.NET’s “Multiple Personality Disorder” (In a Good Way)

In .NET, you can solve this with several personalities:

  1. Basic data structures: Queue<T>, ConcurrentQueue<T>.
  2. Concurrency primitive: System.Threading.Channels.
  3. Process host: BackgroundService.
  4. External messaging systems: Service Bus/RabbitMQ/Kafka.
  5. Real-time client transport: SignalR.

The confusion is not lack of tools; it is choosing the right depth for current scale. .NET Channels (from System.Threading.Channels) are an in-memory, asynchronous producer–consumer pipeline. Think of them as:

A high-performance, async-aware queue with built-in backpressure and coordination.

They are designed for:

  • async/await workflows
  • High-throughput pipelines
  • Fine-grained control over concurrency

Lets compare these options and see why Channels hit the sweet spot for our enrollment orchestrator.

🔁 Channels vs basic data structures

🧵 Queue<T>

  • Not thread-safe
  • No async support
  • No blocking/waiting built in

👉 You’d need to manually add:

  • Locks
  • Signals (Monitor, Semaphore, etc.)

Verdict: Good for simple, single-thread scenarios. Not suitable for modern concurrent pipelines.

⚙️ ConcurrentQueue<T>

  • Thread-safe
  • Lock-free in many cases
  • Still not async-aware

Problems:

  • No natural way to wait for data asynchronously

  • You end up doing:

    while (!queue.TryDequeue(out var item))
    await Task.Delay(10);

    ❌ inefficient polling

👉 Or you combine it with:

  • SemaphoreSlim
  • Manual signaling

Verdict: Better than Queue<T>, but still low-level and incomplete for async workflows.

🚀 .NET Channels

  • Thread-safe ✅
  • Async-first (WriteAsync, ReadAsync) ✅
  • Built-in waiting (no polling) ✅
  • Supports bounded capacity (backpressure) ✅
  • Supports completion & cancellation ✅

Example:

var channel = Channel.CreateBounded<int>(100);

await channel.Writer.WriteAsync(42);
// someone / somewhere else
var item = await channel.Reader.ReadAsync();

No polling, no locks, no hacks.

Verdict: 👉 Best choice for in-process async pipelines

📦 Channels vs messaging systems (RabbitMQ, etc.)

Now this is a different level of abstraction entirely. RabbitMQ (and similar systems) are:

Out-of-process, distributed messaging systems

They provide:

  • Persistence (messages survive crashes)
  • Delivery guarantees (at-least-once, etc.)
  • Cross-service communication
  • Scalability across machines
  • Routing, topics, fanout, etc.
Feature .NET Channels Messaging Systems
Scope In-process Distributed
Persistence ❌ No ✅ Yes
Performance 🚀 Extremely fast 🐢 Slower (network, disk)
Complexity Low High
Reliability Process-bound Durable
Use case Pipelines, background tasks Microservices, integration
  • Channels = “fast pipes inside your app”
  • RabbitMQ = “postal service between apps”

✅ Use Channels when:

  • You’re inside a single app/process
  • You need high throughput
  • You want clean async producer–consumer logic
  • You need backpressure

👉 Example:

  • Background job processing
  • Streaming data pipelines
  • Worker pools

✅ Use ConcurrentQueue when:

  • You need something simple and fast
  • You don’t care about async waiting
  • You’re okay managing coordination manually

✅ Use RabbitMQ (or similar) when:

  • You have multiple services
  • You need durability
  • You need retry and delivery guarantees
  • You need decoupling across systems

👉 Example:

  • Microservices communication
  • Event-driven architectures

.NET Channels and Other Stacks

🟢 Go channels

Go’s channels are the closest conceptual match to .NET Channels.

“Communicate by passing messages, not sharing memory”

Feature .NET Channels Go Channels
Producer–consumer ✅ ✅
Blocking/async reads ✅ (await) ✅ (blocking)
Bounded/unbounded ✅ ✅
Backpressure ✅ ✅
Multiple producers/consumers ✅ ✅

  • Go channels are built into the language and .NET Channels are a library. Go code feels more natural and .NET code is more explicit.

    ch <- 42
x := <-ch

    await channel.Writer.WriteAsync(42);
var x = await channel.Reader.ReadAsync();

⚠️ Key Differences

  • Go channels are idiomatic concurrency but .NET Channels are more of a specialized tool for certain scenarios. You can write concurrent code in .NET without Channels
  • Concurrency model is different, Go is built around goroutines + channels and Channels are the central coordination primitive, while .NET is built around Tasks + async/await and Channels are one option
  • Select/multiplexing is native in Go but not in .NET (you have to simulate it with Task.WhenAny or custom coordination)
select {
  case x := <-ch1:
  case ch2 <- y:
}

☕ Java

Java doesn’t have one equivalent—it has several layers.

  • BlockingQueue (classic Java) from java.util.concurrent that offers thread-blocking model (and its not async-aware)
  • Reactive Streams (modern Java) from org.reactivestreams that offers push-based, asynchronous model

Reactive Streams are used by frameworks like:

  • Project Reactor
  • RxJava
BlockingQueue<Integer> queue = new ArrayBlockingQueue<>(100);
queue.put(42);
int x = queue.take();

Flux.just(1,2,3)
    .map(x -> x * 2)
    .subscribe(System.out::println);

Comparison

Feature .NET Channels BlockingQueue Reactive Streams
Pull model ✅ pull via await ✅ manual take() ❌ push-only
Async/await ✅ native ❌ blocking only ⚠️ via adapters
Backpressure ✅ bounded buffer ✅ blocks when full ✅ built-in
Completion model ✅ Complete() ❌ none ✅ onComplete()
Non-blocking async ✅ fully async ❌ blocking ✅ non-blocking
Simplicity ✅ simple API ✅ trivial ❌ steeper learning curve
Pipelines ⚠️ manual plumbing ❌ none ✅ built-in
Thread-safety ✅ ✅ ✅
Buffering ✅ bounded/unbounded ✅ bounded/unbounded ✅ bounded/unbounded
Error handling ⚠️ manual ❌ ✅ built-in
Composability ⚠️ manual ❌ ✅ high

Note: Channels are “manual plumbing with full control,” Reactive Streams are “dataflow pipelines with operators.”

🌿 Spring Ecosystem

Spring usually operates at a higher abstraction level. It provides:

  • Message channels
  • Routers
  • Transformers
  • Adapters (HTTP, Kafka, etc.)

👉 Conceptually similar naming, but:

Feature .NET Channels Spring Integration
Scope In-process Often distributed
Abstraction Low-level High-level
Control Manual Declarative/config-driven

Spring WebFlux is more comparable to Reactive Streams than to Channels, as it is designed for reactive, non-blocking web applications. Its built on top of Project Reactor and provides a push-based, asynchronous programming model.

Feature .NET Channels Spring (WebFlux + Integration)
Model ✅ Pull via await ❌ Push / declarative (Flux / Integration flows)
Async/await ✅ native ⚠️ reactive operators (Mono/Flux) / adapters
Backpressure ✅ bounded buffer ✅ built-in (reactive) / broker-managed
Completion model ✅ Complete() ✅ onComplete() (WebFlux) / message end events (Integration)
Non-blocking async ✅ fully async ✅ non-blocking (WebFlux)
Simplicity ✅ simple API ❌ steeper learning curve / config-heavy
Pipelines ⚠️ manual plumbing ✅ built-in (operators + integration flows)
Thread-safety ✅ ✅
Buffering ✅ bounded/unbounded ✅ bounded/unbounded (configurable)
Error handling ⚠️ manual ✅ built-in
Composability ⚠️ manual ✅ high (operators + integration patterns)
Scope In-process only Often distributed / multi-transport capable
Abstraction level Low-level High-level / declarative
Ecosystem Minimal 🌐 Large (WebFlux, Integration, Kafka, HTTP, DB connectors)
  • .NET Channels → Low-level, fully async, in-process pipelines. Full manual control, lightweight, simple API.
  • Spring (WebFlux + Integration) → High-level, declarative, reactive, or distributed pipelines. Large ecosystem, built-in error handling, composable operators, multi-transport ready.

Big picture comparison

Ecosystem Equivalent Level
.NET Channels Mid-level
Go Channels Language-level
Java BlockingQueue Low-level
Java Reactor/RxJava High-level
Spring Integration/WebFlux Very high-level
  • If you like Go-style concurrency, .NET Channels feel familiar—but slightly more verbose
  • If you come from Java, Channels feel like a huge upgrade over BlockingQueue
  • If you use Spring reactive, Channels will feel too manual

.NET Channels are one of the few abstractions that hit a sweet spot between control and ergonomics—which is why they’re so powerful but also somewhat underused.

Chosen Architecture

A more robust approach requires decoupling the workflow from the UI: an asynchronous HTTP server to receive requests, a suitable data structure to track workflow state, and a control flow mechanism that triggers processing when new messages arrive. This processing runs in the background service, separate from the request thread, while a push/poll system communicates status updates back to the client. In the .NET ecosystem, this pattern maps naturally to Channel<T> for safe in-memory queues, ASP.NET’s BackgroundService feature for long-running processing, and SignalR for real-time client notifications.

Our implementation intentionally picks a classic middle path:

  1. Multiple producers: HTTP requests enqueue operation IDs.
  2. Single consumer: one hosted background worker processes operations in sequence.
  3. In-memory transport: unbounded Channel<Guid> with SingleReader=true, SingleWriter=false.
  4. Persistent state model: EnrollmentOperation + EnrollmentOperationEvent in DB.
  5. UI delivery: SignalR group push with /enroll/status/{id} polling fallback.

This keeps infrastructure light while still preserving server-side source of truth.

Architecture Diagram

[Browser]
   | POST /enroll
   v
[EnrollmentController] -- create operation --> [EnrollmentOperations table]
   | enqueue operationId
   v
[Channel<Guid>] (multi-producer, single-consumer)
   |
   v
[EnrollmentProcessingService]
   | run workflow via IEnrollmentService
   | append status/events
   v
[EnrollmentOperations + Events]
   |
   +--> [SignalR Hub Group: operationId] --> [Live UI updates]
   |
   +--> [GET /enroll/status/{id}]        --> [Polling fallback]

Implementation Details (Hands-On)

1) Controller schedules work, does not execute it

[HttpPost("/enroll")]
public async Task<IActionResult> Index(EnrollmentViewModel model, CancellationToken cancellationToken)
{
    if (!ModelState.IsValid)
        return View(model);

    var source = new CourseSourceLocator(model.SourceType, model.SourceUri, model.SubPath);
    var operation = await enrollmentOperationService.CreateQueuedAsync(model.LearnerEmail, source);
    await enrollmentQueue.EnqueueAsync(operation.Id, cancellationToken);

    return RedirectToAction(nameof(Progress), new { operationId = operation.Id });
}

Impact: request latency remains short and predictable.

2) Queue implementation is tiny and explicit

public class EnrollmentQueue : IEnrollmentQueue
{
    readonly Channel<Guid> channel = Channel.CreateUnbounded<Guid>(new UnboundedChannelOptions
    {
        SingleReader = true,
        SingleWriter = false
    });

    public ValueTask EnqueueAsync(Guid operationId, CancellationToken cancellationToken = default) =>
        channel.Writer.WriteAsync(operationId, cancellationToken);

    public IAsyncEnumerable<Guid> DequeueAllAsync(CancellationToken cancellationToken) =>
        channel.Reader.ReadAllAsync(cancellationToken);
}

3) Background worker owns orchestration lifecycle

protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
    await foreach (var operationId in queue.DequeueAllAsync(stoppingToken))
    {
        try
        {
            var operation = await getOperationSnapshotAsync(operationId);
            var source = new CourseSourceLocator(
                Enum.Parse<CourseSourceType>(operation.SourceType, ignoreCase: true),
                operation.SourceUri,
                operation.SubPath);

            await updateAndPublishAsync(operationId, s =>
                s.MarkInProgressAsync(operationId, new EnrollmentProgressUpdate("in_progress", "Started", 6)), stoppingToken);

            var enrollment = await enrollmentService.EnrollAsync(operation.LearnerEmail, source, async progress =>
            {
                await updateAndPublishAsync(operationId, s =>
                    s.AppendProgressAsync(operationId, progress), stoppingToken);
            });

            await updateAndPublishAsync(operationId, s =>
                s.MarkCompletedAsync(operationId, enrollment.Id, enrollment.JiraEpicKey), stoppingToken);
        }
        catch (Exception ex)
        {
            await updateAndPublishAsync(operationId, s =>
                s.MarkFailedAsync(operationId, ex.Message), stoppingToken);
        }
    }
}

4) SignalR push plus polling fallback on the client

const connection = new signalR.HubConnectionBuilder()
  .withUrl("/hubs/enrollment")
  .withAutomaticReconnect()
  .build();

connection.on("EnrollmentProgress", (snapshot) => updateProgress(snapshot));
connection.onreconnecting(() => startPolling());
connection.onreconnected(() => {
    connection.invoke("Subscribe", operationId);
    stopPolling();
});

The key detail is fallback: if realtime fails, status polling keeps the UX correct.

Engineering Decisions: Strengths and Tradeoffs

Strengths

  1. Clear separation of concerns: HTTP intake vs workflow execution vs UI notification.
  2. Better reliability semantics than client-driven orchestration.
  3. Strong observability path via persisted state/events.
  4. Incremental architecture: good fit for current scale without premature broker complexity.
  5. Easier incident handling: operation IDs provide a stable trace handle.

Tradeoffs

  1. In-memory channel is not durable through process crash/restart.
  2. Recovery policy for stale queued/in-progress operations is not fully implemented yet.
  3. Single consumer limits throughput by design.
  4. SignalR group subscription model should be hardened with stronger auth checks for multi-tenant scenarios.

These are acceptable if team explicitly acknowledges current reliability envelope.

When Not to Use This Pattern

Do not use this exact setup if:

  1. You need durable queue semantics across process/node failure today.
  2. You need high parallel throughput across many workers now.
  3. You need cross-service fan-out with independent consumers.
  4. You require strict delivery guarantees beyond what in-memory channels provide.

In those cases, keep the orchestration model but replace in-memory transport with a proper broker.

Bottom Line

For Issue #29 we prioritized pragmatic engineering:

  1. Backend owns orchestration.
  2. UI consumes authoritative progress.
  3. Modern .NET primitives provide clean implementation with low operational drag.
  4. Architecture leaves room to evolve without rewriting the workflow model.

If this workflow becomes business-critical, we will need to prioritize:

  1. Startup recovery job to re-enqueue stale operations.
  2. Idempotency strategy for Jira side effects.
  3. Bounded channel + admission/backpressure policy.
  4. Metrics and SLOs (queue depth, processing latency, failure rates).
  5. Migration seam to external broker when durability/scale requires it.