controlled concurrency with semaphores

Performance is not always about doing things faster. Often it is about doing them sensibly. One of the worst mistakes in distributed systems is uncontrolled scale: assuming that because your application can cheaply spawn 100,000 concurrent threads, everything downstream (databases, payment gateways, microservices, third-party APIs) will happily absorb that load.

With Java Virtual Threads (Project Loom, JEP 444), creating a thread is no longer an expensive OS operation. A platform thread reserves about 1MB of native stack by default (on 64-bit Linux) and needs kernel context switches. A Virtual Thread takes a few hundred bytes of heap and switches in userspace.

That improvement also created a dangerous illusion:

"Virtual threads are virtually free. Let’s fire 50,000 parallel HTTP calls to our partner API!"

What happens in production? Failures cascade downstream:

• HTTP 429 Too Many Requests (rate limits hit).
• TCP SYN queue overflow and socket connection timeouts.
• Thread starvation and resource exhaustion in upstream clients.
• Downstream services falling over under sudden bursts of traffic.

This article walks through controlled concurrency and backpressure in Java with java.util.concurrent.Semaphore: how it works internally on AbstractQueuedSynchronizer (AQS), how Virtual Threads unmount while waiting on it, and how to size the limit with Little’s Law.

# concurrency vs throughput vs downstream capacity

In queuing theory, every downstream dependency has a finite Service Capacity (C):

# the fallacy of unbounded concurrency

Once concurrency goes past the downstream system's knee point:

1. Queuing delay explodes: requests spend more time waiting in OS TCP receive buffers and socket backlogs than actually being processed.
2. Context switching and GC overhead: downstream servers burn CPU juggling thousands of half-open TCP connections instead of running business queries.
3. Retry storms: retries from timed-out clients pile more load on top, until useful downstream throughput collapses toward zero.

Backpressure is the feedback loop that tells the upstream producer: "Slow down, only push as much concurrency as downstream can take."

# under the hood: abstractQueuedSynchronizer

java.util.concurrent.Semaphore is not a thin wrapper around synchronized or wait()/notify(). It is built directly on Doug Lea's AbstractQueuedSynchronizer (AQS).

# how aqs state and acquires work

1. Volatile state variable: the available permits live in a single volatile int state inside AQS.
2. Atomic CAS (Compare-And-Swap): when a thread calls semaphore.acquire():

   remaining = current - acquires

   If remaining >= 0, it tries the atomic hardware instruction compareAndSetState(current, remaining). If that succeeds, the thread continues right away with no blocking at all.
3. CLH wait queue: if remaining < 0, the thread creates a shared-mode wait node and appends itself to a lock-free, doubly linked FIFO queue (a CLH variant). Before JDK 14 this was a Node marked with the Node.SHARED constant; since the AQS rewrite in JDK 14 it is a SharedNode subclass, but the idea is the same.
4. FairSync vs NonfairSync:
   • NonfairSync (default): a thread calling acquire can barge in and grab a permit that frees up before the queued threads are unparked. This avoids expensive context switches and gives better throughput.
   • FairSync: strict FIFO. A new thread checks hasQueuedPredecessors() and goes to the tail of the queue if others are already waiting.

# virtual threads interaction & carrier thread pinning

A key question for Java 21+: does waiting on a Semaphore block the underlying OS carrier thread (ForkJoinPool worker)?

# virtual threads unmount cleanly on semaphores

When a Virtual Thread calls semaphore.acquire() and there are no permits left:

1. AQS calls java.util.concurrent.locks.LockSupport.park(this).
2. The Virtual Thread runtime sees the park call and unmounts the Virtual Thread from its OS carrier thread.
3. The Virtual Thread's stack is copied to the Java heap as a suspended Continuation.
4. The OS carrier thread is free right away to run other Virtual Threads.

# sizing with little’s law

How many permits (N) should a Semaphore have? Picking N=10 or N=50 by gut feeling is guessing, not engineering.

Use Little’s Law, a basic result from queuing theory:

L = lambda * W

Where:

• L = average number of concurrent requests in the system (concurrency, i.e. Semaphore permits).
• lambda = sustainable arrival rate / throughput (requests per second, RPS).
• W = average response time (latency in seconds).

# a concrete sizing example

• Say a third-party credit scoring API gives your company an SLA of at most 500 RPS (lambda = 500).
• Under load, their average latency is 80 ms (W = 0.080 s).
• By Little's Law, the maximum concurrency is:

  L = 500 * 0.080 = 40 concurrent requests

With Semaphore(40), your application can get close to the 500 RPS it is allowed while never holding more than 40 connections open to the partner.

Be careful with what that number really guarantees. A semaphore caps concurrency, not rate. If the partner gets faster and latency drops to 40 ms, the same 40 permits allow 40 / 0.040 = 1000 RPS, double the quota, and you start seeing HTTP 429. If latency climbs to 200 ms under load, 40 permits only give you 200 RPS. So when the contract is expressed in requests per second, pair the semaphore with a rate limiter (next section) and treat Little's Law as the starting point for N, then adjust it from measured latency. Also size with a realistic latency under load (p90 or so), not the best-case average.

# production hardening: a resilient concurrency gate

In a critical backend, calling semaphore.acquire() without a timeout is asking for trouble. If downstream hangs, waiting threads pile up forever.

Here is a production-ready Concurrency Gate with:

• Strict timeouts (tryAcquire(timeout)).
• Metrics (Micrometer gauges).
• Fail-fast exception handling, so callers can fall back.

package com.company.resilience.gate;
 
import io.micrometer.core.instrument.Gauge;
import io.micrometer.core.instrument.MeterRegistry;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
 
import java.util.Objects;
import java.util.concurrent.Callable;
import java.util.concurrent.Semaphore;
import java.util.concurrent.TimeUnit;
 
public class ConcurrencyGate {
 
    private static final Logger log = LoggerFactory.getLogger(ConcurrencyGate.class);
 
    private final String name;
    private final Semaphore semaphore;
    private final long acquireTimeoutMs;
 
    public ConcurrencyGate(String name, int maxConcurrentPermits, long acquireTimeoutMs, MeterRegistry registry) {
        this.name = Objects.requireNonNull(name);
        this.semaphore = new Semaphore(maxConcurrentPermits, false); // Non-fair for maximum throughput
        this.acquireTimeoutMs = acquireTimeoutMs;
 
        // Export metrics for Prometheus / Grafana observability
        Gauge.builder("concurrency.gate.permits.available", semaphore, Semaphore::availablePermits)
            .tag("gate", name)
            .register(registry);
 
        Gauge.builder("concurrency.gate.queue.length", semaphore, Semaphore::getQueueLength)
            .tag("gate", name)
            .register(registry);
    }
 
    public static class ConcurrencyLimitExceededException extends RuntimeException {
        public ConcurrencyLimitExceededException(String message) {
            super(message);
        }
    }
 
    public <T> T execute(Callable<T> task) throws Exception {
        boolean acquired = false;
        try {
            acquired = semaphore.tryAcquire(acquireTimeoutMs, TimeUnit.MILLISECONDS);
            if (!acquired) {
                log.warn("Concurrency gate [{}] shed load: Wait queue timeout after {} ms", name, acquireTimeoutMs);
                throw new ConcurrencyLimitExceededException(
                    String.format("Gate [%s] saturated. Queued requests exceeded timeout threshold.", name)
                );
            }
 
            // Execute protected downstream call
            return task.call();
 
        } finally {
            if (acquired) {
                semaphore.release();
            }
        }
    }
}

# running 10,000 tasks with virtual threads

One thing to watch in the example below: the 500 ms acquire timeout is designed for request/response traffic, where shedding load quickly is the right call. In a batch fan-out like this, all 10,000 tasks hit the gate at the same moment. At 40 permits and ~80 ms per call the gate drains about 500 tasks per second, so roughly the first 250 get a permit within 500 ms and the rest fail fast into the fallback. For batch jobs, either give tryAcquire a timeout that matches the whole batch's deadline, or split the batch so that callers wait for a permit instead of being rejected.

@Service
public class ThirdPartyDataAggregator {
 
    private static final Logger log = LoggerFactory.getLogger(ThirdPartyDataAggregator.class);
 
    private final ConcurrencyGate partnerApiGate;
 
    public ThirdPartyDataAggregator(MeterRegistry registry) {
        // Allow maximum 40 concurrent calls, wait up to 500ms before rejecting
        this.partnerApiGate = new ConcurrencyGate("partner-credit-api", 40, 500, registry);
    }
 
    public List<CreditResult> fetchScoresInParallel(List<CustomerRequest> requests) {
        try (var executor = Executors.newVirtualThreadPerTaskExecutor()) {
 
            List<Future<CreditResult>> futures = requests.stream()
                .map(req -> executor.submit(() ->
                    partnerApiGate.execute(() -> callRemoteHttpApi(req))
                ))
                .toList();
 
            return futures.stream()
                .map(f -> {
                    try {
                        return f.get();
                    } catch (Exception e) {
                        log.error("Failed executing request", e);
                        return CreditResult.fallbackDefault();
                    }
                })
                .toList();
        } // Executor auto-joins all virtual threads at close
    }
 
    private CreditResult callRemoteHttpApi(CustomerRequest req) {
        // Standard blocking HTTP client (HttpClient / RestClient)
        // Virtual Thread cleanly unmounts during network socket read!
        return httpScoreClient.getScore(req.customerId());
    }
}

# comparison: semaphore vs rateLimiter vs circuitBreaker

Each resilience pattern limits a different thing. If you use Resilience4j, you don't have to hand-roll the gate: its Bulkhead is a semaphore-based concurrency limit with a max wait time, and its default decorator order (Retry → CircuitBreaker → RateLimiter → TimeLimiter → Bulkhead, outermost first) matches the flow below.

# takeaways

1. Virtual Threads give you scalability, not magic: they remove the local cost of threads. They do not make your database or external network any bigger.
2. Semaphores give you natural backpressure: an AQS-based Semaphore lets upstream Virtual Threads wait in a heap queue without tying up OS carrier threads.
3. Size with Little's Law (L = lambda * W): don't pick permit counts at random. Derive them from the target system's throughput SLA and expected latency.
4. Always use a timeout: acquire permits with tryAcquire(timeout) so a slow downstream doesn't turn into an endless pile-up of waiting requests.

Chỉ là những ghi chép cá nhân với hy vọng mang lại chút giá trị. Nếu thấy hữu ích, đừng ngại chia sẻ cho bạn bè & đồng nghiệp nhé!

Happy coding 😎 👍🏻 🚀 🔥.