The True Code Of Production Systems

It Broke the Moment the Second Server Joined

Four months in production, zero incidents. Then traffic tripled, auto-scaling fired, and a second instance came online at 11:14 PM. By 11:17 the support queue had forty tickets. No code changed. No deployment happened. The only difference: there were now two of everything instead of one.

Gaurav Sharma

28 Jun 2026 • 11 min read

Concept art showing a production application breaking when a second instance joins, highlighting hidden distributed systems issues such as inconsistent caching, duplicate background jobs, session loss, and concurrency conflicts.

This is part of The True Code of Production Systems. The series is about the decisions that only become visible when something breaks in production.
Read the full series at The True Code of Production Systems

The system had been running in production for four months without a single serious incident.

A .NET 8 Web API backed by Azure SQL. Clean architecture. Good test coverage. The team was proud of it. Deployments were smooth. Response times were consistent. The on-call rotation had been quiet.

Then the business ran a promotion. Traffic tripled overnight. Azure App Service auto-scaling did what it was configured to do. A second instance came online at 11:14 PM.

By 11:17 PM the support queue had forty tickets. Users were being randomly logged out mid-session. Some users were seeing data that belonged to other users. A background job that was supposed to run once every thirty minutes had run six times in the last hour and sent six confirmation emails to every affected customer. The payment processing endpoint was throwing intermittent concurrency exceptions that had never appeared in four months of production.

Nothing in the code had changed. No deployment had happened. The only thing that was different was that there were now two instances of the application running instead of one.

The incidents that followed that night were not caused by bugs in the traditional sense. The code was correct. The tests passed. The architecture was reasonable. The problem was a category of assumption that is invisible on a single instance and catastrophic the moment a second one joins. Assumptions so deeply baked into the way developers write code that most engineers do not even know they are making them.

What Single Instance Development Hides From You

Every developer writes code on one machine. One process. One runtime. When you run the application locally, there is exactly one instance of everything. One in-memory cache. One set of static variables. One background job scheduler. One file system. One lock protecting a critical section.

In that world, certain things are true that stop being true the moment a second instance exists.

A value stored in memory is the same value every request sees. A lock acquired by one thread prevents any other thread from entering the same section. A job scheduled to run at midnight runs once. A file written to disk is readable by the next request.

These are not bugs. They are reasonable assumptions for a single process. The problem is that they transfer invisibly into production code and sit there harmlessly until the day auto-scaling fires and a second instance joins the party.

At that point, every assumption that was true for one instance becomes a potential incident for two.

The code does not know there is a second instance. There is no error. No exception on startup. No warning in the logs. The second instance comes up, starts handling requests, and immediately begins contradicting the first instance in ways that produce the kind of failures that take hours to trace because nothing looks obviously wrong.

Ask yourself: if a second instance of your application started right now without warning, which parts of your system would immediately begin behaving differently? Have you ever actually thought through that question?

The In-Memory State That Was Never Meant to Be Distributed

This is the most common version of the problem and the hardest to see coming.

A .NET application maintains some state in memory. Sometimes it is explicit, a static dictionary, a singleton service with instance-level fields. Sometimes it is subtle, an IMemoryCache registration, a local variable in a background service, a list that gets populated on startup and updated throughout the day.

On one instance, this works perfectly. The state is consistent. Every request hitting that instance sees the same view of the world.

// Looks completely reasonable. Is a production timebomb.
public class ProductCatalogueService
{
    private readonly IMemoryCache _cache;
    private readonly AppDbContext _db;

    public ProductCatalogueService(IMemoryCache cache, AppDbContext db)
    {
        _cache = cache;
        _db = db;
    }

    public Product GetProduct(int id)
    {
        return _cache.GetOrCreate(id, entry => _db.Products.Find(id));
    }

    public void UpdateProduct(Product product)
    {
        _db.Products.Update(product);
        _db.SaveChanges();
        _cache.Set(product.Id, product); // Updates Instance 1's cache
        // Instance 2's cache has no idea this happened
    }
}

The moment a second instance exists, the cache is split. Instance 1 has its view. Instance 2 has its own view. A product updated through Instance 1 is updated in Instance 1's cache. Instance 2 continues serving the old version to every request routed to it. Users hitting different instances get different responses to the same request. The inconsistency looks random because the load balancer distributes traffic without any guarantee of which instance a user lands on.

The fix is not complicated once the problem is understood. Shared state needs to live outside the application process. Azure Cache for Redis exists precisely for this reason. State that needs to be consistent across instances goes into Redis, not into memory.

But the fix only happens once the team understands that in-memory state is per-instance, not per-application. That understanding is not always present when the code is being written, because on a single instance, the distinction is invisible.

Ask yourself: does your application hold any state in memory that is updated at runtime and read by subsequent requests? If that state lives in a static field, a singleton, or an IMemoryCache, it is per-instance. On two instances, it is inconsistent.

The Scheduled Job That Runs Twice

Background jobs are where this problem becomes immediately and measurably painful.

A hosted service in .NET is registered with the dependency injection container and starts with the application. On one instance, it runs on schedule, does its work, and everything is fine. On two instances, both hosted services start. Both run on the same schedule. Both do the same work at the same time.

// This job runs fine on one instance
// On two instances it runs twice, simultaneously, every time
public class InvoiceGenerationService : BackgroundService
{
    protected override async Task ExecuteAsync(CancellationToken ct)
    {
        while (!ct.IsCancellationRequested)
        {
            await GenerateMonthlyInvoicesAsync();
            await Task.Delay(TimeSpan.FromHours(1), ct);
        }
    }
}

On the night the second App Service instance came online, the invoice job was in the middle of its hourly run on Instance 1. Instance 2 started, saw it was time to run the job, and began its own run simultaneously. The job had no coordination mechanism. No distributed lock. No check to see whether another instance was already doing this work.

Both instances queried the same Azure SQL table, found the same unbilled orders, generated invoices for all of them, and sent emails to all affected customers. Six runs in one hour. Six emails per customer. The support queue filled up before anyone noticed the job was misbehaving.

The solution is distributed job coordination. Libraries like Hangfire with SQL Server or Azure storage as a backing store handle this correctly by design. Only one instance acquires the job and executes it. Other instances see the job is taken and skip it. The work happens once regardless of how many instances are running.

But none of that is configured by default. A hosted service with a timer runs on every instance, always, with no coordination. The team that built the invoice service never needed coordination before because there was never more than one instance. The assumption was invisible until it was not.

Ask yourself: how many background jobs or hosted services in your application are running on every instance simultaneously right now? For each one, is it safe for that work to happen in parallel across multiple instances, or should it happen only once?

The Lock That Means Nothing Across Processes

This one is particularly deceptive because it looks like correctly written code.

A developer identifies a critical section. Concurrent access to this section would cause a race condition. They protect it with a lock. The code review passes. The tests pass. The race condition never appears in four months of production because there is only one instance and the lock genuinely works.

private static readonly object _lock = new object();

public async Task ProcessOrderAsync(int orderId)
{
    lock (_lock)
    {
        // Check if already processed
        var order = _db.Orders.Find(orderId);
        if (order.IsProcessed) return;

        // Process it
        ProcessOrder(order);
        order.IsProcessed = true;
        _db.SaveChanges();
    }
}

The lock keyword in C# operates within a single process. It prevents two threads in the same process from entering the critical section simultaneously. It has absolutely no awareness of other processes. It cannot communicate with the lock object on Instance 2 because Instance 2 has its own copy of _lock in its own memory space.

When the second instance comes online and a retry mechanism sends the same order to both instances within milliseconds of each other, both instances acquire their own local lock, both check the database and find the order unprocessed, both process it, both mark it as processed. The order is processed twice. The customer is charged twice.

This is not a lock bug. The lock is doing exactly what it was designed to do. The problem is that a single-process synchronization primitive was used to solve a distributed synchronization problem. Those are different problems and they need different tools.

Distributed locking in Azure is typically handled through Azure Blob Storage lease-based locks or Redis distributed locks using the SETNX pattern. Either approach ensures that only one instance across the entire deployment can hold the lock at any given time.

// Redis distributed lock - works across all instances
public async Task ProcessOrderAsync(int orderId)
{
    var lockKey = $"order-processing-{orderId}";
    var lockToken = Guid.NewGuid().ToString(); // unique to this acquirer

    var lockAcquired = await _redis.StringSetAsync(
        lockKey,
        lockToken,
        TimeSpan.FromSeconds(30),
        When.NotExists);

    if (!lockAcquired) return; // Another instance is handling this

    try
    {
        var order = await _db.Orders.FindAsync(orderId);
        if (order.IsProcessed) return;

        ProcessOrder(order);
        order.IsProcessed = true;
        await _db.SaveChangesAsync();
    }
    finally
    {
        // Only delete the lock if we still own it. A plain KeyDelete here
        // would release another instance's lock if ours had already expired
        // mid-operation - reintroducing the exact bug we set out to fix.
        await _redis.ScriptEvaluateAsync(
            "if redis.call('get', KEYS[1]) == ARGV[1] " +
            "then return redis.call('del', KEYS[1]) else return 0 end",
            new RedisKey[] { lockKey },
            new RedisValue[] { lockToken });
    }
}

The original code was not wrong for a single instance system. It was wrong for a distributed system. And the distinction only became visible when the distributed system actually existed.

Ask yourself: are there any critical sections in your application protected only by the lock keyword or a local SemaphoreSlim? On two instances, those locks do not protect anything. They protect one instance from itself.

The Session That Disappears Between Requests

This failure shows up from the user's perspective as a random logout. They are in the middle of doing something. They click a button. They are suddenly unauthenticated. They log in again. It happens again twenty minutes later.

The cause is almost always the same. The application is using in-process session storage or IMemoryCache for session data. Session is stored on the instance that handled the login request. The next request is routed by the load balancer to a different instance. That instance has no knowledge of the session. The user appears unauthenticated.

On one instance this never happens. Every request goes to the same instance. The session is always there. The problem is structurally impossible with a single instance and structurally guaranteed with multiple instances and a round-robin load balancer.

// In Program.cs - this works on one instance
// On two instances, session only exists on the instance that created it
builder.Services.AddSession(options =>
{
    options.IdleTimeout = TimeSpan.FromMinutes(30);
});

// The fix: distributed session backed by Redis
builder.Services.AddStackExchangeRedisCache(options =>
{
    options.Configuration = builder.Configuration["Redis:ConnectionString"];
});
builder.Services.AddSession(options =>
{
    options.IdleTimeout = TimeSpan.FromMinutes(30);
});

The same pattern applies to anything stored in IMemoryCache that is user-specific. A user's preferences loaded and cached on Instance 1 are not available on Instance 2. Data access patterns that work perfectly on a single instance become invisible inconsistencies when requests for the same user bounce between instances.

Azure App Service has a feature called ARR Affinity, sometimes called sticky sessions, which routes a user's requests to the same instance for the duration of a session. This can mask the problem but it does not solve it. It adds a dependency on a specific instance staying healthy and it breaks the moment that instance restarts or is replaced during a deployment.

The correct fix is to move any state that needs to be consistent across a user's session out of process memory and into a distributed store. The application becomes truly stateless. Any instance can handle any request. The session data travels with the request, not with the instance.

Ask yourself: if the instance that handled a user's login request was replaced right now by a fresh instance, what would happen to that user's session? Would they be logged out? Would they lose unsaved state? Would the application throw an exception trying to read something that is no longer there?

The File That Only Exists on One Machine

This is the simplest version of the problem and the one that produces the most confusing errors.

Code writes something to the local file system. A generated report. A temporary file used in a multi-step process. An uploaded file saved locally before processing. A configuration file written at runtime.

On one instance, the write and the read happen on the same machine. The file is always there. On two instances, the write might happen on Instance 1 and the read might happen on Instance 2. The file does not exist on Instance 2. The read fails.

// Writes to the local file system of whichever instance handles this request
public async Task SaveReportAsync(byte[] reportData, string filename)
{
    var path = Path.Combine("/tmp/reports", filename);
    await File.WriteAllBytesAsync(path, reportData);
}

// If this runs on a different instance, the file is not there
public async Task<byte[]> GetReportAsync(string filename)
{
    var path = Path.Combine("/tmp/reports", filename);
    return await File.ReadAllBytesAsync(path); // FileNotFoundException on Instance 2
}

The fix is Azure Blob Storage. Files that need to be accessible across instances go into Blob Storage, not the local file system. The local file system of an App Service instance is local to that instance. It is not shared. It is not replicated. On Azure App Service, the local file system is also ephemeral. A restart wipes it.

This is one of the assumptions that feels so fundamental it never gets questioned. Of course a file written to disk can be read back. That is true on a developer's laptop. In a horizontally scaled production environment, it is only true if the same instance handles both operations.

Ask yourself: does any part of your application write files to the local file system and expect to read them back in a subsequent request? In a multi-instance deployment, that subsequent request may land on a completely different machine.

Why This Keeps Happening

The pattern across every scenario in this article is the same. The code was written on one machine, tested on one machine, and deployed to one machine. Every assumption the code makes about the world is an assumption about a single-instance world. Those assumptions are invisible because they are never violated.

Then production scales. A second instance joins. Every assumption that was invisible becomes a failure.

The engineers who built these systems were not careless. They were building in the environment they had. The problem is that the environment they had was fundamentally different from the environment the code would eventually run in, and nothing in the development or testing process surfaced that difference.

Most teams discover this the hard way. An auto-scaling event during a traffic spike. A promotional campaign that triples load. A Black Friday that finally justifies the infrastructure budget. The second instance joins and the incident begins.

The teams that do not discover it the hard way are the ones who asked the question before the incident did.

Ask yourself: has your application ever been tested running as two simultaneous instances against a shared database? Not load tested. Not performance tested. Functionally tested for the category of failures this article describes. If the answer is no, you have not tested your application in the environment it will eventually run in.

Before Auto-Scaling Fires Next Time

Every application that runs on Azure App Service with auto-scaling enabled will eventually run on more than one instance. The question is not whether the second instance will join. It is whether the application is ready when it does.

Go through this checklist before the next scaling event happens.

[ ] Is there any state stored in static fields, singleton services, or IMemoryCache that is updated at runtime and read by subsequent requests? If yes, it needs to move to Azure Cache for Redis.
[ ] Are there background jobs or hosted services that should run only once regardless of how many instances are running? If yes, they need distributed coordination through Hangfire or Azure-backed scheduling.
[ ] Are there any critical sections protected only by the lock keyword or local synchronization primitives? If yes, they need distributed locking through Redis or Azure Blob Storage leases.
[ ] Is session data stored in-process? If yes, it needs to move to distributed session backed by Redis.
[ ] Does any part of the application write files to the local file system and read them back in subsequent requests? If yes, those files need to go to Azure Blob Storage.
[ ] Has the application ever been run as two simultaneous instances in a test environment to verify these assumptions?

The last item is the most important. The others identify where the problems live. The last one is the only way to know whether they have actually been fixed.