The first time I used a large language model, the part that fascinated me wasn't just that it could answer a question. It was that the answer appeared so effortlessly. I could paste a long document, ask a follow-up, and watch the response stream onto my screen. Something that looks like a simple chat window is actually one of the more interesting distributed-systems problems you can design.
Imagine we have to build our own conversational AI platform. We want it to feel fast, remember conversations, work with files and tools, survive failures, and eventually serve 100 million people every day. What would we build? How would we fit the model into GPU memory? How would we stop its inference bill from swallowing the company?
I want to explain this in the order I wish someone had explained it to me: first the model, then making one model fast, then serving one user, and finally serving 100 million. We'll build a hypothetical production system from first principles, with the calculations in plain sight.
An important distinction: This is not a description of OpenAI's private infrastructure, nor a claim about the parameter count or serving technology of any actual ChatGPT model. Real proprietary architectures, routing policies, throughput, and GPU fleets are not public in the detail required here. I'm combining published research with a deliberately hypothetical model and workload to show how a ChatGPT-like platform could be built. Every hardware count below is a scenario estimate, not a measured fact.
1. What happens when you ask ChatGPT a question?
Let's start with something ordinary: you type “Explain why the sky is blue” and press Enter. At the model level, the job is surprisingly simple: given the text seen so far, predict what token should come next. Repeat that process many times, and those next-token predictions become paragraphs, code, or explanations.
A token is a small piece of text, not necessarily a whole word. “Unbelievable” might be split into several tokens; punctuation may have its own. A tokenizer maps the text into integer IDs. Those IDs become numerical vectors called embeddings. Then a stack of transformer layers repeatedly transforms those vectors, considering how tokens relate to one another. Finally, the model turns its output into scores over possible next tokens, chooses one, appends it to the context, and runs again.
The model doesn't store a literal encyclopedia that it opens for each question. Its learned knowledge is encoded in billions or trillions of numerical weights, also called parameters. Training adjusts those weights. Inference is the separate activity we're studying: using trained weights to answer a live request. Training can take weeks or months; inference must feel responsive within seconds.
Inside a transformer, attention helps each token use relevant information from earlier tokens. Imagine you ask: “Alice gave Bob her notebook. Who owns the notebook now?” Attention helps later tokens relate “her notebook” and “Bob” to the right context. A feed-forward network then processes each token's representation. This attention-plus-computation pattern repeats across many layers. In a causal language model, a token may attend to earlier tokens, but not to future answer tokens that haven't been generated.
One crucial consequence follows: during generation, the next answer token usually depends on the previous answer tokens. You can't simply generate a 400-token answer in one independent parallel pass. Even on astonishingly fast GPUs, that sequential dependency is why output generation is so expensive.
2. Prefill versus decode: one request, two very different jobs
An LLM request has two main stages. Prefill processes the existing input (instructions, previous messages, retrieved files, and your question) in parallel and builds attention state. Decode generates the answer one new token at a time. These stages stress hardware differently.
Read lots of input tokens in parallel.
Mostly compute-intensiveWrite the answer one token at a time.
Often memory-bandwidth-sensitiveDuring prefill, GPUs do big matrix operations on many input tokens at once. This phase often benefits from more raw compute. During decode, the model repeatedly reads weights and earlier attention state while producing relatively few new tokens per request per step. That can make decode memory-bandwidth-limited: the GPU is waiting to move data rather than running out of arithmetic units. The exact bottleneck depends on batch size, architecture, and model.
Two metrics are worth remembering. Time to first token (TTFT) measures how long the user waits before the answer starts appearing. Inter-token latency (ITL) measures the delay between successive pieces of the streamed answer. An architecture can have excellent overall throughput but still feel bad if TTFT or ITL is too high. A good serving system manages all three: TTFT, ITL, and aggregate throughput.
3. The KV cache: how a model avoids rereading its entire past
Without a cache, each time the model produced a new token it would have to repeat attention computations for the preceding context. That is wasteful. Transformer attention calculates three kinds of representations: Query, Key, and Value (Q, K, V). Think of the query as “what information am I looking for?”, keys as “what information does each earlier token advertise?”, and values as “what content should be retrieved?”
KV caching saves the keys and values computed for earlier tokens. When a new token arrives, the model computes that token's new attention state and reads the stored past. It doesn't need to recreate the entire history's keys and values from scratch. This is a cache of intermediate model state, not a database of user conversations; the latter is stored separately and can be reconstructed into a prompt on a later request.
Here's the catch: this cache can be enormous. It grows with context length and the number of simultaneously running sequences. A long conversation is not free just because the model weights already fit in memory.
For a standard attention arrangement, a useful first-order estimate is:
KV bytes = 2 × layers × tokens × KV heads × head dimension × bytes per element
The two represents keys and values. For our later hypothetical model—120 layers, 8 KV heads, head dimension 128, and one byte per KV element—we get 245,760 bytes per token, about 240 KiB. At 8,000 tokens, that is roughly 1.97 GB of KV state for one sequence across the full model. At 32,000 tokens, it's roughly 7.86 GB. At 128,000 tokens, around 31.46 GB. These are design-example numbers, not a general fact about ChatGPT.
So one of the least obvious facts about AI systems is this: concurrent users consume GPU memory even when they are running the same model. Model weights are a fixed cost per replica. KV cache is the growing cost of each active conversation.
4. FlashAttention, PagedAttention, and prefix caching are different optimizations
You'll often hear these terms together, but they solve different problems. Knowing the distinction is a surprisingly useful way to understand modern inference systems.
FlashAttention makes the attention calculation more efficient. Ordinary implementations can create huge intermediate attention matrices and move them repeatedly between high-bandwidth GPU memory and fast on-chip memory. FlashAttention works on tiles (small blocks), carefully reusing on-chip memory and computing exact attention without materializing the full large score matrix. It reduces costly memory movement. It does not magically eliminate every quadratic operation in full-sequence attention, nor is it the same thing as caching earlier tokens. Original paper · FlashAttention-2.
PagedAttention, introduced with vLLM, solves a memory allocation problem. Traditionally, allocating a giant contiguous buffer for each growing KV cache wastes space or causes fragmentation. PagedAttention stores KV entries in small blocks that can be located in different places, much like an operating system manages virtual-memory pages. A mapping tells the attention kernel where to find them. The result is better cache utilization and more requests fitting on the GPU. vLLM paper.
Prefix caching avoids repeating prefill for identical earlier prompt segments. Imagine 10,000 requests share a stable system prompt, or a conversation repeats its earlier turns. If the compatible model and cache keys match, the system can reuse saved KV blocks for those prefixes. Careful keying is essential: sharing cached state between incompatible model versions or user-specific contexts can leak information or produce wrong results. SGLang / RadixAttention.
Avoid moving huge temporary attention matrices back and forth.
Place cache in small blocks, instead of wasting contiguous GPU memory.
Skip repeated prefill when cached prompt beginnings genuinely match.
An analogy: FlashAttention is a chef rearranging the kitchen so ingredients travel less; PagedAttention is an organizer preventing wasted fridge space; prefix caching is keeping already-prepared ingredients for the next valid order. All three can be used together.
5. Continuous batching: don't make new users wait for an entire answer
Suppose five users send messages at nearly the same time. A naive server might process them one at a time. Better, it may collect them into a single fixed batch—but then new requests must wait until the slowest old request finishes. The trick is continuous batching, also called iteration-level batching.
At every token-generation step, completed requests leave and eligible new requests join. The GPU stays busy without tying its schedule to the entire lifetime of one user's answer. This is one reason a modern serving engine such as vLLM or SGLang can deliver many more output tokens per second than a naive model.generate() loop.
There is a tradeoff: larger batches amortize weight reads and improve throughput, but too many active sequences increase KV memory demand and sometimes degrade latency. The scheduler continuously solves a constrained problem: which requests can make progress now, given memory, latency budgets, priority, and GPU load?
Long input prompts create another headache: one enormous prefill can interrupt dozens of active decodes. Two responses are chunked prefill (split long prompts into scheduling-friendly pieces) and prefill/decode disaggregation (run prefill and decode on separate GPU pools, then transfer KV state between them). Disaggregation is powerful when workloads are mixed, but transfer and orchestration overhead can make it unnecessary for small requests. DistServe · SGLang serving docs.
6. Speculative decoding: let a small model draft, then verify
We established that decoding is sequential. What if we let a cheap, fast model guess several future tokens and ask the large model to verify them in parallel?
That is speculative decoding. A small draft model might propose the continuation “The sky appears blue because”. The expensive target model checks this draft efficiently in one verification pass. It accepts the valid prefix and resumes drafting from the first rejected point. With the correct acceptance algorithm, the final sampling distribution is preserved—the small model is accelerating the target, not replacing its decisions.
There are several variants. Classic speculation uses a separate draft model. Medusa and multi-token prediction (MTP) can create drafts with extra prediction heads; EAGLE-style methods use specialized draft mechanisms. These ideas can increase effective tokens per expensive target pass. They help most when verification is cheaper than repeated serial target steps and the draft is reasonably accurate. They may help less if the GPU is already compute-saturated, draft acceptance is poor, or management overhead outweighs the saved work. Speculative decoding paper · Medusa.
Other useful techniques include quantization (using fewer bits for weights and sometimes KV), fused kernels (combining adjacent GPU operations to avoid launches and memory traffic), CUDA graphs for repeated execution shapes, and constrained decoding for valid JSON or tool-call formats. None is a universal free lunch: measure output quality, latency, memory, and reliability together.
7. Mixture of Experts: trillions of weights, not trillions of active operations
Now let's solve a bigger problem: suppose we want an extremely capable model with trillions of learned parameters. If every parameter participates in every token's computation, a dense model can become extraordinarily expensive to serve.
A Mixture of Experts (MoE) transformer replaces some dense feed-forward layers with a collection of smaller specialist networks called experts. A lightweight router scores them for each token and activates only a few. Think of a large company in which every employee has expertise, but you don't summon every employee to every meeting.
per layer
This is token-level routing inside a layer, not the same as sending the whole user request to a different chatbot. A token may consult a different set of experts at every MoE layer. All expert weights still need to be stored or accessible, even if only a fraction runs for a particular token. MoE reduces active arithmetic, but adds complicated cross-GPU communication, expert load imbalance, and scheduling. A suddenly popular kind of prompt can overheat particular experts. Mixtral · DeepSeek-V3 report.
For example, the published DeepSeek-V3 design has 671 billion total parameters and about 37 billion activated per token. That illustrates why total parameters and active parameters are different numbers. It does not tell us the architecture or size of proprietary ChatGPT models.
8. GQA, MLA, and quantization: shrink the memory around attention
The KV cache formula from earlier contains KV heads. This is a major design knob. In standard multi-head attention (MHA), attention heads have their own key/value heads. Grouped-query attention (GQA) lets multiple query heads share a smaller set of key/value heads. If 128 query heads share 8 KV heads, the KV-head component of cache memory is 16 times smaller than the corresponding 128-head arrangement, all else equal. Multi-query attention (MQA) shares even more aggressively. GQA paper.
Another family, Multi-head Latent Attention (MLA), compresses aspects of attention state into a latent representation and reconstructs what the computation needs. It can further reduce cache overhead, with kernel and architecture tradeoffs. The right choice depends on the model's training and implementation; you cannot swap GQA and MLA into an arbitrary frozen checkpoint without care. DeepSeek-V2.
Quantization attacks the bytes used to store numbers. At two bytes per parameter, 3 trillion parameters need about 6 TB just for weights. At roughly one byte per parameter, they require about 3 TB, plus scaling metadata and runtime overhead. Lower-bit formats can shrink them more, but they may introduce quality changes and often require specialized kernels. Model-weight precision, KV precision, and activation precision are separate decisions. Always evaluate actual accuracy and throughput rather than assuming fewer bits always wins.
9. Let's invent a frontier model—and calculate its memory
We now have enough tools to design a concrete hypothetical model. Let's call it FM-3T. It is not an assertion about GPT-6 or any real service.
| Architecture choice | Our hypothetical setting | Why it matters |
|---|---|---|
| Total parameters | 3.2 trillion | Weight-storage footprint |
| Active parameters per token | ~220 billion | Approximate compute per token |
| Transformer layers | 120 | Repeated processing and KV growth |
| Routed experts | 256 per MoE layer | Large sparse expert capacity |
| Selected experts | 8 per token per layer | Only a fraction runs |
| KV heads / head size | 8 / 128 | Lower KV memory via GQA |
| Weight and KV storage | FP8-like, with overhead | Smaller footprint; must validate quality |
| GPU for illustration | B200-class, 192 GB HBM | A hardware sizing reference |
Start with only the model's raw weights. In FP8-like storage, roughly 3.2 trillion × 1 byte ≈ 3.2 TB, plus some quantization metadata. BF16 doubles that to 6.4 TB. An individual 192 GB GPU cannot hold either. At a generous 80% memory budget, 3.2 TB requires at least 21 GPUs (ceil(3,200 / (192×0.8))), before redundancy, activations, KV memory, and networking constraints. If we add scale overhead the floor rises. This is a memory floor, not a deployable low-latency configuration.
A common confusion is thinking that because weights fit on 21 GPUs, 21 GPUs must be the right inference cluster. That ignores how quickly those GPUs can cooperate, whether expert tokens need to cross the network, how much KV cache fits, and how many answers can be generated concurrently. Fitting a model is not the same as serving a model well.
10. One serving replica: why our design chooses 256 GPUs
For this example, imagine one production frontier serving replica spread across 32 machines with 8 B200-class GPUs each—256 GPUs total. This is a hypothetical performance-oriented architecture, not a minimum or a statement about a real vendor. The GPUs jointly serve one logical model replica; not 256 independent full copies of FM-3T.
There are several ways to split model work:
Tensor parallelism (TP) splits large matrix operations across GPUs connected by a very fast local link, such as NVLink. In the example, groups of 8 GPUs cooperate on the attention and shared layers. Expert parallelism (EP) distributes different experts across the 256 GPUs, sending tokens to whichever GPUs own the selected experts. Pipeline parallelism (PP) can place different model layers on different sets of GPUs; this improves some throughput cases but can add latency because each token must travel through each stage. Data parallelism makes independent serving replicas so many user requests can run simultaneously. Context parallelism splits exceptionally long sequences across devices. Megatron-LM · Parallelism for large models.
These are descriptions of how the same physical devices are used. Don't blindly multiply TP × EP × attention-DP: expert-parallel and attention-parallel groups can overlap on the same 256 GPUs.
Inside each 8-GPU machine, high-bandwidth GPU interconnects help split calculations. Between machines, a fast network such as InfiniBand transports expert inputs and outputs. MoE adds an all-to-all communication pattern: send hidden-state vectors to the selected expert owners, compute, then return results. At extreme scale, this network traffic may become a larger bottleneck than arithmetic. Rebalancing hot experts, grouping messages efficiently, and placing related work close together all matter.
Here's the practical reality: if our 256-GPU replica is healthy but each token spends most of its time crossing a congested network, buying more GPU compute won't fix the problem. The serving unit is a computer system, not just a pile of accelerators.
11. How many users can one replica handle?
The right question is not “how many users fit on 256 GPUs?” but how many input and output tokens per second can this specific replica sustain at an acceptable latency? The answer depends on the workload, implementation, sequence lengths, memory allocation, GPU generation, and network fabric. It must be benchmarked. There is no trustworthy one-size-fits-all tokens-per-second number for a hypothetical unreleased model.
Still, we can run a sensitivity exercise. Suppose a highly optimized FM-3T decode replica has an illustrative sustained throughput of 140,000 output tokens per second across its 256 GPUs, under a deliberately throughput-focused workload. That is an assumption, not a published benchmark or an assured 40-tokens-per-second experience for every sequence. If the true number were 100,000 or 180,000, our fleet requirement would change by tens of thousands of GPUs. We'll expose that uncertainty rather than hide it.
The memory side is separate. At an average 8K-token context, each sequence uses roughly 2 GB of KV state across its participating devices. A hypothetical available KV pool of ~31 TB across the 256 GPUs has a theoretical upper bound around 15,000 such sequences. A production scheduler would use a smaller concurrency cap to preserve latency, handle bursts, and avoid fragmentation and bandwidth saturation. This is why KV capacity is a necessary check, but not a guarantee of useful throughput.
For very long prompts, the KV cache consumes dramatically more memory. Short interactive chat and 128K-token document analysis should generally have different scheduling priorities and often distinct serving lanes.
12. Now the real challenge: 100 million daily users
Let's give our new platform a deliberately demanding traffic profile. Assume 100 million daily active users (DAU), each sending an average of 15 messages a day. That is 1.5 billion requests a day. Spread across 86,400 seconds, the average is ~17,361 requests per second. For an illustrative stressed peak, use 8× that average, or ~138,889 requests per second.
The 8× factor is a stress-test assumption for our illustrative total demand, not a claim about actual global ChatGPT traffic or measured peak-to-average ratios. In a real system, we would forecast per region and time zone, model correlated spikes, and avoid applying multiple peak factors to the same requests.
Suppose an average request reads 2,000 prompt tokens (including conversation history and any retrieved context) and produces 400 answer tokens. At our stressed peak, we'd need to process ~278 million input tokens per second and generate ~55.6 million output tokens per second across all model tiers. If 60% of eligible prompt tokens can be reused from prefix caches—a hypothetical and workload-dependent hit rate—the fresh prefill work drops to roughly 111 million input tokens/s. Prefix-cache hits don't eliminate the storage and accounting work of a request, and not all tokens are cacheable.
What about concurrency? If a response takes 10.8 seconds from admission to completion on average at peak, Little's Law gives concurrency ≈ arrival rate × service time ≈ 138,889 × 10.8 ≈ 1.5 million active generations. Five million connected browser tabs would not mean five million active GPU generations: many users might be reading or typing rather than waiting for new output.
This distinction is the economic heart of the system. 100 million daily users do not require 100 million copies of the model. They require enough active serving capacity for the request rate, response length, context, and latency at the same time.
13. A multi-model router prevents the largest model from doing every job
Sending “hello” and “derive a new theorem” through the same expensive frontier replica would be wasteful. A production AI platform can expose one familiar interface while routing requests among several specialized or differently sized model pools.
For our scenario, let's imagine 25% frontier, 50% medium, and 25% small models. This is a system-design choice, not a claim about any particular company's routing. Simpler drafting, formatting, and classification tasks may fit smaller models; advanced reasoning, difficult coding, and complex multimodal analysis may require larger ones. The router must also respect user-selected modes, enterprise policies, supported tools, context length, and quality requirements—not secretly degrade results simply to save money.
The router is not a single magic classifier. It can be a combination of rules, a cheap complexity estimator, safety and tool requirements, recent load, measured quality, and admission-control decisions. We would continuously evaluate false downgrades: a 10% cost saving is a poor trade if critical answers become wrong.
14. Turning tokens into GPU machines: the full capacity plan
Let's make the arithmetic explicit. We will size frontier decode, frontier prefill, medium decode, small decode, and auxiliary models separately. For clarity, use these unverified capacity-planning assumptions:
| Pool | Hypothetical sustained throughput | GPUs per serving unit |
|---|---|---|
| Frontier decode | 140K output tok/s | 256 |
| Frontier prefill | 800K fresh input tok/s | 256 |
| Medium decode | 24K output tok/s | 8 |
| Small decode | 7K output tok/s | 1 |
For the frontier tier, use 600 output tokens and 2,500 input tokens per request, with 45% of prompt tokens requiring fresh prefill. The frontier traffic is ~34,722 requests/s. Decode demand becomes 20.83 million output tokens/s. Divide by 140,000 tokens/s per replica: 149 decode replicas, or 38,144 B200-class GPUs before spare capacity. Add 20% reserve: ~45,800 GPUs.
Prefill uses different arithmetic: 34,722 × 2,500 × 0.45 ≈ 39.1 million fresh input tokens/s. At 800,000 fresh input tokens/s per prefill unit, that is 49 prefill units, or 12,544 GPUs before reserve and ~15,100 after 20% reserve. We are intentionally not making the earlier document's unquantified off-peak adjustment that reduced this to 40 units: it would understate the stated peak.
The medium tier (50% of requests, 400 answer tokens) requires ~27.8 million output tokens/s: about 1,158 eight-GPU units, or ~11,100 GPUs with reserve. The small tier (25% of requests, 250 output tokens) requires ~8.68 million tok/s: about 1,241 single-GPU units, or ~1,500 GPUs with reserve. Add an illustrative 3,500 auxiliary GPUs for embeddings, OCR, moderation, reranking, vision, and tool support.
That yields an order-of-magnitude scenario total around 77,000 GPUs during this aggressive modeled peak, mostly B200-class for the specified frontier pools. Real deployments could differ enormously: different architecture, output mix, token throughput, GPU types, caching, and time-of-day load would change the result. The detailed arithmetic is valuable not because 77,000 is the right real-world number, but because every assumption is visible and contestable.
How many machines? The two frontier pools alone require approximately (45,800 + 15,100) / 8 ≈ 7,600 eight-GPU servers in this scenario. If every remaining GPU also occupied an eight-GPU host—which is not realistic for every auxiliary workload—the entire fleet would be roughly 9,600 eight-GPU-equivalent hosts. Actual physical server counts depend on GPU types, form factors, how GPU pools are packed, CPU-only services, and redundancy. It also excludes separate model-training infrastructure.
The most dangerous mistake in an estimate like this is false precision. Our throughput figures are illustrative, and the 20% reserve is a design assumption, not a universal allowance. A responsible engineer would present ranges and benchmark gates before placing hardware orders.
15. What happens when throughput is lower than the spreadsheet claims?
Let's stress-test the riskiest number: frontier decode throughput. Suppose output length remains 600 tokens and the frontier traffic split remains 25%.
- At 100K output tokens/s per replica, frontier decode needs roughly 64K GPUs with reserve.
- At 140K, the reference case needs roughly 46K GPUs.
- At 180K, the need drops to roughly 36K GPUs.
These are the same workload under different assumed replica performance. They also ignore what happens if longer contexts reduce useful throughput or strict per-user latency caps force smaller batches. The right benchmark should replay realistic prompt/output distributions, not only measure throughput with a synthetic fixed length and infinitely patient users.
Our interactive sizing tool lets you explore how the frontier share, answer length, and throughput affect the estimate. Treat it as a capacity calculator, not a prediction of any company's actual footprint.
16. The full request path, from browser to GPU and back
Let's walk through a concrete chat turn: “Compare these four PDF reports and explain the key differences.” The browser sends the message to a nearby edge location using HTTPS. Edge protection filters abusive requests, a gateway authenticates the caller, checks limits and permissions, and creates a trace ID so operators can follow the work.
The conversation service looks up recent turns. The file service resolves the permitted documents. Retrieval selects relevant sections and table rows. A context builder assembles a bounded prompt: system instructions, selected history, relevant document passages, tool descriptions, and the new question. A request router chooses a model pool, while an admission controller decides whether there is enough KV memory and GPU scheduling capacity to accept the job immediately, queue it briefly, or respond with a transparent busy/retry signal.
The nearest edge filters suspicious traffic and checks basic credentials before a request consumes costly compute.
The prompt reaches the prefill engine. It computes attention state, using compatible prefix cache blocks where available. In a disaggregated system, a transport moves the necessary KV state to decode workers. The decoder then produces tokens and sends them through a streaming gateway to the browser, often using server-sent events (SSE). The user sees the answer growing without waiting for every token to finish.
At completion, the system stores the finalized answer, emits usage/accounting events, and frees temporary KV allocations. Importantly, billing, metrics, and analytics should not become blocking synchronous steps in every streamed token.
17. Uploaded files are their own distributed system
A useful chat assistant is more than a GPU. Consider a 200-page scanned PDF, a spreadsheet with 500,000 rows, a presentation, or a set of images. These must be handled by a dedicated ingestion pipeline rather than pasted directly into a model prompt.
A sensible path begins with a short-lived signed upload URL so the client uploads directly to durable object storage. The platform records file metadata and permissions, checks the actual file type and malware risk, then parses in sandboxed workers. Digitally generated PDFs can often yield structured text; scans need OCR. Tables, images, captions, and page references should retain their structure and provenance. The parser can publish early pages while the rest are processed asynchronously.
Next comes chunking (splitting content into searchable pieces), embeddings (numerical representations used by semantic search), keyword indexing, and optionally summaries. At question time, hybrid retrieval combines keyword matches with vector similarity and a reranker to select the strongest evidence. Citations should carry file and page provenance. For giant spreadsheets, the system should store the data as queryable tables rather than flattening every cell into a multi-million-token prompt.
A critical security requirement is authorization before retrieval. Users must not see snippets belonging to another person or organization. Untrusted documents can contain malicious instructions (prompt injection); treat retrieved text as data, not as commands with the same authority as system instructions. Sandboxed tools, scoped permissions, and confirmation for irreversible actions are more robust than trusting a classifier to remove every malicious instruction.
Why not feed all 600 pages into the model? Sometimes long-context inference is appropriate, but it consumes more prefill compute, larger KV memory, and can make relevant facts harder to retrieve reliably. For many document questions, RAG is cheaper and easier to cite. For exhaustive reasoning across the entire document, combine retrieval with targeted full-context passes or structured analysis.
18. How conversations and memory survive after the GPUs finish
The KV cache is temporary inference state. It does not replace persistent history. A conversation system needs durable message records, conversation metadata, ownership and sharing rules, attachments, optional summaries, and user-managed retention/deletion policies. A relational database may work for user and billing metadata; a horizontally partitioned message store can absorb high-volume chat writes. Files belong in object storage; embeddings in a vector index; hot temporary state in Redis-like caches; analytics events in an append-only pipeline.
For long conversations, the system can't keep re-sending every historical token forever. A context builder chooses recent messages, relevant older turns, explicit saved facts where permitted, and a summary of earlier history. Summaries can omit or distort details, so the system should be able to retrieve original source turns for important claims. Persistent memory, conversation history, retrieval indexes, and GPU KV caching are four different things. They have different access controls, costs, and deletion behavior.
Under multi-device use, sequence numbers and stable message IDs prevent lost or duplicated turns. A retry after a timeout needs an idempotency key or another server-enforced deduplication mechanism so a reconnect doesn't accidentally create two charges or two messages.
19. Tools and agents turn inference into workflows
A question like “What was the weather yesterday?” might need a search tool. “Summarize this spreadsheet and send a report” may need parsing, code execution, and a permissioned email action. Now our system is doing orchestration, not just text generation.
A simple agent loop is: the model proposes a tool call, the orchestrator validates its arguments and permissions, a sandboxed tool runs it, the result returns as an observation, and the model decides what to do next. Every action needs a request ID, deadline, bounded retries, audit record, and clear state. Never assume a timed-out write failed: it may have succeeded while its acknowledgement was lost. Reconcile the outcome before a dangerous retry.
We should also distinguish read-only actions from side effects. Retrying a search is usually safe. Retrying a payment, email send, or SQL update can duplicate an irreversible action. Require user approval for consequential operations and design idempotency at the destination wherever possible. Human handoff and resumable checkpoints matter when multi-step workflows last longer than the browser connection.
Tool calls can spawn a second or third prefill after the model receives results. Capacity planning that counts only one model pass per chat message will therefore underestimate heavy agentic traffic.
20. Scheduling, autoscaling, and the problem of a viral minute
Traffic is never smooth. Imagine a major product launch causes a sharp burst while some GPU racks are already busy. You cannot spin up a 256-GPU distributed replica as quickly as a web server: weights must be staged, machines placed, GPU interconnects initialized, and the model warmed. Startup can take far longer than a user's latency budget.
So the system needs warm spare capacity, queue limits, per-tier budgets, cancellation handling, and predictive scaling. Scale on queued tokens, KV pressure, TTFT, ITL, and available replicas, not simply CPU utilization. In a shortage, stop optional batch jobs first, avoid accepting more work than can be completed, and give honest retry estimates or explicit model alternatives rather than silently timing out requests. Long-context analyses, file processing, and interactive text can run in separate queues with different deadlines.
Admission control is the part that says “not yet” before the GPU is overwhelmed. Backpressure propagates that constraint upstream so callers stop adding more work. Load shedding declines lower-priority or optional work when demand genuinely exceeds available capacity. A platform that promises every request immediate service during a sustained 10× spike will fail at a worse time for everyone.
21. Reliability means recovering from the middle of an answer
Suppose a node fails after streaming 100 out of 400 answer tokens. The system has already delivered partial text to the user, perhaps executed a tool, and maybe written usage events. Blindly rerunning everything risks duplicate side effects and a different answer.
A robust design tracks a stable request ID and the last committed output offset. It keeps a short-lived stream-resumption buffer; if possible, it resumes the stream. If the GPU is gone and the KV cache isn't recoverable, it may re-prefill from verified conversation state and already committed output on a healthy replica, then continue. The output must still be marked as a reconstructed continuation because nondeterministic generation and model-version changes can complicate exact reproduction.
During a regional outage, use global health checks and route new inference to an available region, while honoring data residency and privacy. Avoid casually copying private user files or cross-user KV buffers between regions. The degraded service might provide fresh answers but temporarily lack older files; honesty about missing context is better than inventing it.
A properly designed system uses circuit breakers, exponential backoff with jitter, bounded timeouts, and independent failure domains. At this scale, failures are normal events to be budgeted, not unexpected exceptions.
22. Four regions, two kinds of infrastructure
One plausible geography is North America West, North America East, Europe, and Asia-Pacific. Each region runs the latency-sensitive edge, API, routing, and model-serving services it needs; data is placed according to product and residency requirements. The exact traffic split varies by local time. Follow-the-sun usage means the peak in one region may coincide with spare capacity in another, but cross-region routing incurs latency and may violate data movement rules.
The infrastructure itself has two broad tiers. CPU-heavy services run gateways, retrieval planners, document parsing, metadata databases, streaming connections, and event pipelines. GPU-heavy services run the actual models: frontier, mid, small, embeddings, vision, speech, and rerankers. Kubernetes or a similar orchestrator can manage many services, but multi-node GPU replicas need specialized placement and lifecycle handling to ensure an entire tightly connected model group starts and fails together.
The replica controller needs to know which weights and tokenizer version it is serving, what GPUs belong together, whether the network fabric is healthy, and when the replica is warm enough to accept traffic. Long-running inference models don't behave like stateless HTTP pods.
23. Security, privacy, and trust are part of the architecture
At 100 million users, data security is not a feature you can bolt on later. Authenticate every request, authorize every file and conversation access, isolate organizations, encrypt stored data and network links, rotate secrets, and prevent private user content from appearing in other users' caches, logs, or search results.
Prompt injection is a special risk in a tool-using system: a malicious file might tell the model to leak a secret or ignore instructions. Treat retrieval output as untrusted, restrict tool permissions, sandbox untrusted code and parsers, require approval for sensitive actions, and record auditable decisions. Apply safety checks before and after generation appropriate to the product, including abuse detection and clear escalation paths. Stronger safety rules can increase latency; design an explicit latency and failure policy instead of hoping checks never time out.
For deletion requests, plan the lifecycles of messages, uploaded objects, derived text, vector embeddings, caches, and backups. Removing the visible chat row is not the same as removing every derived artifact. Enterprise customers may also need strict retention controls and residency restrictions.
24. Observability: how do we know the system is healthy?
A GPU fleet this large needs more than a dashboard labeled “GPU utilization.” We should measure TTFT and ITL distributions, queued tokens, decoded tokens/s, prefill cache-hit rates, GPU memory pressure, dropped requests, tool timeouts, file-index freshness, and error rates by region, model, and customer tier.
Trace one request across edge, context assembly, retrieval, model scheduling, prefill, KV transfer, decode, streaming, and storage. Attach correlation IDs to logs without exposing private prompts. Add model-quality signals: accuracy on evaluations, citation correctness, tool execution reliability, and regression tests for safety and prompt injection. Low latency with wrong answers isn't a success.
Model rollout is another distributed-systems problem. A new checkpoint needs compatibility checks for tokenizer, KV layout, tool-call formats, and sampling behavior. Canary a small fraction of traffic, watch quality and latency, then expand. Keep rollback capacity because a new model can be faster in a benchmark yet less stable under real mixed traffic.
25. The cost equation—and the knobs I'd optimize first
If this hypothetical fleet uses approximately 77,000 GPUs under our modeled design peak, a made-up blended $5 per GPU-hour would imply $385,000 per hour, or $9.24 million for 24 hours at that same full fleet size. That's not a real procurement price or an estimate of OpenAI's spend. It ignores CPU clusters, memory, networking, storage, datacenter power, staffing, training, enterprise reservations, and discounted or owned hardware economics. Actual costs require regional utilization curves and specific contracts, not multiplying a peak by 24 hours and pretending every hour looks the same.
The biggest cost levers are fewer unnecessary frontier-model requests, shorter useful outputs, better prefix reuse, efficient KV memory, sustained batch utilization, and a better-performing model/kernel/network stack. Each has a catch: routing can hurt quality, shortening answers can miss details, caching must preserve privacy, and giant batches can harm latency. Optimizing only dollars per token is not enough; optimize cost per successfully completed, correctly answered user task.
There's also an infrastructure price outside the GPU sticker: high-speed fabric, storage for checkpoints, redundant networking, power, cooling, and maintenance reserve. Training a new checkpoint and serving the current one are generally separate capacity exercises. Don't mistakenly add the training fleet to a serving estimate or pretend a brand-new frontier model can be hot-swapped into 256 GPUs instantaneously.
26. What would I build first?
I wouldn't start by buying 77,000 GPUs. I'd begin with one model serving endpoint, measure real TTFT/ITL under representative prompts, and implement cancellation and structured request IDs. Next I'd add persistent conversations and safe file uploads. Then I'd introduce batching, paged KV, prefix caching, and a scheduler with backpressure. Only when a measured bottleneck justifies it would I separate prefill and decode, deploy MoE across nodes, or introduce sophisticated request routing.
From there, build progressively: regional placement and failover, realistic traffic replay, canary model releases, rate-limit and abuse defenses, billing reconciliation, observability, and user-facing reliability. Every major optimization should come with a benchmark showing which bottleneck it fixes and a rollback path if quality changes.
Here's the framework I'd use to review any proposed improvement: What resource is constrained? How does the new technique reduce demand on that resource? What new costs or failure modes does it introduce? What metric proves the user is better off?
27. The full picture
A ChatGPT-like experience isn't one massive model sitting behind an API. It's a chain of separately engineered systems. A transformer turns tokens into predictions; KV caches stop repeated attention work; FlashAttention saves memory traffic; PagedAttention saves cache space; continuous batching fills GPUs; speculative decoding reduces serial target steps; MoE reduces active computation; quantization reduces bytes; and GQA or MLA reduces attention-state memory.
Those optimizations make one serving replica viable. Routing, queues, GPU scheduling, fast networking, durable conversations, file ingestion, RAG, tools, global traffic management, observability, and security make millions of users viable. The exact configuration and hardware bill are model- and workload-specific—and any engineer who gives you a precise fleet count without measuring tokens per second, context mix, and latency targets is guessing.
The most interesting question isn't “how many GPUs does ChatGPT have?” It's “which bottleneck is limiting the next useful answer—and what architectural decision removes that bottleneck without compromising quality or trust?” That's the mindset I find most useful when designing large-scale AI systems.
Further reading: original research behind the techniques
The architecture and calculations above are derived from a hypothetical system-design exercise, supported by publicly documented techniques. For deeper technical treatment, these are excellent starting points:
- Attention Is All You Need — Transformers
- FlashAttention and FlashAttention-2 · Part 2
- vLLM — Efficient Memory Management with PagedAttention
- Orca — Iteration-Level Scheduling
- SGLang — Structured Generation and RadixAttention
- Speculative Decoding · Medusa
- Grouped-Query Attention · DeepSeek-V2 / MLA
- Mixtral of Experts · DeepSeek-V3 Technical Report
- Megatron-LM and tensor parallelism · Large-scale pipeline parallelism
- DistServe: Disaggregating Prefill and Decoding · Mooncake
Written as an educational, independently designed architecture exercise. Figures, model configuration, price assumptions, throughput assumptions, and fleet counts are illustrative—not disclosures about OpenAI or another service.