Imagine opening an AI assistant, typing a question, and seeing an answer appear almost immediately. It feels like one conversation with one brain. Behind that small text box, though, could be a worldwide system of servers, databases, networks, and expensive GPUs.
How would we design that system for 100 million daily users? Let's build it from the outside in, following one message all the way from your browser to the model and back.
A note before we start: This is a hypothetical architecture, distilled from my larger system-design exercise. The traffic numbers, model sizes, GPU counts, and costs are illustrative engineering assumptions—not claims about OpenAI's actual systems, which are not publicly known in this detail. Techniques such as batching, mixture-of-experts, and paged attention are publicly documented.
1. First, understand the size of the problem
Let's assume 100 million people use our AI service each day and each person sends 15 messages. That's 1.5 billion messages a day, or about 17,400 messages every second on average. A busy region may experience much more traffic, so our illustrative design peak is about 139,000 requests per second.
That's the first important lesson: daily users and simultaneous requests are not the same thing. Most people are reading, thinking, or doing something else while the GPUs work. In our scenario, around 1.5 million generations could be active at the modeled peak—not 100 million.
Every request has two expensive parts: reading your prompt and writing the answer. We'll see why that distinction matters when we reach the GPUs.
2. The journey of a single message
Suppose you ask: “Compare the financial results in these four PDF reports.” The model doesn't instantly receive all four PDFs. Several smaller systems prepare the question first.
The nearest edge filters suspicious traffic and checks basic credentials before a request consumes costly compute.
At the edge, a nearby server protects against abusive traffic and checks basic credentials. An API gateway verifies permissions and usage limits. A conversation service finds the messages you've already exchanged. A document pipeline retrieves relevant parts of the PDFs. Then a router chooses an appropriate model, while a scheduler waits for available compute.
Finally, the model generates a response that a streaming gateway sends to you piece by piece. The system also records the completed conversation and usage. Most of these components are independently scalable—because they experience different kinds of load.
3. Your PDFs need their own system
An AI model does not magically search an uploaded 150-page PDF. A file service stores the original, checks it for malicious content, extracts text and tables, and uses OCR when a page is scanned. It then divides the extracted information into useful sections and builds a searchable index.
When you ask a question, retrieval selects the most relevant sections. Imagine a librarian finding the 10 useful pages across four thick books, rather than photocopying all 600 pages and handing them to the model. That's the idea behind retrieval-augmented generation (RAG).
Combining keyword search with semantic search helps find both exact terms and similar meanings. A reranker then decides which passages deserve the model's limited attention. Page references travel with the passages so the eventual answer can cite its sources.
The clever UX choice is progressive processing: let someone ask about the first pages while deeper parsing continues in the background, but clearly say if not every page is ready.
4. Not every question needs the biggest model
Would you use a supercomputer to answer “What is 2 + 2?” Probably not. Our service shouldn't either.
One possible design offers small, medium, and frontier model pools. A router sends simpler requests to smaller models and difficult reasoning or document-comparison work to more capable ones. It must respect the model the user selected, hard capability requirements, and quality constraints.
This is not just an optimization trick. Sending everything to the largest model could make the service much more expensive. But a badly tuned router can save money while silently making answers worse. Teams therefore need evaluations, controlled experiments, and a way to escalate when a smaller model fails.
5. How does a 3-trillion-parameter model fit on a GPU?
It doesn't. At one byte per parameter, 3 trillion parameters require roughly 3 terabytes just for the weights, before accounting for runtime memory. A single GPU cannot hold that. The work and the memory must be shared across many accelerators.
Our thought experiment uses a mixture-of-experts (MoE) model: instead of activating every parameter for every token, a small router chooses a few specialized neural-network blocks, or experts, for each token at each layer. Think of an enormous organization where the right specialists join each task rather than the entire company attending every meeting.
per layer
For illustration, the detailed design uses 256 experts per layer and chooses eight for each token, with a shared expert that always runs. It proposes about 3.2 trillion total parameters, but only roughly 220 billion active parameters per token. These numbers describe our imagined model, not an existing named model.
We also split work across GPUs: tensor parallelism divides calculations, expert parallelism places different experts on different devices, and data parallelism serves more requests with independent model replicas. The catch? GPUs must communicate quickly. More GPUs don't automatically mean a proportionally faster answer.
6. Why the model reads fast but writes slowly
Inference has two phases.
Prefill is when the system reads your prompt and builds an internal representation of the previous tokens. Much of that work can run in parallel. Decode is when it predicts the response one token at a time, and each new token depends on what came before.
Read lots of input tokens in parallel.
Mostly compute-intensiveWrite the answer one token at a time.
Often memory-bandwidth-sensitiveA powerful design separates these phases onto different GPU pools. Prompt-reading machines can handle big chunks of text; answer-writing machines concentrate on producing tokens smoothly. They pass along the intermediate attention state, often called the KV cache.
The KV cache is like the model's working notebook for the current conversation. It avoids rereading the whole prompt from scratch for each new token. But that notebook grows with conversation length. In our illustrative 3T model, an 8,000-token context uses about 2 GB of KV memory per sequence, while 32,000 tokens use about 8 GB. Long conversations are a memory problem as well as a compute problem.
Production serving engines reduce waste using continuous batching (combine different users' decoding steps), paged attention (allocate memory in reusable blocks), and prefix caching (reuse computations for repeated prompt prefixes). Some also use speculative decoding: a smaller model guesses upcoming tokens and the larger model checks them efficiently.
7. What happens when millions of people show up at once?
The answer is scheduling, not simply buying an infinite number of GPUs.
Requests line up in queues. The scheduler considers who has been waiting, service-level commitments, whether a GPU has memory for the context, and how long a request might take. It keeps giant document jobs away from short interactive questions so one long task doesn't freeze everybody else's chat.
When traffic spikes, the system can pause background work, reuse warm capacity, shift eligible traffic to another region, temporarily reduce expensive features, or queue requests. If it must reject a request, it should return a clear retry signal instead of allowing an endless wait.
In the full hypothetical sizing exercise, a heavily assumption-dependent configuration reaches roughly 74,000 GPUs across model serving pools. That is a capacity-planning scenario, not an observed fleet size or a general requirement for 100 million users. Actual capacity depends enormously on model architecture, tokens per request, latency goals, caching, and real measured throughput.
8. If a GPU fails halfway through your answer
What matters is not that machines fail—they will—but what the system does next.
A production design detects the failed model replica, stops sending it new work, and may restart the request elsewhere using the last safely recorded state. The user should see what happened rather than receiving a duplicate answer or a misleading success signal.
For actions with side effects, such as a payment or an external tool call, blind retries are dangerous. The system needs an idempotency key (a stable request identifier), a way to check whether the action already happened, and a clear fallback when the outcome is uncertain.
Other failures have gentler fallback paths: if semantic search is unavailable, keyword search can still retrieve passages; if one region has a problem, another may serve eligible traffic while respecting where customer data is allowed to live. Backups, circuit breakers, bounded queues, and useful error messages are all parts of reliability—not optional polish.
9. Safety and privacy are parts of the architecture
Security starts before the model sees a prompt. The system validates the user's identity, checks access to files and conversations, isolates one organization's data from another's, and encrypts stored information.
Uploaded documents and web pages are untrusted input. A malicious document can contain instructions that attempt to redirect the assistant. The model alone should never decide which tools it may call: a separate permission-enforcing component must check the action and its arguments.
Safety checks may happen before generation, during streaming, and after completion. Observability should measure queue times, token speed, memory pressure, errors, and quality without dumping private conversation text into ordinary logs.
10. The invisible trade-off: speed, quality, and cost
Our original thought experiment estimates around $5.2 million a day to run its illustrative setup. That's not a real provider's operating cost: it's a spreadsheet result driven by guessed GPU prices, utilization, traffic, staffing, and infrastructure.
The useful insight is the shape of the bill. Generating longer answers consumes more decode capacity. Routing simple questions to suitable smaller models can reduce compute. Caching repeated context saves prompt processing. And leaving spare capacity is expensive—but the alternative is an unreliable service when something breaks.
The big takeaway: scaling conversational AI is not simply “put a giant model behind an API.” It is coordinating smart routing, document retrieval, distributed GPUs, memory, streaming, scheduling, and failure recovery so one small chat box stays useful for a very large crowd.
Further reading
For the real techniques behind this hypothetical design, see vLLM and PagedAttention, Orca on iteration-level batching, SGLang on efficient structured generation, Megatron-LM on model parallelism, DeepSeek-V3's mixture-of-experts design, and DistServe on separating prefill from decode.