---
title: How to design an AI chat assistant like ChatGPT, JusInfer.Chat and others
slug: design-ai-chat-assistant
canonical_url: https://blog.juscode.co/design-ai-chat-assistant
published_at: 2026-07-01T09:00:00+00:00
author: jusCode
tags: system design, agents, inference
tldr: Designing an AI chat assistant is two serving-layer patterns: stream tokens in real time over SSE and optimize time-to-first-token, and manage long-running generation on scarce GPUs with a queue and scheduler. Treat the model as a black box; persist and summarize context so cost stays flat.
key_takeaways:
  - Treat the LLM as a black box; you are designing the serving system, not the model.
  - Optimize time-to-first-token and stream tokens over SSE; a blank screen feels broken.
  - GPUs are scarce: queue requests and let a scheduler decide who gets compute, and when.
  - Enforce fairness with rate limits and tiered priority so heavy users cannot starve everyone else.
  - Persist every turn for resumable chats, and summarize old context so cost stays flat.
---

Designing an AI chat assistant, the way it comes up in a system design interview, is really two serving-layer patterns. The model is a commodity you call; the interesting engineering is everything around it. The same patterns power any LLM product, from coding agents to a chat assistant like ChatGPT or JusInfer.Chat.

## The model is the easy part

You are not designing an LLM. Treat the model as an opaque service you call: no training pipelines, no internals. Every interesting decision lives in the layer around it. How do tokens get back to the user fast? How are requests routed onto expensive GPU workers? How do conversations persist and resume? How does cost stay sane as context grows? Scope it to text in, text out, at roughly 200M daily active users.

::figure{template=compare caption="The design lives in the serving layer, not the model" items="The model :: black box, prompt in, tokens out | The serving layer :: streaming, scheduling, persistence, context"}

## Requirements draw the line

Strong candidates spend the first minutes deciding what is in and what is out, then name the constraints that actually shape the architecture.

In scope: send a prompt and receive an AI-generated response; view past chats and resume one, with prior context carried into the next prompt. Deliberately cut: editing or branching messages, images or audio or video, sharing chats, tool calling, browsing, and full-text search over history.

The constraints that shape the design:

- **Low time-to-first-token.** A blank screen after hitting enter feels broken. Optimize latency to the first token over total completion time.
- **Deliberate GPU use.** GPUs are the scarce, expensive resource. The system must decide who gets compute, and when.
- **Durable history.** Conversations persist; users pick up old chats where they left off.
- **Cost that scales sanely.** Longer conversations mean bigger prompts. Inference cost cannot grow unbounded with chat length.

## One request path, one long-running path

A synchronous path for chat history, an asynchronous path for generation, with tokens streamed back the moment they exist.

::figure{template=pipeline caption="Prompt to streamed tokens" items="Client :: web or mobile | API Gateway :: auth, rate limit | Chat Service :: builds context | Queue :: buffers spikes | Scheduler :: who gets GPUs | GPU Workers :: LLM black box"}

The chat service reads and writes a chat-history store, builds the prompt context, and enqueues a generation job. A scheduler assigns the job to a GPU worker, which streams tokens straight back over the open connection and persists the final message when the stream ends.

## Two patterns carry the whole design

Recognize them, and the architecture assembles itself.

**1. Real-time updates.** Generation takes seconds, but the product must feel instant. Instead of waiting for the full response, push each token to the client the moment the model produces it. Perceived latency collapses to the time-to-first-token. Stream over SSE on one HTTP connection, since the flow is one-way and simpler than WebSockets, smooth bursts client-side, and measure TTFT rather than total completion time.

**2. Managing long-running tasks.** A request can run for many seconds on hardware you cannot just add more of. Decouple accepting the request from executing it: enqueue jobs, let a scheduler decide which GPU worker runs what by load and tier, and support cancellation so a stop frees the GPU immediately.

::figure{template=compare caption="Blocking vs streaming: same total time, different product" items="Blocking :: blank screen for about 8s, then the whole answer | Streaming :: first token about 240ms, reading starts immediately"}

## Where interviewers push

**How do tokens get back fast, and smoothly?** Stream over SSE end to end, from worker to service to client. Avoid proxies that buffer, render at a steady cadence even when tokens arrive in bursts, and write the completed message to the database after the stream ends so a slow write never blocks the user.

**How are requests routed across GPU workers?** Workers are stateful while generating and wildly expensive, so naive round-robin wastes them. A scheduler tracks worker load and in-flight generations, batches compatible requests, and admits new work only when capacity exists. Under overload, requests queue rather than crush the fleet; backpressure protects the inference servers.

**How do you keep heavy users from monopolizing GPUs?** Per-user rate limits plus tiered quality of service. Separate queues or priority weights so premium traffic schedules first, while free-tier requests degrade gracefully, waiting longer or falling back to a cheaper model, instead of failing.

**How do you control cost as conversations grow?** Every turn re-sends prior context, so a long chat makes each prompt bigger and more expensive. Cap the context window deliberately: keep recent turns verbatim, compress older ones into a running summary, and drop what no longer matters. The assistant stays coherent without paying to re-read the entire history every turn.


## FAQ

### Why SSE instead of WebSockets?

Token streaming is one-way, server to client. SSE rides on plain HTTP, reconnects automatically, and needs no special infrastructure. WebSockets earn their complexity only when the client must also push data mid-stream.

### Does this design apply beyond chat assistants?

Yes. Any product that calls an LLM, coding agents included, inherits the same two problems: making generation feel instant, and rationing scarce inference capacity. The serving-layer patterns transfer directly.

### Where should the response be written to the database?

After the stream completes, not during it. Persisting per token multiplies writes for no user benefit; persisting at the end keeps the hot path free. Handle a mid-stream crash by marking the message incomplete and letting the client retry.