How the product becomes a platform: one codebase serving many companies, on many domains, in several regions, each with its own private AI — and how far to go with microservices before they start costing more than they return.
Answers three questions: how tenants are isolated, how one build serves many domains, and when a module becomes a serviceWhat runs today at training.flyer.vn is a prototype: static pages, one imaginary company, state in the browser. The product it describes has to serve hundreds of companies, each with its own documents, permissions, branding, domain and AI assistants, in a region of their choosing, with an audit trail a bank would accept. This document is about the distance between those two things.
| The question | The decision | Why this and not the other thing |
|---|---|---|
| How are tenants isolated? | Three tiers on the same code: pooled (shared tables, row-level security), dedicated schema, dedicated cell. Every row, log line, metric and cost record carries tenant_id. | A database per customer is simple to reason about and miserable to operate at a hundred customers: migrations, connection pools, backups and upgrades all multiply. Row-level isolation with a tier escape hatch keeps the common case cheap and the demanding case possible. |
| How does one build serve many domains? | Hostname resolves to a tenant at the edge; branding, language and data follow from the tenant, not from the build. Customer-owned domains get certificates issued automatically. | Building a copy per domain means a copy per domain to patch. We already proved the principle on the marketing site this week: one build, self-canonical on every host. |
| Microservices? | Not yet. One modular monolith plus asynchronous workers, with module boundaries enforced in code. Each service is extracted later against a named pressure, not a diagram. | Microservices trade a compile error for a production incident. With one team and no traffic yet, that trade is a loss. The boundaries still matter, so we draw them now and keep them honest inside one process. |
| Tier | Who | Database | Vector index | Model calls | Blast radius |
|---|---|---|---|---|---|
| Pooled default | Mid-market, 300–5,000 users | Shared tables in the cell database, row-level security on tenant_id | Shared index, one namespace per tenant and scope | Platform keys, region-pinned | Cell |
| Dedicated schema | Large or regulated customers | Own schema, own connection role, in the same cluster | Own index | Platform or customer keys | Schema |
| Dedicated cell | Single-tenant and private-cloud contracts | Own cluster in a dedicated cell, optionally in the customer's account | Own cluster | Customer keys and endpoint | Customer only |
The tier is a row in the tenant registry. The application does not branch on it: the same code opens a connection whose role and search path were chosen at connection time, and writes to an index whose namespace came from the tenant context. Moving a tenant up a tier is a data migration, not a rewrite.
Enforced, not documented. A test walks the schema and fails the build if any table lacks a tenant column and a row-level security policy. A second test runs the common queries as tenant A while tenant B's rows exist, and fails if a single row leaks. Background jobs run inside the same context: a job without a tenant cannot be scheduled.
| Kind of domain | Example | Certificate | Resolves to |
|---|---|---|---|
| Platform domain | enterprise.flyer.vn | Ours | Marketing site and demo tenant |
| Tenant subdomain | acme.flyerenterprise.com | Wildcard, ours | That tenant, immediately on signup |
| Customer domain | learn.acme.com | Issued automatically once the customer points a CNAME at us | That tenant, with their branding |
Resolution is a lookup, not a build step. The edge holds a cached map of hostname to tenant and cell; the shell is served from the same artefact for every customer, and branding — logo, theme colour, default language — arrives as tenant settings. The prototype already carries that model in the product: a company uploads a logo, the palette is derived from it, and the interface language is a tenant default.
training.flyer.vn and enterprise.flyer.vn. Absolute URLs — canonical, hreflang, Open Graph, structured data, sitemap and llms.txt — are rewritten to the requesting host at the edge, and in-page links are root-relative, so every domain is self-canonical without a rebuild. Adding a third domain is attaching it to the project.Search-engine rule for tenant domains. Customer domains and tenant subdomains are marked noindex and excluded from the sitemap. Exactly one public marketing domain is indexed; the others are private front doors that happen to be on the internet.
One API deployable, one worker deployable, one edge layer. Inside the API, modules own their tables and talk to each other through published interfaces, not through each other's data. This is the discipline that makes a later extraction a refactor rather than an excavation.
| Module | Owns | Depends on | Extract later? |
|---|---|---|---|
| Identity & tenancy | Tenants, domains, users, groups, roles, SSO and SCIM config | — | Becomes part of the control plane |
| Catalogue & content | Courses, lessons, versions, media references, SCORM packages | Identity | Unlikely |
| Assignment & compliance | Assignments, rules, recurrence, certifications, attestations | Catalogue, Identity | Unlikely |
| Player & learning records | Attempts, scores, xAPI statements | Catalogue | Candidate write volume |
| Assistant | Assistants, scopes, conversations, citations, feedback | Knowledge, Identity, AI gateway | Candidate latency and scaling |
| Knowledge ingestion | Connectors, documents, chunks, ACL mirror, index state | Identity | First to go |
| Course Studio | Drafts, review state, source links | Knowledge, Catalogue, AI gateway | Later |
| Skills & analytics | Skills, assessments, gap signals, report definitions | Records, Assistant | Candidate read patterns |
| Notifications | Templates, delivery log, digests | Events | Later |
| Metering & billing | Active users, AI usage, quotas, invoices | Events | Control plane |
learning.completed, assignment.assigned, document.indexed, document.purged, question.answered, gap.detected, course.published, user.deprovisioned. Skills, notifications, analytics and metering are all consumers, which is why they can be extracted quietly later.| Service | Extract when | Why it is first or last |
|---|---|---|
| Knowledge ingestion | Crawl work competes with request traffic, or a connector needs a runtime we do not want in the API | Already asynchronous, owns its tables, talks in events. The cheapest extraction and the most likely pressure. |
| Assistant / retrieval | Answer latency is dominated by our own queueing, or AI traffic scales on a different curve from page traffic | Latency-sensitive and bursty; deserves its own autoscaling long before anything else does. |
| Analytics & reporting | Reports interfere with transactional load beyond what a read replica absorbs | Solve with a replica and a warehouse export first. Extraction is the second answer, not the first. |
| Learning records | Statement volume dominates database writes | Append-only and easy to move, but only worth moving at real scale. |
| Control plane | Immediately — it is global while everything else is cell-local | Different lifecycle, different blast radius, different availability requirement. This is the one genuine service on day one. |
The assistants are the reason a customer's security team is in the room, so this layer is designed to be inspected. Everything a model saw and everything it was not allowed to see is recorded on the request path.
| Concern | Mechanism |
|---|---|
| Bring your own model | The gateway is the only component that knows a provider exists. Azure OpenAI, Bedrock, Vertex, or a customer-hosted endpoint are configuration on the tenant, including their own keys held in the cell's secret store. |
| Quality does not regress silently | A golden-question set per assistant, seeded at onboarding and grown from real unanswered questions. Any prompt, model or retrieval change runs against it in CI; a drop in grounded rate blocks the release. |
| Cost is attributable | Tokens are metered per tenant, per assistant, per feature. The same counter drives the customer's usage console, the quota, and our margin dashboard — one number, three audiences. |
| Embeddings stay home | Embedding and re-ranking models run inside the cell, so document text never crosses a region boundary even during indexing. |
| Human oversight | Anything that evaluates a person — scored practice, skills inference — is behind a human-in-the-loop switch that is on by default, with model cards and logs kept for the EU AI Act file. |
A cell is a complete, independent copy of the platform in one region: compute, database, index, storage, queue. Cells do not talk to each other. This is what makes "your data stays in Frankfurt" a property of the topology rather than a promise in a policy document.
| Cell | Region | Default model endpoint | Serves |
|---|---|---|---|
| EU | Frankfurt, recovery in Dublin | Azure OpenAI EU West | European customers, the strictest residency requirements |
| APAC | Singapore, recovery in Sydney | Azure OpenAI or Bedrock, Singapore | Southeast Asia, Japan, Australia |
| US | Virginia, recovery in Oregon | Azure OpenAI East US | North America |
| Dedicated | Customer's own account | Customer's endpoint | Single-tenant contracts |
Moving a tenant between cells is a supported operation, not an incident: freeze writes, snapshot and ship the schema and objects, re-index in the destination, repoint the domain map, verify with the tenant's own golden questions, then release. Measured in hours, rehearsed before it is sold.
What the control plane may hold is deliberately dull: which tenants exist, which cell and tier they are on, which domains point at them, what they are entitled to, and how much they have used. If the control plane is unavailable, existing sessions keep working in each cell; only signup, domain changes and plan changes pause.
| Concern | Approach |
|---|---|
| Observability | Tenant and cell on every log line, span and metric. Dashboards answer "is this customer having a bad time" before they answer "is the fleet healthy", because that is the question support actually receives. |
| Quotas and rate limits | Enforced at the gateway, visible to the customer, with soft warnings before hard stops. |
| Background work | One scheduler, jobs scoped to a tenant, fair queueing, retries with backoff, a dead-letter queue that a human reads. Recurring compliance assignments and connector syncs are the two big producers. |
| Feature flags | Per tenant and per cell, so a feature reaches one design partner before a region, and a flag can be turned off without a deploy. |
| Cost attribution | Infrastructure and model cost per tenant, tracked from the first month, because unit economics discovered late are unit economics discovered painfully. |
| Back office | Provisioning, tier changes, cell moves, impersonation, evidence-pack export. Boring internal tooling, built early, saves an engineer from being an operations department. |
| Target | Value | Note |
|---|---|---|
| Platform availability | 99.9% monthly | Per cell, measured per tenant |
| Assistant availability | 99.5% | Lower because it depends on model providers; failover to a second provider in-region |
| Time to first token | ≤ 1.5 s p95 | Budget: edge and auth 30 ms, permission filter and retrieval 200 ms, re-rank 80 ms, model 900 ms, overhead the rest |
| Page performance | LCP ≤ 2.0 s | Mid-range laptop, cold cache |
| Scale per cell | 200 tenants · 200k users | 10k concurrent learners, 100 questions per second burst |
| Scale per tenant | 50k users · 1M documents | Above this, dedicated tier |
| Index freshness | ≤ 15 min | Source change to citable answer |
| Recovery | RPO 15 min · RTO 4 h | Per cell, rehearsed quarterly |
| Phase | Build | Done when |
|---|---|---|
| 0 · Foundations | Tenant registry and domain map, token shape, row-level security with the isolation test suite, one cell in the EU, the modular monolith skeleton with two real modules | Two tenants coexist on two domains with a passing isolation suite |
| 1 · Product on the platform | Learning core, compliance engine, four connectors with ACL mirroring, the assistant and the model gateway, audit trail, back office | A design partner runs real training and real assistants in production |
| 2 · Scale out | APAC and US cells, custom domains with automatic certificates, ingestion extracted to its own service, skills and analytics, warehouse export | Customers in three regions; a cell move rehearsed end to end |
| 3 · Enterprise | Dedicated tier and private-cloud deployment, customer-managed keys, 21 CFR Part 11, extended enterprise portals, MCP server | First single-tenant contract in production |
| Decision | Chosen | Rejected, and why |
|---|---|---|
| Service topology | Modular monolith plus workers, with named extraction triggers | Microservices from day one — multiplies operational surface before there is traffic to justify it, and freezes boundaries we have not learned yet |
| Tenant isolation | Shared tables with row-level security, tiers for stricter needs | A database per tenant — operationally heavy at a hundred customers. A single shared table with an application-level filter — one forgotten predicate is a breach |
| Regions | Independent cells, no cross-cell traffic | One global database with regional replicas — makes residency a policy question instead of a physical fact |
| Domains | One build, hostname resolves to tenant, certificates issued automatically | A deployment per domain — every domain becomes something to patch and something to forget |
| Model access | A gateway owning every provider call | Providers called from feature code — makes bring-your-own-model, failover, metering and zero retention impossible to guarantee |
| Search | Vector plus keyword with permissions applied before ranking | Vector-only, or filtering after ranking — the second is how permission-aware systems leak |
| Events | Outbox in the writing transaction, relayed to the bus | Publishing directly from application code — loses events precisely when something is already wrong |
| Risk | Mitigation |
|---|---|
| Connector maintenance is a treadmill — every source system changes its API and its permission model | Four connectors at launch, each with its own permission test suite; evaluate a unified connector vendor against building in-house before the fifth |
| Model cost per question erodes the included allowance | Metering from day one, small models for simple questions, answer caching, provider volume pricing, per-tenant cost visible internally from the first month |
| The isolation suite gives false confidence | Add a case with every new table, and commission an external penetration test focused on cross-tenant access before the first enterprise contract |
| Premature extraction turns into a distributed monolith | The trigger table in section 5 is the gate; an extraction proposal without a named pressure is declined |
| Region moves are sold before they are rehearsed | Rehearse on the demo tenant every quarter and record the elapsed time |
Companion documents: the product PRD for scope, positioning and pricing, and the working prototype for the interface this architecture has to serve.