Sanad AI: grounded Arabic and English answers over Twiscope reports
Twiscope's retrieval-augmented assistant answers questions about social-media analytics reports in Arabic or English, with the sources it used. I lead the team that builds it and personally own its ingestion pipeline, deployment, and tracing; this study is explicit about which parts were mine.
- Chat scopes
- 3
- Report types routable
- 7
- Hybrid retrieval
- BM25 + kNN
- Question languages
- AR / EN
The problem
What needed solving
Twiscope produces analytics reports on X keywords and accounts, TikTok keywords, accounts and hashtags, and Instagram hashtags and accounts. Analysts wanted to ask questions about those reports — “which accounts drove the most engagement?”, “how did this hashtag compare with last quarter?” — in Arabic or English, instead of reading dashboards.
A generic chatbot was not acceptable: answers had to be grounded in the report's own data, carry the numbers the report actually contains, compare several reports when asked, and only ever use reports the user's subscription allows.
Constraints
What the design had to respect
- Questions arrive in Modern Standard Arabic, Gulf and Egyptian colloquial Arabic, English, or a mix.
- Access control must be deterministic: subscription entitlements and report scoping are enforced by code, never delegated to a language model.
- The Elasticsearch cluster targeted versions older than 8.14, so native reciprocal rank fusion was not available.
- Reports keep arriving from the Twiscope platform; ingestion has to be idempotent and able to resume a half-processed report.
- Every extra model call adds latency and cost, so the number of LLM calls per question had to stay small and bounded.
Architecture
How the system fits together
A FastAPI service behind Nginx answers chat requests synchronously with async I/O; a separate worker ingests reports from SQS into Elasticsearch. Retrieval is hybrid (lexical and vector) and the language model only sees documents that code has already scoped to the caller's reports.
Request path
Twiscope web app
Calls the assistant with the user's JWT; questions in Arabic or English.
FastAPI API
RS256 JWT verification, then an entitlement check against the subscription service before any retrieval.
Intent classifier
Rules first (greetings, small talk, abuse), an LLM only for ambiguous questions.
LLM index router
Suggests which indices to search; code intersects the answer with a per-platform allow-list.
Retrieval
BGE-M3 embeddings
1024-dimension dense vectors from a self-hosted Text Embeddings Inference server.
BM25 + kNN per index
Two queries per index run in parallel, each filtered to the authorised report and platform.
Reciprocal rank fusion
Client-side, unweighted, k = 60, de-duplicated by document id.
Context builder
Per-report quotas and a character budget; reports with no matches are named explicitly.
Generation
OpenAI GPT models via OpenRouter
One grounded answer per question; temperature 0.3, non-streaming.
Response
Sanitised answer, the source documents used, per-stage timings, and token counts.
Ingestion and operations
SQS worker
Reads a report from PostgreSQL, transforms it per platform, embeds, bulk-indexes, and writes a bilingual summary.
Processing traces
Every stage of every report is recorded, so a failed report resumes from the stage that failed.
Elastic APM
Tracing across the API and both queue workers, plus request and error logs.
Deployment
GitHub Actions on a self-hosted runner: backup, restart, health check, rollback, then a fresh machine image.
Engineering decisions
What was chosen, and why
The model chooses where to look; code decides what may be seen
The LLM router only returns index names. Code intersects them with an allow-list for the platforms in the request, ignores any other hints, and falls back to fixed defaults if the output is empty or invalid. Report and platform filters are built in Python and injected into every Elasticsearch query, and the entitlement check runs before retrieval. A prompt-injected question can change which eligible index is searched, not which reports are visible.
Hybrid retrieval with client-side reciprocal rank fusion
BM25 catches exact Arabic terms, hashtags and handles that embeddings blur; kNN over BGE-M3 catches paraphrases and cross-language matches. Each index gets both queries in parallel and the ranked lists are fused with RRF (k = 60) in the service, because the target cluster predates native RRF. If one sub-query fails, the other list is used on its own.
Always ground on report aggregates
Report-level aggregates and summaries are added to every search regardless of routing, and prompts name each platform's report-total field. Without this, the model treated the handful of retrieved posts as the whole report and produced wrong totals.
Rules before the classifier model
A rule-based first stage answers greetings, small talk and abusive input without retrieval, normalising Arabic text (diacritics, alef and taa marbuta variants) before matching. Only questions the rules can't place pay for an LLM classification call.
Idempotent, resumable ingestion
Document ids are deterministic and scoped to the report, so re-indexing overwrites instead of duplicating. A processing trace per report lets the worker skip completed stages. Transient failures leave the message on the queue for redelivery; malformed or permanently invalid messages are removed.
Degrade rather than fail
If an embedding batch fails during ingestion, those documents are indexed without vectors and stay searchable through BM25. Classifier and router failures fall back to defaults instead of failing the request.
Trade-offs
What those choices cost
LLM index routing
One extra model call per question (latency and spend), bounded by deterministic fallbacks.
Client-side RRF
Two Elasticsearch round-trips per index and a fixed candidate depth of 20 per list; fusion is unweighted.
English-only summary embeddings
One vector space for all questions, but Arabic summary text isn't searched directly.
30-second entitlement cache
Fewer calls to the subscription service; a revoked grant can stay usable for up to 30 seconds per process, and an outage of that service blocks chat.
Comparisons written by the model
Rankings and differences across reports come from the model following strict prompt rules over retrieved numbers, not from deterministic code.
Outcomes
What it delivered
Three chat scopes
Single report, a user's own reports (up to 1,000 per request), and an entitlement-gated all-reports mode — each with its own retrieval strategy.
Answers that show their sources
Every answer returns the documents it used, per-stage timings and token counts, so quality problems can be traced to retrieval or generation.
Seven report types
X keyword and account, TikTok keyword, account and hashtag, Instagram hashtag and account, each with its own eligible indices and ranking fields.
Repeatable delivery
Pushes to the production branch deploy through GitHub Actions with backup, health check and rollback, then bake a new machine image.
My role
- Started the FastAPI service and repository and set its structure.
- Built the first SQS ingestion worker and the report-processing pipeline from PostgreSQL to Elasticsearch, including the X keyword and account query builders.
- Implemented RS256 JWT verification for the API.
- Built the deployment workflows (GitHub Actions → systemd, with backup and rollback, then machine-image refresh).
- Added Elastic APM tracing to the API and both queue workers.
- Built a separate GPU sentiment service for Arabic dialects (three Hugging Face models, batched fp16 inference); it is deployed alongside Sanad but not used by chat.
- As team lead and repo owner, reviewed and merged the team's work, including the router and hybrid-retrieval changes. The retrieval core — classifier, LLM router, hybrid search and RRF, prompts, and the embedding and LLM clients — was designed and written by two teammates.
Scope notes
- No latency, accuracy, cost or traffic numbers are published here; none are measured in a form I can share.
- Answers are not streamed, and the system is not real-time: reports are ingested asynchronously.
- Observability is Elastic APM plus logs in Elasticsearch; there is no Prometheus or Grafana for this service.
- The repository has a large unit-test suite that CI does not yet run. Next improvements: run tests in CI, tighten ingestion error handling and queue visibility windows, and bound fan-out for very large multi-report requests.
Next case study
Fendix
An open-source Go CLI and a hosted platform that turn SAST, DAST, SCA and imported SARIF results into confidence-scored findings and a coverage-aware release decision. I created it and maintain it on my own.
Read it