fraud-detection-design▌
Full model answers to all six Pre_Online_Interview_Questions — pure synthesis of sessions 1 through 4, plus your own projects.
This is pure synthesis
Nothing new to learn here — these six questions get answered by assembling sessions 1 through 4, plus the RAG system, the AWS internship, and the NoProp thesis as proof you have actually built something like this before. The JD's "Required Technical Skills" section is written at a noticeably lower bar than its Responsibilities section — that gap is a signal: solid reasoning and the right instincts carry more weight here than exhaustive production depth on every single point.
How to structure any design answer
What's in this session
| page | question |
|---|---|
| pipeline | Q1 — end-to-end ML pipeline |
| hybrid-infra | Q2 & Q3 — connecting and leveraging on-prem + AWS |
| ci/cd | Q4 — CI/CD pipeline |
| monitoring | Q5 — monitoring across on-prem + AWS |
| aws-services | Q6 — name 5 AWS services |
how would you develop an end-to-end ML pipeline for a fraud detection system across millions of transactions?
The opener
"At bKash's scale, I'd design this as two connected pipelines — an offline pipeline that ingests, engineers features, trains, and registers models, and an online pipeline that serves both a synchronous real-time score and an asynchronous enrichment path. Fraud detection cannot be purely synchronous: you need a fast blocking decision at transaction time, but also a slower, richer analysis afterward that does not hold up the payment. Let me walk through the stages."
The two areas worth going deep on
The offline/online feature store split is the detail that separates a working prototype from a production system — if training computes a feature one way and serving computes it slightly differently, you get train/serve skew, and the model's real-world performance quietly diverges from its offline metrics with no obvious error anywhere. I would enforce a single feature definition — ideally the same code path — that both the batch training job and the low-latency online store pull from.
The sync/async split exists because recency and velocity features — transactions in the last 5 minutes, spend against a 30-day baseline — are often the most predictive signals for fraud, and they need a low-latency store like DynamoDB or Redis to compute at scoring time, not a batch warehouse. The synchronous path only needs to be fast and directionally right; the asynchronous path, consumed off the same stream after the transaction completes, can afford heavier models — graph-based fraud rings, ensembles — and feed account holds or investigation queues without ever touching the transaction's own latency budget.
"The serving layer itself I would build close to how I built my RAG system's API — FastAPI behind Docker and Kubernetes, deployed through CI/CD — just with a much stricter latency budget, and this sync/async split that a Q&A system never needed."
Q2 — how would you connect and leverage on-premise infrastructure and AWS cloud for this? Q3 — how would you use on-premise and AWS cloud to accelerate model training and use the system?
Connectivity, briefly — full detail is in Session 4
Direct Connect, paired with a Transit Gateway via a Direct Connect Gateway, is the production path — dedicated bandwidth and consistent latency for repeatedly moving training data and checkpoints, reaching multiple VPCs through one physical connection. The design-answer version is just: name Direct Connect, name Transit Gateway, and say why over a Site-to-Site VPN.
What actually lives where
This is not just a networking question — it is a data placement question. Regulated fintech environments often carry data residency and access constraints on raw transaction data, so I would want to confirm early what is allowed to leave the on-prem boundary versus what has to be aggregated, tokenized, or feature-engineered first. That is a question I would ask explicitly in an interview like this, not assume an answer to.
Accelerating training
On-prem GPU capacity is the steady-state — fixed cost already sunk, no marginal cloud spend, right for routine scheduled retraining. AWS is the burst — spinning up P/G-family instances, Spot where the job tolerates interruption, when a job needs more parallel capacity than on-prem has: a wide hyperparameter sweep, an architecture search, or an urgent retrain that cannot wait for on-prem capacity to free up.
The JD's own mention of SLURM, Ray, or Kubernetes with GPU node pools for on-prem scheduling is the detail worth extending: run the same Kubernetes scheduler across both on-prem and cloud GPU node pools, so submitting a training job looks identical regardless of where it actually lands — the scheduler decides based on current queue depth and cost, not the engineer.
"and use the system" — where serving lives
Inference should sit wherever it can meet the latency budget most reliably under variable load — for bKash specifically, that is AWS: co-located with the rest of the transaction-processing infrastructure, autoscaling for the kind of surge a mobile financial service sees around Eid, when on-prem capacity planned for an average day would fall over. Training is capacity-planned; serving needs to be elastic — that is the actual reason they end up living in different places.
"I would frame this as: on-prem is our sunk, steady-state training capacity; AWS is elastic — both burst capacity for training and the actual home for serving, because serving needs to scale with transaction volume in a way a fixed on-prem fleet cannot."
how would you set up a CI/CD pipeline for this system?
The answer starts with "two pipelines, not one"
Leading with this distinction immediately is the single highest-value thing to say for this question — a standard software CI/CD pipeline (git-triggered: lint, test, build, deploy) is not enough for an ML system, because the model itself needs its own pipeline, triggered by new data or a schedule, not a code change.
| code CI/CD | model/data CI/CD | |
|---|---|---|
| trigger | git push / PR merge | new data arrival, or a schedule (e.g. nightly) |
| steps | lint, unit test, build image, push to ECR | retrain on fresh data, evaluate offline + business metrics |
| gate | tests pass | new model beats current production model on the metrics that matter |
| output | a deployable image | a versioned model in the registry, promoted or rejected |
That comparison-based promotion gate — new model vs. current production model, not new model vs. a fixed target — is the specifically ML part a code pipeline has no equivalent of. It also means a retrain that produces a worse model is a normal, expected, silent outcome, not a failure to alert on.
Where Jenkins fits
A self-hosted CI runner — Jenkins, or a self-hosted GitHub Actions runner — matters specifically because the training step in the model/data pipeline needs to reach the on-prem GPU cluster. A cloud-only CI service has no path to hardware sitting behind your own firewall; a self-hosted runner does.
"This is the same two-pipeline shape I built for the RAG system, just with the model pipeline doing more — retraining against fresh transaction data instead of a mostly-static document corpus, and a promotion gate that has to weigh false-positive cost against false-negative cost, not just accuracy."
how would you set up monitoring across on-prem infra and AWS cloud, and how would you make sure the model is performing as intended?
Two monitoring domains, not one
Infrastructure monitoring answers "is the system healthy" — GPU utilization and temperature on-prem (a DCGM exporter into Prometheus), CloudWatch for EC2/EKS on the AWS side. Model performance monitoring answers a completely different question — "is the model still doing its job" — and that is the one people forget, because a model can be perfectly healthy infrastructure-wise while quietly getting worse at its actual task.
One pane of glass across both environments
Rather than two separate monitoring stacks, I would run Prometheus on both sides and federate — or remote-write — into one central Grafana, so an on-call engineer is not switching dashboards depending on which environment the alert fired in. CloudWatch stays useful specifically for AWS-managed services Prometheus cannot easily scrape — RDS, Lambda, managed Kafka.
Is the model still doing its job
Data drift — is the incoming transaction feature distribution shifting from what the model trained on (PSI, KS-test, or a tool like Evidently AI). Concept drift — is the relationship between features and actual fraud outcome changing, which for fraud specifically usually means fraudsters adapting tactics, not just natural data shift. And a fraud-specific wrinkle: ground truth lags — a chargeback can take days or weeks to confirm, so you cannot wait for "true" accuracy to catch a problem. I would monitor prediction distribution shift and average confidence score as an earlier, if noisier, leading indicator while the real labels catch up.
On top of the statistical signals, the business metrics that actually matter: false positive rate (legitimate transactions wrongly blocked — a direct hit to customer trust), false negative rate (fraud that got through — a direct dollar loss), and dollars saved versus dollars lost as the summary number leadership actually cares about.
Closing the loop
A defined threshold on any of the above — drift score, false-positive rate, confidence distribution — triggers the model/data CI/CD pipeline from the previous question: retrain, evaluate against the promotion gate, and only replace the production model if the new one actually wins.
"I would treat the ground-truth lag as the trickiest part of this specifically for fraud — unlike a lot of ML monitoring problems, you often do not get to know you were wrong for days or weeks, so the leading indicators, drift and confidence shift, are not a nice-to-have here, they are load-bearing."
name 5 AWS services you would use for this system.
Five, each doing a genuinely different job in the pipeline — naming five random services is weaker than naming five that visibly cover the whole system.
That is ingestion, orchestration, low-latency serving, storage, and observability — one service each, not five variations on compute.
this closes out the verbal rehearsal that has been on deck since early in the prep. Seven reps, timed. Progress saves automatically.
The shape of a design-question answer, out loud
Open with three sentences — scope, the core architectural choice, and what you are about to walk through — before any detail. Then roughly 60% of your time on the core reasoning, the rest split between framing up front and tradeoffs at the end. For Q4 and Q6 specifically, lead with the punchline immediately — "two pipelines, not one" / naming the five — rather than building up to it.
Proof over claims
Every answer in this session already has a bKash pivot built in — the thing to add on top, verbally, is a concrete pointer to your own RAG system or the NoProp thesis wherever it is genuinely relevant. "I built something like this" lands very differently from "here is how you would build this."
The honest-unknown move
If a follow-up goes somewhere none of these six answers cover, name the closest thing you do know, state how you would reason toward an answer, and say plainly that you would want to verify rather than guess with false confidence. That is a stronger answer than a fluent guess, and it is the one framework from this whole prep arc worth having completely automatic.
loading progress…