troubleshooting-scenarios▌
Systematic methodology, plus both written scenarios worked in full — the EC2/Docker networking puzzle and the ECR/CI-CD rollback question.
The methodology, before the specifics
Troubleshooting questions test systematic thinking, not memorized fixes — the same ladder applies whether it's a network problem, a bad deploy, or something neither of you has seen before. Both scenarios below are worked examples of exactly this ladder in action.
What's in this session
| page | covers |
|---|---|
| scenario-1 | EC2/Docker/Nginx unreachable — the full diagnostic ladder |
| scenario-2 | ECR image → production: pre-prod steps, rollout strategy, rollback |
| strategies | rolling vs blue-green vs canary vs shadow, compared |
nginx in a container, listening on port 80, mapped to EC2 host port 8080. Security group allows 8080 inbound. IGW attached. Can't load <ec2-ip>:8080 from outside.
curl localhost:8080 run on the instance only exercises the bottom two layers.
Part a — before touching Docker, two checks
Route table & public IP. An IGW existing in the VPC doesn't mean this specific subnet uses it — confirm the subnet's route table has 0.0.0.0/0 → igw-xxxx, and confirm the instance actually has a public IP or Elastic IP attached. Both are required before a single packet from the internet can reach the ENI at all.
Network ACL. The diagram only confirms the Security Group. NACLs are stateless and subnet-level — even a perfectly correct SG won't help if the NACL doesn't explicitly allow both the inbound request on 8080 and the outbound return traffic on ephemeral ports.
| security group | network acl | |
|---|---|---|
| level | instance / ENI | subnet |
| state | stateful — return traffic auto-allowed | stateless — must allow both directions |
| rules | allow only | allow and deny |
| evaluation | all rules considered | in order, first match wins |
Part b — curl localhost:8080 works, verdict?
Docker and Nginx are proven healthy — that command only exercises the bottom two layers in the stack above, and both clearly work. The problem lives entirely in the path between the internet and the instance's external interface, so I'd re-verify, in order: the Security Group's actual live rule (right port, right protocol — TCP not UDP, right source — 0.0.0.0/0 vs. something narrower that happens to exclude the tester; the diagram says it's fine, but a misread SG rule is one of the most common causes of exactly this symptom, so I'd re-check with fresh eyes rather than trust the diagram), then the Network ACL, then the route table / public IP. If all three are clean, the instance's own OS-level firewall (iptables/firewalld) is next — a rule scoped to the external interface wouldn't touch loopback traffic at all.
"The instinct that actually matters here isn't memorizing the AWS networking stack — it's the isolation technique: curl localhost bisects the problem in one command, proves the app layer clean, and turns a vague 'it's not working' into a specific, narrow search. I'd reach for the same bisection instinct whether it's this or a pod that can't reach a service inside EKS."
a new container image was pushed to ECR. CI/CD checks passed.
Part a — steps before production
Passing CI means the code compiles and unit tests pass — it says nothing about how the image behaves under real conditions. Before production:
Deploy to staging that mirrors production as closely as practical, and run integration/smoke tests against it — not just unit tests.
Pin the exact artifact — deploy by immutable digest or a specific semantic-version tag, never :latest, so what you tested in staging is byte-for-byte what reaches production.
Image scanning (Trivy or equivalent) if it isn't already a CI gate — vulnerabilities in base images slip through even when application code passes every test.
Confirm environment-specific config and secrets are correctly wired for production, not leftover staging values.
Confirm the rollback path before deploying, not after — know the previous approved image is available and redeployable before you need it under pressure.
Part b — a rollout strategy so all users aren't immediately affected
Canary is the direct answer — route a small slice of real traffic (5%, say) to the new version, watch error rate/latency/business metrics against a baseline, then ramp gradually. A rolling update is the close Kubernetes-native cousin — it also avoids an instant full cutover, just without canary's fine-grained percentage control. Full comparison is on the next tab.
Part c — a deployment has gone bad, rollback now
Immediate: shift traffic back to the previous version. If it's canary/blue-green, that's just flipping the routing weight back to 0% new / 100% old. If it's a Kubernetes rolling deployment, kubectl rollout undo deployment/<name> reverts to the last ReplicaSet directly. Either way, this depends on the previous approved image still being available and deployable — which is exactly why keeping it matters.
Q: why is storing a previous approved image important?
A: rollback needs a known-good, already-tested artifact ready to redeploy immediately. Rebuilding from source under incident pressure is slow, and risks pulling in new, untested changes — a moved :latest tag, an updated base image, a dependency bump — at exactly the moment you need certainty most. Same reasoning as pinning exact digests in part a.
Once traffic is back on the known-good version and metrics confirm recovery: root-cause the bad deployment before trying again — don't just roll back and quietly retry the same image.
"For a fraud-scoring service specifically, I'd treat the rollback threshold as a business metric, not just an infra one — a spike in 500s is obvious, but a quiet shift in the false-positive or false-negative rate on live transactions is the one that actually costs money, and canary is what gives you the window to catch that before it's at 100% of traffic."
four patterns worth knowing cold — they differ mainly in who gets exposed to the new version, and how fast you can undo it.
| strategy | who sees the new version | rollback | notes |
|---|---|---|---|
| rolling | gradually, all eventually | slower — pods replaced back one by one, or rollout undo | k8s default; no extra infra needed |
| blue-green | nobody, then everybody — single cutover | fastest — flip traffic back to "blue" | needs 2x infra running at once |
| canary | a small %, growing over time | fast — reduce the % back to 0 | needs traffic-splitting — weighted target groups, ingress, or a mesh |
| shadow | nobody — traffic is mirrored, responses aren't returned to users | n/a — it was never serving real responses | zero user risk; pure validation against real load |
Q: what is a shadow deployment?
A: the new version receives a mirrored copy of live production traffic and processes it in parallel — but its output is never returned to real users; only the current version's response is what they actually see. It validates real-world behavior and load-handling with zero user-facing risk, before the new version ever serves a single real response. Different from canary, which does serve real users — just a limited slice of them.
"For a fraud-scoring model specifically, I'd sequence shadow before canary — shadow to validate the new model's decisions against real transaction volume with zero risk of a bad decline, then canary once I trust its behavior, watching false-positive and false-negative rates before scaling past a small slice of traffic."
seven tasks — a mix of hands-on practice and verbal-answer rehearsal. Progress saves automatically.
loading progress…