The story
Context
A church gate on Sunday morning, a shop counter at closing time, a school office at the end of term. These are the places our software is actually used — and none of them look like an office with fast Wi-Fi.
The problem
Three failure modes show up again and again:
- The network drops exactly when many people need to be recorded at once.
- Several people share one device, with different levels of trust.
- Money moves through retries, webhooks and cash — and must still add up.
Constraints
- Mid-range Android phones and intermittent mobile data.
- Operators with varied literacy and little time.
- Mobile Money and cash rather than cards.
Patterns we reuse
- Queue the work, not the user: Asori’s attendance marks queue on the device with a visible pending count and sync when the connection returns, rechecking permission first.
- Local first where it matters: Estorr saves every sale, stock movement and expense to a local database first, then syncs in order through an outbox.
- Reserve, then charge: Asori’s SMS wallet reserves credit, dispatches from the server and charges only for messages the carrier accepts.
- Idempotent money: payment and settlement handlers can safely receive the same event twice.
- Never erase money: Estorr voids sales with opposite entries instead of deleting them.
- Shared devices, personal accountability: Estorr unlocks a shared shop phone per person with an offline PIN.
Architecture
The server remains the authority. Clients can be optimistic and offline, but totals are re-derived, permissions are rechecked and anomalies are flagged on the server — by scheduled jobs where needed (absentee computation after services, stock checks, daily summaries, fee sweeps).
Design decisions
- Be honest about “offline”: we queue specific actions (attendance, sales), and label everything else as needing a connection.
- Show the cost before the action: SMS previews equal the bill before anything is sent.
- Big targets, few words: Estorr’s home screen has four actions with pictures.
Implementation
These patterns are live in Asori (offline gate attendance, the SMS wallet and its payment webhooks) and built into Estorr (local-first selling, append-only ledgers and shared-device unlock).
Outcome
Ushers keep checking people in when the network fails, administrators see what an SMS campaign costs before they send it, and a shop’s cash position can be trusted at closing time.
What we learned
Reliability is mostly about deciding, action by action, what may happen without the server — and making everything else fail clearly instead of silently.