Research
Validation & known limitations
How well does TripLibre reconstruct a day? This page separates three kinds of evidence: what has been observed in everyday use, what the automated tests guarantee, and the formal benchmarking that has not been done yet.
All accuracy numbers so far come from one person's phone over 17 days, scored against that person's own corrected diary. They describe that sample, not TripLibre in general. Please don't cite them as a general accuracy figure.
Three kinds of evidence
| Evidence | What it shows | What it doesn't | Status |
|---|---|---|---|
| Everyday use | Which situations the app handles and which it gets wrong, seen in the developer's daily use since development began in late September 2026. | Rates. These are observations, not measurements. | Available |
| Automated tests | The engine behaves as specified on synthetic days, every time, and edits survive rebuilds. | How often real days look like the synthetic ones. | Available |
| Personal calibration study | Agreement with a hand-corrected diary on 17 real days, with held-out validation days. | Other people, other phones, other cities. | Available, small sample |
| Formal benchmark | Accuracy, completeness and battery use across participants and devices, against independent ground truth. | — | Planned |
Observed in real-world use
Situations the developer has lived through and checked in the diary, mostly in Philadelphia:
| Scenario | Usually works | Known to go wrong |
|---|---|---|
| Walking between buildings | Walks over a few hundred metres become their own trip. | Buildings under about 100 m apart, with no fix in between, merge into one stay. |
| Indoor GPS drift | Ordinary drift at home or the office is filtered out. | Large venues can drift hundreds of metres and produce duplicate stays 200–500 m apart. |
| Subway | Inferred from a signal blackout with vehicle motion, or from coarse fixes at the two stations. | Fast above-ground rail with clean GPS can read as car. |
| Bus | Usually right at group level (transit). | Often labelled subway or car: bus and road traffic look the same without transit schedules. |
| Rideshare / taxi | Read as car (taxi is never predicted; it counts as car at group level). | Before round 2, periodic GPS holes made many of these look like subway. |
| Short stops | Stops of five minutes or more become visits; shorter ones can be stamped. | Unstamped short stops are missed. |
Automated tests
Run on every push and pull request (.github/workflows/ci.yml). None of them use real location data.
- Core suite (
make test): 112 Swift tests in 11 suites, covering inference, edits, reprocessing, export, analytics, stamps and widgets. Fourteen of them are the synthetic scenarios from the original product brief: home–work–home, a trip chain, a walking trip, subway, car, a multimodal trip, a brief stop that must not become a visit, a long stationary period, GPS drift while stationary, a missing GPS interval, an overnight visit, crossing midnight, airport–flight–airport, and several days without opening the app. Others cover daylight-saving time and check that incremental processing equals a one-shot rebuild. - Calibration regressions (
CalibrationTests): one synthetic fixture per failure class found on real data. Each fails on the engine version before the fix. - Export contract (
make export-check): Swift writes a real export, Python loads it, and the CSV and SQLite versions must agree. - Replay determinism (
make calibrate-smoke): a synthetic fixture replayed twice must give byte-identical output, then the full evaluation pipeline runs on it. - UI tests (
make ui-test): core flows and accessibility audits in the simulator, on demo data.
Personal calibration study
Issue #46 replayed one phone's raw data through the engine
and compared it with the same person's corrected diary. The days were split into calibration days (for tuning) and
validation days (scored once) before any tuning. The full protocol, metrics and adverse tradeoffs are in
docs/research/inference-calibration.md.
| Metric (round 1, 5 validation days) | Before | After |
|---|---|---|
| Whole-day time agreement | 0.827 | 0.939 |
| Visit F1 | 0.85 | 0.86 |
| Trip F1 | 0.86 | 0.86 |
| Visit departure error, mean absolute | 32.0 min | 5.6 min |
Round 2 tuned mode detection. On the 17 calibration days (in-sample), group-level mode accuracy on human-labelled trips rose from 0.51 to 0.82; exact mode accuracy rose from 0.39 to 0.54. The held-out check for round 2 is still pending, because no new days had been recorded after the cutoff.
The study publishes aggregates only. The raw data and per-day reports stay on the developer's machine.
Planned benchmarking
Planned Nothing in this section has been done yet. It defines what a formal evaluation should report, so that results can be compared across versions and studies.
Metrics
| Area | Metric |
|---|---|
| Stays and trips | Precision, recall and F1 under the one-to-one matching rules of the calibration study; split and merge counts. |
| Timing | Signed and absolute arrival and departure error (median, mean, p90). |
| Whole day | Share of minutes where the diary and the truth agree on "at a place" versus "travelling". |
| Mode | Exact and group-level accuracy, and a confusion matrix, on independently labelled trips. |
| Completeness | Share of each day with collection running; days with gaps over 15 minutes; trips flagged has_gap. |
| Battery | Battery drain per day attributable to TripLibre, by collection profile and device model. |
| Correction burden | Edits per person-day and time spent correcting. |
Ground truth
The calibration study used the user's own corrections as truth, which risks agreeing with itself. A formal benchmark should use an independent reference: a researcher-kept log, a scripted route with timestamps, or a second device. It should also report uncertainty (for example, day-level bootstrap intervals) and keep a held-out set.
Reproducible fixtures
Real GPS logs can't be shared without exposing the people in them. Planned fixtures are synthetic or carefully
de-identified (shifted, rotated, time-offset) traces that reproduce a failure class, stored as SQLite files that
hht-calibrate replay can read. See proposing an
inference test case.
Known limitations
- Unobservable stops. A stop with no GPS fix at all can't be detected. The user has to add or stamp it.
- Walks during GPS silence. When the collector is quiet, a walk can be absorbed into a stay or the next trip. Arrival after a silent walk is dated from the next fix.
- Collector holes while driving. Recurring 4–6 minute silences while in a vehicle make routes and distances straight lines across the gaps.
- Nearby buildings merge, and large venues duplicate stays, as described above.
- Mode is the weakest part. Taxi, bus, light rail and commuter rail are never predicted directly; driver versus passenger is never inferred.
- The engine's first answer is lost when you edit. Records are updated in place, so the app's real-world accuracy can only be estimated by replay, not read from use. Keeping the original is Planned.
- No engine version field. A replay is pinned by git commit and configuration, not by a version stored with each record.
- iOS only, and background behaviour depends on iOS power management, which varies by device and iOS version.