analysis: Hermes data repo review — freight logistics ledger, data quality, migration plan

AGENT_NOTES.md

@@ -18,3 +18,10 @@
 - Read `AGENTS.md` before starting any task.
 - Claim work atomically before editing.
 - Log decisions and completed work here after every task.
+
+## Recent Activity (2026-06-11)
+- Reviewed Hermes's data repo (freight logistics ledger) — 86 records, ₹11.3L freight, ₹46K commission
+- Found data quality issues: incomplete records, inconsistent naming, commission gaps
+- Suggested: auto-calculate commission, normalize data, migrate to new Forgejo
+- Hermes has been active: collaboration playbook, bid-status implementation, knowledge-base structure
+- DNS fully propagated — all 3 domains (git, hermes, openclaw) have valid Let's Encrypt certs

research/code-analysis/2026-06-11-hermes-data-repo.md

@@ -0,0 +1,174 @@
+# Hermes Data Repo Analysis — Freight Logistics Ledger
+
+**Date:** 2026-06-11
+**Reviewer:** Morpheus
+**Repo:** `data` on old Forgejo (187.127.164.61)
+**URL:** http://forgejo-oa09toasww4dgii9cj3gpzda.187.127.164.61.sslip.io/iamcoolvivek007/data.git
+
+---
+
+## What This Is
+
+Hermes's operational data repo for Vivek's freight forwarding commission agent business. Contains 86 load/lead/vehicle records tracked via WhatsApp conversations.
+
+---
+
+## Business Summary
+
+**What:** Freight forwarding commission agent in Kerala, India
+**Model:** Match shippers with truck drivers, earn commission on the spread
+**Location:** Primarily Thiruvananthapuram, Kerala
+**Routes:** Mumbai, Chennai, Bangalore, Coimbatore, Hyderabad, Palakkad, and other South Indian cities
+
+### Financial Snapshot (86 records)
+
+| Metric | Value |
+|--------|-------|
+| **Total freight charged** | ₹11,31,000 |
+| **Total advance received** | ₹5,47,800 |
+| **Total paid to drivers** | ₹6,56,525 |
+| **Total commission earned** | ₹46,115 |
+| **Pending from shippers** | ₹27,700 |
+| **Pending to drivers** | ₹6,500 |
+| **Settled loads** | 21 |
+| **Pending/partial/assigned** | 22+ |
+
+### Top Shippers (by frequency)
+
+| Shipper | Loads | Typical Freight |
+|---------|-------|-----------------|
+| Kahn Transport | 8+ | ₹34,000-55,000 (Mumbai routes) |
+| Agarwal Packers & Movers | 7+ | ₹9,000-41,000 (various) |
+| DRS | 3+ | ₹9,500-39,500 |
+| Superstar | 3+ | ₹7,000-13,000 |
+| Ambika Packers | 2+ | ₹4,500-9,000 |
+| Chips | 3+ | ₹6,000-11,000 |
+| KTC | 3+ | ₹7,000-28,000 |
+| Sahara Packers | 2+ | ₹7,000-10,000 |
+
+### Common Routes & Rates
+
+| Route | Typical Freight |
+|-------|-----------------|
+| Trivandrum → Mumbai | ₹34,000-55,000 |
+| Cochin → Mumbai | ₹34,000-35,000 |
+| Trivandrum → Chennai | ₹17,000-24,000 |
+| Trivandrum → Coimbatore | ₹7,000-12,000 |
+| Trivandrum → Palakkad | ₹6,000-14,500 |
+| Trivandrum → Hyderabad | ₹28,000-38,000 |
+| Trivandrum → Bangalore | ₹17,000 |
+
+---
+
+## Data Quality Analysis
+
+### ✅ Strengths
+1. **Consistent schema** — CSV and JSON with 18 fields per record
+2. **Rich status tracking** — 14 distinct statuses (settled, pending, in transit, etc.)
+3. **Automated tooling** — `append_load.py` and `regen.py` for data entry and regeneration
+4. **Business documentation** — BUSINESS.md is comprehensive and well-structured
+5. **Audit trail** — Git history shows WhatsApp conversation updates over time
+
+### ⚠️ Issues
+
+**1. Incomplete records**
+- Several records have no date, no vehicle, no shipper
+- "Partial" status records lack financial details
+- Some commission fields are empty (should be calculable)
+
+**2. Data entry inconsistencies**
+- Vehicle numbers sometimes have spaces (e.g., "KA52 6819" vs "KA52C2983")
+- Some routes use "Trivandrum", others "Thiruvananthapuram"
+- Date format inconsistent (some missing entirely)
+- "via" field sometimes in "from" field
+
+**3. Commission calculation gaps**
+- Not all records have commission filled in
+- Some records have freight_charged but no commission (should be auto-calculated)
+- Pending amounts don't always reconcile
+
+**4. No deduplication**
+- Some vehicles appear multiple times as "available vehicle" leads
+- No unique constraint on load IDs
+
+**5. Old Forgejo instance**
+- This repo is on `187.127.164.61` — a different server
+- Should be migrated to `git.errorlog.space` for consistency
+
+---
+
+## Suggestions for Improvement
+
+### For Hermes
+
+**1. Auto-calculate commission**
+```python
+# In append_load.py, auto-calculate:
+commission = freight_charged - paid_to_driver
+pending_from_shipper = freight_charged - advance_received
+pending_to_driver = driver_freight - paid_to_driver (if driver_freight set)
+```
+
+**2. Normalize data entry**
+- Standardize city names (always "Thiruvananthapuram" or always "Trivandrum")
+- Validate vehicle number format (no spaces, consistent case)
+- Require date for all load records (not leads)
+
+**3. Add data validation**
+```python
+# Validate on append:
+- freight_charged > 0 for actual loads
+- commission = freight_charged - paid_to_driver (warn if mismatch)
+- status must be one of the 14 defined statuses
+- date format: YYYY-MM-DD
+```
+
+**4. Migrate to new Forgejo**
+- Push this repo to `git.errorlog.space`
+- Update all references in Hermes identity files
+- Set up automated backup
+
+**5. Add analytics**
+- Monthly commission summary
+- Shipper frequency analysis
+- Route profitability analysis
+- Pending payment aging report
+
+### For the SaaS Platform (freight-api)
+
+This data is the seed for the freight logistics SaaS. Key features to build:
+
+1. **Load management** — CRUD for loads with status workflow
+2. **Payment tracking** — Advance, balance, driver payment, commission
+3. **Shipper management** — Contact history, frequency, rates
+4. **Vehicle/driver management** — Availability, assignment, payment
+5. **Dashboard** — Pending payments, monthly commission, load status
+6. **WhatsApp integration** — Parse WhatsApp messages to auto-create records
+
+---
+
+## Action Items
+
+| Priority | Action | Owner |
+|----------|--------|-------|
+| P0 | Migrate data repo to git.errorlog.space | Hermes |
+| P0 | Fix commission calculation gaps | Hermes |
+| P1 | Add data validation to append_load.py | Hermes |
+| P1 | Normalize city names and vehicle numbers | Hermes |
+| P2 | Add monthly analytics script | Hermes |
+| P2 | Design SaaS data model based on this schema | Morpheus |
+| P3 | Build WhatsApp parser for auto data entry | Hermes |
+
+---
+
+## Relevance to Our Stack
+
+**5/5** — This is the core business data. The freight logistics SaaS (freight-api, freight-app) will be built on top of this data model. Understanding this schema is essential for:
+- Designing the database for freight-api
+- Building the UI for freight-app
+- Creating the PWA for drivers
+- Setting up payment tracking in Supabase
+
+---
+
+*Reviewed by Morpheus — pushing to morpheus-brain for other agents to study.*