PotholeAI: AI Road Defect Detection & Reporting System
PotholeAI lets a citizen report road damage with a single photo and GPS location, and gives road-maintenance officials an AI-assisted triage dashboard that automatically detects potholes, estimates their real-world severity, avoids duplicate complaints, and ranks every report by urgency - so limited repair budgets go to the most dangerous roads first.
Objective
- Make it effortless for a citizen to report a pothole - one photo, one location, done.
- Automatically detect and count potholes in the photo using computer vision.
- Estimate real physical severity instead of relying on how zoomed-in the photo happens to be.
- Prevent duplicate or spam reports without silently merging genuinely separate complaints.
- Rank every complaint by a single, explainable priority score that factors in traffic, severity, urgency of the citizen's own words, and nearby hospital access.
- Give officials a simple review workflow to approve, reject, or resolve each report.
Tech Stack
- Backend API : FastAPI + Uvicorn
- Database : PostgreSQL with PostGIS, SQLAlchemy (async), GeoAlchemy2
- Frontend : Streamlit
- Detection Model : YOLOv8 (Roboflow inference + HuggingFace hosted model)
- Severity Model : Depth Anything V2 (monocular depth estimation)
- Duplicate Detection : OpenCLIP image embeddings (cosine similarity)
- Geocoding : OpenStreetMap Nominatim API
- Traffic Data : TomTom Traffic Flow API
- Hospital Proximity : OpenStreetMap Overpass API
- LLM : Groq - llama-3.1-8b-instant (urgency scoring & explanations)
Pipeline (Per Complaint)
Image Upload → YOLOv8 Pothole Detection → Depth-Based Severity Estimation → CLIP Embedding → Duplicate Check → Reverse Geocoding → Description Urgency Analysis → Priority Scoring → AI Explanation → Save to Database
Key Features
- One-Photo Complaint Submission : A citizen uploads a single road image with latitude/longitude, an optional description, and an optional email - no account or app install required.
- YOLOv8 Pothole Detection : Runs a pothole-detection model over the image and draws a bounding box and confidence score around every pothole found.
- Depth-Based Severity Scoring (core innovation) : Instead of the old, zoom-biased method of measuring how much of the photo a pothole's bounding box covers, the system runs Depth Anything V2 to build a depth map of the image and measures how far each pothole physically "sinks" below the surrounding road surface. Each pothole is classified LOW, MEDIUM, HIGH, or CRITICAL (inspired by the ASTM D6433 pavement-condition standard), and damage points are summed across every pothole in the image for a single cumulative severity score - so ten shallow cracks and two deep craters are both correctly flagged as serious road conditions, just for different reasons.
- Smart Duplicate Detection : Combines GPS distance, CLIP image similarity (cosine ≥ 0.85), and reporter email to decide the outcome of a new report: block it as a repeat from the same person, silently link it to an existing complaint group if a different person photographed the same pothole, or save it as a brand-new complaint. Every complaint's status is still managed independently - linking never causes an automatic status cascade.
- Priority Scoring (V5 formula) : A single 0–1 urgency score combining 50% live road traffic/importance (TomTom), 20% depth-based severity, 20% description urgency (Groq LLM, with keyword-based fallback), and 10% hotspot density (other complaints within 2km) - plus flat boosts for potholes near hospitals and for high-speed, free-flowing traffic roads.
- Reverse Geocoding & Road Classification : Resolves GPS coordinates into a human-readable address and an OpenStreetMap road classification (motorway, trunk, residential, and so on) used both for display and for the priority formula's traffic fallback.
- AI-Generated Explanations : A Groq LLM writes a short, plain-English explanation addressed to the reviewing official, summarizing why a complaint received its priority score and what the risk factors are.
- Nearby Reports Panel : Shows officials how many other reports exist within a configurable radius (e.g. 500m) of the current complaint, for context - without merging their statuses or forcing a single decision across all of them.
- Official Review Workflow : Officials can approve, reject, or resolve each complaint individually, recording their name and notes for an audit trail. Every action applies only to the single complaint being reviewed.
- Live Dashboard Stats : A sidebar shows running totals - total complaints, total detections, and a breakdown by status - for a quick health check of the whole system.
Streamlit Frontend - Three Tabs
- Submit Report - upload a road image with location and optional details; view AI detection results, priority score, and explanation.
- All Complaints - browse every complaint sorted by priority, filterable by status and road type, with an expandable detail view per report.
- Official Review - pull up a specific complaint, view the annotated image and AI explanation, check nearby reports, and take action.
Challenge
Municipal road-maintenance teams receive scattered, unverified complaints with no way to gauge true urgency. Photos vary wildly in zoom and angle, making severity comparison unreliable, while duplicate reports from the same pothole waste inspection resources. Officials needed a system that could turn a single low-quality citizen photo into a trustworthy, comparable severity and priority signal without manual site visits.
Complexity & Innovation
- Depth-based severity scoring - replacing the industry-standard zoom-biased bounding-box area method with true physical depth estimation via Depth Anything V2, so severity reflects actual road damage rather than photo framing.
- Multi-signal duplicate resolution - combining GPS proximity, CLIP visual embedding similarity, and reporter identity into a single decision tree that blocks, links, or creates complaints without ever merging independent complaint statuses.
- Composite explainable priority formula (V5) - a weighted blend of live traffic data, depth severity, LLM-scored description urgency, and geographic hotspot density, tuned with hospital-proximity and high-speed-road boosts.
- Async geospatial data layer - PostGIS + GeoAlchemy2 for efficient radius queries (nearby reports, hotspot density) at scale.
- LLM-generated, human-readable justifications - translating a numeric priority score into plain-English reasoning officials can act on without needing to understand the underlying model.
Process
- Requirements & data modeling - defined complaint schema, geospatial fields, and severity taxonomy aligned to the ASTM D6433 pavement-condition standard.
- Detection pipeline - integrated and fine-tuned YOLOv8 for pothole detection via Roboflow inference and a HuggingFace-hosted model.
- Severity engine - layered Depth Anything V2 monocular depth estimation on top of detections to compute per-pothole and cumulative damage scores.
- Duplicate-detection logic - built OpenCLIP embedding comparison combined with GPS distance thresholds and reporter identity checks.
- Priority scoring - iterated through multiple formula versions (culminating in V5) blending traffic, severity, urgency, and hotspot signals.
- Backend & API - built async FastAPI services backed by PostgreSQL/PostGIS for geospatial querying.
- Frontend - built a three-tab Streamlit interface for citizen submission, complaint browsing, and official review.
- Testing & rollout - validated against real road-photo datasets and iterated on scoring thresholds with sample municipal data.
Feature Inventory
- One-photo complaint submission with GPS and optional description/email
- YOLOv8 pothole detection with bounding boxes and confidence scores
- Depth-based cumulative severity scoring (LOW/MEDIUM/HIGH/CRITICAL)
- CLIP-based duplicate detection with GPS + email cross-checks
- V5 explainable priority scoring formula
- Reverse geocoding and OpenStreetMap road classification
- Groq LLM-generated plain-English priority explanations
- Nearby-reports contextual panel (configurable radius)
- Official review workflow (approve/reject/resolve with audit trail)
- Live dashboard stats (totals, detections, status breakdown)
Outcome
An end-to-end, AI-triaged civic reporting system that turns a single phone photo into a geo-validated, severity-scored, de-duplicated, and priority-ranked maintenance ticket - with a complete review workflow for government officials.