Smart City & IoTConfidential

Smart City Data Platform

Municipal citizen-engagement platform aggregating waste, climate, warning and geo services from external civic APIs behind a modular Symfony core with Elasticsearch geo-search and reliable mobile push.

Client: Confidential · Public Sector
Role: {{TODO: e.g. Solution Architect / Lead Backend Engineer}}
Period: {{TODO: e.g. 2022–2023}}

Modular DDD core: 11 bounded domain services on Symfony 6.4

Event-driven async ingestion via Symfony Messenger (Doctrine transport)

Elasticsearch geo-search for sub-second municipal-scale radius queries

Change-detection diffing turns raw warning feeds into targeted push alerts

Summary

Overview

The platform aggregates civic services and real-time data for urban residents — waste-collection schedules, climate incentives, civic warnings, a service-provider directory and location-based discovery — and delivers them to citizens through a Next.js dashboard and mobile push. It targets German municipalities automating citizen engagement.

The core challenge was synchronizing many independent external civic APIs (Abfall+, ArcGIS, Climate Go, Warning APIs) into one consistent, freshly indexed dataset, then detecting change in that data so citizens are notified only on genuinely new or updated events — all while keeping geographic queries fast at municipal scale.

Context

The Problem

Civic data lives in fragmented external systems, each with its own schema, freshness and availability. A municipality needs a single trustworthy citizen surface, but it does not own the sources — it must poll them, absorb their drift, and stay responsive even when a source is slow or briefly unavailable.

Constraints

Multi-source ingestion where a slow or failing external API must not block the citizen-facing platform.
Notifications only on real change: raw feeds re-publish unchanged data constantly, so naive polling would spam citizens.
Sub-second geographic browse over a growing set of places and providers, without hand-rolling database geo-index complexity.
Reliable mobile push delivery under transient gateway failures.
24/7 availability expected of a civic service, with eventual consistency acceptable for synced data.

Scope

Requirements

Functional

▹Ingest multi-source civic data (Abfall+ waste, ArcGIS geo, Climate Go incentives, external Warning APIs)
▹Real-time alert push to mobile and web on warning diffs and waste reminders
▹Geospatial browse: city maps and location-based service discovery
▹Service-provider directory with qualification and portfolio profiles
▹User accounts, roles and location preferences via Keycloak
▹Admin panels for push campaigns and user management
▹REST API with OpenAPI 3.0 contracts for web and mobile clients

Non-functional

▹Source isolation — a degraded external API must not cascade to citizen reads
▹Eventual consistency with a few-minute sync lag tolerated
▹Sub-second geo-search latency at municipal scale
▹Reliable async push with retry and backoff
▹PII compliance for user locations, addresses and device tokens
▹Independently scalable containers for core, messenger and web tiers

11 modular domain services as separate bounded contexts4 external civic API families polled on schedules via sync commandsMultiple messenger workers draining async, images, pusha, mail and failed queues in parallelElasticsearch indexes Places for geo and full-text; Redis handles cache and sync locks

System Design

Architecture

100% · drag to pan · scroll to zoom

Rendering diagram…

Modular Domain Services

Eleven bounded contexts (Contact, Provider, CityMap, Warning, WasteCalendar, ClimateGo, Auth, User, Content, PushNotification and supporting modules) keep civic concerns independently maintainable.

Async Ingestion Pipeline

Scheduled sync commands feed Symfony Messenger over the Doctrine transport; workers drain async, images, pusha, mail and failed queues, decoupling external polling from citizen traffic.

Source Adapters

EventSourceService (RSS/JSON loaders) and ArcgisDataSync absorb per-source schema drift and scheduling so a stable V1 API contract is preserved as external feeds change.

Change Detection

WarningChangeEvent and FetchedWarningsCalculateDiffs compute deltas over fetched warnings so citizens are notified only on new or changed events.

Geo Search

CityMapProvider indexes Place documents in Elasticsearch, serving sub-second radius and category queries instead of hand-tuned database geo-indexes.

Push Delivery

The Pusha bundle and SchedulePushaNotification message type deliver mobile push asynchronously with a retry-and-backoff strategy for transient gateway failures.

Stack

Technology Stack

Backend

PHP 8.3Symfony 6.4API Platform 2.6Doctrine ORM 2.11

Data

MySQL 8Elasticsearch 7.17Redis 6.2S3

Frontend

Next.js 12React 17Material-UI 5Keycloak (Auth0/Cognito/Firebase fallbacks)

Messaging / Ops

Symfony Messenger (Doctrine transport)Pusha push gatewayKaniko buildsSentry

Persistence

Data Model

Place (Elasticsearch): geo-location, category, provider reference, searchable metadata for radius and full-text queries
Provider (MySQL): organization profile, services, contact points, portfolio, hero/profile images
WasteCalendar (MySQL): pickup schedules per address, linked to pickup points
Warning/Alert (MySQL): fetched from external API, diff-detected to trigger change notifications
User (MySQL + Keycloak): profile, roles, location preferences, device tokens for push
Subscription (MySQL): user-to-provider notification subscriptions

Reasoning

Key Decisions & Trade-offs

Async-first ingestion via Symfony Messenger (Doctrine transport)

trade-off

Introduces eventual consistency and queue-management complexity, but keeps slow external polling, image processing, mail and push off the request path.

Elasticsearch for geographic and full-text search

trade-off

Adds an index to keep in sync with MySQL, but delivers sub-second radius queries without hand-rolled database geo-index tuning.

External API polling with no internal IoT sensors

trade-off

Simplifies the architecture but couples data freshness to external SLAs, mitigated with source adapters and scheduled sync.

Domain-driven modular bounded contexts

trade-off

Improves maintainability and clear ownership, at the cost of cross-module query complexity such as finding providers near a warning area.

Diff-based change detection before notifying

trade-off

Requires storing and comparing prior state, but prevents notification spam by alerting only on genuinely new or changed civic events.

Operations

Scaling & Reliability

Horizontal: containerized tiers (core, messenger, nginx, webgui) built with Kaniko scale independently; stateless Symfony workers add throughput on demand.

Ingestion: multiple messenger workers process the async, images, pusha, mail and failed queues in parallel, and Redis-backed locks serialize sync tasks so concurrent runs do not collide.

Resilience: the push queue retries transient failures with backoff; external API drift is absorbed by source adapters behind a stable V1 contract, and Elasticsearch sharding carries geo-search load as the place index grows.

What made this hard

Architect's Challenges

→Synchronizing multiple independent external civic APIs into one consistent, freshly indexed dataset
→Detecting change in noisy re-published feeds so citizens are alerted only on real updates
→Serving sub-second geographic browse at municipal scale via Elasticsearch geo-queries
→Delivering reliable mobile push under transient gateway failures with retry and backoff
→Keeping data fresh under an eventual-consistency model without blocking citizen-facing reads

Results

Outcome

▹Async, retry-backed push pipeline with a dedicated failed queue for recovery
▹Stable V1 API contract preserved across external feed drift via source adapters
▹{{TODO: add real metrics — municipalities served, push delivery rate, sync lag, search latency}}