Real Estate News Data Pipeline & API Dashboard

Bringing news from 13 publisher sources into one monitored data pipeline with consistent article records, API access, scheduling, and operator controls.

Web scrapingData pipeline developmentAPI developmentData extractionData automation

Client

Private real estate data client

Status

Private client build

A news data platform built for repeatable operations

The client needed news from multiple UAE, GCC, and regional publishers to arrive in one consistent system. The difficulty was that those publishers used different website platforms, APIs, page structures, and access methods, so the same collection method could not simply be reused everywhere.

Once collected, the articles also needed to become dependable data for other products. The platform stores them in one consistent article structure and exposes them through a read-only API with API keys, quotas, request logs, documentation, and health endpoints.

Product context

Every publisher exposed content differently, and any source could change or stop working over time. The client needed a system where those differences stayed inside the collection layer while source health, failed runs, and API access remained visible to the team.

Challenge

The challenge

Each publisher behaved differently. One source might provide a WordPress API, another might keep useful data inside the page, and another could require its own parsing or access handling. Those external sites could also change without warning, so the team needed to know which source stopped working, when it happened, and what the last collection run actually did.

What we built

We built one operating layer around the 13 publisher integrations. Each source keeps the extraction logic it needs, while article structure, storage, run tracking, scheduling, API access, and operator controls are shared across the platform.

Multi-source ingestion pipeline

Source-specific collectors handle 13 publishers using methods such as WordPress REST APIs, Inertia payloads, Next.js state, Reuters Fusion CMS data, schema.org metadata, and server-rendered HTML.

Shared normalization layer

No matter where an article comes from, the platform stores it in the same structure for headline, summary, URLs, publish dates, authors, categories, tags, images, body text, read time, and source information.

Authenticated API surface

Other products can read the collected data through documented API endpoints protected by API keys, daily limits, usage logs, analytics, and key revocation.

Operator dashboard

The team can see source health, collection runs, scheduler settings, source details, runtime configuration, corpus size, and usage information without going into the server code.

Result

The result

The client now has one system for collecting, storing, monitoring, and serving news from 13 publisher sources. Operators can see how sources are behaving and manage recurring collection without treating every publisher as a separate manual process.

The collected articles also become a reusable data layer. Other products can access consistent records through controlled API keys while the database, collection logic, usage tracking, and operational controls stay behind the platform.

publisher/source integrations handled through source-specific logic

API keys

authenticated read access with quota and usage tracking

Scheduler

manual and scheduled scrape controls for operators

Docker

deployment support for single-service and two-service setups

Client feedback

“The system gave our operators a clearer way to monitor sources, recover from source issues, and use the collected data through a controlled API instead of relying on fragile scripts.”

Name withheld

Operations Lead, Private Data Platform

The impact

Why this mattered

The value came from what these decisions changed for the people using the product and the team responsible for running it.

Source problems became easier to find

The team can see source health, recent runs, failures, and retry behaviour, making it easier to identify when one publisher stops feeding the platform correctly.

Every source produces the same kind of article record

Different publisher formats are converted into one consistent structure, so downstream products do not need separate logic for every source.

Other products can use the data without database access

API keys, quotas, and usage tracking provide a controlled way to serve the collected articles while keeping the underlying database and collection system protected.

More case studies