Data did not live in one place: some in MSSQL, some in MariaDB, some behind third-party APIs. Every time a report was needed, somebody exported tables by hand and joined them together. That was slow, and it could produce a different answer each time it was repeated.

I built pipelines that automate data collection, processing and reporting end to end. The flow looks roughly like this:

1. Collection — scheduled jobs pull data from source systems
2. Normalisation — Python services unify the schema and fix types and encodings
3. Enrichment and classification — LLM integrations label free text, assign categories and extract missing fields
4. Indexing — the result is written to Elasticsearch, becoming searchable and filterable
5. Reporting — finished outputs are delivered to the relevant teams automatically

Orchestration to n8n, the actual work to Python. Scheduling, retries, conditional branching and cross-system triggers live in n8n as a visual flow; the real data processing lives in Python services. That way changing a workflow does not require a code deploy, but heavy processing is not squeezed into a visual tool either.

The LLM step was separated from the deterministic ones. A model call only happens where reasoning is genuinely needed — classification and extracting fields from free text. Schema transformation, joining and validation are done with ordinary code. That split had a direct effect on both cost and the reproducibility of results.

Services talk over REST with a fixed contract. Every step can be run and tested on its own; one step crashing does not lock the whole chain, the work just waits in the queue.

The search layer was moved to Elasticsearch. Searching high-volume datasets with LIKE on a relational database did not scale. Moving indexing and filtering to Elasticsearch reduced both query time and database load.

The collect–join–report loop that used to be manual now runs on its own, results sit in a searchable index, and the same input always produces the same output. Adding a new source means writing only the collection step, not the whole chain.