Data Pipeline Engineer
- Data Pipeline Engineer / Data Pipeline Architect
- 3-month contract (initially)
- Remote Working (UK-based)
- Market Rates
- Outside IR35
The Opportunity:
We are supporting a central government client who are seeking an experienced Data Pipeline Engineer and Architect to help design and build an open-source Data Transformation Layer for a national data exchange platform.
This is a hands-on engineering and architecture role, combining the design of reusable, reliable data pipelines with practical development using Apache NiFi, RDF, SPARQL, Python and Java.
The platform will use open-source, cloud-agnostic technologies to ingest, transform, validate and exchange data securely between systems.
Skills and Experience:
- Proven experience designing and delivering data integration or ETL solutions using Apache NiFi.
- Practical project experience with RDF triples, SPARQL and semantic data modelling.
- Hands-on development experience in both Python and Java.
- Experience designing reusable pipelines across APIs, databases, files and other data sources.
- Understanding of reliable data processing, validation, automated testing and operational monitoring.
- Experience with version control, CI/CD, secure integrations and repeatable deployments.
- Ability to translate high-level requirements into technical designs and production-ready solutions.
Desirable Experience
- Apache Jena and Jena Fuseki.
- Apache Kafka and event-driven data integration.
- SHACL or other RDF validation approaches.
- Custom NiFi processor development.
- Docker, Kubernetes and portable deployment architectures.
- Large RDF datasets, SPARQL optimisation or secure data exchange between organisations.
Key Technologies:
Apache NiFi | RDF | SPARQL | Python | Java | Apache Jena | Fuseki | Kafka | ETL | Data Integration | Open Source
Role and Responsibilities:
- Design and evolve the architecture of a reusable, cloud-agnostic data transformation platform.
- Build, test and maintain Apache NiFi flows, including integrations with APIs, databases, files and messaging systems.
- Develop pipelines to ingest, transform, validate and publish RDF triples, using SPARQL for querying and updates.
- Develop supporting components and integrations using Python and Java.
- Define data mappings, interface contracts, technical specifications and architectural decisions.
- Implement robust processing, including error handling, retries, duplicate management and recovery.
- Establish automated testing, monitoring, version control and repeatable deployment practices.
- Collaborate with architects, developers and stakeholders to translate requirements into working solutions.
- Produce clear technical documentation and support the delivery of reusable platform capabilities.
NB: Candidates must be eligible for BPSS security clearance which will processed following successful interviews (5-7 days).
Applications:
To learn more about this newly created opportunity consulting as a Data Pipeline Engineer / Data Pipeline Architect on an initial 3-month engagement, operating on a remote-working basis for a prominent UK Government Agency; please call and speak with Edward Laing here at ISR Recruitment on 07436 071 872 or please send through a copy of your latest online profile and/or CV (edward@isr.uk.com) for an immediate call back in the strictest confidence.
Please contact Edward here at ISR Recruitment to learn more about our client and how they are leading the way in developing the next generation of technical solutions through innovation and transformational technology?