Data Engineer

The Intermediate Data Engineer will be working within a growing data team in a dynamic telecommunications fibre environment. Working closely with the Data Specialist and senior members of the data team, the incumbent will design, build, and maintain scalable, efficient data pipelines and data platform artefacts that support business operations, network performance monitoring, customer analytics, and strategic decision-making. The ideal candidate will have solid hands-on experience with modern data engineering tooling, particularly Databricks and Python, and will demonstrate a practical understanding of data integration, data quality, and data architecture principles. They will be a self-sufficient contributor who can engage directly with data analysts and data owners to resolve data quality issues and translate business requirements into reliable, high-quality data solutions.

ACCOUNTABILITIES & RESPONSIBILITIES (KEY PERFORMANCE AREAS & TASKS)

• Design, build, and maintain scalable and reliable data pipelines using Databricks and Python, sourcing and transforming data from multiple internal and external systems.
• Develop and maintain ETL/ELT workflows and data transformation logic using Informatica / IDMC to automate the movement and processing of data across the platform.
• Build and maintain data lake and data warehouse artefacts, including tables, schemas, and data models, ensuring they are optimized for performance and scalability.
• Monitor data pipeline health and performance using Grafana dashboards; proactively identify, diagnose, and resolve pipeline failures and data anomalies.
• Implement and maintain data quality checks to ensure the accuracy, completeness, and consistency of data across all platforms.
• Consult data analysts and data owners to understand and resolve data quality issues, translating business requirements into practical data engineering solutions.
• Collaborate with the Data Specialist and business stakeholders to design data solutions that meet reporting, analytics, and operational needs.
• Support integration of structured and unstructured data from internal and external sources, ensuring data is accessible and fit for purpose.
• Apply data governance, security, and compliance standards in line with POPIA and relevant telecom regulatory requirements.
• Document data pipelines, transformation logic, data models, and architectural decisions to maintain an accurate and up-to-date knowledge base.
• Stay current with emerging data engineering tools, technologies, and best practices, and contribute to continuous improvement of the data platform.

Competencies and Minimum Requirements:

Knowledge, qualifications and experience

• Bachelor’s degree in Computer Science, Information Systems, Engineering, or a related field (required)
• Relevant certifications in data engineering, cloud platforms, or database technologies (advantageous)

Technical Knowledge & Skills

Core Stack – Required

• Databricks – hands-on experience building and managing data pipelines, notebooks, and Delta Lake artefacts.
• Python – proficient in Python for data transformation, pipeline orchestration, and automation tasks.
• Informatica / IDMC – experience designing and maintaining data integration workflows and mappings.
• Grafana – ability to configure and interpret monitoring dashboards for pipeline observability.

Additional Technologies – Advantageous

• PowerBI – working knowledge of how PowerBI consumes and models data; ability to publish datasets and support report development is an advantage.
• Relational databases: PostgreSQL and/or Microsoft SQL Server.
• Cloud platforms: AWS or Microsoft Azure.

• Microsoft 365 / SharePoint – familiarity with data stored in or shared via Microsoft collaboration tools.

Core Data Engineering Skills

• Strong proficiency in SQL for data querying, transformation, and analysis.
• Solid understanding of data structures, data modelling, and database design principles.
• Experience with data quality management, data governance, and data security practices.
• Familiarity with data warehouse and data lake concepts, including layered architectures and dimensional modelling.
• Practical understanding of batch and/or streaming data processing patterns.

Skills & Competencies

• Strong analytical and problem-solving skills with close attention to detail.
• Effective communication skills, with the ability to engage both technical and non-technical stakeholders.
• Ability to work collaboratively within a cross-functional data team in a fast-paced environment.
• Proactive approach to identifying and resolving data quality and pipeline issues.

Experience & Skills

• Minimum 3 – 6 years of experience in data engineering or a closely related field.
• Proven hands-on experience with Databricks and Python in a production data engineering environment.
• Demonstrable experience with ETL/ELT tools and frameworks, particularly Informatica / IDMC.
• Experience configuring and maintaining monitoring and observability tooling (e.g., Grafana).
• Solid experience with SQL and relational database environments.
• Experience working with cloud platforms (AWS or Azure) is advantageous.
• Proven experience in the telecommunications or fibre industry is strongly preferred; candidates with relevant experience in other data-intensive industries will also be considered.
• Familiarity with OSS/BSS data environments and telecom data structures is advantageous.
• Knowledge of data governance, data security, and regulatory compliance (e.g., POPIA).