Overview
Principal Data Engineer Jobs in Brisbane, Queensland, Australia at Queensland Government
Title: Principal Data Engineer
Company: Queensland Government
Location: Brisbane, Queensland, Australia
As a Principal Data Engineer, you will be responsible for the design, development, deployment, and ongoing operations of data pipelines to bring data from source destinations into a data lakehouse. The data engineering and data pipelines will involve the use of modelling skills and tools for extract, transform and load operations.
As a Principal Data Engineer You Will
- Designing and developing data pipelines that extract data from various sources, transform it into the desired format, and load it into the appropriate data storage.
- Collaborate with data scientists and analysts to optimize models and algorithms for data quality, security, and governance.
- Integrate data from different sources, including databases, data warehouses, APIs, and external systems.
- Ensure data consistency and integrity during the integration process, performing data validation and cleaning as needed.
- Transform raw data into a usable format by applying data cleansing, aggregation, filtering, and enrichment techniques.
- Optimize data pipelines and data processing workflows for performance, scalability, and efficiency.
- Monitor and tune data systems, identify and resolve performance bottlenecks, and implement strategies to enhance query performance.
- Implement data quality checks and validations to ensure the accuracy, consistency, and completeness of data.
- Construct solutions for the governance of data and algorithms used for analysis, analytical applications, and automated decision making.
Your Mandatory Requirements
- Azure Data Factory, Data Build Tools, Data Vault, Data Bricks experience.
- The ability to design and implement well written code.
- Understanding of Data Lakehouse principals.
- The ability to work in a disciplined Agile team.
- Experience actioning business requirements, translating the requirements into data products, and collaboration with other teams and developers.
- Ability to contribute to Data Lakehouse content including the mapping and wrangling of data assets.
- Understanding of Analytics concepts and experience in working with data visualisation tools like Power BI.
- Knowledge of data governance principles.
- Modelling data, creating datasets, reports, and visualisations.
- Experience using DevOps is desirable.
Occupational group IT & Telecommunications
