Senior Data Engineer
Information Technology | Toronto
The Senior Data Engineer is responsible for expanding and optimizing the current data and data pipeline architecture as well as optimizing data flow and data collection for various teams (Data Analytics, Software, IOT). The Data Engineer is an experienced data pipeline builder and data wrangler who can optimize data systems and be able to build them from the ground up. The Data Engineer will support our software developers, database architects and data scientists on data initiatives and will ensure optimal data delivery architecture. Data Engineers must be self-directed and comfortable supporting the data needs of multiple teams, systems and products.
Essential Responsibilities & Duties
- As and when required, design, build and maintain optimal data pipeline architecture, including for cloud applications.
- Design scalable infrastructure for consuming and processing big data as and when required.
- Manage large, complex data sets and ensure they meet functional / non-functional business requirements.
- Identify, design and implement internal process improvements: automating manual processes, optimizing data delivery, re-designing infrastructure for greater scalability.
- Develop and maintain data warehouse architecture, schematics and relational/non-relational databases for Analytics requirements.
- Build the infrastructure required for optimal extraction, transformation and loading of data from a wide variety of data sources using SQL, XML and other technologies.
- Work with stakeholders including the Business segments, Data Scientists and Software teams to assist with data-related technical issues and support their data infrastructure needs.
- Work with Data Scientists to support key use cases for machine learning to improve functionality in our data systems.
- Create data tools for analytics and data scientists that assist them in building and optimizing our product into an innovative design.
- Examine data using descriptive analytics to answer questions.
- Keep data secure using industry standards and strategies for security.
- Manage data bases and data models as required. Design and build data pipelines and data streams.
- Design and build data services APIs.
- Build dashboard and reports within BI/visualization tool to specification for data visualization.
Education and Experience:
- Bachelor’s or Masters degree in Applied Mathematics, Science, Engineering or equivalent experience
- 5+ years in a Data Engineer position.
- Experience with RDBMS including one of mysql, postgres, MS SQL, or Oracle.
- Experience developing in NoSQL databases such as MongoDB, Cassandra, or Accumulo.
- Experience with Cloud Platforms and with cloud data storage and processing techniques
- Experience with big data tools: Hadoop, Spark, Kafka, etc.
- Experience with data pipeline and workflow management tools: Jenkins, GitLab, Airflow, etc.
- Experience with Kubernetes, Docker, Jenkins
- Experience in use of BI and visualization tools like Spotfire
- Utilize Python and other Big Data tools for data operations
- Worked with Spark, Kafka, Flink, Dataflow and other steaming technologies.