1. Building and maintaining batch and streaming data pipelines critical to Middesks data infrastructure using tools such as Airflow, BigQuery, Dataflow, Dataproc, and VertexAI, and building and managing ETL data pipelines to power machine learning models and APIs.
2. Ensuring the reliability, scalability, and maintainability of systems across the stack.
3. Building, scaling, and extending web scraping platforms and capabilities for real-time data acquisition, and building and managing data gathering web-scrapers in a Python-based environment.
4. Updating and maintaining the interface layer between our application and data platform stack to improve performance as we continue to scale our platform and datasets, including developing RESTful APIs using FastAPI to support data validation, ingestion, and entity resolution for multiple enterprise customers.
5. Collaborating cross-functionally with multiple teams to gather product requirements and conducting design reviews to ensure alignment.
6. Contributing to the continuous improvement of development processes and methodologies, including leveraging open-source tools (PySpark, Delta) to improve data quality, pipeline health and monitoring.
7. Researching and adopting new technologies and open source implementations.
Education and Experience Requirements
The position requires a Bachelors (or foreign educ. equiv.) Degree in Computer Science, Computer Engineering, or a closely related field plus three (3) years of experience in the job offered or related occupation. Experience must include:
a. Building and managing ETL data pipelines to power machine learning models and APIs
b. Building and managing data gathering web-scrapers in a Python-based environment
c. Developing RESTful APIs using FastAPI to support data validation, ingestion, and entity resolution for multiple enterprise customers
d. Leveraging open-source tools (PySpark, Delta) to improve data quality, pipeline health and monitoring
Telecommuting/Remote Work is Permitted
May telecommute
Salary
$185,000 per annum. 40 hours per week; M-F.
Please copy and paste your resume in the email body (do not send attachments; we cannot open them) and email it to candidates at (link removed) with job reference in the subject line.
Thank you.
![]()
