ML Platform Engineer, Evaluation Systems at Bask Health in Remote

ML Platform Engineer, Evaluation Systems

About Bask Health

Bask Health is a software platform that enables entrepreneurs, doctors, and healthcare businesses to launch, manage, and grow direct-to-consumer telehealth businesses.

Our platform brings together patient experiences, clinical workflows, pharmacy operations, and commerce tools in one place.

We're an entrepreneurial, product-focused engineering team that values ownership, technical excellence, and building software that solves real problems. Engineers work closely with product teams and company leadership, take features from idea to production, and have a direct role in shaping the platform.

Engineering at Bask is AI-first and builder-led. We use tools like Cursor and Claude Code throughout development while maintaining high standards for reliability, testing, and software quality.

About the Role

We're looking for an ML Platform Engineer, Evaluation Systems to build and maintain the infrastructure that allows Bask Health to evaluate machine learning models and AI-powered applications reliably, consistently, and at scale.

Bask Health supports direct-to-consumer telehealth businesses through a platform that connects patient experiences, merchant operations, clinical workflows, pharmacy coordination, and commerce capabilities. As machine learning and AI become more integrated into these systems, we need dependable infrastructure for measuring model quality, comparing changes, and understanding how AI systems perform before and after deployment.

You'll help develop the platform services, data pipelines, execution environments, and monitoring capabilities that support machine learning evaluation across engineering teams.

This is a hands-on engineering role focused on the operational infrastructure behind evaluation systems. You'll work with cloud services, distributed processing, model integrations, databases, and automated workflows to support repeatable experiments and large-scale evaluation jobs.

Your work may include building evaluation job orchestration, managing benchmark datasets, developing experiment tracking systems, supporting model comparisons, and creating infrastructure that stores and analyzes evaluation results.

You'll collaborate with machine learning engineers, applied scientists, AI evaluation engineers, and backend engineers to understand their technical requirements and build reusable systems that support different model types and evaluation methodologies.

You'll also help improve the reliability and reproducibility of evaluation workflows. This includes ensuring experiments can be repeated, datasets and configurations are versioned, failures are observable, and results can be compared accurately over time.

Because Bask operates in healthcare-related environments, evaluation infrastructure must support appropriate privacy controls, secure data access, and responsible handling of sensitive information. You'll help ensure evaluation datasets and model outputs are managed according to approved security and operational requirements.

This position emphasizes strong infrastructure engineering, practical machine learning knowledge, and ownership of production platform services. You'll contribute to architecture, build scalable systems, improve operational performance, and help engineering teams evaluate AI capabilities more effectively.

We're looking for an engineer who enjoys building infrastructure for machine learning teams, understands the challenges of reproducible experimentation, and wants to make AI evaluation more dependable and efficient across the organization.

What You'll Do

• Design, build, and maintain machine learning evaluation infrastructure across Bask Health.
• Develop platform services that execute and manage automated model evaluation workloads.
• Build scalable systems for running large batches of model and AI application evaluations.
• Implement job orchestration, scheduling, and execution management for evaluation pipelines.
• Develop infrastructure for storing and versioning evaluation datasets.
• Build systems for tracking model configurations, prompts, parameters, and evaluation results.
• Implement reproducible experiment workflows that support consistent model comparisons.
• Develop automated pipelines for benchmarking machine learning models.
• Build APIs and services that allow engineering teams to submit and monitor evaluation jobs.
• Create infrastructure for comparing evaluation results across model versions and experiments.
• Support evaluation workflows for large language models, predictive models, and retrieval systems.
• Develop asynchronous processing systems for computationally intensive evaluation tasks.
• Implement monitoring for evaluation job performance, failures, latency, and resource usage.
• Build logging and tracing capabilities that help engineers investigate evaluation problems.
• Develop systems for managing evaluation artifacts, reports, and historical results.
• Integrate evaluation infrastructure with model registries and ML development workflows.
• Implement automated quality checks that support controlled model releases.
• Build reliable retry, timeout, and failure recovery mechanisms for evaluation workloads.
• Optimize evaluation execution performance, infrastructure utilization, and operational costs.
• Support distributed evaluation workloads across cloud-based compute resources.
• Collaborate with machine learning engineers to define evaluation infrastructure requirements.
• Work with applied scientists to support reproducible experiments and benchmark comparisons.
• Develop tools that simplify evaluation configuration and execution for engineering teams.
• Implement secure access controls for evaluation datasets and results.
• Ensure evaluation systems respect tenant boundaries and approved data-handling practices.
• Investigate infrastructure incidents and improve evaluation platform reliability.
• Document evaluation infrastructure, deployment procedures, and operational standards.
• Own assigned ML evaluation platform projects from technical design through deployment, monitoring, and continuous improvement.

Requirements

• 3+ years of professional experience in software engineering, machine learning engineering, platform engineering, or a closely related technical field.
• Experience building and maintaining production backend services or infrastructure.
• Strong programming skills in Python, TypeScript, or comparable languages.
• Familiarity with machine learning development and evaluation workflows.
• Experience developing APIs and backend service integrations.
• Understanding of distributed systems and asynchronous processing.
• Experience working with cloud-based infrastructure and production deployments.
• Familiarity with automated testing and continuous integration practices.
• Understanding of machine learning model versioning and experiment reproducibility.
• Experience working with relational databases and structured datasets.
• Familiarity with data pipelines and batch processing workflows.
• Experience designing reliable background job systems.
• Understanding of infrastructure scalability, performance, and fault tolerance.
• Familiarity with logging, monitoring, and production observability.
• Experience implementing retries, timeouts, and error recovery.
• Understanding of authentication, authorization, and secure service communication.
• Familiarity with configuration management and deployment automation.
• Experience writing maintainable, well-tested production code.
• Ability to troubleshoot complex backend or infrastructure issues.
• Understanding of software release processes and operational reliability.
• Experience using Git and collaborative engineering workflows.
• Ability to translate machine learning requirements into scalable infrastructure.
• Strong analytical thinking and problem-solving skills.
• Ability to collaborate effectively with machine learning engineers, applied scientists, and software engineers.
• Ability to independently own engineering projects from design through deployment.

Nice to Have

• Experience building infrastructure for machine learning evaluation or experimentation.
• Familiarity with MLflow, Weights & Biases, or comparable experiment tracking platforms.
• Experience with model registries and artifact management systems.
• Familiarity with large language model evaluation frameworks.
• Experience with LangSmith, DeepEval, Ragas, or comparable AI evaluation tools.
• Experience building distributed evaluation or benchmarking pipelines.
• Familiarity with workflow orchestration tools such as Airflow, Dagster, or Temporal.
• Experience with containerized workloads and distributed compute infrastructure.
• Familiarity with Docker, Kubernetes, or comparable technologies.
• Experience working with AWS Lambda, SQS, EventBridge, or related cloud services.
• Familiarity with AWS SageMaker or comparable ML platform services.
• Experience with serverless infrastructure and infrastructure-as-code tools.
• Familiarity with SST or comparable application deployment frameworks.
• Experience developing data versioning and dataset management systems.
• Understanding of reproducible machine learning experimentation.
• Experience implementing model performance monitoring and drift detection.
• Familiarity with large language model APIs and model comparison workflows.
• Experience building internal developer platforms for machine learning teams.
• Understanding of HIPAA and healthcare data privacy considerations.
• Experience using AI-assisted development tools such as Cursor or Claude Code.