BMO→
Associate AI Evaluation Scientist at BMO in Toronto, Ontario, CA
Skills
Job Description
Application Deadline:
Address:
VIRTUAL59 - REMOTE/TELETRAVAIL - ON - BMOJob Family Group:
About the Team
BMO’s Applied AI team is responsible for building high‑performing, safe, and reliable AI systems that power real banking experiences. The Evaluations group within Applied AI develops the methods, datasets, and tooling used to measure quality, safety, and performance across the AI lifecycle. Working closely with product, engineering, and research partners, the team helps ensure evaluation signals are embedded into experimentation, deployment, and monitoring workflows. The group operates at the intersection of data science, machine learning, and responsible AI, enabling scalable and trustworthy evaluation of advanced AI systems.
About the Role
The Associate AI Evaluation Scientist is an individual contributor role focused on contributing to the data science execution of AI evaluations under the guidance of senior scientists and engineers. This role supports the development and application of evaluation methods, metrics, and datasets that inform model quality, product performance, and AI risk management.
In this role, you will work hands‑on with machine learning models—particularly LLM‑based systems—helping to run evaluations, analyze results, and maintain evaluation workflows. You will gain experience translating evaluation standards into reusable code, collaborating with engineering partners, and communicating findings to technical stakeholders. This position is well suited for an early‑career data scientist who is excited to deepen their skills in applied ML evaluation, experimentation, and responsible AI practices.
Key Responsibilities
- Implement and run established evaluation methods for LLMs and ML systems, including performance, robustness, fairness, and safety‑related metrics.
- Contribute to the creation, curation, and maintenance of evaluation datasets such as golden sets, challenge sets, and test corpora aligned with banking use cases.
- Support the development and maintenance of evaluation pipelines, harnesses, and dashboards used in experimentation and model assessment.
- Perform empirical analyses such as error analysis, metric comparisons, and basic statistical testing to surface model behavior and performance issues.
- Assist with integrating evaluation results into experimentation workflows, model reviews, and deployment readiness checks.
- Document evaluation methodologies, assumptions, and results to support reproducibility and team knowledge sharing.
- Collaborate with senior scientists, engineers, and product partners to understand evaluation requirements and apply best practices.
- Learn and apply responsible AI principles, including reproducibility, transparency, and risk‑aware evaluation.
Qualifications
- 1–3 years of experience in data science, machine learning, analytics, or a related field, including internships, co‑ops, or substantial academic projects.
- Bachelor’s or Master’s degree in Computer Science, Data Science, Statistics, Engineering, or a related quantitative discipline.
- Proficiency in Python and familiarity with data science libraries (e.g., pandas, numpy, scikit‑learn); exposure to PyTorch or TensorFlow is a plus.
- Foundational understanding of statistics, model evaluation metrics, and experimental analysis.
- Experience working with structured or unstructured datasets and writing clear, maintainable analysis code.
- Interest in AI evaluation, model quality, safety, or responsible AI, with a desire to grow deeper technical expertise in this area.
- Strong written and verbal communication skills, with the ability to explain analysis results to technical teammates.
- Familiarity with LLMs, NLP, or ML experimentation workflows is an asset but not required.
Salary:
Pay Type:
The above represents BMO Financial Group’s pay range and type.
Salaries will vary based on factors such as location, skills, experience, education, and qualifications for the role, and may include a commission structure. Salaries for part-time roles will be pro-rated based on number of hours regularly worked. For commission roles, the salary listed above represents BMO Financial Group’s expected target for the first year in this position.
BMO Financial Group’s total compensation package will vary based on the pay type of the position and may include performance-based incentives, discretionary bonuses, as well as other perks and rewards. BMO also offers health insurance, tuition reimbursement, accident and life insurance, and retirement savings plans. To view more details of our benefits, please visit: https://jobs.bmo.com/global/en/Total-Rewards
About Us
At BMO we are driven by a shared Purpose: Boldly Grow the Good in business and life. It calls on us to create lasting, positive change for our customers, our communities and our people. By working together, innovating and pushing boundaries, we transform lives and businesses, and power economic growth around the world.
As a member of the BMO team you are valued, respected and heard, and you have more ways to grow and make an impact. We strive to help you make an impact from day one – for yourself and our customers. We’ll support you with the tools and resources you need to reach new milestones, as you help our customers reach theirs. From in-depth training and coaching, to manager support and network-building opportunities, we’ll help you gain valuable experience, and broaden your skillset.
To find out more visit us at https://jobs.bmo.com/ca/en.
BMO is committed to an inclusive, equitable and accessible workplace. By learning from each other’s differences, we gain strength through our people and our perspectives. Accommodations are available on request for candidates taking part in all aspects of the selection process. To request accommodation, please contact your recruiter.
Note to Recruiters: BMO does not accept unsolicited resumes from any source other than directly from a candidate. Any unsolicited resumes sent to BMO, directly or indirectly, will be considered BMO property. BMO will not pay a fee for any placement resulting from the receipt of an unsolicited resume. A recruiting agency must first have a valid, written and fully executed agency agreement contract for service to submit resumes.