Data Scientist (Dev + QA) – AI Quality Assurance

Position Overview

AriveGuru is seeking a Data Scientist with combined development expertise in Artificial Intelligence and Quality Assurance. The selected candidate will be responsible for developing data science and machine learning solutions and for ensuring that every AI-driven capability we release meets defined standards of accuracy, reliability and consistency.

Candidates must demonstrate working proficiency in AI/data science for both development and software quality assurance practices.

Key Responsibilities

AI and Data Science

• Develop, test and refine machine learning and LLM-based solutions, including retrieval-augmented generation (RAG), classification and information extraction
• Perform exploratory data analysis (EDA), cleaning, and preparation across cloud data platforms and BI tools to support model development and evaluation
• Author, maintain, and version-control analytical scripts and collaborative notebooks within unified data workspaces (e.g., Databricks)
• Design and iterate on prompts using structured, repeatable testing methods

AI Quality Assurance and Evaluation

• Design evaluation datasets, reference (“golden”) answers and test cases to measure the accuracy, relevance and consistency of AI outputs
• Define quality metrics, acceptance criteria and release thresholds for AI-driven features
• Build automated evaluation and regression-testing pipelines in Python, integrated into the development and release workflow
• Design and execute controlled A/B tests and online experiments to validate prompt iterations, model updates, and feature releases
• Track and analyze key AI response and telemetry metrics (e.g., latency, token usage, grounding accuracy, and user feedback) to measure real-world performance
• Identify, document and report failure modes, including hallucinations, incomplete or incorrect responses, bias and formatting errors, and coordinate with engineering teams on resolution
• Apply LLM-as-a-judge methods and calibrate automated scoring against human review
• Monitor production performance, detect model drift and quality degradation, and escalate issues promptly
• Validate the completeness, consistency and correctness of datasets used by our systems

Required Qualifications

AI and Data Science

• Bachelor’s degree in Computer Science, Data Science, Statistics, Mathematics or a related discipline
• Approximately 2 years of hands-on experience in data science, machine learning or AI projects
• Strong proficiency in Python and its data libraries (Pandas, NumPy, Scikit-learn)
• Good working knowledge of SQL and relational databases such as PostgreSQL
• Hands-on experience managing analytical workflows, scripts, and notebooks in cloud data/lakehouse environments (e.g., Databricks) and performing data analysis in BI tools
• Sound understanding of machine learning fundamentals, statistics, experimental design (A/B testing), and evaluation metrics (precision, recall, F1 score, etc.)
• Practical experience with large language models, including prompt engineering, RAG, and an understanding of hallucination and grounding

Quality Assurance

• Working knowledge of QA fundamentals: test planning, test case design, edge-case analysis, defect reporting and regression testing
• Experience testing AI or ML outputs for accuracy, consistency and bias
• Ability to track, benchmark, and report on AI response metrics across model versions
• Ability to build automated tests and evaluation pipelines in Python
• Experience in data quality validation
• Ability to define pass/fail criteria and release standards
• Strong attention to detail and a disciplined, evidence-based approach to verifying results

Preferred Qualifications

• Experience with evaluation frameworks such as Ragas, DeepEval or LangSmith
• Familiarity with LangChain or LlamaIndex
• Knowledge of test automation and API testing tools (Pytest, Postman, Selenium or equivalent)
• Experience with Git, CI/CD pipelines and AWS
• Proficiency in BI platforms and data visualization tools (Power BI, Tableau, Databricks SQL, Matplotlib, or Plotly)
• Exposure to NLP, embeddings and vector databases

Desired Attributes

• Analytical, detail-oriented and methodical
• Strong written and verbal communication skills
• Ability to work independently and collaboratively in a fast-paced environment
• Commitment to continuous learning, including effective use of modern AI tools
• High sense of ownership and accountability for quality

What We Offer

• Opportunity to work on production-grade AI systems where quality standards directly influence outcomes
• Exposure to the complete lifecycle of AI development, evaluation and deployment
• Defined career growth toward senior data science and AI quality leadership roles
• Close collaboration with the leadership and engineering teams