Siri - Evaluation Scientist - Siri Data Organization

Santa Clara Valley (Cupertino), California, United States
Machine Learning and AI

Summary

Posted:
Role Number:200007324
Would you like to play a part in the next revolution in human-computer interaction? Contribute to a product that is redefining mobile and desktop computing, and work with the people who built the intelligent assistant that helps millions of people get things done — just by asking? The vision for the Siri Data Organization is to improve Siri by using data as the voice of our customers. Within this organization, the mission of the Analytics team is to inform the evolution of Siri through measurement and analysis of the user experience. Part of this mission is achieved through human evaluation; as an Evaluation Scientist, you will drive how we design our grading/evaluation tasks and guidelines.

Key Qualifications

  • Academic background in a social quantitative field such as Psychology, Cognitive Science, Sociology, Political Science, Quantitative Linguistics, or other related field.
  • A minimum of a Masters Degree plus 2+ years industry experience or a Bachelor's degree plus 5+ years of industry experience
  • Experience with human evaluation and/or an equivalent type of human-subjects task design (e.g., large scale studies in online environments)
  • Excellent writing skills, especially with regards to guidelines. The tasks you're crafting will drive our data collection, so people must understand what you want them to do!
  • Strong communication skills and the ability to drive ideas. You will work cross-functionally with everyone from our evaluation operations teams to partner engineering teams, ensuring that our evaluation tasks produce meaningful data that can be used across a variety of contexts.
  • Strong analytics skills: You should know how to think about data.
  • Data-querying skills (SQL a must - you must be able to serve yourself data to evaluate the tasks you design)
  • Experience with a scripting language for data processing and development (e.g., Python, R, or Scala)

Description

The Siri Data Organization is seeking a talented Evaluation Scientist to drive our methodologies for measuring how Siri is performing for our users as it can be measured by our evaluation program. Your outputs will help curate data sets and metrics that will be used across Siri, and will impact key decisions on the Siri product. YOU WILL: - Develop and own our various evaluation tasks for curating data sets from our graders. This will include: - Working with our evaluation platform engineering team to bring the tasks to life ensuring that the tasks are designed to elicit reliable labels (clear and concise) - Partnering cross-functionally to ensure that the tasks are providing data that meets the Siri org's needs - Draft and collaborate on our evaluation guidelines to ensure the graders understand how to interact with the workflow - Conduct analyses to understand whether your changes had positive impacts and how to iterate on them in the future

Education & Experience

MS Degree or equivalent Apple is an equal opportunity employer that is committed to inclusion and diversity. We take affirmative action to ensure equal opportunity for all applicants without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, disability, Veteran status, or other legally protected characteristics.

Additional Requirements

  • Apple is an equal opportunity employer that is committed to inclusion and diversity. We take affirmative action to ensure equal opportunity for all applicants without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, disability, Veteran status, or other legally protected characteristics.
  • We will ensure that individuals with disabilities are provided reasonable accommodation to participate in the job application or interview process, to perform essential job functions, and to receive other benefits and privileges of employment. Please contact us to request accommodation.