Hadoop SRE Engineer
Austin, Texas, United States
Software and Services
Apple’s Applied Machine Learning team has built systems for a number of large-scale data science applications. We work on many high-impact projects that serve various Apple lines of business. We use the latest in open source technology and as committers on some of these projects, we are pushing the envelope. Working with multiple lines of business, we manage many streams of Apple-scale data. We bring it all together and extract the value. We do all this with an outstanding group of software engineers, data scientists, SRE/DevOps engineers and managers.
- SRE Experience on Hadoop based technologies - HDFS/Yarn cluster administration, Hive, Spark
- Experience managing Hadoop/YARN clusters with thousands of nodes and 10’s of petabytes of data running 10’s of thousands of jobs
- Have a passion for automation by creating tools using Python, Java or other JVM languages
- Strong expertise in troubleshooting complex production issues.
- Expert understanding of Unix/Linux based operating system
- Excellent problem solving, critical thinking, and interpersonal skills
- Experience deploying and managing CI/CD pipelines
- Experience with Solr cluster administration
- Expertise in configuration management (such as Ansible, salt) for deploying, configuring, and managing servers and systems
- The candidate should be adapt at prioritizing multiple issues in a high pressure environment
- Should be able to understand complex architectures and be comfortable working with multiple teams
- Ability to conduct performance analysis and troubleshoot large scale distributed systems
- Should be highly proactive with a keen focus on improving uptime availability of our mission-critical services
- Comfortable working in a fast paced environment while continuously evaluating emerging technologies
- Proficient in UNIX, command-line tools, and general system debugging
- The position requires solid knowledge of secure coding practices and experience with the open source technologies.
Monitor production, staging, test and development environments for Hadoop/YARN clusters spanning thousands of nodes, in an agile and dynamic organization. You like to automate anything which you do and you documents it for the benefit of others. You are an independent problem-solver who is self-directed and capable of exhibiting deftness to handle multiple simultaneous competing priorities and deliver solutions in a timely manner. Provide incident resolution for all technical production issues. Create and maintain accurate, up-to-date documentation reflecting configuration, and responsible for writing justifications, training users in complex topics, writing status reports, detailing procedures, and interacting with other Apple staff and management. Provide guidance to improve the stability, security, efficiency and scalability of systems. Determine future needs for capacity and investigate new products and/or features. Strong troubleshooting ability will be used daily; will take steps on their own to isolate issues and resolve root cause through investigative analysis in environments where the candidate has little knowledge/experience/documentation. Administer and ensure the proper execution of the backup systems. Provide 24x7 on-call support to handle urgent critical issues.
Education & Experience
BS in computer science with 7-10 years or MS plus 5-7 years experience or related experience.
- Experience with Kubernetes, Docker Swarm, or other container orchestration framework
- Experience building and operating large scale Hadoop/spark data infrastructure used for machine learning in a production environment
- Experience in tuning sophisticated hive and spark queries
- Expertise in debugging Hadoop/spark/hive issues using NameNode, DataNode, NodeManager, spark executor logs.
- Experience in Capacity management on multi tenant Hadoop cluster
- Experience in Workflow and data pipeline orchestration (Airflow, Oozie, Jenkins etc.)
- Experience in jupyter based notebook infrastructure.
- Apple is an Equal Opportunity Employer that is committed to inclusion and diversity. We also take affirmative action to offer employment and advancement opportunities to all applicants, including minorities, women, protected veterans, and individuals with disabilities. Apple will not discriminate or retaliate against applicants who inquire about, disclose, or discuss their compensation or that of other applicants.
- Apple participates in the E-Verify program in certain locations as required by law.
- Apple's committed to working with and providing reasonable accommodation to applicants with physical and mental disabilities.