Associate Systems Software Engineer at Crusoe in San Francisco, CA

Summary: Crusoe is an AI infrastructure company focused on enhancing energy and intelligence for AI workloads. The Associate Systems Software Engineer will be responsible for designing and developing compute applications for virtualized AI platforms, integrating hardware and software, and ensuring system performance and reliability. Responsibilities: - Design highly reliable and performant Linux applications used to manage our virtualization stack across thousands of AI compute servers in multiple global datacenters - Integrate Crusoe applications with a wide variety of hardware and software AI chip-vendor stacks - Build solutions to optimize and monitor virtualized hardware (GPUs, Infiniband/ROCe NICs, Ephemeral Storage, etc.) in cutting-edge AI/HPC environments - Work side by side with our Linux Kernel and Hypervisor teams to ensure our Crusoe applications are seamlessly integrated with a variety of kernels and hypervisors - Analyze and enhance the performance of the entire virtualization stack, from the hypervisor to the virtualized guest OS, with a specific focus on optimizing AI/ML workloads - Diagnose and resolve complex system issues across our virtualization stack (drivers, kernel, hypervisor, guest OS, and crusoe applications) - Conduct thorough code reviews to ensure the highest level of software quality, reliability, and security within compute applications and virtualization stack - Collaborate with other engineering teams, including hardware design, OS development, and AI/ML application teams, to ensure cohesive and integrated product development - Provide technical guidance and mentorship to junior engineers, fostering a culture of technical excellence and collaborative problem-solving within the compute applications team Required Qualifications: - Experience building applications on Linux kernels, specifically pertaining to virtualization, device drivers, memory management, and process scheduling - Solid understanding of hardware devices such as GPUs, CPUs, Infiniband and Ethernet NICs, Ephemeral Disks, and PCI Express - Strong grasp of distributed applications and highly-scalable systems design. Specific focus around communications protocols (GRPC, REST, TCP/IP, etc.), databases (Postgres, Redis), and systems design applications (Pub/Sub, Kafka) - Strong experience building software applications, both at the higher (Golang, Java, Python) and lower (C, C++, Rust) levels. Keen eye for clean, maintainable code, and a unit-test driven mindset - Ability to collaborate with teams across an organization, blocking out noise, and focusing on what needs to get done to get a project across the line - Capable of adapting quickly, eager to research new technology and not get overwhelmed by unfamiliar tech stacks - General knowledge of hypervisors, virtual machine lifecycles, and Linux KVM tooling - Understanding of how to build Gitlab or Github CI/CD pipelines that deliver bug-free code across a multitude of compute platforms Preferred Qualifications: - Experience with virtualization specifically for AI/ML workloads, including GPU virtualization - Previous work debugging or contributing to kernel or hypervisor code, particularly around device management - Experience with configuring thousands of live compute nodes in a bare-metal production environment Required Skills: Linux kernel, Virtualization, Hardware tuning, Distributed systems, Object oriented programming, Low-level systems programming, Linux systems, Device drivers, Memory management, Process scheduling, Hardware integration, GPUs, CPUs, Infiniband, Ethernet NICs, Ephemeral storage, PCI Express, Systems design, Communications protocols, GRPC, REST, TCP/IP, Databases, Postgres, Redis, Software architecture, Golang, Java, Python, C, C++, Rust, Unit testing, Virtual machine lifecycle, Linux KVM, CI/CD pipelines, Gitlab, Github Benefits: Competitive compensation, Restricted Stock Units, Paid time off & paid holidays, Comprehensive health, dental & vision insurance, Employer contributions to HSA account, Paid parental leave, Paid life insurance, short-term and long-term disability, Professional development & tuition reimbursement, Mental health & wellness support, Commuter benefits (parking & transit), Cell phone stipend, 401(k) Retirement plan with company match up to 4% of salary, Volunteer time off