The candidate will be working with programmers and coders to push for the compatibility of our compiler for Data Infrastructure with the Spark engine. Specifically, the candidate will be working on helping us to port the functionalities of our custom Hive and Spark function implementations to get their type signature, column lineage and a general understanding of how secure the functions are and also working on porting the equivalent functionalities of the Hive Query language into our compiler. The candidate will be working on understanding how our spark codebase works at facebook (which is mostly written in scala) and then re-writing the type evolution logic within our python framework of defining input output mappings. The candidate will work with the software engineers on our team to do these translations and ensure it’s correctness as compared to the currently running queries.
Knowledge of the following languages: SQL, Python and Scala. Experience in open source spark is preffered.
BS + 2 years experience, MS + 1 year experience, or PhD in Software Engineering