Lead Data Engineer - Azure & Databricks
New York, NY
| Employer: | NYU Langone Health |
| Category: | Health Care |
| Job Type: | Full Time |
Description | |
|
NYU Langone Health is a world-class, patient-centered, integrated academic medical center, known for its excellence in clinical care, research, and education. It comprises more than 200 locations throughout the New York area, including five inpatient locations, a children's hospital, three emergency rooms and a level 1 trauma center. Also part of NYU Langone Health is the Laura and Isaac Perlmutter Cancer Center, a National Cancer Institute designated comprehensive cancer center, and NYU Grossman School of Medicine, which since 1841 has trained thousands of physicians and scientists who have helped to shape the course of medical history. At NYU Langone Health, equity, diversity, and inclusion are fundamental values. We strive to be a place where our exceptionally talented faculty, staff, and students of all identities can thrive. We embrace diversity, inclusion, and individual skills, ideas, and knowledge. For more information, go to nyulangone.org, and interact with us on LinkedIn, Glassdoor, Indeed, Facebook, Twitter, YouTube and Instagram. Position Summary: We have an exciting opportunity to join our team as a Lead Data Engineer. In this role, the Enterprise Data and Analytics EDA department at NYU Langone Health plays a crucial role in modern healthcare organizations by leveraging data to enhance decision making optimize operations and improve patient outcomes. As a Senior Data Engineer within this department you will play a critical role in designing and developing robust ETL pipelines to integrate diverse data sources This role requires expertise in creating efficient data pipelines handling both streaming and batch data processing and ensuring data integrity throughout the ETL lifecycle The engineer will implement monitoring solutions optimize ETL jobs for performance and provide comprehensive support from data ingestion to final output With in depth knowledge of the Databricks platform and strong analytical skills this position will significantly enhance the department s ability to deliver high quality data driven insights and solutions Job Responsibilities:
Minimum Qualifications: To qualify you must have a Minimum of Bachelor s degree in Computer science Information systems Engineering or Data Science Minimum 10 years of experience in designing developing and optimizing ETL processes 5 years of experience in developing/supporting a data platform in Azure Databricks Proficiency in creating and maintaining efficient data pipelines on the Databricks platform Well rounded in working in a DEV Ops environment supporting processes in data platform in supporting business units and their use cases Strong verbal and written communication skills with the ability to explain complex technical concepts to non technical stakeholders Ability to troubleshoot and resolve issues in a timely manner Experience working in an Agile/Scrum environment Team player with the ability to work collaboratively in a cross functional team Strong analytical and problem solving skills Ability to work independently handle multiple tasks simultaneously and adapt quickly to change with a variety of people and work styles Willingness to learn new technologies and continuously improve skills Preferred Qualifications: In depth knowledge of Databricks platform and technologies including Delta Lake Databricks SQL and Databricks Workflows Experience with Azure cloud platforms and Azure Data Lake cloud storage Knowledge of data warehousing data modeling and best practices Proficiency in programming languages such as Python SQL Scala or R Experience with big data technologies such as Apache Spark Hadoop or Kafka Knowledge of working with Hadoop for migration projects from Hadoop to Databricks Familiarity with DevOps practices and tools such as CI/CD Git etc Knowledge of infrastructure as Code IaC tools like Terraform Experience with implementing data governance and security measures in a cloud environment Qualified candidates must be able to effectively communicate with all levels of the organization. NYU Langone Health provides its staff with far more than just a place to work. Rather, we are an institution you can be proud of, an institution where you'll feel good about devoting your time and your talents. NYU Langone Health is an equal opportunity and affirmative action employer committed to diversity and inclusion in all aspects of recruiting and employment. All qualified individuals are encouraged to apply and will receive consideration without regard to race, color, gender, gender identity or expression, sex, sexual orientation, transgender status, gender dysphoria, national origin, age, religion, disability, military and veteran status, marital or parental status, citizenship status, genetic information or any other factor which cannot lawfully be used as a basis for an employment decision. We require applications to be completed online. If you wish to view NYU Langone Health's EEO policies, please click here. Please click here to view the Federal "EEO is the law" poster or visit https://www.dol.gov/ofccp/regs/compliance/posters/ofccpost.htm for more information. NYU Langone Health provides a salary range to comply with the New York state Law on Salary Transparency in Job Advertisements. The salary range for the role is $81,325.15 - $135,541.73 Annually. Actual salaries depend on a variety of factors, including experience, specialty, education, and hospital need. The salary range or contractual rate listed does not include bonuses/incentive, differential pay or other forms of compensation or benefits. To view the Pay Transparency Notice, please click here | |