- Data Science Recruiters and Staffing SpecialistsBig Data Engineer
Data Science Jobs
- AI Engineer
- AI Product Owner
- Algorithmic Trader
- Artificial Intelligence and Machine Learning Engineer
- Artificial Intelligence Product Owner
- BI Developer
- Big Data Engineer
- Business Intelligence Developer
- Creative Strategist
- Data Analyst
- Data Architect
- Data Engineer
- Data Engineering Manager
- Data Scientist
- Data Scientist, Deep Learning
- Data Solutions Architect
- Database Administrator
- Database Administrator (DBA)
- Database Developer
- Deep Learning Engineer
- Director of Analytics
- Director of Artificial Intelligence
- Director of Data
- Director of Machine Learning
- Engineering Manager, Machine Learning
- ETL Developer
- Fraud Analyst
- Hadoop Engineer
- Head of Data Engineering
- Head of Machine Learning
- Internet of Things Engineer
- Machine Learning Engineer
- ML/AI Engineer
- NLP Data Engineer
- NLP Data Scientist
- NLP Engineer
- Python Developer
- Python Engineer
- Risk Analyst
- Statistician
As the Big Data Engineer, you will be responsible for engaging in the design, development, and maintenance of the big data platform and solutions. This includes analytical solutions that provide visibility and decision support using big data technologies. The role involves administering a Hadoop cluster, developing data integration solutions, and working with data scientists, system administrators, and data architects to ensure the platform meets business demands.
Typical Duties and Responsibilities
- Develop ELT processes from various data repositories and APIs across the enterprise and ensure data quality and process efficiency
- Develop data processing scripts using Spark
- Develop relational and NoSQL data models to help conform data to meet users’ needs using Hive and HBase
- Integrate platform into existing EDW and various operational systems
- Develop administration processes to monitor cluster performance, resource usage, backup and mirroring to ensure a highly available platform
- Address performance and scalability issues in a large-scale data lake environment
Education
- Bachelor’s degree in computer science, information technology, or a related field or equivalent experience
- Master’s degree in computer science, information technology, or a related field or equivalent experience
Required Skills and Experience
- 2 years of experience with big data/Hadoop distribution and ecosystem tools, such as Hive, HBase, Spark, Kafka, NiFi and Oozie
- 2 years of experience developing batch and streaming ETL processes
- 2 years of experience working with relational and NoSQL databases, including modeling and writing complex queries
- Experience with programming languages, such as Python, Java or C#
- Experience with Linux system administration, Linux scripting and basic network skills
- Experience coding against and developing REST API’s