Design and develop of medium to highly complex bigdata data lake and setup real time machine learning/processing pipeline and loads them in to multiple data persistence stores to provide real time and offline analytics. Building indices at Elastic Search to support real-time dash boards at Kibana, and building predictive models to support AI/ML. Experience using Scala on Spark, especially building ETL and complex query models. Experience on Hadoop platform including Big Data tools. Experience in developing shell scripts and running cron/oozie/spark jobs on Hadoop platform. Experience with Kafka, hBase, and Hive. Experience on ElasticSearch and Kibana. Experience in accessing and modeling NoSQL data models, especially with Cassandra. Experience with APIs, JSON, OLTP and real-time data processing. Experience with Java Script. Working experience on Linux and Cloud platforms. Knowledge of Big Data Tools. Experience in building reports using graphana/kibana. Knowledge of business intelligence and analytics industry and best practices. Experience using data wrangling, data engineering, and feature engineering software. Experience in interpreting data models to build user friendly visualizations/dashboards. Experience in statistical techniques and quantitative methodologies that are used in decision making applications. Ability to work independently, and multi-task under short deadlines, based upon general direction. Effective verbal and written communication skills.
Big data engineer
- Responsible for building security analytics data lake
- Setup real time pipline using various big data tools (kafaka, java, spark stremaing, scala, Elastic search, hbase, hive)
- Build real time dashborad using graphna and kibana
- Analyse data and get insights on daily basis.
- Build machine learning pipeline to address frauds experience by verizon
MUST HAVE SKILLS:
3-4 years of JAVA/scala/python experience
1-2 years of SCALA experience
4 years hands on experence in Scala, Spark, Kafka, Realtime Streaming
Hadoop, Hive, NoSQL, SQL, Cassandra,
2 Experince with ELK / Splunk
Experience with streaming technologies
Candidate should be from programming background ( java/scala/python). No DB background
Highly scalable system architectures
CI/CD process in particular - GIT (Bitbucket), Jenkins, Jira, Confluence
Knowledge of Akka
AWS Developer certification
Apply now to have the opportunity to be considered for similar jobs at leading companies in the Seen network for FREE.
Zero stress and one profile that can connect you directly to 1000s of companies.
We’ll take it from there. After you tell us what you’re looking for, we’ll show you off to matches.
Boost your interview skills, map your tech career and seal the deal with 1:1 career coaching.
Join now and Be Seen.