What We Do
Professional Services Strategic Program Delivery Mission Critical Support Field EngineeringBy Technology / Company
What We Do
Professional Services Strategic Program Delivery Mission Critical Support Field EngineeringBy Technology / Company
ShareChat Mohalla Tech Private Limited , North Tower Smartworks ,Vaishnavi Tech Park, Survey No 16/1 & No 17/2 Ambalipura Village, Varthur Hobli, Bangalore East Taluk, Karnataka – 560103
Name | Speaker | Start Time | End Time | Presentation | Recording |
---|---|---|---|---|---|
Welcome and registration | 09:30 AM | 10:00 AM | |||
Realtime Triggers in Kestra | Shruti Mantri, Moveworks | 10:00 AM | 10:30 AM | Slides | YouTube |
Moving from Batch to Realtime : Inspired by a true production incident | Shivji Jha, Nutanix | 10:35 AM | 11:10 AM | Slideshare | YouTube |
Challenges & Learnings with using Kafka/Redpanda at a huge scale | Anuraj Jain, Sharechat | 11:15 AM | 11:50 AM | Slides | YouTube |
Networking break | 11:50 AM | 12:10 PM | |||
Unleashing Data Powerhouses - Benthos | Shivam Yadav & Shubham Dhal, Sharechat | 12:10 PM | 12:45 PM | Slides | YouTube |
Push Query Layer for Stream Processing Systems (Apache Flink) | Avinash Upadhyaya & Pavan Keshavamurthy, Platformatory | 12:45 PM | 01:20 PM | Slides | YouTube |
Lunch and Networking | 01:30 PM | 02:30 PM |
Speaker: Shruti Mantri, Software Engineer at Moveworks
About the talk:
Kestra can serve as a central tool to handle event driven architecture, along with being an orchestration tool.
Speaker: Shivji Jha, Staff Engineer at Nutanix
About the talk: This is a real world account from an Apache Druid cluster in production. A story of 48 hours of debugging, learning and understanding batch vs stream better, filing a couple of issues in Druid open source projects and finally a stable production pipeline again thanks to the Druid community. We will discuss what parts of your design could be impacted, how you should change the related systems so the cascading failures don’t bring down your complete production availability. As an example, we will discuss the bottlenecks we had in overlord, slot issues for Peons in middle managers, coordinator bottlenecks, how to mitigated task and segment flooding, what configs we changed sprinkled with real world numbers and snapshots from our Grafana dashboards.
Finally we will list all the leanings and how we made sure we never repeat the same mistakes in production systems.
A real world account of a production incident showing
Speaker: Anuraj Jain, Software Engineer at Sharechat
About the talk: Kafka is a industry standard data steaming platform and a lot of data intensive companies use it already for their streaming data use-cases, In this talk I am going to cover-up on our (ShareChat’s) Challenges & Leaning’s from using/operating Kafka/Redpanda at a heavy scale (GBs per sec) and overcoming problems with systems, causes for issues etc. It will be a really good leanings/best-practices session for the devs to avoid mistakes in production at a scale. ShareChat already operates big Kafka/Redpanda clusters in production and at a heavy scale. I have done a lot of firefighting around various issues and onboarding/migrating of our services/jobs to Kafka protocol recently.
Devs watching this presentation will be able to understand what all issues can come-up in production with operating Kafka at a high scale with the Kafka system and their services/jobs, they will be able to understand why the issues come-up and what is the resolution, how to deal with a problem and how to avoid mistakes in production.
Speakers: Shivam Yadav and Shubham Dhal, Software Engineers at Sharechat
About the talk: “In this session, we’ll unravel the story of ShareChat’s transition from Java to Benthos for crafting efficient ETL pipelines. With a single configuration file, Benthos effortlessly connects diverse sources and sinks, transforming the way we handle data. Discover how, armed with ““at least once”” guarantees, Benthos emerged as the go-to solution for our stateless pipelines.
Focused on real-world use cases, we’ll explore how Benthos, paired seamlessly with Kafka and Kafka Streams, became the linchpin of our operations. From sending notifications to triggering events for millions of users, to routing posts for reviews, Benthos proved its mettle in simplifying complex tasks.
Learn how we platformized Benthos at ShareChat, deploying over 20 jobs from a single repository. Delve into the specifics of how Benthos with Kafka and Kafka Streams powers our data pipelines, handling tasks such as dumping data to databases, making API calls, and routing events to different message queues. The session will highlight the simplicity and performance benefits we’ve achieved—processing 2K events/sec to an impressive 45K events/sec—without the need for extensive developer code.”
Speaker: Avinash Upadhyaya and Pavan Keshavamurthy, Platformatory
About the talk: One of the problems with stream processing systems is that they exist primarily in the streaming plane. These systems are only capable of processing data, which means it cannot hold an infinite amount of state, unlike databases. Some streaming processing systems work around this, usually by allowing some kind of interface to query the state store directly. But this usually a difficult problem to solve, mainly because the state stores are local and they involve the internals of the system, which aren’t necessarily meant to be exposed. Nonetheless, many stream processing systems do provide a work around to this problem - For example, in Kafka Streams, there is a way to perform interactive queries on the state store across different state stores.
Making this work with Apache Flink is an interesting problem to solve. In this talk, we will talk about how to make state stores queryable. Thus, providing a query layer on top of Flink and therefore embedding Flink into the operational plane.