Chummy Charms

Baby Steps

When LinkedIn started growing its member base, the site’s functionalities got complex by the day. In 2010, they decided to invest in redesigning the infrastructure to facilitate the blooming need of scaling their multiple data pipelines without much hassle. As a result, Kafka, a single, distributed pub-sub platform, was born to handle real-time data streams in each pipeline. In the very next year, Kafka went open-source under Apache and has been used in huge production scales ever since.

What is Apache Kafka?

Kafka is a fast, scalable, durable, fault-tolerant pub-sub messaging system. It’s written in Scala and Java, and uses Apache Zookeeper for reliable distributed coordination. Kafka provides four core APIs, namely Producer, Consumer, Streams, and Connector.

Kafka is composed of a few building blocks. Topic is the feed which represents a stream of records in Kafka. Each topic has one or more partitions which are physical separations of ordered and immutable sequence of records within a topic. Kafka uses the concept of a commit log to append each record, assigned with a sequential integer, the offset in a partition. Producers publish records to topics, while Consumers can subscribe to topics and consume the records. These messaging systems which maintain the published data are called Brokers. One or more such brokers compose a Cluster.

Real world use cases of Kafka

Over big data, fast data is becoming more of a household name lately, as companies are struggling to process real-time data streams. Since Kafka is capable of handling real-time data feeds with high throughput, low latency, and guaranteed reliability, more than a third of the Fortune 500 companies now use Kafka in production.

In its earlier stages, Kafka was used by LinkedIn only for online and offline real-time event consumption, traditional messaging use cases, gathering system health metrics, activity tracking, and feeding the data streams into their Hadoop grid. However, today, Kafka is a critical part of LinkedIn’s central data pipeline, handling over 1.4 trillion messages a day, as a whole ecosystem has been built around it.

Last year in Kafka Summit, impressive facts were revealed about the “Four Comma Club”. Netflix was one of the companies processing over 1 trillion messages a day using Kafka, apart from LinkedIn and Microsoft. This boasts the high scalability this distributed messaging system facilitates. In their Keystone pipeline, Netflix uses Kafka for complex event processing, and real-time monitoring.

Uber uses Apache Kafka in its core infrastructure for online and near real-time event processing. The list goes on and on, from traditional messaging, application monitoring and activity tracking, to usages in recommendation and decision engines, custom preferences and personalization, fraud detection, complex event processing, and ingesting data into Spark and Hadoop.

Why Kafka?

Handling enormous volumes of real-time data streams generated by systems like IoT has erupted pivotal challenges for enterprise giants. Precursory technologies and tools were not equipped to tackle the problems caused due to the scale and speed of these systems. This gave rise to a growing need for real-time analytics rather than traditional big data analytics. Apache Kafka is a fast, scalable, durable, fault-tolerant pub-sub data streaming platform, and hence is endowed to address many of these business problems.

Messaging with Kafka in UltraESB-X

In this article, we are going to discuss integrating Apache Kafka with UltraESB-X using UltraStudio, for real-time messaging. If you want to know more about UltraESB-X, this post would provide a good starting point. UltraStudio provides a graphical IDE to build, test and deploy integration projects without any fuss.

Use Case

Hogwarts is in deep waters lately. After Dolores Umbridge took over the headmastership, not even a pixie could flutter without her knowing. She made sure that school’s webmaster, Prof. Quirinus Quirrell is also dancing under her imperius. All he had to do is to monitor the web activities for the news articles and find which are gaining momentum. Original website looks as follows, with 0 read counts.

UltraESB-X takes care of the magic

Quirrell deligates this task to one of the Slytherin students, Gregory Goyle. As the solution for this, Goyle has decided to use UltraESB-X to integrate this web server with a Kafka server to provide real-time activity tracking. When the audience queries for the full news article as below, the request is used to populate an internal statistics engine to process read counts.

Let’s see step by step — how UltraESB-X takes care of the magic. Here is a final view of the website with desired functionalities.

Let’s break down the requirements into 3 key segments.

1. Retrieve the full news article and record the impression
2. Persist the records, passing data into Hadoop or data warehousing systems for real-time processing and reporting
3. Update the read count back on the server

Prerequisites

You can access the complete source for the website with functionalities from here, and the integration part will be covered throughout this article. Before going into details, make sure that you have the latest version — 17.07.1 — of UltraStudio with you. This tutorial assumes that you have Kafka and Zookeeper installed and started — and that there’s a Kafka topic that we can work with; if you are starting fresh, simply follow the first 3 steps of the Kafka quick-start guide.

Before delving into this simple integration, create a new project and have it ready for action. Your integration flow configuration files should reside in /src/main/conf.

Step-by-step Walkthrough

Segment 1:

First, we’ll see about recording the news article reads. For brevity, only the required configuration parameters are shown.

Step 1: Create a new integration flow.

Step 2: Add an NIO HTTP Ingress Connector from the connectors list and configure it as follows. This connector will be triggered when a news article’s “Read full article” button is clicked.

Step 3: Add an Extract HTTP Query Parameter processing element and configure it as follows. When the “Read full article” button of a news item is clicked, the unique ID of each news item is sent as a query parameter (i.e. http://localhost:8280/hogwarts/news?id=news1) from the website to the back-end to retrieve the full article from a back-end database, queue, file server or some other storage. This processor is used to extract out the unique ID from the request URL and assign it to a variable named key.

Step 4: Add a String Payload Setter processing element and configure it as follows. Now, this processor serves for 2 important requirements.

• including the unique ID of the news article into the message as a metadata for identification in further processing. This is done by composing an XML payload with the value of the key variable (which we injected into the flow in the previous step) added under an id element of the root message element.
• mocking the full article retrieval from a news storage, setting the value of the article element to be a static text — for the sake of simplicity.

The response is now ready for sending back to the website with the full article content, but for activity tracking, we are going to persist the message in a Kafka topic.

Step 5: Add a Kafka Egress Connector from the connectors list and configure it as follows.

NOTE: Bootstrap Servers and Topic Name should be changed according to your configuration. I configured mine with the default server details for a single node Kafka cluster and the topic I had created.

NOTE: Final processing element is used for enabling CORS — thus not a part of the integration logic.

Step 6: Add an Add New Transport Header processing element and configure it as follows.

Now we are all good with the first segment and the final integration flow for this segment 1 should look like this.

Segment 2:

Next, the messages stored in Kafka topics are ready to be ingested to Spark, Storm or any other streaming data processing engine. For comprehensibility, we’ll settle for processing the messages and injecting the read updates of the news articles into an internal statistics engine. In addition to that, we need a processing element to manipulate this statistics engine and update with new impressions.

Writing a custom processing element is easy in UltraStudio; if you need to know more, go through this. Here, we extract out the newsId value from the input and update the statistics engine in the processing element "Impression Injector" as follows.

For this custom processing element to be used effectively, the project should be built. After that, we are ready to create the integration flow for the second segment.

Step 1: Create a new integration flow.

Step 2: Add a Kafka Ingress Connector from the connectors list and configure it as follows. This connector is waiting to be triggered when a new record is published into the Kafka topic, which we published to in the first segment. Whenever it senses a new record, the message is consumed and injected into the integration flow for further processing.

Step 3: Add an XPath String Extractor processing element and configure it as follows. Now, if you do remember, we inserted a title element with the unique id for processing. This processor extracts the value of that element and assigns it to a variable named id.

Step 4: Add the Impression Injector custom processing element and configure it as follows. This processor takes care of reporting new impressions on the news article with the unique ID, which we extracted and assigned to variable id into the Statistics Engine for analytics.

We have the second segment ready and here’s a preview of the final integration flow for this segment.

Segment 3:

Now, we have injected the impression records into the Statistics Engine, but still, the read actions don’t get reflected on the website and it should be fixed. For that, we are going to create another back-end service which the website can be used to call to and retrieve the read counts. We need another custom processing element, “Read Count to Payload Setter” to retrieve the read counts from the Statistics Engine as follows.

With the project built to use our second custom processing element in effect for retrieval of read counts — this is the last straw — we are all geared up.

Step 1: Create a new integration flow.

Step 2: Add an NIO HTTP Ingress Connector from the connectors list and configure it as follows. This connector is used to retrieve the read counts for each news article from the back-end Statistics Engine.

Step 3: Add an Extract HTTP Query Parameter processing element and configure it as follows. We need to update the read count value next to the “Read Full Article” button when the user reads the full article and closes the popup. So, on this action, we bind a backend call with the same unique ID which we used to retrieve the article in the first place, setting it as a query parameter (i.e. http://localhost:8280/hogwarts/stats?id=news1). This processor is used to extract out the unique news ID from the request URL and assign it to a variable named key.

Step 4: Add a Read Count to Payload Setter processing element and configure it as follows. Using the custom processing element, the read count for the queried news article is retrieved from the Statistics Engine and set to the response payload.

NOTE: Just as we did in the Segment 1, we use one final processing element to enable CORS.

Step 5: Add an Add New Transport Header and configure it as follows.

Aaaand… that was the third segment — the home stretch — all the integration flows are good to go. Here’s how the final flow for the segment 3 should look like in UltraStudio.

You can test the flows you created within UltraStudio. Create a new “UltraESB-X Server” run configuration and start the server. Run the website in a simple HTTP server — Python provides a built-in web server; I use serve — and visit http://localhost:5000 in your favourite browser and see if it’s working.

You can also try out this exercise on UltraStudio as a preconfigured sample project.

It all started with the big bang 10 years 13.7 billion years ago. Since then, the earth has gone through many transformations as we witnessed the information age where the digital revolution is reshaping around us. What do you think this relentless quest has brought upon us? Your guess is as good as mine, The Imagination Age. Virtual Reality is just the beginning. But still, the knowledge workers dominate the economy; dissemination of information is of utmost importance. The irony is that I was too naive to contribute. Anyhoo, one of my mentors, Asankha Perera, CEO at Adroitlogic emphasized the power of sharing information at every turn.

Information used to play games with me throughout my academic career. I took the lazy man's road. Google was my friend all along; but, even Google seized feeble at times. My current employment at AdroitLogic put me to the test—I had to do a litmus test—So, I started blogging again; this time about the things that matter, things even my friend, Google couldn’t help me with. Anyways, here are some of the reasons that made me jumping on the bandwagon.

1. An insatiable craving for writing?

Now.. don’t get me wrong. I am not a writer. I’ve never been one because I didn’t want to. Every now and then my actions were slopped onto Google Docs, being saved for a rainy day—as I’m a victim of Transient Global Amnesia—at least I think I am. One thing led to another and now I’m writing.

Not all the writers are quirky recluses; nor gifted with super powers. But, you and I can be writers, if we decide to put an end to watching television since it's poisonous to creativity. You don't want poison anywhere near you, now do you? Writing is a tried-and-true way to enhance learning, organize thinking, improve communication, and develop creativity. Some of the articles I recently read really inspired me to write, thus materialized the long awaited blog. But, will this make me a good writer? The decision is up to you. I think you should write too.

2. Technology transfer

Drastic improvements in almost every sector have pushed idea commercialization to the next levels, mainly due to the technological uprise of enterprise giants. A Business Insider article describes that the food shortages will not be a problem by 2025 with the current rate of development. Technology and its transfer play a huge role in manifesting reality where these were mere dreams. One of the main goals of this blog was to share knowledge and get discussions on technology underway.

3. Noble cause

Have you ever spent days and nights struggling to find solutions for a technical problem, but once you found them you couldn’t understand them? Been there, done that—I’ve had my fair share of searching for the same thing over and over again; reading the same set of articles and answers again and again.

This blog will be about the things I bagged in my ugly endeavours, things I accidentally spilt onto Google Docs. I have had to take a peek and relive in some of those moments every so often. Why conceal the knowledge, when a world could benefit? Thus emerged, Chummy Charms. You will find a string of concernedly compiled articles which will help you to chase away the boogeyman in your nightmares.

In a career of technology, sometimes, following the footsteps is far more valuable than delving into an article with in-depth theories. This blog may include both, you may cherry-pick according to your need.

4. A right-brained guy?

I love to design, to create something that didn't exist. My obsession got worse, as I designed this blog for months including several cycles from the scratch; this piece of art needs some content though. Back in the day, when I was a freelancer, I helped loads of people to blog and live their dreams. Blogging boosts creativity, it definitely did mine. Creativity is proven to be good for health—relieving your stress, enhancing your brain function, nurturing your social life. Who in the right mind doesn’t want that, right? Read, and read more, subscribe to Chummy Charms, gather knowledge, start writing something of your own. Be creative. Be healthy.

5. Technology trends for power readers

Our living standard has been improved due to growing technological applications in day-to-day life. Every industry is boosted by the technology. Owing to this advancement, once mere dreams have become a reality in almost all the fields. Every techno professional must be updated with the technology these days in order to survive.

“Trends come and go, but the style remains forever”.

Learning new things is my passion, thus reading about various technologies and conducting my little experiments. I'm willing to share the little experience I gained over this little period of time making some strong content for the power readers.

6. Adding meaning to life

I’ve been wandering around the blogosphere since 2001 and wanted to be an active part of it for a long time. Then I started but was blogging about various ridiculous things for quite some time. It’s time I took this serious—the power of a blog. Chummy Charms is going to be like the journal of Bilbo Baggins, the diary of Tom Riddle, in which the adventures I encounter during my journey to the centre of the technology world are scratched down. I am open to suggestions, comments that make this more meaningful.

7. Be inspirational

It's no wonder that you and loads of others love to write but don’t have enough drive. I'm not a bookworm, but I read with the intention of being armed with new skills every day, and to admire the attractive work, the masterpieces. Creative blogger Dustin Curtis, the creator of Svbtle used to write striking articles over the years and heavily inspired me. There are life-changing blogs if you read more carefully. I hope I can inspire at least a few of you through Chummy Charms, not today, maybe one day.

8. Sharing

It’s no secret that Facebook, Twitter and other social sites convert others' ideas and thoughts into a constant stream of revenue. Information sharing opens access to a whole new level of individual and community development: promotes trust, enables better decisions, enhances analytical adroitness. If you use more carefully, it builds value. Honestly, I’m not the best at communication. According to my result of the Myers-Briggs test, I am an introvert. I know there are many like me out there. Yes.. I’m talking to you. Being engaged with a blog is a great medium to communicate your ideas. Start sharing your knowledge, add comments, show your love. There is internet full of people waiting to read your views.

9. For a better purpose

Blogging helps web scraping for modern information. Rather than wasting time on movies & games, this drives one to learn new things and sharpen the observational prowess. Crafting an article is not that easy, if it was, everyone would be doing that. It needs a lot of information in your pockets, learning about a wide range of areas, conducting tests on your own. This leads to proactive learning, where one can learn in a more constructive way.

10. Cultivating social life

Blogging gifts you freedom. It extricates the philosopher in you and helps to ease the pain of anyone who’s down in the dumps. More importantly, blogging benefits your social life bulging your network opening up to new and amazing opportunities.

As Winnie once said,

A good blog post should be like a woman’s skirt; long enough to cover the subject and short enough to keep it interesting!