Chummy Charms

Political pioneers all over the world keep on staying dominant and prevailing powers in worldwide matters. It's no surprise that US President Donald Trump is one of the most influential people who remains popular across the news due to his radical decisions. Even the globe's most formidable can't just ignore the fact that analytics play a principal role in everything. However, we ain't here about Trump or his propaganda—we are here about the future of decision making for all it matters.

About a year ago, we started being a part of the digital transformation with the first ever cloud-based IDE for serverless development. It was no cakewalk—we’ve been burning the candle at both ends trying to cover a majority from the AWS’s serverless stack. Working with AWS Kinesis made me realise the beauty of serverless—of course, the exposure to streaming data with Kafka spared me some time going through the rudiments.

Long story short

Did you ever wonder...

• How "Google Search" suggests you when you’re half-typing your query?
• How "Cheapest Airlines" starting to appear everywhere after you searched for a country?
• How online role-playing games adjust according to your decisions?
• How do gambling sites predict the odds of a live game?
• Why were Curry and Thompson benched while Portland was handing the Warriors their worst loss in 73-win NBA season?

The power of real-time streaming data analytics is astonishing indeed. Now, since serverless technology is gaining some momentum, maybe you won’t have to worry about taking risky decisions on your own at all. This post covers the basics of "Serverless Streaming Data Processing" and how it will be an influential component of our decision making in the future.

A Series of Streaming Events

Life is an endless series of events. The technology around us has made it a stream of digital actions emitting streams of data. If you turn back and investigate your life very carefully, you'll see the never-ending string of data you have generated with your every digital action. It could be a lot to digest at first, but let’s explore some scenarios and try to find what applies to you and me.

• Online banking and convenient e-commerce purchasing capabilities
• Ride-sharing, modern-day travelling and transportation
• Industrial equipment and agricultural use cases like monitored machinery, autonomous tractors and precision farming
• Automated power generation and smart grids, Zero-net Buildings, Smart metering
• Real-estate property recommendations based on geo-location, predictive maintenance
• Online dating and matchmaking relying on complex personality patterns and attribute distribution
• Financial trading according to the real-time changes in the stock market, analytical risk management
• Movies, songs and other digital media with a better experience depending on the demographics, preference, and emotions
• Improved web and mobile application experience based on usage
• Dynamic and personalised experiences in online gaming
• Enhanced social media experiences with hyper-personalisation and predictive analytics
• Telemetry from connected devices, or remote data centres from geospatial or spatial services like weather, resource assessment
• Sports analytics to enhance the players’ performance reducing health risks

All these events produce data—lots of it. Due to the frequency of this data emission, it has become an increasing burden to the digital space.

Streaming Data

In a survey conducted last year about data, it’s estimated that with the current pace of data generation,

1.7 MB of data will be created every second for every person on earth by 2020

Data that is poured continuously by a gazillion sources every second has become a fact we can’t just ignore. Big Data disciplinary was an eye-opener for the tech world to apply this once irritating data to do something useful. This same irksome data is collected and analysed by a new species, namely data scientists 😛. Due to the nature of continuity and often being in small sizes (order of Kilobytes) these data flows—usually referred by the moniker streaming data—are collected simultaneously as records and sent in for further processing.

Stream Processing and decision making

A streaming data processing structure usually comprises of two layers—a storage layer and a processing layer. The former is responsible for ordering large streams of records and facilitating persistence and accessibility at high speeds. The processing layer takes care of data consumption, executing computations, and notifying the storage layer to get rid of already processed records. Data processing is done for each record incrementally or by matching over sliding time windows. Processed data is then, subjected to streaming analytics operations and the derived information is used to make context-based decisions. For instance, companies can track public sentiment changes on the products by analysing social media streams continuously—world's most influential nations can intervene in decisive events like presidential elections in other powerful countries—mobile apps can offer personalised recommendations for products based on geo-location of devices, user emotions.

Most applications collect a portion of their data at the outset to produce simple summary reports and take simple decisions such as triggering alarms or calculating a moving average value. When the time flies by, these become more and more sophisticated, and companies might want to access profound insights to perform intricate activities in turn with the aid of Machine Learning algorithms and data analysis techniques. The continual growth of data has made data scientists work around the clock to come up with trailblazing solutions to utilise as much data as possible to fabricate alternate futures with better decisions.

How the world embraces it

Many companies use insights from stream analytics to enhance the visibility of their businesses which allows them to deliver customers a personalised experience. Additionally, near real-time transparency gives these firms the flexibility to promptly address emergencies. The emerging serverless architecture has driven all the leading cloud service platforms to present complementary solutions. Stream processing was made available for serverless application development with fully-managed, cloud-based services for real-time data processing over large Distributed Data Streams.

1. Entertainment got upgraded!

Netflix, the leading online television network in the world, developed a solution which centralises their flow logs using Amazon Kinesis Streams. As a system processing billions of traffic flows every day, this eliminates plenty of complexity for them because of the absence of a database in the architecture. Due to the high scalability and lightning speed, they can discover and address issues as they arise, monitor the application on a massive scale. With the upgraded recommendation algorithm, video transcoding, and licensing popular media, this subsequently grants a seamless experience to the subscribers. With the exponential growth of the subscribers, the company’s responsibilities increase by the day. However, nothing seems to be a problem for Netflix for many years to come since they are considered to have a sound decision-making model.

2. Improving the decisions of the decision makers

As a leading source of integrated and intelligent information for businesses and professionals, Thomson Reuters provide their services to decision makers in a wide range of domains like financing and risk, science, legal, technology. This company built an in-house analytics engine to take full control of data and moved to AWS because they were familiar with its capabilities and scale. The new real-time pipeline attached to Amazon Kinesis stream produces better results in perceptive customer experience with accurate economic forecasts, financial trends for beneficiaries including a range of government activities.

3. Unicorn: a solution to traffic congestion

Jakarta has become a heavily congested city where the motorcycle deemed the most efficient mode of transport. To exploit this business opportunity, GO-JEK—one of the few unicorn businesses in Southeast Asia—started as a call centre for motorcycle taxi bookings. However, to meet the demand in exceeding expectations, the company had to consider expansion. Now with the support of Google Cloud Professional Services, the business architecture built on Cloud Dataflow for stream inference enables them to predict changes in demand effectively.

There are more stories about how cloud platforms like AWS, Google, Microsoft Azure, and IBM Cloud are exploited by companies to make their clients’ lives better and secure.

Shortcomings of Serverless Stream Processing

Serverless stream processing is increasingly becoming a vital part of decision-making engines. However, with the current set of features, it’s not the ideal solution for some scenarios. Implementing real-time analytics for sliding windows and temporal event patterns is not a course for the faint-hearted.

The best way to assimilate never-ending data of this magnitude is through real-time dashboards which requires additional data organisation and persisting. These manoeuvres introduce undesirable latency and data management issues into the context. However, technology is evolving and trying to catch up to the speeds with integration using advanced cloud data management techniques to produce materialised views.

Stream Processing often uses a time-based or record-based window to be processed in contrast to the batch-based processing, which can lead to challenges in use cases that require query re-execution.

Nowadays, application requirements grow beyond aggregated analytics. Increasing the window size seems to be an appropriate temporary solution but, it develops another intractable problem—Memory Management. Modern-day solutions usually provide advanced memory management and scheduling techniques to overcome this, but the world will see further improvements.

Conclusion

All in all, it’s apparent that serverless stream processing has been playing a prominent role around us without us even knowing. With the power of serverless data stream processing, applications can evolve from traditional batch processing to real-time analytics. The revelation of profound insights will result in effective decision making without having to manage infrastructure. Even today, many organizations practise orthodox decision-making strategies based on the analytics derived using the big data clusters that belonged to THE PAST. New horizons of serverless and real-time data processing are now equipped with the power to make effective decisions and create a—more productive, relevant and most importantly secure—world around you.

So do you still think Trump is more powerful?

What do you think?

Share your thoughts. Kudos this.

Originally published at SLAppForge Blog.

Serverless is a misnomer. A perception. It was not materialized until it became a household name, a few years ago. If you’d ask whether there are servers in serverless, the answer—unsurprisingly—is a BIG yes. Now, the million-dollar question—do you have to worry about them?

No.

Back in the day, it was all just bare-metal servers, which were kept on-premises and managed by the organizations themselves. Then came the virtual machine, a piece of software that mimics the hardware-based servers. Cloud was the next big thing inspired by the distributed computing platforms. Containerization, a.k.a. OS-level virtualization came along and gained popularity afterwards. However, later, IaaS (Infrastructure as a Service) had covered some ground due to the emerging popularity of the cloud by then. Provisioning servers became automated with PaaS as the next step of IaaS.

Technology continually evolves, like it or not, so does everything around us. In the last decade, we saw drastic improvements in connectivity, compute power, storage, etc. Serverless started appearing more and more (even got us Baader-Meinhoffed) after Business Development VPs started making impact stories and this continued the trend of PaaS (Platform as a Service) with some significant paradigm shifts. With Amazon Web Services launching Lambda at 2014 ReInvent conference, serverless became inevitable. By today, it is one of the trending topics in the software architecture world.

Diving into Serverless

Serverless computing is a cloud computing model of which applications and services can be built and run without managing any servers. As such, the name has been derived corroborating the fact that provisioning, scaling, and managing of servers are entirely hidden from the operator. Serverless incorporates two different but overlapping realms—BaaS, shorthand for Backend as a Service, and FaaS, shorthand for Function as a Service.

Serverless is not a new kid on the block, even though some misapprehends it to be a relatively new concept. Zimki started offering the first “pay per use” code execution platform in 2006—before they withdrew the service at the end of 2007. Heroku by Salesforce has been silently operating their web application deployment model since 2007. Google App Engine was released in April 2008 with limited features by Google. It started out as a custom Python execution framework and now turned out to be one of the favourite cloud platforms for web application development and hosting.

Brave New Waves

With the event-driven wave hitting the coast, ephemeral, stateless compute containers came into play. Unlike traditional architectures, these compute engines were invoked only once by a 3rd-party—and killed. People started calling monikers—Functions as a Service (FaaS) emerged. This was briskly leveraged by the developers for focusing and implementing own business logic triggered by specific events.

FaaS completely abstracts the infrastructure layer from the developer charging only for the time of use. Quoting AWS Lambda,

Run code without thinking about servers. Pay only for the compute time you consume.

FaaS market is driven mainly by the shift from DevOps to serverless computing lately. Global forecasts predict the serverless market growing at a rate of 32.7%. Industry experts have some enticing views about the progression of serverless technology too.

Serverless, the hero

Introduction of serverless eliminates the woe to manage clusters of servers and the traditional “always on” server requirement. The underlying fleet of servers scales gracefully on its own, provisioning appropriate infrastructure, even to meet massive surges in traffic. They are regularly monitored and managed by the providers—also security patches and other immunization shots are taken care of. This cuts a considerable portion of the budget as it results in notable reductions in operational cost, not to mention the hassle.

In serverless, you only pay as you use; the invocations are billed by fractions of seconds. Implementing a production-ready output from a mere idea is super-fast. Developer productivity for a first-time experience is a significant gain compared to struggling over different software stacks set up internally. AWS and pretty much all the other serverless solution providers give free tier options and come as language agnostic endpoints (except for Google Cloud, others provide multiple runtime/development environments) which makes the first-time hands-on experiences much more relieved.

Serverless, the villain

This does not mean that serverless is all god-like; it might include some hidden costs incurred. Usually, these costs don’t encompass only CPU and RAM usage, but the number of API requests, storage, and network might get their fair share. Sometimes, according to your specific business requirement, an implementation with multiple functions may be more difficult than having one with a single container. Code maintenance may incapacitate you, due to the distributed nature of your overall logic--or for more complex use cases, going with Kubernetes might be advantageous since it’s more mature.

Lack of integration testing facilities on localhost is a huge downside of the existing providers. There is no easy way to emulate but can be overcome to a certain extent by using Serverless Framework or aws-sam-local. Manual machine tuning befitting for your application is impossible because the server layer is intangible.

Provider lock-in can be challenging when moving forward since each serverless platform is a far cry from each other based on their uniqueness and limitations. Because of the cold starts, applications may experience high latency at the beginning.

Serverless Competition

AWS is not the only FaaS provider in the market. There is a handful of other enterprise giants competing for the fattening market. Microsoft Azure enhances event-driven serverless compute experience with Azure Functions. IBM’s serverless domain provides IBM Cloud Functions which is based on Apache OpenWhisk, an open source project. Google Cloud Functions is the Google Cloud Platform’s version of FaaS, which is still in beta.

Different providers provide a stack of fully managed products in their serverless platforms. These backend services require no provisioning, maintaining, or administration. Here’s a depiction of the various components in each serverless platform.

Apart from these, Firebase is the Google Cloud Platform backed (acquired, as usual, in 2014) mobile and web application development platform which consists of multiple integrated services. Most of the serverless providers showcase IoT framework support, mobile backend support and cognitive application development support as well.

Most Amazon web services have availability zone specific endpoints to compensate the latency in the applications around the globe. Developers have the freedom to use these regional endpoints or the default North Virginia endpoint. You can read more about AWS services that support particular regions in here. Just like that, other serverless providers support different availability zones as well.

AWS Lambda and other services have SDKs that support serverless development in multiple programming languages. Amazon recently (2018 January) announced Go support. Read this article to know a great deal about a detailed comparison of different serverless providers.

Many faces of Serverless

Serverless applications let you run code without provisioning or managing servers. Serverless domain empowers a myriad of application opportunities. Some of them are divulged here as follows.

1—Real-time data processing
Real-time processing is found to be coined widely with serverless. Image resizing for viewing on different devices such as desktop computers, smartphones, tablets are done by The Seattle Times using AWS Lambda. SiteSpirit uses IBM cloud data services to build their cloud-based media library.

2—Real-time stream processing
Localytics uses Lambda to process billions of historical and live social media trend data for business users to query. IBM Cloud Functions can be integrated with Apache Kafka supported IBM Message Hub for processing stream data. Plexure, a company engaged in innovative retail businesses based in New Zealand, uses Azure serverless technologies to bring customer experience to the next level—with the help of Azure Functions and Azure Logic Apps, now customers get alerts about the in-store digital display.

3—Extract, Transform, Load (ETL)
Zillow, one of the largest real-estate brands utilizes AWS serverless stack to provide customers with a smooth experience in near-real-time home valuation.

4—IoT Backends
In the IoT domain, there is an increasing growth of applications using serverless platforms. This exciting story explains how GreenQ, a smart residential waste collection company, exploits IBM Bluemix technologies to engage in a fast and more efficient service.

5—Mobile Backends
Bustle.com is a website catering to women which provides news, fashion and many more. They use Lambda and API Gateway for its iOS and website backend. With the move to new serverless architecture, developers no longer worry about managing and provisioning infrastructure, all they focus on is innovating. The mobile app WeatherGods uses IBM Cloud Functions (former IBM OpenWhisk) to notify users about specific weather events (hurricane in FloridaColombo, for example) aligning god avatars according to their preferences.

6—Serverless web applications
TimerCheck.io is a “deceptively simple web service with superpowers” in Eric Hammond’s words. The service is entirely built on Amazon API Gateway and AWS Lambda, and it provides an unlimited number of timers that can run infinitely.

7—SaaS event processing
San Joaquin Valley College (SJVC) is a private junior college in California. To simplify the IT management and provide the students with a better experience, they moved their learning management system to Microsoft Azure and Office 365. With the help of Microsoft FastTrack, SJVC embraced the serverless technology and continued their journey of innovation.

Apart from these most popular ones, you can find more use cases of AWS, Azure, and IBM.

Empowering serverless development

Software architecture world is steadily being absorbed by serverless. It’s not going to be “Serverless Everything” anytime soon as there are still many blanks to be filled. Tooling is improving, but again, there are still gaps to be bridged. The earlier mentioned Serverless framework provides a CLI tool that lets users to be integrated with multiple cloud providers.

Cloud9, once an independent cloud-based IDE was acquired by Amazon to plug development aspect into its serverless stack. However, setting up a Cloud9 development environment requires a new Amazon EC2 instance or your own Linux server. The getting started process is reported to be more complicated than the original Cloud9 (c9.io) by many users. Check this neat trick attempting to come up with a self-sufficient AWS lambda function, which quashes the need of uploading a bundle with third-party dependencies — built locally. Microsoft Azure is powered by Visual Studio, another versatile but heavy cloud IDE with team services.

Think Serverless

SLAppForge, a totally new player, recently announced its beta release of Sigma, a customized editor for the development and deployment of serverless applications. It’s a hybrid model exploiting the beauty of drag-and-drop style and the full power of raw code. Sigma hides the underlying complexities of the platform from the developers using its own, “very” serverless architecture—hich means, now you don’t have to dig into piles and piles of documentation and tutorials.

This powerful IDE is equipped with intuitive code suggestions, context-aware code completion, auto-generated code snippets—still supports only Node.js though. It also takes care of the Continuous Integration, and Continuous Delivery flows of your serverless applications. If you have a web browser (I bet you already do!), a GitHub account and an AWS account, you’re all set—because there’s nothing to install. Deploying your serverless application is minutes away now, not hours, not days.

Conclusion

The paradigm shift to serverless is becoming inescapable by the day, just like was the change from on-premise to cloud. The Big Day that all applications go serverless is not far away. It’s high time to jump on the bandwagon and be a serverless developer since everything is propelling you to be more productive and efficient. Keep aside the worries about infrastructure; focus fully on the business logic. The ball is in your court; jumpstart your serverless development now!

Baby Steps

When LinkedIn started growing its member base, the site’s functionalities got complex by the day. In 2010, they decided to invest in redesigning the infrastructure to facilitate the blooming need of scaling their multiple data pipelines without much hassle. As a result, Kafka, a single, distributed pub-sub platform, was born to handle real-time data streams in each pipeline. In the very next year, Kafka went open-source under Apache and has been used in huge production scales ever since.

What is Apache Kafka?

Kafka is a fast, scalable, durable, fault-tolerant pub-sub messaging system. It’s written in Scala and Java, and uses Apache Zookeeper for reliable distributed coordination. Kafka provides four core APIs, namely Producer, Consumer, Streams, and Connector.

Kafka is composed of a few building blocks. Topic is the feed which represents a stream of records in Kafka. Each topic has one or more partitions which are physical separations of ordered and immutable sequence of records within a topic. Kafka uses the concept of a commit log to append each record, assigned with a sequential integer, the offset in a partition. Producers publish records to topics, while Consumers can subscribe to topics and consume the records. These messaging systems which maintain the published data are called Brokers. One or more such brokers compose a Cluster.

Real world use cases of Kafka

Over big data, fast data is becoming more of a household name lately, as companies are struggling to process real-time data streams. Since Kafka is capable of handling real-time data feeds with high throughput, low latency, and guaranteed reliability, more than a third of the Fortune 500 companies now use Kafka in production.

In its earlier stages, Kafka was used by LinkedIn only for online and offline real-time event consumption, traditional messaging use cases, gathering system health metrics, activity tracking, and feeding the data streams into their Hadoop grid. However, today, Kafka is a critical part of LinkedIn’s central data pipeline, handling over 1.4 trillion messages a day, as a whole ecosystem has been built around it.

Last year in Kafka Summit, impressive facts were revealed about the “Four Comma Club”. Netflix was one of the companies processing over 1 trillion messages a day using Kafka, apart from LinkedIn and Microsoft. This boasts the high scalability this distributed messaging system facilitates. In their Keystone pipeline, Netflix uses Kafka for complex event processing, and real-time monitoring.

Uber uses Apache Kafka in its core infrastructure for online and near real-time event processing. The list goes on and on, from traditional messaging, application monitoring and activity tracking, to usages in recommendation and decision engines, custom preferences and personalization, fraud detection, complex event processing, and ingesting data into Spark and Hadoop.

Why Kafka?

Handling enormous volumes of real-time data streams generated by systems like IoT has erupted pivotal challenges for enterprise giants. Precursory technologies and tools were not equipped to tackle the problems caused due to the scale and speed of these systems. This gave rise to a growing need for real-time analytics rather than traditional big data analytics. Apache Kafka is a fast, scalable, durable, fault-tolerant pub-sub data streaming platform, and hence is endowed to address many of these business problems.

Messaging with Kafka in UltraESB-X

In this article, we are going to discuss integrating Apache Kafka with UltraESB-X using UltraStudio, for real-time messaging. If you want to know more about UltraESB-X, this post would provide a good starting point. UltraStudio provides a graphical IDE to build, test and deploy integration projects without any fuss.

Use Case

Hogwarts is in deep waters lately. After Dolores Umbridge took over the headmastership, not even a pixie could flutter without her knowing. She made sure that school’s webmaster, Prof. Quirinus Quirrell is also dancing under her imperius. All he had to do is to monitor the web activities for the news articles and find which are gaining momentum. Original website looks as follows, with 0 read counts.

UltraESB-X takes care of the magic

Quirrell deligates this task to one of the Slytherin students, Gregory Goyle. As the solution for this, Goyle has decided to use UltraESB-X to integrate this web server with a Kafka server to provide real-time activity tracking. When the audience queries for the full news article as below, the request is used to populate an internal statistics engine to process read counts.

Let’s see step by step — how UltraESB-X takes care of the magic. Here is a final view of the website with desired functionalities.

Let’s break down the requirements into 3 key segments.

1. Retrieve the full news article and record the impression
2. Persist the records, passing data into Hadoop or data warehousing systems for real-time processing and reporting
3. Update the read count back on the server

Prerequisites

You can access the complete source for the website with functionalities from here, and the integration part will be covered throughout this article. Before going into details, make sure that you have the latest version — 17.07.1 — of UltraStudio with you. This tutorial assumes that you have Kafka and Zookeeper installed and started — and that there’s a Kafka topic that we can work with; if you are starting fresh, simply follow the first 3 steps of the Kafka quick-start guide.

Before delving into this simple integration, create a new project and have it ready for action. Your integration flow configuration files should reside in /src/main/conf.

Step-by-step Walkthrough

Segment 1:

First, we’ll see about recording the news article reads. For brevity, only the required configuration parameters are shown.

Step 1: Create a new integration flow.

Step 2: Add an NIO HTTP Ingress Connector from the connectors list and configure it as follows. This connector will be triggered when a news article’s “Read full article” button is clicked.

Step 3: Add an Extract HTTP Query Parameter processing element and configure it as follows. When the “Read full article” button of a news item is clicked, the unique ID of each news item is sent as a query parameter (i.e. http://localhost:8280/hogwarts/news?id=news1) from the website to the back-end to retrieve the full article from a back-end database, queue, file server or some other storage. This processor is used to extract out the unique ID from the request URL and assign it to a variable named key.

Step 4: Add a String Payload Setter processing element and configure it as follows. Now, this processor serves for 2 important requirements.

• including the unique ID of the news article into the message as a metadata for identification in further processing. This is done by composing an XML payload with the value of the key variable (which we injected into the flow in the previous step) added under an id element of the root message element.
• mocking the full article retrieval from a news storage, setting the value of the article element to be a static text — for the sake of simplicity.

The response is now ready for sending back to the website with the full article content, but for activity tracking, we are going to persist the message in a Kafka topic.

Step 5: Add a Kafka Egress Connector from the connectors list and configure it as follows.

NOTE: Bootstrap Servers and Topic Name should be changed according to your configuration. I configured mine with the default server details for a single node Kafka cluster and the topic I had created.

NOTE: Final processing element is used for enabling CORS — thus not a part of the integration logic.

Step 6: Add an Add New Transport Header processing element and configure it as follows.

Now we are all good with the first segment and the final integration flow for this segment 1 should look like this.

Segment 2:

Next, the messages stored in Kafka topics are ready to be ingested to Spark, Storm or any other streaming data processing engine. For comprehensibility, we’ll settle for processing the messages and injecting the read updates of the news articles into an internal statistics engine. In addition to that, we need a processing element to manipulate this statistics engine and update with new impressions.

Writing a custom processing element is easy in UltraStudio; if you need to know more, go through this. Here, we extract out the newsId value from the input and update the statistics engine in the processing element "Impression Injector" as follows.

For this custom processing element to be used effectively, the project should be built. After that, we are ready to create the integration flow for the second segment.

Step 1: Create a new integration flow.

Step 2: Add a Kafka Ingress Connector from the connectors list and configure it as follows. This connector is waiting to be triggered when a new record is published into the Kafka topic, which we published to in the first segment. Whenever it senses a new record, the message is consumed and injected into the integration flow for further processing.

Step 3: Add an XPath String Extractor processing element and configure it as follows. Now, if you do remember, we inserted a title element with the unique id for processing. This processor extracts the value of that element and assigns it to a variable named id.

Step 4: Add the Impression Injector custom processing element and configure it as follows. This processor takes care of reporting new impressions on the news article with the unique ID, which we extracted and assigned to variable id into the Statistics Engine for analytics.

We have the second segment ready and here’s a preview of the final integration flow for this segment.

Segment 3:

Now, we have injected the impression records into the Statistics Engine, but still, the read actions don’t get reflected on the website and it should be fixed. For that, we are going to create another back-end service which the website can be used to call to and retrieve the read counts. We need another custom processing element, “Read Count to Payload Setter” to retrieve the read counts from the Statistics Engine as follows.

With the project built to use our second custom processing element in effect for retrieval of read counts — this is the last straw — we are all geared up.

Step 1: Create a new integration flow.

Step 2: Add an NIO HTTP Ingress Connector from the connectors list and configure it as follows. This connector is used to retrieve the read counts for each news article from the back-end Statistics Engine.

Step 3: Add an Extract HTTP Query Parameter processing element and configure it as follows. We need to update the read count value next to the “Read Full Article” button when the user reads the full article and closes the popup. So, on this action, we bind a backend call with the same unique ID which we used to retrieve the article in the first place, setting it as a query parameter (i.e. http://localhost:8280/hogwarts/stats?id=news1). This processor is used to extract out the unique news ID from the request URL and assign it to a variable named key.

Step 4: Add a Read Count to Payload Setter processing element and configure it as follows. Using the custom processing element, the read count for the queried news article is retrieved from the Statistics Engine and set to the response payload.

NOTE: Just as we did in the Segment 1, we use one final processing element to enable CORS.

Step 5: Add an Add New Transport Header and configure it as follows.

Aaaand… that was the third segment — the home stretch — all the integration flows are good to go. Here’s how the final flow for the segment 3 should look like in UltraStudio.

You can test the flows you created within UltraStudio. Create a new “UltraESB-X Server” run configuration and start the server. Run the website in a simple HTTP server — Python provides a built-in web server; I use serve — and visit http://localhost:5000 in your favourite browser and see if it’s working.

You can also try out this exercise on UltraStudio as a preconfigured sample project.

It all started with the big bang 10 years 13.7 billion years ago. Since then, the earth has gone through many transformations as we witnessed the information age where the digital revolution is reshaping around us. What do you think this relentless quest has brought upon us? Your guess is as good as mine, The Imagination Age. Virtual Reality is just the beginning. But still, the knowledge workers dominate the economy; dissemination of information is of utmost importance. The irony is that I was too naive to contribute. Anyhoo, one of my mentors, Asankha Perera, CEO at Adroitlogic emphasized the power of sharing information at every turn.

Information used to play games with me throughout my academic career. I took the lazy man's road. Google was my friend all along; but, even Google seized feeble at times. My current employment at AdroitLogic put me to the test—I had to do a litmus test—So, I started blogging again; this time about the things that matter, things even my friend, Google couldn’t help me with. Anyways, here are some of the reasons that made me jumping on the bandwagon.

1. An insatiable craving for writing?

Now.. don’t get me wrong. I am not a writer. I’ve never been one because I didn’t want to. Every now and then my actions were slopped onto Google Docs, being saved for a rainy day—as I’m a victim of Transient Global Amnesia—at least I think I am. One thing led to another and now I’m writing.

Not all the writers are quirky recluses; nor gifted with super powers. But, you and I can be writers, if we decide to put an end to watching television since it's poisonous to creativity. You don't want poison anywhere near you, now do you? Writing is a tried-and-true way to enhance learning, organize thinking, improve communication, and develop creativity. Some of the articles I recently read really inspired me to write, thus materialized the long awaited blog. But, will this make me a good writer? The decision is up to you. I think you should write too.

2. Technology transfer

Drastic improvements in almost every sector have pushed idea commercialization to the next levels, mainly due to the technological uprise of enterprise giants. A Business Insider article describes that the food shortages will not be a problem by 2025 with the current rate of development. Technology and its transfer play a huge role in manifesting reality where these were mere dreams. One of the main goals of this blog was to share knowledge and get discussions on technology underway.

3. Noble cause

Have you ever spent days and nights struggling to find solutions for a technical problem, but once you found them you couldn’t understand them? Been there, done that—I’ve had my fair share of searching for the same thing over and over again; reading the same set of articles and answers again and again.

This blog will be about the things I bagged in my ugly endeavours, things I accidentally spilt onto Google Docs. I have had to take a peek and relive in some of those moments every so often. Why conceal the knowledge, when a world could benefit? Thus emerged, Chummy Charms. You will find a string of concernedly compiled articles which will help you to chase away the boogeyman in your nightmares.

In a career of technology, sometimes, following the footsteps is far more valuable than delving into an article with in-depth theories. This blog may include both, you may cherry-pick according to your need.

4. A right-brained guy?

I love to design, to create something that didn't exist. My obsession got worse, as I designed this blog for months including several cycles from the scratch; this piece of art needs some content though. Back in the day, when I was a freelancer, I helped loads of people to blog and live their dreams. Blogging boosts creativity, it definitely did mine. Creativity is proven to be good for health—relieving your stress, enhancing your brain function, nurturing your social life. Who in the right mind doesn’t want that, right? Read, and read more, subscribe to Chummy Charms, gather knowledge, start writing something of your own. Be creative. Be healthy.

5. Technology trends for power readers

Our living standard has been improved due to growing technological applications in day-to-day life. Every industry is boosted by the technology. Owing to this advancement, once mere dreams have become a reality in almost all the fields. Every techno professional must be updated with the technology these days in order to survive.

“Trends come and go, but the style remains forever”.

Learning new things is my passion, thus reading about various technologies and conducting my little experiments. I'm willing to share the little experience I gained over this little period of time making some strong content for the power readers.

6. Adding meaning to life

I’ve been wandering around the blogosphere since 2001 and wanted to be an active part of it for a long time. Then I started but was blogging about various ridiculous things for quite some time. It’s time I took this serious—the power of a blog. Chummy Charms is going to be like the journal of Bilbo Baggins, the diary of Tom Riddle, in which the adventures I encounter during my journey to the centre of the technology world are scratched down. I am open to suggestions, comments that make this more meaningful.

7. Be inspirational

It's no wonder that you and loads of others love to write but don’t have enough drive. I'm not a bookworm, but I read with the intention of being armed with new skills every day, and to admire the attractive work, the masterpieces. Creative blogger Dustin Curtis, the creator of Svbtle used to write striking articles over the years and heavily inspired me. There are life-changing blogs if you read more carefully. I hope I can inspire at least a few of you through Chummy Charms, not today, maybe one day.

8. Sharing

It’s no secret that Facebook, Twitter and other social sites convert others' ideas and thoughts into a constant stream of revenue. Information sharing opens access to a whole new level of individual and community development: promotes trust, enables better decisions, enhances analytical adroitness. If you use more carefully, it builds value. Honestly, I’m not the best at communication. According to my result of the Myers-Briggs test, I am an introvert. I know there are many like me out there. Yes.. I’m talking to you. Being engaged with a blog is a great medium to communicate your ideas. Start sharing your knowledge, add comments, show your love. There is internet full of people waiting to read your views.

9. For a better purpose

Blogging helps web scraping for modern information. Rather than wasting time on movies & games, this drives one to learn new things and sharpen the observational prowess. Crafting an article is not that easy, if it was, everyone would be doing that. It needs a lot of information in your pockets, learning about a wide range of areas, conducting tests on your own. This leads to proactive learning, where one can learn in a more constructive way.

10. Cultivating social life

Blogging gifts you freedom. It extricates the philosopher in you and helps to ease the pain of anyone who’s down in the dumps. More importantly, blogging benefits your social life bulging your network opening up to new and amazing opportunities.

As Winnie once said,

A good blog post should be like a woman’s skirt; long enough to cover the subject and short enough to keep it interesting!