Chummy Charms

Political pioneers all over the world keep on staying dominant and prevailing powers in worldwide matters. It's no surprise that US President Donald Trump is one of the most influential people who remains popular across the news due to his radical decisions. Even the globe's most formidable can't just ignore the fact that analytics play a principal role in everything. However, we ain't here about Trump or his propaganda—we are here about the future of decision making for all it matters.

About a year ago, we started being a part of the digital transformation with the first ever cloud-based IDE for serverless development. It was no cakewalk—we’ve been burning the candle at both ends trying to cover a majority from the AWS’s serverless stack. Working with AWS Kinesis made me realise the beauty of serverless—of course, the exposure to streaming data with Kafka spared me some time going through the rudiments.

Long story short

Did you ever wonder...

• How "Google Search" suggests you when you’re half-typing your query?
• How "Cheapest Airlines" starting to appear everywhere after you searched for a country?
• How online role-playing games adjust according to your decisions?
• How do gambling sites predict the odds of a live game?
• Why were Curry and Thompson benched while Portland was handing the Warriors their worst loss in 73-win NBA season?

The power of real-time streaming data analytics is astonishing indeed. Now, since serverless technology is gaining some momentum, maybe you won’t have to worry about taking risky decisions on your own at all. This post covers the basics of "Serverless Streaming Data Processing" and how it will be an influential component of our decision making in the future.

A Series of Streaming Events

Life is an endless series of events. The technology around us has made it a stream of digital actions emitting streams of data. If you turn back and investigate your life very carefully, you'll see the never-ending string of data you have generated with your every digital action. It could be a lot to digest at first, but let’s explore some scenarios and try to find what applies to you and me.

• Online banking and convenient e-commerce purchasing capabilities
• Ride-sharing, modern-day travelling and transportation
• Industrial equipment and agricultural use cases like monitored machinery, autonomous tractors and precision farming
• Automated power generation and smart grids, Zero-net Buildings, Smart metering
• Real-estate property recommendations based on geo-location, predictive maintenance
• Online dating and matchmaking relying on complex personality patterns and attribute distribution
• Financial trading according to the real-time changes in the stock market, analytical risk management
• Movies, songs and other digital media with a better experience depending on the demographics, preference, and emotions
• Improved web and mobile application experience based on usage
• Dynamic and personalised experiences in online gaming
• Enhanced social media experiences with hyper-personalisation and predictive analytics
• Telemetry from connected devices, or remote data centres from geospatial or spatial services like weather, resource assessment
• Sports analytics to enhance the players’ performance reducing health risks

All these events produce data—lots of it. Due to the frequency of this data emission, it has become an increasing burden to the digital space.

Streaming Data

In a survey conducted last year about data, it’s estimated that with the current pace of data generation,

1.7 MB of data will be created every second for every person on earth by 2020

Data that is poured continuously by a gazillion sources every second has become a fact we can’t just ignore. Big Data disciplinary was an eye-opener for the tech world to apply this once irritating data to do something useful. This same irksome data is collected and analysed by a new species, namely data scientists 😛. Due to the nature of continuity and often being in small sizes (order of Kilobytes) these data flows—usually referred by the moniker streaming data—are collected simultaneously as records and sent in for further processing.

Stream Processing and decision making

A streaming data processing structure usually comprises of two layers—a storage layer and a processing layer. The former is responsible for ordering large streams of records and facilitating persistence and accessibility at high speeds. The processing layer takes care of data consumption, executing computations, and notifying the storage layer to get rid of already processed records. Data processing is done for each record incrementally or by matching over sliding time windows. Processed data is then, subjected to streaming analytics operations and the derived information is used to make context-based decisions. For instance, companies can track public sentiment changes on the products by analysing social media streams continuously—world's most influential nations can intervene in decisive events like presidential elections in other powerful countries—mobile apps can offer personalised recommendations for products based on geo-location of devices, user emotions.

Most applications collect a portion of their data at the outset to produce simple summary reports and take simple decisions such as triggering alarms or calculating a moving average value. When the time flies by, these become more and more sophisticated, and companies might want to access profound insights to perform intricate activities in turn with the aid of Machine Learning algorithms and data analysis techniques. The continual growth of data has made data scientists work around the clock to come up with trailblazing solutions to utilise as much data as possible to fabricate alternate futures with better decisions.

How the world embraces it

Many companies use insights from stream analytics to enhance the visibility of their businesses which allows them to deliver customers a personalised experience. Additionally, near real-time transparency gives these firms the flexibility to promptly address emergencies. The emerging serverless architecture has driven all the leading cloud service platforms to present complementary solutions. Stream processing was made available for serverless application development with fully-managed, cloud-based services for real-time data processing over large Distributed Data Streams.

1. Entertainment got upgraded!

Netflix, the leading online television network in the world, developed a solution which centralises their flow logs using Amazon Kinesis Streams. As a system processing billions of traffic flows every day, this eliminates plenty of complexity for them because of the absence of a database in the architecture. Due to the high scalability and lightning speed, they can discover and address issues as they arise, monitor the application on a massive scale. With the upgraded recommendation algorithm, video transcoding, and licensing popular media, this subsequently grants a seamless experience to the subscribers. With the exponential growth of the subscribers, the company’s responsibilities increase by the day. However, nothing seems to be a problem for Netflix for many years to come since they are considered to have a sound decision-making model.

2. Improving the decisions of the decision makers

As a leading source of integrated and intelligent information for businesses and professionals, Thomson Reuters provide their services to decision makers in a wide range of domains like financing and risk, science, legal, technology. This company built an in-house analytics engine to take full control of data and moved to AWS because they were familiar with its capabilities and scale. The new real-time pipeline attached to Amazon Kinesis stream produces better results in perceptive customer experience with accurate economic forecasts, financial trends for beneficiaries including a range of government activities.

3. Unicorn: a solution to traffic congestion

Jakarta has become a heavily congested city where the motorcycle deemed the most efficient mode of transport. To exploit this business opportunity, GO-JEK—one of the few unicorn businesses in Southeast Asia—started as a call centre for motorcycle taxi bookings. However, to meet the demand in exceeding expectations, the company had to consider expansion. Now with the support of Google Cloud Professional Services, the business architecture built on Cloud Dataflow for stream inference enables them to predict changes in demand effectively.

There are more stories about how cloud platforms like AWS, Google, Microsoft Azure, and IBM Cloud are exploited by companies to make their clients’ lives better and secure.

Shortcomings of Serverless Stream Processing

Serverless stream processing is increasingly becoming a vital part of decision-making engines. However, with the current set of features, it’s not the ideal solution for some scenarios. Implementing real-time analytics for sliding windows and temporal event patterns is not a course for the faint-hearted.

The best way to assimilate never-ending data of this magnitude is through real-time dashboards which requires additional data organisation and persisting. These manoeuvres introduce undesirable latency and data management issues into the context. However, technology is evolving and trying to catch up to the speeds with integration using advanced cloud data management techniques to produce materialised views.

Stream Processing often uses a time-based or record-based window to be processed in contrast to the batch-based processing, which can lead to challenges in use cases that require query re-execution.

Nowadays, application requirements grow beyond aggregated analytics. Increasing the window size seems to be an appropriate temporary solution but, it develops another intractable problem—Memory Management. Modern-day solutions usually provide advanced memory management and scheduling techniques to overcome this, but the world will see further improvements.

Conclusion

All in all, it’s apparent that serverless stream processing has been playing a prominent role around us without us even knowing. With the power of serverless data stream processing, applications can evolve from traditional batch processing to real-time analytics. The revelation of profound insights will result in effective decision making without having to manage infrastructure. Even today, many organizations practise orthodox decision-making strategies based on the analytics derived using the big data clusters that belonged to THE PAST. New horizons of serverless and real-time data processing are now equipped with the power to make effective decisions and create a—more productive, relevant and most importantly secure—world around you.

So do you still think Trump is more powerful?

What do you think?

Share your thoughts. Kudos this.

Originally published at SLAppForge Blog.

Serverless is a misnomer. A perception. It was not materialized until it became a household name, a few years ago. If you’d ask whether there are servers in serverless, the answer—unsurprisingly—is a BIG yes. Now, the million-dollar question—do you have to worry about them?

No.

Back in the day, it was all just bare-metal servers, which were kept on-premises and managed by the organizations themselves. Then came the virtual machine, a piece of software that mimics the hardware-based servers. Cloud was the next big thing inspired by the distributed computing platforms. Containerization, a.k.a. OS-level virtualization came along and gained popularity afterwards. However, later, IaaS (Infrastructure as a Service) had covered some ground due to the emerging popularity of the cloud by then. Provisioning servers became automated with PaaS as the next step of IaaS.

Technology continually evolves, like it or not, so does everything around us. In the last decade, we saw drastic improvements in connectivity, compute power, storage, etc. Serverless started appearing more and more (even got us Baader-Meinhoffed) after Business Development VPs started making impact stories and this continued the trend of PaaS (Platform as a Service) with some significant paradigm shifts. With Amazon Web Services launching Lambda at 2014 ReInvent conference, serverless became inevitable. By today, it is one of the trending topics in the software architecture world.

Diving into Serverless

Serverless computing is a cloud computing model of which applications and services can be built and run without managing any servers. As such, the name has been derived corroborating the fact that provisioning, scaling, and managing of servers are entirely hidden from the operator. Serverless incorporates two different but overlapping realms—BaaS, shorthand for Backend as a Service, and FaaS, shorthand for Function as a Service.

Serverless is not a new kid on the block, even though some misapprehends it to be a relatively new concept. Zimki started offering the first “pay per use” code execution platform in 2006—before they withdrew the service at the end of 2007. Heroku by Salesforce has been silently operating their web application deployment model since 2007. Google App Engine was released in April 2008 with limited features by Google. It started out as a custom Python execution framework and now turned out to be one of the favourite cloud platforms for web application development and hosting.

Brave New Waves

With the event-driven wave hitting the coast, ephemeral, stateless compute containers came into play. Unlike traditional architectures, these compute engines were invoked only once by a 3rd-party—and killed. People started calling monikers—Functions as a Service (FaaS) emerged. This was briskly leveraged by the developers for focusing and implementing own business logic triggered by specific events.

FaaS completely abstracts the infrastructure layer from the developer charging only for the time of use. Quoting AWS Lambda,

Run code without thinking about servers. Pay only for the compute time you consume.

FaaS market is driven mainly by the shift from DevOps to serverless computing lately. Global forecasts predict the serverless market growing at a rate of 32.7%. Industry experts have some enticing views about the progression of serverless technology too.

Serverless, the hero

Introduction of serverless eliminates the woe to manage clusters of servers and the traditional “always on” server requirement. The underlying fleet of servers scales gracefully on its own, provisioning appropriate infrastructure, even to meet massive surges in traffic. They are regularly monitored and managed by the providers—also security patches and other immunization shots are taken care of. This cuts a considerable portion of the budget as it results in notable reductions in operational cost, not to mention the hassle.

In serverless, you only pay as you use; the invocations are billed by fractions of seconds. Implementing a production-ready output from a mere idea is super-fast. Developer productivity for a first-time experience is a significant gain compared to struggling over different software stacks set up internally. AWS and pretty much all the other serverless solution providers give free tier options and come as language agnostic endpoints (except for Google Cloud, others provide multiple runtime/development environments) which makes the first-time hands-on experiences much more relieved.

Serverless, the villain

This does not mean that serverless is all god-like; it might include some hidden costs incurred. Usually, these costs don’t encompass only CPU and RAM usage, but the number of API requests, storage, and network might get their fair share. Sometimes, according to your specific business requirement, an implementation with multiple functions may be more difficult than having one with a single container. Code maintenance may incapacitate you, due to the distributed nature of your overall logic--or for more complex use cases, going with Kubernetes might be advantageous since it’s more mature.

Lack of integration testing facilities on localhost is a huge downside of the existing providers. There is no easy way to emulate but can be overcome to a certain extent by using Serverless Framework or aws-sam-local. Manual machine tuning befitting for your application is impossible because the server layer is intangible.

Provider lock-in can be challenging when moving forward since each serverless platform is a far cry from each other based on their uniqueness and limitations. Because of the cold starts, applications may experience high latency at the beginning.

Serverless Competition

AWS is not the only FaaS provider in the market. There is a handful of other enterprise giants competing for the fattening market. Microsoft Azure enhances event-driven serverless compute experience with Azure Functions. IBM’s serverless domain provides IBM Cloud Functions which is based on Apache OpenWhisk, an open source project. Google Cloud Functions is the Google Cloud Platform’s version of FaaS, which is still in beta.

Different providers provide a stack of fully managed products in their serverless platforms. These backend services require no provisioning, maintaining, or administration. Here’s a depiction of the various components in each serverless platform.

Apart from these, Firebase is the Google Cloud Platform backed (acquired, as usual, in 2014) mobile and web application development platform which consists of multiple integrated services. Most of the serverless providers showcase IoT framework support, mobile backend support and cognitive application development support as well.

Most Amazon web services have availability zone specific endpoints to compensate the latency in the applications around the globe. Developers have the freedom to use these regional endpoints or the default North Virginia endpoint. You can read more about AWS services that support particular regions in here. Just like that, other serverless providers support different availability zones as well.

AWS Lambda and other services have SDKs that support serverless development in multiple programming languages. Amazon recently (2018 January) announced Go support. Read this article to know a great deal about a detailed comparison of different serverless providers.

Many faces of Serverless

Serverless applications let you run code without provisioning or managing servers. Serverless domain empowers a myriad of application opportunities. Some of them are divulged here as follows.

1—Real-time data processing
Real-time processing is found to be coined widely with serverless. Image resizing for viewing on different devices such as desktop computers, smartphones, tablets are done by The Seattle Times using AWS Lambda. SiteSpirit uses IBM cloud data services to build their cloud-based media library.

2—Real-time stream processing
Localytics uses Lambda to process billions of historical and live social media trend data for business users to query. IBM Cloud Functions can be integrated with Apache Kafka supported IBM Message Hub for processing stream data. Plexure, a company engaged in innovative retail businesses based in New Zealand, uses Azure serverless technologies to bring customer experience to the next level—with the help of Azure Functions and Azure Logic Apps, now customers get alerts about the in-store digital display.

3—Extract, Transform, Load (ETL)
Zillow, one of the largest real-estate brands utilizes AWS serverless stack to provide customers with a smooth experience in near-real-time home valuation.

4—IoT Backends
In the IoT domain, there is an increasing growth of applications using serverless platforms. This exciting story explains how GreenQ, a smart residential waste collection company, exploits IBM Bluemix technologies to engage in a fast and more efficient service.

5—Mobile Backends
Bustle.com is a website catering to women which provides news, fashion and many more. They use Lambda and API Gateway for its iOS and website backend. With the move to new serverless architecture, developers no longer worry about managing and provisioning infrastructure, all they focus on is innovating. The mobile app WeatherGods uses IBM Cloud Functions (former IBM OpenWhisk) to notify users about specific weather events (hurricane in FloridaColombo, for example) aligning god avatars according to their preferences.

6—Serverless web applications
TimerCheck.io is a “deceptively simple web service with superpowers” in Eric Hammond’s words. The service is entirely built on Amazon API Gateway and AWS Lambda, and it provides an unlimited number of timers that can run infinitely.

7—SaaS event processing
San Joaquin Valley College (SJVC) is a private junior college in California. To simplify the IT management and provide the students with a better experience, they moved their learning management system to Microsoft Azure and Office 365. With the help of Microsoft FastTrack, SJVC embraced the serverless technology and continued their journey of innovation.

Apart from these most popular ones, you can find more use cases of AWS, Azure, and IBM.

Empowering serverless development

Software architecture world is steadily being absorbed by serverless. It’s not going to be “Serverless Everything” anytime soon as there are still many blanks to be filled. Tooling is improving, but again, there are still gaps to be bridged. The earlier mentioned Serverless framework provides a CLI tool that lets users to be integrated with multiple cloud providers.

Cloud9, once an independent cloud-based IDE was acquired by Amazon to plug development aspect into its serverless stack. However, setting up a Cloud9 development environment requires a new Amazon EC2 instance or your own Linux server. The getting started process is reported to be more complicated than the original Cloud9 (c9.io) by many users. Check this neat trick attempting to come up with a self-sufficient AWS lambda function, which quashes the need of uploading a bundle with third-party dependencies — built locally. Microsoft Azure is powered by Visual Studio, another versatile but heavy cloud IDE with team services.

Think Serverless

SLAppForge, a totally new player, recently announced its beta release of Sigma, a customized editor for the development and deployment of serverless applications. It’s a hybrid model exploiting the beauty of drag-and-drop style and the full power of raw code. Sigma hides the underlying complexities of the platform from the developers using its own, “very” serverless architecture—hich means, now you don’t have to dig into piles and piles of documentation and tutorials.

This powerful IDE is equipped with intuitive code suggestions, context-aware code completion, auto-generated code snippets—still supports only Node.js though. It also takes care of the Continuous Integration, and Continuous Delivery flows of your serverless applications. If you have a web browser (I bet you already do!), a GitHub account and an AWS account, you’re all set—because there’s nothing to install. Deploying your serverless application is minutes away now, not hours, not days.

Conclusion

The paradigm shift to serverless is becoming inescapable by the day, just like was the change from on-premise to cloud. The Big Day that all applications go serverless is not far away. It’s high time to jump on the bandwagon and be a serverless developer since everything is propelling you to be more productive and efficient. Keep aside the worries about infrastructure; focus fully on the business logic. The ball is in your court; jumpstart your serverless development now!