Getting to Real-Time Big Data Faster with Talend and MapR

Post by Talend

Talend and MapR share a common vision around enabling businesses to utilize data as a strategic asset in their armory, so they can be more agile and remain competitive in the Big Data era. A recent survey of CIOs found that up to 80 percent are planning to deploy at least one (if not more) big data projects in 2017. It’s clear that big data has moved past what industry analyst firm Gartner would term ‘the hype cycle’ and into the mainstream. That said there are a few things still holding enterprises back from realizing the full business benefits big data technologies such as Spark and Hadoop can deliver.

Let’s ‘double click’ into some issues facing IT organizations that can hinder them from making progress towards experiencing real-time value from strategic investments in Hadoop and Spark.

Hand Coding

Most IT teams are constantly being asked to do more with less, so it’s tempting to look for quick and easy fixes. Yet, ‘quick and easy’ is not always the best solution for the long term. One example of where most enterprises still tend to think ‘short term’ instead of ‘long term’ is when it comes to hand coding. Oftentimes, hand coding can offer simple fix to an integration challenge. It may even save 20% of your deployment costs in the beginning.  However, the maintenance costs will increase by 200% down the road, according to research by Gartner.1 As big data projects scale, more systems and more coding are involved and maintaining the complex manual coding would be even more challenging with different people supporting the projects. 

Changing Technology

In today’s quickly evolving and competitive marketplace, big data and cloud technologies are moving at a breakneck pace. While you may be leveraging the right tools today, those may very likely be replaced/outdated within the next 18 months. Thus, IT decision makers need to invest wisely in solutions that are open, adaptable and easy to deploy/update within their existing infrastructures as business and market demands change.

Complexity of Data Movement

According to a survey by IDC, 82% of organizations are in some phase of adopting real-time analytics.2 With more big data use cases requiring real-time and streaming capabilities, Hadoop is no longer the “panacea” to everything. Oftentimes enterprises need to set up different clusters for streaming data such as IoT device data, clickstream data, network data, in addition to what they have as Hadoop clusters, enterprise storage and operational databases. The movement of data between these systems becomes a new challenge—creating a siloed architecture (as depicted in the figure below) that not only adds to overhead but also complicates the management of your technology stack.