Is Your Data Integration Platform Container Ready?

Post by Edward Ost, Director, Global Technology Alliances at Talend

Docker Containers are widely used and becoming even more prevalent as companies seek to streamline their operations.  Containers help decouple compute resources from applications, increasing the elasticity and hence the efficiency of IT operations.  DataDog reports that Docker usage has grown 40% over the past year, 18.8% of the DataDog sample customers use Docker.  This represents 15% of the servers monitored by DataDog.  Docker adoption within an organization also grows quickly, typically growing to 5x within 9 months.

Clearly, Docker is here to stay and is quickly becoming a pillar of agile DevOps IT operations.  But what of its impact on the applications running on that infrastructure?  Using Docker is one thing, but following true container architecture principles is another.  Many enterprise applications do not provide a simple path to leverage containers.

Historically the application server was king.  You purchased an enterprise class product, and the expectation was the product was secure, managed, scalable, and highly available.  In fact, each such product typically provided its own approach to these concerns, requiring further integration for each aspect.  Even with the integration of security and monitoring, the application servers themselves remained monolithic applications requiring dedicated resources regardless of the actual usage.  This prevents users from realizing the full benefits of containerization.

Modern, lightweight application designs like Talend offer a simple, container ready path.  Talend data integration Jobs are simple Java applications.  They are autonomous, self-contained applications so they can be easily specified in a DockerFile with full confidence that all dependencies have been addressed.

When the Docker image is launched, it can be parameterized so that inputs, configuration, working and staging areas and output folders are all driven with Docker Volumes at execution time.  Such immutable containers also enable greater security.  Entire Docker images can be marked as read-only.  And Docker registries ensure version controlled repeatability and stability for developers and admins alike.  The image can be disposed of immediately upon completion.

The savings in IT operations are potentially significant.  This is reflected in the Data Dog sample.  The median number of containers per physical server increased from 5 to 9 containers.  This shows the value of Docker in providing more granular, targeted allocation of IT resources.  In addition, the median lifetime of Docker containers is also decreasing.  Users of container management tools like Kubernetes have median container lifetimes of less than one day.  So not only does Docker provide more granular use of IT resources, but more dynamic usage as well.

However, realizing these benefits depends on applications being container ready.  In my next article, I’ll walk you through a simple, repeatable way to deploy your Talend Jobs directly to Docker.