This article explains the new challenges in enterprise integration caused by the advent of big data, and presents some approaches to overcome them.

Big data is the latest buzz-word in the industry. At the basic level, big data is just that: large amounts of data that cannot be handled by conventional systems. With increases in hardware capacity, our definition of what constitutes a conventional system changes, and so does the threshold of big data, which is not something new but has always been around. It's just that the threshold of what constitutes big data has changed. Today, the threshold for big data may be terabytes (1012 bytes). Soon, it will be petabytes (1015 bytes). Twenty years ago, there were very few systems that could process gigabytes (109 bytes) of data in an acceptable time-frame. So gigabytes would have been the lower threshold of big data at that time. different results. This is where it gets interesting. Let us suppose that an enterprise implements a big data solution. The big data processing (refer to the big data processor in the figure) solution could be a shared service across the enterprise. Due to the technical complexity and cost involved in building a big data processor, it is not possible for each division in the organisation to have its own. But due to the business value accruing from big data analysis, sooner or later, different divisions in the enterprise will want their own processing on the big data set. This can be done by moving a subset of the big data, relevant to that division, over to its systems. This is the point at which the integration systems in the enterprise will feel the impact. To enable different divisions in the organisation to ‘do their own stuff’ with data, a subset of the big data will start moving across the integration middleware.