Automating Pattern Discovery with Open Source Software

To keep the competitive edge, enterprises need to discover as much as possible about their clients (the likes, dislikes, favourites) and marketing trends. There is a vast amount of data generated by enterprises (inventory, customer purchases, customer accounts, preferences, etc), which can help them do so. This huge store of data can be analysed with the help of open source pattern discovery algorithms to benefit enterprises.

To leverage the huge amount of data being generated, enterprises today want to look beyond conventional business intelligence, in order to discover the trends and patterns hidden in the data. According to analysts, smart pattern discovery will be one of the most in-demand technologies in the near future. Frequent pattern mining (FPM) and the associated algorithms play an important role in many business scenarios that depend on recognising relevant patterns from transaction databases. Formulating these algorithms is currently getting a good amount of traction in the retail domain. However, they can be used effectively in many other use cases across various domains. In this article, I will share some of my experiences in implementing these algorithms to solve business problems across various domains using a set of open source software.

To provide actionable insights from the huge amount of data available with the modern enterprise, machine learning and related technology is being adopted rapidly across various business domains. These techniques enable the learning of trends or patterns from historical data.

Every enterprise is interested in identifying patterns or trends from the huge gamut of data generated by its applications or coming from the external world. Traditionally, subject matter experts (SMEs) have been tasked with identifying trends and patterns in data, based on their experience. With data volumes surpassing levels that are humanly manageable, enterprises are looking towards tools and technologies that can help identify patterns and that, too, with the least effort.

Though machine learning algorithms can identify patterns, they still need to be validated by SMEs. This is because the quality of patterns identified is largely determined not only by