Four in Ten Retail and E-commerce Organisations Struggle to Find the Most Efficient Tools for Data Acquisition

●      44% of retail and e-commerce organisations struggle to find the most efficient tools for data acquisition

●      39% struggle to find reliable partners to outsource their web scraping activities to

●      40% find it challenging to ensure the quality of the data they are collecting

●      36% found managing and processing large datasets to be challenging

●      52% said they want to collect and utilise more external data, such as web scraped data from competitors’ websites, marketplaces, etc.

●      Half of respondents (50%) stated they wanted to collect more real-time data in order to make faster business decisions

●      The UK’s retail and e-commerce sector has experienced an unprecedented rate of growth throughout the COVID-19 pandemic from 2020 to early 2021

●      Online retail sales hit a 13-year high in 2020, with an increase of 36% year-on-year – the highest rate seen since 2007

More than 40% of retail and e-commerce organisations have struggled to find the most efficient tools or tech stack for data acquisition over the past 12 months to help them rapidly analyse data in real-time resulting in actionable business insights; with 39% of respondents struggling to find reliable partners to outsource their web scraping activities to and 40% finding it challenging to ensure the quality of the data they were collecting.

Over a third (36%) of respondents found managing and processing large datasets to be a challenge. Yet, for retail and e-commerce businesses it is increasingly critical that they have access to data in real-time as pricing intelligence strategies are completely reliant on minimal delays in data delivery and integration.

The findings showcase the challenges of web scraping as data needs to be collected from numerous sources, and in real-time. There are plenty of opportunities for something to break down along this process.

With the boom in online retail and e-commerce set to continue as we progress into 2022 UK retailers and e-commerce businesses are now placing more value than ever before on new, external sources of market data and dynamic pricing information.

Today’s online retail businesses are rife with public web data scraping opportunities, with businesses and consumers alike leaving an increasing amount of highly valuable data available for analysis. Such external data can be scraped from almost any marketplace, as long as it’s public data, to great effect. With it predictions on real-time and future market trends, customer sentiment, dynamic pricing information, etc. can be accurately made.

Aleksandras Šulženko Product Owner at Oxylabs.io comments: “Partly, the drive to develop the E-commerce Scraper API, SEPR Scraper API and Web Scraper API, our dedicated scraping solutions was based on the numerous issues our clients were facing when they used their own data acquisition tools. So, a large proportion of businesses running into scraping issues isn’t all that surprising. Some of these problems, though, can be resolved by outsourcing data acquisition, either to solution providers or Data-as-a-service companies. One thing to note is that most providers are willing to adapt to customer requests. If a solution doesn’t have a feature you need, often the provider will be willing to implement one.”

We have previously established the importance of real-time data in the retail and e-commerce sectors. To add to the rising importance, in terms of having a data strategy for the forthcoming 12 months, over half of all organisations (52%) said they want to collect and utilise more external data, such as web scraped data from competitors’ websites, marketplaces, etc., whilst 50% expressed the wish to collect more real-time data in order to make faster business decisions.

Overall, the picture is clear. Data is the driving force behind the most advanced e-commerce and retail strategies. Companies in this sector are intending to greatly increase their dedication to data, especially to real-time acquisition processes.

Tomas Montvilas Chief Commercial Officer at Oxylabs.io comments: “Real-time data is a comparatively new asset. While some businesses are still adopting it, others are already utilising real-time data to power their business and pricing strategies. In these extremely competitive and dynamic sectors, the importance of real-time data will only increase over time and interruptions will become more costly.”