2024
Massive Data Scraping from Tiktok and Instagram
This project was made to parse the data from TikTok and Instagram platforms precisely. Dataforest’s script parses users' data (account name, main statistics like number of comments, shares, etc.) and collects it into one web solution.
Challenge
* The client provided the list of required data to parse and pointed out that the precise parsed data is their most wanted feature. * For these platforms, data scraping isn’t straightforward due to frequent changes, while the API solutions remain inaccessible due to security measures. * Platforms employ anti-scraping measures such as IP blocking and restrict access to certain content based on geographical locations.
Solution
* To successfully navigate through numerous profiles and data to parse, Dataforest developed a robust scraping infrastructure that can efficiently manage large data loads. Scalable storage solutions and cloud platforms were chosen to organize and store scraped data. * Implemented TikTok’s API authentication process, which helps ensure that scraped data will be accurate, fast, and complete. Dataforest has developed the script and actively monitors all the changes in TikTok’s algorithms and data structures, swiftly adapting the scraping techniques. *Dataforest utilized techniques such as rotating user agents, and using proxy servers to mimic human-like behavior and evade detection, as well as to simulate requests from different geographical locations.
Results
Dataforest has developed 2 separate scrappers, that separately run Instagram and TikTok. These scrapers operate seamlessly and consistently within all the established KPIs set by the client - the script effectively extracts the intended information without significant errors or missing data, leading to 100% data accuracy, while the scraping speed is just up to 5 seconds per 1 id.
Technologies & Tools
PythonPostgreSQLAmazon S3
Project Details
Year
2024
Duration
16
Budget
$100K – $250K
Team Size
8 people
