Bitcoin History Data Processing

The project was built on a third-party provider system, which led to different problems with data processing speed and low data precision. Although the calculation of that amount of data costs a lot if using BigQuery The metrics in this market are usually calculated only per 1 day, which complicates the calculation of historical data.

The data was transferred to Databricks. Calculating metrics using BigData methods reduced the cost and duration of calculations, and the precision was at its highest point. Dataforest developed a script that can calculate the size of each wallet and metrics in it for any period, which made it possible to use historical data and widen the charts and reports presision.

All the client's requests were fulfilled - the migration for their own infrastructure reduces the costs and provides the most precise data consolidation. New charts and reports were constructed in a modern style and were connected to the database alongside the daily metric calculations.