Transforming Financial Data Migration with AI-Powered Automation
The client operates in mid-market finance sector enterprises. Working with large quantities of historical financial data is labour-intensive and prone to errors due to duplication and inconsistencies across multiple systems. With hundreds of people-days needed to complete these types of tasks, the client embarked on testing a strong hypothesis that an AI-powered approach could significantly reduce time and improve accuracy.
Over the years, financial data accumulates across multiple internal and external systems, creating problems of integrity, and duplication requiring manual intervention to manage effectively.
The primary challenge was to determine whether Artificial Intelligence and Machine Learning techniques could effectively accelerate and improve the accuracy of identifying similar, or the same financial information across disparate systems, data integrity and mitigate human intervention to free up time for professionals.
We designed a ‘Proof of Concept’ project to test the feasibility of using AI to address industry challenges while laying the groundwork for future iterations and improvements.
1. AI-Driven Matching Model: Our development team created a custom AI clustering model, enhanced with Natural Language Processing (NLP) techniques, to analyse and group similar datasets. We integrated and tailored an existing open-source financial AI model (FinBERT) which provided the ability to generate higher-dimensional data representations, allowing the system to identify semantic similarities between text and data structures.
2. Graph Structure Integration: To increase the accuracy of data matching, the AI model analysed and incorporated the data structure. A graph-based approach allowed the understanding of relationships between data sets and the propagation of critical information including parent, sibling, and child nodes. This structure provided a more contextual understanding of the data, allowing for the improvement of the process over time.
3. Iterative Learning and Data Enhancement: The system worked with a limited amount of labelled data. So the focus was on building a scalable model that could evolve as more data was attained. The system allows for CSV and Excel uploads to add more data. The solution provided multiple potential matches for datasets. As only an early POC product it does need human oversight to confirm or disprove matches to provide feedback enabling the AI model to learn and fine-tune its predictions as more labelled data becomes available.
- Microsoft .NET
- ASP.Net WebApi
- C#
- FinBERT AI Modelling
- Back end software application development
- Algorithm development
- Testing
- Prototyping
- Natural Language Processing (NLP)
- Workflow mapping
- Proof of Concept (POC)
The project produced promising results, demonstrating the potential of AI to transform financial data migration processes.
1. Increased Efficiency: After feedback and refinement the POC product achieves a 77% match accuracy with the initial dataset, demonstrating the AI-powered product would reduce time required by humans to perform matchmaking. The clustering model’s ability to present multiple potential matches allowed users to select the correct accounts quickly, accelerating the overall process.
2. Insights into Model Limitations: The project highlighted the need for more labelled training data to further enhance the model’s performance. While the initial results were positive, it is recognised that the addition of more comprehensive training data would improve both the accuracy and the scalability of the solution.
3. Foundation for Future Development: The model’s current architecture provided a strong foundation for future iterations. With additional training data and fine-tuning of the AI model, the solution has the potential to achieve even higher accuracy and eventually automate more of the process with limited human intervention.
The project goes a long way to proving the concept that a hugely time intensive task can be mitigated using artificial intelligence and machine learning. The client was satisfied with the initial performance and saw the potential for further improvements.
As the model evolves, we expect it to deliver greater efficiency and accuracy, making it a valuable tool for businesses facing the complex challenge of matching financial data.
We want to hear your questions, your ambitions and understand the requirements of technology to support the people within your organisation. Schedule a no commitment short introductory call.