Geosciences: Solving data-processing complexity
When platform complexity impacts growth and innovation, it’s time for a new approach.
CCQ has several clients working in geoscience and engineering. Data processing has become a mix of desktop applications, legacy scripts and manual processes that many thought was too complex to resolve.
With deep science and mathematical expertise, plus 20-years of accumulated knowledge with these clients, our team had the insight and authority to propose a different approach.
Where many saw unavoidable complexity, CCQ reimagined the problem from the ground up.
Challenge
Clients use specialist sensor technologies to measure and monitor the condition of large-scale infrastructure assets.
CCQ supported their technical platforms as their businesses expand internationally. But as customer project volumes grew, we saw the underpinning data architecture causing difficulties.
Data complexity: Every survey generated multiple streams of specialist data, including multiple specialist sensor streams, imagery and geospatial datasets.
Data volume: Customer projects would generate hundreds of gigabytes daily. Some files are over 100GB each. These have to be downloaded, divided and processed separately, creating terabytes of duplicate data.
Data access: The crucial analysis depended on downloading entire files (hundreds of gigabytes) onto specially configured workstations running desktop software, making access slow, inflexible and difficult to scale.
Manual workflows: Matching the analysis to geographical position demanded significant manual processing and checking, and one mistake in location or timing would force teams into significant rework.
Our clients faced a serious operational bottleneck and growing competitive risk. Their processing output is limited by manual processes, engineers struggling with unwieldy file sizes, and their ability to iterate curtailed by complexity.
Solution
Clients believed their constraints made unifying platforms impossible, but we saw enough potential to investigate further.
Our team are specialist problem solvers with advanced academic qualifications in maths, physics, computer science and engineering.
Successful 10-year+ partnerships gave us the geoscience insight and authority to propose a bold new architectural approach.
A CCQ Tech perspective
We guided them back to first principles, defined the core requirement, ‘faster and easier practically useful infrastructure analysis’. Then we designed a single shared architecture to bring separate workflows together around that purpose.
Before writing any production code, we proved the feasibility of our two hardest assumptions; that location matching could be largely automated, and engineers could inspect enormous files section by section through a browser.
Our solution would ingest and process multiple forms of specialist sensor data, then use time and location to match readings to the correct physical infrastructure network location, aligning data across different customer asset models used by different operators.
Crucially, the new architecture would make it possible to add new sensor types and processing capabilities as platform extensions, rather than creating a new standalone system each time.
Technologies
Microsoft .NET 7 and .NET Core
ASP.Net WebApi
ASP.NET
C# & C# Source Generators
C++
Entity Framework Core
Typescript
Microsoft SQL
Azure
Javascript
Python
Technical disciplines applied
Geospatial and infrastructure network modelling
Graph algorithms and network registration
3D spatial data processing
Sub surface sensor data processing
Proprietary binary formats and entropy compression
High-volume streaming data architecture
Multimodal sensor data synchronisation
Browser-based 3D visualisation
Results
CCQ’s solution removed major constraints on growth for our clients.
By unifying multiple sensor streams, the new platform gives engineers a more connected view of the infrastructure and creates a faster, more scalable way to process increasing volumes of survey data.
Less data duplication: Where teams previously had to split, copy and reprocess enormous source files, the new platform keeps the original data intact and generates the relevant processed view when required, reducing project data storage requirement by up to 70%.
Less rework: Previous corrections would force teams to repeat up to eight processing steps, while the new system reduces that to just two, reducing a week of rework to just a couple of hours.
More specialist capacity: Specialists spend at least 50% less time processing and reprocessing data, giving them more time to interpret findings, produce customer insight and handle greater survey volumes.
Faster innovation: The new system is designed to give clients a repeatable way to accommodate evolving sensor technology and data formats. We proved this by adding a new proprietary data type, creating a functioning first iteration in one month and MVP production readiness within three.
What’s more, the unified data architecture creates a future-ready foundation for AI innovation, for instance by using multimodal AI models to predict asset deterioration or potential failure.
