When platform complexity impacts growth and innovation, it’s time for a new approach.
CCQ has several clients working in geoscience and engineering. For many of these clients, data processing has become a mix of desktop applications, legacy scripts and manual processes that many thought were too complex to resolve.
With deep science and mathematical expertise, plus 20 years of accumulated knowledge with these clients, our team had the insight and authority to propose a different approach.
Where many saw unavoidable complexity, CCQ reimagined the problem from the ground up.
Challenge
Clients use specialist sensor technologies to measure and monitor the condition of large-scale infrastructure assets.
CCQ supported their technical platforms as their businesses expand internationally. But as customer project volumes grew, we saw the underpinning data architecture causing difficulties.
Data complexity: Every survey generated multiple streams of specialist data, including multiple specialist sensor streams, imagery and geospatial datasets.
Data volume: Customer projects would generate hundreds of gigabytes daily. Some files are over 100GB each. These have to be downloaded, divided and processed separately, creating terabytes of duplicate data.
Data access: The crucial analysis depended on downloading entire files (hundreds of gigabytes) onto specially configured workstations running desktop software, making access slow, inflexible and difficult to scale.
Manual workflows: Matching the analysis to geographical position demanded significant manual processing and checking, and one mistake in location or timing would force teams into significant rework.
Our clients faced a serious operational bottleneck and growing competitive risk. Their processing output is limited by manual processes, engineers struggling with unwieldy file sizes, and their ability to iterate curtailed by complexity.
Solution
Clients believed their constraints made unifying platforms impossible, but we saw enough potential to investigate further.
Our team are specialist problem solvers with advanced academic qualifications in maths, physics, computer science and engineering. Successful 10-year+ partnerships gave us the geoscience insight and authority to propose a bold new architectural approach.
A CCQ Tech perspective
We guided them back to first principles, defined the core requirement, ‘faster, easier, practically useful infrastructure analysis’. Then we designed a single shared architecture to bring separate workflows together around that purpose.
Before writing any production code, we proved the feasibility of our two hardest assumptions: that location matching could be largely automated, and that engineers could inspect enormous files section by section through a browser.
Our solution would ingest and process multiple forms of specialist sensor data, then use time and location to match readings to the correct physical infrastructure network location, aligning data across different customer asset models used by different operators.
Crucially, the new architecture makes it possible to add new sensor types and processing capabilities as platform extensions, rather than creating a new standalone system each time.
Technologies
.NET 10
ASP.Net Core, OData & OpenAPI
React 19 & TypeScript
Entity Framework Core 7 PostgreSQL
C# Source Generators and Automated Client Generation
SignalR Real-Time Communications
Three.js, React Three Fiber 7 Geospatial Mapping
TanStack Query, Router & Form
Hangfire Background Processing
SkiaSharp, ImageSharp & PDF Rendering
Modular Plugin-Based Processing Architecture
Docker, Alpine Linux & Caddy
GitHub Actions CI/CD
Microsoft Identity Platform & OIDC
Vite & Bun
C++ and Python
Technical disciplines applied
Geospatial and infrastructure network modelling
Graph algorithms and network registration
3D spatial data processing
Subsurface sensor data processing
Proprietary binary formats and entropy compression
High-volume streaming data architecture
Multimodal sensor data synchronisation
Browser-based 3D visualisation
Results
CCQ’s solution removed major constraints on growth for our clients.
By unifying multiple sensor streams, the new platform gives engineers a more connected view of the infrastructure and a faster, more scalable way to process increasing volumes of survey data.
Less data duplication: Where teams previously had to split, copy and reprocess enormous source files, the new platform keeps the original data intact and generates the relevant processed view when required, reducing project data storage requirement by up to 70%.
Less rework: Previous corrections would force teams to repeat up to 8 processing steps, while the new system cuts that to just 2, turning a week of rework into just a couple of hours.
More specialist capacity: Specialists spend at least 50% less time processing and reprocessing data, giving them more time to interpret findings, produce customer insight and handle greater survey volumes.
Faster innovation: The new system is designed to give clients a repeatable way to accommodate evolving sensor technology and data formats. We proved this by adding a new proprietary data type, creating a functioning first iteration in one month and MVP production readiness within three.
What’s more, the unified data architecture creates a future-ready foundation for AI innovation, for instance by using multimodal AI models to predict asset deterioration or potential failure.
