If you have ever been searching for a new apartment or house in an area that you don't know well,
you have probably wondered how its noise level will be.
ING Analytics Frankfurt Hub's NoiseMap application provides an answer to this question.
NoiseMap collects air traffic data from multiple data sources
over time. Based on the historical traffic data, it calculates the unique noise level for each geolocation using a custom scoring function.
The noise levels are visualized on an interactive map.
Most of the used data sources don't provide historical information on the air traffic but only a snapshot from the time that the
data source is being accessed. This makes maintaining a historical changelog of the air traffic inside NoiseMap a necessity.
The team behind NoiseMap, led by Dr. Nawar Halabi, uses DataCater to continuously extract data from multiple air traffic systems and stream them
to a PostgreSQL-based data warehouse. While streaming the data, multiple filters and transformations are applied, such that
PostgreSQL stores the data in a format suitable for the calculation of the noise level.
“As a Data Engineer, I know how difficult it is to deploy a streaming solution. With DataCater it is made a lot simpler and without writing code. I don't have to worry about scaling and I can manage my pipelines with an easy-to-use online interface.”
Dr. Nawar Halabi (Machine Learning Engineer, ING)
Key advantages of DataCater for this use case
-
DataCater pipelines can seamlessly stream thousands of
events per second, which makes it suitable for the real-time integration of multiple data sources providing live data on the air traffic.
-
In addition to more than 50 no-code functions, DataCater supports Python-based data transformations,
which are used by NoiseMap to transform geolocations.
-
DataCater continuously monitors the health of data
connectors and pipelines. When, for instance, a data
source system becomes temporarily unavailable, the user is
automatically notified such that they are immediately aware of the situation
and can perform an appropriate investigation.
-
DataCater fully automates the management of data connectors and can auto-heal failed connectors, once external data systems become available again.
