© OpenStreetMap contributors

Collecting and Enriching Air Traffic Data in Real-Time

This case study shows how the NoiseMap application by the ING Analytics Frankfurt Hub uses DataCater to collect and enrich air traffic data in real-time.

If you have ever been searching for a new apartment or house in an area that you don't know well, you have probably wondered how its noise level will be. ING Analytics Frankfurt Hub's NoiseMap application provides an answer to this question.

NoiseMap collects air traffic data from multiple data sources over time. Based on the historical traffic data, it calculates the unique noise level for each geolocation using a custom scoring function. The noise levels are visualized on an interactive map.

Most of the used data sources don't provide historical information on the air traffic but only a snapshot from the time that the data source is being accessed. This makes maintaining a historical changelog of the air traffic inside NoiseMap a necessity.

The team behind NoiseMap, led by Dr. Nawar Halabi, uses DataCater to continuously extract data from multiple air traffic systems and stream them to a PostgreSQL-based data warehouse. While streaming the data, multiple filters and transformations are applied, such that PostgreSQL stores the data in a format suitable for the calculation of the noise level.


“As a Data Engineer, I know how difficult it is to deploy a streaming solution. With DataCater it is made a lot simpler and without writing code. I don't have to worry about scaling and I can manage my pipelines with an easy-to-use online interface.”

Dr. Nawar Halabi (Machine Learning Engineer, ING)


Key advantages of DataCater for this use case

  • DataCater pipelines can seamlessly stream thousands of events per second, which makes it suitable for the real-time integration of multiple data sources providing live data on the air traffic.

  • In addition to more than 50 no-code functions, DataCater supports Python-based data transformations, which are used by NoiseMap to transform geolocations.

  • DataCater continuously monitors the health of data connectors and pipelines. When, for instance, a data source system becomes temporarily unavailable, the user is automatically notified such that they are immediately aware of the situation and can perform an appropriate investigation.

  • DataCater fully automates the management of data connectors and can auto-heal failed connectors, once external data systems become available again.

ING-DiBa AG

With more than 9 million customers, ING-DiBa AG is the bank with the third most customers in Germany. It is active in the core business areas of savings, mortgages, securities business, consumer loans, and checking accounts for private customers. ING has been named Germany’s "Most Popular Bank 2019" by the business magazine "Euro".

DataCater

DataCater is the self-service platform for streaming data pipelines, which makes Apache Kafka®-powered streaming data pipelines accessible to data teams. DataCater can be used to integrate a wide range of data systems, such as database systems and web APIs, in real-time.