Eliminating excess computation – Data warehouse optimisation creates significant savings for Fintraffic

Fintraffic’s task is to ensure safe, smooth and environmentally friendly travel on land, water and air. Its traffic data helps companies create new mobility solutions for people and goods.

Various forms of traffic generate massive amounts of data for Fintraffic. Data is generated by airplanes, cars, trains, ships, electronic signalling, weather monitoring equipment, traffic control, maintenance, as well as the producers and users of traffic services. With this data, Fintraffic is able to provide digital services and open data for traffic operators and end users as well as promote the transportation business’ chances of success in a networked world. One solution for making use of data is its analytics service, Fintraffic Datahub. 

Optimisation of the analytics service halves its monthly costs

As part of our continuous service, we help with the monitoring of the Fintraffic Datahub analytics service’s production environment as well as with coordination during incidents.

Processing large data sets requires a lot of computation. Fintraffic set out to find ways to develop data management and to better control the related costs.

“Our question was whether the long data downloads were done at the right time and at the correct volume. Organisations receive data from everywhere and much of the download volume is unnecessary. Every company needs to be critical about its use of data. Nearly everyone could stand to reduce the amount of needless computation. This would help to reduce costs and protect the environment”, says Antero Lundell, Solita Delivery Lead.

“At first, we took a good look at what kinds of actions generate costs – what makes up the tall bars on the cost report. We only started taking action when we had a sufficient understanding of the cost composition. What we did was question operating models based on the data in the cost reports. Things that were unnecessary were removed”, continues Lundell.

As a result of comprehensive data warehouse optimisation, long, half-hour downloads were reduced to a matter of minutes. Significant payloads were optimised and reduced by a factor. Unnecessary downloads were removed. Existing scheduled downloads were adjusted to match the actual use. “How many companies need Saturday figures on Sundays, is it enough to get figures in Monday morning?” asks Lundell.

As a compound result of these measures, we halved the monthly operating costs of the data warehouse.

“With the reduced amount and duration of the data downloads, the costs go down. The cumulative effect can be surprisingly significant over the years, as the small savings start adding up”, says Lundell. 

Optimisation helps all vendors

Fintraffic Datahub, the analytics service, is a multi-vendor platform. In addition to providing the continuous services, Solita is also one of the framework vendors. In the optimisation project, Solita was the coordinator, trainer, and supporter for all of the vendors. Optimisations were carried out by each developer. Seamless cooperation between the vendors was key to achieving a good result.

Solita arranged a workshop for every vendor, in which we focused on different perspectives on service cost and long data downloads. After the workshops, the vendors optimised their parts independently, and reported their findings to all vendors in a transparent manner. The concept Solita used to implement this work is technology-agnostic.

Fintraffic wasn’t the only party to benefit from the work: the vendors improved their competence together and comprehensively improved the operation of Fintraffic Datahub.

“How will a developer optimise the data downloads in their work and learn from it? How will the vendors find the things they needed to improve? How does the monitoring expose expensive and unnecessary operations? These were the questions we wanted to communicate to the vendors, to make them a part of every developer’s daily work”, says Solita Data Architect Mikko Puonti.

“What we did was question whether the data downloads were done at the right time and at the correct volume. We conducted the system towards a more sensible and environmentally friendly operating model. The fewer unnecessary downloads are made, the less power the computing environment consumes”, he adds. 

Environmental friendliness

Fintraffic’s strategy is to aim for the world’s safest, smoothest and most environmentally friendly traffic.

“Looking at our targets, this optimisation work matches them all. In addition to cost savings, this optimisation process resulted in significant results from an environmental perspective”, says Fintraffic Development Manager Tomi Lehtola. 

The annual savings in carbon footprint calculations add up to 10,205 kg CO2e. This nearly matches the average annual Finnish individual carbon footprint, which is 10,300 kg CO2e.

He also has praise for Solita’s way of managing the multi-vendor environment.

“Cooperation with Solita has been smooth as a whole, the people know what they’re doing and have a clear vision, and best of all – we’ve achieved demonstrable results working together”, Lehtola adds.

And the work isn’t over. We’ve prepared a set of instructions based on the current findings for future use.