Solving Data Challenges in the Airline Industry – and Beyond

As the pandemic ends, travel continues to pick up. And that means airlines must be ready–even for the unexpected. The last holiday season proved a challenge for airlines as weather resulted in canceled flights, stranded passengers, incorrectly assigned crews, and, unfortunately for our tech brethren, questions about why airline systems struggled to keep up with fast-changing conditions.

Humans cannot control the weather. But we can respond to the data and information gaps that, in some cases, resulted in the airline industry’s inability to match crews with passengers and airplanes as it attempted to reshuffle resources. No doubt, the airline industry, in general, runs very well. It’s also a long-standing industry with legacy technologies, similar to many industries. I suspect that every airline industry executive would love to have the latest technologies, but rip and replace is often not cost-effective, nor optimal. 

Instead, airlines can incrementally upgrade certain components so that new capabilities can be added. This blog will address three challenges many companies face, including airlines, and data-focused recommendations to meet those challenges. 

All of them are tied to the notion of being able to act in real-time to better serve customers. In the airline industry, this means notifying people of flight delays, via the web, mobile, and any other channels, gate changes, mileage account updates, and enabling consumers to do almost everything they need, seamlessly and easily, even communicate with a gate agent via text or transfer miles to friends, without having to call and speak to someone. These kinds of experiences help drive customer loyalty, which has long been important in the airline industry.

Challenge 1: Over-reliance on legacy technologies and industry data sources. These legacy systems, for the most part, work well and handle a lot of operations. But they don’t enable the multi-channel communication that today’s consumers expect. They also make it harder to build newer applications and can hinder performance and the ability to scale.

Recommendation: In multiple industries, we see companies inserting a central data management system which we like to call a hot data layer because it is responsible for delivering your most important (“hot”) data. That is a data layer that incorporates data from different sources and different legacy IT systems and puts it in one central repository upon which you can build newer applications, especially web and mobile applications. A hot data layer is not just a database for running queries. It is a central repository upon which you can run applications that are mobile or web-based. End users interact with the data directly. A hot data layer is often called a digital integration hub or an operational data store. It has operational data, not analytical data. The data is processed and updated and, if necessary, propagated back to the original store.

Hazelcast serves as the basis for a hot data layer. The data transfer can be done in real time, and it can be done incrementally so the hot data layer has updated information at all times. This means that companies get a real-time view of data across many different sources. Any updates, like changing flights or transferring miles, can be done within this central repository.

Challenge 2: Disconnected data silos. This challenge is common across industries. Distinct data silos were likely purpose-built for a specific set of workloads, applications, and users. The need is to ensure they’re interconnected in a way that allows synchronization in real time. 

Unfortunately, we often see a slow process in synchronizing data across silos so you have inconsistent data across them and different views of a source of truth. Also, as you connect different data silos, you need custom code that adds complexity. Maintaining and optimizing the code between silos creates work that takes away from focusing more on new innovations.

Recommendation: Tie silos together with a microservices architecture where a simple and small application runs different pieces. Furthermore, they’re interconnected by a distributed system that stores data and gives one cohesive view of all data. Automate the transfer of data by using an underlying system that stores all data in a distributed way, keeping that global view of data consistent across each of the microservices. Each microservice operates independently and passes data to each other. That enables real-time applications so that you get that real-time views and processing, and your customers get a real-time experience.

Challenge 3: Batch processing. Companies run batch processing because their systems can’t keep up with real-time workloads. They wait for all the data, or a lot of it, cut off at some point, and run a batch together. This process creates wait times that lead to unsynchronized data and getting stuck in having to wait before another batch can be run. Adding more hardware to handle the load adds costs.

Recommendation: Replace batch processing with stream processing. Use a stream processing engine to process data one element at a time as they come in (which stream processing engines do well), enrich it with historical data, and then deliver information more quickly. A stream processing platform can handle data right away. It allows the synchronization of data immediately across different silos and enables real-time responsiveness. 

The Hazelcast Platform, a real-time stream processing platform, analyzes the stream of incoming data in real-time, and processes or modifies the data for some other use downstream, all in a low-latency storage system. It is a complete, integrated package for building streaming or real-time applications upon a distributed architecture that allows you to scale out or scale down if you need to for slower periods on commodity hardware. It provides a lot of different processing capabilities and a lot of real-time capabilities that can be added on top of existing infrastructure without having to throw any of your existing technologies out. 

ROI for the Long Haul

The return on investment in building IT infrastructure should be viewed with a long-term lens. What capabilities can be added? What defense mechanisms will be strengthened to guard against exceptional failures? What innovative features will set the foundation for growth? In the airline, or really any industry, these challenges can be addressed incrementally. Investing in leading technologies will enable companies to lead their industries in the unfolding real-time economy. No one wants to make customers wait. And, increasingly, customers expect real-time responsiveness.

To learn even more about this topic and how to best address these issues, please check out my video presentation for additional insight.