The crisis businesses are going through right now is essentially a data crisis too. The old ways of doing analytics are turning useless. Legacy data models are crumbling, and businesses are failing to get to grips with the new reality. As a remedy against this shake-up, Itransition offers the service of reactive data analytics—introduced and explained below by Aleksei Shinkarenko, BI Architect at Itransition’s hub of business intelligence consulting.
This crisis became a litmus test of the viability of traditional predictive analytics models, most of which have simply become useless. Today nobody can look at their data and say what exactly is happening or what comes next. Data went into its own kind of self-isolation—it strays from its historic legacy and refuses to reflect a real state of things, at least in a way we used to see it.
How did it all come to this? The thing is, data analytics has been built on the staples that got obsolete in an instant in today’s circumstances:
- Predictive data models that perpetually extend into the future turn false now since data cannot fit into the existing models of the simplified future anymore.
- The ability of data to represent reality as a continuum is now blocked.
- Process automation based on accumulated big data has now shifted back to the manual mode.
- Together with the data model logic and data accuracy, our accumulated knowledge now shows that we have been living under the ‘great illusion’ that most events could be predicted.
As you see, analytics that is built on the continuity principle stops supporting projections and biases during a crisis. What’s more, the current situation exposed the disconnection we preferred to ignore: we chose to speak for mute data ourselves and used analytics to drag data closer to our model of reality that we deemed to be more comfortable. The super-event we have faced—the pandemic exacerbated with a chain of social, economic and political complications—has made data immune to this wishful thinking.
We have to admit that the current situation evidenced by data has turned from a ‘black swan’ into the ‘elephant in the room’. The comfortable models of reality constructed and used since the Great Depression made us turn a blind eye to the fact that even the smallest deviations could represent a massive accumulative effect (on the climate, the global epidemiologic state, etc.) Those small deviations were ignored in the pursuit to bring everything to a common denominator or to average everything out, to draw a trend line or the average temperature. Potentially game-changing outliers were marked as exceptions and muted. We developed a love affair with the thought that our well-being would be never-ending.
Now data has become unchained. It doesn't need us and has become a cause for itself. It is powerful enough to finally dictate its own rules. It moves at breakneck speed and makes old algorithms unfit for keeping up with the change. An unknown algorithm replaced the old ones, and its scale is bigger than our already obsolete analytical mechanisms.
All this has deprived analysts of the ability to see the future. What we are left to do, however, is try to capture small, fragmented, unstructured, flash-like data on the premises of ‘not knowing’.
‘Not knowing’ is the new normal
Having no idea what to do next is not shameful. While everybody tries to predict the future, ‘not knowing’ can be instrumental for analytics in times of crisis.
The root of the data epidemic is that there is too much ambiguous data, and business decision-makers just don’t know what they want to discover with it. They can’t even find a way to articulate their questions.
In my work, I’m responsible for building the logic of data structures, decision flows, and data value chain models. However, to deliver sustainable results, I first help customers understand what exactly they need to analyze. I listen to customers and absorb everything they say, even tiny intentions hidden between the lines. This way I can detect their concerns and reconstruct their visceral demand they can’t quite articulate at the moment. I further zero in on their data to understand whether it can support their objectives.
In many cases, I see a mismatch between accumulated data and expected results. For example, a company may collect data to know more about their customers, but in fact their data is not about customer behavior but mostly about operations. I strive to make our customers see that it’s not enough to just own some data. Numbers can’t fuel decisions without proper interpretation.
To prepare the foundation for this interpretation, we apply the following two major R&D stages.
Stage 1
We get to know the company and try to find out as many whys as we can about their data sources, data warehouses, and data owners, the decisions they make and want to make, and how to turn these decisions into data-driven ones.
We also look for instances where data is manipulated to look convincing for different purposes. There is a general preconception that data should be a single source of truth, where two plus two always equals four. In reality, it’s always corrected by emotions and thus becomes biased. For this reason, bias detection and correction are important parts of our work.
Once a graphical model based on this first research stage is built, we show it to the customer. When they see their data presented as an ‘image’, they start searching for familiar chunks, asking what they can do with this piece of data or how it is going to work—all while talking their way toward the state of their business that they really care about but couldn’t define before. In doing this, they reconstruct their biases.
By following our customers’ own interpretation of the data laid out in front of them, we are able to prepare an accurate replica of their data perception at stage 2.
Stage 2
At this stage, we come back to the customer with feedback in the form of an interactive information display where they can see their data visualized as the answers to their questions and further put this display to use as the model they wish to experiment with in their planning. In other words, we design a platform where data can be interpreted to fuel the thinking process for answering two major questions—what I need to know and what I can do with this knowledge. With their analytics dashboards updated accordingly, they experience one of those a-ha moments—wow, they say, it’s exactly what I had on my mind but couldn’t spell out.
Of course, from the scientific point of view, our approach to data strategy is on the fringe. Traditionally, analytics has been perceived as a mathematical analysis and a logical approach. In our turn, we add a pinch of the irrational and the emotional, which is, frankly speaking, perfectly in tune with the times we live in.
‘Not-knowing’ is a benefit, not a weakness. Instead of lashing out due to your inability to control the situation, it’s high time for probing new ground, experimenting, and adapting gamified analytics.
Harnessing the change in extreme market conditions
Those who are fast on the uptake and ready to transform their analytical mechanisms of decision making will be immune to the deadliest market whirlwinds.
In order to get ready for this transformation, companies need to admit the following:
-
Crisis is permanent.
Crisis is here to stay, as it’s baked into capitalism as a means of renovation. It comes to the surface during such super-events as a pandemic, unleashing uncertainty and unpredictability.
-
Planning based on data continuity is inefficient during a crisis.
There’s no place for linear ‘if-then’ scenarios which are highly susceptible to failure in case of an obstacle. Our actions need to be of a reactive nature.
-
Data is breaking loose during the crisis.
We can’t control it and force it to represent the reality just like we did before by using the methods of repetitiveness and continuity.
The crisis is getting harsher, which means we need to rethink our attitude to data strategy right now. We need to admit that data behaves differently, and this behavior requires reconsideration of the current analytical systems, dashboards, monitoring systems, and new analytical skills. We should have no regrets when redefining the methods and systems that were our allies in decision making during better times.
Reactive data analytics as a vaccine for the business
The accumulated data, small and big, has become irrelevant. Why? Its shelf life has dramatically reduced. Data is speeding up, and it no longer works when made static on a dashboard. To match its lightning-speed development, we at Itransition came up with the new service of reactive data analytics.
We offer data-based micro-projects in order to:
- Quickly establish working metrics
- Design new data points and acquire data for the system
- Find data insights, or growth points
- Provide tools for a quick rebuilding of analytical systems and adapting to the new reality
Our service is based on the HADI (Hypothesis-Action-Data-Insight) cycle where we identify a hypothesis, test it in action, collect data throughout, and then conclude whether it’s going to work and what improvements need to be applied. This route of hypothesis testing is the safest one to take in times of economic chaos and uncertainty.
This service comes in the format of quick insight analytics, which is particularly effective when the customer has only a vague idea about what data is valuable but needs to find it out as soon as possible.
We make a quick analysis of the data structure, business processes, and accumulated data. We can work at any data access level. If the customer can share just a data structure, it’s enough for us to understand whether the existing events allow for collecting data to meet the required objectives.
Together with the customer, we brainstorm what can be done additionally or differently and what data types should be collected. Once the strategy is ready, we put best-suited technologies to work (for example, with the help of Tableau’s data science capabilities) and build an MVP, to be used for fast-testing ideas and delivering tangible results as early as in a few days.
How reactive analytics works: a case study
The customer came to us looking to prevent their helpdesk overload. They needed to see where exactly the overload happened so that they could detect the weak links in the pipeline and react once those links showed a critical level of workload. Unfortunately, this approach could never solve their problem as it was about retroactive response. What they really needed was to have the ability to take action proactively before weak links would send off the alarm, and thus gain the ability to fix them in real time.
When we analyzed the patterns of the anomalies, we saw that they lasted for a very short time—literally minutes, not hours or days. It was a sudden spike on the line which could grow 100% in just 15 minutes.
Consequently, such anomalies could serve as a sign to take an immediate action to curb overload before it actually happens.
With this in mind, we built an impulse monitor that analyzed reasons for the spikes in helpdesk requests, tracked the network load, and delivered overload suspicion alerts in real time.
What if the crisis blocks your data acquisition?
In the current crisis, companies might frequently lose any sense of direction because of a blocked access to their critical data. To help our customers get out of this data lockdown, we offer data acquisition as a service. We apply reverse-engineering to define what data types are needed for the task. Then by means of modeling and prototyping, we generate data and simulate those processes that help companies see the horizon and go forward.
Data acquisition during the lockdown
Just before the crisis, we delivered an MVP to a logistics company. The product was ready for the market release, but during the lockdown their processes halted as did their data acquisition, which was blocked due to external circumstances.
Soon enough, the customer put testing and planned product improvements on hold due to their inability to test hypotheses on real clients, get their feedback, and understand the true value of their service. In a nutshell, they were ready to sell their services but they couldn’t test how they fit the market. Obviously, they got stuck in a loop that could go on for ages. Their pre-crisis methods that were 100% efficient before, couldn’t be applied anymore.
To break this cycle, we offered to manufacture data and simulate product testing in order to still find out the needed insights about the service value and the potential of their data as a product based on the elasticity of the data structure.
Within the scope of a 60-hour project, we built a data sandbox where it was possible to prototype analytics to make it look real even without data being real.
With the help of our data toolkit, we were able to test initial hypotheses by asking the following questions:
- What types of data should be collected and offered to attract more users?
- How to design a flexible topological data structure to make data flow and analysis user-centric?
- Which IoT services and devices allow gathering more intelligence on shipment contracts and key agents?
As a result, we synthesized data, fed it to the MVP, and modeled various scenarios of the product’s future development. When the customer got a chance to see their product’s potential from the data point of view and learn about both preferable and negative scenarios, they could carry on with the project at full speed.
To sum it up, the customer couldn’t move on during the lockdown not because of the data shortage but because of their reluctance to abandon their pre-crisis ideas fed by the bias of sunken cost. They had invested so much effort in their ideas that they couldn’t just put them away and try something new.
We shattered the customer’s belief that a data-based product could become viable only when connected to a firehose of data. Instead, we demonstrated that a viable product could be a result of data structure reorganization. The crisis has actually turned their data structure into a product in its own right.
Once again I’ve seen that trust and sensitivity to the customer’s situation are underrated assets. In spite of our acknowledgement that we had no ready-made answers, the customer stayed with us and agreed to experiment. It proved to be a decisive factor in establishing a whole new level of our relationship. Before the crisis, we were just their technological partner. Now we are a product and business partner who can consult on a wide range of data services and offer data strategy. Instead of being just a developer, we have become a driving force of the customer’s business.
Reactive data analytics is the only way to fast business restructuring
The reactive data analytics strategy is a multi-faceted approach. These are the facets that have proved this approach efficient at responding to abnormal events:
Quantitative data is joined with emotions and a narrative
As mathematical models can’t predict anything anymore, we need to learn how to harness intrinsically fragmented data, previously camouflaged by the illusion of continuity. This illusion has been nurtured by long repetitive cycles; today, however, we can’t count on any such cycles.
Instead, we can take data fragments and analyze them as they are, without any generalizations or classifications. Then, by adding emotions or, rather, connecting numbers with real people’s emotions and connecting the dots by telling a story, we get a desired linear cause-and-effect route that helps us make decisions.
Data is democratized
In times of uncertainty, a rigid hierarchy undermines businesses. Action plans agreed on by all the parties turn irrelevant when the situation radically changes, forcing the parties to start planning all over again. To prevent such a waste of time, a siloed vertical hierarchy should be broken down into the network of micro-teams, or the so-called team of teams. Each team will have access to the unified data but from their own perspective. In a borderless organization, people are able to make data-driven decisions faster at each level.
Reactions are real-time
Data is like flashes on a global map that unpredictably appear here and there and make growth lines irrelevant to the current situation. Today, companies need to operate based on real-time analytics. Where previously there was a dashboard showing the situation developing over a month, now there should be one showing the situation developing over a day, in a heartbeat mode.
This kind of data strategy urges specialists to analyze impulses instead of generalized conclusions. It’s a brand new skill, as real-time reactions don’t provide tools for measuring something as good or bad. Instead, they give a quick representation of how the company’s heart beats, whether it’s pounding, slowed-down, or just healthy.
Infrastructure turns into ultrastructure
The ultrastructure allows zooming in on where data impulses appear. For example, instead of analyzing processes and applying this analysis to process owners, we need to first look at process owners themselves.
For instance, take insurance companies. Car insurance companies typically prefer to set higher premiums for more expensive vehicles. What they should really do is first collect data on prospective car buyers and set the highest premiums for amateur racers, for example, as this buyer category is more likely to get into an accident.
On post-pandemic analytics
The current crisis can become a tipping point for most businesses as consumers have enough time to develop new habits and continue to stick to them beyond the crisis. I’m sure smart business owners are starting to act on their customers’ new behavioral patterns right now and will acquire a different attitude to data strategy as well as planning and prediction models in the future.
Of course, the market will once again gravitate toward the comfortable normal where we will pretend again that data can forecast the future. However, as we have acutely realized that crisis could happen everywhere anytime, we will be taught well by this experience to keep an anti-crisis data strategy kit with ready-made models of reactive analytics. We should get out of the crisis convinced that those businesses that have the foundation for dynamic adaptability and possibility for quick regrouping and transformation of business models will always win.
I hope businesses won’t be reluctant to admit they did pretend data could reflect reality. To connect the dots, everyone was just building projections and drawing them nearer to the existing state of things to predict the future (or so it seemed) and make decisions based on those assumptions. We all were doing this, but now the lines of data and reality are so far from each other that we won’t be able to draw them closer any time soon. If you don’t believe me, just ask yourself: when you look at what data reflects right now, what do you see?
