BI for incident management

Solution

Overview

Itransition developed a unified scalable BI solution, easily customizable by any client via a user interface. We expanded and improved the functionality of the previous solutions and added new features. This flexible approach allowed us to eliminate limitations on report types, with each given company using relevant templates. The solution’s functionality can also be expanded and modified as business needs change. 

The solution’s users are end clients’ employees who need to prepare and send out analytical reports in a format adopted in their departments and come up with suitable reports configuration. As a web application, the solution also supports adaptive design for tablets and smartphones.

The BI solution consists of:

1. Dashboard
   It visualizes incident analytics data in the form of graphs, charts, and diagrams and includes a geolocation map with the overall info on each object and associated incidents. Supporting historical data, the dashboard also shows the dynamics for each section. It includes the following features:

   a set of data visualization widgets preinstalled for each client as per the prior agreement, using terminology specific to their business;

   a flexible data filtering system;

   personal trackers for monitoring business-specific parameters;

   a terminology dictionary.

2. Dashboard configuration tool
   The tool enables the modification of layout arrangement, element dimensions, widget terms vocabulary, and other elements.
3. Tool for modular uploading client data
   The tool can remember the structure of files for subsequent automatic matching with the customer’s data model in the future.

Data upload and update

The customer’s proprietary data model is an incident classification system that can define the incident’s location, its date and time, and the business department where it happened. It also has an additional set of specific metadata used for determining incident details, like the incident’s type, how it occurred, its geolocation, and so on, with the metadata scope approved by end clients in advance. 

The incident classification system has a tree structure, with incidents divided into categories and subcategories. The solution’s algorithms take this structure into account to correctly process and visualize data.

As the previous solutions used different databases, which created some issues, the customer chose Microsoft SQL Server as a database due to its more strict and well-formalized data model. Being table-based, the database is more suitable for predefined schemes and applications that require multi-row operations. Furthermore, it supports foreign keys (the fields that allow for establishing the relationships between two tables) and has a strict set of columns in a table, aligned with the customer’s data clearly defined model scheme. 

The database choice also simplified the architecture and reduced the number of dependencies. While transitioning from MongoDB to SQL Server, we also implemented the accompanying architectural backend logic for working with the new database to improve user experience and performance. In fact, since we re-engineered almost all modules and components in the legacy systems, the end solution turned out a completely new one.

Next, we realized the complex process of multi-step data uploading by system users. The system implies that basic data is uploaded first, followed by the data which can repeat the already submitted information. For instance, a user first defines the business units that can provide reports, and then these business units are subsequently referred to in incident reports. For this purpose, we developed a tool for data uploading and template configuration that allows users to add new data types by defining relations and matching the file structure to the customer’s data model entities. 

For improved user experience, we implemented a flexible UI solution with drag-and-drop features enabled by the React DnD utility. Users can create templates and save them in the system for further use. Employees can also upload files of the same structure as CSV, XLS, and other formats, with the system accurately recognizing them. The solution supports modular uploads of reports, so clients are not required to upload all kinds of data at once, as each business unit can have only their part of the datasets and would upload them independently from each other. We also realized support for batch data uploads in any quantity and as many times as needed. The data contained in uploaded files is then transformed into widgets, graphs, diagrams, and charts and displayed in the dashboard.

The classification of incidents occurs with the help of natural language processing algorithms on the customer’s side and Azure Machine Learning API. Based on the incident’s text description, the system determines its category, assigns relevant tags to uploaded reports, classifies them, and displays them in final reports. Incidents can be classified into different categories, like their root cause, resulting damage, related topics, and so on.

We also implemented a mechanism for updating and synchronizing uploaded and stored data. To overcome the problem of too many simultaneous requests to the database, we enabled caching of frequently used entities in the solution’s RAM while leveraging search in dictionaries.  The solution accesses the database once to obtain existing incident data, compares it utilizing the solution’s RAM, and commits it to the database. Therefore, thousands of commonly used incidents and classification options do not have to be retrieved from the database each time, which minimizes the number of requests and improves performance. 

For example, when data is uploaded, the server receives the file and caches the existing incidents from the database into RAM. Next, the entries from the file and the incidents from the cache are matched by a unique entity identifier. Then the system creates a new incident or, if an incident already exists, it just updates some fields. As a result, we managed to reduce data upload time from three minutes to two seconds.

Dashboard and widgets

Next, we developed the dashboard with widgets as the main application page. Clients can view only essential widgets and configure them by size, arrange and reposition them to their liking, and use terminology specific to their company. We delivered a customized dashboard, where the customer’s admins can change the following:

• configure the set of widgets and their arrangement 
• modify widget size, text information, and titles depending on a corporate client 
• add and remove columns in classification tables 

Our team also developed the ability to create and edit dashboard configurations. Additionally, we integrated the solution with Google translate to automatically translate the uploaded text with pre-configured parameters.

Moreover, we improved the existing widgets for legacy solutions. Previously, to visualize data, each widget performed a series of data transformations to display reports in a widget-compatible view. However, due to excessive and complex data calculation algorithms, these transformations were cumbersome and redundant and contained unnecessary cycles and groupings, data processing could sometimes take a lot of time. We rewrote the logic of these transformations and implemented less complex and more efficient algorithms.

Besides, we significantly improved the map widget, adding points that users can click and view all the information about a specific object and related incident reports. Compared to the previous version, we made it resizable and, by adding dynamically changing color highlighting, allowed users to view activities in the given region. The solution now also supports the display of historical data depending on the date, incident types, and more.

Apart from that, we added a tool for creating trackers, associated with a specific group of incidents and giving a detailed summary of selected parameters. If the client does not want to view all data and prefers working with a particular segment, they can configure certain parameters in the filtering panel and create one or several trackers. This allows users to see incident dynamics in a certain section and define values to be tracked. When the defined values are reached or exceeded, the system notifies the user. The forecasting feature visualizes trends using a trend calculation algorithm.