Custom ML algorithms for an insurance platform

Solution 

Exploratory data analysis 

We conducted the initial project investigation to understand the company’s business needs and project expectations and then began with the EDA (Exploratory Data Analysis) phase. During this stage, our ML engineers analyzed the previous machine learning model and found it lacked scalability and capabilities for more accurate predictions. One of its flaws was its strict connection to the existing clients, while the customer was always acquiring new ones. Itransition also realized that the current solution could be improved not only by retraining the model using more relevant data of larger quantities but also by expanding the list of features the model uses. 

We made a report with our findings for ISC, explaining why it was necessary to: 

• Develop a new model from scratch 

• Carry out an additional feature engineering phase to significantly expand the existing functionality 

• Train the model to be less dependent on existing clients 

Feature engineering 

Before engineering new features, our team carried out data preprocessing and preparation. We analyzed the data to be used and cleaned it from outliers and incorrect data. Then our AI team developed capabilities that transform raw data into features that better represent the underlying business problem. Based on historical data, we created sets of features outlining the criteria for the likelihood of users’ completing the application process. 

Itransition’s AI team created as-is features based on available user data, such as: 

• Socio-demographic characteristics 
• History of client interactions with ISC (time from the first registration, purchased plans, number of registered requests, etc.) 
• Client background (such as the user’s business domain) 

Other features had to be calculated, such as the average insurance price, based on the number of purchases of insurance policies and policy costs. 

We developed 100+ features in close cooperation between our team and ISC specialists, which allowed us to merge our technical data analysis skills and experience with ISC’s domain knowledge. Our AI team trained the model to discard unnecessary features and select the ones that would be the most important prediction factors for the model. 

We experimented with three kinds of data manipulations: 

• Filtering data (using different filters, introducing different data filtering methods) 
• Feature configurations (changing the number of features, adding new features, giving different weights to features) 
• Different combinations of data (data partitioning, removing irrelevant samples, using different data configurations) 

Itransition found the most optimal feature sets and gave the correct weights to each feature for the model. We documented and presented the results of our experiments to the customer along with our interpretation of the solution’s vision to enable experiment tracking and reproducibility. Thus, with the help of sophisticated feature engineering, we ensured the model’s awareness of dynamic market conditions. 

Model training 

After creating the features, we proceeded to model training and ML solution development. Our team tried various decision-tree-based solutions and neural networks, but the final version of the model was created using CatBoost since it demonstrated the best results. 

Itransition’s AI team created an ML solution using 500,000 insurance application records for training the model and perfecting feature selection. We also used 50,000 records for testing to validate the model's accuracy. 

During model training, we modified some features to improve the ML model’s quality and remove features of low importance to the model during decision-making. 

We used the following metrics for assessing the quality of model training: 

• AUC (area under the ROC Curve) measures the quality of the model's predictions regardless of the chosen classification threshold. It also measures how well predictions are ranked rather than their absolute values. 
• F1 score measures model accuracy using the number of correct and incorrect predictions in a given dataset. 

Our team also utilized the model drift metric called PSI (population stability index) for measuring how much data has shifted over time and monitoring the applicability of a model to the current population. Data drift monitoring helps detect and triage issues with data in a working solution. 

As a result of model training and feature selection, we delivered a model that can make decisions based on around 20 most important features. This choice of the optimal feature set allowed us to increase decision-making speed, save costs on computing power resources, and enable rapid iteration and configuration. 

Production monitoring & support 

After we showed the results to the customer and discussed integration strategies, we deployed the model as a microservice and delivered it to the customer’s team for integration together with the solution’s technical documentation. The model was deployed to production by the customer’s team as a microservice and part of ISC’s infrastructure. 

Itransition constantly monitors and trains the ML model when there are new data or use cases. Using the dashboards we built, we also track model drift, i.e. the decay of models' predictive power caused by changes in real-world environments. When the customer wants to improve prediction capabilities but lacks knowledge on how to proceed, we collaborate closely with ISC to understand their goals and objectives. We then conduct analyses to identify potential use cases. Once we clearly understand the requirements, we develop additional features for the predictive model. Throughout this process, we maintain open communication with ISC’s team and work together to refine and validate the model. 

As a technical consultant and partner, Itransition provides guidance and expertise to help the customer improve the created model, thoroughly analyzing its work and identifying areas for improvement. During collaborative syncs, we work with the customer to set goals and establish a roadmap for adding value to the business. We have our unique approach and interpretation of strategies and technologies that would be most beneficial to the customer's business. 

We used the following technologies and tools for the project: 

• Streamlit — demos and results visualization and analysis 
• Flask — API development 
• NumPy, PandasAI, Matplotlib/Seaborn — data analysis and processing frameworks 
• Population Stability Index — developing a feature drift analysis mechanism