Dedicated social game development center
11+ years of dedicated software development, testing and support resulting in a scalable social gaming platform sustaining 100 million users and multiple performance improvements.
Context
The customer is a world leader in land-based, mobile and online gaming with presence in 90 countries, licenses in 250+ jurisdictions, and 3,500+ employees globally. With over 1,000+ granted patents and 3,000+ trademarks, the customer continues to invest in innovative gaming technologies. They offer a wide range of products and services, such as award-winning gaming systems, social gaming solutions, top-performing cabinets, and casino management platforms.
The customer’s goal was to extend their offering on the global market by releasing new products and services while updating the existing ones. Frequent changes to their portfolio required a solution for managing and configuring the entire ecosystem of games and related events, marketing and advertising, as well as user activity analytics. Also, the customer looked for backend development for their gaming solutions.
On top of that, six geographically dispersed teams were working on multiple projects with complicated processes requiring a solid collaboration and communication strategy to achieve common goals. Extensive project documentation had to be structured for fast ramp-up and onboarding of new team members.
To help manage their digital product portfolio and multiple global projects, they were looking for a technological partner to join them as an organic part of their business. Relying on our proven expertise in custom software engineering, support, QA, DevOps, and remote collaboration, the customer chose Itransition to set up a dedicated development center and become an integral part of their product team.
Solution
Starting in 2009, the collaboration was to last for 11 years and counting. During this productive time, we delivered a range of services and solutions stage by stage.
Stage 1: an application manager for Facebook games
Itransition’s team developed a scalable resilient application manager with an advanced back-office capable of handling tens of thousands of concurrent users playing Facebook games. This application manager serves as a central management and configuration hub as well as a control panel for global application delivery, business analytics, and marketing.
We chose to create the solution’s business logic on Ruby on Rails for its high performance, scalability, and straightforward maintenance. At the database level, we used the MySQL Percona server.
The application manager enables the following:
- Setup of targeted marketing campaigns via a custom tool based on the Facebook Ads API
- Payments for services, in-game currency management, and bonus acquisition
- Gaming and user data analytics
- Activation/deactivation of applications and games
- Game settings and ACL configuration
- Facebook application notification management
Stage 2: a mobile gaming engine for land-based casinos
At the second stage of our dedicated center program, we customized the online engine of the customer’s original social slots for use on mobile devices at land-based casinos.
The social slots game had been enduring consistently high loads due to its popularity, yet was hosted on multiple bare-metal servers with no autoscaling. Other challenges included disorganized legacy code and several fixes necessary for a set of interdependent apps.
The goals for this stage were to enable autoscaling and rebrand the game for each specific land-based casino, where visitors would be offered to install the mobile app on their devices.
We developed the backend, focusing on making the deployment and configuration of the mobile game as simple and quick as possible. We deployed the game’s development clone on the Google Cloud Platform infrastructure, ensuring all of the following system components communicated with each other and supported the mobile game client:
Atlas: the component encompasses the API and the back-office as well as contains game settings such as visual assets for the mobile client, slot configurations, etc.
Faye: the component enables direct communication between Atlas and the mobile client.
Segmentation engine: the component allows targeting players by their demographics and gaming preferences.
Beacon: the component enables proximity-triggered notifications and personalized push notifications to players.
We also introduced the following improvements:
- Removing unused modules and libraries.
- Customizing the code to fit the specifics of each land-based casino.
- Improving the user authentication experience by integrating Amazon Cognito.
- Integrating Player Max.
- Enabling autoscaling and running performance tests, which optimized hosting costs and allowed to sustain the load of up to 24,000 concurrent players.
Stage 3: quality assurance
Throughout our engagement in social game development for the customer, software testing was an essential part of all the delivered solutions.
Performance testing
We tested the performance of the application manager to ensure it would sustain the load of 10,000 concurrent users with the response time no longer than 0.2 seconds.
For this, our QA engineers mapped real user journeys for load emulation in cooperation with the software developers working on this product. The team developed load scripts with Apache JMeter and uploaded them to GitHub for the customer to keep track of the progress in real time. Other tools employed for performance testing included Fiddler, Stackdriver, InfluxDB, Grafana, and Redis.
The sub-types of performance testing spanned stress, scalability and load tests.
Stress tests uncovered the bugs preventing the solution from sustaining gradually increasing load and allowed tweaking the application to increase its manageable load tenfold.
Scalability tests helped identify how many machines were to be added to Google Cloud Server Autoscaling to evenly distribute the system load and adjust the performance proportionally to the hardware resources consumed.
Load tests confirmed that the response time remained constant and within the normal range during five-hour intervals identified as the average user session duration.
At the end of performance testing, we compiled a final report detailing the results and our recommendations. Our thorough testing strategy helped the development team fix bugs early and meet the customer’s expectations without reaching critical errors.
Test automation
Itransition provided test automation solutions, development, and feature updates in a test harness to support the customer's new products and verify their quality. The QA team covered the host server with autotests using Rest and WebSocket, rewrote the existing framework using C#, Cucumber (Specflow), and TypeScript, and performed UI testing automation for the admin portal.
We also developed a mobile test automation Java add-on for Appium, a tool allowing the mobile gaming app to communicate with the native device elements (for example, to make purchases on App Store or Google Play). TestNG, part of Appium, was used to organize and run automated tests for the iOS and Android versions of the app.
The complete structure of the interconnected tools used for automating and running mobile tests looks as follows:
Stage 4: support and maintenance
Itransition provided support and maintenance services as well as environment setup automation for the online casino game integrated with Facebook, Google Play, and App Store.
To optimize teamwork and automate environment setup and maintenance, Itransition’s support engineers performed a range of tasks in the following areas:
Environment setup automation: we introduced tools for server configuration, provisioning, and orchestration, and set up automatic virtual environment activation.
Configuration management: we maintained the platform’s consistent performance.
Local testing scripts automation: we adjusted the platform for continuous integration, made automated unit tests part of the deployment process, and automated repetitive test cases for time-efficiency.
Deployment automation and continuous delivery: we deployed applications across the development environments to provide efficient, reliable, and predictable automation results.
Our contribution resulted in the team server configuration time dropping by 2.5 times, the testing time reducing by 5 times, software quality improving by 57%, and deployment accelerating by 3 times.
Process
The dedicated development center has been the underlying engagement model throughout the 11+ years of our cooperation with the customer. Itransition’s team of 80+ people consists of six sub-divisions working on social game development and ancillary services such as support, maintenance, and QA. Each sub-division consists of 10-20 developers, a project manager, a business analyst, and 2-3 QA engineers. The dedicated team growth and optimization plans are reviewed by Itransition and the customer quarterly.
From stage one, we chose Scrum with 2-week sprints as the development methodology to facilitate incremental product delivery. We use Atlassian Jira for planning and task management. Our Scrum Masters manage Agile implementation and coordinate R&D to help the customer achieve maximum business value on every sub-project.
To automate infrastructure management, we appointed a DevOps team. Their implementation of Infrastructure as Code allows the customer to build a new environment from the ground up in minutes. Our DevOps engineers also moved the backend infrastructure to Kubernetes, which further improved automated delivery. To enable continuous integration, we use Jenkins and Bitbucket.
Our project communication modes include sprint planning and review meetings, weekly, monthly and quarterly sync-ups, daily calls, as well as onsite meetings with the customer’s key decision-makers.
Results
With 11+ years of cooperation and 80+ of Itransition’s professionals working on the project, this proved to become one of our biggest and most long-standing accounts ever. By helping this world-leading gaming provider with a range of solutions spanning social game development, cloud computing, software testing, DevOps, and support services, we achieved the following milestones:
- Developing a scalable platform for managing Facebook games with 100 million users per installation, 450,000 daily active users, 2.2 million monthly users, 40,000 requests per minute with the average response time of 45 milliseconds, and 120 GB of data accumulated daily with its further analytical processing.
- Deploying and configuring a mobile game for land-based casinos.
- Optimizing hosting costs and enabling autoscaling with the maximum load value twice as high as the expected target.
- Cutting the server configuration time by 2.5 times, accelerating testing by 5 times, improving software quality by 57%, and speeding up deployment by 3 times.
- Arranging complex collaboration processes across multiple projects and setting up a unified knowledge sharing hub with detailed project documentation.
