DevOps for digital media
Itransition provides DevOps services and IT support for a European digital media publisher with over 1.5 million combined print and digital circulation.
Context
The Economist is a leading digital media publisher with over 1.5 million print and digital subscribers worldwide. They have hundreds of digital assets facing both internal business units and external readers. For instance, the customer creates microsites to cover topics that range from global economic events and business analytics to healthcare research and social trends.
To ensure business continuity and better user experience, The Economist works with external vendors that take care of DevOps and IT issues together with their in-house team. When several employees left the company, they started looking for more DevOps specialists.
In addition to that, the customer faced an issue with several microsites: with about 200 microsites overall, some of them started crashing on a daily basis without any objective reason. There were no changes in traffic, no issues on the server-side, no DNS errors or coding defects. The customer needed specialists to detect and eliminate the problem, as well as provide further support.
As Itransition had previously worked with The Economist and established a good working relationship, they decided to turn to us for DevOps consulting, specifically for solving the issue with microsites. When we eliminated the problem, the customer went on to hire Itransition DevOps specialists for environment setup and automation.
Solution
We started by detecting and fixing the issue with microsites within the scope of the pilot project. Itransition’s specialists found out that the problem was caused by an external API used by the GeoIP module to define website visitors, namely by the API’s slow performance that forced sites to go down.
We solved the problem by implementing a new algorithm for defining the microsites’ visitors. The algorithm includes downloading a publicly available GeoIP database, putting that data on the server, and letting the GeoIP module search in its database instead of waiting for external IP queries to work. This led to better performance and stability. After this pilot project finished, we kept supporting the microsites in case new problems were to occur.
After solving the microsites problem, Itransition’s DevOps experts moved on to the environment setup and maintenance as well as the automation of server deployment.
The microsites are either static or built on WordPress/Drupal. Our specialists develop and upgrade the scripts/templates to build these sites automatically or semi-manually. We created CloudFormation templates to set up the infrastructure and update site versions when needed. When accepting new microsites for DevOps maintenance, we perform joint code review and set up their Git in cooperation with the customer’s team.
An example of a script written by Itransition is the one for ‘housekeeping’ that allows for cost-saving and resource optimization. Before we joined the project, the environments had been managed by different teams since 2015. The customer saw that there were unused resources on Amazon servers, such as volumes, snapshots, file storages, and backups. We wrote a script that used the AWS Command Line Interface tool to check up these resources constantly and notify engineers of the ones that can be cleared up.
The script helps save around 5% of the budget spent on the environment management monthly. The customer also uses the script for other AWS accounts, and the aggregated cost-saving index is up to $4,000 per month. The yearly savings comprise up to $50,000.
We carried out microsites migration when the customer decided to separate the sites belonging to the company’s division responsible for research and analysis of global business and social trends from the rest of the microsites. We created the backup of the file system and the database, and then moved 10 microsites (Staging and Production environments) to a separate AWS account. It allowed for finance management and allocation between different departments, as well as for budget transparency.
After the successful migration, we started providing 24/7 support of these sites. According to the service-level agreement between the customer and Itransition, our specialists are to react to urgent issues within 1 hour, to high-priority issues within 2 business hours, and to spend from 1 to 2 business days on fixing other types of problems.
Process
Itransition’s Level 1 support specialists monitor the sites of The Economist’s business research unit 24/7 and get notifications from Pingdom and Pagerduty. They sort incoming requests and provide basic support and troubleshooting, such as password resets, break/fix instructions, ticket routing, and escalation to Level 2 and Level 3 support.
L1 specialists also notify the customer about issues and investigate their causes upon the customer’s request. For instance, when the system reloaded MySQL during backing-up or log rotation, Pingdom detected a very brief site unavailability. The issue did not require escalating, but we still detected the cause.
L2 support specialists process basic technical infrastructure and configuration issues for all the microsites, such as infrastructure capacity issues, server configuration, and database administration. They manage AWS resources with AWS CloudFormation and use Amazon CloudWatch for monitoring infrastructure, Incapsula for detecting and preventing intrusion, Pingdom for monitoring sites availability, and Pagerduty for escalating issues and getting notifications from Pingdom.
L3 support engineers process and resolve issues that require source code comprehension. They also perform regular maintenance and deliver security patches. They use PHP Composer to manage dependencies of the PHP software and required libraries. L3 engineers also work with Drush to control and administer Drupal websites.
Results
Itransition’s DevOps experts helped The Economist solve the issue of crashing microsites, and continues providing support to allow no more than 2 hours of downtime in case they go down.
In addition, the script written by our team within the scope of environment setup automation saves up to 5% of the monthly budget spent on the environment management for the microsites. Overall, this script allows the customer to save $50,000 per year in total. The relationship we have built with The Economist and the level of provided services led to the customer trusting us with the 24/7 support of the microsites belonging to their business and social research and analysis division.
