Open Source approach helps optimize costs by 20 times

myQuiz is a multicomponent .NET application created to host online quizzes for 10 to 100,000 players at once. From April, 2019, the platform has grown almost 20 times (increasing from 45,000 to 1 million visitors). Inevitably, monthly cloud consumption in myQuiz also jumped from $4,400 to $11,000, so we faced the challenge of optimizing the expenses of the application - and speeding up the delivery of new releases to deliver only positive experience to incoming users.

Users play directly on their mobile devices and compete against each other. They answer quiz questions invented by a host, or purchased in the platform’s marketplace. While, for players it is a fun game based on the speed of their answers, for businesses it is a gamification platform. Brands use the platform to mix fun and promotional content in branded gameplay. Thousands of players can compete in the same quiz held at a certain date and time. As an example, Vodafone used myQuiz to engage 13, 000 fans at the Gulf Cup.

High load capacity and ability to handle a large number of requests per second is a vital component of myQuiz. It must be accessible, quick, and scalable to handle every kind of event: from a small game of a dozen home players to an online party for over 5K users. Being a Microsoft Azure-based web service, the platform can handle these challenges. However, this scalability was not enough for our goals, and we decided to further optimize both the expenses for the application, as well as some steps of our development process.

Step 1. Switched to a parallel component delivery using TFS + Ainsible as an Open Source tool

As we said above, the platform is based on Microsoft Azure and .NET technologies. We use TFS with VisualStudio IDE, and PowerShell for the administrative tasks. The delivery process looked like this:

The parallel delivery of multiple application components could reduce the delivery time and minimize errors in the solution so we started our research of the best tool for the job. The built-in ability of TFS (so-called ‘parallel job’) allows for the defining of multiple agent phases in a single release and the assigning of tasks for each phase. This approach is a convenient way of deploying multicomponent applications.

So this could, in turn, be our ‘after’ delivery scheme:

When running more than one agent phase at a time, you need more parallel ‘jobs’, and normally, would need to pay for every additional job. To avoid additional expenditures, we chose an Open Source solution Ansible - a configuration management tool which can be used for many IT needs. Why Ansible:

• A simple declarative tool (there are ready scripts, which allow for minimizing of developers’ efforts) with detailed documentation
• It is able to run parallel tasks on multiple hosts out-of-the-box, with no licensing
• Scalable and secure (uses SSH or WinRM)

Pairing Ansible with windows services wasn’t an easy task, because Ansible was built to work with Linux. It took WaveAccess some time to research and try different approaches, until we ended up with the following combination:

• Ansible for application deployment (we wrote an Ansible Playbook - a set of scripts to show Ansible how a system should be configured).
• PowerShell as a configuration platform. Being a declarative platform, PowerShell offers a wide range of scripts to perform configuration tasks, setup services, etc, which saves us hours of work.
• Microsoft DSC module to enable windows systems configuration. DSC became a ‘glue’ to stick Ansible and PowerShell together.

From now on, our deployment scheme is:

Step 2. Use the TFS + Ainsible scheme to automatically renew SSL certificates

The strategy that we created in Step1 was then extended to automate another time-consuming task. Some SSL certificates expire every month, this previously had necessitated our developers manually renewing them for each component of myQuiz. Now we have successfully automated it, allowing developers to focus on delivering new features instead of engaging in busy work.

Step 3. Ainsible + Gatling for load testing

To test the application's performance, the myQuiz team uses Gatling Open Source Load testing: there are more than 20+ virtual machines, with 50,000 ‘users’ attacking the app simultaneously. Instead of manual checkout of every machine at once, we automated this task as well.

Despite being an Open-Source in its core, Gatling appears to be excessively costly when it comes to the use of parallel configured simulations: in the case of myQuiz, $19,00 per hour. In addition, it is found on Amazon Web Services hosting, which would require additional payments for the AWS cloud services:

Again, we used Ansible to parallel these ‘attacks’. To achieve this, REST API was used. Every virtual machine in our scale set has its own IP address. They can be reached via Azure REST API. Basically, Gatling containers start with a slight delay of 1 to 3 minutes one after another. However, because of the need to test the true app's load resistance we needed to start them all at once.

The result

Demand for myQuiz is increasing day after day and the question of quality has always been pivotal for us, but now is brought into sharp focus. After myQuiz consumption increased by 20 times, we had to optimize the overall costs for the application, deliver new features faster and perform even more extensive load testing.

Using PowerShell and Ansible OSS technologies we have developed an approach that can be used for any Azure based service, especially, in the case of its rapid growth and expansion. What started as a small project, ended up with multiple appliances:

• To cut costs for parallel delivery of the application to virtual machines in Azure cloud (cost savings is now $40.00 per each packet).
• Cut costs for performance testing: to parallel Gatling simulations (cost savings are now $5,000 to 4,000 per month).
• Eliminate routine: to renew SSL certificates automatically every month, allowing the team to focus on more important tasks.

Open Source approach transforms the way the application is delivered and tested. It becomes easier to overcome today’s challenges along with significantly reducing costs, and still enjoy the freedom of Microsoft Azure cloud.

We used the following Open Source technologies:

Ansible - to parallel delivery of artifacts (SSH certificates, Gatling containers, a ZIP archive with an application version), the core of this OSS project
Redis - a fast key value storage
Gatling - to perform load resistance tests
Selenoid - a cross browser Selenium solution for Docker to perform UI tests
Grafana, Prometheus - to monitor performance tests and to gather metrics every 5 seconds

Thank you for reading! If you enjoyed this article, then probably you'll be happy to learn how to work with Open Source projects in Microsoft Azure!