Achieving code-level visibility in production: How Site24x7 helps HPDK.com troubleshoot and scale its application

Hardware has its limits, and it is the responsibility of the programmer to utilize the full capacity of the hardware. A shortcut to improve performance is adding more servers, but doing so hides a lot of problems in the application design. Over time it might actually prove to be counter-productive as more servers lead to more maintenance, more time to rollout new features, and other inefficiencies. A problem in design can lead to frustrated customers, delayed schedules, lost business opportunities, unhappy employees, war room meetings, and more. Don't wait for doomsday; prepare for the unexpected.

This is where a monitoring tool is crucial for reporting performance over time, offering insights into your code by capturing slow-running transactions, and tracing which modules or functions are causing bottlenecks. You need a tool that can simplify operations and manage your applications without breaking your budget. Site24x7 APM Insight is the perfect solution for monitoring application performance.

Before the advent of managed languages like .NET and Java, profiling an application was a labor-intensive process. Even now, some developers solve performance issues the old-school way, by creating debug builds, printing stack traces of exceptions, writing context-sensitive logs, and undoing the script fixes after the bug is fixed.

Typically, the process of setting up a test environment and cleaning up after a fix is implemented manually takes at least a couple of days. With Site24x7's APM Insight agent, this process is a lot more automated. It happens in real-time and requires little to no change in the code base. All you have to do is install the agent and continue your business as usual.

HPDK.com is a SaaS vendor that provides a platform that helps businesses handle technical and sales support. HPDK.com customers create unique domain called portals, which are then used by their clientele. Clients raise queries, request technical support, and leave feedback on the portal. Technicians are added to the portals to address client concerns or escalate client issues.

As a startup entering the enterprise segment, HPDK.com was often asked by prospective customers whether 5,000 support technicians could access their application all at once. . Being a relatively new company, HPDK.com was built using the latest technology, including numerous libraries and frameworks, modular code, a normalized database, and API-driven clients. They were sure scalability wouldn't be an issue. HPDK.com went ahead with the request, but as 2,000 technicians began accessing customers' portals, performance problems arose. Their response time increased exponentially with each added technician

Although HPDK.com's portals performed fine for smaller clients, they had defects in their code that only became apparent with high volume usage. HPDK.com's application simply had enough resources to mask these underlying problems for small businesses, but when portals were accessed by more than 2,000 technicians at once, the average response time shot up to intolerable limits. It wasn't until a substantial number of technicians simultaneously accessed the portal that HPDK.com realized there was even an issue. Unsure why their application was performing poorly, HPDK.com deployed the Site24x7 APM Insight agent to troubleshoot performance issues as quickly as possible.

HPDK.com immediately set up their test servers, now loaded with APM Insight, and started simulating the performance issues. After collecting data for some time, the Site24x7 console revealed some shocking statistics.

Analysis

As shown in Figure 1, many of HPDK.com's API calls took over 200 seconds to respond. One of them, apart from being slow, was also a frequently called API.

SQL calls only contributed to 10 percent of the response time, but were still very high.

Calls to external servers were much higher in the APIs that took the longest to respond.

The delay was due to four methods called inside the API (Figure 2).

Three of the four methods were called multiple times within a single API request, suggesting unnecessary loops.

Almost all external calls were made by Redis and Cassandra (Figure 3).

Just this information alone wasn't enough to target the bottleneck. They deployed the APM Insight agent to their entire application to get a fuller view of their API. What they found was that the API had nearly 14 million functions called within itself (Figure 4).

The application team was literally searching for the proverbial needle in the haystack. Identifying a bottleneck out of 14 million calls is no easy job. But luckily, the Site24x7 APM Insight agent captures only methods that take a long time, ignoring the ones that execute in milliseconds

A detailed analysis quickly pointed out the following problems:

• One of the methods was called so many times that it contributed to about 25 percent of the response time.
• Two of the Cassandra queries were called multiple times for similar data, but varying parameters.

HPDK.com solved their performance problems by refactoring their API code, removing repeated function calls, and caching the result of SQL queries to reduce the number of external calls.

Once deployed into production, the results were perceptible immediately. What once took nearly 400 seconds took only 12 seconds after the fix was released (Figure 5). A 30x improvement in performance allowed them to scale up to not just 2,000 technicians, but 5,000 technicians without hassle, winning them their enterprise deal fair and square.

Performance and scalability problems are unique. They don't get fixed after just one shot. They have to be monitored continuously because when the code base expands, a seemingly innocuous design change or a one line bug fix may be the source of a sluggish web page. Site24x7 APM Insight will not just allow you to monitor your site, but can also provide you with the insights you need to scale up and respond to issues proactively.