Monitoring Web Applications in Dynamically Growing Environments
The reliable operation of applications, and consequently – customer satisfaction – is the key element in operating a business efficiently. Monitoring processes that can quickly detect a failure, find hidden dependencies and even predict a failure are essential tools in such dynamically growing environments.
You will learn from this article:
- What is application monitoring
- What tools are used to monitor environments
A Quick Introduction to Application Monitoring
The idea of monitoring can contain many different functionalities:
- sending notifications when a failure occurs
- visualizing data collected from various sources
- data mining
- establishing relationships between certain facts and their consequences
- predicting failures
Below, we will discuss the most important application monitoring tools, such as Icinga, Grafana, Kibana, Prometheus, and AWS CloudWatch.
Overview of application monitoring tools
Almost every monitoring system is based on metrics and the thresholds at which alerts are triggered and notifications are sent.
Icinga
Icinga is a tool that actively monitors the components of a given system. In the classic approach, where applications, databases and other components of a web application are hosted on servers, they can be monitored by an NRPE agent. The Icinga server sends them requests for specific sensors and metrics that check, for example, CPU load, available disk memory, length of queues, page loading time, or the number of database queries.
Depending on Icinga’s settings, you can define the tasks to be performed if a critical situation occurs, which can range from notifying the relevant people to performing an automated action.
Grafana
Grafana is a much more visually appealing tool, though inferior in terms of notifications.
It enables data visualization in the form of charts, indicators, and diagrams showing the key performance metrics of an application.
The data sources for Grafana include the databases ElasticSearch, InfluxDB, SQL, or AWS CloudWatch.
A well-constructed dashboard allows you to determine the current and historical status of the entire environment.
Graphs can be populated with relevant comments, such as running a newer version of the application, allowing you to easily find the relationship between the performance of the environment and the version of the application.
Grafana is also suitable for visualizing the metrics checked routinely, allowing a skillful administrator to predict failures – such as running out of memory or disk space – before they occur.
It also allows you to define simple notification rules, although in a much poorer quality than Icinga.
Kibana
Kibana is a dataset mining tool that enables the aggregation of logs from various sources.
In addition to functions such as browsing and efficient search of logs, it allows you to create views with graphs; it also features a timeline module, allowing you to analyze trends or data over time.
In addition to creating graphs and assembling dashboards, you can construct a timeline query that looks for anomalies between data, for example, week-to-week.
The database for Kibana is usually the ElasticSearch index, which can also be used by the aforementioned Grafana. The two tools combined offer a great set for viewing and visualizing the collected metrics.
Prometheus
Prometheus is a very good tool dedicated to modern environments: auto-scaled, dynamic, containerized, and with non-deterministic component lifetimes.
Integration with a directory of services issued by service discovery (e.g. consul, eureka) can be effected in a relatively simple way. With Prometheus, it is also possible to collect data using the http(s) protocol with various exporters.
Prometheus integrates with Grafana, using it as a data visualization engine, and with alert manager as a module for aggregating alerts and sending notifications.
CloudWatch
If our application is running in the AWS cloud, the CloudWatch service provided by Amazon seems like a natural monitoring solution to use.
CloudWatch collects basic metrics from running services automatically and without any configuration, but in addition, we can define our own configurations.
The AWS web panel provides us with integrated functions to view them, create charts, indicators, dashboards, although – if needed – the same data can be accessed via API.
Alerts can also be configured and integrated with SNS notifications or Lambda functions. We can easily program events that integrate with other AWS services, such as scaling an ASG group based on a CloudWatch alert.
Another feature of CloudWatch is the storage of application logs along with the option to view them using SQL queries (Athena) or Insights (GUI interface).
It is worth remembering that CloudWatch can serve as a visualization base for Grafana.
What should be monitored?
Every project should monitor business-critical metrics, such as website loading speed or the handling time for customer requests.
The quick detection of defects in development or acceptance environments helps to avoid situations in which potentially ineffective code is applied in production environments. When designing monitoring systems, one should remember to imitate the normal behavior of Internet users as closely as possible – query applications using public addresses, verify DNS name recognition, and check the validity of certificates.
Features available within specific applications:
- Icinga – notifications, sadly without charts or log analysis.
- Grafana – charts and notifications (limited scope).
- Kibana – charts and log analysis, but no notifications.
- Prometheus – charts (additionally you can link Grafana) and notifications (via alert manager module).
- CloudWatch – the complete set from graphs to log analysis and notifications.
Summary
Well-organized monitoring allows you to not only react to failures but also to effectively predict and prevent them. If a failure does occur, we should be able to determine the cause of the failure using monitoring and define the actions to prevent the failure from occurring.
It is definitely worth taking the time to set up the comprehensive monitoring of environments not only for production but for all stages of software development.
Finally, the most important thing is the satisfaction of the application users, and without measuring its performance, it is hard to guarantee the quality of service.
If you would like to learn more about monitoring services, feel free to take a look at our managed services offering.
