Clare Avieli is the Director of Product Marketing at CA BlazeMeter, where she leads the digital marketing team. She has 15+ years of experience in marketing, PR, and writing, with a specialization in technology. Prior to CA BlazeMeter, Clare led the online and content marketing strategy at Panaya (now Infosys) and was the marketing manager at Nolio, now CA Release Automation.

Become a JMeter and Continuous Testing Pro

Start Learning

Test Your Website Performance NOW! |

arrowPlease enter a URL with http(s)
Jun 21 2016

Which Cloud-Hosted Elastic Stack Solution is Right for You?

The Elastic Stack (Elasticsearch, Logstash, Kibana and Beats) is the world’s most popular open source log analysis platform. With downloads growing from 25m to 55m year over year, Elastic Stack is quickly overtaking existing proprietary solutions and becoming THE first choice for companies shopping for log analysis and management solutions.


Setting up the Elastic Stack is a complicated task that requires a lot of expertise, so many companies these days are instead looking for “hosted Elastic Stack,” or “Elastic Stack as a service,” or “cloud-hosted Elastic Stack” solutions whenever possible.


Cloud-hosted Elastic Stack solutions all seek to offer an easy and scalable way to use the stack, but they differ in many ways. Recognizing the growing role that the Elastic Stack is playing in the world of log analytics, we thought it would be interesting to compare the leading solutions by examining the criteria that is crucial to check when contemplating which Elastic Stack logging platform to use.


What is the Elastic Stack?


The Elastic Stack is comprised of three separate yet homogenous open-source products: Elasticsearch, Logstash, and Kibana. Elasticsearch, probably the more well-known of the three components, is the search engine that powers the stack. Based on Apache Lucene, Elasticsearch can be used to perform full-text and other complex searches. Logstash processes the data before sending it to Elasticsearch for indexing and storage.  Kibana is the visualization tool with which you can view the log messages and create graphs and visualizations.


Why is the Elastic Stack so popular?


The Elastic Stack is popular for a number of reasons. First, IT companies are moving more and more of their architecture -- including logging systems -- to open source technologies. Second, existing logging platforms are too expensive for small and medium-sized companies. The Elastic Stack, for example, might not have all of the features of Splunk, but it does not need that feature-richness. The Elastic Stack is a simple but robust log analysis platform that costs a fraction of the price. The community being created around the Elastic Stack is another reason for the growing popularity.


#1 - AWS Elasticsearch


As the leading cloud service provider, it was just a matter of time until AWS introduced its hosted solution for the Elastic Stack.


Introduced in October last year, this service enables AWS users to setup and configure an Elasticsearch cluster from the AWS Management Console. The service then provisions all the resources for the cluster and launches it.



Monitoring, backups and security is handled by integrations with complimentary AWS services. IAM policies (Identity and Access Management) controls access to Elasticsearch, automatic and manual snapshots store data on S3, and CloudWatch and CloudTrail provide the monitoring and auditing services.


Scalability, which is quite the issue in large production environments, is not automated. Users are expected to monitor the cluster using either CloudWatch, or any other weapon of choice, and manually add resources as the need arises. In case of node failure, the failing node is automatically detected and replaced.


The process of log parsing and mapping needs to be handled manaully by the user, with full support for Logstash.


The service currently ships with Elasticsearch 1.5.2 and it is not clear how upgrades to newer versions are handled and what happens with the data during the transition period.


Pricing for the service is pay as you go and according to EC2 policies. A nice option is to use the Free Tier for development -- with 750 hours per month for a single-AZ t2.micro instance and 10GB a month of optional EBS storage, but you will quickly surpass these limits with a reasonably-sized production logging system. Keeping tabs on your AWS costs is a challenge and will most likely necessitate using a cloud cost management service.


#2 - Elastic Cloud

Elastic -- the commercial entity behind the Elastic Stack -- recently released its own cloud platform called Elastic Cloud, based on AWS infrastructure.


Elastic offers a free trial option for 14 days, with 1GB memory and 16GB for storage. Creating an account is easy, after which you are given access to a management dashboard. From this dashboard you can then configure your new cluster -- its size, region, the number of data centers you want to use, the version of Elasticsearch you wish to use, number of shards and more.


For security and managing user access, you are provided with Elastic Shield (you cannot access Kibana without first enabling and configuring Shield). You also are given access to other Elastic services: Marvel, Sense, Timelion, and Graph.



Backing up your logs is taken care of by snapshots that are taken every 30 minutes and stored on S3.


Like with AWS, scaling is done manually by the user, as is the logging pipeline (aggregation, parsing and mapping).


As for pricing, this of course depends on the cluster setup and your needs. A medium sized cluster for production use, with 64GB of reserved memory and 1TB of reserved storage, will cost around $2600/month.


#3 -


This platform offers a somewhat different approach by providing a more complete Elastic Stack service. has done a good job of publicizing the platform’s architecture, so some information is available on how they process the data coming into the system.


The platform uses Kafka as a message queue for all of the incoming messages, including those from Logstash (queuing is a common best practice for logging systems, and is used so as not to overload Elasticsearch). As part of the service, provides auto parsing, auto sharding, and auto mapping for logs, so the ingestion pipeline is all taken care of automatically. 


There are various log shipping integrations available, so users can quickly integrate their existing environments and establish a shipping pipeline. Generic shippers such as Filebeat, Logstash and rsyslog are all supported, together with native support for AWS, Docker and specific languages.


As with Elastic and AWS, logs can be archived to S3. offers role based access for security and basic user management features.  An important feature to point out is the ability to create alerts, based on saved searches, for events you want to be notified about, either via email or a chatting program (e.g. Slack).


Another nice feature is Elastic Stack Apps -- a collection of pre-made Kibana visualizations for various types of logs (e.g. Apache, AWS, Docker).


There is a free trial available for 14 days restricted to shipping up to 1GB a day. The pro plan costs $89 a month, and includes 1GB a day of log shipping with 14 days retention and S3 archiving.


#4 Sematext Logsene


Logsene is Sematext’s Elastic Stack-as-a-Service offering, and compliments the other analytical tools and services Sematext offers.


Creating a new Logsene application is easy enough, and once created, you can either use the Sematext UI to manage and analyze the logs or the integrated Kibana 4 (you can also hook into your own Kibana or Grafana.) As with, Sematext offer a wide variety of integrations with standard logging methods and platforms, but again -- parsing of logs and the ingestion pipeline is up to the user.


A feature worth mentioning is Live Tail for live viewing of incoming messages -- very useful for monitoring your environment for errors taking place in real time. Sematext also offers an alerting mechanism that integrates with chatting programs so you can get notifications about events.


For monitoring performance metrics in your environment, you can use Sematext’s SPM offering within the same UI, which is useful for identifying correlations in the data.


Sematext includes full role-based access control with owner, admin and user roles, and also provides an on-premise option for users who can’t ship logs to the cloud.


The pro pricing plan costing $60 a month caters for 1GB a day, 7 days retention, and S3 archiving.




To sum things up, we’ve provided a comparison table below that will help you see the big picture. Different challenges necessitate different solutions. It all depends on your specific requirements.


What is clear is that if you’re looking for a log analytics solution based on Elastic Stack as an end-to-end service, the solution offered by is the most functional and most compliant offering. If you’re looking for a searching solution only, Elastic provide an excellent solution. AWS offers a low-cost solution but requires a substantial amount of work to integrate with other AWS services and establishing the log pipeline. Sematext offer a good solution, and we especially liked the concept of live tail, but found the pricing a bit expensive.


arrowPlease enter a URL with http(s)

Interested in writing for our Blog?Send us a pitch!