JackSparrow414

Elasticsearch and ELK in Practice

From setup, queries, and the Java client to logging configuration and collection, ILM, cluster deployment, monitoring, and application optimization.

11 posts · In reading order

  1. Setting Up ELK and Getting Started

    Installing Elasticsearch, Logstash, and Kibana and getting started with the basics.

  2. Querying Elasticsearch

    This article focuses on data queries, an important part of working with Elasticsearch.

  3. Practical Elasticsearch: Common Operations, Logstash Integration, Local IP Handling, and ECS Field Mapping

    After the previous two blog posts, we now have article data and can query it. The next step is to keep building on that foundation.

  4. Generating PEM CA Certificates for ELK, Enabling HTTPS, and Connecting with the Elasticsearch Java Client

    Using ELK 8.2 as an example, this post walks through generating a CA certificate, configuring TLS for internode communication and HTTPS at the HTTP layer, and configuring certificates for Kibana, Logstash, and the Java client. It also covers the differences between the PEM and PKCS#12 formats and the related JDK compatibility issues.

  5. Using the Elasticsearch Java API

    Starting with the Search API request structure and Query DSL, this article explains how to organize Elasticsearch query conditions, sorting, returned fields, and pagination. Examples show how to convert DSL into Java API calls and enable request logging for debugging.

  6. Shipping Tomcat Access Logs from EC2 to ELK with Filebeat and AWS CloudWatch Logs

    An end-to-end approach to sending Tomcat access logs from AWS EC2 to ELK: CloudWatch Agent collects the logs, Filebeat pulls them from CloudWatch Logs, and Logstash parses and indexes them in Elasticsearch. It covers configuration, pipeline verification, and troubleshooting.

  7. Shipping Tomcat access_logs from EC2 to Elasticsearch with Filebeat and AWS CloudWatch Logs, with Automated Log Management via ILM

    In a log collection pipeline from CloudWatch Logs to Elasticsearch, Data Streams and ILM are used to manage index rollover and expiration-based deletion. Drawing on hands-on practice, this article describes the tuning process for Filebeat's memory queue, bulk write, and concurrency parameters, and records issues such as latency, cost, and occasional small amounts of log loss.

  8. Building Elastic Stack from the Official Documentation: A Three-Node Elasticsearch Cluster, Kibana, Filebeat, Metricbeat, and Migration Without Downtime

    Uses Docker Compose to build Elastic Stack with a three-node Elasticsearch cluster, Kibana, Filebeat, and Metricbeat, explaining TLS and monitoring configuration. Drawing on a production migration, it discusses moving from one node to a cluster through domain switching, data migration, and connection verification.

  9. A Practical Guide to Elasticsearch in Application Development, with a Real Optimization Case

    A checklist for using Elasticsearch in application development, covering analyzer validation, field mappings, index planning, lifecycle management, and capacity estimates. A real logging use case illustrates performance problems caused by cleanup and writes, along with improvements using data streams, circuit breakers, and batching.

  10. Automating AWS EC2 Creation, Elasticsearch and Kibana Installation, and OpenTelemetry Monitoring

    AWS CLI scripts create an EC2 instance and internal DNS record, while properties files and User Data organize deployment parameters and initialization. Using Elasticsearch, Kibana, and OpenTelemetry as examples, this article covers server configuration, automated deployment, and verification.

  11. Replacing Database LIKE Queries with Elasticsearch: Approaches and Implementation Details

    For a large order dataset, Elasticsearch replaces database LIKE searches on contact information, followed by database lookups for orders. The article covers index design, historical and incremental synchronization, UTC handling, pagination, database fallback, and synchronization gaps.