Requirements

Imagine you're building the core of a logging and analysis system, a simplified version of Splunk. This system needs to efficiently ingest logs from various sources, store them, and allow users to perform queries with filters based on keywords and timestamps, specifically focusing on "last X minutes/hours/days". The challenge lies in designing a system that’s not just functional, but also highly maintainable, extensible, and capable of handling concurrent queries. Performance is also key, but the focus here is on the data structures and algorithms *within a single process* that enables efficient querying, not on distributed systems architecture. This problem is about crafting elegant, robust, and scalable in-memory data structures and algorithms for log management. Think about how you'd represent log entries, how you'd index them for efficient searching, and how you'd design the query processing pipeline. # Requirements ## Functional Requirements 1. **Log Ingestion:** * Accept log entries from various sources. Each log entry consists of: * A timestamp (e.g., `java.time.Instant`). * A log level (e.g., `INFO`, `WARN`, `ERROR`, `DEBUG`, `TRACE`). * A message (String). * An optional set of key-value pair metadata (e.g., `{"service": "auth", "user_id": "123"}`). * The system should be able to handle a continuous stream of log entries. * The order of insertion of log entries matters. 2. **Querying:** * Support queries with the following criteria: * **Text Search:** Search for log entries containing a specific keyword in the message or metadata values. Should support simple substring matching. * **Timestamp Range:** Filter log entries within a given time window (e.g., "last 5 minutes", "between 10:00 and 11:00"). * **Log Level:** Filter log entries by log level (e.g., show only `ERROR` and `WARN` logs). * **Metadata Filters:** Filter log entries based on specific metadata key-value pairs (e.g., `service = "auth"` AND `user_id = "123"`). Supports equality matching only. * Queries can combine multiple criteria using AND logic. * The system should return a list of matching log entries, sorted by timestamp in ascending order. * Support pagination of query results (e.g., return results in batches of 100 entries). 3. **Time-Based Queries ("Last X"):** * Provide a convenient way to query for logs within the "last X minutes/hours/days". This should be easily configurable and extensible to other time units. ## Non-Functional Requirements 1. **Thread Safety:** The log ingestion and querying components must be thread-safe to handle concurrent requests. Consider multiple ingestion threads and multiple query threads running simultaneously. 2. **Extensibility:** * The system should be easily extensible to support new log sources, query criteria, and output formats. * New metadata filters (beyond simple equality) should be easy to add without modifying core classes. * Adding different storage backends should be possible without impacting the query processing logic. (e.g., consider abstracting the log storage mechanism) 3. **Modularity:** The system should be divided into well-defined modules with clear responsibilities. Favor loose coupling between components. 4. **Testability:** The code should be designed to be easily testable using unit tests and integration tests. Minimize dependencies and use dependency injection where appropriate. 5. **Performance:** While the focus isn't on distributed systems scaling, the in-memory data structures and algorithms should be efficient for searching and filtering large volumes of log data. Consider indexing strategies. Avoid unnecessary object creation. 6. **Error Handling:** Implement proper error handling, including logging and exception management. Gracefully handle invalid queries or corrupted log data. 7. **Configuration:** The system should be configurable, allowing users to adjust parameters such as the maximum number of log entries to store, the indexing strategy, and the pagination size. ## Core Entities 1. **LogEntry:** Represents a single log event. Contains timestamp, log level, message, and metadata. This is the fundamental data structure. 2. **LogIngester:** Responsible for receiving log entries and storing them. Handles concurrency concerns for ingestion. 3. **LogStorage:** An abstraction for storing log entries. Could be an in-memory list, a file-based storage, or a database connection. Define an interface to allow for different implementations. 4. **Query:** Represents a log query. Contains the search criteria (keywords, timestamp range, log levels, metadata filters). 5. **QueryEngine:** Responsible for processing queries and retrieving matching log entries from the `LogStorage`. Optimizes query execution. 6. **Filter:** An abstraction for different filtering criteria (text, timestamp, log level, metadata). Implementations of this interface will apply the specific filter logic. 7. **ResultPage:** Represents a page of query results, including the list of log entries and pagination information.

Design Splunk like log manager system supporting queries with filters on last min, hr, day, week etc.

Requirements

Think like an Architect

Premium Content