Management
Disk Capacity Planning Formula
This document provides the formula and guidelines for calculating disk storage requirements based on Events Per Second (EPS) for log collection systems.
Disk Capacity Calculator
Average log events per second
EPS
Safety buffer: 20%
Long-term storage duration
days
Total Disk Requirement
282.88 GB
with 20% risk margin
Data Stream Buffer
16.80 GB
Temporary message queuing
Analytics Database
46.08 GB
Fast analytics & reporting
Long-term Archive
219.00 GB
Historical log storage
Configuration Store
1.00 GB
System configuration
Risk Factor: 1.20 | Data Stream: min(EPS × 25.5 × RF, 14 × RF) | Analytics: EPS × 0.0064 × 60 × RF | Archive: EPS × 0.005 × Days × RF
Core Formula ∑
Risk_Factor = 1 + (Risk_Percentage / 100)
Data_Stream_Buffer = min(EPS × 25.5 × Risk_Factor, 14 × Risk_Factor)
Analytics_Database = EPS × 0.0064 × 60 × Risk_Factor
Long_term_Archive = EPS × 0.005 × Archive_Days × Risk_Factor
Configuration_Store = 1 GB (fixed)
Total_Disk_GB = Data_Stream_Buffer + Analytics_Database + Long_term_Archive + Configuration_Store
Component Details 🔩
Data Stream Buffer 🛢
- Purpose: Temporary message queuing and buffering
- TTL: 7 days (fixed)
- Capacity: 25.5 MB per EPS per day
- Maximum Size: 14 GB (due to 7-day TTL)
Analytics Database 🧠
- Purpose: Fast analytics and reporting on recent data
- TTL: 60 days (fixed)
- Capacity: 6.4 MB per EPS per day
- Maximum Retention: 2 months
Long-term Archive 📚
- Purpose: Long-term storage of historical logs
- TTL: Configurable (30-1095 days recommended)
- Capacity: 5.0 MB per EPS per day
Configuration Store ⚙️
- Purpose: System configuration and metadata indexing
- Size: 1 GB fixed (does not scale with EPS)
Input Parameters ↪
| Parameter | Default | Min | Max | Description |
|---|---|---|---|---|
| EPS | 100 | 1 | 10000 | Events per second |
| Risk Percentage | 20 | 0 | 100 | Safety margin for capacity planning |
| Archive Days | 365 | 30 | 1095 | Retention period for long-term archive |
Example Calculations 🧮
Example 1: 100 EPS with 20% risk margin
- Risk Factor = 1.2
- Data Stream Buffer = min(100 × 25.5 × 1.2, 14 × 1.2) = 16.8 GB
- Analytics Database = 100 × 0.0064 × 60 × 1.2 = 46.1 GB
- Long-term Archive = 100 × 0.005 × 365 × 1.2 = 219 GB
- Configuration Store = 1 GB
- Total = 283 GB
Example 2: 500 EPS with 50% risk margin
- Risk Factor = 1.5
- Data Stream Buffer = min(500 × 25.5 × 1.5, 14 × 1.5) = 21 GB (capped)
- Analytics Database = 500 × 0.0064 × 60 × 1.5 = 288 GB
- Long-term Archive = 500 × 0.005 × 365 × 1.5 = 1,369 GB
- Configuration Store = 1 GB
- Total = 1,679 GB
Important Notes❗️
- Compression: All values assume compression is enabled:
- Data Stream Buffer: zstd(3)
- Analytics Database: zstd(1)
- Long-term Archive: zstd(3)
- Peak vs Average: Formula uses average EPS. For systems with high peak-to-average ratios, consider using a higher risk percentage.
- Fixed Limits: Data Stream Buffer and Analytics Database have maximum retention periods that cap their growth regardless of EPS.
- Base Metrics: Based on observed data of 6.78M logs/24h = 78.5 EPS average with 641 peak EPS.
This calculator helps you plan disk capacity for:
- Data Stream Buffer: 7-day retention for message queuing
- Analytics Database: 60-day retention for fast queries
- Long-term Archive: Configurable retention for historical data
- Configuration Store: Fixed 1GB for system metadata