Disk space configuration

QoE Stor uses several folders for data storage:

  • /var/qoestor/backend/dump – stores temporary data received by receivers. After loading into the database, this data is deleted.
  • /var/lib/clickhouse/ – the so-called DEFAULT disk. Stores the database schema and data (raw and aggregated logs).
  • /var/lib/clickhouse-hot/ – the so-called HOT disk.
  • /var/lib/clickhouse-cold/ – the so-called COLD disk.

HOT and COLD disks are used to reduce storage costs.

The DEFAULT disk must be super fast (for example, NVMe SSD) to receive data from DPI, save it to the database as raw logs, and perform aggregation (creating aggregated logs).

The HOT disk can store aggregated logs for 1–2 weeks. It should also be an SSD to allow fast report generation.

The COLD disk can store older data. The COLD disk may be a SATA HDD, as reports on older data are rarely generated and can be stored on slower drives.

For information on how to physically separate directories by disks, see the section below on Storage setup on a separate disk.

Storage setup on a separate disk

By default, all data is stored in the /var directory.

Suppose we have connected a separate disk mounted at /storage.

  1. Work as the root user
    sudo su
  2. Stop receivers and the database
    fastor-stop
    fastor-db-stop
  3. Create directories under /storage
    mkdir /storage/qoestor
    mkdir /storage/qoestor/clickhouse
    mkdir /storage/qoestor/clickhouse-hot
    mkdir /storage/qoestor/clickhouse-cold
    mkdir /storage/qoestor/dump
  4. Copy data to the new disk
    cp -r /var/lib/clickhouse/* /storage/qoestor/clickhouse
    cp -r /var/lib/clickhouse-hot/* /storage/qoestor/clickhouse-hot
    cp -r /var/lib/clickhouse-cold/* /storage/qoestor/clickhouse-cold
    cp -r /var/qoestor/backend/dump/* /storage/qoestor/dump
  5. Change ownership of the /storage/qoestor directories
    chown -R clickhouse:clickhouse /storage/qoestor/clickhouse
    chown -R clickhouse:clickhouse /storage/qoestor/clickhouse-hot
    chown -R clickhouse:clickhouse /storage/qoestor/clickhouse-cold
  6. Remove old directories
    rm -rf /var/lib/clickhouse
    rm -rf /var/lib/clickhouse-hot
    rm -rf /var/lib/clickhouse-cold
    rm -rf /var/qoestor/backend/dump
  7. Create symbolic links
    ln -s /storage/qoestor/clickhouse /var/lib/clickhouse
    ln -s /storage/qoestor/clickhouse-hot /var/lib/clickhouse-hot
    ln -s /storage/qoestor/clickhouse-cold /var/lib/clickhouse-cold
    ln -s /storage/qoestor/dump /var/qoestor/backend/dump
  8. Verify links
    readlink -f /var/lib/clickhouse
    readlink -f /var/lib/clickhouse-hot
    readlink -f /var/lib/clickhouse-cold
    readlink -f /var/qoestor/backend/dump
  9. Start the database
    fastor-db-restart
  10. Start receivers
    fastor-restart

Was this information helpful?