Low Disk Space Handling

To work reliably, databases in general and SciDB in particular require a minimum amount of free disk space. Running out of disk space in the midst of a database update can result in data corruption. As a defensive measure, each SciDB instance monitors the free disk space available on the device where its data directory is mounted. If an instance detects that free space has gone below a configurable threshold, SciDB will log the problem and go into read-only mode. If this happens, the system administrator must take steps to increase the free disk space by removing arrays or array versions that are no longer needed, by migrating one or more instance data directories to larger storage devices, and/or by adjusting the low disk space threshold.

Configuring the low disk space threshold

You can configure the low disk space threshold using the low-disk-space-threshold-mb parameter in the SciDB config.ini file. The parameter value is an integer count of mebibytes (MiB). The default value is 1024, that is, one gibibyte (1 GiB). You must restart the SciDB cluster after modifying this configuration parameter to have it take effect.

Encountering low disk space

When SciDB allocates disk space for a WRITE query (that is, a store(), insert(), or similar operation), it also checks the amount of free disk space. If the free space is below the configured threshold, the WRITE query fails, and the error message names the physical device that is low on space:

AFL% store(build(AA, 400 + i), AA);
SystemException in file: src/util/DataStore.cpp function: _throwOnLowDiskSpace \
 line: 922 instance: s0-i0 (0)
Error id: scidb::SCIDB_SE_IO::SCIDB_LE_LOW_DISK_SPACE
Error description: I/O error. Under 1024 MiB (low-disk-space-threshold-mb) free on \
 device /sys/devices/pci0000:00/0000:00:01.3/0000:04:00.0/virtio2/block/vda/vda4, \
 WRITE queries are disabled.  SciDB is in read-only mode.  Ask your system \
 administrator to (a) free up or provision more disk space, and (b) re-enable \
 WRITE queries using lock_arrays(false).
Failed query id: 0.1692235992144189783
AFL%

Once the SCIDB_LE_LOW_DISK_SPACE error occurs, the database is in read-only mode and requires operator intervention. Subsequent WRITE queries fail with the following error:

AFL% append(build(AA, i * i), AA);
SystemException in file: src/system/catalog/SystemCatalog.cpp function: _lockArray \
 line: 4830 instance: s0-i0 (0)
Error id: scidb::SCIDB_SE_SYSCAT::SCIDB_LE_LOW_SPACE_READ_ONLY_MODE
Error description: System catalog error. The database is in read-only mode because \
 low-disk-space-threshold-mb was reached.  Ask your system administrator to \
 (a) free up or provision more disk space, and (b) re-enable WRITE queries \
 using lock_arrays(false).
Failed query id: 0.1692235992630531883
AFL%

The SCIDB_LE_LOW_SPACE_READ_ONLY_MODE error message does not mention the physical device that is low on space. This second query sees the database read-only lock and does not get far enough to attempt disk space allocation, so it cannot know which device is low on space. Search the scidb.log file for the SCIDB_LE_LOW_DISK_SPACE error to find the name of the problem device.

Re-enabling database WRITE queries

The initial SCIDB_LE_LOW_DISK_SPACE error gives the physical name of the device and the physical instance id of the SciDB instance that encountered the shortfall. A system administrator or a user with the needed permissions can free up space with the remove or remove_versions operators (which are still permitted although they are technically WRITE queries). Alternatively, a system administrator can migrate the instance data directory to another, larger device. To ascertain which data directory needs to be migrated, start with the physical instance id mentioned in the SCIDB_LE_LOW_DISK_SPACE error (s0-i0 in the example above) and map it to the data directory, as described in https://paradigm4.atlassian.net/wiki/spaces/scidb/pages/3395883455 .

Beware! A physical instance id of (say) s0-i4 does not necessarily correspond to data directory /scidb_data/mydb/0/4. Read the “Mapping Instance IDs” link above carefully!

Once more space has been made available, the system administrator can restore write access to the database by invoking the lock_arrays(false) operator. Normal database operation can proceed.