ArangoDB Server Options

In this exclusive emergency mode, all networking and HTTP interfaces of the server are disabled. No requests can be made to the server in this mode, and the only way to work with the server in this mode is by using the emergency console.

The server cannot be started in this mode if it is already running in this or another mode.

The default language is used for sorting and comparing strings. The language value is a two-letter language code (ISO-639) or it is composed by a two-letter language code followed by a two letter country code (ISO-3166). For example: de, en, en_US, en_UK.

The default is the system locale of the platform.

Turn on experimental vector index features. If this is enabled downgrading from this version will no longer be possible.

The name (identity) of the group to run the server as.

If you don’t specify this option, the server does not attempt to change its GID, so that the GID the server runs as is the primary group of the user who started the server.

If you specify this option, the server changes its GID after opening ports and reading configuration files, but before accepting connections or opening other files (such as recovery files).

With this option, you can get the sorting and comparing order exactly as it is defined in the ICU standard. The language value can be a two-letter language code (ISO-639), a two-letter language code followed by a two letter country code (ISO-3166), or any other valid ICU locale definition. For example: de, en, en_US, en_UK, de_AT@collation=phonebook.

For the Swedish language (sv), for instance, the correct ICU-based sorting order for letters is 'a','A','b','B','z','Z','å','Ä','ö','Ö'. To get this order, use --icu-language sv. If you use --default-language sv instead, the sorting order will be "A", "a", "B", "b", "Z", "z", "å", "Ä", "Ö", "ö".

Note: You can use only one of the language options, either --icu-language or --default-language. Setting both of them results in an error.

Runs an arangod process as supervisor with another arangod process as child, which acts as the server. In the event that the server unexpectedly terminates due to an internal error, the supervisor automatically restarts the server. Enabling this option implies that the server runs as a daemon.

The name (identity) of the user to run the server as.

If you don’t specify this option, the server does not attempt to change its UID, so that the UID used by the server is the same as the UID of the user who started the server.

If you specify this option, the server changes its UID after opening ports and reading configuration files, but before accepting connections or opening other files (such as recovery files). This is useful if the server must be started with raised privileges (in certain environments) but security considerations require that these privileges are dropped once the server has started work.

Note: You cannot use this option to bypass operating system security. In general, this option (and the related --gid) can lower privileges but not raise them.

While the syscall is generally available since Linux 2.6.x, it is also required that the underlying filesystem supports the splice operation. This is not true for some encrypted filesystems (e.g. ecryptfs), on which splice() calls can fail.

You can set the --use-splice-syscall startup option to false to use a less efficient, but more portable file copying method instead, which should work on all filesystems.

A value of 10 seconds is recommended for regular cluster deployments.

The option value must fall in the range [ 1..4 * NumberOfCores ]. Set it to 0 to automatically choose a sensible number based on the number of cores in the system.

The option value must fall in the range [ 1..arangosearch.commit-threads ]. Set it to 0 to automatically choose a sensible number based on the number of cores in the system.

The option value must fall in the range [ 1..arangosearch.consolidation-threads ]. Set it to 0 to automatically choose a sensible number based on the number of cores in the system.

If set to true, any data retrieval queries on out-of-sync links/indexes fail with the error ‘collection/view is out of sync’ (error code 1481).

If set to false, queries on out-of-sync links/indexes are answered normally, but the returned data may be incomplete.

From version 3.7.5 on, you should set the commit and consolidation thread counts separately via the following options instead:

• --arangosearch.commit-threads
• --arangosearch.consolidation-threads

If either --arangosearch.commit-threads or --arangosearch.consolidation-threads is set, then --arangosearch.threads and arangosearch.threads-limit are ignored. If only the legacy options are set, then the commit and consolidation thread counts are calculated as follows:

• Maximum: The smaller value out of --arangosearch.threads and arangosearch.threads-limit divided by 2, but at least 1.
• Minimum: the maximum divided by 2, but at least 1.

From version 3.7.5 on, you should set the commit and consolidation thread counts separately via the following options instead:

• --arangosearch.commit-threads
• --arangosearch.consolidation-threads

If either --arangosearch.commit-threads or --arangosearch.consolidation-threads is set, then --arangosearch.threads and arangosearch.threads-limit are ignored. If only the legacy options are set, then the commit and consolidation thread counts are calculated as follows:

• Maximum: The smaller value out of --arangosearch.threads and arangosearch.threads-limit divided by 2, but at least 1.
• Minimum: the maximum divided by 2, but at least 1.

Each thread that is involved in the async-registry needs to garbage collect its finished async function calls regularly. This option controls how often this is done in seconds. This can possibly be performance relevant because each involved thread aquires a lock.

You can use this option to limit the maximum line length for individual audit log messages that are written into audit logs by arangod.

Any audit log messages longer than the specified value are truncated and the suffix ... is appended to them.

The default value is 128 MB, which is very high and should effectively mean downwards-compatibility with previous arangod versions, which did not restrict the maximum size of audit log messages.

To write to a file, use file://<filename> with a relative or absolute file path.

To write to a syslog server, use syslog://<facility> or syslog://<facility>/<application-name>.

You can specify this option multiple times to configure the output for multiple targets.

Any occurrence of $PID inside a filename is replaced with the actual process ID at runtime. This allows you to log to process-specific files, e.g. --audit.output file:///var/log/arangod.log.$PID. Note that your terminal may require the dollar sign to be escaped.

You can control whether audit log messages are submitted to a queue and written to disk in batches or if they should be written to disk directly without being queued with this option.

Queueing audit log entries may be beneficial for latency, but can lead to unqueued messages being lost in case of a crash or power loss. Setting this option to false mimics the behavior of version 3.7 and before, where audit log messages were not queued but written in a blocking fashion.

You can control whether the log level is shown in the audit log messages with this option. If you omit it or set it to false, the log level is not shown in messages:

2016-10-03 15:47:26 | server1 | audit-authentication | n/a | database1 | 127.0.0.1:61528 | http basic | credentials wrong | /_api/version

If you set the option to true, the log level is included in the messages:

2016-10-03 15:47:26 | INFO | server1 | audit-authentication | n/a | database1 | 127.0.0.1:61528 | http basic | credentials wrong | /_api/version

This is to speed up the incremental upload process. Note that each instance needs approx. 7 to 10 MB of RAM in addition to arangod. The default should normally be OK for most situations.

This value controls the LZ4-internal acceleration factor for the LZ4 compression. Higher values typically yield less compression in exchange for faster compression and decompression speeds. An increase of 1 commonly leads to a compression speed increase of 3%, and could slightly increase decompression speed.

This value controls the cache’s effective memory usage limit. The user-defined memory limit (i.e. --cache.size) is multipled with this value to create the effective memory limit, from which on the cache will try to free up memory by evicting the oldest entries.

Cache tables with a fill ratio lower than this value will be shrunk by the cache rebalancer.

Cache tables with a fill ratio higher than this value will be inflated in size by the cache rebalancer.

By transparently compressing values in the in-memory edge index cache, more data can be held in memory than without compression.
Storing compressed values can increase CPU usage for the on-the-fly compression and decompression. In case compression is undesired, this option can be set to a very high value, which will effectively disable it. To use compression, set the option to a value that is lower than medium-to-large average payload sizes. It is normally not that useful to compress values that are smaller than 100 bytes.

The server uses a cache system which pools memory across many different cache tables. In order to provide intelligent internal memory management, the system periodically reclaims memory from caches which are used less often and reallocates it to caches which get more activity.

The global caching system, all caches, and all the data contained therein are constrained to this limit.

If there is less than 4 GiB of RAM in the system, default value is 256 MiB. If there is more, the default is (system RAM size - 2 GiB) * 0.25.

You can specify this option multiple times to let the server use a cluster of Agency servers.

Endpoints have the following pattern:

• tcp://ipv4-address:port - TCP/IP endpoint, using IPv4
• tcp://[ipv6-address]:port - TCP/IP endpoint, using IPv6
• ssl://ipv4-address:port - TCP/IP endpoint, using IPv4, SSL encryption
• ssl://[ipv6-address]:port - TCP/IP endpoint, using IPv6, SSL encryption

You must specify at least one endpoint or ArangoDB refuses to start. It is recommended to specify at least two endpoints, so that ArangoDB has an alternative endpoint if one of them becomes unavailable:

--cluster.agency-endpoint tcp://192.168.1.1:4001 --cluster.agency-endpoint tcp://192.168.1.2:4002 ...

The possible values for the option are:

• jwt-all: requires a valid JWT for all accesses to /_admin/cluster and its sub-routes. If you use this configuration, the Cluster and Nodes sections of the web interface are disabled, as they rely on the ability to read data from several cluster APIs.

• jwt-write: requires a valid JWT for write accesses (all HTTP methods except GET) to /_admin/cluster. You can use this setting to allow privileged users to read data from the cluster APIs, but not to do any modifications. Modifications (carried out by write accesses) are then only possible by requests with a valid JWT.

  All existing permission checks for the cluster API routes are still in effect with this setting, meaning that read operations without a valid JWT may still require dedicated other permissions (as in v3.7).

• jwt-compat: no additional access checks are in place for the cluster APIs. However, all existing permissions checks for the cluster API routes are still in effect with this setting, meaning that all operations may still require dedicated other permissions (as in v3.7).

The default value is jwt-compat, which means that this option does not cause any extra JWT checks compared to v3.7.

Setting this option to a value greater than zero makes Coordinators and DB-Servers run period connectivity checks with approximately the specified frequency. The first connectivity check is carried out approximately 15 seconds after server start. Note that a random delay is added to the interval on each server, so that different servers do not execute their connectivity checks all at the same time. Setting this option to a value of zero disables these connectivity checks."

If you don’t set this option, it defaults to the value of the --cluster.min-replication-factor option. If set, the value must be between the values of --cluster.min-replication-factor and --cluster.max-replication-factor.

Note that you can still adjust the replication factor per collection. This value is only the default value used for new collections if no replication factor is specified when creating a collection.

Warning: If you use multiple Coordinators, use the same value on all Coordinators.

The default behavior is to return an HTTP 403 Forbidden status code. You can set the option to 503 to return a 503 Service Unavailable.

If set to true, forces the cluster into creating all future collections with only a single shard and using the same DB-Server as these collections’ shards leader. All collections created this way are eligible for specific AQL query optimizations that can improve query performance and provide advanced transactional guarantees.

Warning: Use the same value on all Coordinators and all DBServers!

This option limits the maximum number of move shards operations that can be made when the Rebalance Shards button is clicked in the web interface. For backwards compatibility, the default value is 10. A value of 0 disables the button.

If you change the value of this setting and restart the servers, no changes are applied to existing collections that would violate the new setting.

Warning: If you use multiple Coordinators, use the same value on all Coordinators.

If you change the value of this setting and restart the servers, no changes are applied to existing collections that would violate the new setting.

Warning: If you use multiple Coordinators, use the same value on all Coordinators.

If you change the value of this setting and restart the servers, no changes are applied to existing collections that would violate the new setting.

Warning: If you use multiple Coordinators, use the same value on all Coordinators.

If specified, the endpoint needs to be in one of the following formats:

• tcp://ipv4-address:port - TCP/IP endpoint, using IPv4
• tcp://[ipv6-address]:port - TCP/IP endpoint, using IPv6
• ssl://ipv4-address:port - TCP/IP endpoint, using IPv4, SSL encryption
• ssl://[ipv6-address]:port - TCP/IP endpoint, using IPv6, SSL encryption

If you don’t specify an endpoint, the server looks up its internal endpoint address in the Agency. If no endpoint can be found in the Agency for the server’s ID, ArangoDB refuses to start.

Examples

Listen only on the interface with the address 192.168.1.1:

--cluster.my-address tcp://192.168.1.1:8530

Listen on all IPv4 and IPv6 addresses which are configured on port 8530:

--cluster.my-address ssl://[::]:8530

If specified, the endpoint needs to be in one of the following formats:

• tcp://ipv4-address:port - TCP/IP endpoint, using IPv4
• tcp://[ipv6-address]:port - TCP/IP endpoint, using IPv6
• ssl://ipv4-address:port - TCP/IP endpoint, using IPv4, SSL encryption
• ssl://[ipv6-address]:port - TCP/IP endpoint, using IPv6, SSL encryption

If you don’t specify an advertised endpoint, no external endpoint is advertised.

Examples

If an external interface is available to this server, you can specify it to communicate with external software / drivers:

--cluster.my-advertised-endpoint tcp://some.public.place:8530

All specifications of endpoints apply.

For a cluster, the possible values are DBSERVER (backend data server) and COORDINATOR (frontend server for external and application access).

Setting this option to a value greater than zero will let a coordinator which cannot send a heartbeat to the agency for the specified time shut down. This is necessary to prevent that a coordinator survives longer than the agency supervision has patience before it removes the coordinator from the agency meta data. Without this it would be possible that a coordinator is still running transactions and committing them, which could, for example, render hotbackups inconsistent.

Warning: If you use multiple DB-Servers, use the same value on all DB-Servers.

Warning: If you use multiple DB-Servers, use the same value on all DB-Servers.

Warning: This option should generally remain untouched and only be changed with great care!

Extend or shorten the timeouts for the internal synchronous replication mechanism between DB-Servers. All such timeouts are affected by this change.

Warning: If you use multiple DB-Servers, use the same value on all DB-Servers.

Warning: This option should generally remain untouched and only be changed with great care!

The minimum timeout in seconds for the internal synchronous replication mechanism between DB-Servers. If replication requests are slow, but the servers are otherwise healthy, timeouts can cause followers to be dropped unnecessarily, resulting in costly resync operations. Increasing this value may help avoid such resyncs. Conversely, decreasing it may cause more resyncs, while lowering the latency of individual write operations.

Warning: If you use multiple DB-Servers, use the same value on all DB-Servers.

Warning: If you use multiple DB-Servers, use the same value on all DB-Servers.

Warning: If you use multiple Coordinators, use the same value on all Coordinators.

This value is used as the default write concern for databases, which in turn is used as the default for collections.

Warning: If you use multiple Coordinators, use the same value on all Coordinators.

If you specify this option, then the server performs a database upgrade instead of starting normally.

A database upgrade first compares the version number stored in the VERSION file in the database directory with the current server version.

If the version number found in the database directory is higher than that of the server, the server considers this is an unintentional downgrade and warns about this. Using the server in these conditions is neither recommended nor supported.

If the version number found in the database directory is lower than that of the server, the server checks whether there are any upgrade tasks to perform. It then executes all required upgrade tasks and prints the status. If one of the upgrade tasks fails, the server exits with an error. Re-starting the server with the upgrade option again triggers the upgrade check and execution until the problem is fixed.

Whether or not you specify this option, the server always perform a version check on startup. If you running the server with a non-matching version number in the VERSION file, the server refuses to start.

If this option is specified together with --database.auto-upgrade, the server will perform a full RocksDB compaction after the database upgrade has completed successfully but before shutting down.

This performs a complete compaction of all column families with both changeLevel and compactBottomMostLevel options enabled, which can help optimize the database files after an upgrade.

The server will exit with an error code if the compaction fails.

This defines the location where all data of a server is stored.

Make sure the directory is writable by the arangod process. You should further not use a database directory which is provided by a network filesystem such as NFS. The reason is that networked filesystems might cause inconsistencies when there are multiple parallel readers or writers or they lack features required by arangod, e.g. flock().

If the maximum number of databases is reached, no additional databases can be created in the deployment. In order to create additional databases, other databases need to be removed first."

Each batch in a dump can grow to at most this size.

Each batch in a dump can grow to at most this size.

The approximate per-server maximum allowed memory usage value for all ongoing dump actions combined.

Each dump action on a server can use at most this many parallel threads. Note that end users can still start multiple dump actions that run in parallel.

The program must output 32 bytes of data on the standard output and exit.

You must secure the encryption key file so that only arangodump, arangorestore, and arangod can access it. You should also ensure that the file is not readable if someone steals your hardware, for example, by encrypting /mytmpfs or creating an in-memory file-system under /mytmpfs.

If set to false, access to any custom Foxx services in the deployment will be forbidden. Access to ArangoDB’s built-in web interface will still be possible though.

Note: when setting this option to false, the management API for Foxx services will automatically be disabled as well. This is the same as manually setting the startup option --foxx.api false.

If set to true, all Foxx services in all databases are synchronized between multiple Coordinators during the startup sequence. This ensures that all Foxx services are up-to-date when a Coordinator reports itself as ready.

In case the option is set to false (i.e. no waiting), the Coordinator completes the startup sequence faster, and the Foxx services are propagated lazily. Until the initialization procedure has completed for the local Foxx apps, any request to a Foxx app is responded to with an HTTP 500 error and a message waiting for initialization of Foxx services in this database. This can cause an unavailability window for Foxx services on Coordinator startup for the initial requests to Foxx apps until the app propagation has completed.

If you don’t use Foxx, you should set this option to false to benefit from a faster Coordinator startup. Deployments relying on Foxx apps being available as soon as a Coordinator is integrated or responding should set this option to true.

The option only has an effect for cluster setups. On single servers all Foxx apps are available from the very beginning.

Note: ArangoDB 3.8 changes the default value to false for this option. In previous versions, this option had a default value of true.

If set to true, the Foxx queues are available and jobs in the queues are executed asynchronously.

If set to false, the queue manager is disabled and any jobs are prevented from being processed, which may reduce CPU load a bit.

Lower values lead to more immediate and more frequent Foxx queue job execution, but make the queue thread wake up and query the queues more often. If set to a low value, the queue thread might cause CPU load.

If you don’t use Foxx queues much, then you may increase this value to make the queues thread wake up less.

Automatically compress outgoing HTTP responses with the deflate or gzip compression format, in case the client request advertises support for this. Compression will only happen for HTTP/1.1 and HTTP/2 connections, if the size of the uncompressed response body exceeds the threshold value controlled by this startup option, and if the response body size after compression is less than the original response body size. Using the value 0 disables the automatic response compression."

If the option is set to true, the server will automatically uncompress incoming HTTP requests with Content-Encodings gzip and deflate even if the request is not authenticated.

Idle keep-alive connections are closed by the server automatically when the timeout is reached. A keep-alive-timeout value of 0 disables the keep-alive feature entirely.

The value contained in this header indicates the current queueing/dequeuing time for requests in the scheduler (in seconds). Client applications and drivers can use this value to control the server load and also react on overload.

You can use this option to control whether user-defined JavaScript code is allowed to be executed on the server by sending HTTP requests to the /_admin/execute API endpoint with an authenticated user account.

The default value is false, which disables the execution of user-defined code. This is also the recommended setting for production. In test environments, it may be convenient to turn the option on in order to send arbitrary setup or teardown commands for execution on the server.

This option is intended to be useful for rolling upgrades. If you set it to true, you can upgrade the underlying ArangoDB packages without influencing the running arangod instance.

Setting this value does only make sense if you use ArangoDB outside of a container solution, like Docker or Kubernetes.

By default, the V8 engine is enabled on single servers and Coordinators. It is disabled by default on Agents and DB-Servers.

It is possible to turn the V8 engine off also on the latter instance types to reduce the footprint of ArangoDB. Turning the V8 engine off on single servers or Coordinators will automatically render certain functionality unavailable or dysfunctional. The affected functionality includes JavaScript transactions, Foxx, AQL user-defined functions, the built-in web interface and some server APIs.

This option is useful to have the garbage collection still work in periods with no or little numbers of requests.

More contexts allow executing more JavaScript actions in parallel, provided that there are also enough threads available. Note that each V8 context uses a substantial amount of memory and requires periodic CPU processing time for garbage collection.

This option configures the maximum number of V8 contexts that can be used in parallel. On server start, only as many V8 contexts are created as are configured by the --javascript.v8-contexts-minimum option. The actual number of available V8 contexts may vary between --javascript.v8-contexts-minimum and --javascript.v8-contexts at runtime. When there are unused V8 contexts that linger around, the server’s garbage collector thread automatically deletes them.

If both --javascript.v8-contexts-max-invocations and --javascript.v8-contexts-max-age are set, then the context is destroyed when either of the specified threshold values is reached.

The actual number of V8 contexts never drops below this value, but it may go up as high as specified by the --javascript.v8-contexts option.

When there are unused V8 contexts that linger around and the number of V8 contexts is greater than --javascript.v8-contexts-minimum, the server’s garbage collector thread automatically deletes them.

You can optionally pass arguments to the V8 JavaScript engine. The V8 engine runs with the default settings unless you explicitly specify them. The options are forwarded to the V8 engine, which parses them on its own. Passing invalid options may result in an error being printed on stderr and the option being ignored.

You need to pass the options as one string, with V8 option names being prefixed with two hyphens. Multiple options need to be separated by whitespace. To get a list of all available V8 options, you can use the value "--help" as follows:

--javascript.v8-options="--help"

Another example of specific V8 options being set at startup:

--javascript.v8-options="--log --no-logfile-per-isolate --logfile=v8.log"

Names and features or usable options depend on the version of V8 being used, and might change in the future if a different version of V8 is being used in ArangoDB. Not all options offered by V8 might be sensible to use in the context of ArangoDB. Use the specific options only if you are sure that they are not harmful for the regular database operation.

Credentials are not written to log files. Nevertheless, some logged data might be sensitive depending on the context of the deployment. For example, if request logging is switched on, user requests and corresponding data might end up in log files. Therefore, a certain care with log files is recommended.

Since the database server offers an API to control logging and query logging data, this API has to be secured properly. By default, the API is accessible for admin users (administrative access to the _system database). However, you can restrict it further to the superuser or disable it altogether:

• true: The /_admin/log API is accessible for admin users.
• jwt: The /_admin/log API is accessible for the superuser only (authentication with JWT superuser token and empty username).
• false: The /_admin/log API is not accessible at all.

This option applies to the control characters, that have hex codes below \x20, and also the character DEL with hex code \x7f.

If you set this option to false, control characters are retained when they have a visible representation, and replaced with a space character in case they do not have a visible representation. For example, the control character \n is visible, so a \n is displayed in the log. Contrary, the control character BEL is not visible, so a space is displayed instead.

If you set this option to true, the hex code for the character is displayed, for example, the BEL character is displayed as \x07.

The default value for this option is true to ensure compatibility with previous versions.

A side effect of turning off the escaping is that it reduces the CPU overhead for the logging. However, this is only noticeable if logging is set to a very verbose level (e.g. debug or trace).

If you set this option to false, Unicode characters are retained and written to the log as-is. For example, 犬 is logged as 犬.

If you set this options to true, any Unicode characters are escaped, and the hex codes for all Unicode characters are logged instead. For example, 犬 is logged as \u72AC.

The default value for this option is set to false for compatibility with previous versions.

A side effect of turning off the escaping is that it reduces the CPU overhead for the logging. However, this is only noticeable if logging is set to a very verbose level (e.g. debug or trace).

You can use this option to disable logging in an extra logging thread. If set to true, any log messages are immediately printed in the thread that triggered the log message. This is non-optimal for performance but can aid debugging. If set to false, log messages are handed off to an extra logging thread, which asynchronously writes the log messages.

You can specify a hostname to be logged at the beginning of each log message (for regular logging) or inside the hostname attribute (for JSON-based logging).

The default value is an empty string, meaning no hostnames is logged. If you set this option to auto, the hostname is automatically determined.

Each log invocation in the ArangoDB source code contains a unique log ID, which can be used to quickly find the location in the source code that produced a specific log message.

Log IDs are printed as 5-digit hexadecimal identifiers in square brackets between the log level and the log topic:

2020-06-22T21:16:48Z [39028] INFO [144fe] {general} using storage engine 'rocksdb' (where 144fe is the log ID).

You can use this option to toggle storing log messages in memory, from which they can be consumed via the /_admin/log HTTP API and via the web interface.

By default, this option is turned on, so log messages are consumable via the API and web interface. Turning this option off disables that functionality, saves a bit of memory for the in-memory log buffers, and prevents potential log information leakage via these means.

You can use this option to control which log messages are preserved in memory (in case --log.in-memory is enabled).

The default value is info, meaning all log messages of types info, warning, error, and fatal are stored in-memory by an instance. By setting this option to warning, only warning, error and fatal log messages are preserved in memory, and by setting the option to error, only error and fatal messages are kept.

This option is useful because the number of in-memory log messages is limited to the latest 2048 messages, and these slots are shared between informational, warning, and error messages by default.

ArangoDB’s log output is grouped by topics. --log.level can be specified multiple times at startup, for as many topics as needed. The log verbosity and output files can be adjusted per log topic.

arangod --log.level all=warning --log.level queries=trace --log.level startup=trace

This sets a global log level of warning and two topic-specific levels (trace for queries and info for startup). Note that --log.level warning does not set a log level globally for all existing topics, but only the general topic. Use the pseudo-topic all to set a global log level.

The same in a configuration file:

[log]
level = all=warning
level = queries=trace
level = startup=trace

The available log levels are:

• fatal: Only log fatal errors.
• error: Only log errors.
• warning: Only log warnings and errors.
• info: Log information messages, warnings, and errors.
• debug: Log debug and information messages, warnings, and errors.
• trace: Logs trace, debug, and information messages, warnings, and errors.

Note that the debug and trace levels are very verbose.

Some relevant log topics available in ArangoDB 3 are:

• agency: Information about the cluster Agency.
• performance: Performance-related messages.
• queries: Executed AQL queries, slow queries.
• replication: Replication-related information.
• requests: HTTP requests.
• startup: Information about server startup and shutdown.
• threads: Information about threads.

You can adjust the log levels at runtime via the PUT /_admin/log/level HTTP API endpoint.

Audit logging (Enterprise Edition): The server logs all audit events by default. Low priority events, such as statistics operations, are logged with the debug log level. To keep such events from cluttering the log, set the appropriate log topics to the info log level.

Note: This option does not include audit log messages. See --audit.max-entry-length instead.

Any log messages longer than the specified value are truncated and the suffix ... is added to them.

The purpose of this option is to shorten long log messages in case there is not a lot of space for log files, and to keep rogue log messages from overusing resources.

The default value is 128 MB, which is very high and should effectively mean downwards-compatibility with previous arangod versions, which did not restrict the maximum size of log messages.

Log entries are pushed on a queue for asynchronous writing unless you enable the --log.force-direct startup option. If you use a slow log output (e.g. syslog), the queue might grow and eventually overflow.

You can configure the upper bound of the queue with this option. If the queue is full, log entries are written synchronously until the queue has space again.

This option allows you to direct the global or per-topic log messages to different outputs. The output definition can be one of the following:

• - for stdout
• + for stderr
• syslog://<syslog-facility>
• syslog://<syslog-facility>/<application-name>
• file://<relative-or-absolute-path>

To set up a per-topic output configuration, use --log.output <topic>=<definition>:

--log.output queries=file://queries.log

The above example logs query-related messages to the file queries.log.

You can specify the option multiple times in order to configure the output for different log topics:

--log.level queries=trace --log.output queries=file:///queries.log --log.level requests=info --log.output requests=file:///requests.log

The above example logs all query-related messages to the file queries.log and HTTP requests with a level of info or higher to the file requests.log.

Any occurrence of $PID in the log output value is replaced at runtime with the actual process ID. This enables logging to process-specific files:

--log.output 'file://arangod.log.$PID'

Note that dollar sign may need extra escaping when specified on a command-line such as Bash.

If you specify --log.file-mode <octalvalue>, then any newly created log file uses octalvalue as file mode. Please note that the umask value is applied as well.

If you specify --log.file-group <name>, then any newly created log file tries to use <name> as the group name. Note that you have to be a member of that group. Otherwise, the group ownership is not changed.

The old --log.file option is still available for convenience. It is a shortcut for the more general option --log.output file://filename.

The old --log.requests-file option is still available. It is a shortcut for the more general option --log.output requests=file://....

To change the log levels for the specified output you can add a comma separated list of topics with their respective level after the output definition, separated by a semicolon: --log.output file:///path/to/file;queries=trace,requests=info --log.output -;all=error

Example: arangod ... --log.prefix "-->"

2020-07-23T09:46:03Z --> [17493] INFO ...

The /_admin/server/api-calls and /_admin/server/aql-queries endpoints provide access to recorded API calls and AQL queries respectively. They are referred to as the recording API.

Since this data might be sensitive depending on the context of the deployment, these endpoints need to be properly secured. By default, the recording API is accessible for admin users (users with administrative access to the _system database). However, you can restrict it further to the superuser or disable it altogether:

• true: The recording API is accessible for admin users.
• jwt: The recording API is accessible for the superuser only (authentication with JWT superuser token and empty username).
• false: The recording API is not accessible at all.

Whether API calls and AQL queries are recorded is independent of this option. It is controlled by the --server.api-call-recording and --server.aql-query-recording startup options.

If you set this option to true, log messages contains a single character with the server’s role. The roles are:

• U: Undefined / unclear (used at startup)
• S: Single server
• C: Coordinator
• P: Primary / DB-Server
• A: Agent

Some log messages can be displayed together with additional information in a structured form. The following parameters are available:

• database: The name of the database.
• username: The name of the user.
• queryid: The ID of the AQL query (on DB-Servers only).
• url: The endpoint path.

The format to enable or disable a parameter is <parameter>=<bool>, or <parameter> to enable it. You can specify the option multiple times to configure multiple parameters:

arangod --log.structured-param database=true --log.structured-param url --log.structured-param username=false

You can adjust the parameter settings at runtime using the /_admin/log/structured HTTP API.

Overview over the different options:

You can use this option to switch the log output to the JSON format. Each log message then produces a separate line with JSON-encoded log data, which can be consumed by other applications.

The object attributes produced for each log message are:

This option is deprecated. Use --log.time-format local-datestring instead.

This option is deprecated. Use --log.time-format timestamp-micros instead.

Automatically compress outgoing HTTP requests in cluster-internal traffic with the deflate, gzip or lz4 compression format. Compression will only happen if the size of the uncompressed request body exceeds the threshold value controlled by this startup option, and if the request body size after compression is less than the original request body size. Using the value 0 disables the automatic compression."

Setting this option to ’none’ will disable compression for cluster-internal requests. To enable compression for cluster-internal requests, set this option to either ‘deflate’, ‘gzip’, ’lz4’ or ‘auto’. The ‘deflate’ and ‘gzip’ compression methods are general purpose, but have significant CPU overhead for performing the compression work. The ’lz4’ compression method compresses slightly worse, but has a lot lower CPU overhead for performing the compression. The ‘auto’ compression method will use ‘deflate’ by default, and ’lz4’ for requests which have a size that is at least 3 times the configured threshold size. The compression method only matters if --network.compress-request-threshold is set to value greater than zero. If the threshold is set to value of 0, then no compression will be performed.

If set to true, using collection names in arbitrary places in AQL expressions is allowed, although using collection names like this is very likely unintended.

For example, consider the following query:

FOR doc IN collection RETURN collection

Here, the collection name is collection, and its usage in the FOR loop is intended and valid. However, collection is also used in the RETURN statement, which is legal but potentially unintended. It should likely be RETURN doc or RETURN doc.someAttribute instead. Otherwise, the entire collection is materialized and returned as many times as there are documents in the collection. This can take a long time and even lead to out-of-memory crashes in the worst case.

If you set the option to false, such unintentional usage of collection names in queries is prohibited, and instead makes the query fail with error 1568 (“collection used as expression operand”).

The default value of the option was true in v3.8, meaning that potentially unintended usage of collection names in queries were still allowed. In v3.9, the default value changes to false. The option is also deprecated from 3.9.0 on and will be removed in future versions. From then on, unintended usage of collection names will always be disallowed.

If a query is eligible for caching and the number of items in the database’s query cache is equal to this threshold value, another cached query result is removed from the cache.

This option only has an effect if the query cache mode is set to either on or demand.

When a query result is inserted into the query results cache, it is checked if the total size of cached results would exceed this value, and if so, another cached query result is removed from the cache before a new one is inserted.

This option only has an effect if the query cache mode is set to either on or demand.

Query results are only eligible for caching if their size does not exceed this setting’s value.

Not storing these results is normally beneficial if you use the query results cache, as queries on system collections are internal to ArangoDB and use space in the query results cache unnecessarily.

Toggles the AQL query results cache behavior. The possible values are:

• off: do not use query results cache
• on: always use query results cache, except for queries that have their cache attribute set to false
• demand: use query results cache only for queries that have their cache attribute set to true

If set to true, AQL queries that produce warnings are instantly aborted and throw an exception. This option can be set to catch obvious issues with AQL queries early.

If set to false, AQL queries that produce warnings are not aborted and return the warnings along with the query results.

You can override the option for each individual AQL query via the failOnWarning attribute.

You can use this option to set a limit on the combined estimated memory usage of all AQL queries (in bytes). If this option has a value of 0, then no global memory limit is in place. This is also the default value and the same behavior as in version 3.7 and older.

If you set this option to a value greater than zero, then the total memory usage of all AQL queries is limited approximately to the configured value. The limit is enforced by each server node in a cluster independently, i.e. it can be set separately for Coordinators, DB-Servers etc. The memory usage of a query that runs on multiple servers in parallel is not summed up, but tracked separately on each server.

If a memory allocation in a query would lead to the violation of the configured global memory limit, then the query is aborted with error code 32 (“resource limit exceeded”).

The global memory limit is approximate, in the same fashion as the per-query memory limit exposed by the option --query.memory-limit is. The global memory tracking has a granularity of 32 KiB chunks.

If both, --query.global-memory-limit and --query.memory-limit, are set, you must set the former at least as high as the latter.

If set to true, all failed AQL queries are logged to the server log. You can use this option during development, or to catch unexpected failed queries in production.

A warning is logged if queries exceed the specified threshold. This is useful for finding queries that use a large amount of memory.

This option allows you to truncate overly long query strings and bind parameter values to a reasonable length in log files.

Yon can use this option to limit the computation time and memory usage when converting complex AQL FILTER conditions into the internal DNF (disjunctive normal form) format. FILTER conditions with a lot of logical branches (AND, OR, NOT) can take a large amount of processing time and memory. This startup option limits the computation time and memory usage for such conditions.

Once the threshold value is reached during the DNF conversion of a FILTER condition, the conversion is aborted, and the query continues with a simplified internal representation of the condition, which cannot be used for index lookups.

Sets a default maximum runtime for AQL queries.

The default value is 0, meaning that the runtime of AQL queries is not limited. If you set it to any positive value, it restricts the runtime of all AQL queries, unless you override it with the maxRuntime query option on a per-query basis.

If a query exceeds the configured runtime, it is killed on the next occasion when the query checks its own status. Killing is best effort-based, so it is not guaranteed that a query will no longer than exactly the configured amount of time.

Warning: This option affects all queries in all databases, including queries issued for administration and database-internal purposes.

The default maximum amount of memory (in bytes) that a single AQL query can use. When a single AQL query reaches the specified limit value, the query is aborted with a resource limit exceeded exception. In a cluster, the memory accounting is done per server, so the limit value is effectively a memory limit per query per server node.

Some operations, namely calls to AQL functions and their intermediate results, are not properly tracked.

You can override the limit by setting the memoryLimit option for individual queries when running them. Overriding the per-query limit value is only possible if the --query.memory-limit-override option is set to true.

The default per-query memory limit value in version 3.8 and later depends on the amount of available RAM. In version 3.7 and older, the default value was 0, meaning “unlimited”.

The default values are:

Available memory:            0      (0MiB)  Limit:            0   unlimited, %mem:  n/a
Available memory:    134217728    (128MiB)  Limit:     33554432     (32MiB), %mem: 25.0
Available memory:    268435456    (256MiB)  Limit:     67108864     (64MiB), %mem: 25.0
Available memory:    536870912    (512MiB)  Limit:    201326592    (192MiB), %mem: 37.5
Available memory:    805306368    (768MiB)  Limit:    402653184    (384MiB), %mem: 50.0
Available memory:   1073741824   (1024MiB)  Limit:    603979776    (576MiB), %mem: 56.2
Available memory:   2147483648   (2048MiB)  Limit:   1288490189   (1228MiB), %mem: 60.0
Available memory:   4294967296   (4096MiB)  Limit:   2576980377   (2457MiB), %mem: 60.0
Available memory:   8589934592   (8192MiB)  Limit:   5153960755   (4915MiB), %mem: 60.0
Available memory:  17179869184  (16384MiB)  Limit:  10307921511   (9830MiB), %mem: 60.0
Available memory:  25769803776  (24576MiB)  Limit:  15461882265  (14745MiB), %mem: 60.0
Available memory:  34359738368  (32768MiB)  Limit:  20615843021  (19660MiB), %mem: 60.0
Available memory:  42949672960  (40960MiB)  Limit:  25769803776  (24576MiB), %mem: 60.0
Available memory:  68719476736  (65536MiB)  Limit:  41231686041  (39321MiB), %mem: 60.0
Available memory: 103079215104  (98304MiB)  Limit:  61847529063  (58982MiB), %mem: 60.0
Available memory: 137438953472 (131072MiB)  Limit:  82463372083  (78643MiB), %mem: 60.0
Available memory: 274877906944 (262144MiB)  Limit: 164926744167 (157286MiB), %mem: 60.0
Available memory: 549755813888 (524288MiB)  Limit: 329853488333 (314572MiB), %mem: 60.0

You can set a global memory limit for the total memory used by all AQL queries that currently execute via the --query.global-memory-limit option.

From ArangoDB 3.8 on, the per-query memory tracking has a granularity of 32 KB chunks. That means checking for memory limits such as “1” (e.g. for testing) may not make a query fail if the total memory allocations in the query don’t exceed 32 KiB. The effective lowest memory limit value that can be enforced is thus 32 KiB. Memory limit values higher than 32 KiB will be checked whenever the total memory allocations cross a 32 KiB boundary.

You can use this option to control whether individual AQL queries can increase their memory limit via the memoryLimit query option. This is the default, so a query that increases its memory limit is allowed to use more memory than set via the --query.memory-limit startup option value.

If the option is set to false, individual queries can only lower their maximum allowed memory usage but not increase it.

You can control how many different query execution plans the AQL query optimizer generates at most for any given AQL query with this option. Normally, the AQL query optimizer generates a single execution plan per AQL query, but there are some cases in which it creates multiple competing plans.

More plans can lead to better optimized queries. However, plan creation has its costs. The more plans are created and shipped through the optimization pipeline, the more time is spent in the optimizer. You can lower the number to make the optimizer stop creating additional plans when it has already created enough plans.

Note that this setting controls the default maximum number of plans to create. The value can still be adjusted on a per-query basis by setting the maxNumberOfPlans attribute for individual queries.

You can use this option to selectively enable or disable AQL query optimizer rules by default. You can specify the option multiple times.

For example, to turn off the rules use-indexes-for-sort and reduce-extraction-to-projection by default, use the following:

--query.optimizer-rules "-use-indexes-for-sort" --query.optimizer-rules "-reduce-extraction-to-projection"

The purpose of this startup option is to be able to enable potential future experimental optimizer rules, which may be shipped in a disabled-by-default state.

If set to true, AQL queries in single server mode also require WITH clauses in AQL queries where a cluster installation would require them.

The option is set to false by default, but you can turn it on in single servers to remove this behavior difference between single servers and clusters, making a later transition from single server to cluster easier.

You can control after what execution time streaming AQL queries are considered “slow” with this option. It exists to give streaming queries a separate, potentially higher timeout value than for regular queries. Streaming queries are often executed in lockstep with application data processing logic, which then also accounts for the queries’ runtime. It is thus expected that the lifetime of streaming queries is longer than for regular queries.

You can control after what execution time an AQL query is considered “slow” with this option. Any slow queries that exceed the specified execution time are logged when they are finished.

You can turn off the tracking of slow queries entirely by setting the option --query.tracking to false.

If set to true, then the bind variables are tracked and shown for all running and slow AQL queries. This also enables the display of bind variable values in the list of cached AQL query results. This option only has an effect if --query.tracking is set to true or if the query results cache is used.

You can disable tracking and displaying bind variable values by setting the option to false.

• 1: a pseudo-random number generator using an implication of the Mersenne Twister MT19937 algorithm
• 2: use a blocking random (or pseudo-random) number generator
• 3: use the non-blocking random (or pseudo-random) number generator supplied by the operating system
• 4: a combination of the blocking random number generator and the Mersenne Twister

You can add custom arguments to invocations of rclone, which is called for uploading and downloading Hot Backups. For example, you can enable debug logging to a separate file on startup as follows:

arangod --rclone.argument "--log-level=DEBUG" --rclone.argument "--log-file=rclone.log" ...

If you use the ArangoDB Starter, you can utilize the ARANGODB_SERVER_DIR environment variable that it sets to generate separate log files for every cluster node, like --all.rclone.argument="--log-file=@ARANGODB_SERVER_DIR@/rclone.log.

Preallocation is turned on by default, but you can turn it off for operating system versions that are known to have issues with it. This option only has an effect on operating systems that support fallocate.

Enabling this option may cause additional CPU and I/O load. You can limit how many index filling operations can execute concurrently with the --rocksdb.max-concurrent-index-fill-tasks startup option.

Set this to false to only (re-)fill in-memory index caches on leaders and save memory on followers. Note that the value of this option should be identical for all DBServers.

When documents are added, modified, or removed, these changes are tracked and a background thread tries to update the index caches accordingly if the feature is enabled, by adding new, updating existing, or deleting and refilling cache entries.

You can enable the feature for individual INSERT, UPDATE, REPLACE, and REMOVE operations in AQL queries, for individual document API requests that insert, update, replace, or remove single or multiple documents, as well as enable it by default using this startup option.

The background refilling is done on a best-effort basis and not guaranteed to succeed, for example, if there is no memory available for the cache subsystem, or during cache grow/shrink operations. A background thread is used so that foreground write operations are not slowed down by a lot. It may still cause additional I/O activity to look up data from the storage engine to repopulate the cache.

This option restricts how many cache entries the background thread for (re-)filling the in-memory index caches can queue at most. This limits the memory usage for the case of the background thread being slower than other operations that invalidate cache entries of edge indexes or cache-enabled persistent indexes.

This may waste some memory but may reduce the number of cross-page I/O operations.

The jemalloc-based memory allocator for the RocksDB block cache will also exclude the block cache contents from coredumps, potentially making generated coredumps a lot smaller. In order to use this option, the executable needs to be compiled with jemalloc support (which is the default on Linux).

The number of bits used to shard the block cache to allow concurrent operations. To keep individual shards at a reasonable size (i.e. at least 512 KiB), keep this value to at most block-cache-shard-bits / 512 KiB. Default: block-cache-size / 2^19.

This is the maximum size of the block cache in bytes. Increasing this value may improve performance. If there is more than 4 GiB of RAM in the system, the default value is (system RAM size - 2GiB) * 0.3.

For systems with less RAM, the default values are:

• 512 MiB for systems with between 2 and 4 GiB of RAM.
• 256 MiB for systems with between 1 and 2 GiB of RAM.
• 128 MiB for systems with less than 1 GiB of RAM.

If you set this option to true, RocksDB tracks all loaded index and filter blocks in the block cache, so that they count towards RocksDB’s block cache memory limit.

If you set this option to false, the memory usage for index and filter blocks is not accounted for.

The default value of --rocksdb.cache-index-and-filter-blocks was false in versions before 3.10, and was changed to true from version 3.10 onwards.

To improve stability of memory usage and avoid untracked memory allocations by RocksDB, it is recommended to set this option to true. Note that tracking index and filter blocks leaves less room for other data in the block cache, so in case servers have unused RAM capacity available, it may be useful to increase the overall size of the block cache.

If set to true, enables verbose logging of RocksDB’s actions into the logfile written by ArangoDB (if the --rocksdb.use-file-logging option is off), or RocksDB’s own log (if the --rocksdb.use-file-logging option is on).

This option is turned off by default, but you can enable it for debugging RocksDB internals and performance.

If set to true, the amount of data in each level of the LSM tree is determined dynamically to minimize the space amplification. Otherwise, the level sizes are fixed. The dynamic sizing allows RocksDB to maintain a well-structured LSM tree regardless of total data size.

Whether the maximum size of the RocksDB block cache is strictly enforced. You can set this option to limit the memory usage of the block cache to at most the specified size. If inserting a data block into the cache would exceed the cache’s capacity, the data block is not inserted. If disabled, a data block may still get inserted into the cache. It is evicted later, but the cache may temporarily grow beyond its capacity limit.

The default value for --rocksdb.enforce-block-cache-size-limit was false before version 3.10, but was changed to true from version 3.10 onwards.

To improve stability of memory usage and prevent exceeding the block cache capacity limit (as configurable via --rocksdb.block-cache-size), it is recommended to set this option to true.

This option allows you to make all writes to the RocksDB storage exclusive and therefore avoid write-write conflicts.

This option was introduced to open a way to upgrade from the legacy MMFiles to the RocksDB storage engine without modifying client application code. You should avoid enabling this option as the use of exclusive locks on collections introduce a noticeable throughput penalty.

Note: The MMFiles engine was removed and this option is a stopgap measure only. This option is thus deprecated, and will be removed in a future version.

Note that format version 6 can only be read by RocksDB versions

= 8.6.0. Thus switching to format version 6 will make the database files incompatible with ArangoDB versions with a lower RocksDB version in case of downgrading.

Compaction of level-0 to level-1 is triggered when this many files exist in level-0. If you set this option to a higher number, it may help bulk writes at the expense of slowing down reads.

When this many files accumulate in level-0, writes are slowed down to --rocksdb.delayed-write-rate to allow compaction to catch up.

When this many files accumulate in level-0, writes are stopped to allow compaction to catch up.

The jobs are submitted to the low priority thread pool. The default value is the number of processors in the system.

The lower this number, the lower the impact of the index cache filling, but the longer it takes to complete.

When reached, force a flush of all column families whose data is backed by the oldest WAL files. If you set this option to a low value, regular flushing of column family data from memtables is triggered, so that WAL files can be moved to the archive.

If you set this option to a high value, regular flushing is avoided but may prevent WAL files from being moved to the archive and being removed.

Transactions store all keys and values in RAM, so large transactions run the risk of causing out-of-memory situations. This setting allows you to ensure that it does not happen by limiting the size of any individual transaction. Transactions whose operations would consume more RAM than this threshold value are aborted automatically with error 32 (“resource limit exceeded”).

If this number is reached before the buffers can be flushed, writes are slowed or stalled.

The default value 0 restores the memory usage pattern of version 3.6. This makes RocksDB not keep any flushed immutable write-buffers in memory.

The recommended value is to set this equal to max-background-flushes. The default value is number of processors / 2.

The default value is number of processors / 2.

Levels above the default of 2 use compression to reduce the disk space requirements for storing data in these levels.

Enabling this option will make RocksDB’s compaction write the document data for different collections/shards into different .sst files. Otherwise the document data from different collections/shards can be mixed and written into the same .sst files.

Enabling this option usually has the benefit of making the RocksDB compaction more efficient when a lot of different collections/shards are written to in parallel. The disavantage of enabling this option is that there can be more .sst files than when the option is turned off, and the disk space used by these .sst files can be higher than if there are fewer .sst files (this is because there is some per-.sst file overhead). In particular on deployments with many collections/shards this can lead to a very high number of .sst files, with the potential of outgrowing the maximum number of file descriptors the ArangoDB process can open. Thus the option should only be enabled on deployments with a limited number of collections/shards.

Enabling this option will make RocksDB’s compaction write the edge index data for different edge collections/shards into different .sst files. Otherwise the edge index data from different edge collections/shards can be mixed and written into the same .sst files.

Enabling this option usually has the benefit of making the RocksDB compaction more efficient when a lot of different edge collections/shards are written to in parallel. The disavantage of enabling this option is that there can be more .sst files than when the option is turned off, and the disk space used by these .sst files can be higher than if there are fewer .sst files (this is because there is some per-.sst file overhead). In particular on deployments with many edge collections/shards this can lead to a very high number of .sst files, with the potential of outgrowing the maximum number of file descriptors the ArangoDB process can open. Thus the option should only be enabled on deployments with a limited number of edge collections/shards.

Enabling this option will make RocksDB’s compaction write the persistent index data for different mdi indexes (also indexes from different collections/shards) into different .sst files. Otherwise the persistent index data from different collections/shards/indexes can be mixed and written into the same .sst files.

Enabling this option usually has the benefit of making the RocksDB compaction more efficient when a lot of different collections/shards/indexes are written to in parallel. The disavantage of enabling this option is that there can be more .sst files than when the option is turned off, and the disk space used by these .sst files can be higher than if there are fewer .sst files (this is because there is some per-.sst file overhead). In particular on deployments with many collections/shards/indexes this can lead to a very high number of .sst files, with the potential of outgrowing the maximum number of file descriptors the ArangoDB process can open. Thus the option should only be enabled on deployments with a limited number of edge collections/shards/indexes.

Enabling this option will make RocksDB’s compaction write the persistent index data for different persistent indexes (also indexes from different collections/shards) into different .sst files. Otherwise the persistent index data from different collections/shards/indexes can be mixed and written into the same .sst files.

Enabling this option usually has the benefit of making the RocksDB compaction more efficient when a lot of different collections/shards/indexes are written to in parallel. The disavantage of enabling this option is that there can be more .sst files than when the option is turned off, and the disk space used by these .sst files can be higher than if there are fewer .sst files (this is because there is some per-.sst file overhead). In particular on deployments with many collections/shards/indexes this can lead to a very high number of .sst files, with the potential of outgrowing the maximum number of file descriptors the ArangoDB process can open. Thus the option should only be enabled on deployments with a limited number of edge collections/shards/indexes.

Enabling this option will make RocksDB’s compaction write the primary index data for different collections/shards into different .sst files. Otherwise the primary index data from different collections/shards can be mixed and written into the same .sst files.

Enabling this option usually has the benefit of making the RocksDB compaction more efficient when a lot of different collections/shards are written to in parallel. The disavantage of enabling this option is that there can be more .sst files than when the option is turned off, and the disk space used by these .sst files can be higher than if there are fewer .sst files (this is because there is some per-.sst file overhead). In particular on deployments with many collections/shards this can lead to a very high number of .sst files, with the potential of outgrowing the maximum number of file descriptors the ArangoDB process can open. Thus the option should only be enabled on deployments with a limited number of collections/shards.

Enabling this option makes RocksDB’s compaction write the index data for different vector indexes (also indexes from different collections/shards) into different .sst files. Otherwise, the index data from different collections/shards/indexes can be mixed and written into the same .sst files.

Enabling this option usually has the benefit of making the RocksDB compaction more efficient when a lot of different collections/shards/indexes are written to in parallel. The disadvantage of enabling this option is that there can be more .sst files than when the option is disabled, and the disk space used by these .sst files can be higher than if there are fewer .sst files because there is some overhead per .sst file. For deployments with many collections/shards/indexes in particular, this can lead to a very high number of .sst files, with the potential of outgrowing the maximum number of file descriptors the ArangoDB process can open. The option should thus only be enabled for deployments with a limited number of edge collections/shards/indexes.

The default value from RocksDB is ~30 days. To avoid periodic auto-compaction and the I/O caused by it, you can set this option to 0.

Automatic synchronization of data from RocksDB’s write-ahead logs to disk is only performed for not-yet synchronized data, and only for operations that have been executed without the waitForSync attribute.

If enabled, dynamically throttles the ingest rate of writes if necessary to reduce chances of compactions getting too far behind and blocking incoming writes.

If the throttling is enabled, it recalculates a new maximum ingestion rate with this frequency.

The actual write rate established by the throttling is the minimum of this value and the value that the regular throttle calculation produces, i.e. this option can be used to set a fixed upper bound on the write rate.

There is normally no need to change this value.

If throttling is enabled, this parameter controls the number of previous intervals to use for throttle value calculation.

There is normally no need to change this value.

The total amount of data to build up in all in-memory buffers (backed by log files). You can use this option together with the block cache size configuration option to limit memory usage.

If set to 0, the memory usage is not limited.

If set to a value larger than 0, this caps memory usage for write buffers but may have an effect on performance. If there is more than 4 GiB of RAM in the system, the default value is (system RAM size - 2 GiB) * 0.5.

For systems with less RAM, the default values are:

• 512 MiB for systems with between 1 and 4 GiB of RAM.
• 256 MiB for systems with less than 1 GiB of RAM.

You can control the number of lock stripes to use for RocksDB’s transaction lock manager with this option. You can use higher values to reduce a potential contention in the lock manager.

The option defaults to the number of available cores, but is increased to a value of 16 if the number of cores is lower.

If set to true, enables writing of RocksDB’s own informational log files into RocksDB’s database directory.

This option is turned off by default, but you can enable it for debugging RocksDB internals and performance.

If set to true, during startup, all .sst files in the engine-rocksdb folder in the database directory are checked for potential corruption and errors. The server process stops after the check and returns an exit code of 0 if the validation was successful, or a non-zero exit code if there is an error in any of the .sst files.

A value of 0 does not restrict the size of the archive, so the leader removes archived WAL files when there are no replication clients needing them. Any non-zero value restricts the size of the WAL files archive to about the specified value and trigger WAL archive file deletion once the threshold is reached. You can use this to get rid of archived WAL files in a disk size-constrained environment.

Note: The value is only a threshold, so the archive may get bigger than the configured value until the background thread actually deletes files from the archive. Also note that deletion from the archive only kicks in after --rocksdb.wal-file-timeout-initial seconds have elapsed after server start.

Archived WAL files are normally deleted automatically after a short while when there is no follower attached that may read from the archive. However, in case when there are followers attached that may read from the archive, WAL files normally remain in the archive until their contents have been streamed to the followers. In case there are slow followers that cannot catch up, this causes a growth of the WAL files archive over time.

You can use the option to force a deletion of WAL files from the archive even if there are followers attached that may want to read the archive. In case the option is set and a leader deletes files from the archive that followers want to read, this aborts the replication on the followers. Followers can restart the replication doing a resync, though, but they may not be able to catch up if WAL file deletion happens too early.

Thus it is best to leave this option at its default value of 0 except in cases when disk size is very constrained and no replication is used.

Data of ongoing transactions is stored in RAM. Transactions that get too big (in terms of number of operations involved or the total size of data created or modified by the transaction) are committed automatically. Effectively, this means that big user transactions are split into multiple smaller RocksDB transactions that are committed individually. The entire user transaction does not necessarily have ACID properties in this case.

If you decrease the value, the server starts the removal of obsolete WAL files earlier after server start. This is useful in testing environments that are space-restricted and do not require keeping much WAL file data at all.

The amount of data to build up in each in-memory buffer (backed by a log file) before closing the buffer and queuing it to be flushed to standard storage. Larger values than the default may improve performance, especially for bulk loads.

You can set this option to false to turn off authentication on the server-side, so that all clients can execute any action without authorization and privilege checks. You should only do this if you bind the server to localhost to not expose it to the public internet

If you set this option to true, then HTTP authentication is only required for requests going to URLs starting with /_, but not for other endpoints. You can thus use this option to expose custom APIs of Foxx microservices without HTTP authentication to the outside world, but prevent unauthorized access of ArangoDB APIs and the admin interface.

Note that checking the URL is performed after any database name prefix has been removed. That means, if the request URL is /_db/_system/myapp/myaction, the URL /myapp/myaction is checked for the /_ prefix.

Authentication still needs to be enabled for the server via --server.authentication in order for HTTP authentication to be forced for the ArangoDB APIs and the web interface. Only setting --server.authentication-system-only is not enough.

If you set this option to false, authentication for requests coming in via UNIX domain sockets is turned off on the server-side. Clients located on the same host as the ArangoDB server can use UNIX domain sockets to connect to the server without authentication. Requests coming in by other means (e.g. TCP/IP) are not affected by this option.

You can specify this option multiple times to let the ArangoDB server listen for incoming requests on multiple endpoints.

The endpoints are normally specified either in ArangoDB’s configuration file or on the command-line with --server.endpoint. ArangoDB supports different types of endpoints:

• tcp://ipv4-address:port - TCP/IP endpoint, using IPv4
• tcp://[ipv6-address]:port - TCP/IP endpoint, using IPv6
• ssl://ipv4-address:port - TCP/IP endpoint, using IPv4, SSL encryption
• ssl://[ipv6-address]:port - TCP/IP endpoint, using IPv6, SSL encryption
• unix:///path/to/socket - Unix domain socket endpoint

You can use http:// as an alias for tcp://, and https:// as an alias for ssl://.

If a TCP/IP endpoint is specified without a port number, then the default port (8529) is used.

If you use SSL-encrypted endpoints, you must also supply the path to a server certificate using the --ssl.keyfile option.

arangod --server.endpoint tcp://127.0.0.1:8529 \
        --server.endpoint ssl://127.0.0.1:8530 \
        --ssl.keyfile server.pem /tmp/data-dir

...
2022-11-07T10:39:30Z [1] INFO [6ea38] {general} using endpoint 'http+ssl://0.0.0.0:8530' for ssl-encrypted requests
2022-11-07T10:39:30Z [1] INFO [6ea38] {general} using endpoint 'http+tcp://0.0.0.0:8529' for non-encrypted requests
2022-11-07T10:39:31Z [1] INFO [cf3f4] {general} ArangoDB (version 3.10.0 [linux]) is ready for business. Have fun!

On one specific ethernet interface, each port can only be bound exactly once. You can look up your available interfaces using the ifconfig command on Linux. The general names of the interfaces differ between operating systems and the hardware they run on. However, every host has typically a so called loopback interface, which is a virtual interface. By convention, it always has the address 127.0.0.1 (IPv4) or ::1 (IPv6), and can only be reached from the very same host. Ethernet interfaces usually have names like eth0, wlan0, eth1:17, le0.

To find out which services already use ports (so ArangoDB can’t bind them anymore), you can use the netstat command. It behaves a little different on each platform; run it with -lnpt on Linux for valuable information.

ArangoDB can also do a so called broadcast bind using tcp://0.0.0.0:8529. This way, it is reachable on all interfaces of the host. This may be useful on development systems that frequently change their network setup, like laptops.

ArangoDB can also listen to IPv6 link-local addresses via adding the zone ID to the IPv6 address in the form [ipv6-link-local-address%zone-id]. However, what you probably want instead is to bind to a local IPv6 address. Local IPv6 addresses start with fd. If you only see a fe80: IPv6 address in your interface configuration but no IPv6 address starting with fd, your interface has no local IPv6 address assigned. You can read more about IPv6 link-local addresses here: https://en.wikipedia.org/wiki/Link-local_address#IPv6 .

To bind to a link-local and local IPv6 address, run ifconfig or equivalent command. The command lists all interfaces and assigned IP addresses. The link-local address may be fe80::6257:18ff:fe82:3ec6%eth0 (IPv6 address plus interface name). A local IPv6 address may be fd12:3456::789a. To bind ArangoDB to it, start arangod with --server.endpoint tcp://[fe80::6257:18ff:fe82:3ec6%eth0]:8529. You can use telnet to test the connection.

Using the whitespace characters in the output may be required to make the metrics output compatible with some processing tools, although Prometheus itself doesn’t need it.

Enabling this option exposes the following additional metrics via the GET /_admin/metrics/v2 endpoint:

• arangodb_document_writes_total
• arangodb_document_writes_replication_total
• arangodb_document_insert_time
• arangodb_document_read_time
• arangodb_document_update_time
• arangodb_document_replace_time
• arangodb_document_remove_time
• arangodb_collection_truncates_total
• arangodb_collection_truncates_replication_total
• arangodb_collection_truncate_time

This option can be used to make DB-Servers export detailed shard usage metrics.

• By default, this option is set to disabled so that no shard usage metrics are exported.

• Set the option to enabled-per-shard to make DB-Servers collect per-shard usage metrics whenever a shard is accessed.

• Set this option to enabled-per-shard-per-user to make DB-Servers collect usage metrics per shard and per user whenever a shard is accessed.

Note that enabling shard usage metrics can produce a lot of metrics if there are many shards and/or users in the system.

Files are sorted alphabetically, the first secret is used for signing + verifying JWT tokens (active secret), and all other secrets are only used to validate incoming JWT tokens (passive secrets). Only one secret needs to verify a JWT token for it to be accepted.

You can reload JWT secrets from disk without restarting the server or the nodes of a cluster deployment via the POST /_admin/server/jwt HTTP API endpoint. You can use this feature to roll out new JWT secrets throughout a cluster.

ArangoDB uses JSON Web Tokens to authenticate requests. Using this option lets you specify a JWT secret stored in a file. The secret must be at most 64 bytes long.

Warning: Avoid whitespace characters in the secret because they may get trimmed, leading to authentication problems:

• Character Tabulation (\t, U+0009)
• End of Line (\n, U+000A)
• Line Tabulation (\v, U+000B)
• Form Feed (\f, U+000C)
• Carriage Return (\r, U+000D)
• Space (U+0020)
• Next Line (U+0085)
• No-Nreak Space (U+00A0)

In single server setups, ArangoDB generates a secret if none is specified.

In cluster deployments which have authentication enabled, a secret must be set consistently across all cluster nodes so they can talk to each other.

ArangoDB also supports an --server.jwt-secret option to pass the secret directly (without a file). However, this is discouraged for security reasons.

You can reload JWT secrets from disk without restarting the server or the nodes of a cluster deployment via the POST /_admin/server/jwt HTTP API endpoint. You can use this feature to roll out new JWT secrets throughout a cluster.

You can specify the maximum size of the queue for asynchronous task execution. If the queue already contains this many tasks, new tasks are rejected until other tasks are popped from the queue. Setting this value may help preventing an instance from being overloaded or from running out of memory if the queue is filled up faster than the server can process requests.

This option determines the maximum number of request processing threads the server is allowed to start for request handling. If this number of threads is already running, arangod does not start further threads for request handling. The default value is max(32, 2 * available cores), so twice the number of CPU cores, but at least 32 threads.

The actual number of request processing threads is adjusted dynamically at runtime and is between --server.minimal-threads and --server.maximal-threads.

This option determines the minimum number of request processing threads the server starts and always keeps around.

There are some countermeasures built into Coordinators to prevent a cluster from being overwhelmed by too many concurrently executing requests.

If a request is executed on a Coordinator but needs to wait for some operation on a DB-Server, the operating system thread executing the request can often postpone execution on the Coordinator, put the request to one side and do something else in the meantime. When the response from the DB-Server arrives, another worker thread continues the work. This is a form of asynchronous implementation, which is great to achieve better thread utilization and enhance throughput.

On the other hand, this runs the risk that work is started on new requests faster than old ones can be finished off. Before version 3.8, this could overwhelm the cluster over time, and lead to out-of-memory situations and other unwanted side effects. For example, it could lead to excessive latency for individual requests.

There is a limit as to how many requests coming from the low priority queue (most client requests are of this type), can be executed concurrently. The default value for this is 4 times as many as there are scheduler threads (see --server.minimal-threads and --server.maximal-threads), which is good for most workloads. Requests in excess of this are not started but remain on the scheduler’s input queue (see --server.maximal-queue-size).

Very occasionally, 4 is already too much. You would notice this if the latency for individual requests is already too high because the system tries to execute too many of them at the same time (for example, if they fight for resources).

On the other hand, in rare cases it is possible that throughput can be improved by increasing the value, if latency is not a big issue and all requests essentially spend their time waiting, so that a high concurrency is acceptable. This increases memory usage, though.

The web interface uses JWT for authentication. However, the session are renewed automatically as long as you regularly interact with the web interface in your browser. You are not logged out while actively using it.

If you set this option to false, then ArangoDB’s statistics gathering is turned off. Statistics gathering causes regular background CPU activity, memory usage, and writes to the storage engine, so using this option to turn statistics off might relieve heavily-loaded instances a bit.

A side effect of setting this option to false is that no statistics are shown in the dashboard of ArangoDB’s web interface, and that the REST API for server statistics at /_admin/statistics returns HTTP 404.

If you set this option to false, then ArangoDB’s statistics gathering is turned off. Statistics gathering causes regular background CPU activity, memory usage, and writes to the storage engine, so using this option to turn statistics off might relieve heavily-loaded instances a bit.

If set to false, no statistics are shown in the dashboard of ArangoDB’s web interface, but the current statistics are available and can be queried using the REST API for server statistics at /_admin/statistics. This is less intrusive than setting the --server.statistics option to false.

ArangoDB’s storage engine is based on RocksDB, see http://rocksdb.org . It is the only available engine from ArangoDB v3.7 onwards.

The storage engine type needs to be the same for an entire deployment. Live switching of storage engines on already installed systems isn’t supported. Configuring the wrong engine (not matching the previously used one) results in the server refusing to start. You may use auto to let ArangoDB choose the previously used one.

This option limits requests from the arangosh to the telemetrics API, but not any other requests to the API.

Requests to the telemetrics API are counted for every 2 hour interval, and then reset. This means after a period of at most 2 hours, the telemetrics API becomes usable again.

The purpose of this option is to keep a deployment from being overwhelmed by too many telemetrics requests issued by arangosh instances that are used for batch processing.

You can use this option to set a high-watermark for the scheduler’s queue fill grade, from which onwards the server starts reporting unavailability via its availability API.

This option has a consequence for the /_admin/server/availability REST API only, which is often called by load-balancers and other availability probing systems.

The /_admin/server/availability REST API returns HTTP 200 if the fill grade of the scheduler’s queue is below the configured value, or HTTP 503 if the fill grade is equal to or above it. This can be used to flag a server as unavailable in case it is already highly loaded.

The default value for this option is 0.75 since version 3.8, i.e. 75%.

To prevent sending more traffic to an already overloaded server, it can be sensible to reduce the default value to even 0.5. This would mean that instances with a queue longer than 50% of their maximum queue capacity would return HTTP 503 instead of HTTP 200 when their availability API is probed.

You can use this option to specify a file with CA certificates that are sent to the client whenever the server requests a client certificate. If you specify a file, the server only accepts client requests with certificates issued by these CAs. Do not specify this option if you want clients to be able to connect without specific certificates.

The certificates in the file must be PEM-formatted.

You can use this option to restrict the server to certain SSL ciphers only, and to define the relative usage preference of SSL ciphers.

The format of the option’s value is documented in the OpenSSL documentation.

To check which ciphers are available on your platform, you may use the following shell command:

> openssl ciphers -v

ECDHE-RSA-AES256-SHA    SSLv3 Kx=ECDH     Au=RSA  Enc=AES(256)  Mac=SHA1
ECDHE-ECDSA-AES256-SHA  SSLv3 Kx=ECDH     Au=ECDSA Enc=AES(256)  Mac=SHA1
DHE-RSA-AES256-SHA      SSLv3 Kx=DH       Au=RSA  Enc=AES(256)  Mac=SHA1
DHE-DSS-AES256-SHA      SSLv3 Kx=DH       Au=DSS  Enc=AES(256)  Mac=SHA1
DHE-RSA-CAMELLIA256-SHA SSLv3 Kx=DH       Au=RSA  Enc=Camellia(256)
Mac=SHA1
...

If you use TLS/SSL encryption by binding the server to an ssl:// endpoint (e.g. --server.endpoint ssl://127.0.0.1:8529), you must use this option to specify the filename of the server’s private key. The file must be PEM-formatted and contain both, the certificate and the server’s private key.

You can generate a keyfile using OpenSSL as follows:

# create private key in file "server.key"
openssl genpkey -out server.key -algorithm RSA -pkeyopt rsa_keygen_bits:2048 -aes-128-cbc

# create certificate signing request (csr) in file "server.csr"
openssl req -new -key server.key -out server.csr

# copy away original private key to "server.key.org"
cp server.key server.key.org

# remove passphrase from the private key
openssl rsa -in server.key.org -out server.key

# sign the csr with the key, creates certificate PEM file "server.crt"
openssl x509 -req -days 365 -in server.csr -signkey server.key -out server.crt

# combine certificate and key into single PEM file "server.pem"
cat server.crt server.key > server.pem

You may use certificates issued by a Certificate Authority or self-signed certificates. Self-signed certificates can be created by a tool of your choice. When using OpenSSL for creating the self-signed certificate, the above commands should create a valid keyfile with a structure like this:

-----BEGIN CERTIFICATE-----
(base64 encoded certificate)
-----END CERTIFICATE-----
-----BEGIN RSA PRIVATE KEY-----
(base64 encoded private key)
-----END RSA PRIVATE KEY-----

For further information please check the manuals of the tools you use to create the certificate.

You can use this option to set various SSL-related options. Individual option values must be combined using bitwise OR.

Which options are available on your platform is determined by the OpenSSL version you use. The list of options available on your platform might be retrieved by the following shell command:

 > grep "#define SSL_OP_.*" /usr/include/openssl/ssl.h

 #define SSL_OP_MICROSOFT_SESS_ID_BUG                    0x00000001L
 #define SSL_OP_NETSCAPE_CHALLENGE_BUG                   0x00000002L
 #define SSL_OP_LEGACY_SERVER_CONNECT                    0x00000004L
 #define SSL_OP_NETSCAPE_REUSE_CIPHER_CHANGE_BUG         0x00000008L
 #define SSL_OP_SSLREF2_REUSE_CERT_TYPE_BUG              0x00000010L
 #define SSL_OP_MICROSOFT_BIG_SSLV3_BUFFER               0x00000020L
 ...

A description of the options can be found online in the OpenSSL documentation: http://www.openssl.org/docs/ssl/SSL_CTX_set_options.html )

Use this option to specify the default encryption protocol to be used. The default value is 9 (generic TLS), which allows the negotiation of the TLS version between the client and the server, dynamically choosing the highest mutually supported version of TLS.

Note that SSLv2 is unsupported as of version 3.4, because of the inherent security vulnerabilities in this protocol. Selecting SSLv2 as protocol aborts the startup.

Sometimes, it is desirable to have the same server use different server keys and certificates when it is contacted under different names. This is what the TLS “server name” extension is for. See https://en.wikipedia.org/wiki/Server_Name_Indication ) for details. for details). With this extension, the client can choose a server name, and the server can, using this information during the TLS handshake, use different server keys and certificate chains.

You can specify the option multiple times. Each value must be a string in the format SERVERNAME=KEYFILENAME. Replace SERVERNAME by a server name and KEYFILENAME by the file name of the key file to be used for that server name.

The format of the keyfile is identical to the one used for the --ssl.keyfile option. The keyfile used by default is the one in the --ssl.keyfile option, and only if there is an exact match between one server name given with --ssl.server-name-indication and the one in the handshake, the server switches to the alternative keyfile.

The maximum value is platform-dependent. If you specify a value higher than defined in the system header’s SOMAXCONN may result in a warning on server start. The actual value used by listen may also be silently truncated on some platforms (this happens inside the listen system call).

If you set this option to true, the socket option SO_REUSEADDR is set on all server endpoints, which is the default. If you set this option to false, it is possible that it takes up to a minute after a server has terminated until it is possible for a new server to use the same endpoint again.

Note: This can be a security risk because it might be possible for another process to bind to the same address and port, possibly hijacking network traffic.

Queries can store intermediate and final results temporarily on disk if a specified threshold is exceeded, to decrease the memory usage. Specify a path to a directory for the temporary data to activate the spillover feature. The directory must not be located underneath the instance’s database directory.

The threshold value to start spilling data onto disk is either a number of rows produced by a query or an amount of memory used in bytes, which you can set as query options (spillOverThresholdNumRows and spillOverThresholdMemoryUsage).

Note: This feature is experimental and is turned off by default. Also, the query results are still built up entirely in memory on Coordinators and single servers for non-streaming queries. To avoid the buildup of the entire query result in RAM, use a streaming query.

ArangoDB uses the path for storing temporary files, for extracting data from uploaded zip files (e.g. for Foxx services), and other things.

Ideally, the temporary path is set to an instance-specific subdirectory of the operating system’s temporary directory. To avoid data loss, the temporary path should not overlap with any directories that contain important data, for example, the instance’s database directory.

If you set the temporary path to the same directory as the instance’s database directory, a startup error is logged and the startup is aborted.

Stream Transactions automatically expire after this period when no further operations are posted into them. Posting an operation into a non-expired Stream Transaction resets the transaction’s timeout to the configured idle timeout.

The lower this value, the more frequently the TTL background thread kicks in and scans all available TTL indexes for expired documents, and the earlier the expired documents are actually removed.

You can configure this value separately from the total removal amount so that the per-collection time window for locking and potential write-write conflicts can be reduced.

In order to avoid “random” load spikes by the background thread suddenly kicking in and removing a lot of documents at once, you can cap the number of to-be-removed documents per thread invocation.

The TTL background thread goes back to sleep once it has removed the configured number of documents in one iteration. If more candidate documents are left for removal, they are removed in subsequent runs of the background thread.