There are a few concepts that are core to Elasticsearch. Understanding these concepts from the outset will tremendously help ease the learning process.

Near Realtime (NRT)

Elasticsearch is a near real time search platform. What this means is there is a slight latency (normally one second) from the time you index a document until the time it becomes searchable.

Cluster

A cluster is a collection of one or more nodes (servers) that together holds your entire data and provides federated indexing and search capabilities across all nodes. A cluster is identified by a unique name which by default is "elasticsearch". This name is important because a node can only be part of a cluster if the node is set up to join the cluster by its name.

Make sure that you don’t reuse the same cluster names in different environments, otherwise you might end up with nodes joining the wrong cluster. For instance you could use logging-dev , logging-stage , and logging-prod for the development, staging, and production clusters.

Note that it is valid and perfectly fine to have a cluster with only a single node in it. Furthermore, you may also have multiple independent clusters each with its own unique cluster name.

Node

A node is a single server that is part of your cluster, stores your data, and participates in the cluster’s indexing and search capabilities. Just like a cluster, a node is identified by a name which by default is a random Marvel character name that is assigned to the node at startup. You can define any node name you want if you do not want the default. This name is important for administration purposes where you want to identify which servers in your network correspond to which nodes in your Elasticsearch cluster.

A node can be configured to join a specific cluster by the cluster name. By default, each node is set up to join a cluster named elasticsearch which means that if you start up a number of nodes on your network and—assuming they can discover each other—they will all automatically form and join a single cluster named elasticsearch .

In a single cluster, you can have as many nodes as you want. Furthermore, if there are no other Elasticsearch nodes currently running on your network, starting a single node will by default form a new single-node cluster named elasticsearch .

Index

An index is a collection of documents that have somewhat similar characteristics. For example, you can have an index for customer data, another index for a product catalog, and yet another index for order data. An index is identified by a name (that must be all lowercase) and this name is used to refer to the index when performing indexing, search, update, and delete operations against the documents in it.

In a single cluster, you can define as many indexes as you want.

Type

Within an index, you can define one or more types. A type is a logical category/partition of your index whose semantics is completely up to you. In general, a type is defined for documents that have a set of common fields. For example, let’s assume you run a blogging platform and store all your data in a single index. In this index, you may define a type for user data, another type for blog data, and yet another type for comments data.

Document

A document is a basic unit of information that can be indexed. For example, you can have a document for a single customer, another document for a single product, and yet another for a single order. This document is expressed in JSON (JavaScript Object Notation) which is an ubiquitous internet data interchange format.

Within an index/type, you can store as many documents as you want. Note that although a document physically resides in an index, a document actually must be indexed/assigned to a type inside an index.

Shards & Replicas

An index can potentially store a large amount of data that can exceed the hardware limits of a single node. For example, a single index of a billion documents taking up 1TB of disk space may not fit on the disk of a single node or may be too slow to serve search requests from a single node alone.

To solve this problem, Elasticsearch provides the ability to subdivide your index into multiple pieces called shards. When you create an index, you can simply define the number of shards that you want. Each shard is in itself a fully-functional and independent "index" that can be hosted on any node in the cluster.

Sharding is important for two primary reasons:

The mechanics of how a shard is distributed and also how its documents are aggregated back into search requests are completely managed by Elasticsearch and is transparent to you as the user.

In a network/cloud environment where failures can be expected anytime, it is very useful and highly recommended to have a failover mechanism in case a shard/node somehow goes offline or disappears for whatever reason. To this end, Elasticsearch allows you to make one or more copies of your index’s shards into what are called replica shards, or replicas for short.

Replication is important for two primary reasons:

To summarize, each index can be split into multiple shards. An index can also be replicated zero (meaning no replicas) or more times. Once replicated, each index will have primary shards (the original shards that were replicated from) and replica shards (the copies of the primary shards). The number of shards and replicas can be defined per index at the time the index is created. After the index is created, you may change the number of replicas dynamically anytime but you cannot change the number shards after-the-fact.

By default, each index in Elasticsearch is allocated 5 primary shards and 1 replica which means that if you have at least two nodes in your cluster, your index will have 5 primary shards and another 5 replica shards (1 complete replica) for a total of 10 shards per index.

Each Elasticsearch shard is a Lucene index. There is a maximum number of documents you can have in a single Lucene index. As of LUCENE-5843 , the limit is 2,147,483,519 (= Integer.MAX_VALUE - 128) documents. You can monitor shard sizes using the _cat/shards api.

With that out of the way, let’s get started with the fun part…

Elasticsearch requires at least Java 7. Specifically as of this writing, it is recommended that you use the Oracle JDK version 1.8.0_25. Java installation varies from platform to platform so we won’t go into those details here. Oracle’s recommended installation documentation can be found on Oracle’s website . Suffice to say, before you install Elasticsearch, please check your Java version first by running (and then install/upgrade accordingly if needed):

java -version
echo $JAVA_HOME

Once we have Java set up, we can then download and run Elasticsearch. The binaries are available from www.elastic.co/downloads along with all the releases that have been made in the past. For each release, you have a choice among a zip or tar archive, or a DEB or RPM package. For simplicity, let’s use the tar file.

Let’s download the Elasticsearch 1.7.6 tar as follows (Windows users should download the zip package):

curl -L -O https://download.elastic.co/elasticsearch/elasticsearch/elasticsearch-1.7.6.tar.gz

Then extract it as follows (Windows users should unzip the zip package):

tar -xvf elasticsearch-1.7.6.tar.gz

It will then create a bunch of files and folders in your current directory. We then go into the bin directory as follows:

cd elasticsearch-1.7.6/bin

And now we are ready to start our node and single cluster (Windows users should run the elasticsearch.bat file):

./elasticsearch

If everything goes well, you should see a bunch of messages that look like below:

./elasticsearch
[2014-03-13 13:42:17,218][INFO ][node           ] [New Goblin] version[1.7.6], pid[2085], build[5c03844/2014-02-25T15:52:53Z]
[2014-03-13 13:42:17,219][INFO ][node           ] [New Goblin] initializing ...
[2014-03-13 13:42:17,223][INFO ][plugins        ] [New Goblin] loaded [], sites []
[2014-03-13 13:42:19,831][INFO ][node           ] [New Goblin] initialized
[2014-03-13 13:42:19,832][INFO ][node           ] [New Goblin] starting ...
[2014-03-13 13:42:19,958][INFO ][transport      ] [New Goblin] bound_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish_address {inet[/192.168.8.112:9300]}
[2014-03-13 13:42:23,030][INFO ][cluster.service] [New Goblin] new_master [New Goblin][rWMtGj3dQouz2r6ZFL9v4g][mwubuntu1][inet[/192.168.8.112:9300]], reason: zen-disco-join (elected_as_master)
[2014-03-13 13:42:23,100][INFO ][discovery      ] [New Goblin] elasticsearch/rWMtGj3dQouz2r6ZFL9v4g
[2014-03-13 13:42:23,125][INFO ][http           ] [New Goblin] bound_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish_address {inet[/192.168.8.112:9200]}
[2014-03-13 13:42:23,629][INFO ][gateway        ] [New Goblin] recovered [1] indices into cluster_state
[2014-03-13 13:42:23,630][INFO ][node           ] [New Goblin] started

Without going too much into detail, we can see that our node named "New Goblin" (which will be a different Marvel character in your case) has started and elected itself as a master in a single cluster. Don’t worry yet at the moment what master means. The main thing that is important here is that we have started one node within one cluster.

As mentioned previously, we can override either the cluster or node name. This can be done from the command line when starting Elasticsearch as follows:

./elasticsearch --cluster.name my_cluster_name --node.name my_node_name

Also note the line marked http with information about the HTTP address ( 192.168.8.112 ) and port ( 9200 ) that our node is reachable from. By default, Elasticsearch uses port 9200 to provide access to its REST API. This port is configurable if necessary.

The REST API

Now that we have our node (and cluster) up and running, the next step is to understand how to communicate with it. Fortunately, Elasticsearch provides a very comprehensive and powerful REST API that you can use to interact with your cluster. Among the few things that can be done with the API are as follows:

curl 'localhost:9200/_cat/health?v'

epoch      timestamp cluster       status node.total node.data shards pri relo init unassign
1394735289 14:28:09  elasticsearch green           1         1      0   0    0    0        0

curl 'localhost:9200/_cat/nodes?v'

curl 'localhost:9200/_cat/nodes?v'
host         ip        heap.percent ram.percent load node.role master name
mwubuntu1    127.0.1.1            8           4 0.00 d         *      New Goblin

curl 'localhost:9200/_cat/indices?v'

curl 'localhost:9200/_cat/indices?v'
health index pri rep docs.count docs.deleted store.size pri.store.size

Elasticsearch provides data manipulation and search capabilities in near real time. By default, you can expect a one second delay (refresh interval) from the time you index/update/delete your data until the time that it appears in your search results. This is an important distinction from other platforms like SQL wherein data is immediately available after a transaction is completed.

Indexing/Replacing Documents

We’ve previously seen how we can index a single document. Let’s recall that command again:

curl -XPUT 'localhost:9200/customer/external/1?pretty' -d '
  "name": "John Doe"
}'

Again, the above will index the specified document into the customer index, external type, with the ID of 1. If we then executed the above command again with a different (or same) document, Elasticsearch will replace (i.e. reindex) a new document on top of the existing one with the ID of 1:

curl -XPUT 'localhost:9200/customer/external/1?pretty' -d '
  "name": "Jane Doe"
}'

The above changes the name of the document with the ID of 1 from "John Doe" to "Jane Doe". If, on the other hand, we use a different ID, a new document will be indexed and the existing document(s) already in the index remains untouched.

curl -XPUT 'localhost:9200/customer/external/2?pretty' -d '
  "name": "Jane Doe"
}'

The above indexes a new document with an ID of 2.

When indexing, the ID part is optional. If not specified, Elasticsearch will generate a random ID and then use it to index the document. The actual ID Elasticsearch generates (or whatever we specified explicitly in the previous examples) is returned as part of the index API call.

This example shows how to index a document without an explicit ID:

curl -XPOST 'localhost:9200/customer/external?pretty' -d '
  "name": "Jane Doe"
}'

Note that in the above case, we are using the POST verb instead of PUT since we didn’t specify an ID.

Sample Dataset

Now that we’ve gotten a glimpse of the basics, let’s try to work on a more realistic dataset. I’ve prepared a sample of fictitious JSON documents of customer bank account information. Each document has the following schema:

{
    "account_number": 0,
    "balance": 16623,
    "firstname": "Bradshaw",
    "lastname": "Mckenzie",
    "age": 29,
    "gender": "F",
    "address": "244 Columbus Place",
    "employer": "Euron",
    "email": "[email protected]",
    "city": "Hobucken",
    "state": "CO"
}

For the curious, I generated this data from www.json-generator.com/ so please ignore the actual values and semantics of the data as these are all randomly generated.

Loading the Sample Dataset

You can download the sample dataset (accounts.json) from here . Extract it to our current directory and let’s load it into our cluster as follows:

curl -XPOST 'localhost:9200/bank/account/_bulk?pretty' --data-binary "@accounts.json"
curl 'localhost:9200/_cat/indices?v'

And the response:

curl 'localhost:9200/_cat/indices?v'
health index pri rep docs.count docs.deleted store.size pri.store.size
yellow bank    5   1       1000            0    424.4kb        424.4kb

Which means that we just successfully bulk indexed 1000 documents into the bank index (under the account type).

Environment Variables

Within the scripts, Elasticsearch comes with built in JAVA_OPTS passed to the JVM started. The most important setting for that is the -Xmx to control the maximum allowed memory for the process, and -Xms to control the minimum allocated memory for the process ( in general, the more memory allocated to the process, the better ).

Most times it is better to leave the default JAVA_OPTS as they are, and use the ES_JAVA_OPTS environment variable in order to set / change JVM settings or arguments.

The ES_HEAP_SIZE environment variable allows to set the heap memory that will be allocated to elasticsearch java process. It will allocate the same value to both min and max values, though those can be set explicitly (not recommended) by setting ES_MIN_MEM (defaults to 256m ), and ES_MAX_MEM (defaults to 1g ).

It is recommended to set the min and max memory to the same value, and enable mlockall .

System Configuration

File Descriptors

Make sure to increase the number of open files descriptors on the machine (or for the user running elasticsearch). Setting it to 32k or even 64k is recommended.

In order to test how many open files the process can open, start it with -Des.max-open-files set to true . This will print the number of open files the process can open on startup.

Alternatively, you can retrieve the max_file_descriptors for each node using the Nodes Info API, with:

curl localhost:9200/_nodes/process?pretty

Virtual memory

Elasticsearch uses a hybrid mmapfs / niofs directory by default to store its indices. The default operating system limits on mmap counts is likely to be too low, which may result in out of memory exceptions. On Linux, you can increase the limits by running the following command as root :

sysctl -w vm.max_map_count=262144

To set this value permanently, update the vm.max_map_count setting in /etc/sysctl.conf .

If you installed Elasticsearch using a package (.deb, .rpm) this setting will be changed automatically. To verify, run sysctl vm.max_map_count .

Memory Settings

Most operating systems try to use as much memory as possible for file system caches and eagerly swap out unused application memory, possibly resulting in the elasticsearch process being swapped. Swapping is very bad for performance and for node stability, so it should be avoided at all costs.

There are three options:

Elasticsearch Settings

elasticsearch configuration files can be found under ES_HOME/config folder. The folder comes with two files, the elasticsearch.yml for configuring Elasticsearch different modules , and logging.yml for configuring the Elasticsearch logging.

The configuration format is YAML . Here is an example of changing the address all network based modules will use to bind and publish to:

network :
    host : 10.0.0.4

Paths

In production use, you will almost certainly want to change paths for data and log files:

path:
  logs: /var/log/elasticsearch
  data: /var/data/elasticsearch

Cluster name

Also, don’t forget to give your production cluster a name, which is used to discover and auto-join other nodes:

cluster:
  name: <NAME OF YOUR CLUSTER>

Make sure that you don’t reuse the same cluster names in different environments, otherwise you might end up with nodes joining the wrong cluster. For instance you could use logging-dev , logging-stage , and logging-prod for the development, staging, and production clusters.

Node name

You may also want to change the default node name for each node to something like the display hostname. By default Elasticsearch will randomly pick a Marvel character name from a list of around 3000 names when your node starts up.

node:
  name: <NAME OF YOUR NODE>

The hostname of the machine is provided in the environment variable HOSTNAME . If on your machine you only run a single elasticsearch node for that cluster, you can set the node name to the hostname using the ${...} notation:

node:
  name: ${HOSTNAME}

Internally, all settings are collapsed into "namespaced" settings. For example, the above gets collapsed into node.name . This means that its easy to support other configuration formats, for example, JSON . If JSON is a preferred configuration format, simply rename the elasticsearch.yml file to elasticsearch.json and add:

Configuration styles

{
    "network" : {
        "host" : "10.0.0.4"
}

It also means that its easy to provide the settings externally either using the ES_JAVA_OPTS or as parameters to the elasticsearch command, for example:

$ elasticsearch -Des.network.host=10.0.0.4

Another option is to set es.default. prefix instead of es. prefix, which means the default setting will be used only if not explicitly set in the configuration file.

Another option is to use the ${...} notation within the configuration file which will resolve to an environment setting, for example:

{
    "network" : {
        "host" : "${ES_NET_HOST}"
}

Additionally, for settings that you do not wish to store in the configuration file, you can use the value ${prompt.text} or ${prompt.secret} and start Elasticsearch in the foreground. ${prompt.secret} has echoing disabled so that the value entered will not be shown in your terminal; ${prompt.text} will allow you to see the value as you type it in. For example:

node:
  name: ${prompt.text}

On execution of the elasticsearch command, you will be prompted to enter the actual value like so:

Enter value for [node.name]:

Elasticsearch will not start if ${prompt.text} or ${prompt.secret} is used in the settings and the process is run as a service or in the background.

The location of the configuration file can be set externally using a system property:

$ elasticsearch -Des.config=/path/to/config/file

Index Settings

Indices created within the cluster can provide their own settings. For example, the following creates an index with memory based storage instead of the default file system based one (the format can be either YAML or JSON):

$ curl -XPUT http://localhost:9200/kimchy/ -d \
index :
    store:
        type: memory
'

Index level settings can be set on the node level as well, for example, within the elasticsearch.yml file, the following can be set:

index :
    store:
        type: memory

This means that every index that gets created on the specific node started with the mentioned configuration will store the index in memory unless the index explicitly sets it . In other words, any index level settings override what is set in the node configuration. Of course, the above can also be set as a "collapsed" setting, for example:

$ elasticsearch -Des.index.store.type=memory

All of the index level configuration can be found within each index module .

Logging

Elasticsearch uses an internal logging abstraction and comes, out of the box, with log4j . It tries to simplify log4j configuration by using YAML to configure it, and the logging configuration file is config/logging.yml . The JSON and properties formats are also supported. Multiple configuration files can be loaded, in which case they will get merged, as long as they start with the logging. prefix and end with one of the supported suffixes (either .yml , .yaml , .json or .properties ) The logger section contains the java packages and their corresponding log level, where it is possible to omit the org.elasticsearch prefix. The appender section contains the destinations for the logs. Extensive information on how to customize logging and all the supported appenders can be found on the log4j documentation .

Additional Appenders and other logging classes provided by log4j-extras are also available, out of the box.

In order to run elasticsearch as a service on your operating system, the provided packages try to make it as easy as possible for you to start and stop elasticsearch during reboot and upgrades.

Linux

Currently our build automatically creates a debian package and an RPM package, which is available on the download page. The package itself does not have any dependencies, but you have to make sure that you installed a JDK.

Each package features a configuration file, which allows you to set the following parameters

Debian/Ubuntu

The debian package ships with everything you need as it uses standard debian tools like update update-rc.d to define the runlevels it runs on. The init script is placed at /etc/init.d/elasticsearch as you would expect it. The configuration file is placed at /etc/default/elasticsearch .

The debian package does not start up the service by default. The reason for this is to prevent the instance to accidentally join a cluster, without being configured appropriately. After installing using dpkg -i you can use the following commands to ensure, that elasticsearch starts when the system is booted and then start up elasticsearch:

sudo update-rc.d elasticsearch defaults 95 10
sudo /etc/init.d/elasticsearch start

Installing the oracle JDK

The usual recommendation is to run the Oracle JDK with elasticsearch. However Ubuntu and Debian only ship the OpenJDK due to license issues. You can easily install the oracle installer package though. In case you are missing the add-apt-repository command under Debian GNU/Linux, make sure have at least Debian Wheezy and the package python-software-properties installed

sudo add-apt-repository ppa:webupd8team/java
sudo apt-get update
sudo apt-get install oracle-java7-installer
java -version

The last command should verify a successful installation of the Oracle JDK. If you want to install java8, you can call apt-get install oracle-java8-installer

RPM based distributions

Using chkconfig

Some RPM based distributions are using chkconfig to enable and disable services. The init script is located at /etc/init.d/elasticsearch , where as the configuration file is placed at /etc/sysconfig/elasticsearch . Like the debian package the RPM package is not started by default after installation, you have to do this manually by entering the following commands

sudo /sbin/chkconfig --add elasticsearch
sudo service elasticsearch start

Using systemd

Distributions like SUSE do not use the chkconfig tool to register services, but rather systemd and its command /bin/systemctl to start and stop services (at least in newer versions, otherwise use the chkconfig commands above). The configuration file is also placed at /etc/sysconfig/elasticsearch . After installing the RPM, you have to change the systemd configuration and then start up elasticsearch

sudo /bin/systemctl daemon-reload
sudo /bin/systemctl enable elasticsearch.service
sudo /bin/systemctl start elasticsearch.service

Also note that changing the MAX_MAP_COUNT setting in /etc/sysconfig/elasticsearch does not have any effect, you will have to change it in /usr/lib/sysctl.d/elasticsearch.conf in order to have it applied at startup.

Windows users can configure Elasticsearch to run as a service to run in the background or start automatically at startup without any user interaction. This can be achieved through service.bat script under bin/ folder which allows one to install, remove, manage or configure the service and potentially start and stop the service, all from the command-line.

c:\elasticsearch-1.7.6\bin>service
Usage: service.bat install|remove|start|stop|manager [SERVICE_ID]

The script requires one parameter (the command to execute) followed by an optional one indicating the service id (useful when installing multiple Elasticsearch services).

The commands available are:

Note that the environment configuration options available during the installation are copied and will be used during the service lifecycle. This means any changes made to them after the installation will not be picked up unless the service is reinstalled.

Based on the architecture of the available JDK/JRE (set through JAVA_HOME ), the appropriate 64-bit(x64) or 32-bit(x86) service will be installed. This information is made available during install:

c:\elasticsearch-{version}bin>service install
Installing service      :  "elasticsearch-service-x64"
Using JAVA_HOME (64-bit):  "c:\jvm\jdk1.7"
The service 'elasticsearch-service-x64' has been installed.

While a JRE can be used for the Elasticsearch service, due to its use of a client VM (as oppose to a server JVM which offers better performance for long-running applications) its usage is discouraged and a warning will be issued.

Customizing service settings

There are two ways to customize the service settings:

There is also a community supported customizable MSI installer available: https://github.com/salyh/elasticsearch-msi-installer (by Hendrik Saly).

The directory layout of an installation is as follows:

The multiple data locations allows to stripe it. The striping is simple, placing whole files in one of the locations, and deciding where to place the file based on the value of the index.store.distributor setting:

Note, there are no multiple copies of the same data, in that, its similar to RAID 0. Though simple, it should provide a good solution for people that don’t want to mess with RAID. Here is how it is configured:

path.data: /mnt/first,/mnt/second

Or the in an array format:

path.data: ["/mnt/first", "/mnt/second"]

Default Paths

Below are the default paths that elasticsearch will use, if not explictly changed.

deb and rpm

zip and tar.gz

We also have repositories available for APT and YUM based distributions. Note that we only provide binary packages, but no source packages, as the packages are created as part of the Elasticsearch build.

We have split the major versions in separate urls to avoid accidental upgrades across major version. For all 0.90.x releases use 0.90 as version number, for 1.0.x use 1.0, for 1.1.x use 1.1 etc.

We use the PGP key D88E42B4 , Elasticsearch Signing Key, with fingerprint

4609 5ACC 8548 582C 1A26 99A9 D27D 666C D88E 42B4

to sign all our packages. It is available from http://pgp.mit.edu .

APT

Download and install the Public Signing Key:

wget -qO - https://packages.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -

Save the repository definition to /etc/apt/sources.list.d/elasticsearch-{branch}.list :

echo "deb http://packages.elastic.co/elasticsearch/1.7/debian stable main" | sudo tee -a /etc/apt/sources.list.d/elasticsearch-1.7.list

Use the echo method described above to add the Elasticsearch repository. Do not use add-apt-repository as it will add a deb-src entry as well, but we do not provide a source package. If you have added the deb-src entry, you will see an error like the following:

Unable to find expected entry 'main/source/Sources' in Release file (Wrong sources.list entry or malformed file)

Just delete the deb-src entry from the /etc/apt/sources.list file and the installation should work as expected.

Run apt-get update and the repository is ready for use. You can install it with:

sudo apt-get update && sudo apt-get install elasticsearch

If two entries exist for the same Elasticsearch repository, you will see an error like this during apt-get update :

Duplicate sources.list entry http://packages.elastic.co/elasticsearch/1.7/debian/ ...`

Examine /etc/apt/sources.list.d/elasticsearch-1.7.list for the duplicate entry or locate the duplicate entry amongst the files in /etc/apt/sources.list.d/ and the /etc/apt/sources.list file.

Configure Elasticsearch to automatically start during bootup. If your distribution is using SysV init, then you will need to run:

sudo update-rc.d elasticsearch defaults 95 10

Otherwise if your distribution is using systemd:

sudo /bin/systemctl daemon-reload
sudo /bin/systemctl enable elasticsearch.service

YUM

Download and install the public signing key:

rpm --import https://packages.elastic.co/GPG-KEY-elasticsearch

Add the following in your /etc/yum.repos.d/ directory in a file with a .repo suffix, for example elasticsearch.repo

[elasticsearch-1.7]
name=Elasticsearch repository for 1.7.x packages
baseurl=http://packages.elastic.co/elasticsearch/1.7/centos
gpgcheck=1
gpgkey=http://packages.elastic.co/GPG-KEY-elasticsearch
enabled=1

And your repository is ready for use. You can install it with:

yum install elasticsearch

Configure Elasticsearch to automatically start during bootup. If your distribution is using SysV init (check with ps -p 1 ), then you will need to run:

The repositories do not work with older rpm based distributions that still use RPM v3, like CentOS5.

chkconfig --add elasticsearch

Otherwise if your distribution is using systemd :

sudo /bin/systemctl daemon-reload
sudo /bin/systemctl enable elasticsearch.service

Elasticsearch can usually be upgraded using a rolling upgrade process, resulting in no interruption of service. This section details how to perform both rolling and restart upgrades.

To determine whether a rolling upgrade is supported for your release, please consult this table:

Before upgrading Elasticsearch, it is a good idea to consult the breaking changes docs.

Back Up Your Data!

Before performing an upgrade, it’s a good idea to back up the data on your system. This will allow you to roll back in the event of a problem with the upgrade. The upgrades sometimes include upgrades to the Lucene libraries used by Elasticsearch to access the index files, and after an index file has been updated to work with a new version of Lucene, it may not be accessible to the versions of Lucene present in earlier Elasticsearch releases.

0.90.x and earlier

To back up a running 0.90.x system, first disable index flushing. This will prevent indices from being flushed to disk while the backup is in process:

$ curl -XPUT 'http://localhost:9200/_all/_settings' -d '{
    "index": {
        "translog.disable_flush": "true"
}'

Then disable reallocation. This will prevent the cluster from moving data files from one node to another while the backup is in process:

$ curl -XPUT 'http://localhost:9200/_cluster/settings' -d '{
    "transient" : {
        "cluster.routing.allocation.disable_allocation": "true"
}'

After reallocation and index flushing are disabled, initiate a backup of Elasticsearch’s data path using your favorite backup method (tar, storage array snapshots, backup software). When the backup is complete and data no longer needs to be read from the Elasticsearch data path, reallocation and index flushing must be re-enabled:

$ curl -XPUT 'http://localhost:9200/_all/_settings' -d '{
    "index": {
        "translog.disable_flush": "false"
$ curl -XPUT 'http://localhost:9200/_cluster/settings' -d '{
    "transient" : {
        "cluster.routing.allocation.disable_allocation": "false"
}'

1.0 and later

To back up a running 1.0 or later system, it is simplest to use the snapshot feature. See the complete instructions for backup and restore with snapshots .

Rolling upgrade process

A rolling upgrade allows the ES cluster to be upgraded one node at a time, with no observable downtime for end users. Running multiple versions of Elasticsearch in the same cluster for any length of time beyond that required for an upgrade is not supported, as shard replication from the more recent version to the previous versions will not work.

Within minor or maintenance releases after release 1.0, rolling upgrades are supported. To perform a rolling upgrade:

This syntax applies to Elasticsearch 1.0 and later:

curl -XPUT localhost:9200/_cluster/settings -d '{
        "transient" : {
            "cluster.routing.allocation.enable" : "none"
(Only applicable to upgrades from ES 1.6.0 to a higher version) There is no problem continuing to index while doing the upgrade. However, you can speed the process considerably
by temporarily stopping non-essential indexing and issuing a manual synced flush.
A synced flush is special kind of flush which can seriously speed up recovery of shards. Elasticsearch automatically
uses it when an index has been inactive for a while (default is 5m) but you can manually trigger it using the following command:

curl -XPOST localhost:9200/_all/_flush/synced

Note that a synced flush call is a best effort operation. It will fail there are any pending indexing operations. It is safe to issue it multiple times if needed.

curl -XPOST 'http://localhost:9200/_cluster/nodes/_local/_shutdown'

curl -XPUT localhost:9200/_cluster/settings -d '{
        "transient" : {
            "cluster.routing.allocation.enable" : "all"
Observe that all shards are properly allocated on all nodes.  Balancing may take some time.
Repeat this process for all remaining nodes.

During a rolling upgrade, primary shards assigned to a node with the higher version will never have their replicas assigned to a node with the lower version, because the newer version may have a different data format which is not understood by the older version.

If it is not possible to assign the replica shards to another node with the higher version — e.g. if there is only one node with the higher version in the cluster — then the replica shards will remain unassigned, i.e. the cluster health will be status yellow . As soon as another node with the higher version joins the cluster, the replicas should be assigned and the cluster health will reach status green .

It may be possible to perform the upgrade by installing the new software while the service is running. This would reduce downtime by ensuring the service was ready to run on the new version as soon as it is stopped on the node being upgraded. This can be done by installing the new version in its own directory and using the symbolic link method outlined above. It is important to test this procedure first to be sure that site-specific configuration data and production indices will not be overwritten during the upgrade process.

Cluster restart upgrade process

Elasticsearch releases prior to 1.0 and releases after 1.0 are not compatible with each other, so a rolling upgrade is not possible. In order to upgrade a pre-1.0 system to 1.0 or later, a full cluster stop and start is required. In order to perform this upgrade:

This syntax is from versions prior to 1.0:

curl -XPUT localhost:9200/_cluster/settings -d '{
    "persistent" : {
    "cluster.routing.allocation.disable_allocation" : true
Stop all Elasticsearch services on all nodes in the cluster.

        curl -XPOST 'http://localhost:9200/_shutdown'

This syntax is from release 1.0 and later:

curl -XPUT localhost:9200/_cluster/settings -d '{
        "persistent" : {
    "cluster.routing.allocation.disable_allocation": false,
        "cluster.routing.allocation.enable" : "all"
}'

The cluster upgrade can be streamlined by installing the software before stopping cluster services. If this is done, testing must be performed to ensure that no production data or configuration files are overwritten prior to restart.

This section discusses the changes that you need to be aware of when migrating your application from Elasticsearch 1.x to Elasticsearch 1.6.

More Like This API

The More Like This API query has been deprecated and will be removed in 2.0. Instead use the More Like This Query .

top_children query

The top_children query has been deprecated and will be removed in 2.0. Instead the has_child query should be used. The top_children query isn’t always faster than the has_child query and the top_children query is often inaccurate. The total hits and any aggregations in the same search request will likely be off.

Snapshot and Restore

Locations of file system repositories has to be now registered using path.repo setting. The path.repo setting can contain one or more repository locations:

path.repo: ["/mnt/daily", "/mnt/weekly"]

If the file system repository location is specified as an absolute path it has to start with one of the locations specified in path.repo . If the location is specified as a relative path, it will be resolved against the first location specified in the path.repo setting.

Starting with version 1.6.1 URL repositories with http: , https: , and ftp: URLs has to be whitelisted by specifying allowed URLs in the repositories.url.allowed_urls setting. This setting supports wildcards in the place of host, path, query, and fragment. For example:

repositories.url.allowed_urls: ["http://www.example.org/root/*", "https://*.mydomain.com/*?*#*"]

This section discusses the changes that you need to be aware of when migrating your application from Elasticsearch 1.x to Elasticsearch 1.5.

Aggregations

The date_histogram aggregation now support a simplified offset option that replaces the previous pre_offset and post_offset which are deprecated in 1.5. Instead of having to specify two separate offset shifts of the underlying buckets, the offset option moves the bucket boundaries in positive or negative direction depending on its argument.

Also for date_histogram , options for pre_zone and post_zone options and the pre_zone_adjust_large_interval parameter are deprecated in 1.5 and replaced by the already existing time_zone option. The behavior of time_zone is equivalent to the former pre_zone option.

This section discusses the changes that you need to be aware of when migrating your application from Elasticsearch 1.x to Elasticsearch 1.4.

Percolator

In indices created with version 1.4.0 or later, percolation queries can only refer to fields that already exist in the mappings in that index. There are two ways to make sure that a field mapping exist:

Aliases

Aliases can include filters which are automatically applied to any search performed via the alias. Filtered aliases created with version 1.4.0 or later can only refer to field names which exist in the mappings of the index (or indices) pointed to by the alias.

Add or update a mapping via the create index or put mapping apis.

Indices APIs

The get warmer api will return a section for warmers even if there are no warmers. This ensures that the following two examples are equivalent:

curl -XGET 'http://localhost:9200/_all/_warmers'
curl -XGET 'http://localhost:9200/_warmers'

The get alias api will return a section for aliases even if there are no aliases. This ensures that the following two examples are equivalent:

curl -XGET 'http://localhost:9200/_all/_aliases'
curl -XGET 'http://localhost:9200/_aliases'

The get mapping api will return a section for mappings even if there are no mappings. This ensures that the following two examples are equivalent:

curl -XGET 'http://localhost:9200/_all/_mappings'
curl -XGET 'http://localhost:9200/_mappings'

Bulk UDP

Bulk UDP has been deprecated and will be removed in 2.0. You should use the standard bulk API instead. Each cluster must have an elected master node in order to be fully operational. Once a node loses its elected master node it will reject some or all operations.

Zen discovery

On versions before 1.4.0.Beta1 all operations are rejected when a node loses its elected master. From 1.4.0.Beta1 only write operations will be rejected by default. Read operations will still be served based on the information available to the node, which may result in being partial and possibly also stale. If the default is undesired then the pre 1.4.0.Beta1 behaviour can be enabled, see: no-master-block

More Like This Field

The More Like This Field query has been deprecated in favor of the More Like This Query restrained set to a specific field . It will be removed in 2.0.

MVEL is deprecated

Groovy is the new default scripting language in Elasticsearch, and is enabled in sandbox mode by default. MVEL has been removed from core, but is available as a plugin: https://github.com/elasticsearch/elasticsearch-lang-mvel

This section discusses the changes that you need to be aware of when migrating your application to Elasticsearch 1.0.

Many users use a proxy with URL-based access control to secure access to Elasticsearch indices. For multi-search , multi-get and bulk requests, the user has the choice of specifying an index in the URL and on each individual request within the request body. This can make URL-based access control challenging.

To prevent the user from overriding the index which has been specified in the URL, add this setting to the config.yml file:

rest.action.multi.allow_explicit_index: false

The default value is true , but when set to false , Elasticsearch will reject requests that have an explicit index specified in the request body.

The index API adds or updates a typed JSON document in a specific index, making it searchable. The following example inserts the JSON document into the "twitter" index, under a type called "tweet" with an id of 1:

$ curl -XPUT 'http://localhost:9200/twitter/tweet/1' -d '{
    "user" : "kimchy",
    "post_date" : "2009-11-15T14:12:12",
    "message" : "trying out Elasticsearch"
}'

The result of the above index operation is:

{
    "_index" : "twitter",
    "_type" : "tweet",
    "_id" : "1",
    "_version" : 1,
    "created" : true
}

Automatic Index Creation

The index operation automatically creates an index if it has not been created before (check out the create index API for manually creating an index), and also automatically creates a dynamic type mapping for the specific type if one has not yet been created (check out the put mapping API for manually creating a type mapping).

The mapping itself is very flexible and is schema-free. New fields and objects will automatically be added to the mapping definition of the type specified. Check out the mapping section for more information on mapping definitions.

Note that the format of the JSON document can also include the type (very handy when using JSON mappers) if the index.mapping.allow_type_wrapper setting is set to true, for example:

$ curl -XPOST 'http://localhost:9200/twitter' -d '{
  "settings": {
    "index": {
      "mapping.allow_type_wrapper": true
{"acknowledged":true}
$ curl -XPUT 'http://localhost:9200/twitter/tweet/1' -d '{
    "tweet" : {
        "user" : "kimchy",
        "post_date" : "2009-11-15T14:12:12",
        "message" : "trying out Elasticsearch"
}'

Automatic index creation can be disabled by setting action.auto_create_index to false in the config file of all nodes. Automatic mapping creation can be disabled by setting index.mapper.dynamic to false in the config files of all nodes (or on the specific index settings).

Automatic index creation can include a pattern based white/black list, for example, set action.auto_create_index to +aaa*,-bbb*,+ccc*,-* (+ meaning allowed, and - meaning disallowed).

Versioning

Each indexed document is given a version number. The associated version number is returned as part of the response to the index API request. The index API optionally allows for optimistic concurrency control when the version parameter is specified. This will control the version of the document the operation is intended to be executed against. A good example of a use case for versioning is performing a transactional read-then-update. Specifying a version from the document initially read ensures no changes have happened in the meantime (when reading in order to update, it is recommended to set preference to _primary ). For example:

curl -XPUT 'localhost:9200/twitter/tweet/1?version=2' -d '{
    "message" : "elasticsearch now has versioning support, double cool!"
}'

NOTE: versioning is completely real time, and is not affected by the near real time aspects of search operations. If no version is provided, then the operation is executed without any version checks.

By default, internal versioning is used that starts at 1 and increments with each update, deletes included. Optionally, the version number can be supplemented with an external value (for example, if maintained in a database). To enable this functionality, version_type should be set to external . The value provided must be a numeric, long value greater or equal to 0, and less than around 9.2e+18. When using the external version type, instead of checking for a matching version number, the system checks to see if the version number passed to the index request is greater than the version of the currently stored document. If true, the document will be indexed and the new version number used. If the value provided is less than or equal to the stored document’s version number, a version conflict will occur and the index operation will fail.

A nice side effect is that there is no need to maintain strict ordering of async indexing operations executed as a result of changes to a source database, as long as version numbers from the source database are used. Even the simple case of updating the elasticsearch index using data from a database is simplified if external versioning is used, as only the latest version will be used if the index operations are out of order for whatever reason.

Version types

Next to the internal & external version types explained above, Elasticsearch also supports other types for specific use cases. Here is an overview of the different version types and their semantics.

NOTE : The external_gte & force version types are meant for special use cases and should be used with care. If used incorrectly, they can result in loss of data.

Operation Type

The index operation also accepts an op_type that can be used to force a create operation, allowing for "put-if-absent" behavior. When create is used, the index operation will fail if a document by that id already exists in the index.

Here is an example of using the op_type parameter:

$ curl -XPUT 'http://localhost:9200/twitter/tweet/1?op_type=create' -d '{
    "user" : "kimchy",
    "post_date" : "2009-11-15T14:12:12",
    "message" : "trying out Elasticsearch"
}'

Another option to specify create is to use the following uri:

$ curl -XPUT 'http://localhost:9200/twitter/tweet/1/_create' -d '{
    "user" : "kimchy",
    "post_date" : "2009-11-15T14:12:12",
    "message" : "trying out Elasticsearch"
}'

Automatic ID Generation

The index operation can be executed without specifying the id. In such a case, an id will be generated automatically. In addition, the op_type will automatically be set to create . Here is an example (note the POST used instead of PUT ):

$ curl -XPOST 'http://localhost:9200/twitter/tweet/' -d '{
    "user" : "kimchy",
    "post_date" : "2009-11-15T14:12:12",
    "message" : "trying out Elasticsearch"
}'

The result of the above index operation is:

{
    "_index" : "twitter",
    "_type" : "tweet",
    "_id" : "6a8ca01c-7896-48e9-81cc-9f70661fcb32",
    "_version" : 1,
    "created" : true
}

Routing

By default, shard placement — or routing — is controlled by using a hash of the document’s id value. For more explicit control, the value fed into the hash function used by the router can be directly specified on a per-operation basis using the routing parameter. For example:

$ curl -XPOST 'http://localhost:9200/twitter/tweet?routing=kimchy' -d '{
    "user" : "kimchy",
    "post_date" : "2009-11-15T14:12:12",
    "message" : "trying out Elasticsearch"
}'

In the example above, the "tweet" document is routed to a shard based on the routing parameter provided: "kimchy".

When setting up explicit mapping, the _routing field can be optionally used to direct the index operation to extract the routing value from the document itself. This does come at the (very minimal) cost of an additional document parsing pass. If the _routing mapping is defined, and set to be required , the index operation will fail if no routing value is provided or extracted.

Parents & Children

A child document can be indexed by specifying its parent when indexing. For example:

$ curl -XPUT localhost:9200/blogs/blog_tag/1122?parent=1111 -d '{
    "tag" : "something"
}'

When indexing a child document, the routing value is automatically set to be the same as its parent, unless the routing value is explicitly specified using the routing parameter.

Timestamp

A document can be indexed with a timestamp associated with it. The timestamp value of a document can be set using the timestamp parameter. For example:

$ curl -XPUT localhost:9200/twitter/tweet/1?timestamp=2009-11-15T14%3A12%3A12 -d '{
    "user" : "kimchy",
    "message" : "trying out Elasticsearch"
}'

If the timestamp value is not provided externally or in the _source , the timestamp will be automatically set to the date the document was processed by the indexing chain. More information can be found on the _timestamp mapping page .

TTL

A document can be indexed with a ttl (time to live) associated with it. Expired documents will be expunged automatically. The expiration date that will be set for a document with a provided ttl is relative to the timestamp of the document, meaning it can be based on the time of indexing or on any time provided. The provided ttl must be strictly positive and can be a number (in milliseconds) or any valid time value as shown in the following examples:

curl -XPUT 'http://localhost:9200/twitter/tweet/1?ttl=86400000' -d '{
    "user": "kimchy",
    "message": "Trying out elasticsearch, so far so good?"
}'

curl -XPUT 'http://localhost:9200/twitter/tweet/1?ttl=1d' -d '{
    "user": "kimchy",
    "message": "Trying out elasticsearch, so far so good?"
}'

curl -XPUT 'http://localhost:9200/twitter/tweet/1' -d '{
    "_ttl": "1d",
    "user": "kimchy",
    "message": "Trying out elasticsearch, so far so good?"
}'

More information can be found on the _ttl mapping page .

Distributed

The index operation is directed to the primary shard based on its route (see the Routing section above) and performed on the actual node containing this shard. After the primary shard completes the operation, if needed, the update is distributed to applicable replicas. The index operation only returns after all shards within the replication group have indexed the document (sync replication).

Write Consistency

To prevent writes from taking place on the "wrong" side of a network partition, by default, index operations only succeed if a quorum (>replicas/2+1) of active shards are available. This default can be overridden on a node-by-node basis using the action.write_consistency setting. To alter this behavior per-operation, the consistency request parameter can be used.

Valid write consistency values are one , quorum , and all .

Note, for the case where the number of replicas is 1 (total of 2 copies of the data), then the default behavior is to succeed if 1 copy (the primary) can perform the write.

Refresh

To refresh the shard (not the whole index) immediately after the operation occurs, so that the document appears in search results immediately, the refresh parameter can be set to true . Setting this option to true should ONLY be done after careful thought and verification that it does not lead to poor performance, both from an indexing and a search standpoint. Note, getting a document using the get API is completely realtime.

Timeout

The primary shard assigned to perform the index operation might not be available when the index operation is executed. Some reasons for this might be that the primary shard is currently recovering from a gateway or undergoing relocation. By default, the index operation will wait on the primary shard to become available for up to 1 minute before failing and responding with an error. The timeout parameter can be used to explicitly specify how long it waits. Here is an example of setting it to 5 minutes:

$ curl -XPUT 'http://localhost:9200/twitter/tweet/1?timeout=5m' -d '{
    "user" : "kimchy",
    "post_date" : "2009-11-15T14:12:12",
    "message" : "trying out Elasticsearch"
          Delete API
            

curl -XGET 'http://localhost:9200/twitter/tweet/1'

{
    "_index" : "twitter",
    "_type" : "tweet",
    "_id" : "1",
    "_version" : 1,
    "found": true,
    "_source" : {
        "user" : "kimchy",
        "postDate" : "2009-11-15T14:12:12",
        "message" : "trying out Elasticsearch"
}

curl -XHEAD -i 'http://localhost:9200/twitter/tweet/1'

curl -XGET 'http://localhost:9200/twitter/tweet/1?_source=false'

curl -XGET 'http://localhost:9200/twitter/tweet/1?_source_include=*.id&_source_exclude=entities'

curl -XGET 'http://localhost:9200/twitter/tweet/1?_source=*.id,retweeted'

curl -XGET 'http://localhost:9200/twitter/tweet/1?fields=title,content'

curl -XGET 'http://localhost:9200/twitter/tweet/1/_source'

curl -XGET 'http://localhost:9200/twitter/tweet/1/_source?_source_include=*.id&_source_exclude=entities'

curl -XHEAD -i 'http://localhost:9200/twitter/tweet/1/_source'

curl -XGET 'http://localhost:9200/twitter/tweet/1?routing=kimchy'

The delete API allows to delete a typed JSON document from a specific index based on its id. The following example deletes the JSON document from an index called twitter, under a type called tweet, with id valued 1:

$ curl -XDELETE 'http://localhost:9200/twitter/tweet/1'

The result of the above delete operation is:

{
    "found" : true,
    "_index" : "twitter",
    "_type" : "tweet",
    "_id" : "1",
    "_version" : 2
}

Versioning

Each document indexed is versioned. When deleting a document, the version can be specified to make sure the relevant document we are trying to delete is actually being deleted and it has not changed in the meantime. Every write operation executed on a document, deletes included, causes its version to be incremented.

Routing

When indexing using the ability to control the routing, in order to delete a document, the routing value should also be provided. For example:

$ curl -XDELETE 'http://localhost:9200/twitter/tweet/1?routing=kimchy'

The above will delete a tweet with id 1, but will be routed based on the user. Note, issuing a delete without the correct routing, will cause the document to not be deleted.

Many times, the routing value is not known when deleting a document. For those cases, when specifying the _routing mapping as required , and no routing value is specified, the delete will be broadcast automatically to all shards.

Parent

The parent parameter can be set, which will basically be the same as setting the routing parameter.

Note that deleting a parent document does not automatically delete its children. One way of deleting all child documents given a parent’s id is to perform a delete by query on the child index with the automatically generated (and indexed) field _parent, which is in the format parent_type#parent_id.

Automatic index creation

The delete operation automatically creates an index if it has not been created before (check out the create index API for manually creating an index), and also automatically creates a dynamic type mapping for the specific type if it has not been created before (check out the put mapping API for manually creating type mapping).

Distributed

The delete operation gets hashed into a specific shard id. It then gets redirected into the primary shard within that id group, and replicated (if needed) to shard replicas within that id group.

Write Consistency

Control if the operation will be allowed to execute based on the number of active shards within that partition (replication group). The values allowed are one , quorum , and all . The parameter to set it is consistency , and it defaults to the node level setting of action.write_consistency which in turn defaults to quorum .

For example, in a N shards with 2 replicas index, there will have to be at least 2 active shards within the relevant partition ( quorum ) for the operation to succeed. In a N shards with 1 replica scenario, there will need to be a single shard active (in this case, one and quorum is the same).

Refresh

The refresh parameter can be set to true in order to refresh the relevant primary and replica shards after the delete operation has occurred and make it searchable. Setting it to true should be done after careful thought and verification that this does not cause a heavy load on the system (and slows down indexing).

Timeout

The primary shard assigned to perform the delete operation might not be available when the delete operation is executed. Some reasons for this might be that the primary shard is currently recovering from a gateway or undergoing relocation. By default, the delete operation will wait on the primary shard to become available for up to 1 minute before failing and responding with an error. The timeout parameter can be used to explicitly specify how long it waits. Here is an example of setting it to 5 minutes:

$ curl -XDELETE 'http://localhost:9200/twitter/tweet/1?timeout=5m'

The update API allows to update a document based on a script provided. The operation gets the document (collocated with the shard) from the index, runs the script (with optional script language and parameters), and index back the result (also allows to delete, or ignore the operation). It uses versioning to make sure no updates have happened during the "get" and "reindex".

Note, this operation still means full reindex of the document, it just removes some network roundtrips and reduces chances of version conflicts between the get and the index. The _source field need to be enabled for this feature to work.

For example, lets index a simple doc:

curl -XPUT localhost:9200/test/type1/1 -d '{
    "counter" : 1,
    "tags" : ["red"]
}'

Scripted updates

Now, we can execute a script that would increment the counter:

curl -XPOST 'localhost:9200/test/type1/1/_update' -d '{
    "script" : "ctx._source.counter += count",
    "params" : {
        "count" : 4
}'

We can add a tag to the list of tags (note, if the tag exists, it will still add it, since its a list):

curl -XPOST 'localhost:9200/test/type1/1/_update' -d '{
    "script" : "ctx._source.tags += tag",
    "params" : {
        "tag" : "blue"
}'

In addition to _source , the following variables are available through the ctx map: _index , _type , _id , _version , _routing , _parent , _timestamp , _ttl .

We can also add a new field to the document:

curl -XPOST 'localhost:9200/test/type1/1/_update' -d '{
    "script" : "ctx._source.name_of_new_field = \"value_of_new_field\""
}'

Or remove a field from the document:

curl -XPOST 'localhost:9200/test/type1/1/_update' -d '{
    "script" : "ctx._source.remove(\"name_of_field\")"
}'

And, we can even change the operation that is executed. This example deletes the doc if the tags field contain blue , otherwise it does nothing ( noop ):

curl -XPOST 'localhost:9200/test/type1/1/_update' -d '{
    "script" : "ctx._source.tags.contains(tag) ? ctx.op = \"delete\" : ctx.op = \"none\"",
    "params" : {
        "tag" : "blue"
}'

Updates with a partial document

The update API also support passing a partial document, which will be merged into the existing document (simple recursive merge, inner merging of objects, replacing core "keys/values" and arrays). For example:

curl -XPOST 'localhost:9200/test/type1/1/_update' -d '{
    "doc" : {
        "name" : "new_name"
}'

If both doc and script is specified, then doc is ignored. Best is to put your field pairs of the partial document in the script itself.

detect_noop

By default if doc is specified then the document is always updated even if the merging process doesn’t cause any changes. Specifying detect_noop as true will cause Elasticsearch to check if there are changes and, if there aren’t, turn the update request into a noop. For example:

curl -XPOST 'localhost:9200/test/type1/1/_update' -d '{
    "doc" : {
        "name" : "new_name"
    "detect_noop": true
}'

If name was new_name before the request was sent then the entire update request is ignored.

Upserts

If the document does not already exist, the contents of the upsert element will be inserted as a new document. If the document does exist, then the script will be executed instead:

curl -XPOST 'localhost:9200/test/type1/1/_update' -d '{
    "script" : "ctx._source.counter += count",
    "params" : {
        "count" : 4
    "upsert" : {
        "counter" : 1
}'

scripted_upsert

If you would like your script to run regardless of whether the document exists or not — i.e. the script handles initializing the document instead of the upsert element — then set scripted_upsert to true :

curl -XPOST 'localhost:9200/sessions/session/dh3sgudg8gsrgl/_update' -d '{
    "script_id" : "my_web_session_summariser",
    "scripted_upsert":true,
    "params" : {
        "pageViewEvent" : {
                "url":"foo.com/bar",
                "response":404,
                "time":"2014-01-01 12:32"
    "upsert" : {}
}'

doc_as_upsert

Instead of sending a partial doc plus an upsert doc, setting doc_as_upsert to true will use the contents of doc as the upsert value:

curl -XPOST 'localhost:9200/test/type1/1/_update' -d '{
    "doc" : {
        "name" : "new_name"
    "doc_as_upsert" : true
}'

Parameters

The update operation supports the following query-string parameters:

curl 'localhost:9200/_mget' -d '{
    "docs" : [
            "_index" : "test",
            "_type" : "type",
            "_id" : "1"
            "_index" : "test",
            "_type" : "type",
            "_id" : "2"
}'

curl 'localhost:9200/test/_mget' -d '{
    "docs" : [
            "_type" : "type",
            "_id" : "1"
            "_type" : "type",
            "_id" : "2"
}'

curl 'localhost:9200/test/type/_mget' -d '{
    "docs" : [
            "_id" : "1"
            "_id" : "2"
}'

curl 'localhost:9200/test/type/_mget' -d '{
    "ids" : ["1", "2"]
}'

curl 'localhost:9200/test/_mget' -d '{
    "ids" : ["1", "1"]
}'

GET /test/_mget/
  "docs" : [
            "_type":"typeA",
            "_id" : "1"
            "_type":"typeB",
            "_id" : "1"
}

curl 'localhost:9200/_mget' -d '{
    "docs" : [
            "_index" : "test",
            "_type" : "type",
            "_id" : "1",
            "_source" : false
            "_index" : "test",
            "_type" : "type",
            "_id" : "2",
            "_source" : ["field3", "field4"]
            "_index" : "test",
            "_type" : "type",
            "_id" : "3",
            "_source" : {
                "include": ["user"],
                "exclude": ["user.location"]
}'

curl 'localhost:9200/_mget' -d '{
    "docs" : [
            "_index" : "test",
            "_type" : "type",
            "_id" : "1",
            "fields" : ["field1", "field2"]
            "_index" : "test",
            "_type" : "type",
            "_id" : "2",
            "fields" : ["field3", "field4"]
}'

curl 'localhost:9200/test/type/_mget?fields=field1,field2' -d '{
    "docs" : [
            "_id" : "1" 
            "_id" : "2",
            "fields" : ["field3", "field4"] 
Returns field1 and field2
Returns field3 and field4

Generated fields

See the section called “Generated fields ” for fields are generated only when indexing.

Routing

You can also specify routing value as a parameter:

curl 'localhost:9200/_mget?routing=key1' -d '{
    "docs" : [
            "_index" : "test",
            "_type" : "type",
            "_id" : "1",
            "_routing" : "key2"
            "_index" : "test",
            "_type" : "type",
            "_id" : "2"
}'

In this example, document test/type/2 will be fetch from shard corresponding to routing key key1 but document test/type/1 will be fetch from shard corresponding to routing key key2 .

Security

See URL-based access control

The bulk API makes it possible to perform many index/delete operations in a single API call. This can greatly increase the indexing speed.

The REST API endpoint is /_bulk , and it expects the following JSON structure:

action_and_meta_data\n
optional_source\n
action_and_meta_data\n
optional_source\n
action_and_meta_data\n
optional_source\n

NOTE : the final line of data must end with a newline character \n .

The possible actions are index , create , delete and update . index and create expect a source on the next line, and have the same semantics as the op_type parameter to the standard index API (i.e. create will fail if a document with the same index and type exists already, whereas index will add or replace a document as necessary). delete does not expect a source on the following line, and has the same semantics as the standard delete API. update expects that the partial doc, upsert and script and its options are specified on the next line.

If you’re providing text file input to curl , you must use the --data-binary flag instead of plain -d . The latter doesn’t preserve newlines. Example:

$ cat requests
{ "index" : { "_index" : "test", "_type" : "type1", "_id" : "1" } }
{ "field1" : "value1" }
$ curl -s -XPOST localhost:9200/_bulk --data-binary "@requests"; echo
{"took":7,"items":[{"create":{"_index":"test","_type":"type1","_id":"1","_version":1}}]}

Because this format uses literal \n 's as delimiters, please be sure that the JSON actions and sources are not pretty printed. Here is an example of a correct sequence of bulk commands:

{ "index" : { "_index" : "test", "_type" : "type1", "_id" : "1" } }
{ "field1" : "value1" }
{ "delete" : { "_index" : "test", "_type" : "type1", "_id" : "2" } }
{ "create" : { "_index" : "test", "_type" : "type1", "_id" : "3" } }
{ "field1" : "value3" }
{ "update" : {"_id" : "1", "_type" : "type1", "_index" : "index1"} }
{ "doc" : {"field2" : "value2"} }

In the above example doc for the update action is a partial document, that will be merged with the already stored document.

The endpoints are /_bulk , /{index}/_bulk , and {index}/{type}/_bulk . When the index or the index/type are provided, they will be used by default on bulk items that don’t provide them explicitly.

A note on the format. The idea here is to make processing of this as fast as possible. As some of the actions will be redirected to other shards on other nodes, only action_meta_data is parsed on the receiving node side.

Client libraries using this protocol should try and strive to do something similar on the client side, and reduce buffering as much as possible.

The response to a bulk action is a large JSON structure with the individual results of each action that was performed. The failure of a single action does not affect the remaining actions.

There is no "correct" number of actions to perform in a single bulk call. You should experiment with different settings to find the optimum size for your particular workload.

If using the HTTP API, make sure that the client does not send HTTP chunks, as this will slow things down.

Versioning

Each bulk item can include the version value using the _version / version field. It automatically follows the behavior of the index / delete operation based on the _version mapping. It also support the version_type / _version_type (see versioning )

Routing

Each bulk item can include the routing value using the _routing / routing field. It automatically follows the behavior of the index / delete operation based on the _routing mapping.

Parent

Each bulk item can include the parent value using the _parent / parent field. It automatically follows the behavior of the index / delete operation based on the _parent / _routing mapping.

Timestamp

Each bulk item can include the timestamp value using the _timestamp / timestamp field. It automatically follows the behavior of the index operation based on the _timestamp mapping.

TTL

Each bulk item can include the ttl value using the _ttl / ttl field. It automatically follows the behavior of the index operation based on the _ttl mapping.

Write Consistency

When making bulk calls, you can require a minimum number of active shards in the partition through the consistency parameter. The values allowed are one , quorum , and all . It defaults to the node level setting of action.write_consistency , which in turn defaults to quorum .

For example, in a N shards with 2 replicas index, there will have to be at least 2 active shards within the relevant partition ( quorum ) for the operation to succeed. In a N shards with 1 replica scenario, there will need to be a single shard active (in this case, one and quorum is the same).

Refresh

The refresh parameter can be set to true in order to refresh the relevant primary and replica shards immediately after the bulk operation has occurred and make it searchable, instead of waiting for the normal refresh interval to expire. Setting it to true can trigger additional load, and may slow down indexing. Due to its costly nature, the refresh parameter is set on the bulk request level and is not supported on each individual bulk item.

Update

When using update action _retry_on_conflict can be used as field in the action itself (not in the extra payload line), to specify how many times an update should be retried in the case of a version conflict.

The update action payload, supports the following options: doc (partial document), upsert , doc_as_upsert , script , params (for script), lang (for script). See update documentation for details on the options. Curl example with update actions:

{ "update" : {"_id" : "1", "_type" : "type1", "_index" : "index1", "_retry_on_conflict" : 3} }
{ "doc" : {"field" : "value"} }
{ "update" : { "_id" : "0", "_type" : "type1", "_index" : "index1", "_retry_on_conflict" : 3} }
{ "script" : "ctx._source.counter += param1", "lang" : "js", "params" : {"param1" : 1}, "upsert" : {"counter" : 1}}
{ "update" : {"_id" : "2", "_type" : "type1", "_index" : "index1", "_retry_on_conflict" : 3} }
{ "doc" : {"field" : "value"}, "doc_as_upsert" : true }

Security

See URL-based access control

Deprecated in 1.5.3.

"Delete by Query will be removed in 2.0: it is problematic since it silently forces a refresh which can quickly cause OutOfMemoryError during concurrent indexing, and can also cause primary and replica to become inconsistent. Instead, use the scroll/scan API to find all matching ids and then issue a bulk request to delete them.

The delete by query API allows to delete documents from one or more indices and one or more types based on a query. The query can either be provided using a simple query string as a parameter, or using the Query DSL defined within the request body. Here is an example:

$ curl -XDELETE 'http://localhost:9200/twitter/tweet/_query?q=user:kimchy'
$ curl -XDELETE 'http://localhost:9200/twitter/tweet/_query' -d '{
    "query" : {
        "term" : { "user" : "kimchy" }
'

The query being sent in the body must be nested in a query key, same as the search api works

Both above examples end up doing the same thing, which is delete all tweets from the twitter index for a certain user. The result of the commands is:

{
    "_indices" : {
        "twitter" : {
            "_shards" : {
                "total" : 5,
                "successful" : 5,
                "failed" : 0
}

Note, delete by query bypasses versioning support. Also, it is not recommended to delete "large chunks of the data in an index", many times, it’s better to simply reindex into a new index.

Multiple Indices and Types

The delete by query API can be applied to multiple types within an index, and across multiple indices. For example, we can delete all documents across all types within the twitter index:

$ curl -XDELETE 'http://localhost:9200/twitter/_query?q=user:kimchy'

We can also delete within specific types:

$ curl -XDELETE 'http://localhost:9200/twitter/tweet,user/_query?q=user:kimchy'

We can also delete all tweets with a certain tag across several indices (for example, when each user has his own index):

$ curl -XDELETE 'http://localhost:9200/kimchy,elasticsearch/_query?q=tag:wow'

Or even delete across all indices:

$ curl -XDELETE 'http://localhost:9200/_all/_query?q=tag:wow'

Request Parameters

When executing a delete by query using the query parameter q , the query passed is a query string using Lucene query parser. There are additional parameters that can be passed:

Request Body

The delete by query can use the Query DSL within its body in order to express the query that should be executed and delete all documents. The body content can also be passed as a REST parameter named source .

Distributed

The delete by query API is broadcast across all primary shards, and from there, replicated across all shards replicas.

Routing

The routing value (a comma separated list of the routing values) can be specified to control which shards the delete by query request will be executed on.

Write Consistency

Control if the operation will be allowed to execute based on the number of active shards within that partition (replication group). The values allowed are one , quorum , and all . The parameter to set it is consistency , and it defaults to the node level setting of action.write_consistency which in turn defaults to quorum .

For example, in a N shards with 2 replicas index, there will have to be at least 2 active shards within the relevant partition ( quorum ) for the operation to succeed. In a N shards with 1 replica scenario, there will need to be a single shard active (in this case, one and quorum is the same).

Limitations

The delete by query does not support the following queries and filters: has_child , has_parent and top_children .

Bulk UDP has been deprecated and will be removed in Elasticsearch 2.0. You should use the standard bulk API instead.

A Bulk UDP service is a service listening over UDP for bulk format requests. The idea is to provide a low latency UDP service that allows to easily index data that is not of critical nature.

The Bulk UDP service is disabled by default, but can be enabled by setting bulk.udp.enabled to true .

The bulk UDP service performs internal bulk aggregation of the data and then flushes it based on several parameters:

The allowed network settings are:

Here is an example of how it can be used:

> cat bulk.txt
{ "index" : { "_index" : "test", "_type" : "type1" } }
{ "field1" : "value1" }
{ "index" : { "_index" : "test", "_type" : "type1" } }
{ "field1" : "value1" }

> cat bulk.txt | nc -w 0 -u localhost 9700

Returns information and statistics on terms in the fields of a particular document. The document could be stored in the index or artificially provided by the user. Term vectors are realtime by default, not near realtime. This can be changed by setting realtime parameter to false .

curl -XGET 'http://localhost:9200/twitter/tweet/1/_termvector?pretty=true'

Optionally, you can specify the fields for which the information is retrieved either with a parameter in the url

curl -XGET 'http://localhost:9200/twitter/tweet/1/_termvector?fields=text,...'

or by adding the requested fields in the request body (see example below). Fields can also be specified with wildcards in similar way to the multi match query .

Return values

Three types of values can be requested: term information , term statistics and field statistics . By default, all term information and field statistics are returned for all fields but no term statistics.

Term information

If the requested information wasn’t stored in the index, it will be computed on the fly if possible. Additionally, term vectors could be computed for documents not even existing in the index, but instead provided by the user.

Start and end offsets assume UTF-16 encoding is being used. If you want to use these offsets in order to get the original text that produced this token, you should make sure that the string you are taking a sub-string of is also encoded using UTF-16.

Term statistics

Setting term_statistics to true (default is false ) will return

By default these values are not returned since term statistics can have a serious performance impact.

Field statistics

Setting field_statistics to false (default is true ) will omit :

Behaviour

The term and field statistics are not accurate. Deleted documents are not taken into account. The information is only retrieved for the shard the requested document resides in. The term and field statistics are therefore only useful as relative measures whereas the absolute numbers have no meaning in this context. By default, when requesting term vectors of artificial documents, a shard to get the statistics from is randomly selected. Use routing only to hit a particular shard.

Example 1

First, we create an index that stores term vectors, payloads etc. :

curl -s -XPUT 'http://localhost:9200/twitter/' -d '{
  "mappings": {
    "tweet": {
      "properties": {
        "text": {
          "type": "string",
          "term_vector": "with_positions_offsets_payloads",
          "store" : true,
          "index_analyzer" : "fulltext_analyzer"
         "fullname": {
          "type": "string",
          "term_vector": "with_positions_offsets_payloads",
          "index_analyzer" : "fulltext_analyzer"
  "settings" : {
    "index" : {
      "number_of_shards" : 1,
      "number_of_replicas" : 0
    "analysis": {
      "analyzer": {
        "fulltext_analyzer": {
          "type": "custom",
          "tokenizer": "whitespace",
          "filter": [
            "lowercase",
            "type_as_payload"
}'

Second, we add some documents:

curl -XPUT 'http://localhost:9200/twitter/tweet/1?pretty=true' -d '{
  "fullname" : "John Doe",
  "text" : "twitter test test test "
curl -XPUT 'http://localhost:9200/twitter/tweet/2?pretty=true' -d '{
  "fullname" : "Jane Doe",
  "text" : "Another twitter test ..."
}'

The following request returns all information and statistics for field text in document 1 (John Doe):

curl -XGET 'http://localhost:9200/twitter/tweet/1/_termvector?pretty=true' -d '{
  "fields" : ["text"],
  "offsets" : true,
  "payloads" : true,
  "positions" : true,
  "term_statistics" : true,
  "field_statistics" : true
}'

Response:

{
    "_id": "1",
    "_index": "twitter",
    "_type": "tweet",
    "_version": 1,
    "found": true,
    "term_vectors": {
        "text": {
            "field_statistics": {
                "doc_count": 2,
                "sum_doc_freq": 6,
                "sum_ttf": 8
            "terms": {
                "test": {
                    "doc_freq": 2,
                    "term_freq": 3,
                    "tokens": [
                            "end_offset": 12,
                            "payload": "d29yZA==",
                            "position": 1,
                            "start_offset": 8
                            "end_offset": 17,
                            "payload": "d29yZA==",
                            "position": 2,
                            "start_offset": 13
                            "end_offset": 22,
                            "payload": "d29yZA==",
                            "position": 3,
                            "start_offset": 18
                    "ttf": 4
                "twitter": {
                    "doc_freq": 2,
                    "term_freq": 1,
                    "tokens": [
                            "end_offset": 7,
                            "payload": "d29yZA==",
                            "position": 0,
                            "start_offset": 0
                    "ttf": 2
}

Example 2

Term vectors which are not explicitly stored in the index are automatically computed on the fly. The following request returns all information and statistics for the fields in document 1 , even though the terms haven’t been explicitly stored in the index. Note that for the field text , the terms are not re-generated.

curl -XGET 'http://localhost:9200/twitter/tweet/1/_termvector?pretty=true' -d '{
  "fields" : ["text", "some_field_without_term_vectors"],
  "offsets" : true,
  "positions" : true,
  "term_statistics" : true,
  "field_statistics" : true
}'

Example 3

Term vectors can also be generated for artificial documents, that is for documents not present in the index. The syntax is similar to the percolator API. For example, the following request would return the same results as in example 1. The mapping used is determined by the index and type .

If dynamic mapping is turned on (default), the document fields not in the original mapping will be dynamically created.

curl -XGET 'http://localhost:9200/twitter/tweet/_termvector' -d '{
  "doc" : {
    "fullname" : "John Doe",
    "text" : "twitter test test test"
}'

Example 4

Additionally, a different analyzer than the one at the field may be provided by using the per_field_analyzer parameter. This is useful in order to generate term vectors in any fashion, especially when using artificial documents. When providing an analyzer for a field that already stores term vectors, the term vectors will be re-generated.

curl -XGET 'http://localhost:9200/twitter/tweet/_termvector' -d '{
  "doc" : {
    "fullname" : "John Doe",
    "text" : "twitter test test test"
  "fields": ["fullname"],
  "per_field_analyzer" : {
    "fullname": "keyword"
}'

Response:

{
  "_index": "twitter",
  "_type": "tweet",
  "_version": 0,
  "found": true,
  "term_vectors": {
    "fullname": {
       "field_statistics": {
          "sum_doc_freq": 1,
          "doc_count": 1,
          "sum_ttf": 1
       "terms": {
          "John Doe": {
             "term_freq": 1,
             "tokens": [
                   "position": 0,
                   "start_offset": 0,
                   "end_offset": 8
          Search APIs
            

Multi termvectors API

Multi termvectors API allows to get multiple termvectors at once. The documents from which to retrieve the term vectors are specified by an index, type and id. But the documents could also be artificially provided. The response includes a docs array with all the fetched termvectors, each element having the structure provided by the termvectors API. Here is an example:

curl 'localhost:9200/_mtermvectors' -d '{
   "docs": [
         "_index": "testidx",
         "_type": "test",
         "_id": "2",
         "term_statistics": true
         "_index": "testidx",
         "_type": "test",
         "_id": "1",
         "fields": [
            "text"
}'

See the termvectors API for a description of possible parameters.

The _mtermvectors endpoint can also be used against an index (in which case it is not required in the body):

curl 'localhost:9200/testidx/_mtermvectors' -d '{
   "docs": [
         "_type": "test",
         "_id": "2",
         "fields": [
            "text"
         "term_statistics": true
         "_type": "test",
         "_id": "1"
}'

And type:

curl 'localhost:9200/testidx/test/_mtermvectors' -d '{
   "docs": [
         "_id": "2",
         "fields": [
            "text"
         "term_statistics": true
         "_id": "1"
}'

If all requested documents are on same index and have same type and also the parameters are the same, the request can be simplified:

curl 'localhost:9200/testidx/test/_mtermvectors' -d '{
    "ids" : ["1", "2"],
    "parameters": {
        "fields": [
                "text"
        "term_statistics": true,
}'

Additionally, just like for the termvectors API, term vectors could be generated for user provided documents. The syntax is similar to the percolator API. The mapping used is determined by _index and _type .

curl 'localhost:9200/_mtermvectors' -d '{
   "docs": [
         "_index": "testidx",
         "_type": "test",
         "doc" : {
            "fullname" : "John Doe",
            "text" : "twitter test test test"
         "_index": "testidx",
         "_type": "test",
         "doc" : {
           "fullname" : "Jane Doe",
           "text" : "Another twitter test ..."
            

Search APIs

Most search APIs are multi-index, multi-type , with the exception of the Explain API endpoints.

Routing

When executing a search, it will be broadcast to all the index/indices shards (round robin between replicas). Which shards will be searched on can be controlled by providing the routing parameter. For example, when indexing tweets, the routing value can be the user name:

$ curl -XPOST 'http://localhost:9200/twitter/tweet?routing=kimchy' -d '{
    "user" : "kimchy",
    "postDate" : "2009-11-15T14:12:12",
    "message" : "trying out Elasticsearch"
'

In such a case, if we want to search only on the tweets for a specific user, we can specify it as the routing, resulting in the search hitting only the relevant shard:

$ curl -XGET 'http://localhost:9200/twitter/tweet/_search?routing=kimchy' -d '{
    "query": {
        "filtered" : {
            "query" : {
                "query_string" : {
                    "query" : "some query string here"
            "filter" : {
                "term" : { "user" : "kimchy" }
'

The routing parameter can be multi valued represented as a comma separated string. This will result in hitting the relevant shards where the routing values match to.

Stats Groups

A search can be associated with stats groups, which maintains a statistics aggregation per group. It can later be retrieved using the indices stats API specifically. For example, here is a search body request that associate the request with two different groups:

{
    "query" : {
        "match_all" : {}
    "stats" : ["group1", "group2"]
          URI Search
            

Search

The search API allows to execute a search query and get back search hits that match the query. The query can either be provided using a simple query string as a parameter , or using a request body .

Multi-Index, Multi-Type

All search APIs can be applied across multiple types within an index, and across multiple indices with support for the multi index syntax . For example, we can search on all documents across all types within the twitter index:

$ curl -XGET 'http://localhost:9200/twitter/_search?q=user:kimchy'

We can also search within specific types:

$ curl -XGET 'http://localhost:9200/twitter/tweet,user/_search?q=user:kimchy'

We can also search all tweets with a certain tag across several indices (for example, when each user has his own index):

$ curl -XGET 'http://localhost:9200/kimchy,elasticsearch/tweet/_search?q=tag:wow'

Or we can search all tweets across all available indices using _all placeholder:

$ curl -XGET 'http://localhost:9200/_all/tweet/_search?q=tag:wow'

Or even search across all indices and all types:

$ curl -XGET 'http://localhost:9200/_search?q=tag:wow'

Request Body Search

URI Search

A search request can be executed purely using a URI by providing request parameters. Not all search options are exposed when executing a search using this mode, but it can be handy for quick "curl tests". Here is an example:

$ curl -XGET 'http://localhost:9200/twitter/tweet/_search?q=user:kimchy'

And here is a sample response:

{
    "_shards":{
        "total" : 5,
        "successful" : 5,
        "failed" : 0
    "hits":{
        "total" : 1,
        "hits" : [
                "_index" : "twitter",
                "_type" : "tweet",
                "_id" : "1",
                "_source" : {
                    "user" : "kimchy",
                    "postDate" : "2009-11-15T14:12:12",
                    "message" : "trying out Elasticsearch"
}

Parameters

The parameters allowed in the URI are:

Name	Description
q	The query string (maps to the query_string query, see Query String Query for more details).
df	The default field to use when no field prefix is defined within the query.
analyzer	The analyzer name to be used when analyzing the query string.
lowercase_expanded_terms	Should terms be automatically lowercased or not. Defaults to true .
analyze_wildcard	Should wildcard and prefix queries be analyzed or not. Defaults to false .
default_operator	The default operator to be used, can be AND or OR . Defaults to OR .
lenient	If set to true will cause format based failures (like providing text to a numeric field) to be ignored. Defaults to false.
explain	For each hit, contain an explanation of how scoring of the hits was computed.
_source	Set to false to disable retrieval of the _source field. You can also retrieve part of the document by using _source_include & _source_exclude (see the request body documentation for more details)
fields	The selective stored fields of the document to return for each hit, comma delimited. Not specifying any value will cause no fields to return.
sort	Sorting to perform. Can either be in the form of fieldName , or fieldName:asc / fieldName:desc . The fieldName can either be an actual field within the document, or the special _score name to indicate sorting based on scores. There can be several sort parameters (order is important).
track_scores	When sorting, set to true in order to still track scores and return them as part of each hit.
timeout	A search timeout, bounding the search request to be executed within the specified time value and bail with the hits accumulated up to that point when expired. Defaults to no timeout.
terminate_after	[ experimental ] This functionality is experimental and may be changed or removed completely in a future release. Elastic will take a best effort approach to fix any issues, but experimental features are not subject to the support SLA of official GA features. The maximum number of documents to collect for each shard, upon reaching which the query execution will terminate early. If set, the response will have a boolean field terminated_early to indicate whether the query execution has actually terminated_early. Defaults to no terminate_after.
from	The starting from index of the hits to return. Defaults to 0 .
size	The number of hits to return. Defaults to 10 .
search_type	The type of the search operation to perform. Can be dfs_query_then_fetch , dfs_query_and_fetch , query_then_fetch , query_and_fetch , count , scan . Defaults to query_then_fetch . See Search Type for more details on the different types of search that can be performed.

Query

Request Body Search

The search request can be executed with a search DSL, which includes the Query DSL , within its body. Here is an example:

$ curl -XGET 'http://localhost:9200/twitter/tweet/_search' -d '{
    "query" : {
        "term" : { "user" : "kimchy" }
'

And here is a sample response:

{
    "_shards":{
        "total" : 5,
        "successful" : 5,
        "failed" : 0
    "hits":{
        "total" : 1,
        "hits" : [
                "_index" : "twitter",
                "_type" : "tweet",
                "_id" : "1",
                "_source" : {
                    "user" : "kimchy",
                    "postDate" : "2009-11-15T14:12:12",
                    "message" : "trying out Elasticsearch"
}

Parameters

timeout A search timeout, bounding the search request to be executed within the specified time value and bail with the hits accumulated up to that point when expired. Defaults to no timeout. See the section called “Time units ”. The starting from index of the hits to return. Defaults to 0 . The number of hits to return. Defaults to 10 . search_type The type of the search operation to perform. Can be dfs_query_then_fetch , dfs_query_and_fetch , query_then_fetch , query_and_fetch . Defaults to query_then_fetch . See Search Type for more. query_cache Set to true or false to enable or disable the caching of search results for requests where ?search_type=count , ie aggregations and suggestions. See Shard query cache . terminate_after [ experimental ] This functionality is experimental and may be changed or removed completely in a future release. Elastic will take a best effort approach to fix any issues, but experimental features are not subject to the support SLA of official GA features. The maximum number of documents to collect for each shard, upon reaching which the query execution will terminate early. If set, the response will have a boolean field terminated_early to indicate whether the query execution has actually terminated_early. Defaults to no terminate_after.

Out of the above, the search_type and the query_cache must be passed as query-string parameters. The rest of the search request should be passed within the body itself. The body content can also be passed as a REST parameter named source .

Both HTTP GET and HTTP POST can be used to execute search with body. Since not all clients support GET with body, POST is allowed as well.

Query

The query element within the search request body allows to define a query using the Query DSL .

{
    "query" : {
        "term" : { "user" : "kimchy" }
            

From / Size

Pagination of results can be done by using the from and size parameters. The from parameter defines the offset from the first result you want to fetch. The size parameter allows you to configure the maximum amount of hits to be returned.

Though from and size can be set as request parameters, they can also be set within the search body. from defaults to 0 , and size defaults to 10 .

{
    "from" : 0, "size" : 10,
    "query" : {
        "term" : { "user" : "kimchy" }
          Source filtering
            

Sort

Allows to add one or more sort on specific fields. Each sort can be reversed as well. The sort is defined on a per field level, with special field name for _score to sort by score.

{
    "sort" : [
        { "post_date" : {"order" : "asc"}},
        "user",
        { "name" : "desc" },
        { "age" : "desc" },
        "_score"
    "query" : {
        "term" : { "user" : "kimchy" }
}

Sort Values

The sort values for each document returned are also returned as part of the response.

Sort Order

The order option can have the following values:

Sort in ascending order Sort in descending order

The order defaults to desc when sorting on the _score , and defaults to asc when sorting on anything else.

Sort mode option

Elasticsearch supports sorting by array or multi-valued fields. The mode option controls what array value is picked for sorting the document it belongs to. The mode option can have the following values:

Pick the lowest value. Pick the highest value. Use the sum of all values as sort value. Only applicable for number based array fields. Use the average of all values as sort value. Only applicable for number based array fields.

Sort mode example usage

In the example below the field price has multiple prices per document. In this case the result hits will be sort by price ascending based on the average price per document.

curl -XPOST 'localhost:9200/_search' -d '{
   "query" : {
   "sort" : [
      {"price" : {"order" : "asc", "mode" : "avg"}}
}'

Sorting within nested objects.

Elasticsearch also supports sorting by fields that are inside one or more nested objects. The sorting by nested field support has the following parameters on top of the already existing sort options:

nested_path Defines the on what nested object to sort. The actual sort field must be a direct field inside this nested object. The default is to use the most immediate inherited nested object from the sort field. nested_filter A filter the inner objects inside the nested path should match with in order for its field values to be taken into account by sorting. Common case is to repeat the query / filter inside the nested filter or query. By default no nested_filter is active.

Nested sorting example

In the below example offer is a field of type nested . Because offer is the closest inherited nested field, it is picked as nested_path . Only the inner objects that have color blue will participate in sorting.

curl -XPOST 'localhost:9200/_search' -d '{
   "query" : {
   "sort" : [
          "offer.price" : {
             "mode" :  "avg",
             "order" : "asc",
             "nested_filter" : {
                "term" : { "offer.color" : "blue" }
}'

Nested sorting is also supported when sorting by scripts and sorting by geo distance.

Missing Values

The missing parameter specifies how docs which are missing the field should be treated: The missing value can be set to _last , _first , or a custom value (that will be used for missing docs as the sort value). For example:

{
    "sort" : [
        { "price" : {"missing" : "_last"} },
    "query" : {
        "term" : { "user" : "kimchy" }
}

If a nested inner object doesn’t match with the nested_filter then a missing value is used.

Ignoring Unmapped Fields

Before 1.4.0 there was the ignore_unmapped boolean parameter, which was not enough information to decide on the sort values to emit, and didn’t work for cross-index search. It is still supported but users are encouraged to migrate to the new unmapped_type instead.

By default, the search request will fail if there is no mapping associated with a field. The unmapped_type option allows to ignore fields that have no mapping and not sort by them. The value of this parameter is used to determine what sort values to emit. Here is an example of how it can be used:

{
    "sort" : [
        { "price" : {"unmapped_type" : "long"} },
    "query" : {
        "term" : { "user" : "kimchy" }
}

If any of the indices that are queried doesn’t have a mapping for price then Elasticsearch will handle it as if there was a mapping of type long , with all documents in this index having no value for this field.

Geo Distance Sorting

Allow to sort by _geo_distance . Here is an example:

{
    "sort" : [
            "_geo_distance" : {
                "pin.location" : [-70, 40],
                "order" : "asc",
                "unit" : "km",
                "mode" : "min",
                "distance_type" : "sloppy_arc"
    "query" : {
        "term" : { "user" : "kimchy" }
distance_type
    How to compute the distance. Can either be sloppy_arc (default), arc (slightly more precise but significantly slower) or plane (faster, but inaccurate on long distances and close to the poles).

Note: the geo distance sorting supports sort_mode options: min , max and avg .

The following formats are supported in providing the coordinates:

Lat Lon as Properties

{
    "sort" : [
            "_geo_distance" : {
                "pin.location" : {
                    "lat" : 40,
                    "lon" : -70
                "order" : "asc",
                "unit" : "km"
    "query" : {
        "term" : { "user" : "kimchy" }
}

Lat Lon as String

Format in lat,lon .

{
    "sort" : [
            "_geo_distance" : {
                "pin.location" : "-70,40",
                "order" : "asc",
                "unit" : "km"
    "query" : {
        "term" : { "user" : "kimchy" }
}

Geohash

{
    "sort" : [
            "_geo_distance" : {
                "pin.location" : "drm3btev3e86",
                "order" : "asc",
                "unit" : "km"
    "query" : {
        "term" : { "user" : "kimchy" }
}

Lat Lon as Array

Format in [lon, lat] , note, the order of lon/lat here in order to conform with GeoJSON .

{
    "sort" : [
            "_geo_distance" : {
                "pin.location" : [-70, 40],
                "order" : "asc",
                "unit" : "km"
    "query" : {
        "term" : { "user" : "kimchy" }
}

Multiple reference points

Multiple geo points can be passed as an array containing any geo_point format, for example

"pin.location" : [[-70, 40], [-71, 42]]
"pin.location" : [{"lat": -70, "lon": 40}, {"lat": -71, "lon": 42}]

and so forth.

The final distance for a document will then be min / max / avg (defined via mode ) distance of all points contained in the document to all points given in the sort request.

Script Based Sorting

Allow to sort based on custom scripts, here is an example:

{
    "query" : {
    "sort" : {
        "_script" : {
            "script" : "doc['field_name'].value * factor",
            "type" : "number",
            "params" : {
                "factor" : 1.1
            "order" : "asc"
}

Track Scores

When sorting on a field, scores are not computed. By setting track_scores to true, scores will still be computed and tracked.

{
    "track_scores": true,
    "sort" : [
        { "post_date" : {"reverse" : true} },
        { "name" : "desc" },
        { "age" : "desc" }
    "query" : {
        "term" : { "user" : "kimchy" }
}

Memory Considerations

When sorting, the relevant sorted field values are loaded into memory. This means that per shard, there should be enough memory to contain them. For string based types, the field sorted on should not be analyzed / tokenized. For numeric types, if possible, it is recommended to explicitly set the type to narrower types (like short , integer and float ).

Fields

Source filtering

Allows to control how the _source field is returned with every hit.

By default operations return the contents of the _source field unless you have used the fields parameter or if the _source field is disabled.

You can turn off _source retrieval by using the _source parameter:

To disable _source retrieval set to false :

{
    "_source": false,
    "query" : {
        "term" : { "user" : "kimchy" }
}

The _source also accepts one or more wildcard patterns to control what parts of the _source should be returned:

For example:

{
    "_source": "obj.*",
    "query" : {
        "term" : { "user" : "kimchy" }
}

Or

{
    "_source": [ "obj1.*", "obj2.*" ],
    "query" : {
        "term" : { "user" : "kimchy" }
}

Finally, for complete control, you can specify both include and exclude patterns:

{
    "_source": {
        "include": [ "obj1.*", "obj2.*" ],
        "exclude": [ "*.description" ]
    "query" : {
        "term" : { "user" : "kimchy" }
          Script Fields
            

Fields

Allows to selectively load specific stored fields for each document represented by a search hit.

{
    "fields" : ["user", "postDate"],
    "query" : {
        "term" : { "user" : "kimchy" }
}

* can be used to load all stored fields from the document.

An empty array will cause only the _id and _type for each hit to be returned, for example:

{
    "fields" : [],
    "query" : {
        "term" : { "user" : "kimchy" }
}

For backwards compatibility, if the fields parameter specifies fields which are not stored ( store mapping set to false ), it will load the _source and extract it from it. This functionality has been replaced by the source filtering parameter.

Field values fetched from the document it self are always returned as an array. Metadata fields like _routing and _parent fields are never returned as an array.

Also only leaf fields can be returned via the field option. So object fields can’t be returned and such requests will fail.

Script fields can also be automatically detected and used as fields, so things like _source.obj1.field1 can be used, though not recommended, as obj1.field1 will work as well.

Partial

When loading data from _source , partial fields can be used to use wildcards to control what part of the _source will be loaded based on include and exclude patterns. For example:

{
    "query" : {
        "match_all" : {}
    "partial_fields" : {
        "partial1" : {
            "include" : "obj1.obj2.*"
}

And one that will also exclude obj1.obj3 :

{
    "query" : {
        "match_all" : {}
    "partial_fields" : {
        "partial1" : {
            "include" : "obj1.obj2.*",
            "exclude" : "obj1.obj3.*"
}

Both include and exclude support multiple patterns:

{
    "query" : {
        "match_all" : {}
    "partial_fields" : {
        "partial1" : {
            "include" : ["obj1.obj2.*", "obj1.obj4.*"],
            "exclude" : "obj1.obj3.*"
          Field Data Fields
            

Script Fields

Allows to return a script evaluation (based on different fields) for each hit, for example:

{
    "query" : {
    "script_fields" : {
        "test1" : {
            "script" : "doc['my_field_name'].value * 2"
        "test2" : {
            "script" : "doc['my_field_name'].value * factor",
            "params" : {
                "factor"  : 2.0
}

Script fields can work on fields that are not stored ( my_field_name in the above case), and allow to return custom values to be returned (the evaluated value of the script).

Script fields can also access the actual _source document indexed and extract specific elements to be returned from it (can be an "object" type). Here is an example:

    {
        "query" : {
        "script_fields" : {
            "test1" : {
                "script" : "_source.obj1.obj2"
    }

Note the _source keyword here to navigate the json-like model.

It’s important to understand the difference between doc['my_field'].value and _source.my_field . The first, using the doc keyword, will cause the terms for that field to be loaded to memory (cached), which will result in faster execution, but more memory consumption. Also, the doc[...] notation only allows for simple valued fields (can’t return a json object from it) and make sense only on non-analyzed or single term based fields.

The _source on the other hand causes the source to be loaded, parsed, and then only the relevant part of the json is returned.

Post filter

Field Data Fields

Allows to return the field data representation of a field for each hit, for example:

{
    "query" : {
    "fielddata_fields" : ["test1", "test2"]
}

Field data fields can work on fields that are not stored.

It’s important to understand that using the fielddata_fields parameter will cause the terms for that field to be loaded to memory (cached), which will result in more memory consumption.

Highlighting

Post filter

The post_filter is applied to the search hits at the very end of a search request, after aggregations have already been calculated. It’s purpose is best explained by example:

Imagine that you are selling shirts, and the user has specified two filters: color:red and brand:gucci . You only want to show them red shirts made by Gucci in the search results. Normally you would do this with a filtered query :

curl -XGET localhost:9200/shirts/_search -d '
  "query": {
    "filtered": {
      "filter": {
        "bool": {
          "must": [
            { "term": { "color": "red"   }},
            { "term": { "brand": "gucci" }}
'

However, you would also like to use faceted navigation to display a list of other options that the user could click on. Perhaps you have a model field that would allow the user to limit their search results to red Gucci t-shirts or dress-shirts .

This can be done with a terms aggregation :

curl -XGET localhost:9200/shirts/_search -d '
  "query": {
    "filtered": {
      "filter": {
        "bool": {
          "must": [
            { "term": { "color": "red"   }},
            { "term": { "brand": "gucci" }}
  "aggs": {
    "models": {
      "terms": { "field": "model" } 
Returns the most popular models of red shirts by Gucci.

But perhaps you would also like to tell the user how many Gucci shirts are available in other colors . If you just add a terms aggregation on the color field, you will only get back the color red , because your query returns only red shirts by Gucci.

Instead, you want to include shirts of all colors during aggregation, then apply the colors filter only to the search results. This is the purpose of the post_filter :

curl -XGET localhost:9200/shirts/_search -d '
  "query": {
    "filtered": {
      "filter": {
        { "term": { "brand": "gucci" }} 
  "aggs": {
    "colors": {
      "terms": { "field": "color" }, 
    "color_red": {
      "filter": {
        "term": { "color": "red" } 
      "aggs": {
        "models": {
          "terms": { "field": "model" } 
  "post_filter": { 
    "term": { "color": "red" }
The main query now finds all shirts by Gucci, regardless of color.
The colors agg returns popular colors for shirts by Gucci.
The color_red agg limits the models sub-aggregation
    to red Gucci shirts.
Finally, the post_filter removes colors other than red
    from the search hits.
            

Highlighting

Allows to highlight search results on one or more fields. The implementation uses either the lucene highlighter , fast-vector-highlighter or postings-highlighter . The following is an example of the search request body:

{
    "query" : {...},
    "highlight" : {
        "fields" : {
            "content" : {}
}

In the above case, the content field will be highlighted for each search hit (there will be another element in each search hit, called highlight , which includes the highlighted fields and the highlighted fragments).

In order to perform highlighting, the actual content of the field is required. If the field in question is stored (has store set to true in the mapping) it will be used, otherwise, the actual _source will be loaded and the relevant field will be extracted from it.

The _all field cannot be extracted from _source , so it can only be used for highlighting if it mapped to have store set to true .

The field name supports wildcard notation. For example, using comment_* will cause all fields that match the expression to be highlighted.

Postings highlighter

If index_options is set to offsets in the mapping the postings highlighter will be used instead of the plain highlighter. The postings highlighter:

• Is faster since it doesn’t require to reanalyze the text to be highlighted: the larger the documents the better the performance gain should be Requires less disk space than term_vectors, needed for the fast vector highlighter Breaks the text into sentences and highlights them. Plays really well with natural languages, not as well with fields containing for instance html markup Treats the document as the whole corpus, and scores individual sentences as if they were documents in this corpus, using the BM25 algorithm

Here is an example of setting the content field to allow for highlighting using the postings highlighter on it:

{
    "type_name" : {
        "content" : {"index_options" : "offsets"}
}

Note that the postings highlighter is meant to perform simple query terms highlighting, regardless of their positions. That means that when used for instance in combination with a phrase query, it will highlight all the terms that the query is composed of, regardless of whether they are actually part of a query match, effectively ignoring their positions.

The postings highlighter does support highlighting of multi term queries, like prefix queries, wildcard queries and so on. On the other hand, this requires the queries to be rewritten using a proper rewrite method that supports multi term extraction, which is a potentially expensive operation.

Fast vector highlighter

If term_vector information is provided by setting term_vector to with_positions_offsets in the mapping then the fast vector highlighter will be used instead of the plain highlighter. The fast vector highlighter:

• Is faster especially for large fields (> 1MB ) Can be customized with boundary_chars , boundary_max_scan , and fragment_offset (see below ) Requires setting term_vector to with_positions_offsets which increases the size of the index Can combine matches from multiple fields into one result. See matched_fields Can assign different weights to matches at different positions allowing for things like phrase matches being sorted above term matches when highlighting a Boosting Query that boosts phrase matches over term matches

Here is an example of setting the content field to allow for highlighting using the fast vector highlighter on it (this will cause the index to be bigger):

{
    "type_name" : {
        "content" : {"term_vector" : "with_positions_offsets"}
}

Force highlighter type

The type field allows to force a specific highlighter type. This is useful for instance when needing to use the plain highlighter on a field that has term_vectors enabled. The allowed values are: plain , postings and fvh . The following is an example that forces the use of the plain highlighter:

{
    "query" : {...},
    "highlight" : {
        "fields" : {
            "content" : {"type" : "plain"}
}

Force highlighting on source

Forces the highlighting to highlight fields based on the source even if fields are stored separately. Defaults to false .

{
    "query" : {...},
    "highlight" : {
        "fields" : {
            "content" : {"force_source" : true}
}

Highlighting Tags

By default, the highlighting will wrap highlighted text in <em> and </em> . This can be controlled by setting pre_tags and post_tags , for example:

{
    "query" : {...},
    "highlight" : {
        "pre_tags" : ["<tag1>"],
        "post_tags" : ["</tag1>"],
        "fields" : {
            "_all" : {}
}

Using the fast vector highlighter there can be more tags, and the "importance" is ordered.

{
    "query" : {...},
    "highlight" : {
        "pre_tags" : ["<tag1>", "<tag2>"],
        "post_tags" : ["</tag1>", "</tag2>"],
        "fields" : {
            "_all" : {}
}

There are also built in "tag" schemas, with currently a single schema called styled with the following pre_tags :

<em class="hlt1">, <em class="hlt2">, <em class="hlt3">,
<em class="hlt4">, <em class="hlt5">, <em class="hlt6">,
<em class="hlt7">, <em class="hlt8">, <em class="hlt9">,
<em class="hlt10">

and </em> as post_tags . If you think of more nice to have built in tag schemas, just send an email to the mailing list or open an issue. Here is an example of switching tag schemas:

{
    "query" : {...},
    "highlight" : {
        "tags_schema" : "styled",
        "fields" : {
            "content" : {}
}

Encoder

An encoder parameter can be used to define how highlighted text will be encoded. It can be either default (no encoding) or html (will escape html, if you use html highlighting tags).

Highlighted Fragments

Each field highlighted can control the size of the highlighted fragment in characters (defaults to 100 ), and the maximum number of fragments to return (defaults to 5 ). For example:

{
    "query" : {...},
    "highlight" : {
        "fields" : {
            "content" : {"fragment_size" : 150, "number_of_fragments" : 3}
}

The fragment_size is ignored when using the postings highlighter, as it outputs sentences regardless of their length.

On top of this it is possible to specify that highlighted fragments need to be sorted by score:

{
    "query" : {...},
    "highlight" : {
        "order" : "score",
        "fields" : {
            "content" : {"fragment_size" : 150, "number_of_fragments" : 3}
}

If the number_of_fragments value is set to 0 then no fragments are produced, instead the whole content of the field is returned, and of course it is highlighted. This can be very handy if short texts (like document title or address) need to be highlighted but no fragmentation is required. Note that fragment_size is ignored in this case.

{
    "query" : {...},
    "highlight" : {
        "fields" : {
            "_all" : {},
            "bio.title" : {"number_of_fragments" : 0}
}

When using fast-vector-highlighter one can use fragment_offset parameter to control the margin to start highlighting from.

In the case where there is no matching fragment to highlight, the default is to not return anything. Instead, we can return a snippet of text from the beginning of the field by setting no_match_size (default 0 ) to the length of the text that you want returned. The actual length may be shorter than specified as it tries to break on a word boundary. When using the postings highlighter it is not possible to control the actual size of the snippet, therefore the first sentence gets returned whenever no_match_size is greater than 0 .

{
    "query" : {...},
    "highlight" : {
        "fields" : {
            "content" : {
                "fragment_size" : 150,
                "number_of_fragments" : 3,
                "no_match_size": 150
}

Highlight query

It is also possible to highlight against a query other than the search query by setting highlight_query . This is especially useful if you use a rescore query because those are not taken into account by highlighting by default. Elasticsearch does not validate that highlight_query contains the search query in any way so it is possible to define it so legitimate query results aren’t highlighted at all. Generally it is better to include the search query in the highlight_query . Here is an example of including both the search query and the rescore query in highlight_query .

{
    "fields": [ "_id" ],
    "query" : {
        "match": {
            "content": {
                "query": "foo bar"
    "rescore": {
        "window_size": 50,
        "query": {
            "rescore_query" : {
                "match_phrase": {
                    "content": {
                        "query": "foo bar",
                        "phrase_slop": 1
            "rescore_query_weight" : 10
    "highlight" : {
        "order" : "score",
        "fields" : {
            "content" : {
                "fragment_size" : 150,
                "number_of_fragments" : 3,
                "highlight_query": {
                    "bool": {
                        "must": {
                            "match": {
                                "content": {
                                    "query": "foo bar"
                        "should": {
                            "match_phrase": {
                                "content": {
                                    "query": "foo bar",
                                    "phrase_slop": 1,
                                    "boost": 10.0
                        "minimum_should_match": 0
}

Note that the score of text fragment in this case is calculated by the Lucene highlighting framework. For implementation details you can check the ScoreOrderFragmentsBuilder.java class. On the other hand when using the postings highlighter the fragments are scored using, as mentioned above, the BM25 algorithm.

Global Settings

Highlighting settings can be set on a global level and then overridden at the field level.

{
    "query" : {...},
    "highlight" : {
        "number_of_fragments" : 3,
        "fragment_size" : 150,
        "tag_schema" : "styled",
        "fields" : {
            "_all" : { "pre_tags" : ["<em>"], "post_tags" : ["</em>"] },
            "bio.title" : { "number_of_fragments" : 0 },
            "bio.author" : { "number_of_fragments" : 0 },
            "bio.content" : { "number_of_fragments" : 5, "order" : "score" }
}

Require Field Match

require_field_match can be set to true which will cause a field to be highlighted only if a query matched that field. false means that terms are highlighted on all requested fields regardless if the query matches specifically on them.

Boundary Characters

When highlighting a field using the fast vector highlighter, boundary_chars can be configured to define what constitutes a boundary for highlighting. It’s a single string with each boundary character defined in it. It defaults to .,!? \t\n .

The boundary_max_scan allows to control how far to look for boundary characters, and defaults to 20 .

Matched Fields

The Fast Vector Highlighter can combine matches on multiple fields to highlight a single field using matched_fields . This is most intuitive for multifields that analyze the same string in different ways. All matched_fields must have term_vector set to with_positions_offsets but only the field to which the matches are combined is loaded so only that field would benefit from having store set to yes .

In the following examples content is analyzed by the english analyzer and content.plain is analyzed by the standard analyzer.

{
    "query": {
        "query_string": {
            "query": "content.plain:running scissors",
            "fields": ["content"]
    "highlight": {
        "order": "score",
        "fields": {
            "content": {
                "matched_fields": ["content", "content.plain"],
                "type" : "fvh"
}

The above matches both "run with scissors" and "running with scissors" and would highlight "running" and "scissors" but not "run". If both phrases appear in a large document then "running with scissors" is sorted above "run with scissors" in the fragments list because there are more matches in that fragment.

{
    "query": {
        "query_string": {
            "query": "running scissors",
            "fields": ["content", "content.plain^10"]
    "highlight": {
        "order": "score",
        "fields": {
            "content": {
                "matched_fields": ["content", "content.plain"],
                "type" : "fvh"
}

The above highlights "run" as well as "running" and "scissors" but still sorts "running with scissors" above "run with scissors" because the plain match ("running") is boosted.

{
    "query": {
        "query_string": {
            "query": "running scissors",
            "fields": ["content", "content.plain^10"]
    "highlight": {
        "order": "score",
        "fields": {
            "content": {
                "matched_fields": ["content.plain"],
                "type" : "fvh"
}

The above query wouldn’t highlight "run" or "scissor" but shows that it is just fine not to list the field to which the matches are combined ( content ) in the matched fields.

Technically it is also fine to add fields to matched_fields that don’t share the same underlying string as the field to which the matches are combined. The results might not make much sense and if one of the matches is off the end of the text then the whole query will fail.

There is a small amount of overhead involved with setting matched_fields to a non-empty array so always prefer

    "highlight": {
        "fields": {
            "content": {}
    }

to

    "highlight": {
        "fields": {
            "content": {
                "matched_fields": ["content"],
                "type" : "fvh"
    }

Phrase Limit

The fast-vector-highlighter has a phrase_limit parameter that prevents it from analyzing too many phrases and eating tons of memory. It defaults to 256 so only the first 256 matching phrases in the document scored considered. You can raise the limit with the phrase_limit parameter but keep in mind that scoring more phrases consumes more time and memory.

If using matched_fields keep in mind that phrase_limit phrases per matched field are considered.

Field Highlight Order

Elasticsearch highlights the fields in the order that they are sent. Per the json spec objects are unordered but if you need to be explicit about the order that fields are highlighted then you can use an array for fields like this:

    "highlight": {
        "fields": [
            {"title":{ /*params*/ }},
            {"text":{ /*params*/ }}
    }

None of the highlighters built into Elasticsearch care about the order that the fields are highlighted but a plugin may.

Search Type

Rescoring

Rescoring can help to improve precision by reordering just the top (eg 100 - 500) documents returned by the query and post_filter phases, using a secondary (usually more costly) algorithm, instead of applying the costly algorithm to all documents in the index.

A rescore request is executed on each shard before it returns its results to be sorted by the node handling the overall search request.

Currently the rescore API has only one implementation: the query rescorer, which uses a query to tweak the scoring. In the future, alternative rescorers may be made available, for example, a pair-wise rescorer.

the rescore phase is not executed when search_type is set to scan or count .

when exposing pagination to your users, you should not change window_size as you step through each page (by passing different from values) since that can alter the top hits causing results to confusingly shift as the user steps through pages.

Query rescorer

The query rescorer executes a second query only on the Top-K results returned by the query and post_filter phases. The number of docs which will be examined on each shard can be controlled by the window_size parameter, which defaults to from and size .

By default the scores from the original query and the rescore query are combined linearly to produce the final _score for each document. The relative importance of the original query and of the rescore query can be controlled with the query_weight and rescore_query_weight respectively. Both default to 1 .

For example:

curl -s -XPOST 'localhost:9200/_search' -d '{
   "query" : {
      "match" : {
         "field1" : {
            "operator" : "or",
            "query" : "the quick brown",
            "type" : "boolean"
   "rescore" : {
      "window_size" : 50,
      "query" : {
         "rescore_query" : {
            "match" : {
               "field1" : {
                  "query" : "the quick brown",
                  "type" : "phrase",
                  "slop" : 2
         "query_weight" : 0.7,
         "rescore_query_weight" : 1.2
'

The way the scores are combined can be controlled with the score_mode :

Score Mode	Description
total	Add the original score and the rescore query score. The default.
multiply	Multiply the original score by the rescore query score. Useful for function query rescores.
avg	Average the original score and the rescore query score.
max	Take the max of original score and the rescore query score.
min	Take the min of the original score and the rescore query score.

Multiple Rescores

It is also possible to execute multiple rescores in sequence:

curl -s -XPOST 'localhost:9200/_search' -d '{
   "query" : {
      "match" : {
         "field1" : {
            "operator" : "or",
            "query" : "the quick brown",
            "type" : "boolean"
   "rescore" : [ {
      "window_size" : 100,
      "query" : {
         "rescore_query" : {
            "match" : {
               "field1" : {
                  "query" : "the quick brown",
                  "type" : "phrase",
                  "slop" : 2
         "query_weight" : 0.7,
         "rescore_query_weight" : 1.2
      "window_size" : 10,
      "query" : {
         "score_mode": "multiply",
         "rescore_query" : {
            "function_score" : {
               "script_score": {
                  "script": "log10(doc['numeric'].value + 2)"
'

The first one gets the results of the query then the second one gets the results of the first, etc. The second rescore will "see" the sorting done by the first rescore so it is possible to use a large window on the first rescore to pull documents into a smaller window for the second rescore.

Search Type

There are different execution paths that can be done when executing a distributed search. The distributed search operation needs to be scattered to all the relevant shards and then all the results are gathered back. When doing scatter/gather type execution, there are several ways to do that, specifically with search engines.

One of the questions when executing a distributed search is how much results to retrieve from each shard. For example, if we have 10 shards, the 1st shard might hold the most relevant results from 0 till 10, with other shards results ranking below it. For this reason, when executing a request, we will need to get results from 0 till 10 from all shards, sort them, and then return the results if we want to ensure correct results.

Another question, which relates to the search engine, is the fact that each shard stands on its own. When a query is executed on a specific shard, it does not take into account term frequencies and other search engine information from the other shards. If we want to support accurate ranking, we would need to first gather the term frequencies from all shards to calculate global term frequencies, then execute the query on each shard using these global frequencies.

Also, because of the need to sort the results, getting back a large document set, or even scrolling it, while maintaining the correct sorting behavior can be a very expensive operation. For large result set scrolling without sorting, the scan search type (explained below) is also available.

Elasticsearch is very flexible and allows to control the type of search to execute on a per search request basis. The type can be configured by setting the search_type parameter in the query string. The types are:

Query Then Fetch

Parameter value: query_then_fetch .

The request is processed in two phases. In the first phase, the query is forwarded to all involved shards . Each shard executes the search and generates a sorted list of results, local to that shard. Each shard returns just enough information to the coordinating node to allow it merge and re-sort the shard level results into a globally sorted set of results, of maximum length size .

During the second phase, the coordinating node requests the document content (and highlighted snippets, if any) from only the relevant shards .

This is the default setting, if you do not specify a search_type in your request.

Dfs, Query Then Fetch

Parameter value: dfs_query_then_fetch .

Same as "Query Then Fetch", except for an initial scatter phase which goes and computes the distributed term frequencies for more accurate scoring.

Count

Parameter value: count .

A special search type that returns the count that matched the search request without any docs (represented in total_hits ), and possibly, including facets as well. In general, this is preferable to the count API as it provides more options.

Scan

Parameter value: scan .

The scan search type disables sorting in order to allow very efficient scrolling through large result sets. See Efficient scrolling with Scroll-Scan for more.

Query And Fetch

Parameter value: query_and_fetch .

The query_and_fetch mode is an internal optimization which is chosen automatically when a query_then_fetch request targets a single shard only. Both phases of query_then_fetch are executed in a single pass. This mode should not be explicitly specified by the user.

Dfs, Query And Fetch

Parameter value: dfs_query_and_fetch .

Same as query_and_fetch , except for an initial scatter phase which goes and computes the distributed term frequencies for more accurate scoring. This mode should not be explicitly specified by the user.

Scroll

While a search request returns a single “page” of results, the scroll API can be used to retrieve large numbers of results (or even all results) from a single search request, in much the same way as you would use a cursor on a traditional database.

Scrolling is not intended for real time user requests, but rather for processing large amounts of data, e.g. in order to reindex the contents of one index into a new index with a different configuration.

Client support for scrolling and reindexing

Some of the officially supported clients provide helpers to assist with scrolled searches and reindexing of documents from one index to another:

See Search::Elasticsearch::Bulk and Search::Elasticsearch::Scroll Python See elasticsearch.helpers.*

The results that are returned from a scroll request reflect the state of the index at the time that the initial search request was made, like a snapshot in time. Subsequent changes to documents (index, update or delete) will only affect later search requests.

In order to use scrolling, the initial search request should specify the scroll parameter in the query string, which tells Elasticsearch how long it should keep the “search context” alive (see Keeping the search context alive ), eg ?scroll=1m .

curl -XGET 'localhost:9200/twitter/tweet/_search?scroll=1m' -d '
    "query": {
        "match" : {
            "title" : "elasticsearch"
'

The result from the above request includes a _scroll_id , which should be passed to the scroll API in order to retrieve the next batch of results.

curl -XGET  'localhost:9200/_search/scroll?scroll=1m'   \
     -d       'c2Nhbjs2OzM0NDg1ODpzRlBLc0FXNlNyNm5JWUc1' 

GET or POST can be used. The URL should not include the index or type name — these are specified in the original search request instead. The scroll parameter tells Elasticsearch to keep the search context open for another 1m . The returned _scroll_id attribute can be passed in the request body or in the query string as ?scroll_id=....

Each call to the scroll API returns the next batch of results until there are no more results left to return, ie the hits array is empty.

The initial search request and each subsequent scroll request returns a new _scroll_id — only the most recent _scroll_id should be used.

If the request specifies aggregations, only the initial search response will contain the aggregations results.

Efficient scrolling with Scroll-Scan

Deep pagination with from and size — e.g. ?size=10&from=10000 — is very inefficient as (in this example) 100,000 sorted results have to be retrieved from each shard and resorted in order to return just 10 results. This process has to be repeated for every page requested.

The scroll API keeps track of which results have already been returned and so is able to return sorted results more efficiently than with deep pagination. However, sorting results (which happens by default) still has a cost.

Normally, you just want to retrieve all results and the order doesn’t matter. Scrolling can be combined with the scan search type to disable any scoring or sorting and to return results in the most efficient way possible. All that is needed is to add search_type=scan to the query string of the initial search request:

curl 'localhost:9200/twitter/tweet/_search?scroll=1m&search_type=scan'  -d '
    "query": {
        "match" : {
            "title" : "elasticsearch"
Setting search_type to scan disables sorting and makes scrolling
    very efficient.

A scanning scroll request differs from a standard scroll request in four ways:

• No score is calculated and sorting is disabled. Results are returned in the order they appear in the index. Aggregations are not supported. The response of the initial search request will not contain any results in the hits array. The first results will be returned by the first scroll request. The size parameter controls the number of results per shard , not per request, so a size of 10 which hits 5 shards will return a maximum of 50 results per scroll request.

If you want the scoring to happen, even without sorting on it, set the track_scores parameter to true .

Keeping the search context alive

The scroll parameter (passed to the search request and to every scroll request) tells Elasticsearch how long it should keep the search context alive. Its value (e.g. 1m , see the section called “Time units ”) does not need to be long enough to process all data — it just needs to be long enough to process the previous batch of results. Each scroll request (with the scroll parameter) sets a new expiry time.

Normally, the background merge process optimizes the index by merging together smaller segments to create new bigger segments, at which time the smaller segments are deleted. This process continues during scrolling, but an open search context prevents the old segments from being deleted while they are still in use. This is how Elasticsearch is able to return the results of the initial search request, regardless of subsequent changes to documents.

Keeping older segments alive means that more file handles are needed. Ensure that you have configured your nodes to have ample free file handles. See the section called “File Descriptors ”.

You can check how many search contexts are open with the nodes stats API :

curl -XGET localhost:9200/_nodes/stats/indices/search?pretty

Clear scroll API

Search contexts are removed automatically either when all results have been retrieved or when the scroll timeout has been exceeded. However, you can clear a search context manually with the clear-scroll API:

curl -XDELETE localhost:9200/_search/scroll \
     -d 'c2Nhbjs2OzM0NDg1ODpzRlBLc0FXNlNyNm5JWUc1' 

The scroll_id can be passed in the request body or in the query string.

Multiple scroll IDs can be passed as comma separated values:

curl -XDELETE localhost:9200/_search/scroll \
     -d 'c2Nhbjs2OzM0NDg1ODpzRlBLc0FXNlNyNm5JWUc1,aGVuRmV0Y2g7NTsxOnkxaDZ' 

All search contexts can be cleared with the _all parameter:

curl -XDELETE localhost:9200/_search/scroll/_all

Preference

Controls a preference of which shard replicas to execute the search request on. By default, the operation is randomized between the shard replicas.

The preference is a query string parameter which can be set to:

_primary The operation will go and be executed only on the primary shards. _primary_first The operation will go and be executed on the primary shard, and if not available (failover), will execute on other shards. _local The operation will prefer to be executed on a local allocated shard if possible. _only_node:xyz Restricts the search to execute only on a node with the provided node id ( xyz in this case). _prefer_node:xyz Prefers execution on the node with the provided node id ( xyz in this case) if applicable. _shards:2,3 Restricts the operation to the specified shards. ( 2 and 3 in this case). This preference can be combined with other preferences but it has to appear first: _shards:2,3;_primary _only_nodes Restricts the operation to nodes specified in node specification https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster.html [ 1.7.0 ] Added in 1.7.0. A custom value will be used to guarantee that the same shards will be used for the same custom value. This can help with "jumping values" when hitting different shards in different refresh states. A sample value can be something like the web session id, or the user name.

For instance, use the user’s session ID to ensure consistent ordering of results for the user:

curl localhost:9200/_search?preference=xyzabc123 -d '
    "query": {
        "match": {
            "title": "elasticsearch"
            

Explain

Enables explanation for each hit on how its score was computed.

{
    "explain": true,
    "query" : {
        "term" : { "user" : "kimchy" }
            

Index Boost

Allows to configure different boost level per index when searching across more than one indices. This is very handy when hits coming from one index matter more than hits coming from another index (think social graph where each user has an index).

{
    "indices_boost" : {
        "index1" : 1.4,
        "index2" : 1.3
          Named Queries and Filters
            

min_score

Exclude documents which have a _score less than the minimum specified in min_score :

{
    "min_score": 0.5,
    "query" : {
        "term" : { "user" : "kimchy" }
}

Note, most times, this does not make much sense, but is provided for advanced use cases.

Elasticsearch Reference [1.7] » Search APIs » Request Body Search » Named Queries and Filters

min_score Inner hits

Named Queries and Filters

Each filter and query can accept a _name in its top level definition.

{
    "filtered" : {
        "query" : {
            "bool" : {
                "should" : [
                    {"match" : { "name.first" : {"query" : "shay", "_name" : "first"} }},
                    {"match" : { "name.last" : {"query" : "banon", "_name" : "last"} }}
        "filter" : {
            "terms" : {
                "name.last" : ["banon", "kimchy"],
                "_name" : "test"
}

The search response will include for each hit the matched_queries it matched on. The tagging of queries and filters only make sense for compound queries and filters (such as bool query and filter, or and and filter, filtered query etc.).

Note, the query filter had to be enhanced in order to support this. In order to set a name, the fquery filter should be used, which wraps a query (just so there will be a place to set a name for it), for example:

{
    "filtered" : {
        "query" : {
            "term" : { "name.first" : "shay" }
        "filter" : {
            "fquery" : {
                "query" : {
                    "term" : { "name.last" : "banon" }
                "_name" : "test"
}

Named queries

The support for the _name option on queries is available from version 0.90.4 and the support on filters is available also in versions before 0.90.4 .

Search Template

Inner hits

This functionality is experimental and may be changed or removed completely in a future release. Elastic will take a best effort approach to fix any issues, but experimental features are not subject to the support SLA of official GA features.

The parent/child and nested features allow the return of documents that have matches in a different scope. In the parent/child case, parent document are returned based on matches in child documents or child document are returned based on matches in parent documents. In the nested case, documents are returned based on matches in nested inner objects.

In both cases, the actual matches in the different scopes that caused a document to be returned is hidden. In many cases, it’s very useful to know which inner nested objects (in the case of nested or children or parent documents), or (in the case of parent/child) caused certain information to be returned. The inner hits feature can be used for this. This feature returns per search hit in the search response additional nested hits that caused a search hit to match in a different scope.

Inner hits can be used by defining a inner_hits definition on a nested , has_child or has_parent query and filter. The structure looks like this:

"<query>" : {
    "inner_hits" : {
        <inner_hits_options>
}

If _inner_hits is defined on a query that supports it then each search hit will contain a inner_hits json object with the following structure:

"hits": [
        "_index": ...,
        "_type": ...,
        "_id": ...,
        "inner_hits": {
           "<inner_hits_name>": {
              "hits": {
                 "total": ...,
                 "hits": [
                       "_type": ...,
                       "_id": ...,
]

Options

Inner hits support the following options:

The offset from where the first hit to fetch for each inner_hits in the returned regular search hits. The maximum number of hits to return per inner_hits . By default the top three matching hits are returned. How the inner hits should be sorted per inner_hits . By default the hits are sorted by the score. The name to be used for the particular inner hit definition in the response. Useful when multiple inner hits have been defined in a single search request. The default depends in which query the inner hit is defined. For has_child query and filter this is the child type, has_parent query and filter this is the parent type and the nested query and filter this is the nested path.

Inner hits also supports the following per document features:

• Highlighting Explain Source filtering Script fields Fielddata fields Include versions

Nested inner hits

The nested inner_hits can be used to include nested inner objects as inner hits to a search hit.

The example below assumes that there is a nested object field defined with the name comments :

{
    "query" : {
        "nested" : {
            "path" : "comments",
            "query" : {
                "match" : {"comments.message" : "[actual query]"}
            "inner_hits" : {} 
The inner hit definition in the nested query. No other options need to be defined.

An example of a response snippet that could be generated from the above search request:

...
"hits": {
  "hits": [
        "_index": "my-index",
        "_type": "question",
        "_id": "1",
        "_source": ...,
        "inner_hits": {
           "comments": { 
              "hits": {
                 "total": ...,
                 "hits": [
                       "_type": "question",
                       "_id": "1",
                       "_nested": {
                          "field": "comments",
                          "offset": 2
                       "_source": ...
The name used in the inner hit definition in the search request. A custom key can be used via the name option.

The _nested metadata is crucial in the above example, because it defines from what inner nested object this inner hit came from. The field defines the object array field the nested hit is from and the offset relative to its location in the _source . Due to sorting and scoring the actual location of the hit objects in the inner_hits is usually different than the location a nested inner object was defined.

By default the _source is returned also for the hit objects in inner_hits , but this can be changed. Either via _source filtering feature part of the source can be returned or be disabled. If stored fields are defined on the nested level these can also be returned via the fields feature.

An important default is that the _source returned in hits inside inner_hits is relative to the _nested metadata. So in the above example only the comment part is returned per nested hit and not the entire source of the top level document that contained the the comment.

Hierarchical levels of nested object fields and inner hits.

If a mapping has multiple levels of hierarchical nested object fields each level can be accessed via dot notated path. For example if there is a comments nested field that contains a votes nested field and votes should directly be returned with the the root hits then the following path can be defined:

{
   "query" : {
      "nested" : {
         "path" : "comments.votes",
         "query" : { ... },
         "inner_hits" : {}
}

This indirect referencing is only supported for nested inner hits.

Parent/child inner hits

The parent/child inner_hits can be used to include parent or child

The examples below assumes that there is a _parent field mapping in the comment type:

{
    "query" : {
        "has_child" : {
            "type" : "comment",
            "query" : {
                "match" : {"message" : "[actual query]"}
            "inner_hits" : {} 
The inner hit definition like in the nested example.

An example of a response snippet that could be generated from the above search request:

...
"hits": {
  "hits": [
        "_index": "my-index",
        "_type": "question",
        "_id": "1",
        "_source": ...,
        "inner_hits": {
           "comment": {
              "hits": {
                 "total": ...,
                 "hits": [
                       "_type": "comment",
                       "_id": "5",
                       "_source": ...
     ...

top level inner hits

Besides defining inner hits on query and filters, inner hits can also be defined as a top level construct alongside the query and aggregations definition. The main reason for using the top level inner hits definition is to let the inner hits return documents that don’t match with the main query. Also inner hits definitions can be nested via the top level notation. Other then that the inner hit definition inside the query should be used, because that is the most compact way for defining inner hits.

The following snippet explains the basic structure of inner hits defined at the top level of the search request body:

"inner_hits" : {
    "<inner_hits_name>" : {
        "<path|type>" : {
            "<path-to-nested-object-field|child-or-parent-type>" : {
                <inner_hits_body>
                [,"inner_hits" : { [<sub_inner_hits>]+ } ]?
    [,"<inner_hits_name_2>" : { ... } ]*
}

Inside the inner_hits definition, first the name if the inner hit is defined then whether the inner_hit is a nested by defining path or a parent/child based definition by defining type . The next object layer contains the name of the nested object field if the inner_hits is nested or the parent or child type if the inner_hit definition is parent/child based.

Multiple inner hit definitions can be defined in a single request. In the <inner_hits_body> any option for features that inner_hits support can be defined. Optionally another inner_hits definition can be defined in the <inner_hits_body> .

An example that shows the use of nested inner hits via the top level notation:

{
    "query" : {
        "nested" : {
            "path" : "comments",
            "query" : {
                "match" : {"comments.message" : "[actual query]"}
    "inner_hits" : {
        "comment" : {
            "path" : { 
                "comments" : { 
                    "query" : {
                        "match" : {"comments.message" : "[different query]"} 
The inner hit definition is nested and requires the path option.
The path option refers to the nested object field comments
A query that runs to collect the nested inner documents for each search hit returned. If no query is defined all nested
    inner documents will be included belonging to a search hit. This shows that it only make sense to the top level
    inner hit definition if no query or a different query is specified.

Additional options that are only available when using the top level inner hits notation:

Defines the nested scope where hits will be collected from. Defines the parent or child type score where hits will be collected from. query Defines the query that will run in the defined nested, parent or child scope to collect and score hits. By default all document in the scope will be matched.

Either path or type must be defined. The path or type defines the scope from where hits are fetched and used as inner hits.

Search Shards API

Search Template

The /_search/template endpoint allows to use the mustache language to pre render search requests, before they are executed and fill existing templates with template parameters.

GET /_search/template
    "template" : {
      "query": { "match" : { "{{my_field}}" : "{{my_value}}" } },
      "size" : "{{my_size}}"
    "params" : {
        "my_field" : "foo",
        "my_value" : "bar",
        "my_size" : 5
}

For more information on how Mustache templating and what kind of templating you can do with it check out the online documentation of the mustache project .

The mustache language is implemented in elasticsearch as a sandboxed scripting language, hence it obeys settings that may be used to enable or disable scripts per language, source and operation as described in scripting docs [ 1.6.0 ] Added in 1.6.0. mustache scripts were always on before and it wasn’t possible to disable them .

More template examples

Filling in a query string with a single value

GET /_search/template
    "template": {
        "query": {
            "match": {
                "title": "{{query_string}}"
    "params": {
        "query_string": "search for these words"
}

Passing an array of strings

GET /_search/template
  "template": {
    "query": {
      "terms": {
        "status": [
          "{{#status}}",
          "{{.}}",
          "{{/status}}"
  "params": {
    "status": [ "pending", "published" ]
}

which is rendered as:

{
"query": {
  "terms": {
    "status": [ "pending", "published" ]
}

Default values

A default value is written as {{var}}{{^var}}default{{/var}} for instance:

{
  "template": {
    "query": {
      "range": {
        "line_no": {
          "gte": "{{start}}",
          "lte": "{{end}}{{^end}}20{{/end}}"
  "params": { ... }
}

When params is { "start": 10, "end": 15 } this query would be rendered as:

{
    "range": {
        "line_no": {
            "gte": "10",
            "lte": "15"
}

But when params is { "start": 10 } this query would use the default value for end :

{
    "range": {
        "line_no": {
            "gte": "10",
            "lte": "20"
}

Conditional clauses

Conditional clauses cannot be expressed using the JSON form of the template. Instead, the template must be passed as a string. For instance, let’s say we wanted to run a match query on the line field, and optionally wanted to filter by line numbers, where start and end are optional.

The params would look like:

{
    "params": {
        "text":      "words to search for",
        "line_no": { 
            "start": 10, 
            "end":   20  
All three of these elements are optional.

We could write the query as:

{
  "query": {
    "filtered": {
      "query": {
        "match": {
          "line": "{{text}}" 
      "filter": {
        {{#line_no}} 
          "range": {
            "line_no": {
              {{#start}} 
                "gte": "{{start}}" 
                {{#end}},{{/end}} 
              {{/start}} 
              {{#end}} 
                "lte": "{{end}}" 
              {{/end}} 
        {{/line_no}} 
Fill in the value of param text
Include the range filter only if line_no is specified
Include the gte clause only if line_no.start is specified
Fill in the value of param line_no.start
Add a comma after the gte clause only if line_no.start
    AND line_no.end are specified
Include the lte clause only if line_no.end is specified
Fill in the value of param line_no.end

As written above, this template is not valid JSON because it includes the section markers like {{#line_no}} . For this reason, the template should either be stored in a file (see the section called “Pre-registered template ”) or, when used via the REST API, should be written as a string:

"template": "{\"query\":{\"filtered\":{\"query\":{\"match\":{\"line\":\"{{text}}\"}},\"filter\":{{{#line_no}}\"range\":{\"line_no\":{{{#start}}\"gte\":\"{{start}}\"{{#end}},{{/end}}{{/start}}{{#end}}\"lte\":\"{{end}}\"{{/end}}}}{{/line_no}}}}}}"

Pre-registered template

You can register search templates by storing it in the config/scripts directory, in a file using the .mustache extension. In order to execute the stored template, reference it by it’s name under the template key:

GET /_search/template
    "template": {
        "file": "storedTemplate" 
    "params": {
        "query_string": "search for these words"
Name of the the query template in config/scripts/, i.e., storedTemplate.mustache.

You can also register search templates by storing it in the elasticsearch cluster in a special index named .scripts . There are REST APIs to manage these indexed templates.

POST /_search/template/<templatename>
    "template": {
        "query": {
            "match": {
                "title": "{{query_string}}"
}

This template can be retrieved by

GET /_search/template/<templatename>

which is rendered as:

{
    "template": {
        "query": {
            "match": {
                "title": "{{query_string}}"
}

This template can be deleted by

DELETE /_search/template/<templatename>

To use an indexed template at search time use:

GET /_search/template
    "template": {
        "id": "templateName" 
    "params": {
        "query_string": "search for these words"
Name of the the query template stored in the .scripts index.
          Aggregations
            

Search Shards API

The search shards api returns the indices and shards that a search request would be executed against. This can give useful feedback for working out issues or planning optimizations with routing and shard preferences.

The index and type parameters may be single values, or comma-separated.

Usage

Full example:

curl -XGET 'localhost:9200/twitter/_search_shards'

This will yield the following result:

{
  "nodes": {
    "JklnKbD7Tyqi9TP3_Q_tBg": {
      "name": "Rl'nnd",
      "transport_address": "inet[/192.168.1.113:9300]"
  "shards": [
        "index": "twitter",
        "node": "JklnKbD7Tyqi9TP3_Q_tBg",
        "primary": true,
        "relocating_node": null,
        "shard": 3,
        "state": "STARTED"
        "index": "twitter",
        "node": "JklnKbD7Tyqi9TP3_Q_tBg",
        "primary": true,
        "relocating_node": null,
        "shard": 4,
        "state": "STARTED"
        "index": "twitter",
        "node": "JklnKbD7Tyqi9TP3_Q_tBg",
        "primary": true,
        "relocating_node": null,
        "shard": 0,
        "state": "STARTED"
        "index": "twitter",
        "node": "JklnKbD7Tyqi9TP3_Q_tBg",
        "primary": true,
        "relocating_node": null,
        "shard": 2,
        "state": "STARTED"
        "index": "twitter",
        "node": "JklnKbD7Tyqi9TP3_Q_tBg",
        "primary": true,
        "relocating_node": null,
        "shard": 1,
        "state": "STARTED"
}

And specifying the same request, this time with a routing value:

curl -XGET 'localhost:9200/twitter/_search_shards?routing=foo,baz'

This will yield the following result:

{
  "nodes": {
    "JklnKbD7Tyqi9TP3_Q_tBg": {
      "name": "Rl'nnd",
      "transport_address": "inet[/192.168.1.113:9300]"
  "shards": [
        "index": "twitter",
        "node": "JklnKbD7Tyqi9TP3_Q_tBg",
        "primary": true,
        "relocating_node": null,
        "shard": 2,
        "state": "STARTED"
        "index": "twitter",
        "node": "JklnKbD7Tyqi9TP3_Q_tBg",
        "primary": true,
        "relocating_node": null,
        "shard": 4,
        "state": "STARTED"
}

This time the search will only be executed against two of the shards, because routing values have been specified.

All parameters:

routing A comma-separated list of routing values to take into account when determining which shards a request would be executed against. preference Controls a preference of which shard replicas to execute the search request on. By default, the operation is randomized between the shard replicas. See the preference documentation for a list of all acceptable values. local A boolean value whether to read the cluster state locally in order to determine where shards are allocated instead of using the Master node’s cluster state. Min Aggregation Aggregations Aggregations grew out of the facets module and the long experience of how users use it (and would like to use it) for real-time data analytics purposes. As such, it serves as the next generation replacement for the functionality we currently refer to as "faceting". Facets provide a great way to aggregate data within a document set context. This context is defined by the executed query in combination with the different levels of filters that can be defined (filtered queries, top-level filters, and facet level filters). While powerful, their implementation is not designed from the ground up to support complex aggregations and is thus limited. The aggregations module breaks the barriers the current facet implementation put in place. The new name ("Aggregations") also indicates the intention here - a generic yet extremely powerful framework for building aggregations - any types of aggregations. An aggregation can be seen as a unit-of-work that builds analytic information over a set of documents. The context of the execution defines what this document set is (e.g. a top-level aggregation executes within the context of the executed query/filters of the search request). There are many different types of aggregations, each with its own purpose and output. To better understand these types, it is often easier to break them into two main families: Bucketing A family of aggregations that build buckets, where each bucket is associated with a key and a document criterion. When the aggregation is executed, all the buckets criteria are evaluated on every document in the context and when a criterion matches, the document is considered to "fall in" the relevant bucket. By the end of the aggregation process, we’ll end up with a list of buckets - each one with a set of documents that "belong" to it. Metric Aggregations that keep track and compute metrics over a set of documents. The interesting part comes next. Since each bucket effectively defines a document set (all documents belonging to the bucket), one can potentially associate aggregations on the bucket level, and those will execute within the context of that bucket. This is where the real power of aggregations kicks in: aggregations can be nested! Bucketing aggregations can have sub-aggregations (bucketing or metric). The sub-aggregations will be computed for the buckets which their parent aggregation generates. There is no hard limit on the level/depth of nested aggregations (one can nest an aggregation under a "parent" aggregation, which is itself a sub-aggregation of another higher-level aggregation). Structuring Aggregations The following snippet captures the basic structure of aggregations: "aggregations" : { "<aggregation_name>" : { "<aggregation_type>" : { <aggregation_body> [,"aggregations" : { [<sub_aggregation>]+ } ]? [,"<aggregation_name_2>" : { ... } ]* } The aggregations object (the key aggs can also be used) in the JSON holds the aggregations to be computed. Each aggregation is associated with a logical name that the user defines (e.g. if the aggregation computes the average price, then it would make sense to name it avg_price ). These logical names will also be used to uniquely identify the aggregations in the response. Each aggregation has a specific type ( <aggregation_type> in the above snippet) and is typically the first key within the named aggregation body. Each type of aggregation defines its own body, depending on the nature of the aggregation (e.g. an avg aggregation on a specific field will define the field on which the average will be calculated). At the same level of the aggregation type definition, one can optionally define a set of additional aggregations, though this only makes sense if the aggregation you defined is of a bucketing nature. In this scenario, the sub-aggregations you define on the bucketing aggregation level will be computed for all the buckets built by the bucketing aggregation. For example, if you define a set of aggregations under the range aggregation, the sub-aggregations will be computed for the range buckets that are defined. Values Source Some aggregations work on values extracted from the aggregated documents. Typically, the values will be extracted from a specific document field which is set using the field key for the aggregations. It is also possible to define a script which will generate the values (per document). The script parameter expects an inline script. Use script_id for indexed scripts and script_file for scripts in the config/scripts/ directory. When both field and script settings are configured for the aggregation, the script will be treated as a value script . While normal scripts are evaluated on a document level (i.e. the script has access to all the data associated with the document), value scripts are evaluated on the value level. In this mode, the values are extracted from the configured field and the script is used to apply a "transformation" over these value/s. When working with scripts, the lang and params settings can also be defined. The former defines the scripting language which is used (assuming the proper language is available in Elasticsearch, either by default or as a plugin). The latter enables defining all the "dynamic" expressions in the script as parameters, which enables the script to keep itself static between calls (this will ensure the use of the cached compiled scripts in Elasticsearch). Scripts can generate a single value or multiple values per document. When generating multiple values, one can use the script_values_sorted settings to indicate whether these values are sorted or not. Internally, Elasticsearch can perform optimizations when dealing with sorted values (for example, with the min aggregations, knowing the values are sorted, Elasticsearch will skip the iterations over all the values and rely on the first value in the list to be the minimum value among all other values associated with the same document). Metrics Aggregations The aggregations in this family compute metrics based on values extracted in one way or another from the documents that are being aggregated. The values are typically extracted from the fields of the document (using the field data), but can also be generated using scripts. Numeric metrics aggregations are a special type of metrics aggregation which output numeric values. Some aggregations output a single numeric metric (e.g. avg ) and are called single-value numeric metrics aggregation , others generate multiple metrics (e.g. stats ) and are called multi-value numeric metrics aggregation . The distinction between single-value and multi-value numeric metrics aggregations plays a role when these aggregations serve as direct sub-aggregations of some bucket aggregations (some bucket aggregations enable you to sort the returned buckets based on the numeric metrics in each bucket). Bucket Aggregations Bucket aggregations don’t calculate metrics over fields like the metrics aggregations do, but instead, they create buckets of documents. Each bucket is associated with a criterion (depending on the aggregation type) which determines whether or not a document in the current context "falls" into it. In other words, the buckets effectively define document sets. In addition to the buckets themselves, the bucket aggregations also compute and return the number of documents that "fell in" to each bucket. Bucket aggregations, as opposed to metrics aggregations, can hold sub-aggregations. These sub-aggregations will be aggregated for the buckets created by their "parent" bucket aggregation. There are different bucket aggregators, each with a different "bucketing" strategy. Some define a single bucket, some define fixed number of multiple buckets, and others dynamically create the buckets during the aggregation process. Caching heavy aggregations Frequently used aggregations (e.g. for display on the home page of a website) can be cached for faster responses. These cached results are the same results that would be returned by an uncached aggregation — you will never get stale results. See Shard query cache for more details. Returning only aggregation results There are many occasions when aggregations are required but search hits are not. For these cases the hits can be ignored by adding search_type=count to the request URL parameters. For example: $ curl -XGET 'http://localhost:9200/twitter/tweet/_search?search_type=count' -d '{ "aggregations": { "my_agg": { "terms": { "field": "text" ' Setting search_type to count avoids executing the fetch phase of the search making the request more efficient. See Search Type for more information on the search_type parameter. Max Aggregation Min Aggregation A single-value metrics aggregation that keeps track and returns the minimum value among numeric values extracted from the aggregated documents. These values can be extracted either from specific numeric fields in the documents, or be generated by a provided script. Computing the min price value across all documents: { "aggs" : { "min_price" : { "min" : { "field" : "price" } } } Response: { "aggregations": { "min_price": { "value": 10 } As can be seen, the name of the aggregation ( min_price above) also serves as the key by which the aggregation result can be retrieved from the returned response. Script Computing the min price value across all document, this time using a script: { "aggs" : { "min_price" : { "min" : { "script" : "doc['price'].value" } } } The script parameter expects an inline script. Use script_id for indexed scripts and script_file for scripts in the config/scripts/ directory. Value Script Let’s say that the prices of the documents in our index are in USD, but we would like to compute the min in EURO (and for the sake of this example, lets say the conversion rate is 1.2). We can use a value script to apply the conversion rate to every value before it is aggregated: { "aggs" : { "min_price_in_euros" : { "min" : { "field" : "price", "script" : "_value * conversion_rate", "params" : { "conversion_rate" : 1.2 Sum Aggregation Max Aggregation A single-value metrics aggregation that keeps track and returns the maximum value among the numeric values extracted from the aggregated documents. These values can be extracted either from specific numeric fields in the documents, or be generated by a provided script. Computing the max price value across all documents { "aggs" : { "max_price" : { "max" : { "field" : "price" } } } Response: { "aggregations": { "max_price": { "value": 35 } As can be seen, the name of the aggregation ( max_price above) also serves as the key by which the aggregation result can be retrieved from the returned response. Script Computing the max price value across all document, this time using a script: { "aggs" : { "max_price" : { "max" : { "script" : "doc['price'].value" } } } The script parameter expects an inline script. Use script_id for indexed scripts and script_file for scripts in the config/scripts/ directory. Value Script Let’s say that the prices of the documents in our index are in USD, but we would like to compute the max in EURO (and for the sake of this example, lets say the conversion rate is 1.2). We can use a value script to apply the conversion rate to every value before it is aggregated: { "aggs" : { "max_price_in_euros" : { "max" : { "field" : "price", "script" : "_value * conversion_rate", "params" : { "conversion_rate" : 1.2 Avg Aggregation Sum Aggregation A single-value metrics aggregation that sums up numeric values that are extracted from the aggregated documents. These values can be extracted either from specific numeric fields in the documents, or be generated by a provided script. Assuming the data consists of documents representing stock ticks, where each tick holds the change in the stock price from the previous tick. { "query" : { "filtered" : { "query" : { "match_all" : {}}, "filter" : { "range" : { "timestamp" : { "from" : "now/1d+9.5h", "to" : "now/1d+16h" }} "aggs" : { "intraday_return" : { "sum" : { "field" : "change" } } } The above aggregation sums up all changes in the today’s trading stock ticks which accounts for the intraday return. The aggregation type is sum and the field setting defines the numeric field of the documents of which values will be summed up. The above will return the following: { "aggregations": { "intraday_return": { "value": 2.18 } The name of the aggregation ( intraday_return above) also serves as the key by which the aggregation result can be retrieved from the returned response. Script Computing the intraday return based on a script: { "aggs" : { "intraday_return" : { "sum" : { "script" : "doc['change'].value" } } } The script parameter expects an inline script. Use script_id for indexed scripts and script_file for scripts in the config/scripts/ directory. Value Script Computing the sum of squares over all stock tick changes: { "aggs" : { "aggs" : { "daytime_return" : { "sum" : { "field" : "change", "script" : "_value * _value" } Stats Aggregation Avg Aggregation A single-value metrics aggregation that computes the average of numeric values that are extracted from the aggregated documents. These values can be extracted either from specific numeric fields in the documents, or be generated by a provided script. Assuming the data consists of documents representing exams grades (between 0 and 100) of students { "aggs" : { "avg_grade" : { "avg" : { "field" : "grade" } } } The above aggregation computes the average grade over all documents. The aggregation type is avg and the field setting defines the numeric field of the documents the average will be computed on. The above will return the following: { "aggregations": { "avg_grade": { "value": 75 } The name of the aggregation ( avg_grade above) also serves as the key by which the aggregation result can be retrieved from the returned response. Script Computing the average grade based on a script: { "aggs" : { "avg_grade" : { "avg" : { "script" : "doc['grade'].value" } } } The script parameter expects an inline script. Use script_id for indexed scripts and script_file for scripts in the config/scripts/ directory. Value Script It turned out that the exam was way above the level of the students and a grade correction needs to be applied. We can use value script to get the new average: { "aggs" : { "aggs" : { "avg_corrected_grade" : { "avg" : { "field" : "grade", "script" : "_value * correction", "params" : { "correction" : 1.2 Extended Stats Aggregation Stats Aggregation A multi-value metrics aggregation that computes stats over numeric values extracted from the aggregated documents. These values can be extracted either from specific numeric fields in the documents, or be generated by a provided script. The stats that are returned consist of: min , max , sum , count and avg . Assuming the data consists of documents representing exams grades (between 0 and 100) of students { "aggs" : { "grades_stats" : { "stats" : { "field" : "grade" } } } The above aggregation computes the grades statistics over all documents. The aggregation type is stats and the field setting defines the numeric field of the documents the stats will be computed on. The above will return the following: { "aggregations": { "grades_stats": { "count": 6, "min": 60, "max": 98, "avg": 78.5, "sum": 471 } The name of the aggregation ( grades_stats above) also serves as the key by which the aggregation result can be retrieved from the returned response. Script Computing the grades stats based on a script: { "aggs" : { "grades_stats" : { "stats" : { "script" : "doc['grade'].value" } } } The script parameter expects an inline script. Use script_id for indexed scripts and script_file for scripts in the config/scripts/ directory. Value Script It turned out that the exam was way above the level of the students and a grade correction needs to be applied. We can use a value script to get the new stats: { "aggs" : { "aggs" : { "grades_stats" : { "stats" : { "field" : "grade", "script" : "_value * correction", "params" : { "correction" : 1.2 Value Count Aggregation Extended Stats Aggregation A multi-value metrics aggregation that computes stats over numeric values extracted from the aggregated documents. These values can be extracted either from specific numeric fields in the documents, or be generated by a provided script. The extended_stats aggregations is an extended version of the stats aggregation, where additional metrics are added such as sum_of_squares , variance , std_deviation and std_deviation_bounds . Assuming the data consists of documents representing exams grades (between 0 and 100) of students { "aggs" : { "grades_stats" : { "extended_stats" : { "field" : "grade" } } } The above aggregation computes the grades statistics over all documents. The aggregation type is extended_stats and the field setting defines the numeric field of the documents the stats will be computed on. The above will return the following: { "aggregations": { "grade_stats": { "count": 9, "min": 72, "max": 99, "avg": 86, "sum": 774, "sum_of_squares": 67028, "variance": 51.55555555555556, "std_deviation": 7.180219742846005, "std_deviation_bounds": { "upper": 100.36043948569201, "lower": 71.63956051430799 } The name of the aggregation ( grades_stats above) also serves as the key by which the aggregation result can be retrieved from the returned response. Standard Deviation Bounds By default, the extended_stats metric will return an object called std_deviation_bounds , which provides an interval of plus/minus two standard deviations from the mean. This can be a useful way to visualize variance of your data. If you want a different boundary, for example three standard deviations, you can set sigma in the request: { "aggs" : { "grades_stats" : { "extended_stats" : { "field" : "grade", "sigma" : 3 sigma controls how many standard deviations +/- from the mean should be displayed

sigma can be any non-negative double, meaning you can request non-integer values such as 1.5 . A value of 0 is valid, but will simply return the average for both upper and lower bounds.

Standard Deviation and Bounds require normality

The standard deviation and its bounds are displayed by default, but they are not always applicable to all data-sets. Your data must be normally distributed for the metrics to make sense. The statistics behind standard deviations assumes normally distributed data, so if your data is skewed heavily left or right, the value returned will be misleading.

Script

Computing the grades stats based on a script:

{
    "aggs" : {
        "grades_stats" : { "extended_stats" : { "script" : "doc['grade'].value" } }
}

The script parameter expects an inline script. Use script_id for indexed scripts and script_file for scripts in the config/scripts/ directory.

Value Script

It turned out that the exam was way above the level of the students and a grade correction needs to be applied. We can use value script to get the new stats:

{
    "aggs" : {
        "aggs" : {
            "grades_stats" : {
                "extended_stats" : {
                    "field" : "grade",
                    "script" : "_value * correction",
                    "params" : {
                        "correction" : 1.2
          Percentiles Aggregation
            

Value Count Aggregation

A single-value metrics aggregation that counts the number of values that are extracted from the aggregated documents. These values can be extracted either from specific fields in the documents, or be generated by a provided script. Typically, this aggregator will be used in conjunction with other single-value aggregations. For example, when computing the avg one might be interested in the number of values the average is computed over.

{
    "aggs" : {
        "grades_count" : { "value_count" : { "field" : "grade" } }
}

Response:

{
    "aggregations": {
        "grades_count": {
            "value": 10
}

The name of the aggregation ( grades_count above) also serves as the key by which the aggregation result can be retrieved from the returned response.

Script

Counting the values generated by a script:

{
    "aggs" : {
        "grades_count" : { "value_count" : { "script" : "doc['grade'].value" } }
}

The script parameter expects an inline script. Use script_id for indexed scripts and script_file for scripts in the config/scripts/ directory.

Percentile Ranks Aggregation

Percentiles Aggregation

A multi-value metrics aggregation that calculates one or more percentiles over numeric values extracted from the aggregated documents. These values can be extracted either from specific numeric fields in the documents, or be generated by a provided script.

This functionality is experimental and may be changed or removed completely in a future release. Elastic will take a best effort approach to fix any issues, but experimental features are not subject to the support SLA of official GA features.

Percentiles show the point at which a certain percentage of observed values occur. For example, the 95th percentile is the value which is greater than 95% of the observed values.

Percentiles are often used to find outliers. In normal distributions, the 0.13th and 99.87th percentiles represents three standard deviations from the mean. Any data which falls outside three standard deviations is often considered an anomaly.

When a range of percentiles are retrieved, they can be used to estimate the data distribution and determine if the data is skewed, bimodal, etc.

Assume your data consists of website load times. The average and median load times are not overly useful to an administrator. The max may be interesting, but it can be easily skewed by a single slow response.

Let’s look at a range of percentiles representing load time:

{
    "aggs" : {
        "load_time_outlier" : {
            "percentiles" : {
                "field" : "load_time" 
The field load_time must be a numeric field

By default, the percentile metric will generate a range of percentiles: [ 1, 5, 25, 50, 75, 95, 99 ] . The response will look like this:

{
   "aggregations": {
      "load_time_outlier": {
         "values" : {
            "1.0": 15,
            "5.0": 20,
            "25.0": 23,
            "50.0": 25,
            "75.0": 29,
            "95.0": 60,
            "99.0": 150
}

As you can see, the aggregation will return a calculated value for each percentile in the default range. If we assume response times are in milliseconds, it is immediately obvious that the webpage normally loads in 15-30ms, but occasionally spikes to 60-150ms.

Often, administrators are only interested in outliers — the extreme percentiles. We can specify just the percents we are interested in (requested percentiles must be a value between 0-100 inclusive):

{
    "aggs" : {
        "load_time_outlier" : {
            "percentiles" : {
                "field" : "load_time",
                "percents" : [95, 99, 99.9] 
Use the percents parameter to specify particular percentiles to calculate

Script

The percentile metric supports scripting. For example, if our load times are in milliseconds but we want percentiles calculated in seconds, we could use a script to convert them on-the-fly:

{
    "aggs" : {
        "load_time_outlier" : {
            "percentiles" : {
                "script" : "doc['load_time'].value / timeUnit", 
                "params" : {
                    "timeUnit" : 1000   
The field parameter is replaced with a script parameter, which uses the
script to generate values which percentiles are calculated on
Scripting supports parameterized input just like any other script

The script parameter expects an inline script. Use script_id for indexed scripts and script_file for scripts in the config/scripts/ directory.

Percentiles are (usually) approximate

There are many different algorithms to calculate percentiles. The naive implementation simply stores all the values in a sorted array. To find the 50th percentile, you simply find the value that is at my_array[count(my_array) * 0.5] .

Clearly, the naive implementation does not scale — the sorted array grows linearly with the number of values in your dataset. To calculate percentiles across potentially billions of values in an Elasticsearch cluster, approximate percentiles are calculated.

The algorithm used by the percentile metric is called TDigest (introduced by Ted Dunning in Computing Accurate Quantiles using T-Digests ).

When using this metric, there are a few guidelines to keep in mind:

• Accuracy is proportional to q(1-q) . This means that extreme percentiles (e.g. 99%) are more accurate than less extreme percentiles, such as the median For small sets of values, percentiles are highly accurate (and potentially 100% accurate if the data is small enough). As the quantity of values in a bucket grows, the algorithm begins to approximate the percentiles. It is effectively trading accuracy for memory savings. The exact level of inaccuracy is difficult to generalize, since it depends on your data distribution and volume of data being aggregated

The following chart shows the relative error on a uniform distribution depending on the number of collected values and the requested percentile:

It shows how precision is better for extreme percentiles. The reason why error diminishes for large number of values is that the law of large numbers makes the distribution of values more and more uniform and the t-digest tree can do a better job at summarizing it. It would not be the case on more skewed distributions.

Compression

Approximate algorithms must balance memory utilization with estimation accuracy. This balance can be controlled using a compression parameter:

{
    "aggs" : {
        "load_time_outlier" : {
            "percentiles" : {
                "field" : "load_time",
                "compression" : 200 
Compression controls memory usage and approximation error

The TDigest algorithm uses a number of "nodes" to approximate percentiles — the more nodes available, the higher the accuracy (and large memory footprint) proportional to the volume of data. The compression parameter limits the maximum number of nodes to 20 * compression .

Therefore, by increasing the compression value, you can increase the accuracy of your percentiles at the cost of more memory. Larger compression values also make the algorithm slower since the underlying tree data structure grows in size, resulting in more expensive operations. The default compression value is 100 .

A "node" uses roughly 32 bytes of memory, so under worst-case scenarios (large amount of data which arrives sorted and in-order) the default settings will produce a TDigest roughly 64KB in size. In practice data tends to be more random and the TDigest will use less memory.

Cardinality Aggregation

Percentile Ranks Aggregation

A multi-value metrics aggregation that calculates one or more percentile ranks over numeric values extracted from the aggregated documents. These values can be extracted either from specific numeric fields in the documents, or be generated by a provided script.

This functionality is experimental and may be changed or removed completely in a future release. Elastic will take a best effort approach to fix any issues, but experimental features are not subject to the support SLA of official GA features.

Please see Percentiles are (usually) approximate and Compression for advice regarding approximation and memory use of the percentile ranks aggregation

Percentile rank show the percentage of observed values which are below certain value. For example, if a value is greater than or equal to 95% of the observed values it is said to be at the 95th percentile rank.

Assume your data consists of website load times. You may have a service agreement that 95% of page loads completely within 15ms and 99% of page loads complete within 30ms.

Let’s look at a range of percentiles representing load time:

{
    "aggs" : {
        "load_time_outlier" : {
            "percentile_ranks" : {
                "field" : "load_time", 
                "values" : [15, 30]
The field load_time must be a numeric field

The response will look like this:

{
   "aggregations": {
      "load_time_outlier": {
         "values" : {
            "15": 92,
            "30": 100
}

From this information you can determine you are hitting the 99% load time target but not quite hitting the 95% load time target

Script

The percentile rank metric supports scripting. For example, if our load times are in milliseconds but we want to specify values in seconds, we could use a script to convert them on-the-fly:

{
    "aggs" : {
        "load_time_outlier" : {
            "percentile_ranks" : {
                "values" : [3, 5],
                "script" : "doc['load_time'].value / timeUnit", 
                "params" : {
                    "timeUnit" : 1000   
The field parameter is replaced with a script parameter, which uses the
script to generate values which percentile ranks are calculated on
Scripting supports parameterized input just like any other script

The script parameter expects an inline script. Use script_id for indexed scripts and script_file for scripts in the config/scripts/ directory.

Geo Bounds Aggregation

Cardinality Aggregation

A single-value metrics aggregation that calculates an approximate count of distinct values. Values can be extracted either from specific fields in the document or generated by a script.

This functionality is experimental and may be changed or removed completely in a future release. Elastic will take a best effort approach to fix any issues, but experimental features are not subject to the support SLA of official GA features.

Assume you are indexing books and would like to count the unique authors that match a query:

{
    "aggs" : {
        "author_count" : {
            "cardinality" : {
                "field" : "author"
}

This aggregation also supports the precision_threshold and rehash options:

{
    "aggs" : {
        "author_count" : {
            "cardinality" : {
                "field" : "author_hash",
                "precision_threshold": 100, 
                "rehash": false 
The precision_threshold options allows to trade memory for accuracy, and
defines a unique count below which counts are expected to be close to
accurate. Above this value, counts might become a bit more fuzzy. The maximum
supported value is 40000, thresholds above this number will have the same
effect as a threshold of 40000.
Default value depends on the number of parent aggregations that multiple
create buckets (such as terms or histograms).
If you computed a hash on client-side, stored it into your documents and want
Elasticsearch to use them to compute counts using this hash function without
rehashing values, it is possible to specify rehash: false. Default value is
true. Please note that the hash must be indexed as a long when rehash is
false.

Counts are approximate

Computing exact counts requires loading values into a hash set and returning its size. This doesn’t scale when working on high-cardinality sets and/or large values as the required memory usage and the need to communicate those per-shard sets between nodes would utilize too many resources of the cluster.

This cardinality aggregation is based on the HyperLogLog++ algorithm, which counts based on the hashes of the values with some interesting properties:

• configurable precision, which decides on how to trade memory for accuracy, excellent accuracy on low-cardinality sets, fixed memory usage: no matter if there are tens or billions of unique values, memory usage only depends on the configured precision.

For a precision threshold of c , the implementation that we are using requires about c * 8 bytes.

The following chart shows how the error varies before and after the threshold:

For all 3 thresholds, counts have been accurate up to the configured threshold (although not guaranteed, this is likely to be the case). Please also note that even with a threshold as low as 100, the error remains under 5%, even when counting millions of items.

Pre-computed hashes

If you don’t want Elasticsearch to re-compute hashes on every run of this aggregation, it is possible to use pre-computed hashes, either by computing a hash on client-side, indexing it and specifying rehash: false , or by using the special murmur3 field mapper, typically in the context of a multi-field in the mapping:

{
    "author": {
        "type": "string",
        "fields": {
            "hash": {
                "type": "murmur3"
}

With such a mapping, Elasticsearch is going to compute hashes of the author field at indexing time and store them in the author.hash field. This way, unique counts can be computed using the cardinality aggregation by only loading the hashes into memory, not the values of the author field, and without computing hashes on the fly:

{
    "aggs" : {
        "author_count" : {
            "cardinality" : {
                "field" : "author.hash"
}

rehash is automatically set to false when computing unique counts on a murmur3 field.

Pre-computing hashes is usually only useful on very large and/or high-cardinality fields as it saves CPU and memory. However, on numeric fields, hashing is very fast and storing the original values requires as much or less memory than storing the hashes. This is also true on low-cardinality string fields, especially given that those have an optimization in order to make sure that hashes are computed at most once per unique value per segment.

Script

The cardinality metric supports scripting, with a noticeable performance hit however since hashes need to be computed on the fly.

{
    "aggs" : {
        "author_count" : {
            "cardinality" : {
                "script": "doc['author.first_name'].value + ' ' + doc['author.last_name'].value"
}

The script parameter expects an inline script. Use script_id for indexed scripts and script_file for scripts in the config/scripts/ directory.

Top hits Aggregation