Hello,
I installed Graylog on Ubuntu 22.04 server about a month ago, and follwoing the recommendation on the official installation page, I installed OpenSearch instead of Elasticsearch, and it’s been working great until last night when I ran apt-get update (I’m not entirely sure it’s the reason for this problem).
Here are the information/troubleshooting I have so far.
dpkg -l | grep -E ".(opensearch|graylog|mongo)."
ii graylog-5.1-repository 1-2 all Package to install Graylog 5.1 GPG key and repository
ii graylog-server 5.1.2-1 amd64 Graylog server
ii mongodb-database-tools 100.7.2 amd64 mongodb-database-tools package provides tools for working with the MongoDB server:
ii mongodb-mongosh 1.10.0 amd64 MongoDB Shell CLI REPL Package
ii mongodb-org 6.0.6 amd64 MongoDB open source document-oriented database system (metapackage)
ii mongodb-org-database 6.0.6 amd64 MongoDB open source document-oriented database system (metapackage)
ii mongodb-org-database-tools-extra 6.0.6 amd64 Extra MongoDB database tools
ii mongodb-org-mongos 6.0.6 amd64 MongoDB sharded cluster query router
ii mongodb-org-server 6.0.6 amd64 MongoDB database server
ii mongodb-org-shell 6.0.6 amd64 MongoDB shell client
ii mongodb-org-tools 6.0.6 amd64 MongoDB tools
ii opensearch 2.8.0 amd64 An open source distributed and RESTful search engine
service graylog-server status
● graylog-server.service - Graylog server
Loaded: loaded (/lib/systemd/system/graylog-server.service; enabled; vendor preset: enabled)
Active: active (running) since Fri 2023-06-16 11:26:47 +03; 30min ago
Docs: http://docs.graylog.org/
Main PID: 3902 (graylog-server)
Tasks: 44 (limit: 43216)
Memory: 226.4M
CPU: 14.174s
CGroup: /system.slice/graylog-server.service
├─3902 /bin/sh /usr/share/graylog-server/bin/graylog-server
└─3903 /usr/share/graylog-server/jvm/bin/java -Xms1g -Xmx1g -server -XX:+UseG1GC -XX:-OmitStackTraceInFastThrow -Djdk.tls.acknowledgeCloseN>
Jun 16 11:26:47 dell-poweredge-r420 systemd[1]: Started Graylog server.
netstat -lptn
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 127.0.0.1:27017 0.0.0.0:* LISTEN 1018/mongod
tcp 0 0 127.0.0.1:8088 0.0.0.0:* LISTEN 1055/influxd
tcp 0 0 127.0.0.1:33060 0.0.0.0:* LISTEN 1319/mysqld
tcp 0 0 127.0.0.53:53 0.0.0.0:* LISTEN 985/systemd-resolve
tcp 0 0 127.0.0.1:3306 0.0.0.0:* LISTEN 1319/mysqld
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1117/sshd: /usr/sbi
tcp 0 0 0.0.0.0:10051 0.0.0.0:* LISTEN 1744/zabbix_server
tcp 0 0 0.0.0.0:10050 0.0.0.0:* LISTEN 1103/zabbix_agentd
tcp6 0 0 ::1:9300 :::* LISTEN 3014/java
tcp6 0 0 ::1:9200 :::* LISTEN 3014/java
tcp6 0 0 :::8086 :::* LISTEN 1055/influxd
tcp6 0 0 :::3000 :::* LISTEN 1709/grafana
tcp6 0 0 :::80 :::* LISTEN 1239/apache2
tcp6 0 0 :::22 :::* LISTEN 1117/sshd: /usr/sbi
tcp6 0 0 127.0.0.1:9200 :::* LISTEN 3014/java
tcp6 0 0 127.0.0.1:9300 :::* LISTEN 3014/java
tcp6 0 0 :::10051 :::* LISTEN 1744/zabbix_server
tcp6 0 0 :::10050 :::* LISTEN 1103/zabbix_agentd
tail /var/log/graylog-server/server.log
2023-06-16T11:58:37.856+03:00 ERROR [VersionProbe] Unable to retrieve version from Elasticsearch node: unexpected end of stream on http://127.0.0.1:9200/... - \n not found: limit=0 content=….
2023-06-16T11:58:37.857+03:00 INFO [VersionProbe] Elasticsearch is not available. Retry #382
2023-06-16T11:58:42.861+03:00 ERROR [VersionProbe] Unable to retrieve version from Elasticsearch node: unexpected end of stream on http://127.0.0.1:9200/... - \n not found: limit=0 content=….
2023-06-16T11:58:42.862+03:00 INFO [VersionProbe] Elasticsearch is not available. Retry #383
2023-06-16T11:58:47.866+03:00 ERROR [VersionProbe] Unable to retrieve version from Elasticsearch node: unexpected end of stream on http://127.0.0.1:9200/... - \n not found: limit=0 content=….
2023-06-16T11:58:47.867+03:00 INFO [VersionProbe] Elasticsearch is not available. Retry #384
2023-06-16T11:58:52.871+03:00 ERROR [VersionProbe] Unable to retrieve version from Elasticsearch node: unexpected end of stream on http://127.0.0.1:9200/... - \n not found: limit=0 content=….
2023-06-16T11:58:52.872+03:00 INFO [VersionProbe] Elasticsearch is not available. Retry #385
2023-06-16T11:58:57.876+03:00 ERROR [VersionProbe] Unable to retrieve version from Elasticsearch node: unexpected end of stream on http://127.0.0.1:9200/... - \n not found: limit=0 content=….
2023-06-16T11:58:57.877+03:00 INFO [VersionProbe] Elasticsearch is not available. Retry #386
server.conf:
is_leader = true
node_id_file = /etc/graylog/server/node-id
password_secret = <sha-hash>
root_username = admin
root_password_sha2 = <sha-hash>
root_email = "<[email protected]>"
root_timezone = Cont/City
bin_dir = /usr/share/graylog-server/bin
data_dir = /var/lib/graylog-server
plugin_dir = /usr/share/graylog-server/plugin
http_bind_address = 192.168.1.1:9000
stream_aware_field_types=false
allow_leading_wildcard_searches = false
allow_highlighting = false
output_batch_size = 500
output_flush_interval = 1
output_fault_count_threshold = 5
output_fault_penalty_seconds = 30
processbuffer_processors = 5
outputbuffer_processors = 3
processor_wait_strategy = blocking
ring_size = 65536
inputbuffer_ring_size = 65536
inputbuffer_processors = 2
inputbuffer_wait_strategy = blocking
message_journal_enabled = true
message_journal_dir = /var/lib/graylog-server/journal
lb_recognition_period_seconds = 3
mongodb_uri = mongodb://localhost/graylog
mongodb_max_connections = 1000
transport_email_enabled = true
transport_email_hostname = smtp.office365.com
transport_email_port = 587
transport_email_use_auth = true
transport_email_auth_username = <[email protected]>
transport_email_auth_password = <mypassword>
transport_email_from_email = <[email protected]>
transport_email_use_tls = true
It might be worth mentioning that after the problem started I ran apt-get upgrade. While doing so I got this message:
Configuration file '/etc/opensearch/opensearch.yml'
==> Modified (by you or by a script) since installation.
==> Package distributor has shipped an updated version.
What would you like to do about it ? Your options are:
Y or I : install the package maintainer's version
N or O : keep your currently-installed version
D : show the differences between the versions
Z : start a shell to examine the situation
The default action is to keep your current version.
*** opensearch.yml (Y/I/N/O/D/Z) [default=N] ?
My answer was Y.
The server was rebooted after the upgrade was done.
Please let me know if more info is required.
Joma_29:
2023-06-16T11:58:57.876+03:00 ERROR [VersionProbe] Unable to retrieve version from Elasticsearch node: unexpected end of stream on http://127.0.0.1:9200/... - \n not found: limit=0 content=….
2023-06-16T11:58:57.877+03:00 INFO [VersionProbe] Elasticsearch is not available. Retry #386
I think here is your problem. Elastic is not running as it should. Did you try to migrate to Opensearch, and also start the service?
That’s good news, so you will need to make you Graylog talk to Opensearch again. It looks as they to not talk to each other.
Can you check if the service is running in the first place? Your netstat-command looks like, but to be sure.
Next check your Graylog-Config, if the right IPs/Ports and users are in there.
Here’s the output of Opensearch status:
service opensearch status
● opensearch.service - OpenSearch
Loaded: loaded (/lib/systemd/system/opensearch.service; enabled; vendor preset: enabled)
Active: active (running) since Fri 2023-06-16 11:14:36 +03; 1h 27min ago
Docs: https://opensearch.org/
Main PID: 3014 (java)
Tasks: 182 (limit: 43216)
Memory: 8.9G
CPU: 5min 36.734s
CGroup: /system.slice/opensearch.service
└─3014 /usr/share/opensearch/jdk/bin/java -Xshare:auto -Dopensearch.networkaddress.cache.ttl=60 -Dopensearch.networkaddress.cache.negative.>
Jun 16 11:14:20 dell-poweredge-r420 systemd[1]: Starting OpenSearch...
Jun 16 11:14:23 dell-poweredge-r420 systemd-entrypoint[3014]: WARNING: A terminally deprecated method in java.lang.System has been called
Jun 16 11:14:23 dell-poweredge-r420 systemd-entrypoint[3014]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.OpenSearch>
Jun 16 11:14:23 dell-poweredge-r420 systemd-entrypoint[3014]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.Ope>
Jun 16 11:14:23 dell-poweredge-r420 systemd-entrypoint[3014]: WARNING: System::setSecurityManager will be removed in a future release
Jun 16 11:14:24 dell-poweredge-r420 systemd-entrypoint[3014]: WARNING: A terminally deprecated method in java.lang.System has been called
Jun 16 11:14:24 dell-poweredge-r420 systemd-entrypoint[3014]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.Security (>
Jun 16 11:14:24 dell-poweredge-r420 systemd-entrypoint[3014]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.Sec>
Jun 16 11:14:24 dell-poweredge-r420 systemd-entrypoint[3014]: WARNING: System::setSecurityManager will be removed in a future release
Jun 16 11:14:36 dell-poweredge-r420 systemd[1]: Started OpenSearch.
The server IP address and users are correct, it’s been working fine for over a month and I haven’t changed anything in server.conf.
But even if there’s a problem in the communication between Graylog and Opensearch, isn’t the webpage supposed to at least open and ask for credentials?
Joma_29:
It might be worth mentioning that after the problem started I ran apt-get upgrade. While doing so I got this message:
Configuration file '/etc/opensearch/opensearch.yml'
==> Modified (by you or by a script) since installation.
==> Package distributor has shipped an updated version.
What would you like to do about it ? Your options are:
Y or I : install the package maintainer's version
N or O : keep your currently-installed version
D : show the differences between the versions
Z : start a shell to examine the situation
The default action is to keep your current version.
*** opensearch.yml (Y/I/N/O/D/Z) [default=N] ?
My answer was Y.
I found opensearch.yml reset to the default template and all my config lines were gone. I added the original config as shown below, after which the server ran without any problem.
cluster.name: graylog
node.name: ${HOSTNAME}
path.data: /var/lib/opensearch
path.logs: /var/log/opensearch
discovery.type: single-node
network.host: 0.0.0.0
action.auto_create_index: false
plugins.security.disabled: true
