Hi All,

I am successfully enabled SSL in graylog and application is up and running fine without any issues now I am trying to secure the communication between the Collector and Graylog by enabling the TLS in beats input

As per the documentation, Graylog itself created a new self signed certificate for the input and in the sidecar beats output configuration I marked Enable TLS support & Insecure TLS connection.

After enabling all these changes I am getting the below error in logs,

2017-11-07T04:51:23.854Z INFO  [InputStateListener] Input [Beats/59b794f268521b07e6b29b5f] is now STOPPING
2017-11-07T04:51:23.856Z INFO  [InputStateListener] Input [Beats/59b794f268521b07e6b29b5f] is now STOPPED
2017-11-07T04:51:23.857Z INFO  [InputStateListener] Input [Beats/59b794f268521b07e6b29b5f] is now TERMINATED
2017-11-07T04:51:23.857Z WARN  [AbstractTcpTransport] TLS key file or certificate file does not exist, creating a self-signed certificate for input [Beats/59b794f268521b07e6b29b5f].
2017-11-07T04:51:23.858Z INFO  [InputStateListener] Input [Beats/59b794f268521b07e6b29b5f] is now STARTING
2017-11-07T04:51:23.948Z INFO  [AbstractTcpTransport] Enabled TLS for input [Beats/59b794f268521b07e6b29b5f]. key-file="/tmp/keyutil_0.0.0.0:null_7936124129426110818.key" cert-file="/tmp/keyutil_0.0.0.0:null_6568254468981381412.crt"
2017-11-07T04:51:23.951Z WARN  [NettyTransport] receiveBufferSize (SO_RCVBUF) for input BeatsInput{title=Beats, type=org.graylog.plugins.beats.BeatsInput, nodeId=null} should be 1048576 but is 212992.
2017-11-07T04:51:23.952Z INFO  [InputStateListener] Input [Beats/59b794f268521b07e6b29b5f] is now RUNNING
2017-11-07T04:51:25.863Z ERROR [AbstractRotationStrategy] Cannot perform rotation at this moment.
2017-11-07T04:51:25.869Z ERROR [AbstractRotationStrategy] Cannot perform rotation at this moment.
2017-11-07T04:51:32.738Z WARN  [AbstractTcpTransport] client auth configured, but no authorized certificates / certificate authorities configured
2017-11-07T04:51:32.740Z ERROR [NettyTransport] Error in Input [Beats/873de4f268521b07e6b29b5f] (channel [id: 0xbda8b30d, /xx.xxx.xxx.xx:1072 => /xx.x.x.x:5044])
org.jboss.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 325700000001324385e6c92516bdb3e14c5f3ff7f927218f4450e57b615277a6a977663a385b5dec3d82841b56f3d53db329292524a3ee7bece90933018f383c53d1c1dff7cef4d3fcf66b3ff66b3ff7fbde1a2e7606a130cf41bc2ebc8d0786907def1103adb40e0914d38889d6da6622fd0b3f7a689e6321817daa139db8e6731e7d1783e3bbfe7faa3097c3e8700c7ac4d5b43cb42668540c73beea0f16978b2ae37a1b503042adb8fdbc06e33987e0abe5d5f7acf0102c1f8676894ec76ec20f0d47257fb48dc74e9c6dbadab7853d9aee32a58078d65a694acb33c59ad8c4af26a4989a9734ad294d327b55a168549113f79bcb2196ccd13227c5f2535f7167b81ce36279acb71ecdaeac41a4ce3a17f44f3e47d10705c59576f866dffc811215d2d1694a52b081c0187c37fdd96e5ddcdbbf2eee6fadb97fbebb2c4a9c56fd8b1f3b1191a6aaee61202c73be59f5efcb43efcadee052e42dbb30fa61fa191922c1229132abe52ae95d4999c13d177083cf3eb8b75f544bfee8cf76d8587d38c4e7b301a677a098de31821f0be1d8c7b85c60799a7149fcbe94d323f9c2a23caaf88f2784aa2584f9a225239915a1f354994af89f2a82f891411a9f4586707cf94581c721757440b49546444454eb45044c58180e28cfeb1b3fbdf010000ffffbe36d8b0
        at org.jboss.netty.handler.ssl.SslHandler.decode(SslHandler.java:857) ~[graylog.jar:?]
        at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:425) ~[graylog.jar:?]
        at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303) ~[graylog.jar:?]
        at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) ~[graylog.jar:?]
        at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [graylog.jar:?]
        at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) [graylog.jar:?]
        at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) [graylog.jar:?]
        at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) [graylog.jar:?]
        at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) [graylog.jar:?]
        at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) [graylog.jar:?]
        at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337) [graylog.jar:?]
        at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) [graylog.jar:?]
        at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) [graylog.jar:?]
        at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) [graylog.jar:?]
        at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) [graylog.jar:?]
        at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176) [graylog.jar:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_144]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_144]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_144]
Please kindly advice If I am doing anything wrong in the setup.
Should I use the graylog-certificate.pem & graylog-key.pem files generated in using HTTPS ???
Thanks,

Ganeshbabu R
Configuration of Graylog collector sidecar
server_url: https://graylogserver.southeastasia.cloudapp.azure.com/api/
update_interval: 10
tls_skip_verify: true
send_status: true
list_log_files:
node_id: graylog-collector-sidecar
collector_id: file:/etc/graylog/collector-sidecar/collector-id
cache_path: /var/cache/graylog/collector-sidecar
log_path: /var/log/graylog/collector-sidecar
log_rotation_time: 86400
log_max_age: 604800
tags:
    - linux
    - apache
    - graylogserver
backends:
    - name: nxlog
      enabled: false
      binary_path: /usr/bin/nxlog
      configuration_path: /etc/graylog/collector-sidecar/generated/nxlog.conf
    - name: filebeat
      enabled: true
      binary_path: /usr/bin/filebeat
      configuration_path: /etc/graylog/collector-sidecar/generated/filebeat.yml
Filebeat file
filebeat:
  prospectors:
  - document_type: log
    encoding: plain
    exclude_lines:
    - Ticket
    fields:
      data: example.org
      gl2_source_collector: f4749ffd-1f9b-4ef1-b065-a8fc32388fa1
    ignore_older: 0
    input_type: log
    paths:
    - /etc/graylog/graylogserver*.csv
    scan_frequency: 10s
    tail_files: false
output:
  logstash:
    hosts:
    - graylogserver.southeastasia.cloudapp.azure.com:5044
path:
  data: /var/cache/graylog/collector-sidecar/filebeat/data
  logs: /var/log/graylog/collector-sidecar
tags:
- linux
- apache
- graylogserver
The “logstash” output in Filebeat is missing all SSL/TLS settings.
Logstash Output | Filebeat Reference [5.6] | Elastic
SSL | Filebeat Reference [5.6] | Elastic
              
Hi @jochen
As you mentioned that logstash output in filebeat don’t have SSL settings. I reconfigured the filebeats output in graylog,

beats output.PNG841×438 31.6 KB
and I given the cert & key file paths and saved it. The SSL changes were reflected in the filebeat.yml file,
output:
  logstash:
    hosts:
    - graylogserver.southeastasia.cloudapp.azure.com:5044
    loadbalance: false
      certificate: /etc/ssl/nginx_crt.crt
      key: /etc/ssl/nginx_key.key
      verification_mode: none
To use SSL I was trying to configure the beats input with TLS enable and below is the changes made in graylog but I am getting the below error in logs after saving it,
beats input.PNG676×656 33.9 KB
2017-11-07T11:34:49.392Z WARN  [AbstractTcpTransport] client auth configured, but no authorized certificates / certificate authorities configured
2017-11-07T11:34:49.396Z ERROR [NettyTransport] Error in Input [Beats/59b794f268521b07e6b29b5f] (channel [id: 0xd64d29c2, /xx.xxx.xx.xx:1148 => /xx.x.x.x:5044])
org.jboss.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 3257000000013243000000b6785e6c8eb14a04311086e3abfcf5dc92648b83a94eae13acb4926be62e036e91e4b88cb2b2e4a97d015951dcc276f8e6fbfef8e09cbb73ce7d2e3864354962025e601f57054367bd9c550c84b38a6173e984834d599b49be82117dd8ef42d8f9fd731adde3573a9394b4960bcd66645b282d02cd5b7b5f5f478bc6f4ded743b1510749eec5893827da79f3dcbdf23e39707e15d6f6daa058c7118070fc2b77cc3f4fe150000ffffe19a4dd6
        at org.jboss.netty.handler.ssl.SslHandler.decode(SslHandler.java:857) ~[graylog.jar:?]
        at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:425) ~[graylog.jar:?]
        at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303) ~[graylog.jar:?]
        at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) ~[graylog.jar:?]
        at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [graylog.jar:?]
        at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) [graylog.jar:?]
        at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) [graylog.jar:?]
        at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) [graylog.jar:?]
        at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) [graylog.jar:?]
        at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) [graylog.jar:?]
        at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337) [graylog.jar:?]
        at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) [graylog.jar:?]
        at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) [graylog.jar:?]
        at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) [graylog.jar:?]
        at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) [graylog.jar:?]
        at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176) [graylog.jar:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_144]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_144]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_144]
2017-11-07T11:34:51.899Z WARN  [AbstractTcpTransport] client auth configured, but no authorized certificates / certificate authorities configured
2017-11-07T11:34:51.909Z ERROR [NettyTransport] Error in Input [Beats/59b794f26807e6b29b5f] (channel [id: 0x4d5f787f, /xx.xx.xx.xxx:51084 => /xx.x.x.x:5044])
**javax.net.ssl.SSLHandshakeException: General SSLEngine problem**
        at sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1478) ~[?:1.8.0_144]
        at sun.security.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:535) ~[?:1.8.0_144]
        at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:813) ~[?:1.8.0_144]
        at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:781) ~[?:1.8.0_144]
        at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624) ~[?:1.8.0_144]
        at org.jboss.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1219) ~[graylog.jar:?]
        at org.jboss.netty.handler.ssl.SslHandler.decode(SslHandler.java:852) ~[graylog.jar:?]
        at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:425) ~[graylog.jar:?]
        at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:310) ~[graylog.jar:?]
        at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) ~[graylog.jar:?]
        at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [graylog.jar:?]
        at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) [graylog.jar:?]
        at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) [graylog.jar:?]
        at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) [graylog.jar:?]
        at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) [graylog.jar:?]
        at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) [graylog.jar:?]
        at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337) [graylog.jar:?]
        at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) [graylog.jar:?]
        at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) [graylog.jar:?]
        at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) [graylog.jar:?]
        at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) [graylog.jar:?]
        at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176) [graylog.jar:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_144]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_144]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_144]
Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
        at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) ~[?:1.8.0_144]
        at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1728) ~[?:1.8.0_144]
        at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:304) ~[?:1.8.0_144]
        at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:296) ~[?:1.8.0_144]
        at sun.security.ssl.ServerHandshaker.clientCertificate(ServerHandshaker.java:1906) ~[?:1.8.0_144]
        at sun.security.ssl.ServerHandshaker.processMessage(ServerHandshaker.java:233) ~[?:1.8.0_144]
        at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1026) ~[?:1.8.0_144]
        at sun.security.ssl.Handshaker$1.run(Handshaker.java:966) ~[?:1.8.0_144]
        at sun.security.ssl.Handshaker$1.run(Handshaker.java:963) ~[?:1.8.0_144]
        at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_144]
        at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1416) ~[?:1.8.0_144]
        at org.jboss.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1393) ~[graylog.jar:?]
        at org.jboss.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1256) ~[graylog.jar:?]
        ... 19 more
Caused by: java.security.cert.CertificateException: No X509TrustManager implementation available
        at sun.security.ssl.DummyX509TrustManager.checkClientTrusted(SSLContextImpl.java:1191) ~[?:1.8.0_144]
        at sun.security.ssl.ServerHandshaker.clientCertificate(ServerHandshaker.java:1893) ~[?:1.8.0_144]
        at sun.security.ssl.ServerHandshaker.processMessage(ServerHandshaker.java:233) ~[?:1.8.0_144]
        at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1026) ~[?:1.8.0_144]
        at sun.security.ssl.Handshaker$1.run(Handshaker.java:966) ~[?:1.8.0_144]
        at sun.security.ssl.Handshaker$1.run(Handshaker.java:963) ~[?:1.8.0_144]
        at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_144]
        at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1416) ~[?:1.8.0_144]
        at org.jboss.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1393) ~[graylog.jar:?]
        at org.jboss.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1256) ~[graylog.jar:?]
        ... 19 more
Please correct me If I am doing anything wrong in the setup
Thanks,

Ganeshbabu R
              
@jochen
Kindly advice if I am doing anything wrong in the above SSL setup of beats input.
I couldn’t able to fix the problem
Thanks,

Ganeshbabu R
              
Hi, i’m not sure, but it seems that certificate for beats input is not correct, as your logs said:
“java.security.cert.CertificateException: No X509TrustManager implementation available”
Have you removed the passphrase from certificate key?

Is your crt in pem encoded format ?
hope this helps.
              
Hi @zionio
I generated the cert & key files using the below command,
sudo openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout /etc/ssl/nginx_key.key -out /etc/ssl/nginx_crt.crt
Have you removed the passphrase from certificate key?
I haven’t given any passphrase while generated key
Is your crt in pem encoded format ?
No its not in pem format
I just followed this documentation
Please kindly advice
Thanks,

Ganeshbabu R
              
Hi @jochen
2017-11-07T11:34:49.396Z ERROR [NettyTransport] Error in Input [Beats/59b794f268521b07e6b29b5f] (channel [id: 0xd64d29c2, /xx.xxx.xx.xx:1148 => /xx.x.x.x:5044])
org.jboss.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record:
Due to wrong configuration in the beats inputs I was getting the above error and I given the correct path of the cert & key files.

the filebeat are started harvesting the files and below is the details from the /var/log/collector-sidecar/filebeat
2017-11-13T13:57:11Z INFO Starting Registrar
2017-11-13T13:57:11Z INFO Start sending events to output
2017-11-13T13:57:11Z INFO Starting spooler: spool_size: 2048; idle_timeout: 5s
2017-11-13T13:57:11Z INFO Harvester started for file: /etc/graylog/data74.csv
2017-11-13T13:57:11Z INFO Harvester started for file: /etc/graylog/data71.csv
2017-11-13T13:57:11Z INFO Harvester started for file: /etc/graylog/data72.csv
2017-11-13T13:57:11Z INFO Harvester started for file: /etc/graylog/data73.csv
2017-11-13T13:57:41Z INFO Non-zero metrics in the last 30s: filebeat.harvester.open_files=4 filebeat.harvester.running=4 filebeat.harvester.started=4 libbeat.publisher.published_events=1130
2017-11-13T13:57:41Z ERR Connecting error publishing events (retrying): dial tcp 52.187.191.6:5044: i/o timeout
not sure whether this issue is anything related to firewall or port open…
below is the response of netstat -tuplen
netstat info.PNG984×354 11.1 KB
It would be very helpful if you could share your thoughts.
Thanks