Recently enabled Mattermost on Gitlab (since updated to 8.5.5) which is configured to have both as virtual hosts on Apache. The installation is on CentOS 6.7 with SCL’d version of httpd24 which includes
proxy_wstunnel_module
.
Mattermost is working except for a problem with Web Sockets. I have found lots of comments and configurations about this but don’t seem to be able to get this working. Unsure if my configuration is incorrect or if this is a problem with using the SCL’d version of httpd24.
The configuration file for Apache is:
<VirtualHost *:80>
LogLevel debug
ServerName mattermost.example.com
ServerAlias mattermost
DocumentRoot /opt/gitlab/embedded/service/mattermost/web/
ProxyPreserveHost On
# ProxyRequests Off
# Ensure that encoded slashes are not decoded but left in their encoded state.
# http://doc.gitlab.com/ce/api/projects.html#get-single-project
AllowEncodedSlashes NoDecode
# AllowEncodedSlashes On
RewriteEngine on
RewriteCond %{REQUEST_URI} ^/api/v1/websocket [NC,OR]
RewriteCond %{HTTP:UPGRADE} ^WebSocket$ [NC,OR]
RewriteCond %{HTTP:CONNECTION} ^Upgrade$ [NC]
RewriteRule .* ws://127.0.0.1:8065%{REQUEST_URI} [P,QSA,L]
RewriteCond %{DOCUMENT_ROOT}/%{REQUEST_FILENAME} !-f
RewriteRule .* http://127.0.0.1:8065%{REQUEST_URI} [P,QSA,L]
# RequestHeader set X-Forwarded-Proto "https"
# Prevent apache from sending incorrect 304 status updates
RequestHeader unset If-Modified-Since
RequestHeader unset If-None-Match
<Location /api/v1/websocket>
Require all granted
ProxyPassReverse ws://127.0.0.1:8065/api/v1/websocket
ProxyPassReverseCookieDomain 127.0.0.1 mattermost.example.com
</Location>
<Location />
Require all granted
ProxyPassReverse http://127.0.0.1:8065/
ProxyPassReverseCookieDomain 127.0.0.1 mattermost.example.com
</Location>
LogFormat "%{X-Forwarded-For}i %l %u %t \"%r\" %>s %b" common_forwarded
ErrorLog /var/log/httpd/mattermost.example.com_error.log
CustomLog /var/log/httpd/mattermost.example.com_forwarded.log common_forwarded
CustomLog /var/log/httpd/mattermost.example.com_access.log combined env=!dontlog
CustomLog /var/log/httpd/mattermost.example.com.log combined
</VirtualHost>
But with this, I still get the warning message Please check connection, Mattermost unreachable. If issue persists, ask administrator to check WebSocket port. at the top of page when in Mattermost.
This is my first time with WebSockets and Apache, so am a little unsure as to what I should see in the logs, etc. In trying to resolve the problem, I tried accessing the api directly to check the responses and logs. If I use the URL http://mattermost.example.com/api/v1/websocket directly, i.e. not via Apache but through Gitlab, I get back:
Bad Request
{"id":"api.web_socket.connect.upgrade.app_error","message":"Failed to upgrade websocket connection","detailed_error":"","request_id":"ihsn1f8se3nmtfcx74ezq8omhr","status_code":500,"is_oauth":false}
But when going through Apache VirtualHost, I get back GitLab Mattermost needs your help: and the URL is adjusted to http://mattermost.example.com/ws:/127.0.0.1:8065/api/v1/websocket
In the logs I see:
"GET /ws://127.0.0.1:8065/api/v1/websocket?session_token_index=0 HTTP/1.1" 301
Any suggestions/guidance very much appreciated.
Bit more digging and I have found the following in the logs:
[Mon Mar 21 07:39:17.030403 2016] [authz_core:debug] [pid 28334] mod_authz_core.c(802): [client {ip_addr}:52922] AH01626: authorization result of Require all granted: granted
[Mon Mar 21 07:39:17.030458 2016] [authz_core:debug] [pid 28334] mod_authz_core.c(802): [client {ip_addr}:52922] AH01626: authorization result of <RequireAny>: granted
[Mon Mar 21 07:39:17.030522 2016] [proxy:debug] [pid 28334] mod_proxy.c(1100): [client {ip_addr}:52922] AH01143: Running scheme http handler (attempt 0)
[Mon Mar 21 07:39:17.030537 2016] [proxy_wstunnel:debug] [pid 28334] mod_proxy_wstunnel.c(328): [client {ip_addr}:52922] AH02450: declining URL http://mattermost.example.com/ws://127.0.0.1:8065/api/v1/websocket?session_token_index=0
[Mon Mar 21 07:39:17.030552 2016] [proxy_ajp:debug] [pid 28334] mod_proxy_ajp.c(708): [client {ip_addr}:52922] AH00894: declining URL http://mattermost.example.com/ws://127.0.0.1:8065/api/v1/websocket?session_token_index=0
[Mon Mar 21 07:39:17.030566 2016] [proxy_fcgi:debug] [pid 28334] mod_proxy_fcgi.c(944): [client {ip_addr}:52922] AH01076: url: http://mattermost.example.com/ws://127.0.0.1:8065/api/v1/websocket?session_token_index=0 proxyname: (null) proxyport: 0
[Mon Mar 21 07:39:17.030579 2016] [proxy_fcgi:debug] [pid 28334] mod_proxy_fcgi.c(950): [client {ip_addr}:52922] AH01077: declining URL http://mattermost.example.com/ws://127.0.0.1:8065/api/v1/websocket?session_token_index=0
[Mon Mar 21 07:39:17.030595 2016] [proxy:debug] [pid 28334] proxy_util.c(2020): AH00942: HTTP: has acquired connection for (*)
[Mon Mar 21 07:39:17.030611 2016] [proxy:debug] [pid 28334] proxy_util.c(2072): [client {ip_addr}:52922] AH00944: connecting http://mattermost.example.com/ws://127.0.0.1:8065/api/v1/websocket?session_token_index=0 to mattermost.example.com:80
[Mon Mar 21 07:39:17.031538 2016] [proxy:debug] [pid 28334] proxy_util.c(2194): [client {ip_addr}:52922] AH00947: connected /ws://127.0.0.1:8065/api/v1/websocket?session_token_index=0 to mattermost.example.com:80
Which has a declining URL http://mattermost.example.com/ws://127.0.0.1:8065/...
which looks like it hasn’t switch protocols.
I’m having the same issue with Apache 2.4.6 and CentOS 7 with https.
“Please check connection, Mattermost unreachable. If issue persists, ask administrator to check WebSocket port.” message and all connections through wss:// return 301.
<VirtualHost *:443>
ServerName mattermost.example.com
ServerSignature Off
# TLS
SSLEngine on
SSLProtocol all -SSLv2
SSLHonorCipherOrder on
Header add Strict-Transport-Security: "max-age=15768000;includeSubdomains"
SSLCompression Off
SSLCertificateKeyFile /.../privkey.pem
SSLCertificateFile /.../cert.pem
SSLCertificateChainFile /.../chain.pem
ProxyPreserveHost On
RewriteEngine On
SSLProxyEngine On
RewriteCond %{REQUEST_URI} ^/api/v1/websocket [NC,OR]
RewriteCond %{HTTP:UPGRADE} ^WebSocket$ [NC,OR]
RewriteCond %{HTTP:CONNECTION} ^Upgrade$ [NC]
RewriteRule .* ws://127.0.0.1:8065%{REQUEST_URI} [P,QSA,L]
RewriteCond %{DOCUMENT_ROOT}/%{REQUEST_FILENAME} !-f
RewriteRule .* http://127.0.0.1:8065%{REQUEST_URI} [P,QSA,L]
# Be sure to uncomment the next 2 lines if https is used
RequestHeader set X-Forwarded-Proto "https"
# Prevent apache from sending incorrect 304 status updates
RequestHeader unset If-Modified-Since
RequestHeader unset If-None-Match
<Location /api/v1/websocket>
Require all granted
ProxyPassReverse ws://127.0.0.1:8065/api/v1/websocket
ProxyPassReverseCookieDomain 127.0.0.1 mattermost.example.com
</Location>
<Location />
Require all granted
ProxyPassReverse http://127.0.0.1:8065/
ProxyPassReverseCookieDomain 127.0.0.1 mattermost.example.com
</Location>
ProxyPreserveHost On
ProxyRequests Off
</VirtualHost>
I tried a couple of further configuration changes, including compiling mod_proxy_wstunnel.c
for Apache 2.2. Couldn’t get it working and could no longer spend any more time investigating. I have now moved the installation to CentOS 7 where it appears to be working.
Closed, but not resolved.