java.lang.IllegalArgumentException: The character [_] is never valid in a domain name.
    org.apache.tomcat.util.http.parser.HttpParser$DomainParseState.next(HttpParser.java:926)
    org.apache.tomcat.util.http.parser.HttpParser.readHostDomainName(HttpParser.java:822)
    org.apache.tomcat.util.http.parser.Host.parse(Host.java:71)
    org.apache.tomcat.util.http.parser.Host.parse(Host.java:45)
    org.apache.coyote.AbstractProcessor.parseHost(AbstractProcessor.java:288)
    org.apache.coyote.http11.Http11Processor.prepareRequest(Http11Processor.java:809)
    org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:384)
    org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
    org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:834)
    org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1415)
    org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
    java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    java.lang.Thread.run(Thread.java:748)

On versions prior to 6.0, the above error will not display - and you will receive an HTTP 400 error without any other information, such as the following:

To confirm that you are being impacted by this issue while on a version lower than 6.0, we recommend enabling Apache Tomcat access logging and reviewing the error in the Apache access log.

To enable Apache Tomcat access logging, we recommend setting the Tomcat access log properties in your bitbucket.properties file to something like the following values:

server.tomcat.accesslog.enabled=true
server.tomcat.accesslog.directory=${bitbucket.home/log}
server.tomcat.accesslog.prefix=tomcat-access
server.tomcat.accesslog.pattern=%t [%I] %{X-AREQUESTID}o %h (%{X-Forwarded-For}i) "%r" %s %b

Cause

When upgrading to Bitbucket Server 5.13.0 or later, the version of Apache Tomcat is upgraded to 8.5.31 or above. Within this Apache Tomcat update, the following change was introduced:

• " Enable strict validation of the provided host name and port for all connectors. Requests with invalid host names and/or ports will be rejected with a 400 response. (markt) " - Apache Tomcat 8.5.31 Release Notes

If you are receiving the above error, this means that the "Host" http header is being formatted in a way where it is now being denied access by Apache Tomcat, and the request is now receiving a 400 HTTP response. A known cause of the "Host" http header being modified is when a reverse proxy modifies this http header before passing the http request to the proxied server (Bitbucket Server's Apache Tomcat instance).

Resolution

Locate where the "Host" http header is being modified before it reaches Apache Tomcat, and modify it to remove any special characters/incorrect formatting.

Here is some example portions of two nginx.conf files where the 'Host' http header was being modified to include an '_' (an invalid character), and then the change made to the "proxy_set_header" portion to remove the invalid character and allow access to Bitbucket:

proxy_set_header Host "Bitbucket_Server_Proxy"; proxy_pass http://bitbucket.mybitbucket.com.upstream; proxy_redirect off; upstream bitbucket.mybitbucket.com.upstream { server localhost:7990; proxy_set_header Host "BitbucketServerProxy"; proxy_pass http://bitbucket.mybitbucket.com.upstream; proxy_redirect off; upstream bitbucket.mybitbucket.com.upstream { server localhost:7990;