Platform notice: Server and Data Center only.
This article only applies to Atlassian products on the
Server and Data Center platforms
.
Support for Server* products
ended on February 15th 2024
. If you are running a Server product, you can visit the
Atlassian Server end of support
announcement to review your migration options.
*Except Fisheye and Crucible
Summary
Under heavy load, Bitbucket may log errors of "Too many open files".
At the same time, Git operations may fail and web interface may not be fully functional.
Environment
This issue may occur in various versions of Bitbucket Server and Data Center on Linux.
For example, it has been seen in the version 5.16.11.
Diagnosis
The application logs (eg.
atlassian-bitbucket.log
) may show multiple errors similar to the following:
2021-02-02 23:59:58,584 ERROR https-jsse-nio-8443-Acceptor-0 o.apache.tomcat.util.net.NioEndpoint Socket accept failed
java.io.IOException: Too many open files
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at sun.nio.ch.ServerSocketChannelImpl.accept(Unknown Source)
at sun.nio.ch.ServerSocketChannelImpl.accept(Unknown Source)
at java.lang.Thread.run(Unknown Source)
... 1 frame trimmed
Cause
Bitbucket's process, combined with its child (Git) processes, has too many open files, and has reached the maximum limit set in the Operating System.
UNIX systems have a limit on the number of files that can be concurrently open by any one process. This error indicates that the limit has been reached and Bitbucket is unable to open additional files to complete the on-going operations.
Solution based on the limits.conf file
In order to resolve this issue, you will need to allow Bitbucket to open more files than it is currently allowed to.
This involves a change in the configuration of the Operating System and a change in Bitbucket's startup procedure.
-
Log into Bitbucket's Linux server as Bitbucket's user (eg.
atlbitbucket
), open terminal window and run the following command, to find the currently applied limit:
In some instances, the limit of 64K may still be too restrictive, and you may need to raise it to 128K (134217728).
We do not recommend specifying an unlimited value, even if your Linux Operating System allows it.
-
Edit Bitbucket's
$BITBUCKET_INSTALL_DIR/bin/_start-webapp.sh
script.
Find the line that begins with:
MAX_OPEN_FILES=
and change it to the same limit you specified in the previous step.
In this example, this line should be:
-
Restart the server (including the Operating System) for the new settings to be applied.
-
If this is Bitbucket Data Center with multiple nodes, repeat the steps 1 through 4 for each node, one-by-one.
In step 4, to prevent the downtime, you may use the
rolling restart
of all nodes.
-
To verify if the new limit has been correctly applied, you may use one of the following two methods:
-
Generate a Support Zip (in Data Center, one from each Bitbucket node), extract it, open
application-properties/application.xml
and look for the line similar to:
This method allows you to confirm the change on each node of Bitbucket Data Center.
-
In Bitbucket's web interface, navigate to
Administration
>
Troubleshooting and support tools
>
System Information
.
Next, scroll down to the last section:
Operating System
, where you should see:
Max File Descriptors
65,536
This method only allows to confirm the change on the node to which your browser is currently connected.
Solution when installed as a systemd service
When Bitbucket is installed as a
systemd service
you may need to update the service unit file as described below.
If you are sure Bitbucket is running as a systemd service, go straight to Step 2.
-
Check if Bitbucket is configured as a systemd service. This helps to identify if this is the case and the service name, in case you are not sure.