Troubleshooting

Can I Use Backtrace In Conjunction With Another Crash And Error Reporting Library? ​

For example, can I use Backtrace with Firebase Crashlytics?

Yes, it is possible to use Backtrace in conjunction with Firebase Crashlytics on Android and iOS. However, there are some limitations to keep in mind:

1. Only one native crash signal handler can be active per session. Generally, the last crash handler that was initialized is used, but there have been instances of non-deterministic behavior where both handlers are called, only the first initialized handler is called, or neither are called.

2. It is possible to disable the native functionality in Backtrace while keeping other features enabled, such as Crash Free metrics, Breadcrumbs, and Exception reporting. Refer to the SDK documentation for instructions on how to do this. A similar approach may be possible with the other crash reporting library as well.

Unhosted Symbols ​

In some cases, third-party platforms may offer debugging symbols as a direct download. Since these symbols are not available on a public symbol server, they are not included in our defaults. Here are a few known cases:

1. Renaming of a third-party library like Electron. For example, you can find Electron's releases at this GitHub repository: https://github.com/electron/electron/releases .
2. Debug symbol packages for your preferred Linux distribution. You can find more information in the Ubuntu Wiki .

The following example focuses on an unsymbolicated Electron crash:

1. To locate the symbol filename and debug ID, you can hover over the missing symbol from a crash or check the project's missing symbols list. Both will provide you with the filename (for example, Electron Framework) and the debug ID (for example, 45E07FE79CAC31169A0459CA63D608A70).

2. If you are using a public Electron release, you can find their symbols at the GitHub repository . Navigate to the version of Electron you are using.

3. If you are unsure of the version in use, you can locate the version attribute of the given crash. Let's assume the attribute is 11.4.7 .

   • For macOS, locating the matching symbol file can be a bit tricky. However, in this example, I found a match in electron-v11.4.7-darwin-x64-symbols.zip specifically in the folder \electron-v11.4.7-darwin-x64-symbols.zip\breakpad_symbols\Electron Framework\45E07FE79CAC31169A0459CA63D608A70 .

4. After locating the symbol file, you may need to update the filename to match the debug console in your Backtrace project. In this case, you would rename the symbol with the debug ID 45E07FE79CAC31169A0459CA63D608A70 to yourApplicationsName .

5. To upload the symbol, it is best practice to add it to an active zip or tar archive. You can then use curl or the webUI to upload the archive containing the symbol(s). The following support documents provide guidance for both processes:

   • HTTP API
   • Missing Symbols

6. All crash data present in a project before uploading the symbols must be reprocessed to account for the newly added symbols. Subsequent reports are processed with the newly added symbols. For more information, see Object Reprocessing .

Missing Symbols ​

Symbol Fundamentals ​

Symbols play a crucial role in generating callstacks for minidumps by associating memory addresses in a dump with the corresponding function names. In 64-bit builds, debug information is stored in executable files, which are necessary in addition to symbol files for complete callstack resolution. Backtrace supports various methods of providing symbols and executables, including custom symbol server integration, automated uploads in the build pipeline, manual uploads through the UI or via curl.

Determining Missing Symbols ​

To resolve missing symbols, the first step is to identify which symbols are missing.

Missing Symbol Files ​

Backtrace indicates that a crash report is missing symbols by setting an attribute, _missing_symbols , to 1, and in the debug view UI (in the callstack and the global annotations sections).

When a crash report is missing standard symbol files, in the debug view you'll see a memory address in the callstack with an orange triangle missing symbol icon.

You can hover over callstack frames missing symbols to see more information (like the file path and debug ID)

and you can click the blue missing symbol header to see additional missing symbol information.

Missing Executable Files ​

The debug view similarly indicates when a crash report is missing debug information from executable files, also known as Call Frame Information (CFI). The callstack section will show a guessed function name or a memory address in gray font, and the frame will be enclosed in <> .

Upon hover, you'll see a "missing CFI" message with details about the missing file.

If either of the filename or the debug ID are missing in this hover message, then you can search the global annotations modules section for whichever information (the debug ID or filename) was provided to find the missing portion from the corresponding symbol file entry.

These options should be sufficient in determining what symbols are missing from a crash report, but always feel free to reach out to our support team (by opening a support ticket, via the in-app widget, or emailing [email protected] for additional assistance.

Supplying Missing Symbols ​

For Backtrace to provide complete callstack resolution, the missing symbols must be supplied and the previously received crashes corresponding to those symbols must be reprocessed . How this is completed depends on users' symbol upload workflow.

Manual Upload ​

If you are uploading symbols manually, your next steps will be to retrieve and upload those symbols to Backtrace. After uploading missing symbols, corresponding crashes should be reprocessed in order for the symbols to apply.

Custom Symbol Server Integration ​

If you are using a custom symbol server , then your next step will be to verify that the symbol exists on the symbol server in the proper directory structure (that is, <url>/<object_name>/<debug_id>/<file> ). If it does not, then you know why it did not apply.

If it does exist, and reprocessing does not correct, then there are a few additional things to check. These items to check live in the symbol server section of your project settings (navigate from the Main Menu > Project Settings > Symbols > Symbol Servers ), unless otherwise noted.

• White list: If this optional configuration is enabled, then a symbol's name must exist on the white list in order for Backtrace to download it from the symbol server.
• Skip list: Symbol name and debug ID combinations that fail to download from a custom symbol server are added to a skip list so that requests are not made for symbols which do not exist on the server. If a symbol that previously did not exist on the symbol server was uploaded, its name and debug ID may exist on this list. Removing the name and ID from the skiplist and reprocessing the crash object will allow the symbol to download. The symbols can removed from the skip list tab of the symbol server entry, or via our command line tool, morgue .
• Logs: The symbol server entry's logs tab can be searched for additional information about the symbol download attempts and results / error messages. Searching the logs for the symbol's debug ID is generally most efficient. Note that we'll log a 200 upon successful download of a symbol, and common errors logged for failed downloads include 404, 401, etc.
• Misc: Backtrace applies downloaded symbols to all threads, but only downloads new symbols from the faulting thread. If the missing symbol is in a nonfaulting thread, it will not be downloaded from a custom symbol server. To workaround this, a symbol can be manually downloaded from the symbol server, uploaded to Backtrace, and then the crash can be reprocessed. Alternatively, the crash can be resubmitted with the thread in question marked faulting (using a modifier ) and later deleted as needed. After the symbol is downloaded from the symbol server, it will apply to all threads of new or reprocessed crashes.

Processing Missing Symbols on a Non-Faulting Thread ​

For a number of reasons (storage / performance), Backtrace does not by default process symbols which are only relevant to the non-faulting thread. However, the debugger does have a feature that will let you pull that information if it is relevant to your debugging.

First, there are a few prerequisites to enabling this functionality:

• You must be an admin
• Your instance's _BACKTRACE_PROMPT_MISSING_SYMBOLS window variable must be set to true . This can be changed by SSH into your coronerd instance and editing the .../coronerd/coronerd.conf file. The system must detect missing symbols on the non-faulting thread when you open an error report in the debugger (this is checked / happens automatically) If all of the above are true, you will see an icon/button show up on the top right corner of the debugger as shown below.

Pressing this button will bring up a window confirming whether or not you want to download + process the symbols that are missing. Confirming this will kick off the process for you (it might take a few seconds to complete).

You must have actually provided / uploaded the symbols in question. If you haven't done so, then the missing symbols errors wont be resolved. If that is the case, you can follow the guide outlined about around how to upload missing symbols.

Resolving Missing Symbols ​

Once missing symbols are uploaded or obtained from a symbol server, and the relevant crashes are reprocessed, the callstack will resolve completely.

If there are any questions or remaining issues, always feel free to reach out to our support team by opening a support ticket, via the in-app widget, or emailing [email protected] .

Why Are Aggregate Filters Removed In Certain Views? ​

Aggregation filters are used to filter the results of aggregations, which are performed on a per-group basis. For instance, in the triage view, the results are grouped by fingerprint by default, and aggregation filters are applied to each group's aggregations. In the explore view, users have the option to either group all results into one group or group them based on arbitrary attributes.

In an upcoming release, aggregation filters will be implemented to support flame graphs.

However, aggregation filters will not be available in the debugger or list views, as these views utilize selection queries that do not involve aggregations.

Error Processing Modifiers ​

Backtrace supports the following modifier attributes:

• _mod_faulting_tid : This attribute can be set on the object submission of minidump objects. If set, it specifies the thread to use for deduplication. The value is the integer identifier of a thread. Typically, this is used to incorporate watchdog systems for detecting hung threads.

• _mod_sync : This attribute can be set on the submission of any error object. It specifies that the object processor in Backtrace must respond only after the object has been evaluated and indexed. This may impact response latencies but allows for more comprehensive responses and advanced workflows, such as leveraging submission buckets. It is also valuable during debugging.

• _mod_log : This attribute can be set on the submission of any error object. Enabling it includes a detailed trace buffer of object processing in the response. This may impact response latencies but is valuable for debugging purposes.

The following modifier is considered advanced (consult Support before enabling):

• _mod_fingerprint : This attribute can be set on submission. Its value must be sha256 . If set, it bypasses the default fingerprinting algorithms and creates a fingerprint based on the SHA256 for a given crash.

How Much Disk Space Do Crash Dumps Take in Backtrace? ​

To estimate the total size of your crash dump metadata, you can use the following formula:

8 bytes * number of attributes * number of crashes

To estimate the total disk space used by the crash dumps themselves, you can multiply the average size of your crash dumps by the estimated volume.

How Do I Delete a Project? ​

Reach out to support if you wish to delete a project.

We are aware this is not optimal and plan to allow users to manage project deletions themselves soon.

What Version of jemalloc Does Ptrace Support? ​

Any of version of jemalloc before version 5 is supported with ptrace .

Improving Unwinding Accuracy for Windows ​

On Windows, 64-bit applications store some unwinding information exclusively in the executable object (.exe or .dll file). Therefore, it is advisable to include the executable code of your application and library when uploading symbols. These files can be uploaded as standalone files, but it is essential to ensure that the file base name matches the base name of the corresponding .pdb file. For instance, if the executable is named Editor.exe, the debug information should be stored in Editor.pdb. It is crucial to maintain this naming consistency, as the executable's name must match the associated .pdb file. If the executable is uploaded as Word.exe, it will not be possible to pair it with Editor.pdb. To ensure proper pairing, it is recommended to use a symbol archive.

How Can I Login to the Morgue Tool Via SSO? ​

Direct login to Morgue using an SSO-only account is not supported. However, there is a workaround available:

1. Start by logging into the Backtrace UI as you normally would.
2. Next, go to https://youruniverse.sp.backtrace.io/api/session and copy the token from the JSON data displayed on that page.

To log in to Morgue using this token, run the following command:

morgue login https://youruniverse.sp.backtrace.io --token=

replacing token with the token you copied from the api/session link mentioned earlier.

The token may be reset during server restarts or maintenance. If you encounter any messages regarding invalid tokens or authentication issues, repeat the process described above.

missing_symbol Attribute ​

The _missing_symbol attribute is a boolean that indicates whether an object has any private missing symbols. A private missing symbol refers to a symbol that is not included in either the blocklist or allowlist of our symbol management daemon.

You can utilize the _missing_symbol attribute to identify crashes that either have missing symbols or do not have missing symbols.

Changing the Displayed Timezone ​

Backtrace suggests installing a browser extension to modify the displayed timezone in our Web UI. According to feedback from Backtrace users, the Change Timezone extension for Chrome has been reported to work effectively. You can find it at this link: Chrome Web Store - Change Timezone .

For Firefox users, there is a similar plugin available at: Firefox Add-ons - Change Timezone .

How Do I Backup My Backtrace On-Premises Server? ​

To backup your on-premises coronerd installation, you'll need to backup the settings and the data folders.

Settings ​

Coronerd stores its settings in /etc/coronerd - make sure to include this entire folder in your backup. This will include your coronerd.conf file as well as your coronerd.db configuration database - you will need both to restore your system.

The path to coronerd.db is configured in /etc/coronerd/coronerd.conf in the setting configuration - if you've relocated this file, refer to this setting for the proper path.

Data ​

By default, coronerd stores its data under /var/coronerd - make sure to include this folder and all subfolders in your backup.

This path is determined by "storage": { "disks": [ ... ] } in your coronerd.conf file, so if you're using an alternate location, verify it here.

For Users of Symbold ​

If you're using symbold, make sure to include /var/symbold in your backup.

This path is determined by "symbold": { "symbol_path": "..." } in coronerd.conf .

For Users of backtrace-saml ​

If you're also running the backtrace-saml service, make sure to backup the folder /opt/backtrace/backtrace-saml , which should also include the saml.json file. Depending on your configuration, this may be on the same or different server than where coronerd is running - see the "saml": { "provider": "..." } setting in your coronerd.conf .

Folder Structure ​

Dump files and snapshots are stored in /var/coronerd///_objects

Query data is stored in /var/coronerd///crdb and /var/coronerd///database

Can We Upload Different Versions of Symbol Files For One Executable? ​

You can upload symbol files for different versions of the same executable, and the Backtrace system will ensure that they are correctly identified without overwriting older versions. It will then match crash dumps to the corresponding symbol files.

This matching process is achieved by comparing an internal identifier present in both the dump and symbol files. For more details about our symbol management system, see, Symbolication .

I Need Help Registering For a Free Trial ​

The Subdomain field is a user-entered field that plays a crucial role in accessing your Backtrace instance. After you have completed the registration process, you can sign in to Backtrace by visiting: https:// [subdomain] .sp.backtrace.io.

While entering a value in the Subdomain field, there are certain limitations to consider. If you try to input a value that is already being used by another user, our system will reject it, and you will need to choose a different value.

What Happens If I Hit My Crash Limit? ​

After you reach the limits of your current plan , Backtrace will no longer accept additional crash submissions. However, you can continue using the Console UI without any disruption. The crash submission limit will be reset at the beginning of each month.

If you require a higher crash submission limit, we recommend upgrading to a plan with a higher limit or exploring our Enterprise offering.

Even though invalid object submissions will not be visible in your Backtrace instance, they still count towards your submission limit. If you submit invalid objects, you may notice a difference between your upper submission limit and the number of crashes reported in Backtrace.

Vectored Exceptions With Crashpad ​

To handle Vectored Exceptions with Crashpad, one does need to do the following:

1. Integrate Crashpad.
2. Create a handler function with the following signature: LONG WINAPI (PEXCEPTION_POINTERS)
3. Register the global exception handler, or use the handler in a __try /__except block.

Exception Codes ​

0xC0000005 - Access Violation 0xE06D7363 - External Exception 0xC0000374 - Heap Corruption 0xC000008E - Float Divide By Zero 0xC0000094 - Integer Divide By Zero