Timestamps, Time Zones, Time Ranges, and Date Formats

We support several options for timestamps, time zones, time ranges, and dates. When collecting log data, the timestamp attached to messages is vital, both for the integrity of the data in your account, and for accurate query results.

Because of the importance of timestamps, Sumo Logic indexes the timestamp of each message, making sure that data relevant to a query’s time range is returned properly in search results, which allows you to reconstruct a correct event timeline.

Timestamps ​

The timestamp is the part of a log message that marks the time that an event occurred. During ingestion, we can detect the message timestamp, convert it to Unix epoch time (the number of milliseconds since midnight, January 1, 1970 UTC), and index it. The timestamp is parsed either using the default timestamp parsing settings, or a custom format that you specify, including the time zone.

When configuring a Source, you can choose to use the default timestamp parsing settings, or you can specify a custom format for us to parse timestamps in your log messages. The Enable Timestamp Parsing option is selected by default. If it's deselected, no timestamp information is parsed at all. Instead, we stamp logs with the time at which the messages are processed.

Timestamp considerations ​

By default, we can automatically detect timestamps in your log messages. Automatic detection identifies timestamps in common formats and prefers timestamps that appear early in the message.

If your log messages from a Source contain multiple timestamps, timestamps in unusual formats, or a mix of distinct timestamp formats, you have two options:

• Configure a Source for each log format
• Configure a custom timestamp format for your Source

The Collector assumes that all log messages coming from a particular Source will have timestamps that are close together. If a message comes through that appears to be more than one day earlier or later than recent messages from that Source it will be auto-corrected to match the current time. You can stop this auto-correction by explicitly configuring a custom timestamp format on your Source.

The Collector also assumes that all log messages coming from a particular Source will have timestamps that are within a window of -1 year through +2 days compared to the current time. Any log message with a parsed timestamp outside of that window is automatically re-stamped with the current time. You must contact Sumo Logic Support to adjust this auto-correction behavior. See how to ingest old or historical data for further details.

Automated timestamp parsing ​

Collectors can automatically parse any of the following timestamp formats. If more than one valid timestamp is detected in a log message, the Collector will select the timestamp that appears "furthest left" in the message.

Unix epoch timestamps ​

Unix epoch timestamps are supported in the following formats:

• 10-digit epoch time. The 10 digits must be surrounded by brackets (or followed by a comma) and at the very start of the message. For example, [1234567890] or [1234567890, other] followed by the rest of the message.
• 13-digit epoch time. The 13 digits must be at the very start of the message. For example, 1234567890123... followed by the rest of the message.
• 16-digit epoch time. The 16 digits must be at the very start of the message. For example, 1234567890123... followed by the rest of the message.
• 19-digit epoch time. The 19 digits must be at the very start of the message. For example, 1496756806.655123456... followed by the rest of the message.
• We also recognize the time format for the Akamai log delivery service. The format is 13 digits with a period before the last three (ms) digits. For example, 1234567890.123 .
• Comma-separated values where the 5th value from the start of the message is a 10 digit epoch time. For example, field1, field2, field3, field4, 1234567890
• JSON formatted property called "timestamp" followed by a 13-digit epoch time. For example: "timestamp":"123456789013" .
• Format of Cisco Fortigate/Meraki log message:

  ​<134>1 1439277406.903768018 Store_020026
  flows src=<redact> dst=72.245.34.184 protocol=udp
  sport=62118 dport=53 pattern: 1 all
  
• Format of Linux audit message:

  type=PATH msg=audit(1439992022.365:83931889): item=0
  name="/usr/sbin/ss" inode=91193416 dev=08:02
  mode=0100755 ouid=0 ogid=0 rdev=00:00
  

Specifying a custom timestamp format ​

Our Collectors can automatically parse most timestamps without any issues. However, if you see timestamp parsing issues, you can manually specify the timestamp format in the Sumo Logic UI when configuring a new Source or editing the timestamp information for an existing Source.

1. Do one of the following:
   • If you're configuring a new Source, proceed to the next step.
   • To edit the timestamp settings for an existing Source, navigate to Manage Data > Collection > Collection . Then click Edit to the right of the Source name and go to step 2.
2. Navigate to the Advanced Options for Logs section.
3. For Timestamp Format , select Specify a format .
   
4. In the Format field, enter the timestamp format the Collector should use to parse timestamps in your log.
   
   
   note

   If the timestamp format is in epoch time, enter epoch in the Format field.

   requirements

   • Your custom timestamp format must follow our supported timestamp conventions .
   • When you specify a custom format, provide us with the timestamp format (and optionally a regex) to help locate the desired timestamp in your log line format. If you don't provide a locator, we’ll scan the entire log message for a timestamp matching the given format by default. You can also test some sample log lines and see if we can parse the new format.
   • When providing multiple custom formats, specify the most common format first. The Collector will process each custom format in the order provided. Once a timestamp is located, no further timestamp checking is done.
   • If no timestamps are located that match your custom formats, the Collector will still attempt to automatically locate the log's timestamp.
5. The Timestamp locator is a regular expression with a capture group matching the timestamp in your log messages.
   The timestamp locator must:
   • be provided for 16-digit epoch or 19-digit epoch timestamps. Otherwise, this field is not necessary.
   • be a valid Java regular expression. Otherwise, this error message will be displayed: Unable to validate timestamp formats. The timestamp locator regex your-regex is invalid. The timestamp locator regex your-regex uses matching features which are not supported.
   • be an RE2-compliant regular expression, for example: \[time=(.*?)\] . Otherwise, this error message will be displayed: Unable to validate timestamp formats. The timestamp locator regex your-regex uses matching features which are not supported.
   • contain one unnamed capture group. When we extract timestamps, we only scan the portion of each log message that is captured by this group. If a log message does not match the locator expression, then your timestamp format cannot be applied to that message. If the regex doesn't contain one unnamed capture group, this error message will be displayed: Unable to validate timestamp formats. The timestamp locator regex your-regex does not contain a single unnamed capture group. The timestamp locator regex your-regex uses matching features which are not supported
     tip

     If you use quotes in the timestamp locator regular expression, you may see issues in the display after you save. The regular expression is not actually changed and can still be used to locate your timestamp.

6. If you have more than one custom timestamp format that you want to add, click + Add . The ordering of formats is significant. Each provided timestamp format is tested, in the order specified, until a matching format is found. The first matching format determines the final message timestamp. If none of the provided formats match a particular message, the Collector will attempt to automatically determine the message's timestamp.
7. Next, we recommend testing a few log lines from your data against your specified formats and locators. Enter sample log messages to test the timestamp formats you want to extract.
   
8. Click Test once your log lines are entered. The results display with the timestamp parsed and format matches (if any).
   You should see one of the following messages:
   • Format matched. In this example, the format of yyyy/MM/dd HH:mm:ss was matched and highlighted in green. This was the first format provided so it returns as 1(format: yyyy/MM/dd HH:mm:ss locator: \[time=(.*?)\]) The Effective message time would be 2023-01-15 02:12.000 +0000.
   • None of the custom timestamp format was matched. While the custom formats were not found in the log, there's still an auto detected timestamp highlighted in orange, 2023-06-01 02:12:12.259667 that we can use. The Effective message time is going to be 2023-06-01 02:12:12.259 +0000
   • Unable to parse any timestamp . No part of the sample log line "This line shouldn't parse" has a parseable timestamp and so the timestamp will be the current time.
9. Make any edits as needed to ensure your timestamps are parsed correctly.
10. Click Save to save your custom timestamp formats.

Using _format for troubleshooting ​

You can use _format to see how the timestamp is parsed from the log file. Assign _format an alias to return it in your search results, for example:

| _format as timestampFormat

The fields returned in the search results of _format are:

t:<parse type>,o:<offset>,l:<length>,p:<date_format>

where <parse type> can take the values:

• fail — Failed to locate timestamp.
• cache — Success, cached format.
• def — Success, default (user-specified) format.
• full — Success, from "full" parsing against library of patterns.
• none — Local/receipt time because timestamp parsing is not enabled for this source.
• ac1 — Auto-corrected by the "window-based" heuristic (what we call "auto-correction" today). Sumo Logic assumes that all log messages coming from a particular Source will have timestamps that are close together. If a message comes through that appears to be more than one day earlier or later than recent messages from that source, it will be auto-corrected to match the current time. You can stop this auto-correction by explicitly configuring a custom timestamp format on your Source.