列存储索引 - 查询性能 - SQL Server | Microsoft Learn

link管理

链接快照平台

输入网页链接，自动生成快照
标签化管理网页链接

相关文章推荐

强悍的大蒜 · Prevent Cross-Site ...· 10 月前 ·

愉快的单车 · SQLite – PHP | 菜鸟教程· 2 年前 ·

严肃的打火机 · vue.js - ...· 2 年前 ·

面冷心慈的饭盒 · Android WebView 详解 - ...· 2 年前 ·

聪明伶俐的白开水 · Android开发中日期工具类DateUti ...· 2 年前 ·

paths: - /var/log/containers/*-${data.kubernetes.container.id}.log exclude_lines: ["^\\s+[\\-`('.|_]"] # drop asciiart lines processors: - decode_json_fields: fields: ["message"] target: "json_message" process_array: true processors: - drop_event: when.or: - and: - regexp: message: '^\d+\.\d+\.\d+\.\d+ ' - equals: fileset.name: error - and: - not: regexp: message: '^\d+\.\d+\.\d+\.\d+ ' - equals: fileset.name: access - add_cloud_metadata: - add_kubernetes_metadata: - add_docker_metadata: output.elasticsearch: hosts: ['${ELASTICSEARCH_HOST:elasticsearch}:${ELASTICSEARCH_PORT:9200}'] username: ${ELASTICSEARCH_USERNAME} password: ${ELASTICSEARCH_PASSWORD} setup.kibana: host: '${KIBANA_HOST:kibana}:${KIBANA_PORT:5601}' setup.dashboards.enabled: true setup.template.enabled: true setup.ilm: policy_file: /etc/indice-lifecycle.json setup.template.settings: index.number_of_shards: 1 index.number_of_replicas: 2 apiVersion: v1 kind: ConfigMap metadata: namespace: kube-logging name: filebeat-indice-lifecycle labels: app: filebeat data: indice-lifecycle.json: |- "policy": { "phases": { "hot": { "actions": { "rollover": { "max_age": "1d" "delete": { "min_age": "1d", "actions": { "delete": {} apiVersion: apps/v1 kind: DaemonSet metadata: namespace: kube-logging name: filebeat labels: app: filebeat spec: selector: matchLabels: app: filebeat template: metadata: labels: app: filebeat spec: serviceAccountName: filebeat terminationGracePeriodSeconds: 30 # hostNetwork: true # dnsPolicy: ClusterFirstWithHostNet containers: - name: filebeat image: docker.elastic.co/beats/filebeat:7.6.1 args: [ "-c", "/etc/filebeat.yml", "-e", - name: ELASTICSEARCH_HOST value: elasticsearch - name: ELASTICSEARCH_PORT value: "9200" - name: ELASTICSEARCH_USERNAME value: elastic # - name: ELASTICSEARCH_PASSWORD # valueFrom: # secretKeyRef: # name: elasticsearch-pw-elastic # key: password - name: KIBANA_HOST value: kibana - name: KIBANA_PORT value: "5601" - name: NODE_NAME valueFrom: fieldRef: fieldPath: spec.nodeName securityContext: runAsUser: 0 privileged: true # for openshift resources: limits: memory: 200Mi requests: cpu: 100m memory: 100Mi volumeMounts: - name: config mountPath: /etc/filebeat.yml readOnly: true subPath: filebeat.yml - name: filebeat-indice-lifecycle mountPath: /etc/indice-lifecycle.json readOnly: true subPath: indice-lifecycle.json - name: data mountPath: /usr/share/filebeat/data - name: varlog mountPath: /var/log readOnly: true - name: varlibdockercontainers mountPath: /var/lib/docker/containers readOnly: true - name: dockersock mountPath: /var/run/docker.sock volumes: - name: config configMap: defaultMode: 0600 name: filebeat-config - name: filebeat-indice-lifecycle configMap: defaultMode: 0600 name: filebeat-indice-lifecycle - name: varlog hostPath: path: /var/log - name: varlibdockercontainers hostPath: path: /var/lib/docker/containers - name: dockersock hostPath: path: /var/run/docker.sock - name: data emptyDir: {} apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: namespace: kube-logging name: filebeat subjects: - kind: ServiceAccount name: filebeat namespace: kube-logging roleRef: kind: ClusterRole name: filebeat apiGroup: rbac.authorization.k8s.io apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: namespace: kube-logging name: filebeat labels: app: filebeat rules: - apiGroups: [""] resources: - namespaces - pods verbs: - get - watch - list apiVersion: v1 kind: ServiceAccount metadata: namespace: kube-logging name: filebeat labels: app: filebeat

It works, but the filebeat uses its hostname to connect to k8s:

2020-05-29T12:09:04.394Z	INFO	kubernetes/util.go:94	kubernetes: Using pod name filebeat-5zkvv and namespace kube-logging to discover kubernetes node
2020-05-29T12:09:04.402Z	INFO	kubernetes/util.go:100	kubernetes: Using node ip-10-20-2-27.ec2.internal discovered by in cluster pod node query
It doesn't look good because the hostname of filebeat pod will appear in the ES. It will be good to see the node's name in which the filebeat pod works instead of pod's name.
In the official example I found that we have to use those parameters (but in my example file I didn't use that)
      hostNetwork: true
      dnsPolicy: ClusterFirstWithHostNet
When I added those fields the filebeat didn't discover any kubernetes entities because in this case the node's hostname was used but there weren't any pods with that name:
2020-05-29T12:03:33.279Z	INFO	kubernetes/util.go:94	kubernetes: Using pod name ip-10-20-0-115.ec2.internal and namespace kube-logging to discover kubernetes node
2020-05-29T12:03:33.283Z	ERROR	kubernetes/util.go:97	kubernetes: Querying for pod failed with error: pods "ip-10-20-0-115.ec2.internal" not found
What do I want? I want filebeat works with "hostNetwork: true" correctly.
If you login to your pod by

kubectl exec -it filebeat-xxxxx --namespace=kube-logging -- /bin/bash

Check out your filesystem, where exactly are your container logs?

Are they somewhere under the default location /var/lib/docker/containers or maybe under /var/log/containers ?

Because this would be an additional fact you need to care about.
              The logs are here /var/log/containers. It is set in ConfigMap as you see for autodiscover.

I added this field host: {HOSTNAME} with hostnetwork false. When I don't use the host network the var {HOSTNAME} is "filebeat-5zkvv" (the hostname of pod). It works by default but in this case I see the agent.hostname in ES with pod's name but not node's name.

When I move to host network true the ${HOSTNAME} var is node's name and in this case filebeat can't autodiscover using node's name as a name of pod for autodiscover because there aren't pods with node's name.
              Maybe you can add some additional information to add_kubernetes_metadata ?
      - add_kubernetes_metadata:
          namespace: 'kube-system'
          in_cluster: true
          default_matchers.enabled: false
          default_indexers.enabled: false
          host: ${NODE_NAME}
          matchers:
            - logs_path:
                logs_path: '/var/log/containers'
                resource_type: 'pod'
      - add_docker_metadata:
Maybe you need to reconfigure you namespace setting if you want to use other ones.
And I used the setting additionally (You commented it out)
      dnsPolicy: ClusterFirstWithHostNet
Because I had this error regarding the docker host as well at first.
              May be you don't understand me 

I am saying that filebeat uses it's hostname when it runs autodiscover.

Filebeat's hostname depends on hostNetwork parameter.

When the hostNetwork is true filebeat uses hostname of kubernets node. In this way filebeat can't discover anything.

When the hostNetwork is false filebeat uses its pod name. In this way filebeat discovers everything that I need but in Elasticsearch I found that filebeat's data has the field agent.hostname with pod's name, not node's name. In this approach the pod's name is useless because we need to node's name on what filebeat works.
              I think I understand you and I had the same issue as well. Maybe I post you my whole yaml file if that helps. If not I am not sure if this can be a kubernetes specific topic
I have a local kubernetes installation:
>kubectl version
Client Version: version.Info{Major:"1", Minor:"11+", GitVersion:"v1.11.0+d4cacc0", GitCommit:"d4cacc0", GitTreeState:"clean", BuildDate:"2018-10-10T16:38:01Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"windows/amd64"}
Server Version: version.Info{Major:"1", Minor:"16+", GitVersion:"v1.16.6-beta.0", GitCommit:"e7f962ba86f4ce7033828210ca3556393c377bcc", GitTreeState:"clean", BuildDate:"2020-01-15T08:18:29Z", GoVersion:"go1.13.5", Compiler:"gc", Platform:"linux/amd64"}
Do you maybe use a cloud platform like Google GKE or Amazon EKS?
apiVersion: v1
kind: ConfigMap
metadata:
  namespace: kube-logging
  name: filebeat-config
  labels:
    app: filebeat
data:
  filebeat.yml: |-
    filebeat.modules:
      - module: system
        auth:
          enabled: true
    filebeat.autodiscover:
      providers:
        - type: kubernetes
          host: ${HOSTNAME}
          templates:
            - condition:
                  - equals:
                      kubernetes.namespace: cis
                  - equals:
                      kubernetes.namespace: kube-logging
              config:
                - type: container
                  paths:
                    - /var/log/containers/*-${data.kubernetes.container.id}.log
                  exclude_lines: ["^\\s+[\\-`('.|_]"]  # drop asciiart lines
                  processors:
                    - decode_json_fields:
                        fields: ["message"]
                        target: "json_message"
                        process_array: true
    processors:
      - drop_event:
          when.or:
              - and:
                  - regexp:
                      message: '^\d+\.\d+\.\d+\.\d+ '
                  - equals:
                      fileset.name: error
              - and:
                  - not:
                      regexp:
                          message: '^\d+\.\d+\.\d+\.\d+ '
                  - equals:
                      fileset.name: access
      - add_cloud_metadata:
      - add_kubernetes_metadata:
          namespace: 'kube-system'
          in_cluster: true
          default_matchers.enabled: false
          default_indexers.enabled: false
          host: ${NODE_NAME}
          matchers:
            - logs_path:
                logs_path: '/var/log/containers'
                resource_type: 'pod'
      - add_docker_metadata:
    output.elasticsearch:
      hosts: ['${ELASTICSEARCH_HOST:elasticsearch}:${ELASTICSEARCH_PORT:9200}']
    #  username: ${ELASTICSEARCH_USERNAME}
    #  password: ${ELASTICSEARCH_PASSWORD}
    #setup.kibana:
    #  host: '${KIBANA_HOST:kibana}:${KIBANA_PORT:5601}'
    #setup.dashboards.enabled: true
    setup.template.enabled: true
    setup.ilm:
      policy_file: /etc/indice-lifecycle.json
    setup.template.settings:
      index.number_of_shards: 1
    #  index.number_of_replicas: 2
      index.number_of_replicas: 0
apiVersion: v1
kind: ConfigMap
metadata:
  namespace: kube-logging
  name: filebeat-indice-lifecycle
  labels:
    app: filebeat
data:
  indice-lifecycle.json: |-
      "policy": {
        "phases": {
          "hot": {
            "actions": {
              "rollover": {
                "max_age": "1d"
          "delete": {
            "min_age": "1d",
            "actions": {
              "delete": {}
apiVersion: apps/v1
kind: DaemonSet
metadata:
  namespace: kube-logging
  name: filebeat
  labels:
    app: filebeat
spec:
  selector:
    matchLabels:
      app: filebeat
  template:
    metadata:
      labels:
        app: filebeat
    spec:
      serviceAccountName: filebeat
      terminationGracePeriodSeconds: 30
      hostNetwork: true
      dnsPolicy: ClusterFirstWithHostNet
      containers:
      - name: filebeat
        image: docker.elastic.co/beats/filebeat:7.7.0
        args: [
          "-c", "/etc/filebeat.yml",
          "-e",
        - name: ELASTICSEARCH_HOST
          value: elasticsearch
        - name: ELASTICSEARCH_PORT
          value: "9200"
        #- name: ELASTICSEARCH_USERNAME
        #  value: elastic
        # - name: ELASTICSEARCH_PASSWORD
        #   valueFrom:
        #     secretKeyRef:
        #       name: elasticsearch-pw-elastic
        #       key: password
#        - name: KIBANA_HOST
#          value: kibana
#        - name: KIBANA_PORT
#          value: "5601"
        - name: NODE_NAME
          valueFrom:
            fieldRef:
              fieldPath: spec.nodeName
        securityContext:
          runAsUser: 0
          privileged: true      # for openshift
        resources:
          limits:
            memory: 200Mi
          requests:
            cpu: 100m
            memory: 100Mi
        volumeMounts:
        - name: config
          mountPath: /etc/filebeat.yml
          readOnly: true
          subPath: filebeat.yml
        - name: filebeat-indice-lifecycle
          mountPath: /etc/indice-lifecycle.json
          readOnly: true
          subPath: indice-lifecycle.json
        - name: data
          mountPath: /usr/share/filebeat/data
        - name: varlog
          mountPath: /var/log
          readOnly: true
        - name: varlibdockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true
        - name: dockersock
          mountPath: /var/run/docker.sock
      volumes:
      - name: config
        configMap:
          defaultMode: 0600
          name: filebeat-config
      - name: filebeat-indice-lifecycle
        configMap:
          defaultMode: 0600
          name: filebeat-indice-lifecycle
      - name: varlog
        hostPath:
          path: /var/log
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers
      - name: dockersock
        hostPath:
          path: /var/run/docker.sock
      - name: data
        emptyDir: {}
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  namespace: kube-logging
  name: filebeat
subjects:
- kind: ServiceAccount
  name: filebeat
  namespace: kube-logging
roleRef:
  kind: ClusterRole
  name: filebeat
  apiGroup: rbac.authorization.k8s.io
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  namespace: kube-logging
  name: filebeat
  labels:
    app: filebeat
rules:
- apiGroups: [""]
  resources:
  - namespaces
  - pods
  verbs:
  - get
  - watch
  - list
apiVersion: v1
kind: ServiceAccount
metadata:
  namespace: kube-logging
  name: filebeat
  labels:
    app: filebeat