Yeah, I agree. My first thought was the same, set the default sample rate to 10000, but then i saw terabits of traffic, when the reality is just a few gigs. Worked my way down to 1 where i got the sum of all flows to match up with the actual interface traffic.
I have added the sampler-table option and i have added a sampler map that specifies the mode to be random 1 out of 10000 - at this point it seems the software on the ASR is bugging, and its such an old router that i'm not going to fiddle around more with it, it was more for testing.
I'll try it out on our Juniper edges and see how that goes. :-)
Really good work on this, i like it alot so far - and thanks for your help.
I've got the sampler-table enabled, two different routers (both ASR1001-X), Akvorado is still claiming it's not getting the sampler table though
this is my sanitised output, looks like flows are incoming but the web interface is still showing no flows / exporters. 1.1.1.1 & 2.2.2.2 are the real routers, 172.20.0.1 is what appears before I delete the conntrack entries.
curl -s http://127.0.0.1:8081/api/v0/inlet/metrics | grep '^akvorado_inlet'
akvorado_inlet_core_classifier_cache_hits 3138
akvorado_inlet_core_classifier_cache_misses 33
akvorado_inlet_core_flows_errors{error="SNMP cache miss",exporter="1.1.1.1"} 5
akvorado_inlet_core_flows_errors{error="SNMP cache miss",exporter="172.20.0.1"} 163
akvorado_inlet_core_flows_errors{error="SNMP cache miss",exporter="2.2.2.2"} 16
akvorado_inlet_core_flows_forwarded{exporter="1.1.1.1"} 460
akvorado_inlet_core_flows_forwarded{exporter="2.2.2.2"} 597
akvorado_inlet_core_flows_http_clients 0
akvorado_inlet_core_flows_received{exporter="1.1.1.1"} 465
akvorado_inlet_core_flows_received{exporter="172.20.0.1"} 163
akvorado_inlet_core_flows_received{exporter="2.2.2.2"} 613
akvorado_inlet_flow_decoder_count{name="netflow"} 721
akvorado_inlet_flow_decoder_error_count{name="netflow"} 90
akvorado_inlet_flow_decoder_netflow_count{exporter="1.1.1.1",version="9"} 354
akvorado_inlet_flow_decoder_netflow_count{exporter="172.20.0.1",version="9"} 111
akvorado_inlet_flow_decoder_netflow_count{exporter="2.2.2.2",version="9"} 256
akvorado_inlet_flow_decoder_netflow_delay_summary_seconds{exporter="1.1.1.1",version="9",quantile="0.5"} 2
akvorado_inlet_flow_decoder_netflow_delay_summary_seconds{exporter="1.1.1.1",version="9",quantile="0.9"} 2
akvorado_inlet_flow_decoder_netflow_delay_summary_seconds{exporter="1.1.1.1",version="9",quantile="0.99"} 2
akvorado_inlet_flow_decoder_netflow_delay_summary_seconds_sum{exporter="1.1.1.1",version="9"} 930
akvorado_inlet_flow_decoder_netflow_delay_summary_seconds_count{exporter="1.1.1.1",version="9"} 465
akvorado_inlet_flow_decoder_netflow_delay_summary_seconds{exporter="172.20.0.1",version="9",quantile="0.5"} 2
akvorado_inlet_flow_decoder_netflow_delay_summary_seconds{exporter="172.20.0.1",version="9",quantile="0.9"} 2
akvorado_inlet_flow_decoder_netflow_delay_summary_seconds{exporter="172.20.0.1",version="9",quantile="0.99"} 2
akvorado_inlet_flow_decoder_netflow_delay_summary_seconds_sum{exporter="172.20.0.1",version="9"} 327
akvorado_inlet_flow_decoder_netflow_delay_summary_seconds_count{exporter="172.20.0.1",version="9"} 163
akvorado_inlet_flow_decoder_netflow_delay_summary_seconds{exporter="2.2.2.2",version="9",quantile="0.5"} 2
akvorado_inlet_flow_decoder_netflow_delay_summary_seconds{exporter="2.2.2.2",version="9",quantile="0.9"} 2
akvorado_inlet_flow_decoder_netflow_delay_summary_seconds{exporter="2.2.2.2",version="9",quantile="0.99"} 2
akvorado_inlet_flow_decoder_netflow_delay_summary_seconds_sum{exporter="2.2.2.2",version="9"} 1231
akvorado_inlet_flow_decoder_netflow_delay_summary_seconds_count{exporter="2.2.2.2",version="9"} 613
akvorado_inlet_flow_decoder_netflow_errors_count{error="template not found",exporter="1.1.1.1"} 1
akvorado_inlet_flow_decoder_netflow_errors_count{error="template not found",exporter="172.20.0.1"} 50
akvorado_inlet_flow_decoder_netflow_errors_count{error="template not found",exporter="2.2.2.2"} 39
akvorado_inlet_flow_decoder_netflow_flowset_records_sum{exporter="1.1.1.1",type="DataFlowSet",version="9"} 465
akvorado_inlet_flow_decoder_netflow_flowset_records_sum{exporter="1.1.1.1",type="OptionsDataFlowSet",version="9"} 401
akvorado_inlet_flow_decoder_netflow_flowset_records_sum{exporter="1.1.1.1",type="OptionsTemplateFlowSet",version="9"} 74
akvorado_inlet_flow_decoder_netflow_flowset_records_sum{exporter="1.1.1.1",type="TemplateFlowSet",version="9"} 26
akvorado_inlet_flow_decoder_netflow_flowset_records_sum{exporter="172.20.0.1",type="DataFlowSet",version="9"} 163
akvorado_inlet_flow_decoder_netflow_flowset_records_sum{exporter="172.20.0.1",type="OptionsDataFlowSet",version="9"} 117
akvorado_inlet_flow_decoder_netflow_flowset_records_sum{exporter="172.20.0.1",type="OptionsTemplateFlowSet",version="9"} 25
akvorado_inlet_flow_decoder_netflow_flowset_records_sum{exporter="172.20.0.1",type="TemplateFlowSet",version="9"} 7
akvorado_inlet_flow_decoder_netflow_flowset_records_sum{exporter="2.2.2.2",type="DataFlowSet",version="9"} 613
akvorado_inlet_flow_decoder_netflow_flowset_records_sum{exporter="2.2.2.2",type="OptionsDataFlowSet",version="9"} 224
akvorado_inlet_flow_decoder_netflow_flowset_records_sum{exporter="2.2.2.2",type="OptionsTemplateFlowSet",version="9"} 32
akvorado_inlet_flow_decoder_netflow_flowset_records_sum{exporter="2.2.2.2",type="TemplateFlowSet",version="9"} 4
akvorado_inlet_flow_decoder_netflow_flowset_sum{exporter="1.1.1.1",type="DataFlowSet",version="9"} 254
akvorado_inlet_flow_decoder_netflow_flowset_sum{exporter="1.1.1.1",type="OptionsDataFlowSet",version="9"} 74
akvorado_inlet_flow_decoder_netflow_flowset_sum{exporter="1.1.1.1",type="OptionsTemplateFlowSet",version="9"} 74
akvorado_inlet_flow_decoder_netflow_flowset_sum{exporter="1.1.1.1",type="TemplateFlowSet",version="9"} 26
akvorado_inlet_flow_decoder_netflow_flowset_sum{exporter="172.20.0.1",type="DataFlowSet",version="9"} 79
akvorado_inlet_flow_decoder_netflow_flowset_sum{exporter="172.20.0.1",type="OptionsDataFlowSet",version="9"} 25
akvorado_inlet_flow_decoder_netflow_flowset_sum{exporter="172.20.0.1",type="OptionsTemplateFlowSet",version="9"} 25
akvorado_inlet_flow_decoder_netflow_flowset_sum{exporter="172.20.0.1",type="TemplateFlowSet",version="9"} 7
akvorado_inlet_flow_decoder_netflow_flowset_sum{exporter="2.2.2.2",type="DataFlowSet",version="9"} 220
akvorado_inlet_flow_decoder_netflow_flowset_sum{exporter="2.2.2.2",type="OptionsDataFlowSet",version="9"} 32
akvorado_inlet_flow_decoder_netflow_flowset_sum{exporter="2.2.2.2",type="OptionsTemplateFlowSet",version="9"} 32
akvorado_inlet_flow_decoder_netflow_flowset_sum{exporter="2.2.2.2",type="TemplateFlowSet",version="9"} 4
akvorado_inlet_flow_decoder_netflow_templates_count{exporter="1.1.1.1",obs_domain_id="256",template_id="269",type="template",version="9"} 26
akvorado_inlet_flow_decoder_netflow_templates_count{exporter="1.1.1.1",obs_domain_id="6",template_id="256",type="options_template",version="9"} 27
akvorado_inlet_flow_decoder_netflow_templates_count{exporter="1.1.1.1",obs_domain_id="6",template_id="257",type="options_template",version="9"} 27
akvorado_inlet_flow_decoder_netflow_templates_count{exporter="1.1.1.1",obs_domain_id="6",template_id="259",type="options_template",version="9"} 20
akvorado_inlet_flow_decoder_netflow_templates_count{exporter="172.20.0.1",obs_domain_id="256",template_id="256",type="template",version="9"} 1
akvorado_inlet_flow_decoder_netflow_templates_count{exporter="172.20.0.1",obs_domain_id="256",template_id="269",type="template",version="9"} 6
akvorado_inlet_flow_decoder_netflow_templates_count{exporter="172.20.0.1",obs_domain_id="6",template_id="256",type="options_template",version="9"} 6
akvorado_inlet_flow_decoder_netflow_templates_count{exporter="172.20.0.1",obs_domain_id="6",template_id="257",type="options_template",version="9"} 13
akvorado_inlet_flow_decoder_netflow_templates_count{exporter="172.20.0.1",obs_domain_id="6",template_id="258",type="options_template",version="9"} 1
akvorado_inlet_flow_decoder_netflow_templates_count{exporter="172.20.0.1",obs_domain_id="6",template_id="259",type="options_template",version="9"} 5
akvorado_inlet_flow_decoder_netflow_templates_count{exporter="2.2.2.2",obs_domain_id="256",template_id="256",type="template",version="9"} 4
akvorado_inlet_flow_decoder_netflow_templates_count{exporter="2.2.2.2",obs_domain_id="6",template_id="256",type="options_template",version="9"} 20
akvorado_inlet_flow_decoder_netflow_templates_count{exporter="2.2.2.2",obs_domain_id="6",template_id="257",type="options_template",version="9"} 8
akvorado_inlet_flow_decoder_netflow_templates_count{exporter="2.2.2.2",obs_domain_id="6",template_id="258",type="options_template",version="9"} 4
akvorado_inlet_flow_input_udp_bytes{exporter="1.1.1.1",listener="0.0.0.0:2055",worker="1"} 29544
akvorado_inlet_flow_input_udp_bytes{exporter="172.20.0.1",listener="0.0.0.0:2055",worker="1"} 3208
akvorado_inlet_flow_input_udp_bytes{exporter="172.20.0.1",listener="0.0.0.0:2055",worker="5"} 6924
akvorado_inlet_flow_input_udp_bytes{exporter="2.2.2.2",listener="0.0.0.0:2055",worker="2"} 36164
akvorado_inlet_flow_input_udp_in_drops{listener="0.0.0.0:2055",worker="1"} 0
akvorado_inlet_flow_input_udp_in_drops{listener="0.0.0.0:2055",worker="2"} 0
akvorado_inlet_flow_input_udp_in_drops{listener="0.0.0.0:2055",worker="5"} 0
akvorado_inlet_flow_input_udp_packets{exporter="1.1.1.1",listener="0.0.0.0:2055",worker="1"} 254
akvorado_inlet_flow_input_udp_packets{exporter="172.20.0.1",listener="0.0.0.0:2055",worker="1"} 20
akvorado_inlet_flow_input_udp_packets{exporter="172.20.0.1",listener="0.0.0.0:2055",worker="5"} 59
akvorado_inlet_flow_input_udp_packets{exporter="2.2.2.2",listener="0.0.0.0:2055",worker="2"} 220
akvorado_inlet_flow_input_udp_summary_size_bytes{exporter="1.1.1.1",listener="0.0.0.0:2055",worker="1",quantile="0.5"} 124
akvorado_inlet_flow_input_udp_summary_size_bytes{exporter="1.1.1.1",listener="0.0.0.0:2055",worker="1",quantile="0.9"} 176
akvorado_inlet_flow_input_udp_summary_size_bytes{exporter="1.1.1.1",listener="0.0.0.0:2055",worker="1",quantile="0.99"} 176
akvorado_inlet_flow_input_udp_summary_size_bytes_sum{exporter="1.1.1.1",listener="0.0.0.0:2055",worker="1"} 29544
akvorado_inlet_flow_input_udp_summary_size_bytes_count{exporter="1.1.1.1",listener="0.0.0.0:2055",worker="1"} 254
akvorado_inlet_flow_input_udp_summary_size_bytes{exporter="172.20.0.1",listener="0.0.0.0:2055",worker="1",quantile="0.5"} 176
akvorado_inlet_flow_input_udp_summary_size_bytes{exporter="172.20.0.1",listener="0.0.0.0:2055",worker="1",quantile="0.9"} 176
akvorado_inlet_flow_input_udp_summary_size_bytes{exporter="172.20.0.1",listener="0.0.0.0:2055",worker="1",quantile="0.99"} 224
akvorado_inlet_flow_input_udp_summary_size_bytes_sum{exporter="172.20.0.1",listener="0.0.0.0:2055",worker="1"} 3208
akvorado_inlet_flow_input_udp_summary_size_bytes_count{exporter="172.20.0.1",listener="0.0.0.0:2055",worker="1"} 20
akvorado_inlet_flow_input_udp_summary_size_bytes{exporter="172.20.0.1",listener="0.0.0.0:2055",worker="5",quantile="0.5"} 124
akvorado_inlet_flow_input_udp_summary_size_bytes{exporter="172.20.0.1",listener="0.0.0.0:2055",worker="5",quantile="0.9"} 176
akvorado_inlet_flow_input_udp_summary_size_bytes{exporter="172.20.0.1",listener="0.0.0.0:2055",worker="5",quantile="0.99"} 176
akvorado_inlet_flow_input_udp_summary_size_bytes_sum{exporter="172.20.0.1",listener="0.0.0.0:2055",worker="5"} 6924
akvorado_inlet_flow_input_udp_summary_size_bytes_count{exporter="172.20.0.1",listener="0.0.0.0:2055",worker="5"} 59
akvorado_inlet_flow_input_udp_summary_size_bytes{exporter="2.2.2.2",listener="0.0.0.0:2055",worker="2",quantile="0.5"} 176
akvorado_inlet_flow_input_udp_summary_size_bytes{exporter="2.2.2.2",listener="0.0.0.0:2055",worker="2",quantile="0.9"} 224
akvorado_inlet_flow_input_udp_summary_size_bytes{exporter="2.2.2.2",listener="0.0.0.0:2055",worker="2",quantile="0.99"} 224
akvorado_inlet_flow_input_udp_summary_size_bytes_sum{exporter="2.2.2.2",listener="0.0.0.0:2055",worker="2"} 36164
akvorado_inlet_flow_input_udp_summary_size_bytes_count{exporter="2.2.2.2",listener="0.0.0.0:2055",worker="2"} 220
akvorado_inlet_flow_summary_decoding_time_seconds{name="netflow",quantile="0.5"} 4.5835e-05
akvorado_inlet_flow_summary_decoding_time_seconds{name="netflow",quantile="0.9"} 5.9645e-05
akvorado_inlet_flow_summary_decoding_time_seconds{name="netflow",quantile="0.99"} 8.550400000000001e-05
akvorado_inlet_flow_summary_decoding_time_seconds_sum{name="netflow"} 0.03266854500000002
akvorado_inlet_flow_summary_decoding_time_seconds_count{name="netflow"} 721
akvorado_inlet_geoip_db_refresh_total{database="asn"} 1
akvorado_inlet_geoip_db_refresh_total{database="country"} 1
akvorado_inlet_kafka_brokers_incoming_byte_rate{broker="1001"} 5.706640724363773
akvorado_inlet_kafka_brokers_inflight_requests{broker="1001"} 0
akvorado_inlet_kafka_brokers_outgoing_byte_rate{broker="1001"} 220.2626865814584
akvorado_inlet_kafka_brokers_request_latency_seconds_bucket{broker="1001",le="0.5"} 2
akvorado_inlet_kafka_brokers_request_latency_seconds_bucket{broker="1001",le="0.9"} 4
akvorado_inlet_kafka_brokers_request_latency_seconds_bucket{broker="1001",le="0.99"} 202
akvorado_inlet_kafka_brokers_request_latency_seconds_bucket{broker="1001",le="+Inf"} 26
akvorado_inlet_kafka_brokers_request_latency_seconds_sum{broker="1001"} 263
akvorado_inlet_kafka_brokers_request_latency_seconds_count{broker="1001"} 26
akvorado_inlet_kafka_brokers_request_rate{broker="1001"} 0.09720276637938277
akvorado_inlet_kafka_brokers_request_size_bucket{broker="1001",le="0.5"} 2253
akvorado_inlet_kafka_brokers_request_size_bucket{broker="1001",le="0.9"} 2443
akvorado_inlet_kafka_brokers_request_size_bucket{broker="1001",le="0.99"} 2603
akvorado_inlet_kafka_brokers_request_size_bucket{broker="1001",le="+Inf"} 26
akvorado_inlet_kafka_brokers_request_size_sum{broker="1001"} 52541
akvorado_inlet_kafka_brokers_request_size_count{broker="1001"} 26
akvorado_inlet_kafka_brokers_response_bytes_bucket{broker="1001",le="0.5"} 60
akvorado_inlet_kafka_brokers_response_bytes_bucket{broker="1001",le="0.9"} 60
akvorado_inlet_kafka_brokers_response_bytes_bucket{broker="1001",le="0.99"} 60
akvorado_inlet_kafka_brokers_response_bytes_bucket{broker="1001",le="+Inf"} 26
akvorado_inlet_kafka_brokers_response_bytes_sum{broker="1001"} 1516
akvorado_inlet_kafka_brokers_response_bytes_count{broker="1001"} 26
akvorado_inlet_kafka_brokers_response_rate{broker="1001"} 0.09720276637938277
akvorado_inlet_kafka_producer_batch_bytes_bucket{le="0.5"} 2210
akvorado_inlet_kafka_producer_batch_bytes_bucket{le="0.9"} 2415
akvorado_inlet_kafka_producer_batch_bytes_bucket{le="0.99"} 2557
akvorado_inlet_kafka_producer_batch_bytes_bucket{le="+Inf"} 25
akvorado_inlet_kafka_producer_batch_bytes_sum 51361
akvorado_inlet_kafka_producer_batch_bytes_count 25
akvorado_inlet_kafka_producer_compression_ratio_bucket{le="0.5"} 650
akvorado_inlet_kafka_producer_compression_ratio_bucket{le="0.9"} 700
akvorado_inlet_kafka_producer_compression_ratio_bucket{le="0.99"} 723
akvorado_inlet_kafka_producer_compression_ratio_bucket{le="+Inf"} 25
akvorado_inlet_kafka_producer_compression_ratio_sum 16262
akvorado_inlet_kafka_producer_compression_ratio_count 25
akvorado_inlet_kafka_producer_record_send_rate 4.309717029506775
akvorado_inlet_kafka_producer_records_per_request_bucket{le="0.5"} 44
akvorado_inlet_kafka_producer_records_per_request_bucket{le="0.9"} 48
akvorado_inlet_kafka_producer_records_per_request_bucket{le="0.99"} 51
akvorado_inlet_kafka_producer_records_per_request_bucket{le="+Inf"} 25
akvorado_inlet_kafka_producer_records_per_request_sum 1023
akvorado_inlet_kafka_producer_records_per_request_count 25
akvorado_inlet_kafka_sent_bytes_total{exporter="1.1.1.1"} 138354
akvorado_inlet_kafka_sent_bytes_total{exporter="2.2.2.2"} 170622
akvorado_inlet_kafka_sent_messages_total{exporter="1.1.1.1"} 460
akvorado_inlet_kafka_sent_messages_total{exporter="2.2.2.2"} 597
akvorado_inlet_snmp_cache_expired 0
akvorado_inlet_snmp_cache_exporters 2
akvorado_inlet_snmp_cache_hit 2131
akvorado_inlet_snmp_cache_miss 351
akvorado_inlet_snmp_cache_refresh 0
akvorado_inlet_snmp_cache_refresh_runs 2
akvorado_inlet_snmp_cache_size 22
akvorado_inlet_snmp_poller_coalesced_count 330
akvorado_inlet_snmp_poller_failure_requests{error="ifalias missing",exporter="172.20.0.1"} 195
akvorado_inlet_snmp_poller_failure_requests{error="ifdescr missing",exporter="172.20.0.1"} 195
akvorado_inlet_snmp_poller_failure_requests{error="ifspeed missing",exporter="172.20.0.1"} 195
akvorado_inlet_snmp_poller_pending_requests 0
akvorado_inlet_snmp_poller_seconds{exporter="1.1.1.1",quantile="0.5"} 0.018010881
akvorado_inlet_snmp_poller_seconds{exporter="1.1.1.1",quantile="0.9"} 0.018153703
akvorado_inlet_snmp_poller_seconds{exporter="1.1.1.1",quantile="0.99"} 0.018153703
akvorado_inlet_snmp_poller_seconds_sum{exporter="1.1.1.1"} 0.07203823100000001
akvorado_inlet_snmp_poller_seconds_count{exporter="1.1.1.1"} 4
akvorado_inlet_snmp_poller_seconds{exporter="172.20.0.1",quantile="0.5"} 0.00028853
akvorado_inlet_snmp_poller_seconds{exporter="172.20.0.1",quantile="0.9"} 0.000380991
akvorado_inlet_snmp_poller_seconds{exporter="172.20.0.1",quantile="0.99"} 0.010112424
akvorado_inlet_snmp_poller_seconds_sum{exporter="172.20.0.1"} 0.056360141000000016
akvorado_inlet_snmp_poller_seconds_count{exporter="172.20.0.1"} 110
akvorado_inlet_snmp_poller_seconds{exporter="2.2.2.2",quantile="0.5"} 0.009843966
akvorado_inlet_snmp_poller_seconds{exporter="2.2.2.2",quantile="0.9"} 0.009946579
akvorado_inlet_snmp_poller_seconds{exporter="2.2.2.2",quantile="0.99"} 0.010002395
akvorado_inlet_snmp_poller_seconds_sum{exporter="2.2.2.2"} 0.157686227
akvorado_inlet_snmp_poller_seconds_count{exporter="2.2.2.2"} 16
akvorado_inlet_snmp_poller_success_requests{exporter="1.1.1.1"} 5
akvorado_inlet_snmp_poller_success_requests{exporter="2.2.2.2"} 17
root@test-box:/opt/akvorado#
docker-compose exec clickhouse clickhouse-client
ClickHouse client version 22.3.8.39 (official build).
Connecting to localhost:9000 as user default.
Connected to ClickHouse server version 22.3.8 revision 54455.
3d5b47e40d5c :) show tables
SHOW TABLES
Query id: 8fc4a98f-5722-44ba-8e9d-e85b14ae5a27
┌─name───────────────────────────────────────────┐
│ .inner_id.a1b7d466-f23d-4e6b-8a52-dcb46fe4174a │
│ asns │
│ exporters │
│ flows │
│ flows_1h0m0s │
│ flows_1h0m0s_consumer │
│ flows_1m0s │
│ flows_1m0s_consumer │
│ flows_5m0s │
│ flows_5m0s_consumer │
│ networks │
│ protocols │
└────────────────────────────────────────────────┘
12 rows in set. Elapsed: 0.002 sec.
3d5b47e40d5c :) show create table flow_1_raw
SHOW CREATE TABLE flow_1_raw
Query id: 33ac3e17-c6bb-4837-a3c8-81bc946ea80a
0 rows in set. Elapsed: 0.001 sec.
Received exception from server (version 22.3.8):
Code: 390. DB::Exception: Received from localhost:9000. DB::Exception: Table flow_1_raw
doesn't exist. (CANNOT_GET_CREATE_TABLE_QUERY)
3d5b47e40d5c :) select * from system.errors order by last_error_time desc limit 5 format Vertical
SELECT *
FROM system.errors
ORDER BY last_error_time DESC
LIMIT 5
FORMAT Vertical
Query id: 0d0e6904-e15e-44d7-85cd-d5d34e423fd2
Row 1:
──────
name: BAD_ARGUMENTS
code: 36
value: 17
last_error_time: 2022-07-16 14:28:44
last_error_message: Number of consumers can not be bigger than 2
last_error_trace: [188171898,355476465,355473321,368396378,362775801,362754591,362789451,366162671,366153109,378257466,378321305,431538415,431547713,433370377,433358944,139840717014537,139840716116275]
remote: 0
Row 2:
──────
name: FILE_DOESNT_EXIST
code: 107
value: 1
last_error_time: 2022-07-16 14:28:31
last_error_message: Cannot open file /var/lib/clickhouse/store/2dc/2dc5bbb2-1664-48b9-ba20-3b33ee3c6f2c/flow_1_raw.sql, errno: 2, strerror: No such file or directory
last_error_trace: [188171898,188177002,354874105,354887995,354886971,363800863,363799574,366162671,366153109,378257466,378321305,431538415,431547713,433370377,433358944,139840717014537,139840716116275]
remote: 0
Row 3:
──────
name: CANNOT_GET_CREATE_TABLE_QUERY
code: 390
value: 1
last_error_time: 2022-07-16 14:28:31
last_error_message: Table flow_1_raw
doesn't exist
last_error_trace: [188171898,354887525,363800863,363799574,366162671,366153109,378257466,378321305,431538415,431547713,433370377,433358944,139840717014537,139840716116275]
remote: 0
Row 4:
──────
name: CANNOT_OPEN_FILE
code: 76
value: 1
last_error_time: 2022-07-16 14:14:36
last_error_message: Cannot open certificate file: /etc/clickhouse-server/server.crt.
last_error_trace: [188171898,375898552,375895335,188828972,188662635,188763399,188731818,431639526,188667828,188657031,188148298,139840715088003,185963502]
remote: 0
4 rows in set. Elapsed: 0.003 sec.
3d5b47e40d5c :)
There are missing tables, check what you get with:
docker-compose logs akvorado-orchestrator | grep migration
I think that the error comes from Number of consumers can not be bigger than 2
. Did you set a particular value for the number of consumers in the configuration? It is either bounded by the number of CPUs or by the number of partitions for the Kafka topic.
This message is repeated fairly often -
akvorado-orchestrator_1 | {"level":"error","step":"create raw flows table","error":"code: 36, message: Number of consumers can not be bigger than 2","time":"2022-07-16T14:34:44Z","caller":"akvorado/orchestrator/clickhouse/migrations.go:106","module":"akvorado/orchestrator/clickhouse","message":"cannot execute migration step"}
I haven't configured anything outside of your documentation, basically geoIP details and removing fake-exporters are the only changes
The virtual-machine it's running on only has 2 v-cpu
Yes I ran the command: curl -s http://akvorado/api/v0/inlet/flows\?limit=1
This is the command that never completes and just sits there until I kill
it off. The second command $ curl -s http://akvorado/api/v0/inlet/metrics
| grep '^akvorado_inlet' returns data instantly.
akvorado_inlet_core_classifier_cache_hits 0
akvorado_inlet_core_classifier_cache_misses 0
akvorado_inlet_core_flows_errors{error="SNMP cache
miss",exporter="103.118.x.x"} 1.092484e+06
akvorado_inlet_core_flows_errors{error="SNMP cache
miss",exporter="103.118.x.x"} 251405
akvorado_inlet_core_flows_errors{error="SNMP cache
miss",exporter="172.20.0.1"} 18357
akvorado_inlet_core_flows_errors{error="input and output interfaces
missing",exporter="103.118.x.x"} 620
akvorado_inlet_core_flows_errors{error="input and output interfaces
missing",exporter="103.118.x.x"} 1
akvorado_inlet_core_flows_http_clients 0
akvorado_inlet_core_flows_received{exporter="103.118.x.x"} 1.093104e+06
akvorado_inlet_core_flows_received{exporter="103.118.x.x"} 251406
akvorado_inlet_core_flows_received{exporter="172.20.0.1"} 18357
akvorado_inlet_flow_decoder_count{name="netflow"} 408140
akvorado_inlet_flow_decoder_error_count{name="netflow"} 1234
akvorado_inlet_flow_decoder_netflow_count{exporter="103.118.x.x",version="9"}
285922
akvorado_inlet_flow_decoder_netflow_count{exporter="103.118.x.x",version="9"}
116732
akvorado_inlet_flow_decoder_netflow_count{exporter="172.20.0.1",version="9"}
akvorado_inlet_flow_decoder_netflow_delay_summary_seconds{exporter="103.118.x.x",version="9",quantile="0.5"}
akvorado_inlet_flow_decoder_netflow_delay_summary_seconds{exporter="103.118.x.x",version="9",quantile="0.9"}
akvorado_inlet_flow_decoder_netflow_delay_summary_seconds{exporter="103.118.x.x",version="9",quantile="0.99"}
4.295596e+06
akvorado_inlet_flow_decoder_netflow_delay_summary_seconds_sum{exporter="103.118.x.x",version="9"}
4.7629673497e+10
akvorado_inlet_flow_decoder_netflow_delay_summary_seconds_count{exporter="103.118.x.x",version="9"}
1.093104e+06
akvorado_inlet_flow_decoder_netflow_delay_summary_seconds{exporter="103.118.x.x",version="9",quantile="0.5"}
akvorado_inlet_flow_decoder_netflow_delay_summary_seconds{exporter="103.118.x.x",version="9",quantile="0.9"}
akvorado_inlet_flow_decoder_netflow_delay_summary_seconds{exporter="103.118.x.x",version="9",quantile="0.99"}
4.295604e+06
akvorado_inlet_flow_decoder_netflow_delay_summary_seconds_sum{exporter="103.118.x.x",version="9"}
1.4100228921e+10
akvorado_inlet_flow_decoder_netflow_delay_summary_seconds_count{exporter="103.118.x.x",version="9"}
251406
akvorado_inlet_flow_decoder_netflow_delay_summary_seconds{exporter="172.20.0.1",version="9",quantile="0.5"}
akvorado_inlet_flow_decoder_netflow_delay_summary_seconds{exporter="172.20.0.1",version="9",quantile="0.9"}
akvorado_inlet_flow_decoder_netflow_delay_summary_seconds{exporter="172.20.0.1",version="9",quantile="0.99"}
akvorado_inlet_flow_decoder_netflow_delay_summary_seconds_sum{exporter="172.20.0.1",version="9"}
8.80048738e+08
akvorado_inlet_flow_decoder_netflow_delay_summary_seconds_count{exporter="172.20.0.1",version="9"}
18357
akvorado_inlet_flow_decoder_netflow_errors_count{error="template not
found",exporter="103.118.x.x"} 765
akvorado_inlet_flow_decoder_netflow_errors_count{error="template not
found",exporter="103.118.x.x"} 6
akvorado_inlet_flow_decoder_netflow_errors_count{error="template not
found",exporter="172.20.0.1"} 463
akvorado_inlet_flow_decoder_netflow_flowset_records_sum{exporter="103.118.x.x",type="DataFlowSet",version="9"}
1.093104e+06
akvorado_inlet_flow_decoder_netflow_flowset_records_sum{exporter="103.118.x.x",type="OptionsDataFlowSet",version="9"}
akvorado_inlet_flow_decoder_netflow_flowset_records_sum{exporter="103.118.x.x",type="OptionsTemplateFlowSet",version="9"}
akvorado_inlet_flow_decoder_netflow_flowset_records_sum{exporter="103.118.x.x",type="TemplateFlowSet",version="9"}
akvorado_inlet_flow_decoder_netflow_flowset_records_sum{exporter="103.118.x.x",type="DataFlowSet",version="9"}
251406
akvorado_inlet_flow_decoder_netflow_flowset_records_sum{exporter="103.118.x.x",type="OptionsDataFlowSet",version="9"}
akvorado_inlet_flow_decoder_netflow_flowset_records_sum{exporter="103.118.x.x",type="OptionsTemplateFlowSet",version="9"}
akvorado_inlet_flow_decoder_netflow_flowset_records_sum{exporter="103.118.x.x",type="TemplateFlowSet",version="9"}
akvorado_inlet_flow_decoder_netflow_flowset_records_sum{exporter="172.20.0.1",type="DataFlowSet",version="9"}
18357
akvorado_inlet_flow_decoder_netflow_flowset_records_sum{exporter="172.20.0.1",type="TemplateFlowSet",version="9"}
akvorado_inlet_flow_decoder_netflow_flowset_sum{exporter="103.118.x.x",type="DataFlowSet",version="9"}
284877
akvorado_inlet_flow_decoder_netflow_flowset_sum{exporter="103.118.x.x",type="OptionsDataFlowSet",version="9"}
akvorado_inlet_flow_decoder_netflow_flowset_sum{exporter="103.118.x.x",type="OptionsTemplateFlowSet",version="9"}
akvorado_inlet_flow_decoder_netflow_flowset_sum{exporter="103.118.x.x",type="TemplateFlowSet",version="9"}
akvorado_inlet_flow_decoder_netflow_flowset_sum{exporter="103.118.x.x",type="DataFlowSet",version="9"}
115745
akvorado_inlet_flow_decoder_netflow_flowset_sum{exporter="103.118.x.x",type="OptionsDataFlowSet",version="9"}
akvorado_inlet_flow_decoder_netflow_flowset_sum{exporter="103.118.x.x",type="OptionsTemplateFlowSet",version="9"}
akvorado_inlet_flow_decoder_netflow_flowset_sum{exporter="103.118.x.x",type="TemplateFlowSet",version="9"}
akvorado_inlet_flow_decoder_netflow_flowset_sum{exporter="172.20.0.1",type="DataFlowSet",version="9"}
akvorado_inlet_flow_decoder_netflow_flowset_sum{exporter="172.20.0.1",type="TemplateFlowSet",version="9"}
akvorado_inlet_flow_decoder_netflow_templates_count{exporter="103.118.x.x",obs_domain_id="524288",template_id="320",type="template",version="9"}
akvorado_inlet_flow_decoder_netflow_templates_count{exporter="103.118.x.x",obs_domain_id="524288",template_id="576",type="options_template",version="9"}
akvorado_inlet_flow_decoder_netflow_templates_count{exporter="103.118.x.x",obs_domain_id="524288",template_id="320",type="template",version="9"}
akvorado_inlet_flow_decoder_netflow_templates_count{exporter="103.118.x.x",obs_domain_id="524288",template_id="576",type="options_template",version="9"}
akvorado_inlet_flow_decoder_netflow_templates_count{exporter="172.20.0.1",obs_domain_id="524288",template_id="320",type="template",version="9"}
akvorado_inlet_flow_input_udp_bytes{exporter="103.118.x.x",listener="
0.0.0.0:2055",worker="0"} 9.6471576e+07
akvorado_inlet_flow_input_udp_bytes{exporter="103.118.x.x",listener="
0.0.0.0:2055",worker="0"} 2.3393172e+07
akvorado_inlet_flow_input_udp_bytes{exporter="172.20.0.1",listener="
0.0.0.0:2055",worker="0"} 230436
akvorado_inlet_flow_input_udp_bytes{exporter="172.20.0.1",listener="
0.0.0.0:2055",worker="2"} 586048
akvorado_inlet_flow_input_udp_bytes{exporter="172.20.0.1",listener="
0.0.0.0:2055",worker="3"} 9690
akvorado_inlet_flow_input_udp_bytes{exporter="172.20.0.1",listener="
0.0.0.0:2055",worker="4"} 810524
akvorado_inlet_flow_input_udp_in_drops{listener="0.0.0.0:2055",worker="0"} 0
akvorado_inlet_flow_input_udp_in_drops{listener="0.0.0.0:2055",worker="2"} 0
akvorado_inlet_flow_input_udp_in_drops{listener="0.0.0.0:2055",worker="3"} 0
akvorado_inlet_flow_input_udp_in_drops{listener="0.0.0.0:2055",worker="4"} 0
akvorado_inlet_flow_input_udp_packets{exporter="103.118.x.x",listener="
0.0.0.0:2055",worker="0"} 284877
akvorado_inlet_flow_input_udp_packets{exporter="103.118.x.x",listener="
0.0.0.0:2055",worker="0"} 115745
akvorado_inlet_flow_input_udp_packets{exporter="172.20.0.1",listener="
0.0.0.0:2055",worker="0"} 1135
akvorado_inlet_flow_input_udp_packets{exporter="172.20.0.1",listener="
0.0.0.0:2055",worker="2"} 1896
akvorado_inlet_flow_input_udp_packets{exporter="172.20.0.1",listener="
0.0.0.0:2055",worker="3"} 45
akvorado_inlet_flow_input_udp_packets{exporter="172.20.0.1",listener="
0.0.0.0:2055",worker="4"} 2400
akvorado_inlet_flow_input_udp_summary_size_bytes{exporter="103.118.x.x",listener="
0.0.0.0:2055",worker="0",quantile="0.5"} 352
akvorado_inlet_flow_input_udp_summary_size_bytes{exporter="103.118.x.x",listener="
0.0.0.0:2055",worker="0",quantile="0.9"} 352
akvorado_inlet_flow_input_udp_summary_size_bytes{exporter="103.118.x.x",listener="
0.0.0.0:2055",worker="0",quantile="0.99"} 352
akvorado_inlet_flow_input_udp_summary_size_bytes_sum{exporter="103.118.x.x",listener="
0.0.0.0:2055",worker="0"} 9.6471576e+07
akvorado_inlet_flow_input_udp_summary_size_bytes_count{exporter="103.118.x.x",listener="
0.0.0.0:2055",worker="0"} 284877
akvorado_inlet_flow_input_udp_summary_size_bytes{exporter="103.118.x.x",listener="
0.0.0.0:2055",worker="0",quantile="0.5"} 188
akvorado_inlet_flow_input_udp_summary_size_bytes{exporter="103.118.x.x",listener="
0.0.0.0:2055",worker="0",quantile="0.9"} 352
akvorado_inlet_flow_input_udp_summary_size_bytes{exporter="103.118.x.x",listener="
0.0.0.0:2055",worker="0",quantile="0.99"} 352
akvorado_inlet_flow_input_udp_summary_size_bytes_sum{exporter="103.118.x.x",listener="
0.0.0.0:2055",worker="0"} 2.3393172e+07
akvorado_inlet_flow_input_udp_summary_size_bytes_count{exporter="103.118.x.x",listener="
0.0.0.0:2055",worker="0"} 115745
akvorado_inlet_flow_input_udp_summary_size_bytes{exporter="172.20.0.1",listener="
0.0.0.0:2055",worker="0",quantile="0.5"} NaN
akvorado_inlet_flow_input_udp_summary_size_bytes{exporter="172.20.0.1",listener="
0.0.0.0:2055",worker="0",quantile="0.9"} NaN
akvorado_inlet_flow_input_udp_summary_size_bytes{exporter="172.20.0.1",listener="
0.0.0.0:2055",worker="0",quantile="0.99"} NaN
akvorado_inlet_flow_input_udp_summary_size_bytes_sum{exporter="172.20.0.1",listener="
0.0.0.0:2055",worker="0"} 230436
akvorado_inlet_flow_input_udp_summary_size_bytes_count{exporter="172.20.0.1",listener="
0.0.0.0:2055",worker="0"} 1135
akvorado_inlet_flow_input_udp_summary_size_bytes{exporter="172.20.0.1",listener="
0.0.0.0:2055",worker="2",quantile="0.5"} NaN
akvorado_inlet_flow_input_udp_summary_size_bytes{exporter="172.20.0.1",listener="
0.0.0.0:2055",worker="2",quantile="0.9"} NaN
akvorado_inlet_flow_input_udp_summary_size_bytes{exporter="172.20.0.1",listener="
0.0.0.0:2055",worker="2",quantile="0.99"} NaN
akvorado_inlet_flow_input_udp_summary_size_bytes_sum{exporter="172.20.0.1",listener="
0.0.0.0:2055",worker="2"} 586048
akvorado_inlet_flow_input_udp_summary_size_bytes_count{exporter="172.20.0.1",listener="
0.0.0.0:2055",worker="2"} 1896
akvorado_inlet_flow_input_udp_summary_size_bytes{exporter="172.20.0.1",listener="
0.0.0.0:2055",worker="3",quantile="0.5"} NaN
akvorado_inlet_flow_input_udp_summary_size_bytes{exporter="172.20.0.1",listener="
0.0.0.0:2055",worker="3",quantile="0.9"} NaN
akvorado_inlet_flow_input_udp_summary_size_bytes{exporter="172.20.0.1",listener="
0.0.0.0:2055",worker="3",quantile="0.99"} NaN
akvorado_inlet_flow_input_udp_summary_size_bytes_sum{exporter="172.20.0.1",listener="
0.0.0.0:2055",worker="3"} 9690
akvorado_inlet_flow_input_udp_summary_size_bytes_count{exporter="172.20.0.1",listener="
0.0.0.0:2055",worker="3"} 45
akvorado_inlet_flow_input_udp_summary_size_bytes{exporter="172.20.0.1",listener="
0.0.0.0:2055",worker="4",quantile="0.5"} NaN
akvorado_inlet_flow_input_udp_summary_size_bytes{exporter="172.20.0.1",listener="
0.0.0.0:2055",worker="4",quantile="0.9"} NaN
akvorado_inlet_flow_input_udp_summary_size_bytes{exporter="172.20.0.1",listener="
0.0.0.0:2055",worker="4",quantile="0.99"} NaN
akvorado_inlet_flow_input_udp_summary_size_bytes_sum{exporter="172.20.0.1",listener="
0.0.0.0:2055",worker="4"} 810524
akvorado_inlet_flow_input_udp_summary_size_bytes_count{exporter="172.20.0.1",listener="
0.0.0.0:2055",worker="4"} 2400
akvorado_inlet_flow_summary_decoding_time_seconds{name="netflow",quantile="0.5"}
2.0909e-05
akvorado_inlet_flow_summary_decoding_time_seconds{name="netflow",quantile="0.9"}
3.5289e-05
akvorado_inlet_flow_summary_decoding_time_seconds{name="netflow",quantile="0.99"}
5.3262000000000006e-05
akvorado_inlet_flow_summary_decoding_time_seconds_sum{name="netflow"}
9.975279888000186
akvorado_inlet_flow_summary_decoding_time_seconds_count{name="netflow"}
408140
akvorado_inlet_geoip_db_refresh_total{database="asn"} 1
akvorado_inlet_geoip_db_refresh_total{database="country"} 1
akvorado_inlet_snmp_cache_expired 0
akvorado_inlet_snmp_cache_exporters 0
akvorado_inlet_snmp_cache_hit 0
akvorado_inlet_snmp_cache_miss 2.622388e+06
akvorado_inlet_snmp_cache_refresh 0
akvorado_inlet_snmp_cache_refresh_runs 157
akvorado_inlet_snmp_cache_size 0
akvorado_inlet_snmp_poller_breaker_open_count{exporter="103.118.x.x"} 963349
akvorado_inlet_snmp_poller_breaker_open_count{exporter="103.118.x.x"} 203378
akvorado_inlet_snmp_poller_breaker_open_count{exporter="172.20.0.1"} 16379
akvorado_inlet_snmp_poller_coalesced_count 2.515993e+06
akvorado_inlet_snmp_poller_failure_requests{error="get",exporter="103.118.x.x"}
akvorado_inlet_snmp_poller_failure_requests{error="get",exporter="103.118.x.x"}
akvorado_inlet_snmp_poller_failure_requests{error="get",exporter="172.20.0.1"}
akvorado_inlet_snmp_poller_pending_requests 5
akvorado_inlet_snmp_poller_retry_requests{exporter="103.118.x.x"} 16174
akvorado_inlet_snmp_poller_retry_requests{exporter="103.118.x.x"} 12934
akvorado_inlet_snmp_poller_retry_requests{exporter="172.20.0.1"} 232
On Mon, Jul 18, 2022 at 3:28 PM Vincent Bernat ***@***.***> wrote:
Look at the metrics. The troublesooting doc
<
https://github.com/vincentbernat/akvorado/blob/main/console/data/docs/05-troubleshooting.md>
has been updated recently. Be sure to check it.
Reply to this email directly, view it on GitHub
<
#6 (reply in thread)>,
or unsubscribe
<
https://github.com/notifications/unsubscribe-auth/AVKJAOWOZTNITDKRPZG2PPDVUTTOBANCNFSM53TMM4MA>
You are receiving this because you commented.Message ID:
***@***.***>
root@flow:/docker/akvorado# curl -s http://localhost:8081/api/v0/inlet/metrics | grep '^akvorado_inlet' | grep _errors
akvorado_inlet_core_flows_errors{error="SNMP cache miss",exporter="103.118.x.x"} 1.099362e+06
akvorado_inlet_core_flows_errors{error="SNMP cache miss",exporter="103.118.x.x"} 252977
akvorado_inlet_core_flows_errors{error="SNMP cache miss",exporter="172.20.0.1"} 18357
akvorado_inlet_core_flows_errors{error="input and output interfaces missing",exporter="103.118.x.x"} 620
akvorado_inlet_core_flows_errors{error="input and output interfaces missing",exporter="103.118.x.x"} 1
akvorado_inlet_flow_decoder_netflow_errors_count{error="template not found",exporter="103.118.x.x"} 765
akvorado_inlet_flow_decoder_netflow_errors_count{error="template not found",exporter="103.118.x.x"} 6
akvorado_inlet_flow_decoder_netflow_errors_count{error="template not found",exporter="172.20.0.1"} 463
root@flow:/docker/akvorado# curl -s http://localhost:8081/api/v0/inlet/metrics | grep '^akvorado_inlet_core_flows_forwarded'
root@flow:/docker/akvorado#
root@flow:/docker/akvorado# curl -s http://localhost:8081/api/v0/inlet/flows\?limit=1
root@flow:/docker/akvorado# ^C
I have it configured now. I did not see anywhere it was a requirement for
the system to work.
Flows are showing up now, just need to give it some time to see if it's
correct.