The goal is to get the average power (watt) per minute for a set of hosts for the past week, day, hour, 5 min. I’m hoping I can get some help with performance issues as compared to the same setup in ElasticSearch.
What I’m observing is that the time taken to retrieve this information for the exact same data set on the exact same hardware Timescale takes 5x longer (~4000 ms) vs. ElasticSearch (~800 ms). Even limiting the query to the last day Timescale takes 2.5x as long (~1100ms) vs. ElasticSearch (~426 ms).
I have tried various combinations of the below settings, and while some combinations do improve performance slightly it is not enough of an improvement to resolve this issue.
- Chunk Interval: 1 hour, 1 day, 1 week
- Replication-factor: 1, 3
- Partition Column: series_id, NONE
- Regular Hypertable
Regardless of the combination of settings used above, the problem seems to always center on the Custom Scan (AsyncAppend) step taking nearly all the time.
Setup:
- Timescale: 1 Access Node, 3 data nodes
- ElasticSearch: 4 nodes
Hardware Summary:
- CPU: 2.5 GHz, 32 cores (AMD Epyc)
- Memory: 256 GB DDR4
- Disk: 3 TB SSD
- Network: 10 Gbps
In addition I have setup Timescale running in docker containers (1 access, 3 data) on my Macbook Pro (Intel version), dumped the data from my test system and loaded it into my local setup. While running it this way does improve performance (as expected due to lower network overhead, contention etc.) it is still 2x slower than ElasticSearch.
Timescale Design Summary:
- Metric data is separated from metadata (schema is setup like Promscale with some modifications)
- Timestamps are UNIX epoch with millisecond resolution
- Row count: 20062588
- Replication-factor: 3
- Compression: FALSE
- Partition: series_id
- Chunk Interval: 8600000
- Timescale Version: 2.6.0
- Postgres Version: 14.2
ElasticSearch Design Summary:
- An index for each day
- Metadata and metric data stored together as a single flat json doc/object
- Replication-factor: 3
- Version: 6.8.23
Query:
Timescale does not resolve an expression and ship the results to the data nodes so we instead convert to an array of ID’s which can be shipped. We use a CTE for situations where we may want to group by and return other labels with the result.
with power_series_data as (
select id, labels
from series_telemetry_power s
where
s.labels && (select coalesce(array_agg(l.id), array[]::int[]) from label l where l.key = 'Location' and substring(l.value from '^x[0-9]+c[0-9]+s[0-9]+b[0-9]+$') in ('x9000c1s0b0','x9000c1s0b1','x9000c1s1b0','x9000c1s1b1','x9000c1s2b0','x9000c1s2b1','x9000c1s4b0','x9000c1s4b1')))
select
time_bucket(60000, m.ts) time,
avg(val) Power
from telemetry_power m
where
array[m.series_id] && (select coalesce(array_agg(id), array[]::integer[]) from power_series_data)
and m.ts between 1650997919334 and 1651602719334
group by time
order by 1 desc;
Plan:
All tables vacuum analyze before running
set track_io_timing = on;
explain (analyze, buffers, settings, verbose);
QUERY PLAN
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Finalize GroupAggregate (cost=431047.77..431055.27 rows=200 width=16) (actual time=3823.647..3841.920 rows=10078 loops=1)
Output: (time_bucket('60000'::bigint, ts)), avg(val)
Group Key: (time_bucket('60000'::bigint, ts))
Buffers: shared hit=19
InitPlan 2 (returns $1)
-> Aggregate (cost=8.68..8.69 rows=1 width=32) (actual time=0.189..0.194 rows=1 loops=1)
Output: COALESCE(array_agg(s.id), '{}'::integer[])
Buffers: shared hit=19
InitPlan 1 (returns $0)
-> Aggregate (cost=2.19..2.20 rows=1 width=32) (actual time=0.104..0.106 rows=1 loops=1)
Output: COALESCE(array_agg(l.id), '{}'::integer[])
Buffers: shared hit=3
-> Index Only Scan using label_key_value_id_key on public.label l (cost=0.27..2.18 rows=1 width=4) (actual time=0.057..0.097 rows=8 loops=1)
Output: l.key, l.value, l.id
Index Cond: (l.key = 'Location'::text)
Filter: ("substring"(l.value, '^x[0-9]+c[0-9]+s[0-9]+b[0-9]+$'::text) = ANY ('{x9000c1s0b0,x9000c1s0b1,x9000c1s1b0,x9000c1s1b1,x9000c1s2b0,x9000c1s2b1,x9000c1s4b0,x9000c1s4b1}'::text[]))
Rows Removed by Filter: 19
Heap Fetches: 0
Buffers: shared hit=3
-> Bitmap Heap Scan on public.series_telemetry_power s (cost=2.23..6.47 rows=4 width=4) (actual time=0.155..0.181 rows=40 loops=1)
Output: s.id, s.metric_id, s.labels, s.delete_epoch
Recheck Cond: ((s.labels)::integer[] && $0)
Heap Blocks: exact=7
Buffers: shared hit=19
-> Bitmap Index Scan on series_telemetry_power_labels_idx (cost=0.00..2.23 rows=4 width=0) (actual time=0.144..0.145 rows=40 loops=1)
Index Cond: ((s.labels)::integer[] && $0)
Buffers: shared hit=12
-> Sort (cost=431039.08..431040.58 rows=600 width=40) (actual time=3823.626..3830.866 rows=30234 loops=1)
Output: (time_bucket('60000'::bigint, ts)), (PARTIAL avg(val))
Sort Key: (time_bucket('60000'::bigint, ts)) DESC
Sort Method: quicksort Memory: 5020kB
Buffers: shared hit=19
-> Custom Scan (AsyncAppend) (cost=312.60..431011.40 rows=600 width=40) (actual time=3802.641..3813.805 rows=30234 loops=1)
Output: (time_bucket('60000'::bigint, ts)), (PARTIAL avg(val))
Buffers: shared hit=19
-> Append (cost=312.60..431011.40 rows=600 width=40) (actual time=0.008..7.325 rows=30234 loops=1)
-> Custom Scan (DataNodeScan) (cost=312.60..103004.65 rows=200 width=40) (actual time=0.005..1.317 rows=10078 loops=1)
Output: (time_bucket('60000'::bigint, m.ts)), (PARTIAL avg(m.val))
Relations: Aggregate on (public.telemetry_power m)
Data node: dn_ld01
Fetcher Type: Row by row
Chunks: _dist_hyper_125_59475_chunk, _dist_hyper_125_59474_chunk, _dist_hyper_125_59473_chunk, _dist_hyper_125_59472_chunk, _dist_hyper_125_59471_chunk, _dist_hyper_125_59470_chunk, _dist_hyper_125_59469_chunk, _dist_hyper_125_59468_chunk, _dist_hyper_125_59467_chunk, _dist_hyper_125_59466_chunk, _dist_hyper_125_59465_chunk, _dist_hyper_125_59464_chunk, _dist_hyper_125_59463_chunk, _dist_hyper_125_59462_chunk, _dist_hyper_125_59461_chunk, _dist_hyper_125_59460_chunk, _dist_hyper_125_59459_chunk, _dist_hyper_125_59458_chunk, _dist_hyper_125_59457_chunk, _dist_hyper_125_59456_chunk, _dist_hyper_125_59455_chunk, _dist_hyper_125_59454_chunk, _dist_hyper_125_59453_chunk, _dist_hyper_125_59452_chunk, _dist_hyper_125_59451_chunk, _dist_hyper_125_59450_chunk, _dist_hyper_125_59449_chunk, _dist_hyper_125_59448_chunk, _dist_hyper_125_59447_chunk, _dist_hyper_125_59446_chunk, _dist_hyper_125_59445_chunk, _dist_hyper_125_59444_chunk, _dist_hyper_125_59443_chunk, _dist_hyper_125_59442_chunk, _dist_hyper_125_59441_chunk, _dist_hyper_125_59440_chunk, _dist_hyper_125_59439_chunk, _dist_hyper_125_59438_chunk, _dist_hyper_125_59437_chunk, _dist_hyper_125_59436_chunk, _dist_hyper_125_59435_chunk, _dist_hyper_125_59434_chunk, _dist_hyper_125_59433_chunk, _dist_hyper_125_59432_chunk, _dist_hyper_125_59431_chunk, _dist_hyper_125_59430_chunk, _dist_hyper_125_59429_chunk, _dist_hyper_125_59428_chunk, _dist_hyper_125_59427_chunk, _dist_hyper_125_59426_chunk, _dist_hyper_125_59425_chunk, _dist_hyper_125_59424_chunk, _dist_hyper_125_59423_chunk, _dist_hyper_125_59422_chunk, _dist_hyper_125_59421_chunk, _dist_hyper_125_59420_chunk, _dist_hyper_125_59419_chunk, _dist_hyper_125_59418_chunk, _dist_hyper_125_59417_chunk, _dist_hyper_125_59416_chunk, _dist_hyper_125_59415_chunk, _dist_hyper_125_59414_chunk, _dist_hyper_125_59413_chunk, _dist_hyper_125_59412_chunk, _dist_hyper_125_59411_chunk, _dist_hyper_125_59410_chunk, _dist_hyper_125_59409_chunk, _dist_hyper_125_59408_chunk, _dist_hyper_125_59407_chunk, _dist_hyper_125_59405_chunk, _dist_hyper_125_59404_chunk
Remote SQL: SELECT public.time_bucket(60000::bigint, ts), _timescaledb_internal.partialize_agg(avg(val)) FROM public.telemetry_power WHERE _timescaledb_internal.chunks_in(public.telemetry_power.*, ARRAY[96682, 96681, 96680, 96679, 96678, 96677, 96676, 96675, 96674, 96673, 96672, 96671, 96670, 96669, 96668, 96667, 96666, 96665, 96664, 96663, 96662, 96661, 96660, 96659, 96658, 96657, 96656, 96655, 96654, 96653, 96652, 96651, 96650, 96649, 96648, 96647, 96646, 96645, 96644, 96643, 96642, 96641, 96640, 96639, 96638, 96637, 96636, 96635, 96634, 96633, 96632, 96631, 96630, 96629, 96628, 96627, 96626, 96625, 96624, 96623, 96622, 96621, 96620, 96619, 96618, 96617, 96616, 96615, 96614, 96612, 96611]) AND ((ARRAY[series_id] && $1::integer[])) AND ((ts >= 1650997919334::bigint)) AND ((ts <= 1651602719334::bigint)) GROUP BY 1
-> Custom Scan (DataNodeScan) (cost=425.30..157478.07 rows=200 width=40) (actual time=0.002..1.213 rows=10078 loops=1)
Output: (time_bucket('60000'::bigint, m_1.ts)), (PARTIAL avg(m_1.val))
Relations: Aggregate on (public.telemetry_power m)
Data node: dn_ld02
Fetcher Type: Row by row
Chunks: _dist_hyper_125_59564_chunk, _dist_hyper_125_59563_chunk, _dist_hyper_125_59562_chunk, _dist_hyper_125_59561_chunk, _dist_hyper_125_59560_chunk, _dist_hyper_125_59559_chunk, _dist_hyper_125_59558_chunk, _dist_hyper_125_59557_chunk, _dist_hyper_125_59556_chunk, _dist_hyper_125_59555_chunk, _dist_hyper_125_59554_chunk, _dist_hyper_125_59553_chunk, _dist_hyper_125_59552_chunk, _dist_hyper_125_59550_chunk, _dist_hyper_125_59544_chunk, _dist_hyper_125_59543_chunk, _dist_hyper_125_59542_chunk, _dist_hyper_125_59541_chunk, _dist_hyper_125_59540_chunk, _dist_hyper_125_59539_chunk, _dist_hyper_125_59538_chunk, _dist_hyper_125_59537_chunk, _dist_hyper_125_59536_chunk, _dist_hyper_125_59535_chunk, _dist_hyper_125_59534_chunk, _dist_hyper_125_59533_chunk, _dist_hyper_125_59532_chunk, _dist_hyper_125_59531_chunk, _dist_hyper_125_59530_chunk, _dist_hyper_125_59529_chunk, _dist_hyper_125_59528_chunk, _dist_hyper_125_59527_chunk, _dist_hyper_125_59526_chunk, _dist_hyper_125_59525_chunk, _dist_hyper_125_59524_chunk, _dist_hyper_125_59523_chunk, _dist_hyper_125_59522_chunk, _dist_hyper_125_59521_chunk, _dist_hyper_125_59520_chunk, _dist_hyper_125_59519_chunk, _dist_hyper_125_59518_chunk, _dist_hyper_125_59517_chunk, _dist_hyper_125_59516_chunk, _dist_hyper_125_59515_chunk, _dist_hyper_125_59514_chunk, _dist_hyper_125_59513_chunk, _dist_hyper_125_59512_chunk, _dist_hyper_125_59511_chunk, _dist_hyper_125_59510_chunk, _dist_hyper_125_59509_chunk, _dist_hyper_125_59508_chunk, _dist_hyper_125_59507_chunk, _dist_hyper_125_59506_chunk, _dist_hyper_125_59505_chunk, _dist_hyper_125_59504_chunk, _dist_hyper_125_59503_chunk, _dist_hyper_125_59502_chunk, _dist_hyper_125_59500_chunk, _dist_hyper_125_59499_chunk, _dist_hyper_125_59498_chunk, _dist_hyper_125_59497_chunk, _dist_hyper_125_59496_chunk, _dist_hyper_125_59495_chunk, _dist_hyper_125_59494_chunk, _dist_hyper_125_59493_chunk, _dist_hyper_125_59492_chunk, _dist_hyper_125_59491_chunk, _dist_hyper_125_59490_chunk, _dist_hyper_125_59489_chunk, _dist_hyper_125_59487_chunk, _dist_hyper_125_59486_chunk
Remote SQL: SELECT public.time_bucket(60000::bigint, ts), _timescaledb_internal.partialize_agg(avg(val)) FROM public.telemetry_power WHERE _timescaledb_internal.chunks_in(public.telemetry_power.*, ARRAY[96771, 96770, 96769, 96768, 96767, 96766, 96765, 96764, 96763, 96762, 96761, 96760, 96759, 96757, 96751, 96750, 96749, 96748, 96747, 96746, 96745, 96744, 96743, 96742, 96741, 96740, 96739, 96738, 96737, 96736, 96735, 96734, 96733, 96732, 96731, 96730, 96729, 96728, 96727, 96726, 96725, 96724, 96723, 96722, 96721, 96720, 96719, 96718, 96717, 96716, 96715, 96714, 96713, 96712, 96711, 96710, 96709, 96707, 96706, 96705, 96704, 96703, 96702, 96701, 96700, 96699, 96698, 96697, 96696, 96694, 96693]) AND ((ARRAY[series_id] && $1::integer[])) AND ((ts >= 1650997919334::bigint)) AND ((ts <= 1651602719334::bigint)) GROUP BY 1
-> Custom Scan (DataNodeScan) (cost=452.27..170525.68 rows=200 width=40) (actual time=0.001..1.291 rows=10078 loops=1)
Output: (time_bucket('60000'::bigint, m_2.ts)), (PARTIAL avg(m_2.val))
Relations: Aggregate on (public.telemetry_power m)
Data node: dn_ld03
Fetcher Type: Row by row
Chunks: _dist_hyper_125_59673_chunk, _dist_hyper_125_59672_chunk, _dist_hyper_125_59671_chunk, _dist_hyper_125_59670_chunk, _dist_hyper_125_59669_chunk, _dist_hyper_125_59668_chunk, _dist_hyper_125_59667_chunk, _dist_hyper_125_59666_chunk, _dist_hyper_125_59665_chunk, _dist_hyper_125_59664_chunk, _dist_hyper_125_59663_chunk, _dist_hyper_125_59662_chunk, _dist_hyper_125_59661_chunk, _dist_hyper_125_59660_chunk, _dist_hyper_125_59659_chunk, _dist_hyper_125_59658_chunk, _dist_hyper_125_59654_chunk, _dist_hyper_125_59653_chunk, _dist_hyper_125_59652_chunk, _dist_hyper_125_59651_chunk, _dist_hyper_125_59650_chunk, _dist_hyper_125_59649_chunk, _dist_hyper_125_59648_chunk, _dist_hyper_125_59647_chunk, _dist_hyper_125_59646_chunk, _dist_hyper_125_59645_chunk, _dist_hyper_125_59644_chunk, _dist_hyper_125_59643_chunk, _dist_hyper_125_59642_chunk, _dist_hyper_125_59641_chunk, _dist_hyper_125_59640_chunk, _dist_hyper_125_59639_chunk, _dist_hyper_125_59638_chunk, _dist_hyper_125_59637_chunk, _dist_hyper_125_59636_chunk, _dist_hyper_125_59635_chunk, _dist_hyper_125_59634_chunk, _dist_hyper_125_59633_chunk, _dist_hyper_125_59632_chunk, _dist_hyper_125_59631_chunk, _dist_hyper_125_59630_chunk, _dist_hyper_125_59629_chunk, _dist_hyper_125_59628_chunk, _dist_hyper_125_59627_chunk, _dist_hyper_125_59626_chunk, _dist_hyper_125_59625_chunk, _dist_hyper_125_59624_chunk, _dist_hyper_125_59623_chunk, _dist_hyper_125_59622_chunk, _dist_hyper_125_59621_chunk, _dist_hyper_125_59620_chunk, _dist_hyper_125_59619_chunk, _dist_hyper_125_59615_chunk, _dist_hyper_125_59614_chunk, _dist_hyper_125_59613_chunk, _dist_hyper_125_59606_chunk, _dist_hyper_125_59602_chunk, _dist_hyper_125_59601_chunk, _dist_hyper_125_59600_chunk, _dist_hyper_125_59598_chunk, _dist_hyper_125_59597_chunk, _dist_hyper_125_59593_chunk, _dist_hyper_125_59583_chunk, _dist_hyper_125_59582_chunk, _dist_hyper_125_59581_chunk, _dist_hyper_125_59580_chunk, _dist_hyper_125_59579_chunk, _dist_hyper_125_59578_chunk, _dist_hyper_125_59577_chunk, _dist_hyper_125_59575_chunk, _dist_hyper_125_59574_chunk
Remote SQL: SELECT public.time_bucket(60000::bigint, ts), _timescaledb_internal.partialize_agg(avg(val)) FROM public.telemetry_power WHERE _timescaledb_internal.chunks_in(public.telemetry_power.*, ARRAY[96880, 96879, 96878, 96877, 96876, 96875, 96874, 96873, 96872, 96871, 96870, 96869, 96868, 96867, 96866, 96865, 96861, 96860, 96859, 96858, 96857, 96856, 96855, 96854, 96853, 96852, 96851, 96850, 96849, 96848, 96847, 96846, 96845, 96844, 96843, 96842, 96841, 96840, 96839, 96838, 96837, 96836, 96835, 96834, 96833, 96832, 96831, 96830, 96829, 96828, 96827, 96826, 96822, 96821, 96820, 96813, 96809, 96808, 96807, 96805, 96804, 96800, 96790, 96789, 96788, 96787, 96786, 96785, 96784, 96782, 96781]) AND ((ARRAY[series_id] && $1::integer[])) AND ((ts >= 1650997919334::bigint)) AND ((ts <= 1651602719334::bigint)) GROUP BY 1
Settings: effective_cache_size = '161555MB', effective_io_concurrency = '900', enable_partitionwise_aggregate = 'on', jit = 'off', max_parallel_workers = '32', max_parallel_workers_per_gather = '16', random_page_cost = '1.1', work_mem = '16000MB'
Query Identifier: 8067911393521483881
Planning:
Buffers: shared hit=31575
Planning Time: 202.581 ms
Execution Time: 3844.555 ms
(63 rows)
Schema:
Table "public.telemetry_power"
Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
-----------+------------------+-----------+----------+---------+---------+-------------+--------------+-------------
ts | bigint | | not null | | plain | | |
val | double precision | | | | plain | | |
series_id | integer | | | | plain | | |
Indexes:
"telemetry_power_series_id_ts_idx" btree (series_id, ts DESC)
"telemetry_power_ts_idx" btree (ts DESC)
Triggers:
ts_insert_blocker BEFORE INSERT ON telemetry_powerFOR EACH ROW EXECUTE FUNCTION _timescaledb_internal.insert_blocker()
Child tables: <snip>
Access method: heap
Options: autovacuum_vacuum_threshold=50000, autovacuum_analyze_threshold=50000
Table "public.series_telemetry_power"
Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
--------------+-------------+-----------+----------+---------+----------+-------------+--------------+-------------
id | integer | | not null | | plain | | |
metric_id | integer | | not null | | plain | | |
labels | label_array | | not null | | extended | | |
delete_epoch | bigint | | | | plain | | |
Partition of: series FOR VALUES IN (654)
Partition constraint: ((metric_id IS NOT NULL) AND (metric_id = 654))
Indexes:
"series_pkey_654" PRIMARY KEY, btree (id)
"series_telemetry_power_labels_idx" gin (labels)
"series_labels_id_654" UNIQUE CONSTRAINT, btree (labels) INCLUDE (id)
Check constraints:
"series_telemetry_power_labels_check" CHECK (labels[1] = 413 AND labels[1] IS NOT NULL)
"series_telemetry_power_metric_id_check" CHECK (metric_id = 654)
Access method: heap
Options: autovacuum_vacuum_threshold=100, autovacuum_analyze_threshold=100
Table "public.label"
Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
--------+---------+-----------+----------+-----------------------------------+----------+-------------+--------------+-------------
id | integer | | not null | nextval('label_id_seq'::regclass) | plain | | |
key | text | | | | extended | | |
value | text | | | | extended | | |
Indexes:
"label_pkey" PRIMARY KEY, btree (id) INCLUDE (key, value)
"label_key_value_id_key" UNIQUE CONSTRAINT, btree (key, value) INCLUDE (id)
Check constraints:
"label_id_check" CHECK (id > 0)
Access method: heap
Table "public.label_key_position"
Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
-----------------+---------+-----------+----------+---------+----------+-------------+--------------+-------------
metric_category | text | | | | extended | | |
metric_name | text | | | | extended | | |
key | text | | | | extended | | |
pos | integer | | | | plain | | |
Indexes:
"label_key_position_metric_category_metric_name_key_pos_key" UNIQUE CONSTRAINT, btree (metric_category, metric_name, key) INCLUDE (pos)
Access method: heap
ElasticSearch Sample Doc:
"_index" : "telemetry-2022.04.28",
"_type" : "kafka-connect",
"_id" : "telemetry-2022.04.28+1+86136091",
"_score" : 0.0,
"_source" : {
"TelemetrySource" : "nC",
"MessageId" : "Telemetry.Power",
"Timestamp" : 1651176279136,
"Location" : "x9000c1s2b1n0",
"ParentalContext" : "Chassis",
"PhysicalContext" : "VoltageRegulator",
"PhysicalSubContext" : "Input",
"ParentalIndex" : null,
"Index" : 0,
"DeviceSpecificContext" : null,
"Value" : 1043.0
}
ElasticSearch Query:
{"search_type":"query_then_fetch","ignore_unavailable":true,"index":["telemetry-2022.04.22","telemetry-2022.04.27","telemetry-2022.04.28","telemetry-2022.04.29","telemetry-2022.04.30","telemetry-2022.05.01","telemetry-2022.05.02","telemetry-2022.05.03"]}
{"size":0,"query":{"bool":{"filter":[{"range":{"Timestamp":{"gte":1650663853916,"lte":1651602719334,"format":"epoch_millis"}}},{"query_string":{"analyze_wildcard":true,"query":"MessageId = CrayTelemetry.Power and Location = (\"x9000c1s0b0\" OR \"x9000c1s0b1\" OR \"x9000c1s1b0\" OR \"x9000c1s1b1\" OR \"x9000c1s2b0\" OR \"x9000c1s2b1\" OR \"x9000c1s4b0\" OR \"x9000c1s4b1\")"}}]}},"aggs":{"2":{"date_histogram":{"interval":"1m","field":"Timestamp","min_doc_count":1,"extended_bounds":{"min":1650663853916,"max":1651602719334},"format":"epoch_millis"},"aggs":{"3":{"avg":{"field":"Value"}}}}}}