Timescale flies when you’re having fun: 2021 in review
Welcome to the last blog post of the year!
It has become a Timescale tradition to celebrate #12DaysofTimescale, where we share our favorite content pieces published the previous year. But 2021 was such an awesome year for us in Team Timescale, that we didn’t want to end it there.
Today, we are celebrating the accomplishments that made us proud in 2021.
In 2021, Timescale delivered substantial improvements to our category-defining relational database for time-series. Early in our history, Timescale made the decision to build on PostgreSQL, the world’s fastest-growing database. Today, that foundation enables us to move quickly and deliver the functionality and dependability that developers demand when building data-intensive applications.
We shipped Timescale, a cloud database that makes it incredibly easy for developers to build and deploy data-intensive applications. We advanced the state-of-the-art in SQL with new hyperfunctions and function pipelines, tools that significantly improve developer productivity by making it easier to manipulate and analyze time-series data using the SQL you know and love. We made significant investments in the observability business by embracing OpenTelemetry and making Promscale your unified store for metrics, logs, and traces.
4,000,000 TimescaleDB instances are active today.
Developers count on us every day. We are proud of our contributions to the developer community and look forward to next year when we can continue to delight our customers as they build and deploy data-intensive applications.
Read on for a summary of everything we’ve shipped this year.
If you’re new to Timescale, check out Timescale; you can create a free account to get started (100% free for 30 days and no credit card required).
Once you are using Timescale, join the Timescale Community and ask us any questions about time-series data, TimescaleDB, PostgreSQL, and more. And, for those who share our mission of serving developers worldwide 🌏 and want to join our fully remote, global team, we are hiring broadly across many roles!
In October, we launched the new and improved Timescale. Timescale provides a balance between a familiar developer platform and a flexible database for modern applications while retaining the ease and scalability of modern cloud services.
Timescale represents our vision for a modern data platform that combines the ease of use expected from a cloud service with the flexibility that developers need. We call this the database cloud. The database cloud provides developers with a familiar interface so that they can bring their existing skills to solve the complex problems of today, be it in financial services, web-scale infrastructure, IoT, or more. The database cloud gives developers the ability to dig deep into the internals of the database and truly understand how their data is managed. This year, we shipped many new features for Timescale, including:
- Multi-region support
- One-click support for multi-node services
- VPC peering
- Storage autoscaling with zero downtime and configurable limits
- Explorer dashboard for an easy, visual in-console interface
- Automated, zero-downtime upgrades during maintenance windows
- Point-in-time recovery via automated, continuous incremental backups
- Guided demo with pre-loaded data for first-time users
- PDF invoices
- Additional emails for billing
- One-click database forking
New features in TimescaleDB 🐯
At Timescale, we focus every day on delighting our customers with powerful and easy-to-use features. In 2021, we delivered 13 TimescaleDB releases, including:
- Support for PG14. We want TimescaleDB users to benefit from the latest improvements introduced by the Postgres community, so we included support for PostgreSQL 14 only one month after its release.
- Schema modifications and inserts in compressed hypertables. In TimescaleDB 2.3 and above, you can directly insert data into compressed hypertables (vs decompressing, inserting, and then re-compressing chunks manually).
- Drastic performance optimizations through Skip Scan. We implemented a technique that gave users 26x-8000x faster
DISTINCTqueries, not only in TimescaleDB hypertables but in their regular PostgreSQL tables as well.
- Experimental schema. In TimescaleDB 2.4 we introduced our experimental schema, where we develop and ship features at an even faster pace than we normally do - to later “graduate” these experimental features once they reach full maturity for normal production usage.
- More flexibility in
time_bucket_ng(). A good example of the use of the experimental schema is
time_bucket_ng(), which expands the popular
time_bucket()to also support buckets by months, years, and timezones.
Apart from these major feature releases, we improved the functionality of distributed hypertables, adding support for advanced TimescaleDB features like compression policies, continuous aggregates, triggers, gap-fill, and Skip Scan. We also added advanced functionality in regards to multi-node operations, like the ability to move and copy chunks between data nodes, distributed restore points for backup/restore, and managing distributed objects and DDL.
And… We closed 728 issues in Github! Thank you so much to everyone who reported bugs and who helped us improve TimescaleDB: this wouldn’t be possible without you! We also love hearing which features you would like to see next (closing those issues feels especially satisfying). Please, keep sharing your suggestions and feedback with us!
In July, we launched TimescaleDB hyperfunctions, a series of SQL functions within TimescaleDB that make it easier to manipulate and analyze time-series data in PostgreSQL with fewer lines of code. You can use hyperfunctions to calculate percentile approximations of data, compute time-weighted averages, downsample and smooth data, and perform faster
COUNT DISTINCT queries using approximations. Moreover, hyperfunctions are “easy” to use: you call a hyperfunction using the same SQL syntax you know and love.
We’ve already delivered all these hyperfunctions:
- average / average_y / average_x
- corr (correlation coefficient)
- determination_coeff (R squared)
- kurtosis / kurtosis_y / kurtosis_x
- skewness / skewness_y / skewness_x
- stddev / stddev_y / stddev_x
- sum/ sum_y / sum_x
- variance / variance_y / variance_x
- counter_agg (point form)
And in October, we launched function pipelines, a new capability that introduces functional programming concepts inside PostgreSQL (and SQL) using custom operators. Function pipelines radically improve the developer ergonomics of analyzing data in PostgreSQL and SQL, by applying principles from functional programming and popular tools like Python’s Pandas and PromQL.
SELECT device id, sum(abs_delta) as volatility FROM ( SELECT device_id, abs(val - lag(val) OVER (PARTITION BY device_id ORDER BY ts)) as abs_delta FROM measurements WHERE ts >= now() - '1 day'::interval) calc_delta GROUP BY device_id;
Pop quiz: What does this query do?
Even if you are a SQL expert, queries like this can be quite difficult to read - and even harder to express. Complex data analysis in SQL can be hard.
Function pipelines let you express that same query like this:
SELECT device_id, timevector(ts, val) -> sort() -> delta() -> abs() -> sum() as volatility FROM measurements WHERE ts >= now() - '1 day'::interval GROUP BY device_id;
Function pipelines: Building functional programming into PostgreSQL using custom operators
One Promscale for all 🔍
In October, we made major improvements to Promscale. Promscale started as a data store for Prometheus metrics, but quickly matured in 2021 to become a unified backend for observability data - supporting both Prometheus metrics and OpenTelemetry traces. And this is only the beginning (stay tuned!)
Apart from the support for OpenTelemetry tracing in Promscale, we released many tools and improvements related to Promscale and observability, including:
- Tobs, a tool that allows you to deploy a full observability stack for Kubernetes in one command. The stack includes Promscale, TimescaleDB, Prometheus, OpenTelemetry Operator, Jaeger, and Grafana (including pre-built dashboards for Kubernetes)
- Prom-migrator, a universal migration tool for Prometheus data
- Multi-tenancy in Promscale
- Support for OpenMetrics exemplars, continuous aggregates, Prometheus metadata, high availability setups, PG14, text + JSON ingest API, and multi-node
- Innumerable bug and performance fixes
All of us at Timescale are so proud to work for you every day ❤️ We hope the year ahead brings you great joy and prosperity. Please, stay safe during this holiday season!
We wish you happy holidays and a happy New Year.
On to 2022!