Grow Worry-Free: Storage Autoscaling on Timescale Cloud

Time-series data is everywhere, and it drives decision-making in every industry. Time-series data collectively represents how a system, process, or behavior changes over time. Understanding these changes helps us to solve complex problems across numerous industries, including observability, financial services, Internet of Things, and even professional American football. Analyzing this data across the time dimension enables developers to understand what is happening right now, how that is changing, and why that is changing.

And time-series is also relentless, often at high rates and volumes. To help developers keep up, we built Timescale Cloud: a database cloud for relational and time-series workloads, built on PostgreSQL, and architected around our vision of a modern cloud service:  easy, scalable, familiar, and flexible.

We recently announced larger disk plans of up to 16TB per node on Timescale Cloud, enabling developers to effectively store and analyze 200+ TB of data (thanks to TimescaleDB’s hybrid row-column native compression, with compression rates of 94-97%).

Today, to continue with this theme of delivering a worry free platform for time-series that scales with your data needs, we’re announcing the release of storage autoscaling on Timescale Cloud.

What is storage autoscaling?

Storage autoscaling helps you keep your database operations running smoothly and avoid the hassle of dealing with full disks, by automatically increasing your database storage, with user-defined limits to keep costs under control (and avoid unexpected bills). Storage autoscaling (and manual disk resizing) happens fully online, without service downtime.

Storage autoscaling is on by default. You now create a database service on Timescale Cloud with a single click. Your service starts small, but the platform continuously monitors your storage consumption and automatically increases your capacity when needed. From 10GB up to 16TB (or 200+TB of effective storage). Easy to use and highly cost effective.

But Timescale Cloud also empowers developers with greater control. The platform allows developers to easily set configurable limits for auto-scaling (or turn it on or off on a per-service basis). When enabled with limits (which can be modified at any time), your storage will automatically scale up to your specified limit.  And then other full disk protection mechanisms will kick in if and when you hit your limits.

While managed database services save you the hassle of administering your own database, storage autoscaling helps you save even more time by avoiding dealing with operational blockages caused by full disks, and at the same time ensuring that your users and customers experience uninterrupted service. Furthermore, storage autoscaling gives you full control over your future costs, with transparent pricing and user-defined limits to ensure that you never scale unexpectedly beyond your budget.

In addition to automatically scaling up storage as required, Timescale Cloud also includes full disk protection mechanisms, which puts the database into read-only mode once the autoscaling limit is reached and the database is full, ensuring that the database doesn’t crash from new data attempting to be inserted into it (read more here about Timescale Cloud’s storage safety mechanisms.)  It’s another example of how Timescale Cloud is focused on providing a worry-free platform for developers.

Learn more about storage autoscaling

Read on for more about what storage autoscaling is, the problem it solves, and why and how to use storage autoscaling in Timescale Cloud.

For existing Timescale Cloud users, storage autoscaling is enabled by default on all new services created, just visit the Autoscaling options under the Operations tab in your console.

If you’re new to Timescale, create a free account to get started with a fully-managed Timescale Cloud service (100% free for 30 days, no credit card required).

Once you are using TimescaleDB, please join the Timescale community and ask any questions you may have about time-series data, databases, and more.

Special thanks to Ivan Tolstosheyev and the entire Timescale Cloud team behind the development of this feature 🙏.

Full Disks Are a Pain

We’ve all experienced it. You’re trying to take that latest puppy video or photo to share in your group text, but your storage is full. You’re left trying to figure out which caches to clear or other pictures to delete.

Nobody likes a full disk, including your database and the operating system it uses. You try to insert more data into your database, and the “write-ahead-log” (WAL) it uses to ensure all data is reliably and atomically written has no place to write its log. Try to add an index, and there’s no place on disk to store the index pages. And, even if you don’t directly write any new data to the database, things are happening (or, perhaps more accurately, not happening) in the background. Temp files can’t be written. File system blocks can't be allocated. Unexpected things go wrong.

As a database cloud that prioritizes ease and scale, Timescale Cloud provides built-in safety mechanisms to deal with these issues on your TimescaleDB services. Timescale Cloud will alert you whenever you approach storage limitations on your account, put your service in a read-only state so that your data is not lost, and give you an opportunity to configure your service so that you can resume collecting everything that matters. Many of these mechanisms are transparent and “just work” (as one example, Timescale Cloud employs system balloon files for an additional layer of “defense in depth” to full disks) and you can read more about the existing measure in our blog post announcing automated disk management.

But today, we wanted to share a bit more about our new capabilities for storage autoscaling and why we built them.

There is plenty of friction involved in triaging a database with a full disk (even with protections in place to ensure a database doesn’t crash, like putting it into read-only mode). It requires manual intervention from developers in order to provision more storage or perhaps to move, delete or compress data to save space. This intervention at worst takes away from engineering time that could be used for feature development, or at a minimum is just plain inconvenient.

Moreover, in the case you choose to provision more storage, it’s difficult to know exactly how much storage you’ll need with most DBaaS or serverless solutions. If you choose a disk that’s too small, you run the risk of having to manually intervene again in the near future, but if you choose a disk size that’s too big you end up paying for more capacity than you actually need as you wait for your disk to fill up.

Storage autoscaling on Timescale Cloud was designed to solve these problems, by removing the pain of dealing with full disks and minimizing the time that developers spend worrying about their database.

Enter Storage Autoscaling

How it works

With storage autoscaling on Timescale Cloud, you can start with as little storage as you need but set an autoscaling limit to give you the peace of mind that you’ll be able to scale up smoothly to storage levels you may need in the future.

With storage autoscaling enabled, when a service hits 85% of the current storage capacity, the storage plan will autoscale to the next available plan. (For example, if you’re on the 25GB storage plan, you’ll be autoscaled to the 50GB. And if you’re on the 9TB plan, you’ll be autoscaled to the 10TB plan.)

This means that you’ll never pay for more than the storage plan your service is currently using and only pay for larger storage plans when you autoscale to that tier. You can start at 10GB and automatically scale up to 16TB as your workload increases over time with no manual intervention. Moreover,  storage autoscaling (and manual disk resizing) happens fully online, without service downtime.

Autoscaling limits

You set an autoscaling limit, which is the maximum disk capacity that your service’s storage will reach with autoscaling enabled. This gives you full control over your future costs and helps keep your operations running smoothly with predictable, transparent pricing.

This saves you from potentially nasty billing surprises that many users have experienced on serverless data platforms, where their services automatically scale opaquely without any insight into limits or why their costs increased.

What can you do if your limit is reached?

Once your autoscaling limit has been reached, your disks enter “read-only” mode, meaning data can no longer be inserted. From here, you’ll be notified to take action. You have three options: first, you can reduce disk usage by compressing data and altering your native compression settings, second, you can enable data retention policies or manually drop chunks, or third, you can update your autoscaling limit to a higher tier. You can alter your autoscaling configuration anytime under your service’s operations tab.

Get Started Today

For existing Timescale Cloud users, storage autoscaling is enabled by default on all new services created, just visit the Autoscaling options under the Operations tab in your console.

If you’re new to Timescale, create a free account to get started with a fully-managed Timescale Cloud service (100% free for 30 days, no credit card required).

Once you are using TimescaleDB, please join the Timescale community and ask any questions you may have about time-series data, databases, and more.