“There’s an expectation in our industry that if a company holds tens of terabytes of data, they must have a sizable DBA team to manage it. With TimescaleDB, that’s simply not the case. We have one person working part-time on database management, that’s it! TimescaleDB allows us to be extremely efficient when it comes to maintenance and operations.”— Pawel Suwala, CTO, k6
k6’s load testing SaaS platform (cloud and open source) is used by thousands of developers around the world. As their client base continues to grow, so does the amount of tests they need to run – and they need them to run simultaneously and quickly. Just one user’s configuration test could produce gigabytes of data every minute, all of which is time-series data. At scale, this number is massive.
The k6 Cloud needed a database that could support hundreds of clients generating a huge amount of time-series data, so that they could deliver test results rapidly and enable customers to monitor and analyze how their metrics change over time.
Originally, the team used Cassandra to store the load test results and associated metrics that they surface for their users. But, as they looked ahead to the next generation of their application, they knew they needed to address certain Cassandra limitations, like its limited query capabilities and difficulty storing all of their data.
As long-time PostgreSQL users (and fans), the k6 team was instantly drawn to TimescaleDB, since it’s built on Postgres - a stable and reliable technology, expands on its capabilities, and supports full SQL. After evaluating many available options and doing their due diligence, TimescaleDB won the selection race.
They’ve been using TimescaleDB for over two years, and, currently, during a period of high demand, k6 Cloud will
INSERTnearly 200K rows into their master TimescaleDB per second (loads are particularly high before Black Friday and Cyber Monday, given their popularity with eCommerce brands). At the same time, they replicate that data to a TimescaleDB replica server and
SELECTat least 200K rows per second. Thanks to TimescaleDB’s purpose-built time-series architecture, k6 Cloud’s insert rates remain stable during intense periods — which will only become more frequent as they continue to gain popularity with developers worldwide.
Since TimescaleDB’s architecture is based on hypertables and chunks, k6 saw a large performance boost compared to vanilla PostgreSQL. Their current setup configures hypertables into 6-hour chunks, and they hold entire chunks in memory.
To put this in perspective, the k6 Cloud stores between 25-30 Tebibytes (TiB) or 27-32 Terabytes (TB) in TimescaleDB. On average, they ingest around 100 Gibibyte (GiB) or 107 Gigabytes (GB) of compressed disk storage every day – and during periods of high demand, triples to 350GiB. They retain data based on the needs of their customers, but on average they delete between 50-100GiB every 24 hours.
In addition to memory and storage-based improvements, with TimescaleDB, the team can perform correlations, aggregate group-bys on URLs, join time-series with metadata, and use full SQL to work across datasets. As a result, k6 gets better query performance, runs more complex queries, and can focus on their goal: helping their customers catch performance issues before they make it into production.
“Using an established technology with a vast amount of learning resources on the web helps to shorten the time for onboarding new developers into the codebase and infrastructure.
Hiring becomes much easier because the majority of Developers and DevOps Engineers have used PostgreSQL or SQL queries in the past.”
With the current growth of the k6 Cloud, the team expects to double the size of their main TimescaleDB storage within the next 12 months (this means that they could be hitting over 60TB of storage). k6 is confident in TimescaleDB’s ability to scale with them - and their ever-increasing data volume - as their business continues to rapidly expand.
Since TimescaleDB runs seamlessly behind the scenes, they’re free to concentrate on growing their user base and providing the superior load testing and reporting experience developers expect and need.
And, when it comes time to grow the team, they know they can continue to recruit and onboard new members quickly, since their stack builds on a well-established technology.
It’s full steam ahead for the team at k6!