Aug 10, 2023
This is an installment of our “Community Member Spotlight” series, where we invite our customers to share their work, shining a light on their success and inspiring others with new ways to use technology to solve problems.
In this edition, Rob Robinett and Gwyn Griffiths, the creators of WsprDaemon, join us to share the work they’re doing to allow amateur radio enthusiasts to analyze transmission data and understand trends, be it their own personal noise levels or much larger space weather patterns.
Amateur radio is a hobby for some three million people worldwide (see “What is Amateur Radio?” to learn more) and its technical scope is vast, examples include: designing and building satellites, devising bandwidth-efficient data communications protocols, and creating novel, low noise antennas for use in urban locations.
Our project, WsprDaemon, focuses on amateurs who use the (amateur-developed!) open-source Weak Signal Propagation Reporter (WSPR): a protocol that uses low-power radio transmissions that probe the earth’s ionosphere to identify radio propagation paths and provide insights on space weather. On a typical day, 2,500 amateurs may report 2.7 million WSPR “spots” to the wsprnet database, whose webpage interface allows simple queries on data collected in the last two weeks.
Radio signals that end up in the WsprDaemon TimescaleDB database may be received on a wide variety of antennas, from the 94-foot tower-supported multiband array in Northern Utah pictured above, to more modest 3-foot installations that you may see in many suburban or urban locations.
We have a small, two-member volunteer core team, and we’re supported by a dozen or so beta testers and radio specialists (we typically have people from six countries on a weekly Zoom meeting). Rob Robinett, based in Berkeley California, is CEO of TV equipment manufacturer Mystic Video and he’s founded a series of Silicon Valley startups. He recently “rediscovered” amateur radio - after an absence of more than 40 years - and he's applying his software expertise to developing systems that measure short wave radio transmission conditions.
Gwyn Griffiths, based in Southampton, UK, returned to amateur radio after retiring from a career as an ocean technologist, where he worked with sensors and data from ships, undersea moorings, and robotics underwater vehicles. Gwyn focuses on the TimescaleDB components, devises Grafana dashboards to help inspire WsprDaemon users to create their own, and writes our step-by-step guides (check them out here).
WsprDaemon ingests data from the wsprnet database into TimescaleDB, allowing users to access historical data (remember, the wsprnet database shows online data from the last two weeks, and our project allows users to use older data for more robust analysis) and enabling a rich range of SQL queries.
Additionally, TimescaleDB facilitates our Grafana dashboards; seeing a month of “spots” gives users a far deeper understanding about their own data, enables comparisons with other datasets, and provides a platform for further experimentation and creative graphics.
Our TimescaleDB application caters to a wide spectrum of radio amateurs, from casual hobbyists to third-party developers:
The key measurement for a WSPR receiver is the signal-to-noise ratio (SNR): how strong an incoming signal is compared with the background noise. But, there is also vital metadata, including the location of the receiver and transmitter, the operating radio frequency, and most critically - time. On average, our database takes in about 4,000 “sets” of this data from a given transmitter, typically 400kB, every two minutes.
This below shows an example of SNR from three transmitters, in New York State, Italy, and Virginia.
The seven-day time-series data shown in this dashboard example provides rich information for its receiver, station KD2OM in New York State:
While SNR is the main measurement, our users are also interested in aggregate metadata functions, which provide an overview of daily changes in the ionosphere. Our project allows them to run these more complex queries, and we bring in complementary public domain data, such as the case below where we pull in data from the US Space Weather Prediction Center.
In this example, the top panel of the Grafana dashboard uses the WsprDaemon dataset to display a simple count of the “spots” in each 10 minute time bucket with, on the second y-axis, a measure of the planetary geomagnetic disturbance index (kp) from the US Space Weather Prediction Center. In 2020, we’re at the minimum of the 11-year sunspot cycle, so our current space weather is generally very quiet, but we’re anticipating disturbances - as well as more WSPR spots - as the sun becomes more active over the next four years.
The second panel is a heat map that shows the variation in distance between the receiver in Belgium and the transmitters it’s heard over time.
The third panel shows the direction of arrival at the receiver, while the bottom panel helps the user interpret all of this data, showing the local noise level and instances of overload.
Editor’s note: see our Grafana Series Override blog post to learn how (and why) to use two y-axes to more accurately plot your data and understand trends.
As evidenced above, a big advantage for our users is our ability to bring disparate datasets together into one database, with one graphical visualisation tool.
Our initial need was for a database and visualisation tool for radio noise measurements from a handful of stations. A colleague suggested Influx and Grafana, and kindly set up a prototype for us. We were hooked.
We sought to expand to cover a larger range of data sets from several thousand sources. The Influx documentation was great, and we had an extended application running quickly. Initially, our query time performance was excellent, but, as we accumulated weeks of data we hit the cardinality issue. Query times became unacceptably long, and we looked for an alternative time-series database solution.
We quickly came across an objective article on how to solve the cardinality problem that led us to adopt TimescaleDB. The biggest factor in choosing TimescaleDB was that it solved our cardinality problem, but there were also “nice to have” features, such as the easy-to-use tool to migrate our data from Influx and the underlying use of PostgreSQL. But, we did miss Influx’s comprehensive single-source documentation.
Editor’s Note: Because we think it’s important to remain balanced and let our community members’ voice shine through, we don’t edit mentions of alternative technologies (favorable or unfavorable 🙂).
Our initial TimescaleDB implementation is on a DigitalOcean Droplet (2 cores, 4GB memory 100GB SSD disk), but we are moving to our own 16 core, 192GB memory Dell server and a back-up (we’re evaluating query performance as our user base grows).
As noted above, the way TimescaleDB has solved the issue of cardinality was a big selling point for us, and it’s what makes the WsprDaemon site performant for our users.
We use bash Linux shell scripts with Python to gather the data that populates our database tables. We find that batch upload using psycopg2.extras.execute_batch works well for us, and our users use a variety of methods to access WsprDaemon, including Node.js and psql via its command line interface.
We already make extensive use of Grafana dashboards, and we expect to expand our capabilities - adding global map panels is just one example. But, even after extensive searching, it’s not always straightforward or obvious how to obtain the best, streamlined end result.
For example, creating an animation that shows the global geographic distribution of receivers and transmitters by hour requires us to export data to CSV using psql, import the file into Octave, generate maps on an azimuthal equidistant projection, save these map files as PNG, and then import into ImageJ to generate an AVI file.
Editor’s Note: To learn more about building Grafana visualizations with TimescaleDB, check out our step-by-step tutorials, blogs, and “Guide to Grafana” webinar series.
Our future path includes collaboration, both with others in the global amateur radio community and more data sources. We continually learn about people who have neat solutions for handling and visualising data, and, by sharing knowledge and experience, we can collectively grow and improve the tools we offer this great community. We’re keen to expand our connections to other third party data sources, such as space weather, to help our users better interpret their results.
We’re non-professional database users, so we only feel qualified to speak to others with a similar minimal level of prior familiarity.
As you evaluate time-series database options, of course, read independent comparisons and the links they provide, but also look carefully at insightful, fair-minded comparisons from TimescaleDB, e.g., on SQL vs Flux. Try to assess the advantages of different approaches for your application, current and future skill sets, and requirements.
We believe that data analytics for radio amateurs is in its infancy. We’re detailing our approach and experience with TimescaleDB and Grafana in a paper at the 39th gathering of the Digital Communications Conference in September 2020 (for the first time, the Conference will be completely virtual, which is likely to enable a larger-than-usual participation from around the world). We’ll feature some nice examples of how self inner joins help pull out features and trends from comparisons, as well as many other capabilities of interest to our users.
We’d like to thank Rob & Gwyn for sharing their story, as well as for their work to create open-source, widely distributed queries, graphs, and tools. Their dedication to making transmission data accessible and consumable for the global amateur radio community is yet another testament to how technology and creativity combine to breed amazing solutions.
We’re always keen to feature new community projects and stories on our blog. If you have a story or project you’d like to share, reach out on Slack (@lacey butler), and we’ll go from there.
Additionally, if you’re looking for more ways to get involved and show your expertise, check out the Timescale Heroes program.